Artificial Brains Need Sleep Too – Desperate AI Researchers Discover Way to Stabilize Neuromorphic Processors

Neural Network Sleeping

Neural network sleeping illustration.

States that resemble sleep-like cycles in simulated neural networks quell the instability that comes with uninterrupted self-learning in artificial analogs of brains.

No one can say whether androids will dream of electric sheep, but they will almost certainly need periods of rest that offer benefits similar to those that sleep provides to living brains, according to new research from Los Alamos National Laboratory.

“We study spiking neural networks, which are systems that learn much as living brains do,” said Los Alamos National Laboratory computer scientist Yijing Watkins. “We were fascinated by the prospect of training a neuromorphic processor in a manner analogous to how humans and other biological systems learn from their environment during childhood development.”

Watkins and her research team found that the network simulations became unstable after continuous periods of unsupervised learning. When they exposed the networks to states that are analogous to the waves that living brains experience during sleep, stability was restored. “It was as though we were giving the neural networks the equivalent of a good night’s rest,” said Watkins.

Dreaming Droid

Sleep may be as vital to intelligent machines of the future as it is to us. Credit: Los Alamos National Laboratory

The discovery came about as the research team worked to develop neural networks that closely approximate how humans and other biological systems learn to see. The group initially struggled with stabilizing simulated neural networks undergoing unsupervised dictionary training, which involves classifying objects without having prior examples to compare them to.

“The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”

The researchers characterize the decision to expose the networks to an artificial analog of sleep as nearly a last-ditch effort to stabilize them. They experimented with various types of noise, roughly comparable to the static you might encounter between stations while tuning a radio. The best results came when they used waves of so-called Gaussian noise, which includes a wide range of frequencies and amplitudes. They hypothesize that the noise mimics the input received by biological neurons during slow-wave sleep. The results suggest that slow-wave sleep may act, in part, to ensure that cortical neurons maintain their stability and do not hallucinate.

The group’s next goal is to implement their algorithm on Intel’s Loihi neuromorphic chip. They hope allowing Loihi to sleep from time to time will enable it to stably process information from a silicon retina camera in real time. If the findings confirm the need for sleep in artificial brains, we can probably expect the same to be true of androids and other intelligent machines that may come about in the future.

Watkins will be presenting the research at the Women in Computer Vision Workshop on June 14 in Seattle.

Reference: “Using Sinusoidally-Modulated Noise as a Surrogate for Slow-Wave Sleep to Accomplish Stable Unsupervised Dictionary Learning in a Spike-Based Sparse Coding Model” by Yijing Watkins, Edward Kim, Andrew Sornborger and Garrett T. Kenyon, 14 June 2020, CVPR Women in Computer Vision Workshop, 2020-06-14 (Seattle, Washington, United States).
DOI: 10.1109/CVPRW50498.2020.00188

Funding: NNSA NA-22, ASC Beyond Moore Program.

7 Comments on "Artificial Brains Need Sleep Too – Desperate AI Researchers Discover Way to Stabilize Neuromorphic Processors"

  1. So, these are looking to be more and more “human”? So it follows – if it needs sleep, then might it also be subject to other similarities? Might it not become psychotic? Sadistic? Murderous? Racist? If it can love, then it surely can also hate. We see this possibility – yet we will still create this thing – its intellect and robot bodies far superior to ours – and think that we can live with it – even control it like a dog. I do not think so.

  2. You will shortly find that AL will, as it develops further, will need exercise,a companion, site seeing vacations,visits to extreme climates and more of what we humans need to have to be happy and to satisfy our curosity.
    Also, a great next experiment for AL is to put a virtual reality screen over the eyes with a program of your choice and monitor his reactions. You will be amazed!

  3. Thomas Kimlicka | June 11, 2020 at 12:16 am | Reply

    Data–on Star Trek didn’t need sleep!?

  4. I’m not sure why the author says we can expect android’s and other intelligent machines in the future to require sleep. The study co-author even made clear this only applies to using spiking neuromorphic processors, and does not apply to the typical platforms for artificial intelligence/machine learning. Is the the suggestion this platform will become the norm?

  5. From Wikipedia:

    A small explainer on Spiking Neural Networks:

    Forums
    New posts
    Members
    Marees Search
    New posts
    Find threads
    Watched
    Search forums
    Mark forums read
    Forums Real Life Stuff HardForum Tech News
    Why some Artificial Brains made of spiking neural networks Need to Sleep
    Thread starterMarees Start dateMay 19, 2020
    Jump to new
    Unwatch

    •••
    May 19, 2020
    Add bookmark
    #1
    Marees
    Limp Gawd
    JoinedSep 29, 2018
    Messages447
    “Our results make the surprising prediction that slow-wave sleep may be essential for any spiking neural network, or indeed any organism with a nervous system, to be able to learn from its environment.”
    Do they dream of electronic sheep, when they sleep ?

    https://www.insidescience.org/news/why-artificial-brains-need-sleep

    Scientists found that although Spiking Neural Networks could learn to identify the data it was trained to look for, when such training went uninterrupted long enough, its neurons began to continuously fire no matter what signals they received.

    Watkins recalled that “almost in desperation,” they tried having the simulation essentially undergo deep sleep. They exposed it to cycles of oscillating noise, roughly corresponding to the slow brain waves seen in deep sleep, which restored the simulation to stability. The researchers suggest this simulation of slow-wave sleep may help “prevent neurons from hallucinating the features they’re looking for in random noise,” Watkins said.

    In most artificial neural networks, a neuron’s output is a number that alters continuously as the input it is fed changes. This is roughly analogous to the number of signals a biological neuron might fire over a span of time.

    In contrast, in a spiking neural network, a neuron “spikes,” or generates an output signal, only after it receives a certain amount of input signals over a given time, more closely mimicking how real biological neurons behave.

    Since spiking neural networks only rarely fire spikes, they shuffle around much less data than typical artificial neural networks and in principle require much less power and communication bandwidth.

  6. Oops how do I edit a post to remove stuff that was copied in unnecessarily?

  7. Trying this again without the unnecessary stuff

    From Wikipedia for SPIKING Neural Network:

    In most artificial neural networks, a neuron’s output is a number that alters continuously as the input it is fed changes. This is roughly analogous to the number of signals a biological neuron might fire over a span of time.

    In contrast, in a spiking neural network, a neuron “spikes,” or generates an output signal, only after it receives a certain amount of input signals over a given time, more closely mimicking how real biological neurons behave.

    Since spiking neural networks only rarely fire spikes, they shuffle around much less data than typical artificial neural networks and in principle require much less power and communication bandwidth.

Leave a Reply to Marees Cancel reply

Email address is optional. If provided, your email will not be published or shared.