TITLE: StrangeLoop 4: Computing Like The Brain
AUTHOR: Eugene Wallingford
DATE: September 27, 2012 5:33 PM
Tuesday morning kicked off with a keynote address by
Jeff Hawkins entitled "Computing Like The Brain". Hawkins is
a company he co-founded in 2005, after having founding
the Redwood Neuroscience Institute and two companies most
technophiles will recognize: Palm and Handspring.
Hawkins said he has devoted his professional life to
understanding machine science. He recalls reading an article
by Francis Crick in Scientific American as a youth
and being inspired to study neuroscience. It was a data-rich,
theory-poor discipline, one crying out for abstractions to
unify our understanding of how the brain works from the mass
of data we were collecting. He says he dedicated life then
to discovering principles of how the brain works, especially
and to build computer systems that implement these principles.
The talk began with a primer on the neocortex, which can be
thought of as a predictive modeling system to controls human
intelligence. If we take into account all the components of
what we think of as our five senses, the brain has millions of
sensors that constantly stream data to the neocortex. Its job
is to build an on-line model from this streaming data. It
constantly predicts what he expects to receive next, detects
anomalies, updates itself, and produces actions. When the
neocortex updates, we learn.
On this view, the brain doesn't "compute". It is a
memory system. (I immediately thought of Roger Schank,
his views on AI, and case-based reasoning...) The brain is
really one memory algorithm operating over all of our sensory
inputs. The key elements of this memory system are:
Hawkins spoke briefly about hierarchy and sequence memory, but
he quickly moved into the idea of sparse distributed
representation (SDR). This can be contrasted to the dense,
localized memory of traditional computer systems. For example,
ASCII code consists of seven bits, all combinations of which
we use to represent a single character. Capital 'A' is 65, or
1000001; the digit '5' is 55, or 0110111. The coincidence of
'5' and 55 notwithstanding, the individual bits of an ASCII
code don't mean anything. Change one bit, and you get a
different character, sometimes a very different one.
An SDR uses a large number of bits, with only a few set to 1.
Hawkins said that typically only ~ 2% of the bits are "on".
Each bit in an SDR has specific meaning, one that has been
learned through memory updating, not assigned. He then
demonstrated several properties of an SDR, such as how it can
be used to detect similarities, how it can do "store-and-compare"
using only indices, and how it can perform remarkably well
using on a sampling of the indices. Associative look-up in the
brain's SDR produces surprisingly few errors, and those tend to
be related to the probe, corresponding to similar situations
The first takeaway point of the talk was this: Intelligent
systems of the future will be built using sparse distributed
At this point, my note-taking slowed. I am not a biologist, so
most of what Hawkins was describing lies far outside my area of
expertise. So I made a master note -- gotta get this guy's
book! -- and settled into more focused listening.
(It turns out that
a former student
recommended Hawkins's book,
to me a year or two ago. I should have listened to Allyn then
and read it!)
One phrase that made me smile later in the talk was the
semantic meaning of the wrongness. Knowing why something
is wrong, or how, is a huge step up on "just" being wrong.
Hawkins referred to this in particular as part of the subtlety
of making predictions.
To close, Hawkins offered some conjectures. He thinks that the
future of machine intelligence will depend on us developing
more and better theory to explain how the brain works,
especially in the areas of hierarchy and attention. The most
compelling implementation will be an embodied intelligence,
with embedded agents distributed across billions of sensors.
We need better hardware in order to create faster systems.
recall that the brain is more a memory systems than a
computation device, so better memory is as or more important
than better processors. Finally, we need to find a way to
increase the level connectivity among components. Neurons have
tens or hundreds of connections to other neurons, and these can
be grown or strengthened dynamically. Currently, our computer
chips are not good at this.
Where will breakthrough applications come from? He's not sure.
In the past, breakthrough applications of technologies have not
always been where we expected them.
I gotta read more. As a student of AI, I was never been all
that interested in neurobiology or even its implications for
my discipline. The cognitive level has always excited me more.
But Hawkins makes an interesting case that the underlying
technologies we need to reach the cognitive level will look
more like our brains than today's computers.
- a hierarchy of regions,
- sequence memory, and
- sparse distributed representation.