April 28, 2017 11:27 AM

Data Compression and the Complexity of Consciousness

Okay, so this is cool:

Neuroscientists stimulate the brain with brief pulses of energy and then record the echoes that bounce back. Dreamless sleep and general anaesthesia return simple echoes; brains in conscious states produce more complex patterns. Then comes a little inspiration from data compression:

Excitingly, we can now quantify the complexity of these echoes by working out how compressible they are, similar to how simple algorithms compress digital photos into JPEG files. The ability to do this represents a first step towards a "consciousness-meter" that is both practically useful and theoretically motivated.

This made me think of Chris Ford's StrangeLoop 2015 talk about using compression to understand music. Using compressibility as a proxy for complexity gives us a first opportunity to understand all sorts of phenomena about which we are collecting data. Kolmogorov complexity is a fun tool for programmers to wield.

The passage above is from an Aeon article on the study of consciousness. I found it an interesting read from beginning to end.


Posted by Eugene Wallingford | Permalink | Categories: Computing, Patterns, Software Development

April 15, 2017 10:39 AM

Science Seeks Regularity

A week or so ago I tweeted that Carver Mead was blowing my mind: an electron a mile long! I read about that idea in this Spectator interview that covers both Mead's personal life and his professional work in engineering. Well worth a read.

Mead is not satisfied with the current state of physics and biology, or at least with the incomplete theories that we seem to have accepted in lieu of a more coherent conceptual understanding of how the world works. Ultimately, he sides with Einstein in his belief that there is a more coherent explanation:

I think Einstein was being a scientist in the truest sense in his response to the Copenhagen interpretation. He said that none of us would be scientists if deep down we didn't believe there is a set of regularities in the operation of physical law. That is a matter of faith. It is not something anybody has proven, but none of us would be scientists if we didn't have that faith.

Like Einstein, Mead believes that unpredictability at the lowest levels of a system does not imply intrinsic uncertainty. We need a different view that brings regularities to the forefront of our theories.

I also like this line from near the end of the interview:

People don't even know where to put the decimal point.

Mead says this as part of his assurance that artificial intelligence is nowhere near the level of what even the fruit fly can do, let alone the human brain. A lot has happened in AI during fifteen years since this interview; a computer program even beats our best players in Go now. Still, there is so much that we don't understand and cannot replicate.

I wonder if Mead's "decimal point" aphorism also might apply, metaphorically, to his view of the areas of science in which we have settled for, or are currently stuck with, unsatisifying theories. Our mathematical models cover a lot of ground, decimal point-wise, but there is still a simpler, more coherent picture to see. Maybe, though, that is the engineer in Mead showing through.


Posted by Eugene Wallingford | Permalink | Categories: General

April 07, 2017 1:33 PM

Two Very Different Kinds of Student

The last sentence of each of these passages reminds me of some of the students over the years.

First this, from Paul Callaghan's The Word Chain Kata:

One common way to split problems is via the "generate and test" pattern, where one component suggests results and the other one discards invalid ones. (In my experience, undergrad programmers are experts in this pattern, although they tend to use it as a [software development] methodology--but that's another story.)

When some students learn to program for the first time, they start out by producing code that looks like something they have seen before, trying it out, and then tinkering with it until it works or until they become exasperated. (I always hope that they act on their exasperation by asking me for help, rather than by giving up.) These students usually understand little bits of the code locally, but they don't really understand the program or function as a whole. Yet, somehow, they find a way to make it work.

It's surprising how far some students can get in a course of study by programming this way. (That's why Callaghan calling the approach a methodology made me smile.) It's unfortunate, too, because eventually the approach hits a wall when problems and domains become more challenging. Or when they run into a course where they program in Racket, in which one misplaced parenthesis can cause an otherwise perfect piece of code to implode. Lisp-like languages do not provide a supportive environment for this kind of "programming by wandering around".

And then there's this, from Andy Hertzfeld's fun little story about the writing of the classic manual Inside Macintosh:

Pretty soon, I figured out that if Caroline had trouble understanding something, it probably meant that the design was flawed. On a number of occasions, I told her to come back tomorrow after she asked a penetrating question, and revised the API to fix the flaw that she had pointed out. I began to imagine her questions when I was coding something new, which made me work harder to get things clearer before I went over them with her.

In this story, Caroline is not a student, but a young new writer assigned to the Mac documentation team. Still, she reminds me of students who are as delightful to work with as generate-and-test programmers can be frustrating. These students pay attention. They ask good questions, ones that often challenge the unstated assumptions underlying what we have explained before. At first, this can seem frustrating to us teachers, because we have to formulate answers for things that should be obvious. But that's the point: they aren't obvious, at least not to everyone, and us thinking they are obvious is inhibiting our teaching.

Last semester, I had one team of students in my compilers class that embodied this attitude. They asked questions no one had ever bothered to ask me before. At first, I thought, "How can these guys not understand such basic material?" Like Hertzfeld, though, pretty soon I figured out that their questions were exposing holes or weaknesses in my lectures, examples, and sample code. I began to anticipate their questions as I prepared for class. Their questions helped me see ways to make my course better.

As along so many other dimensions, part of the challenge in teaching CS is the wide variation in the way students study, learn, and approach their courses. It is also a part of the great fun of teaching, especially when you encounter the Carolines and Averys who push me to get better.


Posted by Eugene Wallingford | Permalink | Categories: Software Development, Teaching and Learning

April 02, 2017 12:02 PM

Reading an Interview with John McPhee Again, for the First Time

"This weekend I enjoyed Peter Hessler's interview of McPhee in The Paris Review, John McPhee, The Art of Nonfiction No. 3."

That's a direct quote from this blog. Don't remember it? I don't blame you; neither do I. I do remember blogging about McPhee back when, but as I read the same Paris Review piece again last Sunday and this, I had no recollection of reading it before, no sense of déjà vu at all.

Sometimes having a memory like mine is a blessing: I occasionally get to read something for the first time again. If you read my blog, then you get to read my first impressions for a second time.

I like this story that McPhee told about Bob Bingham, his editor at The New Yorker:

Bingham had been a writer-reporter at The Reporter magazine. So he comes to work at The New Yorker, to be a fact editor. Within the first two years there, he goes out to lunch with his old high-school friend Gore Vidal. And Gore says, What are you doing as an editor, Bobby? What happened to Bob Bingham the writer? And Bingham says, Well, I decided that I would rather be a first-rate editor than a second-rate writer. And Gore Vidal draws himself up and says, And what is wrong with a second-rate writer?

I can just hear the faux indignation in Vidal's voice.

McPhee talked a bit about his struggle over several years to write a series of books on geology, which had grown out of an idea for a one-shot "Talk of the Town" entry. The interviewer asked him if he ever thought about abandoning the topic and moving on to something he might enjoy more. McPhee said:

The funny thing is that you get to a certain point and you can't quit. Because I always worried: if you quit, you'll quit again. The only way out was to go forward, to learn your way and write your way out of it.

I know that feeling. Sometimes, I really do need to quit something and move on, but I always wonder whether quitting this time will make it easier to do next time. Because sometimes, I need to stick it out and, as McPhee says, learn my way out of the difficulty. I have no easy answers for knowing when quitting is the right thing to do.

Toward the end of the interview, the conversation turned to the course McPhee teaches at Princeton, once called "the literature of fact". The university first asked him to teach on short notice, over the Christmas break in 1974, and he accepted immediately. Not everyone thought it was a good idea:

One of my dear friends, an English teacher at Deerfield, told me: Do not do this. He said, Teachers are a dime a dozen -- writers aren't. But my guess is that I've been more productive as a writer since I started teaching than I would have been if I hadn't taught. In the overall crop rotation, it's a complementary job: I'm looking at other people's writing, and the pressure's not on me to do it myself. But then I go back quite fresh.

I know a lot of academics who feel this way. Then again, it's a lot easier to stay fresh in one's creative work if one has McPhee's teaching schedule, rather than a full load of courses:

My schedule is that I teach six months out of thirty-six, and good Lord, that leaves a lot of time for writing, right?

Indeed it does. Indeed it does.

On this reading of the interview, I marked only two passages that I wrote about last time. One came soon after the above response, on how interacting with students is its own reward. The other was a great line about the difference between mastering technique and having something to say: You demonstrated you know how to saddle a horse. Now go find the horse.

That said, I unconsciously channeled this line from McPhee just yesterday:

Writing teaches writing.

We had a recruitment event on campus, and I was meeting with a dozen or so prospective students and their entourages. We were talking about our curriculum, and I said a few words about our senior project courses. Students generally like these courses, even though they find them difficult. The students have never had to write a big program over the course of several months, and it's harder than it looks. The people who hire our graduates like these courses, too, because they know that these courses are places where students really begin to learn to program.

In the course of my remarks, I said something to the effect, "You can learn a lot about programming in classes where you study languages and techniques and theory, but ultimately you learn to write software by writing software. That's what the project courses are all about." There were a couple of experienced programmers in the audience, and they were all nodding their heads. They know McPhee is right.


Posted by Eugene Wallingford | Permalink | Categories: General, Software Development, Teaching and Learning

March 29, 2017 4:16 PM

Working Through A Problem Manually

This week, I have been enjoying Eli Bendersky's two-article series "Adventures in JIT Compilation":

Next I'll follow his suggestion and read the shorter How to JIT - An Introduction.

Bendersky is a good teacher, at least in the written form, and I am picking up a lot of ideas for my courses in programming languages and compilers. I recommend his articles and his code highly.

In Part 2, Bendersky says something that made me think of my students:

One of my guiding principles through the field of programming is that before diving into the possible solutions for a problem (for example, some library for doing X) it's worth working through the problem manually first (doing X by hand, without libraries). Grinding your teeth over issues for a while is the best way to appreciate what the shrinkwrapped solution/library does for you.

The presence or absence of this attitude is one of the crucial separators among CS students. Some students come into the program with this mindset already in place, and they are often the ones who advance most quickly in the early courses. Other students don't have this mindset, either by interest or by temperament. They prefer to solve problems quickly using canned libraries and simple patterns. These students are often quite productive, but they sometimes soon hit a wall in their learning. When a student rides along the surface of what they are told in class, never digging deeper, they tend to have a shallow knowledge of how things work in their own programs. Again, this can lead to a high level of productivity, but it also produces brittle knowledge. When something changes, or the material gets more difficult, they begin to struggle. A few of the students eventually develop new habits and move nicely into the group of students who likes to grind. The ones who don't make the transition continue to struggle and begin to enjoy their courses less.

There is a rather wide variation among undergrad CS students, both in their goals and in their preferred styles or working and learning. This variation is one of the challenges facing profs who hope to reaching the full spectrum of students in their classes. And helping students to develop new attitudes toward learning and doing is always a challenge.


Posted by Eugene Wallingford | Permalink | Categories: Computing, Teaching and Learning

March 22, 2017 4:50 PM

Part of the Fun of Programming

As I got ready for class yesterday morning, I decided to refactor a piece of code. No big deal, right? It turned out to be a bigger deal than I expected. That's part of the fun of programming.

The function in question is a lexical addresser for a little language we use as a specimen in my Programming Languages course. My students had been working on a design, and it was time for us to build a solution as a group. Looking at my code from the previous semester, I thought that changing the order of two cases would make for a better story in class. The cases are independent, so I swapped them and ran my tests.

The change broke my code. It turns out that the old "else" clause had been serving as a convenient catch-all and was only working correctly due to an error in another function. Swapping the cases exposed the error.

Ordinarily, this wouldn't be a big deal, either. I would simply fix the code and give my students a correct solution. Unfortunately, I had less than an hour before class, so I now found myself in a scramble to find the bug, fix it, and make the changes to my lecture notes that had motivated the refactor in the first place. Making changes like this under time pressure is rarely a good idea... I was tempted to revert to the previous version, teach class, and make the change after class. But I am a programmer, dogged and often foolhardy, so I pressed on. With a few minutes to spare, I closed the editor on my lecture notes and synced the files to my teaching machine. I was tired and still had a little nervous energy coursing through me, but I felt great. That's part of the fun of programming.

I will say this: Boy, was I glad to have my test suite! It was incomplete, of course, because I found an error in my program. But the tests I did have helped me to know that my bug fix had not broken something else unexpectedly. The error I found led to several new tests that make the test suite stronger.

This experience was fresh in my mind this morning when I read "Physics Was Paradise", an interview with Melissa Franklin, a distinguished experimental particle physicist at Harvard. At one point, Franklin mentioned taking her first physics course in high school. The interviewer asked if physics immediately stood out as something she would dedicate her life to. Franklin responded:

Physics is interesting, but it didn't all of a sudden grab me because introductory physics doesn't automatically grab people. At that time, I was still interested in being a writer or a philosopher.

I took my first programming class in high school and, while I liked it very much, it did not cause me to change my longstanding intention to major in architecture. After starting in the architecture program, I began to sense that, while I liked architecture and had much to learn from it, computer science was where my future lay. Maybe somewhere deep in my mind was memory of an experience like the one I had yesterday, battling a piece of code and coming out with a sense of accomplishment and a desire to do battle again. I didn't feel the same way when working on problems in my architecture courses.

Intro CS, like intro physics, doesn't always snatch people away from their goals and dreams. But if you enjoy the fun of programming, eventually it sneaks up on you.


Posted by Eugene Wallingford | Permalink | Categories: Computing, Personal, Software Development

March 18, 2017 11:42 AM

Hidden Figures, Douglas Engelbart Edition

At one point in this SOHP interview, Douglas Engelbart describes his work at the Ames Research Center after graduating from college. He was an electrical engineer, building and maintaining wind tunnels, paging systems, and other special electronics. Looking to make a connection between this job and his future work, the interviewer asked, "Did they have big computers running the various operations?" Engelbart said:

I'll tell you what a computer was in those days. It was an underpaid woman sitting there with a hand calculator, and they'd have rooms full of them, that's how they got their computing done. So you'd say, "What's your job?" "I'm a computer."

Later in the interview, Engelbart talks about how his experience working with radar in the Navy contributed to his idea for a symbol-manipulating system that could help people deal with complexity and urgency. He viewed the numeric calculations done by the human computers at Ames as being something different. Still, I wonder how much this model of parallel computing contributed to his ideas, if only implcitly.


Posted by Eugene Wallingford | Permalink | Categories: Computing

March 17, 2017 9:27 AM

What It Must Feel Like to be Ivan Sutherland

In The Victorian Internet, Tom Standage, says this about the unexpected challenge facing William Cooke and Samuel Morse, the inventors of the telegraph:

[They] had done the impossible and constructed working telegraphs. Surely the world would fall at their feet. Building the prototypes, however, turned out to be the easy part. Convincing people of their significance was far more of a challenge.

That must be what it feels like to be Ivan Sutherland. Or Alan Kay, for that matter.


Posted by Eugene Wallingford | Permalink | Categories: Computing

March 16, 2017 8:50 AM

Studying Code Is More Like Natural Science Than Reading

A key passage from Peter Seibel's 2014 essay, Code Is Not Literature:

But then it hit me. Code is not literature and we are not readers. Rather, interesting pieces of code are specimens and we are naturalists. So instead of trying to pick out a piece of code and reading it and then discussing it like a bunch of Comp Lit. grad students, I think a better model is for one of us to play the role of a 19th century naturalist returning from a trip to some exotic island to present to the local scientific society a discussion of the crazy beetles they found: "Look at the antenna on this monster! They look incredibly ungainly but the male of the species can use these to kill small frogs in whose carcass the females lay their eggs."
The point of such a presentation is to take a piece of code that the presenter has understood deeply and for them to help the audience understand the core ideas by pointing them out amidst the layers of evolutionary detritus (a.k.a. kludges) that are also part of almost all code. One reasonable approach might be to show the real code and then to show a stripped down reimplementation of just the key bits, kind of like a biologist staining a specimen to make various features easier to discern.

My scientist friends often like to joke that CS isn't science, even as they admire the work that computer scientists and programmers do. I think Seibel's essay expresses nicely one way in which studying software really is like what natural scientists do. True, programs are created by people; they don't exist in the world as we find it. (At least programs in the sense of code written by humans to run on a computer.) But they are created under conditions that look a lot more like biological evolution than, say, civil engineering.

As Hal Abelson says in the essay, most real programs end up containing a lot of stuff just to make it work in the complex environments in which they operate. The extraneous stuff enables the program to connect to external APIs and plug into existing frameworks and function properly in various settings. But the extraneous stuff is not the core idea of the program.

When we study code, we have to slash our way through the brush to find this core. When dealing with complex programs, this is not easy. The evidence of adaptation and accretion obscures everything we see. Many people do what Seibel does when they approach a new, hairy piece of code: they refactor it, decoding the meaning of the program and re-coding it in a structure that communicates their understanding in terms that express how they understand it. Who knows; the original program may well have looked like this simple core once, before it evolved strange appendages in order to adapt to the condition in which it needed to live.

The folks who helped to build the software patterns community recognized this. They accepted that every big program "in the wild" is complex and full of cruft. But they also asserted that we can study such programs and identify the recurring structures that enable complex software both to function as intended and to be open to change and growth at the hands of programmers.

One of the holy grails of software engineering is to find a way to express the core of a system in a clear way, segregating the extraneous stuff into modules that capture the key roles that each piece of cruft plays. Alas, our programs usually end up more like the human body: a mass of kludges that intertwine to do some very cool stuff just well enough to succeed in a harsh world.

And so: when we read code, we really do need to bring the mindset and skills of a scientist to our task. It's not like reading People magazine.


Posted by Eugene Wallingford | Permalink | Categories: Computing, Patterns, Software Development

March 10, 2017 2:51 PM

Reading Is A Profoundly Creative Act

This comes from Laura Miller, a book reviewers and essayist for Slate, in a Poets & Writers interview:

I also believe that reading is a profoundly creative act, that every act of reading is a collaboration between author and reader. I don't understand why more people aren't interested in this alchemy. It's such an act of grace to give someone else ten or fifteen hours out of your own irreplaceable life, and allow their voice, thoughts, and imaginings into your head.

I think this is true of all reading, whether fiction or nonfiction, literary or technical. I often hear CS profs tell their students to read "actively" by trying code out in an interpreter, asking continually what the author means, and otherwise engaging with the material. Students who do have a chance to experience what Miller describes: turning over a few hours of their irreplaceable lives to someone who understands a topic well, allow their voice, thoughts, and imaginings into their heads, and coming out on the other end of the experience with new thoughts -- and maybe even a new mind.


Posted by Eugene Wallingford | Permalink | Categories: General, Teaching and Learning