Session 18

Lexical Analysis:
Computing Lexical Addresses

CS 3540
Programming Languages and Paradigms

A Warm-Up Exercise

Annotate the variable references in this expression from our little language with their lexical addresses:

    (lambda (f)
      ((lambda (h)
         (lambda (n)
           ((f (h h)) n)))
       (lambda (h)
         (lambda (n)
           ((f (h h)) n)))))

Recall that each lambda expression creates a new block in which its formal parameters -- which are new local variables -- have meaning.

Solution. This expression has five lambda expressions, so it has five blocks. The outermost block contains two blocks, each of which contains another. The variable references n, h, and f are to variables declared in three different blocks, so their depths will be 0, 1, and 2, respectively. Each reference is to the first and only variable declared in that block, so its position is 0. The two (lambda (h) ...) expressions are identical, so the lexical addresses of the two sets of variables are identical. So:

    (lambda (f)
      ((lambda (h)
         (lambda (n)
           (((f : 2 0) ((h : 1 0) (h : 1 0))) (n : 0 0))))
       (lambda (h)
         (lambda (n)
           (((f : 2 0) ((h : 1 0) (h : 1 0))) (n : 0 0))))))

As we saw at the end of Session 17, once we have computed lexical addresses for all variable references, we no longer really need the variable references themselves. What's more, we don't even really need the variable declarations, either! They are sugar.

To remind yourselves of that, try Exercise 3 from last session's closing exercise. Reconstruct a legal Racket expression from this lexical address expression:

     (lambda 3                                    ;; Problem 3
        ((: 0 1) ((lambda 2
                     ((: 0 0) (: 1 0) (: 0 1)))
                  (: 0 2))))

Can we reuse any of the variable names from the outer block when we declare the inner block? Which ones?

Not every expression that looks like a lexical address expression can be "reverse engineered" in this way. Consider Exercise 2 from last session's close:

     (lambda (x)                                  ;; Problem 2
        (lambda (x)
           (: 1 0)))

Why isn't this legal? Fortunately, any time we start with a legal expression and compute lexical addresses for its variables, the result is a legal lexical address expression that can be reversed -- losing only the actual names used.

You may have noticed that the first expression above is a combinator. Indeed, it is the Y combinator, famous in programming language theory because it shows that it is possible to create even recursive local functions without a name. But how can a function call itself if it doesn't have a name? If you would like to find out, check out this optional reading and the associated code. Warning: this one may make your head hurt for a few minutes.

Now, onward.

Where Are We?

For the last couple of weeks, we have been devoting our attention to the idea of a syntactic abstraction, that is, a language feature that is not strictly necessary because we could conceivably do without it. We can do without such "syntactic sugar" because we have an equivalent way to express the same idea using other language features. For the same reason, the interpreter doesn't really need to understand the feature in order to determine the meaning of our program.

At this point, we have seen that a number of common language features are in fact syntactic abstractions:

This list is longer than it was last time. In your reading for today, you examined the idea of local recursive functions. The syntax is identical to a let expression, but its semantics are bit different. Look at factorial-aps.rkt to study the syntax and the behavior.

Like other local variables, local recursive functions are a syntactic abstraction. Unlike other local variables, they require something more complex than a simple rewrite to an application of a lambda expression.

In last session, we saw that variable names are not necessary: they are really syntactic sugar. I supported this claim by showing how a piece of code without explicit variable references can convey the same information as one that uses variable names.

Today, we will take a deeper journey into the idea of lexical addressing, which we first explored in Session 17, by writing a program that does lexical addressing. The program will make the idea clearer by putting it into a concrete program that you can run and modify. Writing the program will give you another opportunity to create a processor for a little language using Structural Recursion. Using Structural Recursion, this problem is tricky but manageable; without it, this problem might seem like a killer!

An Exercise in Lexical Addressing

Here is the BNF description of a new version of our little language, the small Racket-like language we've been using:

     <exp> ::= <varref>
             | (lambda (<var>*) <exp>)     ; 0 or more
             | (<exp>+)                    ; 1 or more -- app
             | (if <exp> <exp> <exp>)

Notice that this version of the language allows functions with zero or more parameters, and so applications with zero or more arguments. It also has variable references and a standard if-then-else expression.

Write a function named (lexical-address exp), where exp is any expression in our language. lexical-address returns an equivalent expression with every variable reference v replaced by its lexical address, in the form of a list (v : d p), as described Session 17.

Let's allow our expressions to contain free variables. In order to do that, we have to make an assumption similar to the one we make when we use a Racket interpreter: the free variables are bound at the "top level". We can imagine that the expression to process is contained within a lambda expression that binds references to any variables that occur free in the expression. This will account for system primitives such as eq? and cons.

For example:

    > (lexical-address 'a)
    (a : 0 0)

    > (lexical-address '(if a b c))
    (if (a : 0 0) (b : 0 1) (c : 0 2))

    > (lexical-address '(lambda (a b c)
                          (if (eq? b c)
                              ((lambda (c) (cons a c)) a)
                              b)) )
    (lambda (a b c)
       (if ((eq? : 1 0) (b : 0 1) (c : 0 2))
           ((lambda (c) ((cons : 2 1) (a : 1 0) (c : 0 0))) (a : 0 0))
           (b : 0 1)))

    > (lexical-address '(lambda (f)
                          ((lambda (h)
                             (lambda (n)
                               ((f (h h)) n)))
                           (lambda (h)
                             (lambda (n)
                               ((f (h h)) n))))))
    (lambda (f)
      ((lambda (h)
         (lambda (n)
           (((f : 2 0) ((h : 1 0) (h : 1 0))) (n : 0 0))))
       (lambda (h)
         (lambda (n)
           (((f : 2 0) ((h : 1 0) (h : 1 0))) (n : 0 0))))))

Here is some code to use in building your solution.

Today's .zip file contains all three of these items, along with a set ADT used by free-vars. You don't need these files to write lexical-address, but you should review their interfaces before beginning.

... time passes as students work and think and work. And then we reach the end of our time.

Now that you've worked on this for a while, you have begun to discover some of the secrets to building a solution, among them several things you already know:

Soon, if not yet, you may hit on these ideas:

With these ideas, you can build this function! Give it a serious try before our next session.

Wrap Up

Eugene Wallingford ..... ..... March 14, 2019