PL: Lecture #21  Tuesday, November 20th
(text file)

Scoping problems

You should be able to guess what’s this problem about. The basic problem of these macros is that they cannot be used reliably — the name that is produced by the macro can shadow a name that is in a completely different place, therefore destroying lexical scope. For example, in:

(let ((val 4))
  (orelse #f val))

the val in the macro shadows the use of this name in the above. One way to solve this is to write macros that look like this:

(define-macro (orelse <expr1> <expr2>)
  `(let ((%%!my*internal*var-do-not-use!%% ,<expr1>))
    (if %%!my*internal*var-do-not-use!%%
      %%!my*internal*var-do-not-use!%%
      ,<expr2>)))

or:

(define-macro (orelse <expr1> <expr2>)
  `(let ((i-am-using-orelse-so-i-should-not-use-this-name ,<expr1>))
    (if i-am-using-orelse-so-i-should-not-use-this-name
      i-am-using-orelse-so-i-should-not-use-this-name
      ,<expr2>)))

or (this is actually similar to using UUIDs):

(define-macro (orelse <expr1> <expr2>)
  `(let ((eli@barzilay.org/foo/bar/2002-02-02-10:22:22 ,<expr1>))
    (if eli@barzilay.org/foo/bar/2002-02-02-10:22:22
      eli@barzilay.org/foo/bar/2002-02-02-10:22:22
      ,<expr2>)))

Which is really not too good because such obscure variables tend to clobber each other too, in all kinds of unexpected ways.

Another way is to have a function that gives you a different variable name every time you call it:

(define-macro (orelse <expr1> <expr2>)
  (let ((temp (gensym)))
    `(let ((,temp ,<expr1>))
      (if ,temp
        ,temp
        ,<expr2>))))

but this is not safe either since there might still be clashes of these names (eg, if they’re using a counter that is specific to the current process, and you start a new process and load code that was generated earlier). The Lisp solution for this (which Racket’s gensym function implements as well) is to use uninterned symbols — symbols that have their own identity, much like strings, and even if two have the same name, they are not eq?.

Note also that there is the mirror side of this problem — what happens if we try this:

(let ([if 123]) (orelse #f #f))

? This leads to capture in the other direction — the code above shadows the if binding that the macro produces.

Some Schemes will allow something like

(define-macro (foo x)
  `(,mul-list ,x))

but this is a hack since the macro outputs something that is not a pure s-expression (and it cannot work for a syntactic keyword like if). Specifically, it is not possible to write the resulting expression (to a compiled file, for example).

We will ignore this for a moment.


Another problem — manageability of these transformations.

Quasiquotes gets us a long way, but it is still insufficient.

For example, lets write a Racket bind that uses lambda for binding. The transformation we now want is:

(bind ((var expr) ...)
  body)
-->
((lambda (var ...) body)
expr ...)

The code for this looks like this:

(define-macro (bind var-expr-list body)
  (cons (list 'lambda (map car var-expr-list) body)
        (map cadr var-expr-list)))

This already has a lot more pitfalls. There are lists and conses that you should be careful of, there are maps and there are cadrs that would be catastrophic if you use cars instead. The quasiquote syntax is a little more capable — you can write this:

(define-macro (bind var-expr-list body)
  `((lambda ,(map car var-expr-list) ,body)
    ,@(map cadr var-expr-list)))

where “,@” is similar to “,” but the unquoted expression should evaluate to a list that is spliced into its surrounding list (that is, its own parens are removed and it’s made into elements in the containing list). But this is still not as readable as the transformation you actually want, and worse, it is not checking that the input syntax is valid, which can lead to very confusing errors.

This is yet another problem — if there is an error in the resulting syntax, the error will be reported in terms of this result rather than the syntax of the code. There is no easy way to tell where these errors are coming from. For example, say that we make a common mistake: forget the “@” character in the above macro:

(define-macro (bind var-expr-list body)
  `((lambda ,(map car var-expr-list) ,body)
    ,(map cadr var-expr-list)))

Now, someone else (the client of this macro), tries to use it:

> (bind ((x 1) (y 2)) (+ x y))
procedure application: expected procedure,
given: 1; arguments were: 2

Yes? Now what? Debugging this is difficult, since in most cases it is not even clear that you were using a macro, and in any case the macro comes from code that you have no knowledge of and no control over. [The problem in this specific case is that the macro expands the code to:

((lambda (x y) (+ x y))
(1 2))

so Racket will to use 1 as a function and throw a runtime error.]

Adding error checking to the macro results in this code:

(define-macro (bind var-expr-list body)
  (if (andmap (lambda (var-expr)
                (and (list? var-expr)
                    (= 2 (length var-expr))
                    (symbol? (car var-expr))))
              var-expr-list)
    `((lambda ,(map car var-expr-list) ,body)
      ,@(map cadr var-expr-list))
    (error 'bind "bad syntax whaaaa!")))

Such checks are very important, yet writing this is extremely tedious.

Scheme (and Racket) Macros

Scheme, Racket included (and much extended), has a solution that is better than defmacro: it has define-syntax and syntax-rules. First of all, define-syntax is used to create the “magical connection” between user code and some macro transformation code that does some rewriting. This definition:

(define-syntax foo
  ...something...)

makes foo be a special syntax that, when used in the head of an expression, will lead to transforming the expression itself, where the result of this transformation is what gets used instead of the original expression. The “...something...” in this code fragment should be a transformation function — one that consumes the expression that is to be transformed, and returns the new expression to run.

Next, syntax-rules is used to create such a transformation in an easy way. The idea is that what we thought to be an informal specification of such rewrites, for example:

`let' can be defined as the following transformation:
  (let ((x v) ...) body ...)
  --> ((lambda (x ...) body ...) v ...)

and

`let*' is defined with two transformation rules:
1. (let* () body ...)
  --> (let () body ...)
2. (let* ((x1 v1) (x2 v2) ...) body ...)
  --> (let ((x1 v1)) (let* ((x2 v2) ...) body ...))

can actually be formalized by automatically creating a syntax transformation function from these rule specifications. (Note that this example has round parentheses so we don’t fall into the illusion that square brackets are different: the resulting transformation would be the same.) The main point is to view the left hand side as a pattern that can match some forms of syntax, and the right hand side as producing an output that can use some matched patterns.

syntax-rules is used with such rewrite specifications, and it produces the corresponding transformation function. For example, this:

(syntax-rules () ;*** ignore this "()" for now
  [(x y) (y x)])

evaluates to a function that is somewhat similar to:

(lambda (expr)
  (if (and (list? expr) (= 2 (length expr)))
    (list (second expr) (first expr))
    (error "bad syntax")))

but match is a little closer, since it uses similar input patterns:

(lambda (expr)
  (match expr
    [(list x y) (list y x)]
    [else (error "bad syntax")]))

Such transformations are used in a define-syntax expression to tie the transformer back into the compiler by hooking it on a specific keyword. You can now appreciate how all this work when you see how easy it is to define macros that are very tedious with define-macro. For example, the above bind:

(define-syntax bind
  (syntax-rules ()
    [(bind ((x v) ...) body ...)
    ((lambda (x ...) body ...) v ...)]))

and let* with its two rules:

(define-syntax let*
  (syntax-rules ()
    [(let* () body ...)
    (let () body ...)]
    [(let* ((x v) (xs vs) ...) body ...)
    (let ((x v)) (let* ((xs vs) ...) body ...))]))

These transformations are so convenient to follow, that Scheme specifications (and reference manuals) describe forms by specifying their definition. For example, the Scheme report, specifies let* as a “derived form”, and explains its semantics via this transformation.

The input patterns in these rules are similar to match patterns, and the output patterns assemble an s-expression using the matched parts in the input. For example:

(x y) --> (y x)

does the thing you expect it to do — matches a parenthesized form with two sub-forms, and produces a form with the two sub-forms swapped. The rules for “...” on the left side are similar to match, as we have seen many times, and on the right side it is used to splice a matched sequence into the resulting expression and it is required to use the ... for sequence-matched pattern variables. For example, here is a list of some patterns, and a description of how they match an input when used on the left side of a transformation rule and how they produce an output expression when they appear on the right side:

Some examples of transformations that would be very tedious to write code manually for:


This is solving the problems of easy code — no need for list, cons etc, not even for quasiquotes and tedious syntax massaging. But there were other problems. First, there was a problem of bad scope, one that was previously solved with a gensym:

(define-macro (orelse <expr1> <expr2>)
  (let ((temp (gensym)))
    `(let ((,temp ,<expr1>))
      (if ,temp ,temp ,<expr2>))))

Translating this to define-syntax and syntax-rules we get something simpler:

(define-syntax orelse
  (syntax-rules ()
    [(orelse <expr1> <expr2>)
    (let ((temp <expr1>))
      (if temp temp <expr2>))]))

Even simpler, Racket has a macro called define-syntax-rule that expands to a define-syntax combined with a syntax-rules — using it, we can write:

(define-syntax-rule (orelse <expr1> <expr2>)
  (let ((temp <expr1>))
    (if temp temp <expr2>)))

This looks like like a function — but you must remember that it is a transformation rule specification which is a very different beast, as we’ll see.

The main thing here is that Racket takes care of making bindings follow the lexical scope rules:

(let ([temp 4])
  (orelse #f temp))

works fine. In fact, it fully respects the scoping rules: there is no confusion between bindings that the macro introduces and bindings that are introduced where the macro is used. (Think about different colors for bindings introduced by the macro and other bindings.) It’s fine with many cases that are much harder to cope with otherwise (eg, cases where there is no gensym magic solution):

(let ([if +])
  (orelse 1 1))

(let ([if +])
  (if (orelse 1 1) 10 100)) ; two different `if's here

You can even use both:

(let ([if #f] [temp 4])
  (orelse if temp))

and use DrRacket’s macro debugger to see how the various bindings get colored differently.

define-macro advocates will claim that it is difficult to make a macro that intentionally plants an identifier. Think about a loop macro that has an abort that can be used inside its body. Or an if-it form that is like if, but makes it possible to use the condition’s value in the “then” branch as an it binding. It is possible with all Scheme macro systems to “break hygiene” in such ways, and we will later see how to do this in Racket. However, Racket also provides a better way to deal with such problems (think about it being always “bound to a syntax error”, but locally rebound in an if-it form).

Scheme macros are said to be hygienic — a term used to specify that they respect lexical scope. (All of this can get much more important in the presence of a module system, since you can write a module that provides transformations rules, not just values and functions.) There are several implementations of hygienic macro systems across Scheme implementations.


The way that Scheme implementations achieve hygiene in a macro system is by making it deal with more than just raw S-expressions. Roughly speaking, it deals with syntax objects that are sort of a wrapper structure around S-expression, carrying additional information. The important part of this information when it gets to dealing with hygiene is the “lexical scope” — which can roughly be described as having identifiers be represented as symbols plus a “color” which represents the scope. This way such systems can properly avoid confusing identifiers with the same name that come from different scopes.

There was also the problem of making debugging difficult, because a macro can introduce errors that are “coming out of nowhere”. In the implementation that we work with, this is solved by adding yet more information to these syntax objects — in addition to the underlying S-expression and the lexical scope, they also contain source location information. This allows Racket (and DrRacket) to locate the source of a specific syntax error, so locating the offending code is easy. DrRacket’s macro debugger heavily relies on this information to provide a very useful tool — since writing macros can easily become a hard job.

Finally, there was the problem of writing bad macros. For example, it is easy to forget that you’re dealing with a macro definition and write:

(define-syntax-rule (twice x) (+ x x))

just because you want to inline the addition — but in this case you end up duplicating the input expression which can have a disastrous effect. For example:

(twice (twice (twice (twice (twice (twice (twice (twice 1))))))))

expands to a lot of code to compile.

Another example is:

(define-syntax-rule (with-increment var expr)
  (let ([var (add1 var)]) expr))
...
(with-increment (* foo 2)
  ...code...)

the problem here is that (* foo 2) will be used as an identifier to be bound by the let expression — which can lead to a confusing syntax error.

Racket provides many tools to help macro programmers — in addition to a user-interface tool like the macro debugger there are also programmer-level tools where you can reject an input if it doesn’t contain an identifier at a certain place etc. Still, writing macros is much harder than writing functions — some of these problems are inherent to the problem that macros solve; for example, you may want a twice macro that replicates an expression. By specifying a transformation to the core language, a macro writer has full control over which expressions get evaluated and how, which identifiers are binding instances, and how is the scope of the given expression is shaped.

Meta Macros

One of the nice results of syntax-rules dealing with the subtle points of identifiers and scope is that things works fine even when we “go up a level”. For example, the short define-syntax-rule form that we’ve seen is itself a defined as a simple macro:

(define-syntax define-syntax-rule
  (syntax-rules ()
    [(define-syntax-rule (name P ...) B)
    (define-syntax name
      (syntax-rules ()
        [(name P ...) B]))]))

In fact, this is very similar to something that we have already seen: the rewrite form that we have used in Schlac is implemented in just this way. The only difference is that rewrite requires an actual => token to separate the input pattern from the output template. If we just use it in a syntax rule:

(define-syntax rewrite
  (syntax-rules ()
    [(rewrite (name P ...) => B)
    (define-syntax name
      (syntax-rules ()
        [(name P ...) B]))]))

it won’t work. Racket treats the above => just like any identifier, which in this case acts as a pattern variable which matches anything. The solution to this is to list the => as a keyword which is expected to appear in the macro use as-is — and that’s what the mysterious () of syntax-rules is used for: any identifier listed there is taken to be such a keyword. This makes the following version

(define-syntax rewrite
  (syntax-rules (=>)
    [(rewrite (name P ...) => B)
    (define-syntax name
      (syntax-rules ()
        [(name P ...) B]))]))

do what we want and throw a syntax error unless rewrite is used with an actual => in the proper place.

Lazy Constructions in an Eager Language

PLAI §37 (has some examples)

This is not really lazy evaluation, but it gets close, and provides the core useful property of easy-to-use infinite lists.

(define-syntax-rule (cons-stream x y)
  (cons x (lambda () y)))
(define stream? pair?)
(define null-stream null)
(define null-stream? null?)
(define stream-first first)
(define (stream-rest s) ((rest s)))

Using it:

(define ones (cons-stream 1 ones))
(define (stream-map f s)
  (if (null-stream? s)
    null-stream
    (cons-stream (f (stream-first s))
                (stream-map f (stream-rest s)))))
(define (stream-map2 f s1 s2)
  (if (null-stream? s1)
    null-stream
    (cons-stream (f (stream-first s1) (stream-first s2))
                (stream-map2 f (stream-rest s1)
                                (stream-rest s2)))))
(define ints (cons-stream 0 (stream-map2 + ones ints)))

Actually, all Scheme implementations come with a generalized tool for (local) laziness: a delay form that delays computation of its body expression, and a force function that forces such promises. Here is a naive implementation of this:

(define-type promise
  [make-promise (-> Any)])

(define-syntax-rule (delay expr)
  (make-promise (lambda () expr)))

(define (force p)
  (cases p [(make-promise thunk) (thunk)]))

Proper definitions of delay/force cache the result — and practical ones can get pretty complex, for example, in order to allow tail calls via promises.

Recursive Macros

Syntax transformations can be recursive. For example, we have seen how let* can be implemented by a transformation that uses two rules, one of which expands to another use of let*:

(define-syntax let*
  (syntax-rules ()
    [(let* () body ...)
    (let () body ...)]
    [(let* ((x v) (xs vs) ...) body ...)
    (let ((x v)) (let* ((xs vs) ...) body ...))]))

When Racket expands a let* expression, the result may contain a new let* which needs extending as well. An important implication of this is that recursive macros are fine, as long as the recursive case is using a smaller expression. This is just like any form of recursion (or loop), where you need to be looping over a well-founded set of values — where each iteration uses a new value that is closer to some base case.

For example, consider the following macro:

(define-syntax-rule (while condition body ...)
  (when condition
    body ...
    (while condition body ...)))

It seems like this is a good implementation of a while loop — after all, if you were to implement it as a function using thunks, you’d write very similar code:

(define (while condition-thunk body-thunk)
  (when (condition-thunk)
    (body-thunk)
    (while condition-thunk body-thunk)))

But if you look at the nested while form in the transformation rule, you’ll see that it is exactly the same as the input form. This means that this macro can never be completely expanded — it specifies infinite code! In practice, this makes the (Racket) compiler loop forever, consuming more and more memory. This is unlike, for example, the recursive let* rule which uses one less binding-value pair than specified as its input.

The reason that the function version of while is fine is that it iterates using the same code, and the condition thunk will depend on some state that converges to a base case (usually the body thunk will perform some side-effects that makes the loop converge). But in the macro case there is no evaluation happening, if the transformed syntax contains the same input pattern, we end up having a macro that expands infinitely.

The correct solution for a while macro is therefore to use plain recursion using a local recursive function:

(define-syntax-rule (while condition body ...)
  (letrec ([loop (lambda ()
                  (when condition
                    body ...
                    (loop)))])
    (loop)))

A popular way to deal with macros like this that revolve around a specific control flow is to separate them into a function that uses thunks, and a macro that does nothing except wrap input expressions as thunks. In this case, we get this solution:

(define (while/proc condition-thunk body-thunk)
  (when (condition-thunk)
    (body-thunk)
    (while/proc condition-thunk body-thunk)))

(define-syntax-rule (while condition body ...)
  (while/proc (lambda () condition)
              (lambda () body ...)))

Another example: a simple loop.

Here is an implementation of a macro that does a simple arithmetic loop:

(define-syntax for
  (syntax-rules (= to do)
    [(for x = m to n do body ...)
    (letrec ([loop (lambda (x)
                      (when (<= x n)
                        body ...
                        (loop (+ x 1))))])
      (loop m))]))

(Note that this is not complete code: it suffers from the usual problem of multiple evaluations of the n expression. We’ll deal with it soon.)

This macro combines both control flow and lexical scope. Control flow is specified by the loop (done, as usual in Racket, as a tail-recursive function) — for example, it determines how code is iterated, and it also determines what the for form will evaluate to (it evaluates to whatever when evaluates to, the void value in this case). Scope is also specified here, by translating the code to a function — this code makes x have a scope that covers the body so this is valid:

(for i = 1 to 3 do (printf "i = ~s\n" i))

but it also makes the boundary expression n be in this scope, making this:

(for i = 1 to (if (even? i) 10 20) do (printf "i = ~s\n" i))

valid. In addition, while evaluating the condition on each iteration might be desirable, in most cases it’s not — consider this example:

(for i = 1 to (read) do (printf "i = ~s\n" i))

This is easily solved by using a let to make the expression evaluate just once:

(define-syntax for
  (syntax-rules (= to do)
    [(for x = m to n do body ...)
    (let ([n* n]
          [m* m]) ; execution order
      (letrec ([loop (lambda (x)
                        (when (<= x n*)
                          body ...
                          (loop (+ x 1))))])
        (loop m*)))]))

which makes the previous use result in a “reference to undefined identifier: i” error.

Furthermore, the fact that we have a hygienic macro system means that it is perfectly fine to use nested for expressions:

(for a = 1 to 9 do
  (for b = 1 to 9 do (printf "~s,~s " a b))
  (newline))

The transformation is, therefore, completely specifying the semantics of this new form.

Extending this syntax is easy using multiple transformation rules — for example, say that we want to extend it to have a step optional keyword. The standard idiom is to have the step-less pattern translated into one that uses step 1:

(for x = m to n do body ...)
--> (for x = m to n step 1 do body ...)

Usually, you should remember that syntax-rules tries the patterns one by one until a match is found, but in this case there is no problems because the keywords make the choice unambiguous:

(define-syntax for
  (syntax-rules (= to do step)
    [(for x = m to n do body ...)
    (for x = m to n step 1 do body ...)]
    [(for x = m to n step d do body ...)
    (let ([n* n]
          [m* m]
          [d* d])
      (letrec ([loop (lambda (x)
                        (when (<= x n*)
                          body ...
                          (loop (+ x d*))))])
        (loop m*)))]))

(for i = 1 to 10 do (printf "i = ~s\n" i))
(for i = 1 to 10 step 2 do (printf "i = ~s\n" i))

We can even extend it to do a different kind of iteration, for example, iterate over list:

(define-syntax for
  (syntax-rules (= to do step in)
    [(for x = m to n do body ...)
    (for x = m to n step 1 do body ...)]
    [(for x = m to n step d do body ...)
    (let ([n* n]
          [m* m]
          [d* d])
      (letrec ([loop (lambda (x)
                        (when (<= x n*)
                          body ...
                          (loop (+ x d*))))])
        (loop m*)))]
    ;; list
    [(for x in l do body ...)
    (for-each (lambda (x) body ...) l)]))

(for i in (list 1 2 3 4) do (printf "i = ~s\n" i))

(for i in (list 1 2 3 4) do
  (for i = 0 to i do (printf "i = ~s  " i))
  (newline))