PL: Lecture #5  Tuesday, September 25th
(text file)

The match Form

The syntax for match is

(match value
  [pattern result-expr]
  ...)

The value is matched against each pattern, possibly binding names in the process, and if a pattern matches it evaluates the result expression. The simplest form of a pattern is simply an identifier — it always matches and binds that identifier to the value:

(match (list 1 2 3)
  [x x]) ; evaluates to the list

Another simple pattern is a quoted symbol, which matches that symbol. For example:

(match foo
  ['x "yes"]
  [else "no"])

will evaluate to "yes" if foo is the symbol x, and to "no" otherwise. Note that else is not a keyword here — it happens to be a pattern that always succeeds, so it behaves like an else clause except that it binds else to the unmatched-so-far value.

Many patterns look like function application — but don’t confuse them with applications. A (list x y z) pattern matches a list of exactly three items and binds the three identifiers; or if the “arguments” are themselves patterns, match will descend into the values and match them too. More specifically, this means that patterns can be nested:

(match (list 1 2 3)
  [(list x y z) (+ x y z)]) ; evaluates to 6
(match '((1) (2) 3)
  [(list (list x) (list y) z) (+ x y z)]) ; also 6

There is also a cons pattern that matches a non-empty list and then matches the first part against the head for the list and the second part against the tail of the list.

In a list pattern, you can use ... to specify that the previous pattern is repeated zero or more times, and bound names get bound to the list of respective matching. One simple consequent is that the (list hd tl ...) pattern is exactly the same as (cons hd tl), but being able to repeat an arbitrary pattern is very useful:

> (match '((1 2) (3 4) (5 6) (7 8))
    [(list (list x y) ...) (list x y)])
'((1 3 5 7) (2 4 6 8))

A few more useful patterns:

id              -- matches anything, binds `id' to it
_              -- matches anything, but does not bind
(number: n)    -- matches any number and binds it to `n'
(symbol: s)    -- same for symbols
(string: s)    -- strings
(sexpr: s)      -- S-expressions (needed sometimes for Typed Racket)
(and pat1 pat2) -- matches both patterns
(or pat1 pat2)  -- matches either pattern (careful with bindings)

Note that the foo: patterns are all specific to our #lang pl, they are not part of #lang racket or #lang typed/racket

The patterns are tried one by one in-order, and if no pattern matches the value, an error is raised.

Note that ... in a list pattern can follow any pattern, including all of the above, and including nested list patterns.

Here are a few examples — you can try them out with #lang pl untyped at the top of the definitions window. This:

(match x
  [(list (symbol: syms) ...) syms])

matches x against a pattern that accepts only a list of symbols, and binds syms to those symbols. And here’s an example that matches a list of any number of lists, where each of the sub-lists begins with a symbol and then has any number of numbers. Note how the n and s bindings get values for a list of all symbols and a list of lists of the numbers:

> (define (foo x)
    (match x
      [(list (list (symbol: s) (number: n) ...) ...)
      (list 'symbols: s 'numbers: n)]))
> (foo (list (list 'x 1 2 3) (list 'y 4 5)))
'(symbols: (x y) numbers: ((1 2 3) (4 5)))

Here is a quick example for how or is used with two literal alternatives, how and is used to name a specific piece of data, and how or is used with a binding:

> (define (foo x)
    (match x
      [(list (or 1 2 3)) 'single]
      [(list (and x (list 1 _)) 2) x]
      [(or (list 1 x) (list 2 x)) x]))
> (foo (list 3))
'single
> (foo (list (list 1 99) 2))
'(1 99)
> (foo (list 1 10))
10
> (foo (list 2 10))
10

Semantics (= Evaluation)

PLAI §2

Back to BNF — now, meaning.

An important feature of these BNF specifications: we can use the derivations to specify meaning (and meaning in our context is “running” a program (or “interpreting”, “compiling”, but we will use “evaluating”)). For example:

<AE> ::= <num>        ; <AE> evaluates to the number
      | <AE1> + <AE2> ; <AE> evaluates to the sum of evaluating
                      ;      <AE1> and <AE2>
      | <AE1> - <AE2> ; ... the subtraction of <AE2> from <AE1>
                              (... roughly!)

To do this a little more formally:

a. eval(<num>) = <num> ;*** special rule: translate syntax to value
b. eval(<AE1> + <AE2>) = eval(<AE1>) + eval(<AE2>)
c. eval(<AE1> - <AE2>) = eval(<AE1>) - eval(<AE2>)

Note the completely different roles of the two +s and -s. In fact, it might have been more correct to write:

a. eval("<num>") = <num>
b. eval("<AE1> + <AE2>") = eval("<AE1>") + eval("<AE2>")
c. eval("<AE1> - <AE2>") = eval("<AE1>") - eval("<AE2>")

or even using a marker to denote meta-holes in these strings:

a. eval("$<num>") = <num>
b. eval("$<AE1> + $<AE2>") = eval("$<AE1>") + eval("$<AE2>")
c. eval("$<AE1> - $<AE2>") = eval("$<AE1>") - eval("$<AE2>")

but we will avoid pretending that we’re doing that kind of string manipulation. (For example, it will require specifying what does it mean to return <num> for $<num> (involves string->number), and the fragments on the right side mean that we need to specify these as substring operations.)

Note that there’s a similar kind of informality in our BNF specifications, where we assume that <foo> refers to some terminal or non-terminal. In texts that require more formal specifications (for example, in RFC specifications), each literal part of the BNF is usually double-quoted, so we’d get

<AE> ::= <num> | <AE1> "+" <AE2> | <AE1> "-" <AE2>

An alternative popular notation for eval(X) is [[X]]:

a. [[<num>]] = <num>
b. [[<AE1> + <AE2>]] = [[<AE1>]] + [[<AE2>]]
c. [[<AE1> - <AE2>]] = [[<AE1>]] - [[<AE2>]]

Is there a problem with this definition? Ambiguity:

eval(1 - 2 + 3) = ?

Depending on the way the expression is parsed, we can get either a result of 2 or -4:

eval(1 - 2 + 3) = eval(1 - 2) + eval(3)          [b]
                = eval(1) - eval(2) + eval(3)    [c]
                = 1 - 2 + 3                      [a,a,a]
                = 2

eval(1 - 2 + 3) = eval(1) - eval(2 + 3)          [c]
                = eval(1) - (eval(2) + eval(3))  [a]
                = 1 - (2 + 3)                    [a,a,a]
                = -4

Again, be very aware of confusing subtleties which are extremely important: We need parens around a sub-expression only in one side, why? — When we write:

eval(1 - 2 + 3) = ... = 1 - 2 + 3

we have two expressions, but one stands for an input syntax, and one stands for a real mathematical expression.

In a case of a computer implementation, the syntax on the left is (as always) an AE syntax, and the real expression on the right is an expression in whatever language we use to implement our AE language.

Like we said earlier, ambiguity is not a real problem until the actual parse tree matters. With eval it definitely matters, so we must not make it possible to derive any syntax in multiple ways or our evaluation will be non-deterministic.


Quick exercise:

We can define a meaning for <digit>s and then <num>s in a similar way:

<NUM> ::= <digit> | <digit> <NUM>

eval(0) = 0
eval(1) = 1
eval(2) = 2
...
eval(9) = 9

eval(<digit>) = <digit>
eval(<digit> <NUM>) = 10*eval(<digit>) + eval(<NUM>)

Is this exactly what we want? — Depends on what we actually want…

Side-note: Compositionality

The example of

<NUM> ::= <digit> | <NUM> <digit>

being a language that is easier to write an evaluator for leads us to an important concept — compositionality. This definition is easier to write an evaluator for, since the resulting language is compositional: the meaning of an expression — for example 123 — is composed out of the meaning of its two parts, which in this BNF are 12 and 3. Specifically, the evaluation of <NUM> <digit> is 10 * the evaluation of the first, plus the evaluation of the second. In the <digit> <NUM> case this is more difficult — the meaning of such a number depends not only on the meaning of the two parts, but also on the <NUM> syntax:

eval(<digit> <NUM>) =
  eval(<digit>) * 10^length(<NUM>) + eval(<NUM>)

This this case this can be tolerable, since the meaning of the expression is still made out of its parts — but imperative programming (when you use side effects) is much more problematic since it is not compositional (at least not in the obvious sense). This is compared to functional programming, where the meaning of an expression is a combination of the meanings of its subexpressions. For example, every sub-expression in a functional program has some known meaning, and these all make up the meaning of the expression that contains them — but in an imperative program we can have a part of the code be x++ — and that doesn’t have a meaning by itself, at least not one that contributes to the meaning of the whole program in a direct way.

(Actually, we can have a well-defined meaning for such an expression: the meaning is going from a world where x is a container of some value N, to a world where the same container has a different value N+1. You can probably see now how this can make things more complicated. On an intuitive level — if we look at a random part of a functional program we can tell its meaning, so building up the meaning of the whole code is easy, but in an imperative program, the meaning of a random part is pretty much useless.)

Implementing an Evaluator

Now continue to implement the semantics of our syntax — we express that through an eval function that evaluates an expression.

We use a basic programming principle — splitting the code into two layers, one for parsing the input, and one for doing the evaluation. Doing this avoids the mess we’d get into otherwise, for example:

(define (eval sexpr)
  (match sexpr
    [(number: n) n]
    [(list '+ left right) (+ (eval left) (eval right))]
    [(list '- left right) (- (eval left) (eval right))]
    [else (error 'eval "bad syntax in ~s" sexpr)]))

This is messy because it combines two very different things — syntax and semantics — into a single lump of code. For this particular kind of evaluator it looks simple enough, but this is only because it’s simple enough that all we do is replace constructors by arithmetic operations. Later on things will get more complex, and bundling the evaluator with the parser will be more problematic. (Note: the fact that we can replace constructors with the run-time operators mean that we have a very simple, calculator-like language, and that we can, in fact, “compile” all programs down to a number.)

If we split the code, we can easily include decisions like making

{+ 1 {- 3 "a"}}

syntactically invalid. (Which is not, BTW, what Racket does…) (Also, this is like the distinction between XML syntax and well-formed XML syntax.)

An additional advantage is that by using two separate components, it is simple to replace each one, making it possible to change the input syntax, and the semantics independently — we only need to keep the same interface data (the AST) and things will work fine.

Our parse function converts an input syntax to an abstract syntax tree (AST). It is abstract exactly because it is independent of any actual concrete syntax that you type in, print out etc.

Implementing The AE Language

Back to our eval — this will be its (obvious) type:

(: eval : AE -> Number)
;; consumes an AE and computes
;; the corresponding number

which leads to some obvious test cases:

(equal? 3 (eval (parse "3")))
(equal? 7 (eval (parse "{+ 3 4}")))
(equal? 6 (eval (parse "{+ {- 3 4} 7}")))

which from now on we will write using the new test form that the #lang pl language provides:

(test (eval (parse "3"))            => 3)
(test (eval (parse "{+ 3 4}"))      => 7)
(test (eval (parse "{+ {- 3 4} 7}")) => 6)

Note that we’re testing only at the interface level — only running whole functions. For example, you could think about a test like:

(test (parse "{+ {- 3 4} 7}")
      => (Add (Sub (Num 3) (Num 4)) (Num 7)))

but the details of parsing and of the constructor names are things that nobody outside of our evaluator cares about — so we’re not testing them. In fact, we shouldn’t even mention parse in these tests, since it is not part of the public interface of our users; they only care about using it as a compiler-like black box. (This is sometimes called “integration tests”.) We’ll address this shortly.

Like everything else, the structure of the recursive eval code follows the recursive structure of its input. In HtDP terms, our template is:

(: eval : AE -> Number)
(define (eval expr)
  (cases expr
    [(Num n)  ... n ...]
    [(Add l r) ... (eval l) ... (eval r) ...]
    [(Sub l r) ... (eval l) ... (eval r) ...]))

In this case, filling in the gaps is very simple

(: eval : AE -> Number)
(define (eval expr)
  (cases expr
    [(Num n)  n]
    [(Add l r) (+ (eval l) (eval r))]
    [(Sub l r) (- (eval l) (eval r))]))

We now further combine eval and parse into a single run function that evaluates an AE string.

(: run : String -> Number)
;; evaluate an AE program contained in a string
(define (run str)
  (eval (parse str)))

This function becomes the single public entry point into our code, and the only thing that should be used in tests that verify our interface:

(test (run "3")            => 3)
(test (run "{+ 3 4}")      => 7)
(test (run "{+ {- 3 4} 7}") => 6)

The resulting full code is:

#lang pl

#| BNF for the AE language:
  <AE> ::= <num>
          | { + <AE> <AE> }
          | { - <AE> <AE> }
          | { * <AE> <AE> }
          | { / <AE> <AE> }
|#

;; AE abstract syntax trees
(define-type AE
  [Num Number]
  [Add AE AE]
  [Sub AE AE]
  [Mul AE AE]
  [Div AE AE])

(: parse-sexpr : Sexpr -> AE)
;; parses s-expressions into AEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n) (Num n)]
    [(list '+ lhs rhs) (Add (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '- lhs rhs) (Sub (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '* lhs rhs) (Mul (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '/ lhs rhs) (Div (parse-sexpr lhs) (parse-sexpr rhs))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

(: parse : String -> AE)
;; parses a string containing an AE expression to an AE AST
(define (parse str)
  (parse-sexpr (string->sexpr str)))

(: eval : AE -> Number)
;; consumes an AE and computes the corresponding number
(define (eval expr)
  (cases expr
    [(Num n)  n]
    [(Add l r) (+ (eval l) (eval r))]
    [(Sub l r) (- (eval l) (eval r))]
    [(Mul l r) (* (eval l) (eval r))]
    [(Div l r) (/ (eval l) (eval r))]))

(: run : String -> Number)
;; evaluate an AE program contained in a string
(define (run str)
  (eval (parse str)))

;; tests
(test (run "3") => 3)
(test (run "{+ 3 4}") => 7)
(test (run "{+ {- 3 4} 7}") => 6)

(Note that the tests are done with a test form, which we mentioned above.)

For anyone who thinks that Racket is a bad choice, this is a good point to think how much code would be needed in some other language to do the same as above.

Intro to Typed Racket

The plan:

Types

Typed Racket will be both similar to and very different from anything you’ve seen before.

Why types?

–> They will help you. A lot.

Structuring programs

Why Typed Racket?

Racket is the language we all know, and it has the benefits that we discussed earlier. Mainly, it is an excellent language for experimenting with programming languages.

[Also: the development of Typed Racket is happening here in Northeastern, and will benefit from your feedback.]

How is Typed Racket different from Racket

Examples

(: digit-num : Number -> (U Number String))
(define (digit-num n)
  (cond [(<= n 9)    1]
        [(<= n 99)  2]
        [(<= n 999)  3]
        [(<= n 9999) 4]
        [else        "a lot"]))

(: fact : Number -> Number)
(define (fact n)
  (if (zero? n)
    1
    (* n (fact (- n 1)))))

(: helper : Number Number -> Number)
(define (helper n acc)
  (if (zero? n)
    acc
    (helper (- n 1) (* acc n))))
(: fact : Number -> Number)
(define (fact n)
  (helper n 1))

(: fact : Number -> Number)
(define (fact n)
  (: helper : Number Number -> Number)
  (define (helper n acc)
    (if (zero? n)
      acc
      (helper (- n 1) (* acc n))))
  (helper n 1))

(: every? : (All (A) (A -> Boolean) (Listof A) -> Boolean))
;; Returns false if any element of lst fails the given pred,
;; true if all pass pred.
(define (every? pred lst)
  (or (null? lst)
      (and (pred (first lst))
          (every? pred (rest lst)))))

(define-type AE
  [Num Number]
  [Add AE AE]
  [Sub AE AE])

;; the only difference in the following definition is
;; using (: <name> : <type>) instead of ";; <name> : <type>"

(: parse-sexpr : Sexpr -> AE)
;; parses s-expressions into AEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n) (Num n)]
    [(list '+ left right)
    (Add (parse-sexpr left) (parse-sexpr right))]
    [(list '- left right)
    (Sub (parse-sexpr left) (parse-sexpr right))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

More interesting examples

Bindings & Substitution

We now get to an important concept: substitution.

Even in our simple language, we encounter repeated expressions. For example, if we want to compute the square of some expression:

{* {+ 4 2} {+ 4 2}}

Why would we want to get rid of the repeated sub-expression?

So, the normal way to avoid redundancy is to introduce an identifier. Even when we speak, we might say: “let x be 4 plus 2, multiply x by x”.

(These are often called “variables”, but we will try to avoid this name: what if the identifier does not change (vary)?)

To get this, we introduce a new form into our language:

{with {x {+ 4 2}}
  {* x x}}

We expect to be able to reduce this to:

{* 6 6}

by substituting 6 for x in the body sub-expression of with.

A little more complicated example:

{with {x {+ 4 2}}
  {with {y {* x x}}
    {+ y y}}}
[add]  = {with {x 6} {with {y {* x x}} {+ y y}}}
[subst]= {with {y {* 6 6}} {+ y y}}
[mul]  = {with {y 36} {+ y y}}
[subst]= {+ 36 36}
[add]  = 72

WAE: Adding Bindings to AE

PLAI §3

To add this to our language, we start with the BNF. We now call our language “WAE” (With+AE):

<WAE> ::= <num>
        | { + <WAE> <WAE> }
        | { - <WAE> <WAE> }
        | { * <WAE> <WAE> }
        | { / <WAE> <WAE> }
        | { with { <id> <WAE> } <WAE> }
        | <id>

Note that we had to introduce two new rules: one for introducing an identifier, and one for using it. This is common in many language specifications, for example define-type introduces a new type, and it comes with cases that allows us to destruct its instances.

For <id> we need to use some form of identifiers, the natural choice in Racket is to use symbols. We can therefore write the corresponding type definition:

(define-type WAE
  [Num  Number]
  [Add  WAE WAE]
  [Sub  WAE WAE]
  [Mul  WAE WAE]
  [Div  WAE WAE]
  [Id  Symbol]
  [With Symbol WAE WAE])

The parser is easily extended to produce these syntax objects:

(: parse-sexpr : Sexpr -> WAE)
;; parses s-expressions into WAEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n)    (Num n)]
    [(symbol: name) (Id name)]
    [(list 'with (list (symbol: name) named) body)
    (With name (parse-sexpr named) (parse-sexpr body))]
    [(list '+ lhs rhs) (Add (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '- lhs rhs) (Sub (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '* lhs rhs) (Mul (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '/ lhs rhs) (Div (parse-sexpr lhs) (parse-sexpr rhs))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

But note that this parser is inconvenient — if any of these expressions:

{* 1 2 3}
{foo 5 6}
{with x 5 {* x 8}}
{with {5 x} {* x 8}}

would result in a “bad syntax” error, which is not very helpful. To make things better, we can add another case for with expressions that are malformed, and give a more specific message in that case:

(: parse-sexpr : Sexpr -> WAE)
;; parses s-expressions into WAEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n)    (Num n)]
    [(symbol: name) (Id name)]
    [(list 'with (list (symbol: name) named) body)
    (With name (parse-sexpr named) (parse-sexpr body))]
    [(cons 'with more)
    (error 'parse-sexpr "bad `with' syntax in ~s" sexpr)]
    [(list '+ lhs rhs) (Add (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '- lhs rhs) (Sub (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '* lhs rhs) (Mul (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '/ lhs rhs) (Div (parse-sexpr lhs) (parse-sexpr rhs))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

and finally, to group all of the parsing code that deals with with expressions (both valid and invalid ones), we can use a single case for both of them:

(: parse-sexpr : Sexpr -> WAE)
;; parses s-expressions into WAEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n)    (Num n)]
    [(symbol: name) (Id name)]
    [(cons 'with more)
    ;; go in here for all sexpr that begin with a 'with
    (match sexpr
      [(list 'with (list (symbol: name) named) body)
        (With name (parse-sexpr named) (parse-sexpr body))]
      [else (error 'parse-sexpr "bad `with' syntax in ~s" sexpr)])]
    [(list '+ lhs rhs) (Add (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '- lhs rhs) (Sub (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '* lhs rhs) (Mul (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '/ lhs rhs) (Div (parse-sexpr lhs) (parse-sexpr rhs))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

And now we’re done with the syntactic part of the with extension.

Quick note — why would we indent With like a normal function in code like this

(With 'x
      (Num 2)
      (Add (Id 'x) (Num 4)))

instead of an indentation that looks like a let

(With 'x (Num 2)
  (Add (Id 'x) (Num 4)))

?

The reason for this is that the second indentation looks like a binding construct (eg, the indentation used in a let expression), but With is not a binding form — it’s a plain function because it’s at the Racket level. You should therefore keep in mind the huge difference between that With and the with that appears in WAE programs:

{with {x 2}
  {+ x 4}}

Another way to look at it: imagine that we intend for the language to be used by Spanish/Chinese/German speakers. In this case we would translate “with”:

{con {x 2} {+ x 4}}
{he  {x 2} {+ x 4}}
{mit {x 2} {+ x 4}}

but we will not do the same for With if we (the language implementors) are English speakers.

Evaluation of with

Now, to make this work, we will need to do some substitutions.

We basically want to say that to evaluate:

{with {id WAE1} WAE2}

we need to evaluate WAE2 with id substituted by WAE1. Formally:

eval( {with {id WAE1} WAE2} )
  = eval( subst(WAE2,id,WAE1) )

There is a more common syntax for substitution (quick: what do I mean by this use of “syntax”?):

eval( {with {id WAE1} WAE2} )
  = eval( WAE2[WAE1/id] )

Side-note: this syntax originates with logicians who used [x/v]e, and later there was a convention that mimicked the more natural order of arguments to a function with e[x->v], and eventually both of these got combined into e[v/x] which is a little confusing in that the left-to-right order of the arguments is not the same as for the subst function.

Now all we need is an exact definition of substitution.

Note that substitution is not the same as evaluation, it’s only a part of the evaluation process. In the previous examples, when we evaluated the expression we did substitutions as well as the usual arithmetic operations that were already part of the AE evaluator. In this last definition there is still a missing evaluation step, see if you can find it.

So let us try to define substitution now:

Substitution (take 1): e[v/i]
To substitute an identifier i in an expression e with an expression v, replace all identifiers in e that have the same name i by the expression v.

This seems to work with simple expressions, for example:

{with {x 5} {+ x x}}  -->  {+ 5 5}
{with {x 5} {+ 10 4}}  -->  {+ 10 4}

however, we crash with an invalid syntax if we try:

{with {x 5} {+ x {with {x 3} 10}}}
  -->  {+ 5 {with {5 3} 10}} ???

— we got to an invalid expression.

To fix this, we need to distinguish normal occurrences of identifiers, and ones that are used as new bindings. We need a few new terms for this:

  1. Binding Instance: a binding instance of an identifier is one that is used to name it in a new binding. In our <WAE> syntax, binding instances are only the <id> position of the with form.

  2. Scope: the scope of a binding instance is the region of program text in which instances of the identifier refer to the value bound in the binding instance. (Note that this definition actually relies on a definition of substitution, because that is what is used to specify how identifiers refer to values.)

  3. Bound Instance (or Bound Occurrence): an instance of an identifier is bound if it is contained within the scope of a binding instance of its name.

  4. Free Instance (or Free Occurrence): An identifier that is not contained in any binding instance of its name is said to be free.

Using this we can say that the problem with the previous definition of substitution is that it failed to distinguish between bound instances (which should be substituted) and binding instances (which should not). So we try to fix this:

Substitution (take 2): e[v/i]
To substitute an identifier i in an expression e with an expression v, replace all instances of i that are not themselves binding instances with the expression v.

First of all, check the previous examples:

{with {x 5} {+ x x}}  -->  {+ 5 5}
{with {x 5} {+ 10 4}}  -->  {+ 10 4}

still work, and

{with {x 5} {+ x {with {x 3} 10}}}
  -->  {+ 5 {with {x 3} 10}}
  -->  {+ 5 10}

also works. However, if we try this:

{with {x 5}
  {+ x {with {x 3}
        x}}}

we get:

-->  {+ 5 {with {x 3} 5}}
-->  {+ 5 5}
-->  10

but we want that to be 8: the inner x should be bound by the closest with that binds it.

The problem is that the new definition of substitution that we have respects binding instances, but it fails to deal with their scope. In the above example, we want the inner with to shadow the outer with’s binding for x.

Substitution (take 3): e[v/i]
To substitute an identifier i in an expression e with an expression v, replace all instances of i that are not themselves binding instances, and that are not in any nested scope, with the expression v.

This avoids bad substitution above, but it is now doing things too carefully:

{with {x 5} {+ x {with {y 3} x}}}

becomes

-->  {+ 5 {with {y 3} x}}
-->  {+ 5 x}

which is an error because x is unbound (and there is reasonable no rule that we can specify to evaluate it).

The problem is that our substitution halts at every new scope, in this case, it stopped at the new y scope, but it shouldn’t have because it uses a different name. In fact, that last definition of substitution cannot handle any nested scope.

Revise again:

Substitution (take 4): e[v/i]
To substitute an identifier i in an expression e with an expression v, replace all instances of i that are not themselves binding instances, and that are not in any nested scope of i, with the expression v.

which, finally, is a good definition. This is just a little too mechanical. Notice that we actually refer to all instances of i that are not in a scope of a binding instance of i, which simply means all free occurrences of i — free in e (why? — remember the definition of “free”?):

Substitution (take 4b): e[v/i]
To substitute an identifier i in an expression e with an expression v, replace all instances of i that are free in e with the expression v.

Based on this we can finally write the code for it:

(: subst : WAE Symbol WAE -> WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to) ; returns expr[to/from]
  (cases expr
    [(Num n) expr]
    [(Add l r) (Add (subst l from to) (subst r from to))]
    [(Sub l r) (Sub (subst l from to) (subst r from to))]
    [(Mul l r) (Mul (subst l from to) (subst r from to))]
    [(Div l r) (Div (subst l from to) (subst r from to))]
    [(Id name) (if (eq? name from) to expr)]
    [(With bound-id named-expr bound-body)
    (if (eq? bound-id from)
      expr          ;*** don't go in!
      (With bound-id
            named-expr
            (subst bound-body from to)))]))

… and this is just the same as writing a formal “paper version” of the substitution rule.

We still have bugs: but we’ll need some more work to get to them.


Before we find the bugs, we need to see when and how substitution is used in the evaluation process.

To modify our evaluator, we will need rules to deal with the new syntax pieces — with expressions and identifiers.

When we see an expression that looks like:

{with {x E1} E2}

we continue by evaluating E1 to get a value V1, we then substitute the identifier x with the expression V1 in E2, and continue by evaluating this new expression. In other words, we have the following evaluation rule:

eval( {with {x E1} E2} )
  = eval( E2[eval(E1)/x] )

So we know what to do with with expressions. How about identifiers? The main feature of subst, as said in the purpose statement, is that it leaves no free instances of the substituted variable around. This means that if the initial expression is valid (did not contain any free variables), then when we go from

{with {x E1} E2}

to

E2[E1/x]

the result is an expression that has no free instances of x. So we don’t need to handle identifiers in the evaluator — substitutions make them all go away.

We can now extend the formal definition of AE to that of WAE:

eval(...) = ... same as the AE rules ...
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(id) = error!

If you’re paying close attention, you might catch a potential problem in this definition: we’re substituting eval(E1) for x in E2 — an operation that requires a WAE expression, but eval(E1) is a number. (Look at the type of the eval definition we had for AE, then look at the above definition of subst.) This seems like being overly pedantic, but we it will require some resolution when we get to the code. The above rules are easily coded as follows:

(: eval : WAE -> Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
  (cases expr
    [(Num n) n]
    [(Add l r) (+ (eval l) (eval r))]
    [(Sub l r) (- (eval l) (eval r))]
    [(Mul l r) (* (eval l) (eval r))]
    [(Div l r) (/ (eval l) (eval r))]
    [(With bound-id named-expr bound-body)
    (eval (subst bound-body
                  bound-id
                  (Num (eval named-expr))))] ;***
    [(Id name) (error 'eval "free identifier: ~s" name)]))

Note the Num expression in the marked line: evaluating the named expression gives us back a number — we need to convert this number into a syntax to be able to use it with subst. The solution is to use Num to convert the resulting number into a numeral (the syntax of a number). It’s not an elegant solution, but it will do for now.

Finally, here are a few test cases. We use a new test special form which is part of the course plugin. The way to use test is with two expressions and an => arrow — DrRacket evaluates both, and nothing will happen if the results are equal. If the results are different, you will get a warning line, but evaluation will continue so you can try additional tests. You can also use an =error> arrow to test an error message — use it with some text from the expected error, ? stands for any single character, and * is a sequence of zero or more characters. (When you use test in your homework, the handin server will abort when tests fail.) We expect these tests to succeed (make sure that you understand why they should succeed).

;; tests
(test (run "5") => 5)
(test (run "{+ 5 5}") => 10)
(test (run "{with {x {+ 5 5}} {+ x x}}") => 20)
(test (run "{with {x 5} {+ x x}}") => 10)
(test (run "{with {x {+ 5 5}} {with {y {- x 3}} {+ y y}}}") => 14)
(test (run "{with {x 5} {with {y {- x 3}} {+ y y}}}") => 4)
(test (run "{with {x 5} {+ x {with {x 3} 10}}}") => 15)
(test (run "{with {x 5} {+ x {with {x 3} x}}}") => 8)
(test (run "{with {x 5} {+ x {with {y 3} x}}}") => 10)
(test (run "{with {x 5} {with {y x} y}}") => 5)
(test (run "{with {x 5} {with {x x} x}}") => 5)
(test (run "{with {x 1} y}") =error> "free identifier")

Putting this all together, we get the following code; trying to run this code will raise an unexpected error…

#lang pl

#| BNF for the WAE language:
    <WAE> ::= <num>
            | { + <WAE> <WAE> }
            | { - <WAE> <WAE> }
            | { * <WAE> <WAE> }
            | { / <WAE> <WAE> }
            | { with { <id> <WAE> } <WAE> }
            | <id>
|#

;; WAE abstract syntax trees
(define-type WAE
  [Num  Number]
  [Add  WAE WAE]
  [Sub  WAE WAE]
  [Mul  WAE WAE]
  [Div  WAE WAE]
  [Id  Symbol]
  [With Symbol WAE WAE])

(: parse-sexpr : Sexpr -> WAE)
;; parses s-expressions into WAEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n)    (Num n)]
    [(symbol: name) (Id name)]
    [(cons 'with more)
    (match sexpr
      [(list 'with (list (symbol: name) named) body)
        (With name (parse-sexpr named) (parse-sexpr body))]
      [else (error 'parse-sexpr "bad `with' syntax in ~s" sexpr)])]
    [(list '+ lhs rhs) (Add (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '- lhs rhs) (Sub (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '* lhs rhs) (Mul (parse-sexpr lhs) (parse-sexpr rhs))]
    [(list '/ lhs rhs) (Div (parse-sexpr lhs) (parse-sexpr rhs))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

(: parse : String -> WAE)
;; parses a string containing a WAE expression to a WAE AST
(define (parse str)
  (parse-sexpr (string->sexpr str)))

(: subst : WAE Symbol WAE -> WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
  (cases expr
    [(Num n) expr]
    [(Add l r) (Add (subst l from to) (subst r from to))]
    [(Sub l r) (Sub (subst l from to) (subst r from to))]
    [(Mul l r) (Mul (subst l from to) (subst r from to))]
    [(Div l r) (Div (subst l from to) (subst r from to))]
    [(Id name) (if (eq? name from) to expr)]
    [(With bound-id named-expr bound-body)
    (if (eq? bound-id from)
      expr
      (With bound-id
            named-expr
            (subst bound-body from to)))]))

(: eval : WAE -> Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
  (cases expr
    [(Num n) n]
    [(Add l r) (+ (eval l) (eval r))]
    [(Sub l r) (- (eval l) (eval r))]
    [(Mul l r) (* (eval l) (eval r))]
    [(Div l r) (/ (eval l) (eval r))]
    [(With bound-id named-expr bound-body)
    (eval (subst bound-body
                  bound-id
                  (Num (eval named-expr))))]
    [(Id name) (error 'eval "free identifier: ~s" name)]))

(: run : String -> Number)
;; evaluate a WAE program contained in a string
(define (run str)
  (eval (parse str)))

;; tests
(test (run "5") => 5)
(test (run "{+ 5 5}") => 10)
(test (run "{with {x {+ 5 5}} {+ x x}}") => 20)
(test (run "{with {x 5} {+ x x}}") => 10)
(test (run "{with {x {+ 5 5}} {with {y {- x 3}} {+ y y}}}") => 14)
(test (run "{with {x 5} {with {y {- x 3}} {+ y y}}}") => 4)
(test (run "{with {x 5} {+ x {with {x 3} 10}}}") => 15)
(test (run "{with {x 5} {+ x {with {x 3} x}}}") => 8)
(test (run "{with {x 5} {+ x {with {y 3} x}}}") => 10)
(test (run "{with {x 5} {with {y x} y}}") => 5)
(test (run "{with {x 5} {with {x x} x}}") => 5)
(test (run "{with {x 1} y}") =error> "free identifier")