The rest of the notes, Sunday, September 25th
=============================================
***This is provided for reference, and can change once we
go over this material in class.***


# Bindings & Substitution
We now get to an important concept: substitution.
Even in our simple language, we encounter repeated expressions. For
example, if we want to compute the square of some expression:
{* {+ 4 2} {+ 4 2}}
Why would we want to get rid of the repeated subexpression?
* It introduces a redundant computation. In this example, we want to
avoid computing the same subexpression a second time.
* It makes the computation more complicated than it could be without the
repetition. Compare the above with:
with x = {+ 4 2},
{* x x}
* This is related to a basic fact in programming that we have already
discussed: duplicating information is always a bad thing. Among other
bad consequences, it can even lead to bugs that could not happen if we
wouldn't duplicate code. A toy example is "fixing" one of the numbers
in one expression and forgetting to fix the corresponding one:
{* {+ 4 2} {+ 4 1}}
Real world examples involve much more code, which make such bugs very
difficult to find, but they still follow the same principle.
* This gives us more expressive power  we don't just say that we want
to multiply two expressions that both happen to be `{+ 4 2}`, we say
that we multiply the `{+ 4 2}` expression by *itself*. It allows us
to express identity of two values as well as using two values that
happen to be the same.
So, the normal way to avoid redundancy is to introduce an identifier.
Even when we speak, we might say: "let x be 4 plus 2, multiply x by x".
(These are often called "variables", but we will try to avoid this name:
what if the identifier does not change (vary)?)
To get this, we introduce a new form into our language:
{with {x {+ 4 2}}
{* x x}}
We expect to be able to reduce this to:
{* 6 6}
by substituting 6 for `x` in the body subexpression of `with`.
A little more complicated example:
{with {x {+ 4 2}}
{with {y {* x x}}
{+ y y}}}
[add] = {with {x 6} {with {y {* x x}} {+ y y}}}
[subst]= {with {y {* 6 6}} {+ y y}}
[mul] = {with {y 36} {+ y y}}
[subst]= {+ 36 36}
[add] = 72

# WAE: Adding Bindings to AE
> [PLAI §3]
To add this to our language, we start with the BNF. We now call our
language "WAE" (With+AE):
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

Note that we had to introduce *two* new rules: one for introducing an
identifier, and one for using it. This is common in many language
specifications, for example `definetype` introduces a new type, and it
comes with `cases` that allows us to destruct its instances.
For `` we need to use some form of identifiers, the natural choice
in Racket is to use symbols. We can therefore write the corresponding
type definition:
(definetype WAE
[Num Number]
[Add WAE WAE]
[Sub WAE WAE]
[Mul WAE WAE]
[Div WAE WAE]
[Id Symbol]
[With Symbol WAE WAE])
The parser is easily extended to produce these syntax objects:
(: parsesexpr : Sexpr > WAE)
;; parses sexpressions into WAEs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
But note that this parser is inconvenient  if any of these
expressions:
{* 1 2 3}
{foo 5 6}
{with x 5 {* x 8}}
{with {5 x} {* x 8}}
would result in a "bad syntax" error, which is not very helpful. To
make things better, we can add another case for `with` expressions that
are malformed, and give a more specific message in that case:
(: parsesexpr : Sexpr > WAE)
;; parses sexpressions into WAEs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[(cons 'with more)
(error 'parsesexpr "bad `with' syntax in ~s" sexpr)]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
and finally, to group all of the parsing code that deals with `with`
expressions (both valid and invalid ones), we can use a single case for
both of them:
(: parsesexpr : Sexpr > WAE)
;; parses sexpressions into WAEs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
;; go in here for all sexpr that begin with a 'with
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
And now we're done with the syntactic part of the `with` extension.
> Quick note  why would we indent `With` like a normal function in
> code like this
>
> (With 'x
> (Num 2)
> (Add (Id 'x) (Num 4)))
>
> instead of an indentation that looks like a `let`
>
> (With 'x (Num 2)
> (Add (Id 'x) (Num 4)))
>
> ?
>
> The reason for this is that the second indentation looks like a
> binding construct (eg, the indentation used in a `let` expression),
> but `With` is *not* a binding form  it's a *plain function* because
> it's at the Racket level. You should therefore keep in mind the huge
> difference between that `With` and the `with` that appears in WAE
> programs:
>
> {with {x 2}
> {+ x 4}}
>
> Another way to look at it: imagine that we intend for the language to
> be used by Spanish/Chinese/German/French speakers. In this case we
> would translate "`with`":
>
> {con {x 2} {+ x 4}}
> {he {x 2} {+ x 4}}
> {mit {x 2} {+ x 4}}
> {avec {x 2} {+ x 4}}
> {c {x 2} {+ x 4}}
>
> but we will *not* do the same for `With` if we (the language
> implementors) are English speakers.

# Evaluation of `with`
Now, to make this work, we will need to do some substitutions.
We basically want to say that to evaluate:
{with {id WAE1} WAE2}
we need to evaluate `WAE2` with id substituted by `WAE1`. Formally:
eval( {with {id WAE1} WAE2} )
= eval( subst(WAE2,id,WAE1) )
There is a more common syntax for substitution (quick: what do I mean by
this use of "syntax"?):
eval( {with {id WAE1} WAE2} )
= eval( WAE2[WAE1/id] )
> Sidenote: this syntax originates with logicians who used `[x/v]e`,
> and later there was a convention that mimicked the more natural order
> of arguments to a function with `e[x>v]`, and eventually both of
> these got combined into `e[v/x]` which is a little confusing in that
> the lefttoright order of the arguments is not the same as for the
> `subst` function.
Now all we need is an exact definition of substitution.
> Note that substitution is not the same as evaluation, it's only a part
> of the evaluation process. In the previous examples, when we
> evaluated the expression we did substitutions as well as the usual
> arithmetic operations that were already part of the AE evaluator. In
> this last definition there is still a missing evaluation step, see if
> you can find it.
So let us try to define substitution now:
> Substitution (take 1): `e[v/i]` \
> To substitute an identifier `i` in an expression `e` with an
> expression `v`, replace all identifiers in `e` that have the same
> name `i` by the expression `v`.
This seems to work with simple expressions, for example:
{with {x 5} {+ x x}} > {+ 5 5}
{with {x 5} {+ 10 4}} > {+ 10 4}
however, we crash with an invalid syntax if we try:
{with {x 5} {+ x {with {x 3} 10}}}
> {+ 5 {with {5 3} 10}} ???
 we got to an invalid expression.
To fix this, we need to distinguish normal occurrences of identifiers,
and ones that are used as new bindings. We need a few new terms for
this:
1. Binding Instance: a binding instance of an identifier is one that is
used to name it in a new binding. In our `` syntax, binding
instances are only the `` position of the `with` form.
2. Scope: the scope of a binding instance is the region of program text
in which instances of the identifier refer to the value bound in the
binding instance. (Note that this definition actually relies on a
definition of substitution, because that is what is used to specify
how identifiers refer to values.)
3. Bound Instance (or Bound Occurrence): an instance of an identifier is
bound if it is contained within the scope of a binding instance of
its name.
4. Free Instance (or Free Occurrence): An identifier that is not
contained in any binding instance of its name is said to be free.
Using this we can say that the problem with the previous definition of
substitution is that it failed to distinguish between bound instances
(which should be substituted) and binding instances (which should not).
So we try to fix this:
> Substitution (take 2): `e[v/i]` \
> To substitute an identifier `i` in an expression `e` with an
> expression `v`, replace all instances of `i` that are not
> themselves binding instances with the expression `v`.
First of all, check the previous examples:
{with {x 5} {+ x x}} > {+ 5 5}
{with {x 5} {+ 10 4}} > {+ 10 4}
still work, and
{with {x 5} {+ x {with {x 3} 10}}}
> {+ 5 {with {x 3} 10}}
> {+ 5 10}
also works. However, if we try this:
{with {x 5}
{+ x {with {x 3}
x}}}
we get:
> {+ 5 {with {x 3} 5}}
> {+ 5 5}
> 10
but we want that to be `8`: the inner `x` should be bound by the closest
`with` that binds it.
The problem is that the new definition of substitution that we have
respects binding instances, but it fails to deal with their scope. In
the above example, we want the inner `with` to *shadow* the outer
`with`'s binding for `x`.
> Substitution (take 3): `e[v/i]` \
> To substitute an identifier `i` in an expression `e` with an
> expression `v`, replace all instances of `i` that are not themselves
> binding instances, and that are not in any nested scope, with the
> expression `v`.
This avoids bad substitution above, but it is now doing things too
carefully:
{with {x 5} {+ x {with {y 3} x}}}
becomes
> {+ 5 {with {y 3} x}}
> {+ 5 x}
which is an error because `x` is unbound (and there is reasonable no
rule that we can specify to evaluate it).
The problem is that our substitution halts at every new scope, in this
case, it stopped at the new `y` scope, but it shouldn't have because it
uses a different name. In fact, that last definition of substitution
cannot handle any nested scope.
Revise again:
> Substitution (take 4): `e[v/i]` \
> To substitute an identifier `i` in an expression `e` with an
> expression `v`, replace all instances of `i` that are not themselves
> binding instances, and that are not in any nested scope of `i`, with
> the expression `v`.
which, finally, is a good definition. This is just a little too
mechanical. Notice that we actually refer to all instances of `i` that
are not in a scope of a binding instance of `i`, which simply means all
*free occurrences* of `i`  free in `e` (why?  remember the
definition of "free"?):
> Substitution (take 4b): `e[v/i]` \
> To substitute an identifier `i` in an expression `e` with an
> expression `v`, replace all instances of `i` that are free in `e`
> with the expression `v`.
Based on this we can finally write the code for it:
(: subst : WAE Symbol WAE > WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to) ; returns expr[to/from]
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(if (eq? boundid from)
expr ;*** don't go in!
(With boundid
namedexpr
(subst boundbody from to)))]))
... and this is just the same as writing a formal "paper version" of the
substitution rule.
We still have bugs: but we'll need some more work to get to them.

Before we find the bugs, we need to see when and how substitution is
used in the evaluation process.
To modify our evaluator, we will need rules to deal with the new syntax
pieces  `with` expressions and identifiers.
When we see an expression that looks like:
{with {x E1} E2}
we continue by *evaluating* `E1` to get a value `V1`, we then substitute
the identifier `x` with the expression `V1` in `E2`, and continue by
evaluating this new expression. In other words, we have the following
evaluation rule:
eval( {with {x E1} E2} )
= eval( E2[eval(E1)/x] )
So we know what to do with `with` expressions. How about identifiers?
The main feature of `subst`, as said in the purpose statement, is that
it leaves no free instances of the substituted variable around. This
means that if the initial expression is valid (did not contain any free
variables), then when we go from
{with {x E1} E2}
to
E2[E1/x]
the result is an expression that has *no* free instances of `x`. So we
don't need to handle identifiers in the evaluator  substitutions make
them all go away.
We can now extend the formal definition of AE to that of WAE:
eval(...) = ... same as the AE rules ...
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(id) = error!
If you're paying close attention, you might catch a potential problem in
this definition: we're substituting `eval(E1)` for `x` in `E2`  an
operation that requires a WAE expression, but `eval(E1)` is a number.
(Look at the type of the `eval` definition we had for AE, then look at
the above definition of `subst`.) This seems like being overly
pedantic, but we it will require some resolution when we get to the
code. The above rules are easily coded as follows:
(: eval : WAE > Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))]
[(Div l r) (/ (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))] ;***
[(Id name) (error 'eval "free identifier: ~s" name)]))
Note the `Num` expression in the marked line: evaluating the named
expression gives us back a number  we need to convert this number
into a syntax to be able to use it with `subst`. The solution is to use
`Num` to convert the resulting number into a numeral (the syntax of a
number). It's not an elegant solution, but it will do for now.
Finally, here are a few test cases. We use a new `test` special form
which is part of the course plugin. The way to use `test` is with two
expressions and an `=>` arrow  DrRacket evaluates both, and nothing
will happen if the results are equal. If the results are different, you
will get a warning line, but evaluation will continue so you can try
additional tests. You can also use an `=error>` arrow to test an error
message  use it with some text from the expected error, `?` stands
for any single character, and `*` is a sequence of zero or more
characters. (When you use `test` in your homework, the handin server
will abort when tests fail.) We expect these tests to succeed (make
sure that you understand *why* they should succeed).
;; tests
(test (run "5") => 5)
(test (run "{+ 5 5}") => 10)
(test (run "{with {x {+ 5 5}} {+ x x}}") => 20)
(test (run "{with {x 5} {+ x x}}") => 10)
(test (run "{with {x {+ 5 5}} {with {y { x 3}} {+ y y}}}") => 14)
(test (run "{with {x 5} {with {y { x 3}} {+ y y}}}") => 4)
(test (run "{with {x 5} {+ x {with {x 3} 10}}}") => 15)
(test (run "{with {x 5} {+ x {with {x 3} x}}}") => 8)
(test (run "{with {x 5} {+ x {with {y 3} x}}}") => 10)
(test (run "{with {x 5} {with {y x} y}}") => 5)
(test (run "{with {x 5} {with {x x} x}}") => 5)
(test (run "{with {x 1} y}") =error> "free identifier")
Putting this all together, we get the following code; trying to run this
code will raise an unexpected error...
#lang pl
# BNF for the WAE language:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

#
;; WAE abstract syntax trees
(definetype WAE
[Num Number]
[Add WAE WAE]
[Sub WAE WAE]
[Mul WAE WAE]
[Div WAE WAE]
[Id Symbol]
[With Symbol WAE WAE])
(: parsesexpr : Sexpr > WAE)
;; parses sexpressions into WAEs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > WAE)
;; parses a string containing a WAE expression to a WAE AST
(define (parse str)
(parsesexpr (string>sexpr str)))
(: subst : WAE Symbol WAE > WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(if (eq? boundid from)
expr
(With boundid
namedexpr
(subst boundbody from to)))]))
(: eval : WAE > Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))]
[(Div l r) (/ (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))]
[(Id name) (error 'eval "free identifier: ~s" name)]))
(: run : String > Number)
;; evaluate a WAE program contained in a string
(define (run str)
(eval (parse str)))
;; tests
(test (run "5") => 5)
(test (run "{+ 5 5}") => 10)
(test (run "{with {x {+ 5 5}} {+ x x}}") => 20)
(test (run "{with {x 5} {+ x x}}") => 10)
(test (run "{with {x {+ 5 5}} {with {y { x 3}} {+ y y}}}") => 14)
(test (run "{with {x 5} {with {y { x 3}} {+ y y}}}") => 4)
(test (run "{with {x 5} {+ x {with {x 3} 10}}}") => 15)
(test (run "{with {x 5} {+ x {with {x 3} x}}}") => 8)
(test (run "{with {x 5} {+ x {with {y 3} x}}}") => 10)
(test (run "{with {x 5} {with {y x} y}}") => 5)
(test (run "{with {x 5} {with {x x} x}}") => 5)
(test (run "{with {x 1} y}") =error> "free identifier")

Oops, this program still has problems that were caught by the tests 
we encounter unexpected free identifier errors. What's the problem now?
In expressions like:
{with {x 5}
{with {y x}
y}}
we forgot to substitute `x` in the expression that `y` is bound to. We
need to the recursive substitute in both the with's body expression as
well as its named expression:
(: subst : WAE Symbol WAE > WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(if (eq? boundid from)
expr
(With boundid
(subst namedexpr from to) ;*** new
(subst boundbody from to)))]))
And *still* we have a problem... Now it's
{with {x 5}
{with {x x}
x}}
that halts with an error, but we want it to evaluate to `5`! Carefully
trying out our substitution code reveals the problem: when we substitute
`5` for the outer `x`, we don't go inside the inner `with` because it
has the same name  but we *do* need to go into its named expression.
We need to substitute in the named expression even if the identifier is
the *same* one we're substituting:
(: subst : WAE Symbol WAE > WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]))
The complete (and, finally, correct) version of the code is now:
;;; <<>>
#lang pl
# BNF for the WAE language:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

#
;; WAE abstract syntax trees
(definetype WAE
[Num Number]
[Add WAE WAE]
[Sub WAE WAE]
[Mul WAE WAE]
[Div WAE WAE]
[Id Symbol]
[With Symbol WAE WAE])
(: parsesexpr : Sexpr > WAE)
;; parses sexpressions into WAEs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > WAE)
;; parses a string containing a WAE expression to a WAE AST
(define (parse str)
(parsesexpr (string>sexpr str)))
# Formal specs for `subst':
(`N' is a , `E1', `E2' are s, `x' is some ,
`y' is a *different* )
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]}
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
#
(: subst : WAE Symbol WAE > WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]))
# Formal specs for `eval':
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2)
eval({ E1 E2}) = eval(E1)  eval(E2)
eval({* E1 E2}) = eval(E1) * eval(E2)
eval({/ E1 E2}) = eval(E1) / eval(E2)
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
#
(: eval : WAE > Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))]
[(Div l r) (/ (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))]
[(Id name) (error 'eval "free identifier: ~s" name)]))
(: run : String > Number)
;; evaluate a WAE program contained in a string
(define (run str)
(eval (parse str)))
;; tests
(test (run "5") => 5)
(test (run "{+ 5 5}") => 10)
(test (run "{with {x 5} {+ x x}}") => 10)
(test (run "{with {x {+ 5 5}} {+ x x}}") => 20)
(test (run "{with {x 5} {with {y { x 3}} {+ y y}}}") => 4)
(test (run "{with {x {+ 5 5}} {with {y { x 3}} {+ y y}}}") => 14)
(test (run "{with {x 5} {+ x {with {x 3} 10}}}") => 15)
(test (run "{with {x 5} {+ x {with {x 3} x}}}") => 8)
(test (run "{with {x 5} {+ x {with {y 3} x}}}") => 10)
(test (run "{with {x 5} {with {y x} y}}") => 5)
(test (run "{with {x 5} {with {x x} x}}") => 5)
(test (run "{with {x 1} y}") =error> "free identifier")

Reminder:
* We started doing substitution, with a `let`like form: `with`.
* Reasons for using bindings:
 Avoid writing expressions twice.
* More expressive language (can express identity).
* Duplicating is bad! ("DRY": *Don't Repeat Yourself*.)
* Avoids *static* redundancy.
 Avoid redundant computations.
* More than *just* an optimization when it avoids exponential
resources.
* Avoids *dynamic* redundancy.
* BNF:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

Note that we had to introduce two new rules: one for introducing an
identifier, and one for using it.
* Type definition:
(definetype WAE
[Num Number]
[Add WAE WAE]
[Sub WAE WAE]
[Mul WAE WAE]
[Div WAE WAE]
[Id Symbol]
[With Symbol WAE WAE])
* Parser:
(: parsesexpr : Sexpr > WAE)
;; parses sexpressions into WAEs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s"
sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
* We need to define substitution. Terms:
1. Binding Instance.
2. Scope.
3. Bound Instance.
4. Free Instance.
* After lots of attempts:
> e[v/i]  To substitute an identifier `i` in an expression `e`
> with an expression `v`, replace all instances of `i` that are free
> in `e` with the expression `v`.
* Implemented the code, and again, needed to fix a few bugs:
(: subst : WAE Symbol WAE > WAE)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]))
(Note that the bugs that we fixed clarify the exact way that our
scopes work: in `{with {x 2} {with {x {+ x 2}} x}}`, the scope of the
first `x` is the `{+ x 2}` expression.)
* We then extended the AE evaluation rules:
eval(...) = ... same as the AE rules ...
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(id) = error!
and noted the possible type problem.
* The above translated into a Racket definition for an `eval` function
(with a hack to avoid the type issue):
(: eval : WAE > Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))]
[(Div l r) (/ (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))]
[(Id name) (error 'eval "free identifier: ~s" name)]))

# Formal Specs
Note the formal definitions that were included in the WAE code. They
are ways of describing pieces of our language that are more formal than
plain English, but still not as formal (and as verbose) as the actual
code.
A formal definition of `subst`:
(`N` is a ``, `E1`, `E2` are ``s, `x` is some ``, `y` is a
*different* ``)
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]}
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
And a formal definition of `eval`:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2)
eval({ E1 E2}) = eval(E1)  eval(E2)
eval({* E1 E2}) = eval(E1) * eval(E2)
eval({/ E1 E2}) = eval(E1) / eval(E2)
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])

# Lazy vs Eager Evaluation
As we have previously seen, there are two basic approaches for
evaluation: either eager or lazy. In lazy evaluation, bindings are used
for sort of textual references  it is only for avoiding writing an
expression twice, but the associated computation is done twice anyway.
In eager evaluation, we eliminate not only the textual redundancy, but
also the computation.
Which evaluation method did our evaluator use? The relevant piece of
formalism is the treatment of `with`:
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
And the matching piece of code is:
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))]
How do we make this lazy?
In the formal equation:
eval({with {x E1} E2}) = eval(E2[E1/x])
and in the code:
(: eval : WAE > Number)
;; evaluates WAE expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
namedexpr))] ;*** no eval and no Num wrapping
[(Id name) (error 'eval "free identifier: ~s" name)]))
We can verify the way this works by tracing `eval` (compare the trace
you get for the two versions):
> (trace eval) ; (put this in the definitions window)
> (run "{with {x {+ 1 2}} {* x x}}")
Ignoring the traces for now, the modified WAE interpreter works as
before, specifically, all tests pass. So the question is whether the
language we get is actually different than the one we had before. One
difference is in execution speed, but we can't really notice a
difference, and we care more about meaning. Is there any program that
will run differently in the two languages?
The main feature of the lazy evaluator is that it is not evaluating the
named expression until it is actually needed. As we have seen, this
leads to duplicating computations if the bound identifier is used more
than once  meaning that it does not eliminate the dynamic redundancy.
But what if the bound identifier is not used at all? In that case the
named expression simply evaporates. This is a good hint at an
expression that behaves differently in the two languages  if we add
division to both languages, we get a different result when we try
running:
{with {x {/ 8 0}} 7}
The eager evaluator stops with an error when it tries evaluating the
division  and the lazy evaluator simply ignores it.
Even without division, we get a similar behavior for
{with {x y} 7}
but it is questionable whether the fact that this evaluates to 7 is
correct behavior  we really want to forbid program that use free
variable.
Furthermore, there is an issue with name capturing  we don't want to
substitute an expression into a context that captures some of its free
variables. But our substitution allows just that, which is usually not
a problem because by the time we do the substitution, the named
expression should not have free variables that need to be replaced.
However, consider evaluating this program:
{with {y x}
{with {x 2}
{+ x y}}}
under the two evaluation regimens: the eager version stops with an
error, and the lazy version succeed. This points at a bug in our
substitution, or rather not dealing with an issue that we do not
encounter.
So the summary is: as long as the initial program is correct, both
evaluation regimens produce the same results. If a program contains
free variables, they might get captured in a naive lazy evaluator
implementation (but this is a bug that should be fixed). Also, there
are some cases where eager evaluation runs into a runtime problem which
does not happen in a lazy evaluator because the expression is not used.
It is possible to prove that when you evaluate an expression, if there
is an error that can be avoided, lazy evaluation will always avoid it,
whereas an eager evaluator will always run into it. On the other hand,
lazy evaluators are usually slower than eager evaluator, so it's a speed
vs. robustness tradeoff.
Note that with lazy evaluation we say that an identifier is bound to an
expression rather than a value. (Again, this is why the eager version
needed to wrap `eval`'s result in a `Num` and this one doesn't.)
(It is possible to change things and get a more well behaved
substitution, we basically will need to find if a capture might happen,
and rename things to avoid it. For example,
{with {y E1} E2}[v/x]
if `x' and `y' are equal
= {with {y E1[v/x]} E2} = {with {x E1[v/x]} E2}
if `y' has a free occurrence in `v'
= {with {y1 E1[v/x]} E2[y1/y][v/x]} ; `y1' is "fresh"
otherwise
= {with {x E1[v/x]} E2[v/x]}
With this, we might have gone through this path in evaluating the above:
{with {y x} {with {x 2} {+ x y}}}
{with {x₁ 2} {+ x₁ x}} ; note that x₁ is a fresh name, not x
{+ 2 x}
error: free `x`
But you can see that this is much more complicated (more code: requires
a `freein` predicate, being able to invent new *fresh* names, etc).
And it's not even the end of that story...)

# de Bruijn Indexes
This whole story revolves around names, specifically, name capture is a
problem that should always be avoided (it is one major source of PL
headaches).
But are names the only way we can use bindings?
There is a least one alternative way: note that the only thing we used
names for are for references. We don't really care what the name is,
which is pretty obvious when we consider the two WAE expressions:
{with {x 5} {+ x x}}
{with {y 5} {+ y y}}
or the two Racket function definitions:
(define (foo x) (list x x))
(define (foo y) (list y y))
Both of these show a pair of expressions that we should consider as
equal in some sense (this is called "alphaequality"). The only thing
we care about is what variable points where: the binding structure is
the only thing that matters. In other words, as long as DrRacket
produces the same arrows when we use Check Syntax, we consider the
program to be the same, regardless of name choices (for argument names
and local names, not for global names like `foo` in the above).
The alternative idea uses this principle: if all we care about is where
the arrows go, then simply get rid of the names... Instead of
referencing a binding through its name, just specify which of the
surrounding scopes we want to refer to. For example, instead of:
{with {x 5} {with {y 6} {+ x y}}}
we can use a new "reference" syntax  `[N]`  and use this instead
of the above:
{with 5 {with 6 {+ [1] [0]}}}
So the rules for `[N]` are  `[0]` is the value bound in the current
scope, `[1]` is the value from the next one up etc.
Of course, to do this translation, we have to know the precise scope
rules. Two more complicated examples:
{with {x 5} {+ x {with {y 6} {+ x y}}}}
is translated to:
{with 5 {+ [0] {with 6 {+ [1] [0]}}}}
(note how `x` appears as a different reference based on where it
appeared in the original code.) Even more subtle:
{with {x 5} {with {y {+ x 1}} {+ x y}}}
is translated to:
{with 5 {with {+ [0] 1} {+ [1] [0]}}}
because the inner `with` does not have its own named expression in its
scope, so the named expression is immediately in the scope of the outer
`with`.
This is called "de Bruijn Indexes": instead of referencing identifiers
by their name, we use an index into the surrounding binding context.
The major disadvantage, as can be seen in the above examples, is that it
is not convenient for humans to work with. Specifically, the same
identifier is referenced using different numbers, which makes it hard to
understand what some code is doing. After all, *abstractions* are the
main thing we deal with when we write programs, and having labels make
the bindings structure much easier to understand than scope counts.
However, practically all compilers use this for compiled code (think
about stack pointers). For example, GCC compiles this code:
{
int x = 5;
{
int y = x + 1;
return x + y;
}
}
to:
subl $8, %esp
movl $5, 4(%ebp) ; int x = 5
movl 4(%ebp), %eax
incl %eax
movl %eax, 8(%ebp) ; int y = %eax
movl 8(%ebp), %eax
addl 4(%ebp), %eax

# Functions & Function Values
> [PLAI §4]
Now that we have a form for local bindings, which forced us to deal with
proper substitutions and everything that is related, we can get to
functions. The concept of a function is itself very close to
substitution, and to our `with` form. For example, when we write:
{with {x 5}
{* x x}}
then the `{* x x}` body is itself parametrized over some value for `x`.
If we take this expression and take out the `5`, we're left with
something that has all of the necessary ingredients of a function  a
bunch of code that is parameterized over some input identifier:
{with {x}
{* x x}}
We only need to replace `with` and use a proper name that indicates that
it's a function:
{fun {x}
{* x x}}
Now we have a new form in our language, one that should have a function
as its meaning. As we have seen in the case of `with` expressions, we
also need a new form to *use* these functions. We will use `call` for
this, so that
{call {fun {x} {* x x}}
5}
will be the same as the original `with` expression that we started with
 the `fun` expression is like the `with` expression with no value,
and applying it on `5` is providing that value back:
{with {x 5}
{* x x}}
Of course, this does not help much  all we get is a way to use local
bindings that is more verbose from what we started with. What we're
really missing is a way to *name* these functions. If we get the right
evaluation rules, we can evaluate a `fun` expression to some value 
which will allow us to bind it to a variable using `with`. Something
like this:
{with {sqr {fun {x} {* x x}}}
{+ {call sqr 5}
{call sqr 6}}}
In this expression, we say that `x` is the formal parameter (or
argument), and the `5` and `6` are actual parameters (sometimes
abbreviated as formals and actuals). Note that naming functions often
helps, but many times there are small functions that are fine to specify
without a name  for example, consider a twostage addition function,
where there is no apparent good name for the returned function:
{with {add {fun {x}
{fun {y}
{+ x y}}}}
{call {call add 8} 9}}

# Implementing First Class Functions
> [PLAI §6] (uses some stuff from [PLAI §5], which we do later)
This is a simple plan, but it is directly related to how functions are
going to be used in our language. We know that `{call {fun {x} E1} E2}`
is equivalent to a `with` expression, but the new thing here is that we
do allow writing just the `{fun ...}` expression by itself, and
therefore we need to have some meaning for it. The meaning, or the
value of this expression, should roughly be "an expression that needs a
value to be plugged in for `x`". In other words, our language will have
these new kinds of values that contain an expression to be evaluated
later on.
There are three basic approaches that classify programming languages in
relation to how the deal with functions:
1. First order: functions are not real values. They cannot be used or
returned as values by other functions. This means that they cannot
be stored in data structures. This is what most "conventional"
languages used to have in the past. (You will be implementing such a
language in homework 4.)
An example of such a language is the Beginner Student language that
is used in HtDP, where the language is intentionally firstorder to
help students write correct code (at the early stages where using a
function as a value is usually an error). It's hard to find
practical modern languages that fall in this category.
2. Higher order: functions can receive and return other functions as
values. This is what you get with C and modern Fortran.
3. First class: functions are values with all the rights of other
values. In particular, they can be supplied to other functions,
returned from functions, stored in data structures, and new functions
can be created at runtime. (And most modern languages have first
class functions.)
The last category is the most interesting one. Back in the old days,
complex expressions were not firstclass in that they could not be
freely composed. This is still the case in machinecode: as we've seen
earlier, to compute an expression such as
(b + sqrt(b^2  4*a*c)) / 2a
you have to do something like this:
x = b * b
y = 4 * a
y = y * c
x = x  y
x = sqrt(x)
y = b
x = y + x
y = 2 * a
s = x / y
In other words, every intermediate value needs to have its own name.
But with proper ("highlevel") programming languages (at least most of
them...) you can just write the original expression, with no names for
these values.
With firstclass functions something similar happens  it is possible
to have complex expressions that consume and return functions, and they
do not need to be named.
What we get with our `fun` expression (if we can make it work) is
exactly this: it generates a function, and you can choose to either bind
it to a name, or not. The important thing is that the value exists
independently of a name.
This has a major effect on the "personality" of a programming language
as we will see. In fact, just adding this feature will make our
language much more advanced than languages with just higherorder or
firstorder functions.

Quick Example: the following is working JavaScript code, that uses first
class functions.
function foo(x) {
function bar(y) { return x + y; }
return bar;
}
function main() {
var f = foo(1);
var g = foo(10);
return [f(2), g(2)];
}
Note that the above definition of `foo` does *not* use an anonymous
"lambda expression"  in Racket terms, it's translated to
(define (foo x)
(define (bar y) (+ x y))
bar)
The returned function is not anonymous, but it's not really named
either: the `bar` name is bound only inside the body of `foo`, and
outside of it that name no longer exists since it's not its scope. It
gets used in the printed form if the function value is displayed, but
this is merely a debugging aid. The anonymous `lambda` version that is
common in Racket can be used in JavaScript too:
function foo(x) {
return function(y) { return x + y; }
}
> Sidenote: GCC includes extensions that allow internal function
> definitions, but it still does not have first class functions 
> trying to do the above is broken:
>
> #include
> typedef int(*int2int)(int);
> int2int foo(int x) {
> int bar(int y) { return x + y; }
> return bar;
> }
> int main() {
> int2int f = foo(1);
> int2int g = foo(10);
> printf(">> %d, %d\n", f(2), g(2));
> }

## Sidenote: how important is it to have *anonymous* functions?
You'll see many places where people refer to the feature of firstclass
functions as the ability to create *anonymous* functions, but this is a
confusion and it's not accurate. Whether a function has a name or not
is not the important question  instead, the important question is
whether functions can exist with no *bindings* that refers to them.
As a quick example in Racket:
(define (foo x)
(define (bar y) (+ x y))
bar)
in Javascript:
function foo(x) {
function bar(y) {
return x + y;
}
return bar;
}
and in Python:
def foo(x):
def bar(y):
return x + y
return bar
In all three of these, we have a `foo` function that returns a function
*named* `bar`  but the `bar` name, is only available in the scope of
`foo`. The fact that the name is displayed as part of the textual
rendering of the function value is merely a debugging feature.

# The FLANG Language
Now for the implementation  we call this new language FLANG.
First, the BNF:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
And the matching type definition:
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG] ; No namedexpression
[Call FLANG FLANG])
The parser for this grammar is, as usual, straightforward:
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
We also need to patch up the substitution function to deal with these
things. The scoping rule for the new function form is, unsurprisingly,
similar to the rule of `with`, except that there is no extra expression
now, and the scoping rule for `call` is the same as for the arithmetic
operators:
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]}
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
{call E1 E2}[v/x] = {call E1[v/x] E2[v/x]}
{fun {y} E}[v/x] = {fun {y} E[v/x]}
{fun {x} E}[v/x] = {fun {x} E}
And the matching code:
(: subst : FLANG Symbol FLANG > FLANG)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]
[(Call l r) (Call (subst l from to) (subst r from to))]
[(Fun boundid boundbody)
(if (eq? boundid from)
expr
(Fun boundid (subst boundbody from to)))]))

Now, before we start working on an evaluator, we need to decide on what
exactly do we use to represent values of this language. Before we had
functions, we had only number values and we used Racket numbers to
represent them. Now we have two kinds of values  numbers and
functions. It seems easy enough to continue using Racket numbers to
represent numbers, but what about functions? What should be the result
of evaluating
{fun {x} {+ x 1}}
? Well, this is the new toy we have: it should be a function value,
which is something that can be used just like numbers, but instead of
arithmetic operations, we can `call` these things. What we need is a
way to avoid evaluating the body expression of the function  *delay*
it  and instead use some value that will contain this delayed
expression in a way that can be used later.
To accommodate this, we will change our implementation strategy a
little: we will use our syntax objects for numbers (`(Num n)` instead of
just `n`), which will be a little inconvenient when we do the arithmetic
operations, but it will simplify life by making it possible to evaluate
functions in a similar way: simply return their own syntax object as
their values. The syntax object has what we need: the body expression
that needs to be evaluated later when the function is called, and it
also has the identifier name that should be replaced with the actual
input to the function call. This means that evaluating:
(Add (Num 1) (Num 2))
now yields
(Num 3)
and a number `(Num 5)` evaluates to `(Num 5)`.
In a similar way, `(Fun 'x (Num 2))` evaluates to `(Fun 'x (Num 2))`.
Why would this work? Well, because `call` will be very similar to
`with`  the only difference is that its arguments are ordered a
little differently, being retrieved from the function that is applied
and the argument.
The formal evaluation rules are therefore treating functions like
numbers, and use the syntax object to represent both values:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2)
eval({ E1 E2}) = eval(E1)  eval(E2)
eval({* E1 E2}) = eval(E1) * eval(E2)
eval({/ E1 E2}) = eval(E1) / eval(E2)
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(FUN) = FUN ; assuming FUN is a function expression
eval({call E1 E2})
= eval(Ef[eval(E2)/x]) if eval(E1) = {fun {x} Ef}
= error! otherwise
Note that the last rule could be written using a translation to a `with`
expression:
eval({call E1 E2})
= eval({with {x E2} Ef}) if eval(E1) = {fun {x} Ef}
= error! otherwise
And alternatively, we could specify `with` using `call` and `fun`:
eval({with {x E1} E2}) = eval({call {fun {x} E2} E1})
There is a small problem in these rules which is intuitively seen by the
fact that the evaluation rule for a `call` is expected to be very
similar to the one for arithmetic operations. We now have two kinds of
values, so we need to check the arithmetic operation's arguments too:
eval({+ E1 E2}) = N1 + N2
if eval(E1), eval(E2) evaluate to numbers N1, N2
otherwise error!
...
The corresponding code is:
(: eval : FLANG > FLANG) ;*** note return type
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr] ;*** change here
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))] ;*** no `(Num ...)'
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr] ;*** similar to `Num'
[(Call (Fun boundid boundbody) argexpr) ;*** nested pattern
(eval (subst boundbody ;*** just like `with'
boundid
(eval argexpr)))]
[(Call something argexpr)
(error 'eval "`call' expects a function, got: ~s" something)]))
Note that the `Call` case is doing the same thing we do in the `With`
case. In fact, we could have just *generated* a `With` expression and
evaluate that instead:
...
[(Call (Fun boundid boundbody) argexpr)
(eval (With boundid argexpr boundbody))]
...
The `arithop` function is in charge of checking that the input values
are numbers (represented as FLANG numbers), translating them to plain
numbers, performing the Racket operation, then rewrapping the result in
a `Num`. Note how its type indicates that it is a higherorder
function.
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper (note the H.O. type, and note the hack of the `val`
;; name which is actually an AST that represents a runtime value)
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
It uses the following function to convert FLANG numbers to Racket
numbers. (Note that `else` is almost always a bad idea since it can
prevent the compiler from showing you places to edit code  but this
case is an exception since we never want to deal with anything other
than `Num`s.) The reason that this function is relatively trivial is
that we chose the easy way and represented numbers using Racket numbers,
but we could have used strings or anything else.
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
We can also make things a little easier to use if we make `run` convert
the result to a number:
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str))])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
Adding few simple tests we get:
;; The Flang interpreter
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
subst:
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]} ; if y =/= x
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
{call E1 E2}[v/x] = {call E1[v/x] E2[v/x]}
{fun {y} E}[v/x] = {fun {y} E[v/x]} ; if y =/= x
{fun {x} E}[v/x] = {fun {x} E}
eval:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2) \ if both E1 and E2
eval({ E1 E2}) = eval(E1)  eval(E2) \ evaluate to numbers
eval({* E1 E2}) = eval(E1) * eval(E2) / otherwise error!
eval({/ E1 E2}) = eval(E1) / eval(E2) /
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(FUN) = FUN ; assuming FUN is a function expression
eval({call E1 E2}) = eval(Ef[eval(E2)/x])
if eval(E1)={fun {x} Ef}, otherwise error!
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
(: subst : FLANG Symbol FLANG > FLANG)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]
[(Call l r) (Call (subst l from to) (subst r from to))]
[(Fun boundid boundbody)
(if (eq? boundid from)
expr
(Fun boundid (subst boundbody from to)))]))
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
(: eval : FLANG > FLANG)
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))]
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr]
[(Call (Fun boundid boundbody) argexpr)
(eval (subst boundbody
boundid
(eval argexpr)))]
[(Call something argexpr)
(error 'eval "`call' expects a function, got: ~s" something)]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str))])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)

There is still a problem with this version. First a question  if
`call` is similar to arithmetic operations (and to `with` in what it
actually does), then how come the code is different enough that it
doesn't even need an auxiliary function?
Second question: what *should* happen if we evaluate these code
snippets:
(run "{with {add {fun {x}
{fun {y}
{+ x y}}}}
{call {call add 8} 9}}")
(run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
(run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
Third question, what *will* happen if we do the above?
What we're missing is an evaluation of the function expression, in case
it's not a literal `fun` form. The following fixes this:
(: eval : FLANG > FLANG)
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))]
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr)]) ;*** need to evaluate this!
(cases fval
[(Fun boundid boundbody)
(eval (subst boundbody
boundid
(eval argexpr)))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
The complete code is:
;;; <<>>
;; The Flang interpreter
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
subst:
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]} ; if y =/= x
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
{call E1 E2}[v/x] = {call E1[v/x] E2[v/x]}
{fun {y} E}[v/x] = {fun {y} E[v/x]} ; if y =/= x
{fun {x} E}[v/x] = {fun {x} E}
eval:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2) \ if both E1 and E2
eval({ E1 E2}) = eval(E1)  eval(E2) \ evaluate to numbers
eval({* E1 E2}) = eval(E1) * eval(E2) / otherwise error!
eval({/ E1 E2}) = eval(E1) / eval(E2) /
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(FUN) = FUN ; assuming FUN is a function expression
eval({call E1 E2}) = eval(Ef[eval(E2)/x])
if eval(E1)={fun {x} Ef}, otherwise error!
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
(: subst : FLANG Symbol FLANG > FLANG)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]
[(Call l r) (Call (subst l from to) (subst r from to))]
[(Fun boundid boundbody)
(if (eq? boundid from)
expr
(Fun boundid (subst boundbody from to)))]))
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
(: eval : FLANG > FLANG)
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))]
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr)])
(cases fval
[(Fun boundid boundbody)
(eval (subst boundbody
boundid
(eval argexpr)))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str))])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {add {fun {x}
{fun {y}
{+ x y}}}}
{call {call add 8} 9}}")
=> 17)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Introducing Racket's `lambda`
`fun` & `lambda`
difference between lambda and simple values
not being able to do recursive functions with `let`
let* as a derived form
let with lambda in Racket > can be a derived form
how `if` can be used to implement `and` and `or` as derived forms
Newtonian syntax vs. a lambda expression.
Don't be fooled into making a bogus connection between Racket's syntax,
and its `unique` powers... The fact is that it is not the only language
that has this capability. For example, this:
(define (f g) (g 2 3))
(f +) ==> 5
(f *) ==> 6
(f (lambda (x y) (+ (square x) (square y)))) ==> 13
Can be written in JavaScript like this:
function f(g) { return g(2,3); }
function square(x) { return x*x; }
console.log(f(function (x,y) { return square(x) + square(y); }));
or in ES6 JavaScript:
let f = (g) => g(2,3);
let square = (x) => x*x;
console.log(f((x,y) => square(x) + square(y)));
In Perl:
sub f { my ($g) = @_; return $g>(2,3); }
sub square { my ($x) = @_; return $x * $x; }
print f(sub { my ($x, $y) = @_; return square($x) + square($y); });
In Ruby:
def f(g) g.call(2,3) end
def square(x) x*x end
puts f(lambda{x,y square(x) + square(y)})
etc. Even [Java has lambda expressions], and recently
[C++ added them too].
[Java has lambda expressions]:
http://www.drdobbs.com/jvm/lambdaexpressionsinjava8/240166764
[C++ added them too]:
http://www.cprogramming.com/c++11/c++11lambdaclosures.html

# Using Functions as Objects
A very important aspect of Racket  using "higher order" functions 
functions that get and return functions. Here is a very simple example:
(define (f x) (lambda () x))
(define a (f 2))
(a) > 2
(define b (f 3))
(b) > 3
Note: what we get is actually an object that remembers (by the
substitution we're doing) a number. How about:
(define aa (f a))
(aa) > # (this is a)
((aa)) > 2
Take this idea to the next level:
(define (kons x y)
(lambda (b)
(if b x y)))
(define (kar p) (p #t))
(define (kdr p) (p #f))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
Or, with types:
(: kons : (All (A B) A B > (Boolean > (U A B))))
(define (kons x y)
(lambda (b)
(if b x y)))
(: kar : (All (T) (Boolean > T) > T))
(define (kar p) (p #t))
(: kdr : (All (T) (Boolean > T) > T))
(define (kdr p) (p #f))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
Even more  why should the internal function expect a boolean and
choose what to return? We can simply expect a function that will take
the two values and return one:
(define (kons x y) (lambda (s) (s x y)))
(define (kar p) (p (lambda (x y) x)))
(define (kdr p) (p (lambda (x y) y)))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
And a typed version, using our own constructor to make it a little less
painful:
(definetype (Kons A B) = ((A B > (U A B)) > (U A B)))
(: kons : (All (A B) A B > (Kons A B)))
(define (kons x y) (lambda (s) (s x y)))
(: kar : (All (A B) (Kons A B) > (U A B)))
(define (kar p) (p (lambda (x y) x)))
(: kdr : (All (A B) (Kons A B) > (U A B)))
(define (kdr p) (p (lambda (x y) y)))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
Note that the `Kons` type definition is the same as:
(definetype Kons = (All (A B) (A B > (U A B)) > (U A B)))
so `All` is to polymorphic type definitions what `lambda` is for
function definitions.
Finally in JavaScript:
function kons(x,y) { return function(s) { return s(x, y); } }
function kar(p) { return p(function(x,y){ return x; }); }
function kdr(p) { return p(function(x,y){ return y; }); }
a = kons(1,2);
b = kons(3,4);
console.log('a = <' + kar(a) + ',' + kdr(a) + '>' );
console.log('b = <' + kar(b) + ',' + kdr(b) + '>' );
Or with ES6 *arrow functions*, the function definitionss become:
const kons = (x,y) => s => s(x,y);
const kar = p => p((x,y) => x);
const kdr = p => p((x,y) => y);

# Using `definetype` for new "type aliases"
As seen in these examples, there is another way to use `definetype`,
using a `=` to create a new type name "alias" for an *existing* type.
For example:
(definetype Strings = (Listof String))
These uses of `definetype` do not define any new kind of type, they are
essentially a convenience tool for making code shorter and more
readable.
(definetype NumericFunction = Number > Number)
(: square : NumericFunction)
(define (square n) (* n n))
Note in particular that this can also be used to define "alias type
constructors" too: somewhat similar to creating new "type functions".
For example:
(definetype (BinaryFun In Out) = In In > Out)
(: diagonal : (BinaryFun Natural Number))
(define (diagonal width height)
(sqrt (+ (* width width) (* height height))))
This is something that we will only need in a few rare cases.

# Currying
A *curried* function is a function that, instead of accepting two (or
more) arguments, accepts only one and returns a function that accepts
the rest. For example:
(: plus : Number > (Number > Number))
(define (plus x)
(lambda (y)
(+ x y)))
It's easy to write functions for translating between normal and curried
versions.
(define (currify f)
(lambda (x)
(lambda (y)
(f x y))))
Typed version of that, with examples:
(: currify : (All (A B C) (A B > C) > (A > (B > C))))
;; convert a doubleargument function to a curried one
(define (currify f)
(lambda (x) (lambda (y) (f x y))))
(: add : Number Number > Number)
(define (add x y) (+ x y))
(: plus : Number > (Number > Number))
(define plus (currify add))
(test ((plus 1) 2) => 3)
(test (((currify add) 1) 2) => 3)
(test (map (plus 1) '(1 2 3)) => '(2 3 4))
(test (map ((currify add) 1) '(1 2 3)) => '(2 3 4))
(test (map ((currify +) 1) '(1 2 3)) => '(2 3 4))
Usages  common with H.O. functions like map, where we want to *fix*
one argument.
When dealing with such higherorder code, the types are very helpful,
since every arrow corresponds to a function:
(: currify : (All (A B C) (A B > C) > (A > (B > C))))
It is common to make the `>` function type associate to the right, so
you can find this type written as:
currify : (A B > C) > (A > B > C)
or even as
currify : (A B > C) > A > B > C
but that can be a little confusing...

# Using HigherOrder & Anonymous Functions
Say that we have a function for estimating derivatives of a function at
a specific point:
(define dx 0.01)
(: deriv : (Number > Number) Number > Number)
;; compute the derivative of `f' at the given point `x'
(define (deriv f x)
(/ ( (f (+ x dx)) (f x)) dx))
(: integrate : (Number > Number) Number > Number)
;; compute an integral of `f' at the given point `x'
(define (integrate f x)
(: loop : Number Number > Number)
(define (loop y acc)
(if (> y x)
(* acc dx)
(loop (+ y dx) (+ acc (f y)))))
(loop 0 0))
And say that we want to try out various functions given some `plot`
function that draws graphs of numeric functions, for example:
(plot sin)
The problem is that `plot` expects a single `(Number > Number)`
function  if we want to try it with a derivative, we can do this:
(: sinderiv : Number > Number)
;; the derivative of sin
(define sinderiv (lambda (x) (deriv sin x)))
(plot sinderiv)
But this will get very tedious very fast  it is much simpler to use
an anonymous function:
(plot (lambda (x) (deriv sin x)))
we can even verify that our derivative is correct by comparing a known
function to its derivative
(plot (lambda (x) ( (deriv sin x) (cos x))))
But it's still not completely natural to do these things  you need to
explicitly combine functions, which is not too convenient. Instead of
doing this, we can write H.O. functions that will work with functional
inputs and outputs. For example, we can write a function to subtract
functions:
(: fsub : (Number > Number) (Number > Number)
> (Number > Number))
;; subtracts two numeric 1argument functions
(define (fsub f g)
(lambda (x) ( (f x) (g x))))
and the same for the derivative:
(: fderiv : (Number > Number) > (Number > Number))
;; compute the derivative function of `f'
(define (fderiv f)
(lambda (x) (deriv f x)))
Now we can try the same in a much easier way:
(plot (fsub (fderiv sin) cos))
More than that  our `fderiv` could be created from `deriv`
automatically:
(: currify : (All (A B C) (A B > C) > (A > B > C)))
;; convert a doubleargument function to a curried one
(define (currify f)
(lambda (x) (lambda (y) (f x y))))
(: fderiv : (Number > Number) > (Number > Number))
;; compute the derivative function of `f'
(define fderiv (currify deriv))
Same principle with `fsub`: we can write a function that converts a
binary arithmetical function into a function that operates on unary
numeric function. But to make things more readable we can define new
types for unary and binary numeric functions:
(definetype UnaryFun = (Number > Number))
(definetype BinaryFun = (Number Number > Number))
(: binop>fbinop : BinaryFun > (UnaryFun UnaryFun > UnaryFun))
;; turns an arithmetic binary operator to a function operator
(define (binop>fbinop op)
(lambda (f g)
(lambda (x) (op (f x) (g x)))))
(: fsub : UnaryFun UnaryFun > UnaryFun)
;; functional pointwise subtraction
(define fsub (binop>fbinop ))
We can do this with anything  developing a rich library of functions
and functionals (functions over functions) is extremely easy... Here's
a pretty extensive yet very short library of functions:
#lang pl untyped
(define (currify f)
(lambda (x) (lambda (y) (f x y))))
(define (binop>fbinop op)
(lambda (f g)
(lambda (x) (op (f x) (g x)))))
(define (compose f g)
(lambda (x) (f (g x))))
(define dx 0.01)
(define (deriv f x)
(/ ( (f (+ x dx)) (f x)) dx))
(define (integrate f x)
(define over? (if (< x 0) < >))
(define step (if (< x 0)  +))
(define add (if (< x 0)  +))
(define (loop y acc)
(if (over? y x)
(* acc dx)
(loop (step y dx) (add acc (f y)))))
(loop 0 0))
(define fadd (binop>fbinop +))
(define fsub (binop>fbinop ))
(define fmul (binop>fbinop *))
(define fdiv (binop>fbinop /))
(define fderiv (currify deriv))
(define fintegrate (currify integrate))
;; ...
This is written in the "untyped dialect" of the class language, but it
should be easy now to add the types.
Examples:
;; want to verify that `integrate' is the opposite of `deriv':
;; take a function, subtract it from its derivative's integral
(plot (fsub sin (fintegrate (fderiv sin))))
;; want to magnify the errors?  here's how you magnify:
(plot (compose ((currify *) 5) sin))
;; so:
(plot (compose ((currify *) 20)
(fsub sin (fintegrate (fderiv sin)))))

## Sidenote: "PointFree" combinators
> Forming functions without using `lambda` (or an implicit `lambda`
> using a `define` syntactic sugar) is called *pointfree style*. It's
> especially popular in Haskell, where it is easier to form functions
> this way because of implicit currying and a large number of higher
> level function combinators. If used too much, it can easily lead to
> obfuscated code.

## This is not Runtime Code Generation
All of this is similar to runtime code generation, but not really. The
only thing that `fderiv` does is take a function and store it somewhere
in the returned function, then when that function receives a number, it
uses the stored function and send it to deriv with the number. We could
simply write deriv as what `fderiv` is  which is the *real*
derivative function:
(define (deriv f)
(lambda (x)
(/ ( (f (+ x dx)) (f x)) dx)))
but again, this is not faster or slower than the plain `deriv`.
However, there are some situations where we can do some of the
computation on the firststage argument, saving work from the second
stage. Here is a cookedtoexaggeration example  we want a function
that receives two inputs `x`, `y` and returns `fib(x)*y`, but we must
use a stupid `fib`:
(define (fib n)
(if (<= n 1)
n
(+ (fib ( n 1)) (fib ( n 2)))))
The function we want is:
(define (bogus x y)
(* (fib x) y))
If we currify it as usual (or just use `currify`), we get:
(define (bogus x)
(lambda (y)
(* (fib x) y)))
And try this several times:
(define bogus36 (bogus 36))
(map bogus36 '(1 2 3 4 5))
But in the definition of `bogus`, notice that `(fib x)` does not depend
on `y`  so we can rewrite it a little differently:
(define (bogus x)
(let ([fibx (fib x)])
(lambda (y)
(* fibx y))))
and trying the above again is much faster now:
(define bogus36 (bogus 36))
(map bogus36 '(1 2 3 4 5))
This is therefore not doing any kind of runtime code generation, but it
*enables* doing similar optimizations in our code. A proper RTCG
facility would recompile the curried function for a given first input,
and (hopefully) automatically achieve the optimization that we did in a
manual way.

# Substitution Caches
> [PLAI §5] (called "deferred substitutions" there)
Evaluating using substitutions is very inefficient  at each scope, we
copy a piece of the program AST. This includes all function calls which
implies an impractical cost (function calls should be *cheap*!).
To get over this, we want to use a cache of substitutions.
Basic idea: we begin evaluating with no cached substitutions, then
collect them as we encounter bindings.
[Implies another change for our evaluator: we don't really substitute
identifiers until we get to them; when we reach an identifier, it is no
longer an error  we must consult the substitution cache.]

# Implementation of Cache Functionality
First, we need a type for a substitution cache. For this we will use a
list of lists of two elements each  a name and its value FLANG:
;; a type for substitution caches:
(definetype SubstCache = (Listof (List Symbol FLANG)))
We need to have an empty substitution cache, a way to extend it, and a
way to look things up:
(: emptysubst : SubstCache)
(define emptysubst null)
(: extend : Symbol FLANG SubstCache > SubstCache)
;; extend a given substitution cache with a new mapping
(define (extend id expr sc)
(cons (list id expr) sc))
(: lookup : Symbol SubstCache > FLANG)
;; lookup a symbol in a substitution cache, return the value it is
;; bound to (or throw an error if it isn't bound)
(define (lookup name sc)
(cond [(null? sc) (error 'lookup "no binding for ~s" name)]
[(eq? name (first (first sc))) (second (first sc))]
[else (lookup name (rest sc))]))
Actually, the reason to use such list of lists is that Racket has a
builtin function called `assq` that will do this kind of search (`assq`
is a search in an association list using `eq?` for the key comparison).
This is a version of `lookup` that uses `assq`:
(define (lookup name sc)
(let ([cell (assq name sc)])
(if cell
(second cell)
(error 'lookup "no binding for ~s" name))))

# Formal Rules for Cached Substitutions
The formal evaluation rules are now different. Evaluation carries along
a *substitution cache* that begins its life as empty: so `eval` needs an
extra argument. We begin by writing the rules that deal with the cache,
and use the above function names for simplicity  the behavior of the
three definitions can be summed up in a single rule for `lookup`:
lookup(x,emptysubst) = error!
lookup(x,extend(x,E,sc)) = E
lookup(x,extend(y,E,sc)) = lookup(x,sc) if `x' is not `y'
And now we can write the new rules for `eval`
eval(N,sc) = N
eval({+ E1 E2},sc) = eval(E1,sc) + eval(E2,sc)
eval({ E1 E2},sc) = eval(E1,sc)  eval(E2,sc)
eval({* E1 E2},sc) = eval(E1,sc) * eval(E2,sc)
eval({/ E1 E2},sc) = eval(E1,sc) / eval(E2,sc)
eval(x,sc) = lookup(x,sc)
eval({with {x E1} E2},sc) = eval(E2,extend(x,eval(E1,sc),sc))
eval({fun {x} E},sc) = {fun {x} E}
eval({call E1 E2},sc)
= eval(Ef,extend(x,eval(E2,sc),sc))
if eval(E1,sc) = {fun {x} Ef}
= error! otherwise
Note that there is no mention of `subst`  the whole point is that we
don't really do substitution, but use the cache instead. The `lookup`
rules, and the places where `extend` is used replaces `subst`, and
therefore specifies our scoping rules.
Also note that the rule for `call` is still very similar to the rule for
`with`, but it looks like we have lost something  the interesting bit
with substituting into `fun` expressions.

# Evaluating with Substitution Caches
Implementing the new `eval` is easy now  it is extended in the same
way that the formal `eval` rule is extended:
(: eval : FLANG SubstCache > FLANG)
;; evaluates FLANG expressions by reducing them to expressions
(define (eval expr sc)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l sc) (eval r sc))]
[(Sub l r) (arithop  (eval l sc) (eval r sc))]
[(Mul l r) (arithop * (eval l sc) (eval r sc))]
[(Div l r) (arithop / (eval l sc) (eval r sc))]
[(With boundid namedexpr boundbody)
(eval boundbody
(extend boundid (eval namedexpr sc) sc))]
[(Id name) (lookup name sc)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr sc)])
(cases fval
[(Fun boundid boundbody)
(eval boundbody
(extend boundid (eval argexpr sc) sc))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
Again, note that we don't need `subst` anymore, but the rest of the code
(the data type definition, parsing, and `arithop`) is exactly the same.
Finally, we need to make sure that `eval` is initially called with an
empty cache. This is easy to change in our main `run` entry point:
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) emptysubst)])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
The full code (including the same tests, but not including formal rules
for now) follows. Note that one test does not pass.
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; a type for substitution caches:
(definetype SubstCache = (Listof (List Symbol FLANG)))
(: emptysubst : SubstCache)
(define emptysubst null)
(: extend : Symbol FLANG SubstCache > SubstCache)
;; extend a given substitution cache with a new mapping
(define (extend name val sc)
(cons (list name val) sc))
(: lookup : Symbol SubstCache > FLANG)
;; lookup a symbol in a substitution cache, return the value it is
;; bound to (or throw an error if it isn't bound)
(define (lookup name sc)
(let ([cell (assq name sc)])
(if cell
(second cell)
(error 'lookup "no binding for ~s" name))))
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
(: eval : FLANG SubstCache > FLANG)
;; evaluates FLANG expressions by reducing them to expressions
(define (eval expr sc)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l sc) (eval r sc))]
[(Sub l r) (arithop  (eval l sc) (eval r sc))]
[(Mul l r) (arithop * (eval l sc) (eval r sc))]
[(Div l r) (arithop / (eval l sc) (eval r sc))]
[(With boundid namedexpr boundbody)
(eval boundbody
(extend boundid (eval namedexpr sc) sc))]
[(Id name) (lookup name sc)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr sc)])
(cases fval
[(Fun boundid boundbody)
(eval boundbody
(extend boundid (eval argexpr sc) sc))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) emptysubst)])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> "???")
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Dynamic and Lexical Scopes
This seems like it should work, and it even worked on a few examples,
except for one which was hard to follow. Seems like we have a bug...
Now we get to a tricky issue that managed to be a problem for *lots* of
language implementors, including the first version of Lisp. Lets try to
run the following expression  try to figure out what it will evaluate
to:
(run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
We expect it to return `7` (at least I do!), but we get `9` instead...
The question is  *should* it return `9`?
What we have arrived to is called *dynamic scope*. Scope is determined
by the dynamic runtime environment (which is represented by our
substitution cache). This is *almost always* undesirable, as I hope to
convince you.
Before we start, we define two scope options for a programming language:
* Static Scope (also called Lexical Scope): In a language with static
scope, each identifier gets its value from the scope of its
definition, not its use.
* Dynamic Scope: In a language with dynamic scope, each identifier gets
its value from the scope of its use, not its definition.
Racket uses lexical scope, our new evaluator uses dynamic, the old
substitutionbased evaluator was static etc.
As a sideremark, Lisp began its life as a dynamicallyscoped language.
The artifacts of this were (sortof) dismissed as an implementation bug.
When Scheme was introduced, it was the first Lisp dialect that used
strictly lexical scoping, and Racket is obviously doing the same. (Some
Lisp implementations used dynamic scope for interpreted code and lexical
scope for compiled code!) In fact, Emacs Lisp is the only *live*
dialects of Lisp that is still dynamically scoped by default. To see
this, compare a version of the above code in Racket:
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(f 4))))
and the Emacs Lisp version (which looks almost the same):
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(funcall f 4))))
which also happens when we use another function on the way:
(defun blah (func val)
(funcall func val))
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(blah f 4))))
and note that renaming identifiers can lead to different code  change
that `val` to `x`:
(defun blah (func x)
(funcall func x))
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(blah f 4))))
and you get `8` because the argument name changed the `x` that the
internal function sees!
Consider also this Emacs Lisp function:
(defun returnx ()
x)
which has no meaning by itself (`x` is unbound),
(returnx)
but can be given a dynamic meaning using a `let`:
(let ((x 5)) (returnx))
or a function application:
(defun foo (x)
(returnx))
(foo 5)
There is also a dynamicallyscoped language in the course languages:
#lang pl dynamic
(define x 123)
(define (getx) x)
(define (bar1 x) (getx))
(define (bar2 y) (getx))
(test (getx) => 123)
(test (let ([x 456]) (getx)) => 456)
(test (getx) => 123)
(test (bar1 999) => 999)
(test (bar2 999) => 123)
(define (foo x) (define (helper) (+ x 1)) helper)
(test ((foo 0)) => 124)
;; and *much* worse:
(define (add x y) (+ x y))
(test (let ([+ *]) (add 6 7)) => 42)
Note how bad the last example gets: you basically cannot call any
function and know in advance what it will do.
There are some cases where dynamic scope can be useful in that it allows
you to "remotely" customize any piece of code. A good example of where
this is taken to an extreme is Emacs: originally, it was based on an
ancient Lisp dialect that was still dynamically scoped, but it retained
this feature even when practically all Lisp dialects moved on to having
lexical scope by default. The reason for this is that the danger of
dynamic scope is also a way to make a very open system where almost
anything can be customized by changing it "remotely". Here's a concrete
example for a similar kind of dynamic scope usage that makes a very
hackable and open system:
#lang pl dynamic
(define tax% 6.25)
(define (withtax n)
(+ n (* n (/ tax% 100))))
(withtax 10) ; how much do we pay?
(let ([tax% 17.0]) (withtax 10)) ; how much would we pay in Israel?
;; make that into a function
(define iltax% 17.0)
(define (maoverilsaving n)
( (let ([tax% iltax%]) (withtax n))
(withtax n)))
(maoverilsaving 10)
;; can even control that: how much would we save if
;; the tax in israel went down one percent?
(let ([iltax% ( iltax% 1)]) (maoverilsaving 10))
;; or change both: how much savings in NH instead of MA?
(let ((tax% 0.0) (iltax% tax%)) (maoverilsaving 1000))
Obviously, this power to customize everything is also the main source of
problems with getting no guarantees for code. A common way to get the
best of both worlds is to have *controllable* dynamic scope. For
example, Common Lisp also has lexical scope everywhere by default, but
some variables can be declared as *special*, which means that they are
dynamically scoped. The main problem with that is that you can't tell
when a variable is special by just looking at the code that uses it, so
a more popular approach is the one that is used in Racket: all bindings
are always lexically scoped, but there are *parameters* which are a kind
of dynamically scoped value containers  but they are bound to plain
(lexically scoped) identifiers. Here's the same code as above,
translated to Racket with parameters:
#lang racket
(define tax% (makeparameter 6.5)) ; create the dynamic container
(define (withtax n)
(+ n (* n (/ (tax%) 100)))) ; note how its value is accessed
(withtax 10) ; how much do we pay?
(parameterize ([tax% 17.0]) (withtax 10)) ; not a `let'
;; make that into a function
(define iltax% (makeparameter 17.0))
(define (maoverilsaving n)
( (parameterize ([tax% (iltax%)]) (withtax n))
(withtax n)))
(maoverilsaving 10)
(parameterize ([iltax% ( (iltax%) 1)]) (maoverilsaving 10))
The main point here is that the points where a dynamically scoped value
is used are under the programmer's control  you cannot "customize"
what `` is doing, for example. This gives us back the guarantees that
we like to have (= that code works), but of course these points are
predetermined, unlike an environment where everything can be customized
including things that are unexpectedly useful.
> As a sidenote, after many decades of debating this, Emacs has finally
> added lexical scope in its core language, but this is still determined
> by a flag  a global `lexicalbinding` variable.

# Dynamic versus Lexical Scope
And back to the discussion of whether we should use dynamic or lexical
scope:
* The most important fact is that we want to view programs as executed
by the normal substituting evaluator. Our original motivation was to
optimize evaluation only  not to *change* the semantics! It
follows that we want the result of this optimization to behave in the
same way. All we need is to evaluate:
(run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
in the original evaluator to get convinced that `7` should be the
correct result (note also that the same code, when translated into
Racket, evaluates to `7`).
(Yet, this is a very important optimization, which without it lots of
programs become too slow to be feasible, so you might claim that
you're fine with the modified semantics...)
* It does not allow using functions as objects, for example, we have
seen that we have a functional representation for pairs:
(define (kons x y)
(lambda (n)
(match n
['first x]
['second y]
[else (error ...)])))
(define mypair (kons 1 2))
If this is evaluated in a dynamicallyscoped language, we do get a
function as a result, but the values bound to `x` and `y` are now
gone! Using the substitution model we substituted these values in,
but now they were only held in a cache which no has no entries for
them...
In the same way, currying would not work, our nice `deriv` function
would not work etc etc etc.
* Makes reasoning impossible, because any piece of code behaves in a way
that *cannot* be predicted until runtime. For example, if dynamic
scoping was used in Racket, then you wouldn't be able to know what
this function is doing:
(define (foo)
x)
As it is, it will cause a runtime error, but if you call it like
this:
(let ([x 1])
(foo))
then it will return `1`, and if you later do this:
(define (bar x)
(foo))
(let ([x 1])
(bar 2))
then you would get `2`!
These problems can be demonstrated in Emacs Lisp too, but Racket goes
one step further  it uses the same rule for evaluating a function
as well as its values (Lisp uses a different namespace for
functions). Because of this, you cannot even rely on the following
function:
(define (add x y)
(+ x y))
to always add `x` and `y`!  A similar example to the above:
(let ([+ ])
(add 1 2))
would return `1`!
* Many socalled "scripting" languages begin their lives with dynamic
scoping. The main reason, as we've seen, is that implementing it is
extremely simple (no, *nobody* does substitution in the real world!
(Well, *almost* nobody...)).
Another reason is that these problems make life impossible if you want
to use functions as object like you do in Racket, so you notice them
very fast  but in a `normal` language without firstclass
functions, problems are not as obvious.
* For example, bash has `local` variables, but they have dynamic scope:
x="the global x"
print_x() { echo "The current value of x is \"$x\""; }
foo() { local x="x from foo"; print_x; }
print_x; foo; print_x
Perl began its life with dynamic scope for variables that are declared
`local`:
$x="the global x";
sub print_x { print "The current value of x is \"$x\"\n"; }
sub foo { local($x); $x="x from foo"; print_x; }
print_x; foo; print_x;
When faced with this problem, "the Perl way" was, obviously, not to
remove or fix features, but to pile them up  so `local` *still*
behaves in this way, and now there is a `my` declaration which
achieves proper lexical scope (and every serious Perl programmer knows
that you should always use `my`)...
There are other examples of languages that changed, and languages that
want to change (e.g, nobody likes dynamic scope in Emacs Lisp, but
there's just too much code now).
* This is still a tricky issue, like any other issue with bindings. For
example, googling got me quickly to [a Python blog post] which is
confused about what "dynamic scoping" is... It claims that Python
uses dynamic scope (Search for "Python uses dynamic as opposed to
lexical scoping"), yet python always used lexical scope rules, as can
be seen by translating their code to Racket (ignore sideeffects in
this computation):
(define (orangejuice)
(* x 2))
(define x 3)
(define y (orangejuice)) ; y is now 6
(define x 1)
(define y (orangejuice)) ; y is now 2
or by trying this in Python:
def orange_juice():
return x*2
def foo(x):
return orange_juice()
foo(2)
The real problem of python (pre 2.1, and pre 2.2 without the funny
from __future__ import nested_scope
line) is that it didn't create closures, which we will talk about
shortly.
[a Python blog post]:
https://folk.idi.ntnu.no/mlh/hetland_org/writing/instantpython.html
* Another example, which is an indicator of how easy it is to mess up
your scope is the following Ruby bug  running in `irb`:
% irb
irb(main):001:0> x = 0
=> 0
irb(main):002:0> lambda{x x}.call(5)
=> 5
irb(main):003:0> x
=> 5
(This is a bug due to weird scoping rules for variables, which was
fixed in newer versions of Ruby. See [this Ruby rant] for details, or
read about [Ruby and the principle of unwelcome surprise] for
additional gems (the latter is gone, so you'll need the [web
archive](https://archive.org/) to read it).)
[this Ruby rant]:
http://innig.net/software/ruby/closuresinruby
[Ruby and the principle of unwelcome surprise]:
http://ceaude.twoticketsplease.de/articles/rubyandtheprincipleofunwelcomesurprise.html
* Another thing to consider is the fact that compilation is something
that you do based only on the lexical structure of programs, since
compilers never actually run code. This means that dynamic scope
makes compilation close to impossible.
* There are some advantages for dynamic scope too. Two notable ones
are:
 Dynamic scope makes it easy to have a "configuration variable"
easily change for the extent of a calling piece of code (this is
used extensively in Emacs, for example). The thing is that usually
we want to control which variables are "configurable" in this way,
statically scoped languages like Racket often choose a separate
facility for these. To rephrase the problem of dynamic scoping,
it's that *all* variables are modifiable.
The same can be said about functions: it is sometimes desirable to
change a function dynamically (for example, see "Aspect Oriented
Programming"), but if there is no control and all functions can
change, we get a world where no code can every be reliable.
 It makes recursion immediately available  for example,
{with {f {fun {x} {call f x}}}
{call f 0}}
is an infinite loop with a dynamically scoped language. But in a
lexically scoped language we will need to do some more work to get
recursion going.

# Implementing Lexical Scope: Closures and Environments
So how do we fix this?
Lets go back to the root of the problem: the new evaluator does not
behave in the same way as the substituting evaluator. In the old
evaluator, it was easy to see how functions can behave as objects that
remember values. For example, when we do this:
{with {x 1}
{fun {y}
{+ x y}}}
the result was a function value, which actually was the syntax object
for this:
{fun {y} {+ 1 y}}
Now if we call this function from someplace else like:
{with {f {with {x 1} {fun {y} {+ x y}}}}
{with {x 2}
{call f 3}}}
it is clear what the result will be: f is bound to a function that adds
1 to its input, so in the above the later binding for `x` has no effect
at all.
But with the caching evaluator, the value of
{with {x 1}
{fun {y}
{+ x y}}}
is simply:
{fun {y} {+ x y}}
and there is no place where we save the 1  *that's* the root of our
problem. (That's also what makes people suspect that using `lambda` in
Racket and any other functional language involves some inefficient
coderecompiling magic.) In fact, we can verify that by inspecting the
returned value, and see that it does contain a free identifier.
Clearly, we need to create an object that contains the body and the
argument list, like the function syntax object  but we don't do any
substitution, so in addition to the body an argument name(s) we need to
remember that we still need to substitute `x` by `1` . This means that
the pieces of information we need to know are:
 formal argument(s): {y}
 body: {+ x y}
 pending substitutions: [1/x]
and that last bit has the missing `1`. The resulting object is called a
`closure` because it closes the function body over the substitutions
that are still pending (its environment).
So, the first change is in the value of functions which now need all
these pieces, unlike the `Fun` case for the syntax object.
A second place that needs changing is the when functions are called.
When we're done evaluating the `call` arguments (the function value and
the argument value) but before we apply the function we have two
*values*  there is no more use for the current substitution cache at
this point: we have finished dealing with all substitutions that were
necessary over the current expression  we now continue with
evaluating the body of the function, with the new substitutions for the
formal arguments and actual values given. But the body itself is the
same one we had before  which is the previous body with its suspended
substitutions that we *still* did not do.
Rewrite the evaluation rules  all are the same except for evaluating
a `fun` form and a `call` form:
eval(N,sc) = N
eval({+ E1 E2},sc) = eval(E1,sc) + eval(E2,sc)
eval({ E1 E2},sc) = eval(E1,sc)  eval(E2,sc)
eval({* E1 E2},sc) = eval(E1,sc) * eval(E2,sc)
eval({/ E1 E2},sc) = eval(E1,sc) / eval(E2,sc)
eval(x,sc) = lookup(x,sc)
eval({with {x E1} E2},sc) = eval(E2,extend(x,eval(E1,sc),sc))
eval({fun {x} E},sc) = <{fun {x} E}, sc>
eval({call E1 E2},sc1)
= eval(Ef,extend(x,eval(E2,sc1),sc2))
if eval(E1,sc1) = <{fun {x} Ef}, sc2>
= error! otherwise
As a side note, these substitution caches are a little more than "just a
cache" now  they actually hold an *environment* of substitutions in
which expression should be evaluated. So we will switch to the common
*environment* name now.:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2, env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
In case you find this easier to follow, the "flat algorithm" for
evaluating a `call` is:
1. f := evaluate E1 in env1
2. if f is not a <{fun ...},...> closure then error!
3. x := evaluate E2 in env1
4. new_env := extend env_of(f) by mapping arg_of(f) to x
5. evaluate (and return) body_of(f) in new_env
Note how the scoping rules that are implied by this definition match the
scoping rules that were implied by the substitutionbased rules. (It
should be possible to prove that they are the same.)
The changes to the code are almost trivial, except that we need a way to
represent `<{fun {x} Ef}, env>` pairs.

The implication of this change is that we now cannot use the same type
for function syntax and function values since function values have more
than just syntax. There is a simple solution to this  we never do
any substitutions now, so we don't need to translate values into
expressions  we can come up with a new type for values, separate from
the type of abstract syntax trees.
When we do this, we will also fix our hack of using FLANG as the type of
values: this was merely a convenience since the AST type had cases for
all kinds of values that we needed. (In fact, you should have noticed
that Racket does this too: numbers, strings, booleans, etc are all used
by both programs and syntax representation (sexpressions)  but note
that function values are *not* used in syntax.) We will now implement a
separate `VAL` type for runtime values.
First, we need now a type for such environments  we can use `Listof`
for this:
;; a type for environments:
(definetype ENV = (Listof (List Symbol VAL)))
but we can just as well define a new type for environment values:
(definetype ENV
[EmptyEnv]
[Extend Symbol VAL ENV])
Reimplementing `lookup` is now simple:
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id val restenv)
(if (eq? id name) val (lookup name restenv))]))
... we don't need `extend` because we get `Extend` from the type
definition, and we also get `(EmptyEnv)` instead of `emptysubst`.
We now use this with the new type for values  two variants of these:
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV]) ; argname, body, scope
And now the new implementation of `eval` which uses the new type and
implements lexical scope:
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
We also need to update `arithop` to use `VAL` objects. The full code
follows  it now passes all tests, including the example that we used
to find the problem.
;;; <<>>
;; The Flang interpreter, using environments
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2,env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype ENV
[EmptyEnv]
[Extend Symbol VAL ENV])
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id val restenv)
(if (eq? id name) val (lookup name restenv))]))
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Fixing an Overlooked Bug
Incidentally, this version fixes a bug we had previously in the
substitution version of FLANG:
(run "{with {f {fun {y} {+ x y}}}
{with {x 7}
{call f 1}}}")
This bug was due to our naive `subst`, which doesn't avoid capturing
renames. But note that since that version of the evaluator makes its
way from the outside in, there is no difference in semantics for *valid*
programs  ones that don't have free identifiers.
(Reminder: This was *not* a dynamically scoped language, just a bug that
happened when `x` wasn't substituted away before `f` was replaced with
something that refers to `x`.)

# Lexical Scope using Racket Closures
> [PLAI §11] (without the last part about recursion)
An alternative representation for an environment.
We've already seen how firstclass functions can be used to implement
"objects" that contain some information. We can use the same idea to
represent an environment. The basic intuition is  an environment is
a *mapping* (a function) between an identifier and some value. For
example, we can represent the environment that maps `'a` to `1` and `'b`
to `2` (using just numbers for simplicity) using this function:
(: mymap : Symbol > Number)
(define (mymap id)
(cond [(eq? 'a id) 1]
[(eq? 'b id) 2]
[else (error ...)]))
An empty mapping that is implemented in this way has the same type:
(: emptymapping : Symbol > Number)
(define (emptymapping id)
(error ...))
We can use this idea to implement our environments: we only need to
define three things  `EmptyEnv`, `Extend`, and `lookup`. If we
manage to keep the contract to these functions intact, we will be able
to simply plug it into the same evaluator code with no other changes.
It will also be more convenient to define `ENV` as the appropriate
function type for use in the `VAL` type definition instead of using the
actual type:
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
Now we get to `EmptyEnv`  this is expected to be a function that
expects no arguments and creates an empty environment, one that behaves
like the `emptymapping` function defined above. We could define it
like this (changing the `emptymapping` type to return a `VAL`):
(define (EmptyEnv) emptymapping)
but we can skip the need for an extra definition and simply return an
empty mapping function:
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error ...)))
(The unRackety name is to avoid replacing previous code that used the
`EmptyEnv` name for the constructor that was created by the type
definition.)
The next thing we tackle is `lookup`. The previous definition that was
used is:
(: lookup : Symbol ENV > VAL)
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id val restenv)
(if (eq? id name) val (lookup name restenv))]))
How should it be modified now? Easy  an environment is a mapping: a
Racket function that will do the searching job itself. We don't need to
modify the contract since we're still using `ENV`, except a different
implementation for it. The new definition is:
(: lookup : Symbol ENV > VAL)
(define (lookup name env)
(env name))
Note that `lookup` does almost nothing  it simply delegates the real
work to the `env` argument. This is a good hint for the error message
that empty mappings should throw 
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
Finally, `Extend`  this was previously created by the variant case of
the ENV type definition:
[Extend Symbol VAL ENV]
keeping the same type that is implied by this variant means that the new
`Extend` should look like this:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
...)
The question is  how do we extend a given environment? Well, first,
we know that the result should be mapping  a `symbol > VAL` function
that expects an identifier to look for:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
...))
Next, we know that in the generated mapping, if we look for `id` then
the result should be `val`:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
...)))
If the `name` that we're looking for is not the same as `id`, then we
need to search through the previous environment:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(lookup name restenv))))
But we know what `lookup` does  it simply delegates back to the
mapping function (which is our `rest` argument), so we can take a direct
route instead:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name)))) ; same as (lookup name restenv)
To see how all this works, try out extending an empty environment a few
times and examine the result. For example, the environment that we
began with:
(define (mymap id)
(cond [(eq? 'a id) 1]
[(eq? 'b id) 2]
[else (error ...)]))
behaves in the same way (if the type of values is numbers) as
(Extend 'a 1 (Extend 'b 2 (EmptyEnv)))
The new code is now the same, except for the environment code:
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2,env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# More Closures (on both levels)
Racket closures (= functions) can be used in other places too, and as we
have seen, they can do more than encapsulate various values  they can
also hold the behavior that is expected of these values.
To demonstrate this we will deal with closures in our language. We
currently use a variant that holds the three pieces of relevant
information:
[FunV Symbol FLANG ENV]
We can replace this by a functional object, which will hold the three
values. First, change the `VAL` type to hold functions for `FunV`
values:
(definetype VAL
[NumV Number]
[FunV (? > ?)])
And note that the function should somehow encapsulate the same
information that was there previously, the question is *how* this
information is going to be done, and this will determine the actual
type. This information plays a role in two places in our evaluator 
generating a closure in the `Fun` case, and using it in the `Call` case:
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody ;***
(Extend boundid ;***
(eval argexpr env) ;***
fenv))] ;***
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]
we can simply fold the marked functionality bit of `Call` into a Racket
function that will be stored in a `FunV` object  this piece of
functionality takes an argument value, extends the closure's environment
with its value and the function's name, and continues to evaluate the
function body. Folding all of this into a function gives us:
(lambda (argval)
(eval boundbody (Extend boundid argval env)))
where the values of `boundbody`, `boundid`, and `val` are known at the
time that the `FunV` is *constructed*. Doing this gives us the
following code for the two cases:
[(Fun boundid boundbody)
(FunV (lambda (argval)
(eval boundbody (Extend boundid argval env))))]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV proc) (proc (eval argexpr env))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]
And now the type of the function is clear:
(definetype VAL
[NumV Number]
[FunV (VAL > VAL)])
And again, the rest of the code is unmodified:
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype VAL
[NumV Number]
[FunV (VAL > VAL)])
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV (lambda (argval)
(eval boundbody (Extend boundid argval env))))]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV proc) (proc (eval argexpr env))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Types of Evaluators
What we did just now is implement lexical environments and closures in
the language we implement using lexical environments and closures in our
own language (Racket)!
This is another example of embedding a feature of the host language in
the implemented language, an issue that we have already discussed.
There are many examples of this, even when the two languages involved
are different. For example, if we have this bit in the C implementation
of Racket:
// Disclaimer: not real Racket code
Racket_Object *eval_and(int argc, Racket_Object *argv[]) {
Racket_Object *tmp;
if ( argc != 2 )
signal_racket_error("bad number of arguments");
else if ( racket_eval(argv[0]) != racket_false &&
(tmp = racket_eval(argv[1])) != racket_false )
return tmp;
else
return racket_false;
}
then the special semantics of evaluating a Racket `and` form is being
inherited from C's special treatment of `&&`. You can see this by the
fact that if there is a bug in the C compiler, then it will propagate to
the resulting Racket implementation too. A different solution is to not
use `&&` at all:
// Disclaimer: not real Racket code
Racket_Object *eval_and(int argc, Racket_Object *argv[]) {
Racket_Object *tmp;
if ( argc != 2 )
signal_racket_error("bad number of arguments");
else if ( racket_eval(argv[0]) != racket_false )
return racket_eval(argv[1]);
else
return racket_false;
}
and we can say that this is even better since it evaluates the second
expression in tail position. But in this case we don't really get that
benefit, since C itself is not doing tailcall optimization as a
standard feature (though some compilers do so under some circumstances).
We have seen a few different implementations of evaluators that are
quite different in flavor. They suggest the following taxonomy.
* A ___syntactic evaluator___ is one that uses its own language to
represent expressions and semantic runtime values of the evaluated
language, implementing all the corresponding behavior explicitly.
* A ___meta evaluator___ is an evaluator that uses language features of
its own language to directly implement behavior of the evaluated
language.
While our substitutionbased FLANG evaluator was close to being a
syntactic evaluator, we haven't written any purely syntactic evaluators
so far: we still relied on things like Racket arithmetics etc. The most
recent evaluator that we have studied, is even more of a *meta*
evaluator than the preceding ones: it doesn't even implement closures
and lexical scope, and instead, it uses the fact that Racket itself has
them.
With a good match between the evaluated language and the implementation
language, writing a meta evaluator can be very easy. With a bad match,
though, it can be very hard. With a syntactic evaluator, implementing
each semantic feature will be somewhat hard, but in return you don't
have to worry as much about how well the implementation and the
evaluated languages match up. In particular, if there is a particularly
strong mismatch between the implementation and the evaluated language,
it may take less effort to write a syntactic evaluator than a meta
evaluator. As an exercise, we can build upon our latest evaluator to
remove the encapsulation of the evaluator's response in the VAL type.
The resulting evaluator is shown below. This is a true meta evaluator:
it uses Racket closures to implement FLANG closures, Racket function
application for FLANG function application, Racket numbers for FLANG
numbers, and Racket arithmetic for FLANG arithmetic. In fact, ignoring
some small syntactic differences between Racket and FLANG, this latest
evaluator can be classified as something more specific than a meta
evaluator:
* A ___metacircular evaluator___ is a meta evaluator in which the
implementation and the evaluated languages are the same.
(Put differently, the trivial nature of the evaluator clues us in to the
deep connection between the two languages, whatever their syntactic
differences may be.)

# Feature Embedding
We saw that the difference between lazy evaluation and eager evaluation
is in the evaluation rules for `with` forms, function applications, etc:
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
is eager, and
eval({with {x E1} E2}) = eval(E2[E1/x])
is lazy. But is the first rule *really* eager? The fact is that the
only thing that makes it eager is the fact that our understanding of the
mathematical notation is eager  if we were to take math as lazy, then
the description of the rule becomes a description of lazy evaluation.
Another way to look at this is  take the piece of code that
implements this evaluation:
(: eval : FLANG > Number)
;; evaluates FLANG expressions by reducing them to numbers
(define (eval expr)
(cases expr
...
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))]
...))
and the same question applies: is this really implementing eager
evaluation? We know that this is indeed eager  we can simply try it
and check that it is, but it is only eager because we are using an eager
language for the implementation! If our own language was lazy, then the
evaluator's implementation would run lazily, which means that the above
applications of the `eval` and the `subst` functions would also be lazy,
making our evaluator lazy as well.
This is a general phenomena where some of the semantic features of the
language we use (math in the formal description, Racket in our code)
gets *embedded* into the language we implement.
Here's another example  consider the code that implements
arithmetics:
(: eval : FLANG > Number)
;; evaluates FLANG expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
...))
what if it was written like this:
FLANG eval(FLANG expr) {
if (is_Num(expr))
return num_of_Num(expr);
else if (is_Add(expr))
return eval(lhs_of_Add(expr)) + eval(rhs_of_Add(expr));
else if ...
...
}
Would it still implement unlimited integers and exact fractions? That
depends on the language that was used to implement it: the above syntax
suggests C, C++, Java, or some other relative, which usually come with
limited integers and no exact fractions. But this really depends on the
language  even our own code has unlimited integers and exact
rationals only because Racket has them. If we were using a language
that didn't have such features (there are such Scheme implementations),
then our implemented language would absorb these (lack of) features too,
and its own numbers would be limited in just the same way. (And this
includes the syntax for numbers, which we embedded intentionally, like
the syntax for identifiers).
The bottom line is that we should be aware of such issues, and be very
careful when we talk about semantics. Even the language that we use to
communicate (semiformal logic) can mean different things.

Aside: read "Reflections on Trusting Trust" by Ken Thompson
(You can skip to the "Stage II" part to get to the interesting stuff.)
(And when you're done, look for "XcodeGhost" to see a relevant example,
and don't miss the leaked document on the wikipedia page...)

Here is yet another variation of our evaluator that is even closer to a
metacircular evaluator. It uses Racket values directly to implement
values, so arithmetic operations become straightforward. Note
especially how the case for function application is similar to
arithmetics: a FLANG function application translates to a Racket
function application. In both cases (applications and arithmetics) we
don't even check the objects since they are simple Racket objects  if
our language happens to have some meaning for arithmetics with
functions, or for applying numbers, then we will inherit the same
semantics in our language. This means that we now specify less behavior
and fall back more often on what Racket does.
We use Racket values with this type definition:
(definetype VAL = (U Number (VAL > VAL)))
And the evaluation function can now be:
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) n] ;*** return the actual number
[(Add l r) (+ (eval l env) (eval r env))]
[(Sub l r) ( (eval l env) (eval r env))]
[(Mul l r) (* (eval l env) (eval r env))]
[(Div l r) (/ (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(lambda ([argval : VAL]) ;*** return the racket function
;; note that this requires input type specifications since
;; typed racket can't guess the right one
(eval boundbody (Extend boundid argval env)))]
[(Call funexpr argexpr)
((eval funexpr env) ;*** trivial like the arithmetics!
(eval argexpr env))]))
Note how the arithmetics implementation is simple  it's a direct
translation of the FLANG syntax to Racket operations, and since we don't
check the inputs to the Racket operations, we let Racket throw type
errors for us. Note also how function application is just like the
arithmetic operations: a FLANG application is directly translated to a
Racket application.
However, this does not work quite as simply in Typed Racket. The whole
point of typechecking is that we never run into type errors  so we
cannot throw back on Racket errors since code that might produce them is
forbidden! A way around this is to perform explicit checks that
guarantee that Racket cannot run into type errors. We do this with the
following two helpers that are defined inside `eval`:
(: evalN : FLANG > Number)
(define (evalN e)
(let ([n (eval e env)])
(if (number? n)
n
(error 'eval "got a nonnumber: ~s" n))))
(: evalF : FLANG > (VAL > VAL))
(define (evalF e)
(let ([f (eval e env)])
(if (function? f)
f
(error 'eval "got a nonfunction: ~s" f))))
Note that Typed Racket is "smart enough" to figure out that in `evalF`
the result of the recursive evaluation has to be either `Number` or
`(VAL > VAL)`; and since the `if` throws out on numbers, we're left
with `(VAL > VAL)` functions, not just any function.
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
;; Values are plain Racket values, no new VAL wrapper;
;; (but note that this is a recursive definition)
(definetype VAL = (U Number (VAL > VAL)))
;; Define a type for functional environments
(definetype ENV = (Symbol > VAL))
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(: evalN : FLANG > Number)
(define (evalN e)
(let ([n (eval e env)])
(if (number? n)
n
(error 'eval "got a nonnumber: ~s" n))))
(: evalF : FLANG > (VAL > VAL))
(define (evalF e)
(let ([f (eval e env)])
(if (function? f)
f
(error 'eval "got a nonfunction: ~s" f))))
(cases expr
[(Num n) n]
[(Add l r) (+ (evalN l) (evalN r))]
[(Sub l r) ( (evalN l) (evalN r))]
[(Mul l r) (* (evalN l) (evalN r))]
[(Div l r) (/ (evalN l) (evalN r))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(lambda ([argval : VAL])
(eval boundbody (Extend boundid argval env)))]
[(Call funexpr argexpr)
((evalF funexpr)
(eval argexpr env))]))
(: run : String > VAL) ; no need to convert VALs to numbers
;; evaluate a FLANG program contained in a string
(define (run str)
(eval (parse str) (EmptyEnv)))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Recursion, Recursion, Recursion
> [PLAI §9]
There is one major feature that is still missing from our language: we
have no way to perform recursion (therefore no kind of loops). So far,
we could only use recursion when we had *names*. In FLANG, the only way
we can have names is through `with` which not good enough for recursion.
To discuss the issue of recursion, we switch to a "broken" version of
(untyped) Racket  one where a `define` has a different scoping rules:
the scope of the defined name does *not* cover the defined expression.
Specifically, in this language, this doesn't work:
#lang pl broken
(define (fact n)
(if (zero? n) 1 (* n (fact ( n 1)))))
(fact 5)
In our language, this translation would also not work (assuming we have
`if` etc):
{with {fact {fun {n}
{if {= n 0} 1 {* n {call fact { n 1}}}}}}
{call fact 5}}
And similarly, in plain Racket this won't work if `let` is the only tool
you use to create bindings:
(let ([fact (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))])
(fact 5))
In the brokenscope language, the `define` form is more similar to a
mathematical definition. For example, when we write:
(define (F x) x)
(define (G y) (F y))
(G F)
it is actually shorthand for
(define F (lambda (x) x))
(define G (lambda (y) (F y)))
(G F)
we can then replace defined names with their definitions:
(define F (lambda (x) x))
(define G (lambda (y) (F y)))
((lambda (y) (F y)) (lambda (x) x))
and this can go on, until we get to the actual code that we wrote:
((lambda (y) ((lambda (x) x) y)) (lambda (x) x))
This means that the above `fact` definition is similar to writing:
fact := (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))
(fact 5)
which is not a wellformed definition  it is *meaningless* (this is a
formal use of the word "meaningless"). What we'd really want, is to
take the *equation* (using `=` instead of `:=`)
fact = (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))
and find a solution which will be a value for `fact` that makes this
true.
If you look at the Racket evaluation rules handout on the web page, you
will see that this problem is related to the way that we introduced the
Racket `define`: there is a handwavy explanation that talks about
*knowing* things.
The big question is: can we define recursive functions without Racket's
magical `define` form?
> Note: This question is a little different than the question of
> implementing recursion in our language  in the Racket case we have
> no control over the implementation of the language. As it will
> eventually turn out, implementing recursion in our own language will
> be quite easy when we use mutation in a specific way. So the question
> that we're now facing can be phrased as either "can we get recursion
> in Racket without Racket's magical definition forms?" or "can we get
> recursion in our interpreter without mutation?".

# Recursion without the Magic
> [PLAI §22.4] (we go much deeper)
> Note: This explanation is similar to the one you can find in "The Why
> of Y", by Richard Gabriel.
To implement recursion without the `define` magic, we first make an
observation: this problem does *not* come up in a dynamicallyscoped
language. Consider the `let`version of the problem:
#lang pl dynamic
(let ([fact (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))])
(fact 5))
This works fine  because by the time we get to evaluate the body of
the function, `fact` is already bound to itself in the current dynamic
scope. (This is another reason why dynamic scope is perceived as a
convenient approach in new languages.)
Regardless, the problem that we have with lexical scope is still there,
but the way things work in a dynamic scope suggest a solution that we
can use now. Just like in the dynamic scope case, when `fact` is
called, it does have a value  the only problem is that this value is
inaccessible in the lexical scope of its body.
Instead of trying to get the value in via lexical scope, we can imitate
what happens in the dynamically scoped language by passing the `fact`
value to itself so it can call itself (going back to the original code
in the brokenscope language):
(define (fact self n) ;***
(if (zero? n) 1 (* n (self ( n 1)))))
(fact fact 5) ;***
except that now the recursive call should still send itself along:
(define (fact self n)
(if (zero? n) 1 (* n (self self ( n 1))))) ;***
(fact fact 5)
The problem is that this required rewriting calls to `fact`  both
outside and recursive calls inside. To make this an acceptable
solution, calls from both places should not change. Eventually, we
should be able to get a working `fact` definition that uses just
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))
The first step in resolving this problem is to curry the `fact`
definition.
(define (fact self) ;***
(lambda (n) ;***
(if (zero? n)
1
(* n ((self self) ( n 1)))))) ;***
((fact fact) 5) ;***
Now `fact` is no longer our factorial function  it's a function that
constructs it. So call it `makefact`, and bind `fact` to the actual
factorial function.
(define (makefact self) ;***
(lambda (n)
(if (zero? n) 1 (* n ((self self) ( n 1))))))
(define fact (makefact makefact)) ;***
(fact 5) ;***
We can try to do the same thing in the body of the factorial function:
instead of calling `(self self)`, just bind `fact` to it:
(define (makefact self)
(lambda (n)
(let ([fact (self self)]) ;***
(if (zero? n)
1
(* n (fact ( n 1))))))) ;***
(define fact (makefact makefact))
(fact 5)
This works fine, but if we consider our original goal, we need to get
that local `fact` binding outside of the `(lambda (n) ...)`  so we're
left with a definition that uses the factorial expression as is. So,
swap the two lines:
(define (makefact self)
(let ([fact (self self)]) ;***
(lambda (n) ;***
(if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact (makefact makefact))
(fact 5)
But the problem is that this gets us into an infinite loop because we're
trying to evaluate `(self self)` too ea(ge)rly. In fact, if we ignore
the body of the `let` and other details, we basically do this:
(define (makefact self) (self self)) (makefact makefact)
reducesugar>
(define makefact (lambda (self) (self self))) (makefact makefact)
replacedefinition>
((lambda (self) (self self)) (lambda (self) (self self)))
renameidentifiers>
((lambda (x) (x x)) (lambda (x) (x x)))
And this expression has an interesting property: it reduces to itself,
so evaluating it gets stuck in an infinite loop.
So how do we solve this? Well, we know that `(self self)` *should* be
the same value that is the factorial function itself  so it must be a
oneargument function. If it's such a function, we can use a value that
is equivalent, except that it will not get evaluated until it is needed,
when the function is called. The trick here is the observation that
`(lambda (n) (add1 n))` is really the same as `add1` (provided that
`add1` is a oneargument function), except that the `add1` part doesn't
get evaluated until the function is called. Applying this trick to our
code produces a version that does not get stuck in the same infinite
loop:
(define (makefact self)
(let ([fact (lambda (n) ((self self) n))]) ;***
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact (makefact makefact))
(fact 5)
Continuing from here  we know that
(let ([x v]) e) is the same as ((lambda (x) e) v)
(remember how we derived `fun` from a `with`), so we can turn that `let`
into the equivalent function application form:
(define (makefact self)
((lambda (fact) ;***
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))
(lambda (n) ((self self) n)))) ;***
(define fact (makefact makefact))
(fact 5)
And note now that the (lambda (fact) ...) expression is everything that
we need for a recursive definition of `fact`  it has the proper
factorial body with a plain recursive call. It's almost like the usual
value that we'd want to define `fact` as, except that we still have to
abstract on the recursive value itself. So lets move this code into a
separate definition for `factstep`:
(define factstep ;***
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makefact self)
(factstep ;***
(lambda (n) ((self self) n))))
(define fact (makefact makefact))
(fact 5)
We can now proceed by moving the `(makefact makefact)` self
application into its own function which is what creates the real
factorial:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makefact self)
(factstep
(lambda (n) ((self self) n))))
(define (makerealfact) (makefact makefact)) ;***
(define fact (makerealfact)) ;***
(fact 5)
Rewrite the `makefact` definition using an explicit `lambda`:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define makefact ;***
(lambda (self) ;***
(factstep
(lambda (n) ((self self) n)))))
(define (makerealfact) (makefact makefact))
(define fact (makerealfact))
(fact 5)
and fold the functionality of `makefact` and `makerealfact` into a
single `makefact` function by just using the value of `makefact`
explicitly instead of through a definition:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerealfact)
(let ([make (lambda (self) ;***
(factstep ;***
(lambda (n) ((self self) n))))]) ;***
(make make)))
(define fact (makerealfact))
(fact 5)
We can now observe that `makerealfact` has nothing that is specific to
factorial  we can make it take a "core function" as an argument:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerealfact core) ;***
(let ([make (lambda (self)
(core ;***
(lambda (n) ((self self) n))))])
(make make)))
(define fact (makerealfact factstep)) ;***
(fact 5)
and call it `makerecursive`:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerecursive core) ;***
(let ([make (lambda (self)
(core
(lambda (n) ((self self) n))))])
(make make)))
(define fact (makerecursive factstep)) ;***
(fact 5)
We're almost done now  there's no real need for a separate
`factstep` definition, just use the value for the definition of `fact`:
(define (makerecursive core)
(let ([make (lambda (self)
(core
(lambda (n) ((self self) n))))])
(make make)))
(define fact
(makerecursive
(lambda (fact) ;***
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))) ;***
(fact 5)
turn the `let` into a function form:
(define (makerecursive core)
((lambda (make) (make make)) ;***
(lambda (self) ;***
(core (lambda (n) ((self self) n)))))) ;***
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
do some renamings to make things simpler  `make` and `self` turn to
`x`, and `core` to `f`:
(define (makerecursive f) ;***
((lambda (x) (x x)) ;***
(lambda (x) (f (lambda (n) ((x x) n)))))) ;***
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
or we can manually expand that first (lambda (x) (x x)) application to
make the symmetry more obvious (not really surprising because it started
with a `let` whose purpose was to do a selfapplication):
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n)))) ;***
(lambda (x) (f (lambda (n) ((x x) n)))))) ;***
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
And we finally got what we were looking for: a general way to define
*any* recursive function without any magical `define` tricks. This also
work for other recursive functions:
#lang pl broken
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n))))
(lambda (x) (f (lambda (n) ((x x) n))))))
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
(define fib
(makerecursive
(lambda (fib)
(lambda (n) (if (<= n 1) n (+ (fib ( n 1)) (fib ( n 2))))))))
(fib 8)
(define length
(makerecursive
(lambda (length)
(lambda (l) (if (null? l) 0 (+ (length (rest l)) 1))))))
(length '(x y z))
A convenient tool that people often use on paper is to perform a kind of
a syntactic abstraction: "assume that whenever I write (twice foo) I
really meant to write (foo foo)". This can often be done as plain
abstractions (that is, using functions), but in some cases  for
example, if we want to abstract over definitions  we just want such a
rewrite rule. (More on this towards the end of the course.) The
brokenscope language does provide such a tool  `rewrite` extends the
language with a rewrite rule. Using this, and our `makerecursive`, we
can make up a recursive definition form:
(rewrite (define/rec (f x) E)
=> (define f (makerecursive (lambda (f) (lambda (x) E)))))
In other words, we've created our own "magical definition" form. The
above code can now be written in almost the same way it is written in
plain Racket:
#lang pl broken
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n))))
(lambda (x) (f (lambda (n) ((x x) n))))))
(rewrite (define/rec (f x) E)
=> (define f (makerecursive (lambda (f) (lambda (x) E)))))
;; examples
(define/rec (fact n) (if (zero? n) 1 (* n (fact ( n 1)))))
(fact 5)
(define/rec (fib n) (if (<= n 1) n (+ (fib ( n 1)) (fib ( n 2)))))
(fib 8)
(define/rec (length l) (if (null? l) 0 (+ (length (rest l)) 1)))
(length '(x y z))
Finally, note that makerecursive is limited to 1argument functions
only because of the protection from eager evaluation. In any case, it
can be used in any way you want, for example,
(makerecursive (lambda (f) (lambda (x) f)))
is a function that *returns itself* rather than calling itself. Using
the rewrite rule, this would be:
(define/rec (f x) f)
which is the same as:
(define (f x) f)
in plain Racket.

# The Core of `makerecursive`
As in Racket, being able to express recursive functions is a fundamental
property of the language. It means that we can have loops in our
language, and that's the essence of making a language powerful enough to
be TMequivalent  able to express undecidable problems, where we
don't know whether there is an answer or not.
The core of what makes this possible is the expression that we have seen
in our derivation:
((lambda (x) (x x)) (lambda (x) (x x)))
which reduces to itself, and therefore has no value: trying to evaluate
it gets stuck in an infinite loop. (This expression is often called
"Omega".)
This is the key for creating a loop  we use it to make recursion
possible. Looking at our final `makerecursive` definition and ignoring
for a moment the "protection" that we need against being stuck
prematurely in an infinite loop:
(define (makerecursive f)
((lambda (x) (x x)) (lambda (x) (f (x x)))))
we can see that this is almost the same as the Omega expression  the
only difference is that application of `f`. Indeed, this expression
(the result of (makerecursive F) for some `F`) reduces in a similar way
to Omega:
((lambda (x) (x x)) (lambda (x) (F (x x))))
((lambda (x) (F (x x))) (lambda (x) (F (x x))))
(F ((lambda (x) (F (x x))) (lambda (x) (F (x x)))))
(F (F ((lambda (x) (F (x x))) (lambda (x) (F (x x))))))
(F (F (F ((lambda (x) (F (x x))) (lambda (x) (F (x x)))))))
...
which means that the actual value of this expression is:
(F (F (F ...forever...)))
This definition would be sufficient if we had a lazy language, but to
get things working in a strict one we need to bring back the protection.
This makes things a little different  if we use `(protect f)` to be a
shorthand for the protection trick,
(rewrite (protect f) => (lambda (x) (f x)))
then we have:
(define (makerecursive f)
((lambda (x) (x x)) (lambda (x) (f (protect (x x))))))
which makes the (makerecursive F) evaluation reduce to
(F (protect (F (protect (F (protect (...forever...)))))))
and this is still the same result (as long as `F` is a singleargument
function).
(Note that `protect` cannot be implemented as a plain function!)

# Denotational Explanation of Recursion
> Note: This explanation is similar to the one you can find in "The
> Little Schemer" called "(Y Y) Works!", by Dan Friedman and Matthias
> Felleisen.
The explanation that we have now for how to derive the `makerecursive`
definition is fine  after all, we did manage to get it working. But
this explanation was done from a kind of an operational point of view:
we knew a certain trick that can make things work and we pushed things
around until we got it working like we wanted. Instead of doing this,
we can reapproach the problem from a more declarative point of view.
So, start again from the same broken code that we had (using the
brokenscope language):
(define fact
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))
This is as broken as it was when we started: the occurrence of `fact` in
the body of the function is free, which means that this code is
meaningless. To avoid the compilation error that we get when we run
this code, we can substitute *anything* for that `fact`  it's even
better to use a replacement that will lead to a runtime error:
(define fact
(lambda (n) (if (zero? n) 1 (* n (777 ( n 1)))))) ;***
This function will not work in a similar way to the original one  but
there is one case where it *does* work: when the input value is `0`
(since then we do not reach the bogus application). We note this by
calling this function `fact0`:
(define fact0 ;***
(lambda (n) (if (zero? n) 1 (* n (777 ( n 1))))))
Now that we have this function defined, we can use it to write `fact1`
which is the factorial function for arguments of `0` or `1`:
(define fact0
(lambda (n) (if (zero? n) 1 (* n (777 ( n 1))))))
(define fact1
(lambda (n) (if (zero? n) 1 (* n (fact0 ( n 1))))))
And remember that this is actually just shorthand for:
(define fact1
(lambda (n)
(if (zero? n)
1
(* n ((lambda (n)
(if (zero? n)
1
(* n (777 ( n 1)))))
( n 1))))))
We can continue in this way and write `fact2` that will work for n<=2:
(define fact2
(lambda (n) (if (zero? n) 1 (* n (fact1 ( n 1))))))
or, in full form:
(define fact2
(lambda (n)
(if (zero? n)
1
(* n ((lambda (n)
(if (zero? n)
1
(* n ((lambda (n)
(if (zero? n)
1
(* n (777 ( n 1)))))
( n 1)))))
( n 1))))))
If we continue this way, we *will* get the true factorial function, but
the problem is that to handle *any* possible integer argument, it will
have to be an infinite definition! Here is what it is supposed to look
like:
(define fact0 (lambda (n) (if (zero? n) 1 (* n (777 ( n 1))))))
(define fact1 (lambda (n) (if (zero? n) 1 (* n (fact0 ( n 1))))))
(define fact2 (lambda (n) (if (zero? n) 1 (* n (fact1 ( n 1))))))
(define fact3 (lambda (n) (if (zero? n) 1 (* n (fact2 ( n 1))))))
...
The true factorial function is `factinfinity`, with an infinite size.
So, we're back at the original problem...
To help make things more concise, we can observe the repeated pattern in
the above, and extract a function that abstracts this pattern. This
function is the same as the `factstep` that we have seen previously:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact0 (factstep 777))
(define fact1 (factstep fact0))
(define fact2 (factstep fact1))
(define fact3 (factstep fact2))
...
which is actually:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact0 (factstep 777))
(define fact1 (factstep (factstep 777)))
(define fact2 (factstep (factstep (factstep 777))))
...
(define fact
(factstep (factstep (factstep (... (factstep 777) ...)))))
Do this a little differently  rewrite `fact0` as:
(define fact0
((lambda (mk) (mk 777))
factstep))
Similarly, `fact1` is written as:
(define fact1
((lambda (mk) (mk (mk 777)))
factstep))
and so on, until the real factorial, which is still infinite at this
stage:
(define fact
((lambda (mk) (mk (mk (... (mk 777) ...))))
factstep))
Now, look at that `(lambda (mk) ...)`  it is an infinite expression,
but for every actual application of the resulting factorial function we
only need a finite number of `mk` applications. We can guess how many,
and as soon as we hit an application of `777` we know that our guess is
too small. So instead of `777`, we can try to use the maker function to
create and use the next.
To make things more explicit, here is the expression that is our
`fact0`, without the definition form:
((lambda (mk) (mk 777))
factstep)
This function has a very low guess  it works for 0, but with 1 it
will run into the `777` application. At this point, we want to somehow
invoke `mk` again to get the next level  and since `777` *does* get
applied, we can just replace it with `mk`:
((lambda (mk) (mk mk))
factstep)
The resulting function works just the same for an input of `0` because
it does not attempt a recursive call  but if we give it `1`, then
instead of running into the error of applying `777`:
(* n (777 ( n 1)))
we get to apply `factstep` there:
(* n (factstep ( n 1)))
and this is still wrong, because `factstep` expects a function as an
input. To see what happens more clearly, write `factstep` explicitly:
((lambda (mk) (mk mk))
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
The problem is in what we're going to pass into `factstep`  its
`fact` argument will not be the factorial function, but the `mk`
function constructor. Renaming the `fact` argument as `mk` will make
this more obvious (but not change the meaning):
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n (mk ( n 1)))))))
It should now be obvious that this application of `mk` will not work,
instead, we need to apply it on some function and *then* apply the
result on `( n 1)`. To get what we had before, we can use `777` as a
bogus function:
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n ((mk 777) ( n 1)))))))
This will allow one recursive call  so the definition works for both
inputs of `0` and `1`  but not more. But that `777` is used as a
maker function now, so instead, we can just use `mk` itself again:
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n ((mk mk) ( n 1)))))))
And this is a *working* version of the real factorial function, so make
it into a (nonmagical) definition:
(define fact
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n ((mk mk) ( n 1))))))))
But we're not done  we "broke" into the factorial code to insert that
`(mk mk)` application  that's why we dragged in the actual value of
`factstep`. We now need to fix this. The expression on that last line
(lambda (n) (if (zero? n) 1 (* n ((mk mk) ( n 1)))))
is close enough  it is `(factstep (mk mk))`. So we can now try to
rewrite our `fact` as:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact
((lambda (mk) (mk mk))
(lambda (mk) (factstep (mk mk)))))
... and would fail in a familiar way! If it's not familiar enough, just
rename all those `mk`s as `x`s:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact
((lambda (x) (x x))
(lambda (x) (factstep (x x)))))
We've run into the eagerness of our language again, as we did before.
The solution is the same  the `(x x)` is the factorial function, so
protect it as we did before, and we have a working version:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact
((lambda (x) (x x))
(lambda (x) (factstep (lambda (n) ((x x) n))))))
The rest should not be surprising now... Abstract the recursive making
bit in a new `makerecursive` function:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerecursive f)
((lambda (x) (x x))
(lambda (x) (f (lambda (n) ((x x) n))))))
(define fact (makerecursive factstep))
and now we can do the first reduction inside `makerecursive` and write
the `factstep` expression explicitly:
#lang pl broken
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n))))
(lambda (x) (f (lambda (n) ((x x) n))))))
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
and this is the same code we had before.

# The Y Combinator
Our `makerecursive` function is usually called the *fixpoint operator*
or the *Y combinator*.
It looks really simple when using the lazy version (remember: our
version is the eager one):
(define Y
(lambda (f)
((lambda (x) (f (x x)))
(lambda (x) (f (x x))))))
> Note that if we *do* allow a recursive definition for Y itself, then
> the definition can follow the definition that we've seen:
>
> (define (Y f) (f (Y f)))
And this all comes from the loop generated by:
((lambda (x) (x x)) (lambda (x) (x x)))
This expression, which is also called *Omega* (the `(lambda (x) (x x))`
part by itself is usually called *omega* and then `(omega omega)` is
*Omega*), is also the idea behind many deep mathematical facts. As an
example for what it does, follow the next rule:
I will say the next sentence twice:
"I will say the next sentence twice".
(Note the usage of colon for the first and quotes for the second 
what is the equivalent of that in the lambda expression?)
By itself, this just gets you stuck in an infinite loop, as Omega does,
and the Y combinator adds `F` to that to get an infinite chain of
applications  which is similar to:
I will say the next sentence twice:
"I will hop on one foot and then say the next sentence twice".
Sidenote:
[see this SO question](https://stackoverflow.com/q/25228394/128595)
and my answer, which came from the PLQ implementation.

## The main property of Y
`factstep` is a function that given any limited factorial, will
generate a factorial that is good for one more integer input. Start
with `777`, which is a factorial that is good for nothing (because it's
not a function), and you can get `fact0` as
fact0 == (factstep 777)
and that's a good factorial function only for an input of `0`. Use that
with `factstep` again, and you get
fact1 == (factstep fact0) == (factstep (factstep 777))
which is the factorial function when you only look at input values of
`0` or `1`. In a similar way
fact2 == (factstep fact1)
is good for `0`...`2`  and we can continue as much as we want, except
that we need to have an infinite number of applications  in the
general case, we have:
factn == (factstep (factstep (factstep ... 777)))
which is good for `0`...`n`. The *real* factorial would be the result
of running `factstep` on itself infinitely, it *is* `factinfinity`.
In other words (here `fact` is the *real* factorial):
fact = factinfinity == (factstep (factstep ...infinitely...))
but note that since this is really infinity, then
fact = (factstep (factstep ...infinitely...))
= (factstep fact)
so we get an equation:
fact = (factstep fact)
and a solution for this is going to be the real factorial. The solution
is the *fixedpoint* of the `factstep` function, in the same sense that
`0` is the fixed point of the `sin` function because
0 = (sin 0)
And the Y combinator does just that  it has this property:
(makerecursive f) = (f (makerecursive f))
or, using the more common name:
(Y f) = (f (Y f))
This property encapsulates the real magical power of Y. You can see how
it works: since `(Y f) = (f (Y f))`, we can add an `f` application to
both sides, giving us `(f (Y f)) = (f (f (Y f)))`, so we get:
(Y f) = (f (Y f)) = (f (f (Y f))) = (f (f (f (Y f)))) = ...
= (f (f (f ...)))
and we can conclude that
(Y factstep) = (factstep (factstep ...infinitely...))
= fact

# Yet another explanation for Y
Here's another explanation of how the Y combinator works. Remember that
our `factstep` function was actually a function that generates a
factorial function based on some input, which is supposed to be the
factorial function:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
As we've seen, you can apply this function on a version of factorial
that is good for inputs up to some n, and the result will be a factorial
that is good for those values up to n+1. The question is *what is the
fixpoint of `factstep`*? And the answer is that if it maps factₙ
factorial to factₙ₊₁, then the input will be equal to the output on the
*infinitieth* `fact`, which is the *actual* factorial. Since Y is a
fixpoint combinator, it gives us exactly that answer:
(define therealfactorial (Y factstep))

# Typing the Y Combinator
Typing the Y combinator is a tricky issue. For example, in standard ML
you must write a new type definition to do this:
datatype 'a t = T of 'a t > 'a
val y = fn f => (fn (T x) => (f (fn a => x (T x) a)))
(T (fn (T x) => (f (fn a => x (T x) a))))
> Can you find a pattern in the places where `T` is used?
>  Roughly speaking, that type definition is
>
> ;; `t' is the type name, `T' is the constructor (aka the variant)
> (definetype (RecTypeOf t)
> [T ((RecTypeOf t) > t)])
>
> First note that the two `fn a => ...` parts are the same as our
> protection, so ignoring that we get:
>
> val y = fn f => (fn (T x) => (f (x (T x))))
> (T (fn (T x) => (f (x (T x)))))
>
> if you now replace `T` with `Quote`, things make more sense:
>
> val y = fn f => (fn (Quote x) => (f (x (Quote x))))
> (Quote (fn (Quote x) => (f (x (Quote x)))))
>
> and with our syntax, this would be:
>
> (define (Y f)
> ((lambda (qx)
> (cases qx
> [(Quote x) (f (x qx))]))
> (Quote
> (lambda (qx)
> (cases qx
> [(Quote x) (f (x qx))])))))
>
> it's not really quotation  but the analogy should help: it uses
> `Quote` to distinguish functions as values that are applied (the `x`s)
> from functions that are passed as arguments.
In OCaml, this looks a little different:
# type 'a t = T of ('a t > 'a) ;;
type 'a t = T of ('a t > 'a)
# let y f = (fun (T x) > x (T x))
(T (fun (T x) > fun z > f (x (T x)) z)) ;;
val y : (('a > 'b) > 'a > 'b) > 'a > 'b =
# let fact = y (fun fact n > if n < 1 then 1 else n * fact(n1)) ;;
val fact : int > int =
# fact 5 ;;
 : int = 120
but OCaml has also a `rectypes` command line argument, which will make
it infer the type by itself:
# let y f = (fun x > x x) (fun x > fun z > f (x x) z) ;;
val y : (('a > 'b) > 'a > 'b) > 'a > 'b =
# let fact = y (fun fact n > if n < 1 then 1 else n * fact(n1)) ;;
val fact : int > int =
# fact 5 ;;
 : int = 120
The translation of this to `#lang pl` is a little verbose because we
don't have autocurrying, and because we need to declare input types to
functions, but it's essentially a direct translation of the above:
(definetype (RecTypeOf t)
[T ((RecTypeOf t) > t)])
(: Y : (All (A B) ((A > B) > (A > B)) > (A > B)))
(define (Y f)
((lambda ([x : (RecTypeOf (A > B))])
(cases x
[(T x) (x (T x))]))
(T (lambda ([x : (RecTypeOf (A > B))])
(cases x
[(T x) (lambda ([z : A])
((f (x (T x))) z))])))))
(define fact
(Y (lambda ([fact : (Integer > Integer)])
(lambda ([n : Integer])
(if (< n 1) 1 (* n (fact (sub1 n))))))))
(fact 5)
It is also possible to write this expression in "plain" Typed Racket,
without a userdefined type  and we need to start with a proper type
definition. First of all, the type of Y should be straightforward: it
is a fixpoint operation, so it takes a `T > T` function and produces
its fixpoint. The fixpoint itself is some `T` (such that applying the
function on it results in itself). So this gives us:
(: makerecursive : (T > T) > T)
However, in our case `makerecursive` computes a *functional* fixpoint,
for unary `S > T` functions, so we should narrow down the type
(: makerecursive : ((S > T) > (S > T)) > (S > T))
Now, in the body of `makerecursive` we need to add a type for the `x`
argument which is behaving in a weird way: it is used both as a function
and as its own argument. (Remember  I will say the next sentence
twice: "I will say the next sentence twice".) We need a recursive type
definition helper (not a new type) for that:
(definetype (Tau S T) = (Rec this (this > (S > T))))
This type is tailored for our use of `x`: it is a type for a function
that will *consume itself* (hence the `Rec`) and spit out the value that
the `f` argument consumes  an `S > T` function.
The resulting full version of the code:
(: makerecursive : (All (S T) ((S > T) > (S > T)) > (S > T)))
(definetype (Tau S T) = (Rec this (this > (S > T))))
(define (makerecursive f)
((lambda ([x : (Tau S T)]) (f (lambda (z) ((x x) z))))
(lambda ([x : (Tau S T)]) (f (lambda (z) ((x x) z))))))
(: fact : Number > Number)
(define fact (makerecursive
(lambda ([fact : (Number > Number)])
(lambda ([n : Number])
(if (zero? n)
1
(* n (fact ( n 1))))))))
(fact 5)

# Lambda Calculus  Schlac
> [PLAI §22] (we do much more)
We know that many constructs that are usually thought of as primitives
are not really needed  we can implement them ourselves given enough
tools. The question is how far can we go?
The answer: as far as we want. For example:
(define foo((lambda(f)((lambda(x)(x x))(lambda(x)(f(x x)))))(lambda(
f)(lambda(x)(((x(lambda(x)(lambda(x y)y))(lambda(x y)x))(x(lambda(x)
(lambda(x y)y))(lambda(x y)x))(((x(lambda (p)(lambda(s)(s(p(lambda(x
y)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(s) (s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x))(lambda(x)(lambda(x y)y))(lambda
(x y)x)))(lambda(f x)(f x))((f((x(lambda(p)(lambda(s)(s(p(lambda(x y
)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(y s)(s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x)))(lambda(n)(lambda(f x)(f(n f x)
)))(f((((x(lambda(p)(lambda(s)(s(p (lambda(x y)y))(lambda(f x)(f((p(
lambda(x y)y))f x))))))(lambda(s)(s(lambda(f x) x)(lambda(f x)x))))(
lambda(x y)x))(lambda(p)(lambda(s)(s(p(lambda(x y)y))(lambda(f x)(f(
(p(lambda(x y)y))f x))))))(lambda(s)(s(lambda(f x)x)(lambda(f x)x)))
)(lambda(x y)x)))))))))
We begin with a very minimal language, which is based on the Lambda
Calculus. In this language we get a very minimal set of constructs and
values.
In DrRacket, this we will use the Schlac language level (stands for
"SchemeRacket as Lambda Calculus"). This language has a
Racketlike syntax, but don't be confused  it is *very* different
from Racket. The only constructs that are available in this language
are: lambda expressions of at least one argument, function application
(again, at least one argument), and simple definition forms which are
similar to the ones in the "Broken define" language  definitions are
used as shorthand, and cannot be used for recursive function definition.
They're also only allowed at the toplevel  no local helpers, and a
definition is not an expression that can appear anywhere. The BNF is
therefore:
::= ...
::=
 (define )
::=
 (lambda ( ...) )
 ( ...)
Since this language has no primitive values (other than functions),
Racket numbers and booleans are also considered identifiers, and have no
builtin value that come with the language. In addition, all functions
and function calls are curried, so
(lambda (x y z) (z y x))
is actually shorthand for
(lambda (x) (lambda (y) (lambda (z) ((z y) x))))
The rules for evaluation are simple, there is one very important rule
for evaluation which is called "beta reduction":
((lambda (x) E1) E2) > E1[E2/x]
where substitution in this context requires being careful so you won't
capture names. This requires you to be able to do another kind of
transformation which is called "alpha conversion", which basically says
that you can rename identifiers as long as you keep the same binding
structure (eg, a valid renaming does not change the deBruijn form of
the expression). There is one more rule that can be used, *eta
conversion* which says that `(lambda (x) (f x))` is the same as `f` (we
used this rule above when deriving the Y combinator).
One last difference between Schlac and Racket is that Schlac is a *lazy*
language. This will be important since we do not have any builtin
special forms like `if`.
Here is a Schlac definition for the identity function:
(define identity (lambda (x) x))
and there is not much that we can do with this now:
> identity
#
> (identity identity)
#
> (identity identity identity)
#
(In the last expression, note that `(id id id)` is shorthand for `((id
id) id)`, and since `(id id)` is the identity, applying that on `id`
returns it again.)

# Church Numerals
So far, it seems like it is impossible to do anything useful in this
language, since all we have are functions and applications. We know how
to write the identity function, but what about other values? For
example, can you write code that evaluates to zero?
> What's zero? I only know how to write functions!
>
> (Turing Machine programmer: "What's a function?
>  I only know how to write 0s and 1s!")
The first thing we therefore need is to be able to *encode* numbers as
functions. For zero, we will use a function of two arguments that
simply returns its second value:
(define 0 (lambda (f) (lambda (x) x)))
or, more concisely
(define 0 (lambda (f x) x))
This is the first step in an encoding that is known as *Church
Numerals*: an encoding of natural numbers as functions. The number zero
is encoded as a function that takes in a function and a second value,
and applies the function zero times on the argument (which is really
what the above definition is doing). Following this view, the number
one is going to be a function of two arguments, that applies the first
on the second one time:
(define 1 (lambda (f x) (f x)))
and note that `1` is just like the identity function (as long as you
give it a function as its first input, but this is always the case in
Schlac). The next number on the list is two  which applies the first
argument on the second one twice:
(define 2 (lambda (f x) (f (f x))))
We can go on doing this, but what we really want is a way to perform
arbitrary arithmetic. The first requirement for that is an `add1`
function that increments its input (an encoded natural number) by one.
To do this, we write a function that expects an encoded number:
(define add1 (lambda (n) ...))
and this function is expected to return an encoded number, which is
always a function of `f` and `x`:
(define add1 (lambda (n) (lambda (f x) ...)))
Now, in the body, we need to apply `f` on `x` n+1 times  but remember
that `n` is a function that will do `n` applications of its first
argument on its second:
(define add1 (lambda (n) (lambda (f x) ... (n f x) ...)))
and all we have left to do now is to apply `f` one more time, yielding
this definition for `add1`:
(define add1 (lambda (n) (lambda (f x) (f (n f x)))))
Using this, we can define a few useful numbers:
(define 1 (add1 0))
(define 2 (add1 1))
(define 3 (add1 2))
(define 4 (add1 3))
(define 5 (add1 4))
This is all nice theoretically, but how can we make sure that it is
correct? Well, Schlac has a few additional builtin functions that
translate Church numerals into Racket numbers. To try our definitions
we use the `>nat` (read: to natural number):
(>nat 0)
(>nat 5)
(>nat (add1 (add1 5)))
You can now verify that the identity function is really the same as the
number 1:
(>nat identity)
We can even write a test case, since Schlac contains the `test` special
form, but we have to be careful in that  first of all, we cannot test
whether functions are equal (why?) so we must use `>nat`, but
(test (>nat (add1 (add1 5))) => 7)
will not work since `7` is undefined. To overcome this, Schlac has a
`backdoor` for primitive Racket values  just use a quote:
(test (>nat (add1 (add1 5))) => '7)
We can now define natural number addition  one simple idea is to get
two encoded numbers `m` and `n`, then start with `x`, apply `f` on it
`n` times by using it as a function, then apply `f` `m` more times on
the result in the same way:
(define + (lambda (m n) (lambda (f x) (m f (n f x)))))
or equivalently:
(define + (lambda (m n f x) (m f (n f x))))
Another idea is to use `add1` and increment `n` by `m` using `add1`:
(define + (lambda (m n) (m add1 n)))
(>nat (+ 4 5))
We can also define multiplication of `m` and `n` quite easily  begin
with addition  `(lambda (x) (+ n x))` is a function that expects an
`x` and returns `(+ x n)`  it's an incrementbyn function. But
since all functions and applications are curried, this is actually the
same as `(lambda (x) ((+ n) x))` which is the same as `(+ n)`. Now,
what we want to do is repeat this operation `m` times over zero, which
will add `n` to zero `m` times, resulting in `m` * `n`. The definition
is therefore:
(define * (lambda (m n) (m (+ n) 0)))
(>nat (* 4 5))
(>nat (+ 4 (* (+ 2 5) 5)))
An alternative approach is to consider
(lambda (x) (n f x))
for some encoded number `n` and a function `f`  this function is like
`f`^`n` (f composed n times with itself). But remember that this is
shorthand for
(lambda (x) ((n f) x))
and we know that `(lambda (x) (foo x))` is just like `foo` (if it is a
function), so this is equivalent to just
(n f)
So `(n f)` is `f`^`n`, and in the same way `(m g)` is `g`^`m`  if we
use `(n f)` for `g`, we get `(m (n f))` which is n selfcompositions of
`f`, selfcomposed m times. In other words, `(m (n f))` is a function
that is like `m`*`n` applications of `f`, so we can define
multiplication as:
(define * (lambda (m n) (lambda (f) (m (n f)))))
which is the same as
(define * (lambda (m n f) (m (n f))))
The same principle can be used to define exponentiation (but now we have
to be careful with the order since exponentiation is not commutative):
(define ^ (lambda (m n) (n (* m) 1)))
(>nat (^ 3 4))
And there is a similar alternative here too 
* a Church numeral `m` is the mselfcomposition function,
* and `(1 m)` is just like `m`^`1` which is the same as `m`
(`1`=`identity`)
* and `(2 m)` is just like `m`^`2`  it takes a function `f`, self
composes it `m` times, and self composes the result `m` times  for
a total of `f`^`(m*m)`
* and `(3 m)` is similarly `f`^`(m*m*m)`
* so `(n m)` is `f`^`(m^n)` (note that the first `^` is
selfcompositions, and the second one is a mathematical exponent)
* so `(n m)` is a function that returns `m`^`n` selfcompositions of an
input function,
Which means that `(n m)` is the Church numeral for `m`^`n`, so we get:
(define ^ (lambda (m n) (n m)))
which basically says that any number encoding `n` is also the `?`^`n`
operation.
All of this is was not too complicated  but all so far all we did is
write functions that increment their inputs in various ways. What about
`sub1`? For that, we need to do some more work  we will need to
encode booleans.

# More Encodings
Our choice of encoding numbers makes sense  the idea is that the main
feature of a natural number is repeating something a number of times.
For booleans, the main property we're looking for is choosing between
two values. So we can encode true and false by functions of two
arguments that return either the first or the second argument:
(define #t (lambda (x y) x))
(define #f (lambda (x y) y))
Note that this encoding of `#f` is really the same as the encoding of
`0`, so we have to know what type to expect an use the proper operations
(this is similar to C, where everything is just integers). Now that we
have these two, we can define `if`:
(define if (lambda (c t e) (c t e)))
it expects a boolean which is a function of two arguments, and passes it
the two expressions. The `#t` boolean will simply return the first, and
the `#f` boolean will return the second. Strictly speaking, we don't
really need this definition, since instead of writing `(if c t e)`, we
can simply write `(c t e)`. In any case, we need the language to be
lazy for this to work. To demonstrate this, we'll intentionally use the
quote backdoor to use a nonfunctional value, using this will normally
result in an error:
(+ '1 '2)
But testing our `if` definition, things work just fine:
(if #t (+ 4 5) (+ 1 2))
and we see that DrRacket leaves the second addition expression in red,
which indicates that it was not executed. We can also make sure that
even when it is defined as a function, it is still working fine because
the language is lazy:
(if #f ((lambda (x) (x x)) (lambda (x) (x x))) 3)
What about `and` and `or`? Simple, `or` takes two arguments, and
returns either true or false if one of the inputs is true:
(define or (lambda (a b) (if a #t (if b #t #f))))
but `(if b #t #f)` is really the same as just `b` because it must be a
boolean (we cannot use more than one "truty" or "falsy" values):
(define or (lambda (a b) (if a #t b)))
also, if `a` is true, we want to return `#t`, but that is exactly the
value of `a`, so:
(define or (lambda (a b) (if a a b)))
and finally, we can get rid of the `if` (which is actually breaking the
`if` abstraction, if we encode booleans in some other way):
(define or (lambda (a b) (a a b)))
Similarly, you can convince yourself that the definition of `and` is:
(define and (lambda (a b) (a b a)))
Schlac has toRacket conversion functions for booleans too:
(>bool (or #f #f))
(>bool (or #f #t))
(>bool (or #t #f))
(>bool (or #t #t))
and
(>bool (and #f #f))
(>bool (and #f #t))
(>bool (and #t #f))
(>bool (and #t #t))
A `not` function is quite simple  one alternative is to choose from
true and false in the usual way:
(define not (lambda (a) (a #f #t)))
and another is to return a function that switches the inputs to an input
boolean:
(define not (lambda (a) (lambda (x y) (a y x))))
which is the same as
(define not (lambda (a x y) (a y x)))
We can now put numbers and booleans together: we define a `zero?`
function.
(define zero? (lambda (n) (n (lambda (x) #f) #t)))
(test (>bool (and (zero? 0) (not (zero? 3)))) => '#t)
(Good question: is this fast?)
(Note that it is better to test that the value is explicitly `#t`, if we
just use `(test (>bool ...))` then the test will work even if the
expression in question evaluated to some bogus value.)
The idea is simple  if `n` is the encoding of zero, it will return
it's second argument which is `#t`:
(zero? 0) > ((lambda (f n) n) (lambda (x) #f) #t) > #t
if `n` is an encoding of a bigger number, then it is a selfcomposition,
and the function that we give it is one that always returns `#f`, no
matter how many times it is selfcomposed. Try `2` for example:
(zero? 2) > ((lambda (f n) (f (f n))) (lambda (x) #f) #t)
> ((lambda (x) #f) ((lambda (x) #f) #t))
> #f
Now, how about an encoding for compound values? A minimal approach is
what we use in Racket  a way to generate pairs (`cons`), and encode
lists as chains of pairs with a special value at the end (`null`).
There is a natural encoding for pairs that we have previously seen  a
pair is a function that expects a selector, and will apply that on the
two values:
(define cons (lambda (x y) (lambda (s) (s x y))))
Or, equivalently:
(define cons (lambda (x y s) (s x y)))
To extract the two values from a pair, we need to pass a selector that
consumes two values and returns one of them. In our framework, this is
exactly what the two boolean values do, so we get:
(define car (lambda (x) (x #t)))
(define cdr (lambda (x) (x #f)))
(>nat (+ (car (cons 2 3)) (cdr (cons 2 3))))
We can even do this:
(define 1st car)
(define 2nd (lambda (l) (car (cdr l))))
(define 3rd (lambda (l) (car (cdr (cdr l)))))
(define 4th (lambda (l) (car (cdr (cdr (cdr l))))))
(define 5th (lambda (l) (car (cdr (cdr (cdr (cdr l)))))))
or write a `listref` function:
(define listref (lambda (l n) (car (n cdr l))))
Note that we don't need a recursive function for this: our encoding of
natural numbers makes it easy to "iterate N times". What we get with
this encoding is essentially free naturalnumber recursion.
We now need a special `null` value to mark list ends. This value should
have the same number of arguments as a `cons` value (one: a
selector/boolean function), and it should be possible to distinguish it
from other values. We choose
(define null (lambda (s) #t))
Testing the list encoding:
(define l123 (cons 1 (cons 2 (cons 3 null))))
(>nat (2nd l123))
And as with natural numbers and booleans, Schlac has builtin facility
to convert encoded lists to Racket values, except that this requires
specifying the type of values in a list so it's a higherorder function:
((>listof >nat) l123)
which ("as usual") can be written as
(>listof >nat l123)
We can even do this:
(>listof (>listof >nat) (cons l123 (cons l123 null)))
Defining `null?` is now relatively easy (and it's actually already used
by the above `>listof` conversion). The following definition
(define null? (lambda (x) (x (lambda (x y) #f))))
works because if `x` is null, then it simply ignores its argument and
returns `#t`, and if it's a pair, then it uses the input selector, which
always returns `#f` in its turn. Using some arbitrary `A` and `B`:
(null? (cons A B))
> ((lambda (x) (x (lambda (x y) #f))) (lambda (s) (s A B)))
> ((lambda (s) (s A B)) (lambda (x y) #f))
> ((lambda (x y) #f) A B)
> #f
(null? null)
> ((lambda (x) (x (lambda (x y) #f))) (lambda (s) #t))
> ((lambda (s) #t) (lambda (x y) #f))
> #t
We can use the Y combinator to create recursive functions  we can
even use the rewrite rules facility that Schlac contains (the same one
that we have previously seen):
(define Y
(lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x))))))
(rewrite (define/rec f E) => (define f (Y (lambda (f) E))))
and using it:
(define/rec length
(lambda (l)
(if (null? l)
0
(add1 (length (cdr l))))))
(>nat (length l123))
And to complete this, um, journey  we're still missing subtraction.
There are many ways to solve the problem of subtraction, and for a
challenge try to come up with a solution yourself. One of the clearer
solutions uses a simple idea  begin with a pair of two zeroes
`<0,0>`, and repeat this transformation `n` times: `` > ``.
After `n` steps, we will have ``  so we get:
(define inccons (lambda (p) (cons (cdr p) (add1 (cdr p)))))
(define sub1 (lambda (n) (car (n inccons (cons 0 0)))))
(>nat (sub1 5))
And from this the road is short to general subtraction, `m``n` is
simply `n` applications of `sub1` on `m`:
(define  (lambda (m n) (n sub1 m)))
(test (>nat ( 3 2)) => '1)
(test (>nat ( (* 4 (* 5 5)) 5)) => '95)
We now have a normallooking language, and we're ready to do anything we
want. Here are two popular examples:
(define/rec fact
(lambda (x)
(if (zero? x) 1 (* x (fact (sub1 x))))))
(test (>nat (fact 5)) => '120)
(define/rec fib
(lambda (x)
(if (or (zero? x) (zero? (sub1 x)))
1
(+ (fib ( x 1)) (fib ( x 2))))))
(test (>nat (fib (* 5 2))) => '89)
To get generalized arithmetic capability, Schlac has yet another
builtin facility for translating Racket natural numbers into Church
numerals:
(>nat (fib (nat> '10)))
... and to get to that frightening expression in the beginning, all you
need to do is replace all definitions in the `fib` definition over and
over again until you're left with nothing but lambda expressions and
applications, then reformat the result into some cute shape. For extra
fun, you can look for immediate applications of lambda expressions and
reduce them manually.
All of this is in the following code:
;;; <<>>
;; Making Schlac into a practical language (not an interpreter)
#lang pl schlac
(define identity (lambda (x) x))
;; Natural numbers
(define 0 (lambda (f x) x))
(define add1 (lambda (n) (lambda (f x) (f (n f x)))))
;; same as:
;; (define add1 (lambda (n) (lambda (f x) (n f (f x)))))
(define 1 (add1 0))
(define 2 (add1 1))
(define 3 (add1 2))
(define 4 (add1 3))
(define 5 (add1 4))
(test (>nat (add1 (add1 5))) => '7)
(define + (lambda (m n) (m add1 n)))
(test (>nat (+ 4 5)) => '9)
;; (define * (lambda (m n) (m (+ n) 0)))
(define * (lambda (m n f) (m (n f))))
(test (>nat (* 4 5)) => '20)
(test (>nat (+ 4 (* (+ 2 5) 5))) => '39)
;; (define ^ (lambda (m n) (n (* m) 1)))
(define ^ (lambda (m n) (n m)))
(test (>nat (^ 3 4)) => '81)
;; Booleans
(define #t (lambda (x y) x))
(define #f (lambda (x y) y))
(define if (lambda (c t e) (c t e))) ; not really needed
(test (>nat (if #t 1 2)) => '1)
(test (>nat (if #t (+ 4 5) (+ '1 '2))) => '9)
(define and (lambda (a b) (a b a)))
(define or (lambda (a b) (a a b)))
;; (define not (lambda (a) (a #f #t)))
(define not (lambda (a x y) (a y x)))
(test (>bool (and #f #f)) => '#f)
(test (>bool (and #t #f)) => '#f)
(test (>bool (and #f #t)) => '#f)
(test (>bool (and #t #t)) => '#t)
(test (>bool (or #f #f)) => '#f)
(test (>bool (or #t #f)) => '#t)
(test (>bool (or #f #t)) => '#t)
(test (>bool (or #t #t)) => '#t)
(test (>bool (not #f)) => '#t)
(test (>bool (not #t)) => '#f)
(define zero? (lambda (n) (n (lambda (x) #f) #t)))
(test (>bool (and (zero? 0) (not (zero? 3)))) => '#t)
;; Lists
(define cons (lambda (x y s) (s x y)))
(define car (lambda (x) (x #t)))
(define cdr (lambda (x) (x #f)))
(test (>nat (+ (car (cons 2 3)) (cdr (cons 2 3)))) => '5)
(define 1st car)
(define 2nd (lambda (l) (car (cdr l))))
(define 3rd (lambda (l) (car (cdr (cdr l)))))
(define 4th (lambda (l) (car (cdr (cdr (cdr l))))))
(define 5th (lambda (l) (car (cdr (cdr (cdr (cdr l)))))))
(define null (lambda (s) #t))
(define null? (lambda (x) (x (lambda (x y) #f))))
(define l123 (cons 1 (cons 2 (cons 3 null))))
;; Note that `>listof' is a H.O. converter
(test ((>listof >nat) l123) => '(1 2 3))
(test (>listof >nat l123) => '(1 2 3)) ; same as the above
(test (>listof (>listof >nat) (cons l123 (cons l123 null)))
=> '((1 2 3) (1 2 3)))
;; Subtraction is tricky
(define inccons (lambda (p) (cons (cdr p) (add1 (cdr p)))))
(define sub1 (lambda (n) (car (n inccons (cons 0 0)))))
(test (>nat (sub1 5)) => '4)
(define  (lambda (a b) (b sub1 a)))
(test (>nat ( 3 2)) => '1)
(test (>nat ( (* 4 (* 5 5)) 5)) => '95)
(test (>nat ( 2 4)) => '0) ; this is "natural subtraction"
;; Recursive functions
(define Y
(lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x))))))
(rewrite (define/rec f E) => (define f (Y (lambda (f) E))))
(define/rec length
(lambda (l)
(if (null? l)
0
(add1 (length (cdr l))))))
(test (>nat (length l123)) => '3)
(define/rec fact
(lambda (x)
(if (zero? x) 1 (* x (fact (sub1 x))))))
(test (>nat (fact 5)) => '120)
(define/rec fib
(lambda (x)
(if (or (zero? x) (zero? (sub1 x)))
1
(+ (fib (sub1 x)) (fib (sub1 (sub1 x)))))))
(test (>nat (fib (* 5 2))) => '89)
#
;; Fullyexpanded Fibonacci
(define fib
((lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x)))))
(lambda (f)
(lambda (x)
((lambda (c t e) (c t e))
((lambda (a b) (a a b))
((lambda (n)
(n (lambda (x) (lambda (x y) y)) (lambda (x y) x)))
x)
((lambda (n)
(n (lambda (x) (lambda (x y) y)) (lambda (x y) x)))
((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
x)))
((lambda (n) (lambda (f x) (f (n f x)))) (lambda (f x) x))
((lambda (x y)
(x (lambda (n) (lambda (f x) (f (n f x)))) y))
(f ((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
x))
(f ((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
x)))))))))
;; The same after reducing all immediate function applications
(define fib
((lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x)))))
(lambda (f)
(lambda (x)
(((x (lambda (x) (lambda (x y) y)) (lambda (x y) x))
(x (lambda (x) (lambda (x y) y)) (lambda (x y) x))
(((x (lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x))
(lambda (x) (lambda (x y) y))
(lambda (x y) x)))
(lambda (f x) (f x))
((f ((x (lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (y s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x)))
(lambda (n) (lambda (f x) (f (n f x))))
(f ((((x (lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x))
(lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x)))))))))
;; Cute reformatting of the above:
(define fib((lambda(f)((lambda(x)(x x))(lambda(x)(f(x x)))))(lambda(
f)(lambda(x)(((x(lambda(x)(lambda(x y)y))(lambda(x y)x))(x(lambda(x)
(lambda(x y)y))(lambda(x y) x))(((x(lambda(p)(lambda(s)(s(p(lambda(x
y)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(s) (s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x))(lambda(x)(lambda(x y)y))(lambda
(x y)x)))(lambda(f x)(f x))((f((x(lambda(p)(lambda(s)(s(p(lambda(x y
)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(y s)(s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x)))(lambda(n)(lambda(f x)(f(n f x)
)))(f((((x(lambda(p)(lambda(s)(s(p (lambda(x y)y))(lambda(f x)(f((p(
lambda(x y) y))f x))))))(lambda(s)(s(lambda(f x)x)(lambda(f x)x))))(
;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
;; `(cons 0 0)'
lambda(x y)x))(lambda(p)(lambda(s)(s(p(lambda(x y)y))(lambda(f x)(f(
(p(lambda(x y)y))f x))))))(lambda(s)(s(lambda(f x)x)(lambda(f x)x)))
)(lambda(x y)x)))))))))
And for extra fun:
(λ(f)(λ
(x)(((x(λ(
x)(λ(x y)y)
)(λ(x y)x))(
x(λ(x)(λ(x y)
y))(λ(x y
)x))(((
x(λ(p)(
λ(s)(s
(p (λ(
x y)y))
(λ(f x
)(f((p(
λ(x y)
y))f x
))))))(
λ(s)(s(
λ(f x)x)
(λ(f x)x)
)))(λ(x y)
x))(λ(x)(λ(
x y)y)) (λ(
x y) x)))(λ(
f x)(f x))((f
((x(λ(p )(λ (s
)(s(p( λ(x y)
y))(λ ( f x)(f(
(p (λ( x y)y)
)f x))) )))(λ(
y s)(s (λ (f x
)x)(λ( f x)x)
)))(λ( x y)x))
)(λ(n) (λ (f
x)(f (n f x)))
)(f((( (x(λ(p)
(λ(s)(s (p( λ(
x y )y ))(λ(f
x) (f(( p(λ(x y
)y)) f x)))))
)(λ(s)( s(λ(f x
)x)(λ( f x)x)
))) (λ (x y)x
))(λ(p )(λ(s)(
s(p(λ( x y)y)
)(λ (f x)(f((
p(λ (x y)y)) f
x)))))) (λ(s)(
s(λ (f x)x)(λ
(f x)x) )))(λ(
x y)x) ))))))
#

# Alternative Church Encoding
Finally, note that this is just one way to encode things  other
encodings are possible. One alternative encoding is in the following
code  it uses a list of `N` falses as the encoding for `N`. This
encoding makes it easier to `add1` (just `cons` another `#f`), and to
`sub1` (simply `cdr`). The tradeoff is that some arithmetics operations
becomes more complicated, for example, the definition of `+` requires
the fixpoint combinator. (As expected, some people want to see what can
we do with a language without recursion, so they don't like jumping to Y
too fast.)
;;; <<>>
;; An alternative "Church" encoding: use lists to encode numbers
#lang pl schlac
(define identity (lambda (x) x))
;; Booleans (same as before)
(define #t (lambda (x y) x))
(define #f (lambda (x y) y))
(define if (lambda (c t e) (c t e))) ; not really needed
(test (>bool (if #t #f #t))
=> '#f)
(test (>bool (if #f ((lambda (x) (x x)) (lambda (x) (x x))) #t))
=> '#t)
(define and (lambda (a b) (a b a)))
(define or (lambda (a b) (a a b)))
(define not (lambda (a x y) (a y x)))
(test (>bool (and #f #f)) => '#f)
(test (>bool (and #t #f)) => '#f)
(test (>bool (and #f #t)) => '#f)
(test (>bool (and #t #t)) => '#t)
(test (>bool (or #f #f)) => '#f)
(test (>bool (or #t #f)) => '#t)
(test (>bool (or #f #t)) => '#t)
(test (>bool (or #t #t)) => '#t)
(test (>bool (not #f)) => '#t)
(test (>bool (not #t)) => '#f)
;; Lists (same as before)
(define cons (lambda (x y s) (s x y)))
(define car (lambda (x) (x #t)))
(define cdr (lambda (x) (x #f)))
(define 1st car)
(define 2nd (lambda (l) (car (cdr l))))
(define 3rd (lambda (l) (car (cdr (cdr l)))))
(define 4th (lambda (l) (car (cdr (cdr (cdr l))))))
(define 5th (lambda (l) (car (cdr (cdr (cdr (cdr l)))))))
(define null (lambda (s) #t))
(define null? (lambda (x) (x (lambda (x y) #f))))
;; Natural numbers (alternate encoding)
(define 0 identity)
(define add1 (lambda (n) (cons #f n)))
(define zero? car) ; tricky
(define sub1 cdr) ; this becomes very simple
;; Note that we could have used something more straightforward:
;; (define 0 null)
;; (define add1 (lambda (n) (cons #t n))) ; cons anything
;; (define zero? null?)
;; (define sub1 (lambda (l) (if (zero? l) l (cdr l))))
(define 1 (add1 0))
(define 2 (add1 1))
(define 3 (add1 2))
(define 4 (add1 3))
(define 5 (add1 4))
(test (>nat* (add1 (add1 5))) => '7)
(test (>nat* (sub1 (sub1 (add1 (add1 5))))) => '5)
(test (>bool (and (zero? 0) (not (zero? 3)))) => '#t)
(test (>bool (zero? (sub1 (sub1 (sub1 3))))) => '#t)
;; listofnumbers tests
(define l123 (cons 1 (cons 2 (cons 3 null))))
(test (>listof >nat* l123) => '(1 2 3))
(test (>listof (>listof >nat*) (cons l123 (cons l123 null)))
=> '((1 2 3) (1 2 3)))
;; Recursive functions
(define Y
(lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x))))))
(rewrite (define/rec f E) => (define f (Y (lambda (f) E))))
;; note that this example is doing something silly now
(define/rec length
(lambda (l)
(if (null? l)
0
(add1 (length (cdr l))))))
(test (>nat* (length l123)) => '3)
;; addition becomes hard since it requires a recursive definition
;; (define/rec +
;; (lambda (m n) (if (zero? n) m (+ (add1 m) (sub1 n)))))
;; (test (>nat* (+ 4 5)) => '9)
;; faster alternative:
(define/rec +
(lambda (m n)
(if (zero? m) n
(if (zero? n) m
(add1 (add1 (+ (sub1 m) (sub1 n))))))))
(test (>nat* (+ 4 5)) => '9)
;; subtraction is similar to addition
;; (define/rec 
;; (lambda (m n) (if (zero? n) m ( (sub1 m) (sub1 n)))))
;; (test (>nat* ( (+ 4 5) 4)) => '5)
;; but this is not "natural subtraction": doesn't work when n>m,
;; because (sub1 0) does not return 0.
;; a solution is like alternative form of +:
(define/rec 
(lambda (m n)
(if (zero? m) 0
(if (zero? n) m
( (sub1 m) (sub1 n))))))
(test (>nat* ( (+ 4 5) 4)) => '5)
(test (>nat* ( 2 5)) => '0)
;; alternatively, could change sub1 above:
;; (define sub1 (lambda (n) (if (zero? n) n (cdr n))))
;; we can do multiplication in a similar way
(define/rec *
(lambda (m n)
(if (zero? m) 0
(+ n (* (sub1 m) n)))))
(test (>nat* (* 4 5)) => '20)
(test (>nat* (+ 4 (* (+ 2 5) 5))) => '39)
;; and the rest of the examples
(define/rec fact
(lambda (x)
(if (zero? x) 1 (* x (fact (sub1 x))))))
(test (>nat* (fact 5)) => '120)
(define/rec fib
(lambda (x)
(if (or (zero? x) (zero? (sub1 x)))
1
(+ (fib (sub1 x)) (fib (sub1 (sub1 x)))))))
(test (>nat* (fib (* 5 2))) => '89)
#
;; Fullyexpanded Fibonacci (note: much shorter than the previous
;; encoding, but see how Y appears twice  two "((lambda" pairs)
(define fib((lambda(f)((lambda(x)(x x))(lambda(x)(f(x x)))))(lambda(
f)(lambda(x)(((((x(lambda(x y)x))(x(lambda(x y)x)))((x(lambda(x y)y)
)(lambda(x y)x)))(lambda(s)(s(lambda(x y)y)(lambda(x)x))))((((lambda
(f)((lambda(x)(x x))(lambda(x)(f(x x))))) (lambda(f)(lambda(m n)((m(
lambda(x y)x))n (((n(lambda(x y)x)) m)(lambda(s)((s (lambda(x y)y))(
lambda(s)((s (lambda(x y)y))((f(m(lambda(x y)y)))(n(lambda(x y)y))))
))))))))(f(x(lambda(x y)y))))(f((x(lambda(x y)y))(lambda(x y)y))))))
)))
#

# Implementing `definetype` + `cases` in Schlac
Another interesting way to implement lists follows the pattern matching
approach, where both pairs and the null value are represented by a
function that serves as a kind of a `match` dispatcher. This function
takes in two inputs  if it is the representation of null then it will
return the first input, and if it is a pair, then it will apply the
second input on the two parts of the pair. This is implemented as
follows (with type comments to make it clear):
;; null : List
(define null
(lambda (n p)
n))
;; cons : A List > List
(define cons
(lambda (x y)
(lambda (n p)
(p x y))))
This might seem awkward, but it follows the intended use of pairs and
null as a matchlike construct. Here is an example, with the equivalent
Racket code on the side:
;; Sums up a list of numbers
(define (sum l)
(l ; (match l
0 ; ['() 0]
(lambda (x xs) ; [(cons x xs)
(+ x (sum xs))))) ; (+ x (sum xs))])
In fact, it's easy to implement our selectors and predicate using this:
(define null? (lambda (l) (l #t (lambda (x xs) #f))))
(define car (lambda (l) (l #f (lambda (x xs) x))))
(define cdr (lambda (l) (l #f (lambda (x xs) xs))))
;; in the above `#f' is really any value, since it
;; should be an error alternatively:
(define car (lambda (l)
(l ((lambda (x) (x x)) (lambda (x) (x x))) ; "error"
(lambda (x y) x))))
The same approach can be used to define any kind of new data type in a
way that looks like our own `definetype` definitions. For example,
consider a muchsimplified definition of the AE type we've seen early in
the semester, and a matching `eval` definition as an example for using
`cases`:
(definetype AE
[Num Number]
[Add AE AE])
(: eval : AE > Number)
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]))
We can follow the above approach now to write Schlac code that more than
being equivalent, is also very similar in nature. Note that the type
definition is replaced by two definitions for the two constructors:
(define Num (lambda (n) (lambda (num add) (num n ))))
(define Add (lambda (l r) (lambda (num add) (add l r))))
(define/rec eval
(lambda (expr) ; `expr` is always a (lambda (num add) ...), and it
; expects a unary `num` argument and a binary `add`
(expr (lambda (n) n)
(lambda (l r) (+ (eval l) (eval r))))))
(test (>nat (eval (Add (Num 1) (Num 2)))) => '3)
We can even take this further: the translations from `definetype` and
`cases` are mechanical enough that we could implement them almost
exactly via rewrites (there are a subtle change in that we're list field
names rather than types):
(rewrite (definetype ignoredType [Variant arg ...] ...)
=> (define Variant
(lambda (arg ...)
(lambda (Variant ...) (Variant arg ...))))
...)
(rewrite (cases value [(ignoredVariant arg ...) result] ...)
=> (value (lambda (arg ...) result) ...))
And using that, an evluator is simple:
(definetype AE [Num n] [Add l r] [Sub l r] [Mul l r])
(define/rec eval
(lambda (expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))])))
(test (>nat (eval (Mul (Add (Num 1) (Num 2))
(Sub (Num 4) (Num 2)))))
=> '6)

# Recursive Environments
> [PLAI §11.5]
What we really need for recursion, is a special kind of an environment,
one that can refer to itself. So instead of doing (note: `call`s
removed for readability):
{with {fact {fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}
{fact 5}}
which does not work for the usual reasons, we want to use some
{rec {fact {fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}
{fact 5}}
that will do the necessary magic.
One way to achieve this is using the Y combinator as we have seen  a
kind of a "constructor" for recursive functions. We can do that in a
similar way to the `rewrite` rule that we have seen in Schlac 
translate the above expression to:
{with {fact {makerec
{fun {fact}
{fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}}}
{fact 5}}
or even:
{with {fact {{fun {f} {{fun {x} {f {x x}}} {fun {x} {f {x x}}}}}
{fun {fact}
{fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}}}
{fact 5}}
Now, we will see how it can be used in *our* code to implement a
recursive environment.
If we look at what `with` does in
{with {fact {fun {n}
{if {zero? n} 1 {* n {call fact { n 1}}}}}}
{call fact 5}}
then we can say that to evaluate this expression, we evaluate the body
expression in an extended environment that contains `fact`, even if a
bogus one that is good for `0` only  the new environment is created
with something like this:
extend("fact", makefactclosure(), env)
so we can take this whole thing as an operation over `env`
addfact(env) := extend("fact", makefactclosure(), env)
This gives us the firstlevel fact. But `fact` itself is still
undefined in `env`, so it cannot call itself. We can try this:
addfact(addfact(env))
but that still doesn't work, and it will never work no matter how far we
go:
addfact(addfact(addfact(addfact(addfact(...env...)))))
What we really want is infinity: a place where addfact works and the
result is the same as what we've started with  we want to create a
"magical" environment that makes this possible:
let magicenv = ???
such that:
addfact(magicenv) = magicenv
which basically gives us the illusion of being at the infinity point.
This magicenv thing is exactly the *fixedpoint* of the `addfact`
operation. We can use:
magicenv = rec(addfact)
and following the main property of the Y combinator, we know that:
magicenv = rec(addfact) ; def. of magicenv
= addfact(rec(addfact)) ; Y(f) = f(Y(f))
= addfact(magicenv) ; def. of magicenv
What does all this mean? It means that if we have a fixedpoint
operator at the level of the implementation of our environments, then we
can use it to implement a recursive binder. In our case, this means
that a fixpoint in Racket can be used to implement a recursive language.
But we have that  Racket does have recursive functions, so we should
be able to use that to implement our recursive binder.
There are two ways that make it possible to write recursive functions in
Racket. One is to define a function, and use its name to do a recursive
call  using the Racket formal rules, we can see that we said that we
mark that we now *know* that a variable is bound to a value. This is
essentially a sideeffect  we modify what we know, which corresponds
to modifying the global environment. The second way is a new form:
`letrec`. This form is similar to `let`, except that the scope that is
established includes the named expressions  it is exactly what we
want `rec` to do. A third way is using recursive local definitions, but
that is equivalent to using `letrec`, more on this soon.

# Recursion: Racket's `letrec`
So we want to add recursion to our language, practically. We already
know that Racket makes it possible to write recursive functions, which
is possible because of the way it implements its "global environment":
our evaluator can only *extend* an environment, while Racket *modifies*
its global environment. This means that whenever a function is defined
in the global environment, the resulting closure will have it as its
environment "pointer", but the global environment was not extended 
it stays the same, and was just modified with one additional binding.
But Racket has another, a bit more organized way of using recursion:
there is a special localbinding construct that is similar to `let`, but
allows a function to refer to itself. It is called `letrec`:
(letrec ([fact (lambda (n)
(if (zero? n)
1
(* n (fact ( n 1)))))])
(fact 5))
Some people may remember that there was a third way for creating
recursive functions: using local definition in function bodies. For
example, we have seen things like:
(define (length list)
(define (helper list len)
(if (null? list)
len
(helper (rest list) (+ len 1))))
(helper list 0))
This looks like the same kind of environment magic that happens with a
global `define`  but actually, Racket defines the meaning of internal
definitions using `letrec`  so the above code is exactly the same as:
(define (length list)
(letrec ([helper (lambda (list len)
(if (null? list)
len
(helper (rest list) (+ len 1))))])
(helper list 0)))
The scoping rules for a `letrec` is that the scope of the bound name
covers both the body *and* the named expression. Furthermore, multiple
names can be bound to multiple expressions, and the scope of each name
covers all named expression as well as the body. This makes it easy to
define mutually recursive functions, such as:
(letrec ([even? (lambda (n) (if (zero? n) #t (odd? ( n 1))))]
[odd? (lambda (n) (if (zero? n) #f (even? ( n 1))))])
(even? 99))
But it is not a required functionality  it could be done with a
single recursive binding that contains several functions:
(letrec ([even+odd
(list (lambda (n)
(if (zero? n) #t ((second even+odd) ( n 1))))
(lambda (n)
(if (zero? n) #f ((first even+odd) ( n 1)))))])
((first even+odd) 99))
This is basically the same problem we face if we want to use the Y
combinator for mutually recursive bindings. The above solution is
inconvenient, but it can be improved using more `let`s to have easier
name access. For example:
(letrec ([even+odd
(list (lambda (n)
(let ([even? (first even+odd)]
[odd? (second even+odd)])
(if (zero? n) #t (odd? ( n 1)))))
(lambda (n)
(let ([even? (first even+odd)]
[odd? (second even+odd)])
(if (zero? n) #f (even? ( n 1))))))])
(let ([even? (first even+odd)]
[odd? (second even+odd)])
(even? 99)))

# Implementing Recursion using `letrec`
We will see how to add a similar construct to our language  for
simplicity, we will add a `rec` form that handles a single binding:
{rec {fact {fun {n}
{if {= 0 n}
1
{* n {fact { n 1}}}}}}
{fact 5}}
Using this, things can get a little tricky. What should we get if we
do:
{rec {x x} x}
? Currently, it seems like there is no point in using any expression
except for a *function* expression in a `rec` expression, so we will
handle only these cases.
(BTW, under what circumstances would nonfunction values be useful in a
letrec?)

One way to achieve this is to use the same trick that we have recently
seen: instead of reimplementing language features, we can use existing
features in our own language, which hopefully has the right
functionality in a form that can be reused to in our evaluator.
Previously, we have seen a way to implement environments using Racket
closures:
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: lookup : Symbol ENV > VAL)
(define (lookup name env)
(env name))
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
We can use this implementation, and create circular environments using
Racket's `letrec`. The code for handling a `with` expressions is:
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
It looks like we should be able to handle `rec` in a similar way (the
AST constructor name is `WRec` ("withrec") so it doesn't collide with
TR's `Rec` constructor for recursive types):
[(WRec boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
but this won't work because the named expression is evaluated
prematurely, in the previous environment. Instead, we will move
everything that needs to be done, including evaluation, to a separate
`extendrec` function:
[(WRec boundid namedexpr boundbody)
(eval boundbody
(extendrec boundid namedexpr env))]
Now, the `extendrec` function needs to provide the new, "magically
circular" environment. Following what we know about the arguments to
`extendrec`, and the fact that it returns a new environment (= a lookup
function), we can sketch a rough definition:
(: extendrec : Symbol FLANG ENV > ENV) ; FLANG, not VAL!
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(lambda (name)
(if (eq? name id)
... something that uses expr to get a value ...
(restenv name))))
What should the missing expression be? It can simply evaluate the
object given itself:
(define (extendrec id expr restenv)
(lambda (name)
(if (eq? name id)
(eval expr ...this environment...)
(restenv name))))
But how do we get *this environment*, before it is defined? Well, the
environment is itself a Racket *function*, so we can use Racket's
`letrec` to make the function refer to itself recursively:
(define (extendrec id expr restenv)
(letrec ([recenv (lambda (name)
(if (eq? name id)
(eval expr recenv)
(restenv name)))])
recenv))
It's a little more convenient to use an internal definition, and add a
type for clarity:
(define (extendrec id expr restenv)
(: recenv : Symbol > VAL)
(define (recenv name)
(if (eq? name id)
(eval expr recenv)
(restenv name)))
recenv)
This works, but there are several problems:
1. First, we no longer do a simple lookup in the new environment.
Instead, we evaluate the expression on *every* such lookup. This
seems like a technical point, because we do not have sideeffects in
our language (also because we said that we want to handle only
function expressions). Still, it wastes space since each evaluation
will allocate a new closure.
2. Second, a related problem  what happens if we try to run this:
{rec {x x} x}
? Well, we do that stuff to extend the current environment, then
evaluate the body in the new environment, this body is a single
variable reference:
(eval (Id 'x) thenewenv)
so we look up the value:
(lookup 'x thenewenv)
which is:
(thenewenv 'x)
which goes into the function which implements this environment, there
we see that `name` is the same as `name1`, so we return:
(eval expr recenv)
but the `expr` here is the original namedexpression which is itself
`(Id 'x)`, and we're in an infinite loop.
We can try to get over these problems using another binding. Racket
allows several bindings in a single `letrec` expression or multiple
internal function definitions, so we change `extendrec` to use the
newlycreated environment:
(define (extendrec id expr restenv)
(: recenv : Symbol > VAL)
(define (recenv name)
(if (eq? name id)
val
(restenv name)))
(: val : VAL)
(define val (eval expr recenv))
recenv)
This runs into an interesting type error, which complains about possibly
getting some `Undefined` value. It does work if we switch to the
untyped language for now (using `#lang pl untyped`)  and it seems to
run fine too. But it raises more questions, beginning with: what is the
meaning of:
(letrec ([x ...]
[y ...x...])
...)
or equivalently, an internal block of
(define x ...)
(define y ...x...)
? Well, DrRacket seems to do the "right thing" in this case, but what
about:
(letrec ([y ...x...]
[x ...])
...)
? As a hint, see what happens when we now try to evaluate the
problematic
{rec {x x} x}
expression, and compare that with the result that you'd get from Racket.
This also clarifies the type error that we received.
It should be clear now why we want to restrict usage to just binding
recursive functions. There are no problems with such definitions
because when we evaluate a `fun` expression, there is no evaluation of
the body, which is the only place where there are potential references
to the same function that is defined  a function's body is *delayed*,
and executed only when the function is applied later.
But the biggest question that is still open: we just implemented a
circular environment using Racket's own circular environment
implementation, and that does not explain how they are actually
implemented. The cycle of pointers that we've implemented depends on
the cycle of pointers that Racket uses, and that is a black box we want
to open up.
For reference, the complete code is below.
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }
 { rec { } }

 { fun { } }
 { call }
Evaluation rules:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({rec {x E1} E2},env) = ???
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2,env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[WRec Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'rec more)
(match sexpr
[(list 'rec (list (symbol: name) named) body)
(WRec name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `rec' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(: recenv : Symbol > VAL)
(define (recenv name)
(if (eq? name id)
val
(restenv name)))
(: val : VAL)
(define val (eval expr recenv))
recenv)
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(WRec boundid namedexpr boundbody)
(eval boundbody
(extendrec boundid namedexpr env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Implementing `rec` Using Cyclic Structures
> [PLAI §10]
Looking at the arrows in the environment diagrams, what we're really
looking for is a closure that has an environment pointer which is the
same environment in which it was defined. This will make it possible
for `fact` to be bound to a closure that can refer to *itself* since its
environment is the same one in which it is defined. However, so far we
have no tools that makes it possible to do this.
What we need is to create a "cycle of pointers", and so far we do not
have a way of achieving that: when we create a closure, we begin with an
environment which is saved in the slot's environment slot, but we want
that closure to be the value of a binding in that same environment.

# Boxes and Mutation
To actually implement a circular structure, we will now use
*sideeffects*, using a new kind of Racket value which supports
mutation: a box. A box value is built with the `box` constructor:
(define mything (box 7))
the value is retrieved with the `unbox' function,
(* 6 (unbox mything))
and finally, the value can be changed with the `setbox!` function.
(setbox! mything 17)
(* 6 (unbox mything))
An important thing to note is that `setbox!` is much like `display`
etc, it returns a value that is not printed in the Racket REPL, because
there is no point in using the result of a `setbox!`, it is called for
the sideeffect it generates. (Languages like C blur this distinction
between returning a value and a sideeffect with its assignment
statement.)
As a side note, we now have side effects of two kinds: mutation of
state, and I/O (at least the O part). (Actually, there is also infinite
looping that can be viewed as another form of a side effect.) This
means that we're now in a completely different world, and lots of new
things can make sense now. A few things that you should know about:
* We never used more than one expression in a function body because
there was no point in it, but now there is. To evaluate a sequence of
Racket expressions, you wrap them in a `begin` expression.
* In most places you don't actually need to use `begin`  these are
places that are said to have an *implicit* `begin`: the body of a
function (or any lambda expression), the body of a `let` (and
`let`relatives), the consequence positions in `cond`, `match`, and
`cases` clauses and more. One of the common places where a `begin` is
used is in an `if` expression (and some people prefer using `cond`
instead when there is more than a single expression).
* `cond` without an `else` in the end can make sense, if all you're
using it it for is sideeffects.
* `if` could get a single expression which is executed when the
condition is true (and an unspecified value is used otherwise), but
our language (as well as the default Racket language) always forbids
this  there are convenient special forms for a onesided `if`s:
`when` & `unless`, and they can have any number of expressions (they
have an implicit `begin`). They have an advantage of saying "this
code does some sideeffects here" more explicit.
* There is a function called `foreach` which is just like `map`, except
that it doesn't collect the list of results, it is used only for
performing side effects.
When any one of these things is used (in Racket or other languages), you
can tell that sideeffects are involved, because there is no point in
any of them otherwise. In addition, any name that ends with a `!`
("bang") is used to mark a function that changes state (usually a
function that only changes state).
So how do we create a cycle? Simple, boxes can have any value, and they
can be put in other values like lists, so we can do this:
#lang pl untyped
(define foo (list 1 (box 3)))
(setbox! (second foo) foo)
and we get a circular value. (Note how it is printed.) And with types:
#lang pl
(: foo : (List Number (Boxof Any)))
(define foo (list 1 (box 3)))
(setbox! (second foo) foo)

# Types for Boxes
Obviously, `Any` is not too great  it is the most generic type, so it
provides the least information. For example, notice that
(unbox (second foo))
returns the right list, which is equal to `foo` itself  but if we try
to grab some part of the resulting list:
(second (unbox (second foo)))
we get a type error, because the result of the `unbox` is `Any`, so
Typed Racket knows nothing about it, and won't allow you to treat it as
a list. It is not too surprising that the type constructor that can
help in this case is `Rec` which we have already seen  it allows a
type that can refer to itself:
#lang pl
(: foo : (Rec this (List Number (Boxof (U #f this)))))
(define foo (list 1 (box #f)))
(setbox! (second foo) foo)
Note that either `foo` or the value in the box are both printed with a
`Rec` type  the value in the box can't just have a `(U #f this)`
type, since `this` doesn't mean anything in there, so the whole type
needs to still be present.
There is another issue to be aware of with `Boxof` types. For most type
constructors (like `Listof`), if `T1` is a subtype of `T2`, then we also
know that`(Listof T1)` is a subtype of `(Listof T2)`. This makes the
following code typecheck:
#lang pl
(: foo : (Listof Number) > Number)
(define (foo l)
(first l))
(: bar : Integer > Number)
(define (bar x)
(foo (list x)))
Since the `(Listof Integer)` is a subtype of the `(Listof Number)` input
for `foo`, the application typechecks. But this is *not* the same for
the output type, for example  if we change the `bar` type to:
(: bar : Integer > Integer)
we get a type error since `Number` is not a subtype of `Integer`. So
subtypes are required to "go up" on the input side and "down" on the
other. So, in a sense, the fact that boxes are mutable means that their
contents can be considered to be on the other side of the arrow, which
is why for such `T1` subtype of `T2`, it is `(Boxof T2)` that is a
subtype of `(Boxof T1)`, instead of the usual. For example, this
doesn't work:
#lang pl
(: foo : (Boxof Number) > Number)
(define (foo b)
(unbox b))
(: bar : Integer > Number)
(define (bar x)
(: b : (Boxof Integer))
(define b (box x))
(foo b))
And you can see why this is the case  the marked line is fine given a
`Number` contents, so if the type checker allows passing in a box
holding an integer, then that expression would mutate the contents and
make it an invalid value.
However, boxes are not only mutable, they hold a value that can be read
too, which means that they're on *both* sides of the arrow, and this
means that `(Boxof T1)` is a subtype of `(Boxof T2)` if `T2` is a
subtype of `T1` *and* `T1` is a subtype of `T2`  in other words, this
happens only when `T1` and `T2` are the same type. (See below for an
extended demonstration of all of this.)
Note also that this demonstration requires that extra `b` definition, if
it's skipped:
(define (bar x)
(foo (box x)))
then this will typecheck again  Typed Racket will just consider the
context that requires a box holding a `Number`, and it is still fine to
initialize such a box with an `Integer` value.
> As a side comment, this didn't always work. Earlier in its existence,
> Typed Racket would always choose a specific type for values, which
> would lead to confusing errors with boxes. For example, the above
> would need to be written as
>
> (define (bar x)
> (foo (box (ann x : Number))))
>
> to prevent Typed Racket from inferring a specific type. This is no
> longer the case, but there can still be some surprises. A similar
> annotation was needed in the case of a list holding a selfreferential
> box, to avoid the initial `#f` from getting a specificbutwrong type.

# `Boxof`'s Lack of Subtyping
The lack of any subtype relations between `(Boxof T)` and `(Boxof S)`
regardless of `S` and `T` can roughly be explained as follows.
First, a box is a container that you can pull a value out of  which
makes it similar to lists. In the case of lists, we have:
if: S subtypeof T
then: (Listof S) subtypeof (Listof T)
This is true for all such containers that you can pull a value out of:
if you expect to pull a `T` but you're given a container of a subtype
`S`, then things are still fine. Such "containers" include functions
that produce a value  for example:
if: S subtypeof T
then: Q > S subtypeof Q > T
However, functions also have the other side, where things are different
 instead of a side of some *produced* value, it's the side of the
*consumed* value. We get the opposite rule there:
if: T subtypeof S
then: S > Q subtypeof T > Q
To see why this is right, use `Number` and `Integer` for `S` and `T`:
if: Integer subtypeof Number
then: Number > Q subtypeof Integer > Q
so  if you expect a function that takes a number is a *subtype* of
one that takes an integer; in other words, every function that takes a
number is also a function that takes an integer, but not the other way.
To summarize all of this, when you make the output type of a function
"smaller" (more constrained), the resulting type is smaller, but on the
input side things are flipped  a bigger input type means a more
constrained function.
Now, a `(Boxof T)` is a producer of `T` when you pull a value out of the
box, but it's also a consumer of `T` when you put such a value in it.
This means that  using the above analogy  the `T` is on both sides
of the arrow. This means that
if: S subtypeof T *and* T subtypeof S
then: (Boxof S) subtypeof (Boxof T)
which is actually:
if: S isthesametypeas T
then: (Boxof S) isthesametypeas (Boxof T)
A different way to look at this conclusion is to consider the function
type of `(A > A)`: when is it a subtype of some other `(B > B)`? Only
when `A` is a subtype of `B` and `B` is a subtype of `A`, which means
that this happens only when `A` and `B` are the same type.
(Side note: this is related to the fact that in logic, `P => Q` is
roughly equivalent to `not(P) or Q`  the left side, `P`, is inside a
negation. It also explains why in `((S > T) > Q)` the `S` obeys the
first rule, as if it was on the right side  because it's negated
twice.)
The following piece of code makes the analogy to function types more
formally. Boxes behave as if their contents is on both sides of a
function arrow  on the right because they're readable, and on the
left because they're writable, which the conclusion that a `(Boxof A)`
type is a subtype of itself and no other `(Boxof B)`.
#lang pl
;; a type for a "readonly" box
(definetype (Boxof/R A) = (> A))
;; Boxof/R constructor
(: box/r : (All (A) A > (Boxof/R A)))
(define (box/r x) (lambda () x))
;; we can see that (Boxof/R T1) is a subtype of (Boxof/R T2)
;; if T1 is a subtype of T2 (this is not surprising, since
;; these boxes are similar to any other container, like lists):
(: foo1 : Integer > (Boxof/R Integer))
(define (foo1 b) (box/r b))
(: bar1 : (Boxof/R Number) > Number)
(define (bar1 b) (b))
(test (bar1 (foo1 123)) => 123)
;; a type for a "writeonly" box
(definetype (Boxof/W A) = (A > Void))
;; Boxof/W constructor
(: box/w : (All (A) A > (Boxof/W A)))
(define (box/w x) (lambda (new) (set! x new)))
;; in contrast to the above, (Boxof/W T1) is a subtype of
;; (Boxof/W T2) if T2 is a subtype of T1, *not* the other way
;; (and note how this is related to A being on the *left* side
;; of the arrow in the `Boxof/W' type):
(: foo2 : Number > (Boxof/W Number))
(define (foo2 b) (box/w b))
(: bar2 : (Boxof/W Integer) Integer > Void)
(define (bar2 b new) (b new))
(test (bar2 (foo2 123) 456))
;; combining the above two into a type for a "read/write" box
(definetype (Boxof/RW A) = (A > A))
;; Boxof/RW constructor
(: box/rw : (All (A) A > (Boxof/RW A)))
(define (box/rw x) (lambda (new) (let ([old x]) (set! x new) old)))
;; this combines the above two: `A' appears on both sides of the
;; arrow, so (Boxof/RW T1) is a subtype of (Boxof/RW T2) if T1
;; is a subtype of T2 (because there's an A on the right) *and*
;; if T2 is a subtype of T1 (because there's another A on the
;; left)  and that can happen only when T1 and T2 are the same
;; type. So this is a type error:
;; (: foo3 : Integer > (Boxof/RW Integer))
;; (define (foo3 b) (box/rw b))
;; (: bar3 : (Boxof/RW Number) Number > Number)
;; (define (bar3 b new) (b new))
;; (test (bar3 (foo3 123) 456) => 123)
;; ** Expected (Number > Number), but got (Integer > Integer)
;; And this a type error too:
;; (: foo3 : Number > (Boxof/RW Number))
;; (define (foo3 b) (box/rw b))
;; (: bar3 : (Boxof/RW Integer) Integer > Integer)
;; (define (bar3 b new) (b new))
;; (test (bar3 (foo3 123) 456) => 123)
;; ** Expected (Integer > Integer), but got (Number > Number)
;; The two types must be the same for this to work:
(: foo3 : Integer > (Boxof/RW Integer))
(define (foo3 b) (box/rw b))
(: bar3 : (Boxof/RW Integer) Integer > Integer)
(define (bar3 b new) (b new))
(test (bar3 (foo3 123) 456) => 123)

# Implementing a Circular Environment
We now use this to implement `rec` in the following way:
1. Change environments so that instead of values they hold boxes of
values: `(Boxof VAL)` instead of `VAL`, and whenever `lookup` is
used, the resulting boxed value is unboxed,
2. In the `WRec` case, create the new environment with some temporary
binding for the identifier  any value will do since it should not
be used (when named expressions are always `fun` expressions),
3. Evaluate the expression in the new environment,
4. Change the binding of the identifier (the box) to the result of this
evaluation.
The resulting definition is:
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(let ([newcell (box (NumV 42))])
(let ([newenv (Extend id newcell restenv)])
(let ([value (eval expr newenv)])
(setbox! newcell value)
newenv))))
Racket has another `let` relative for such cases of multiplenested
`let`s  `let*`. This form is a derived form  it is defined as a
shorthand for using nested `let`s. The above is therefore exactly the
same as this code:
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(let* ([newcell (box (NumV 42))]
[newenv (Extend id newcell restenv)]
[value (eval expr newenv)])
(setbox! newcell value)
newenv))
This `let*` form can be read almost as a C/Javaish kind of code:
fun extend_rec(id, expr, rest_env) {
new_cell = new NumV(42);
new_env = Extend(id, new_cell, rest_env);
value = eval(expr, new_env);
*new_cell = value;
return new_env;
}
The code can be simpler if we fold the evaluation into the `setbox!`
(since `value` is used just there), and if use `lookup` to do the
mutation  since this way there is no need to hold onto the box. This
is a bit more expensive, but since the binding is guaranteed to be the
first one in the environment, the addition is just one quick step. The
only binding that we need is the one for the new environment, which we
can do as an internal definition, leaving us with:
(: extendrec : Symbol FLANG ENV > ENV)
(define (extendrec id expr restenv)
(define newenv (Extend id (box (NumV 42)) restenv))
(setbox! (lookup id newenv) (eval expr newenv))
newenv)
A complete rehacked version of FLANG with a `rec` binding follows:
;;; <<>>
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[WRec Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons (or 'with 'rec) more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[(list 'rec (list (symbol: name) named) body)
(WRec name (parsesexpr named) (parsesexpr body))]
[(cons x more)
(error 'parsesexpr "bad `~s' syntax in ~s" x sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype ENV
[EmptyEnv]
[Extend Symbol (Boxof VAL) ENV])
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
(: lookup : Symbol ENV > (Boxof VAL))
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id boxedval restenv)
(if (eq? id name) boxedval (lookup name restenv))]))
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(define newenv (Extend id (box (NumV 42)) restenv))
(setbox! (lookup id newenv) (eval expr newenv))
newenv)
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (box (eval namedexpr env)) env))]
[(WRec boundid namedexpr boundbody)
(eval boundbody
(extendrec boundid namedexpr env))]
[(Id name) (unbox (lookup name env))]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (box (eval argexpr env)) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Variable Mutation
> [PLAI §12] and [PLAI §13] (different: adds boxes to the language)
>
> [PLAI §14] (that's what we do)
The code that we now have implements recursion by *changing* bindings,
and to make that possible we made environments hold boxes for all
bindings, therefore bindings are *all* mutable now. We can use this to
add more functionality to our evaluator, by allowing changing any
variable  we can add a `set!` form:
{set! }
to the evaluator that will modify the value of a variable. To implement
this functionality, all we need to do is to use `lookup` to retrieve
some box, then evaluate the expression and put the result in that box.
The actual implementation is left as a homework exercise.
One thing that should be considered here is  all of the expressions
in our language evaluate to some value, the question is what should be
the value of a `set!` expression? There are three obvious choices:
1. return some bogus value,
2. return the value that was assigned,
3. return the value that was previously in the box.
Each one of these has its own advantage  for example, C uses the
second option to `chain` assignments (eg, `x = y = 0`) and to allow side
effects where an expression is expected (eg, `while (x = x1) ...`).
The third one is useful in cases where you might use the old value that
is overwritten  for example, if C had this behavior, you could `pop`
a value from a linked list using something like:
first(stack = rest(stack));
because the argument to `first` will be the old value of `stack`, before
it changed to be its `rest`. You could also swap two variables in a
single expression: `x = y = x`.
(Note that the expression `x = x + 1` has the meaning of C's `++x` when
option (2) is used, and `x++` when option (3) is used.)
Racket chooses the first option, and we will do the same in our
language. The advantage here is that you get no discounts, therefore
you must be explicit about what values you want to return in situations
where there is no obvious choice. This leads to more robust programs
since you do not get other programmers that will rely on a feature of
your code that you did not plan on.
In any case, the modification that introduces mutation is small, but it
has a tremendous effect on our language: it was true for Racket, and it
is true for FLANG. We have seen how mutation affects the language
subset that we use, and in the extension of our FLANG the effect is even
stronger: since *any* variable can change (no need for explicit
`box`es). In other words, a binding is not always the same  in can
change as a result of a `set!` expression. Of course, we could extend
our language with boxes (using Racket boxes to implement FLANG boxes),
but that will be a little more verbose.
> Note that Racket does have a `set!` form, and in addition, fields in
> structs can be made modifiable. However, we do not use any of these.
> At least not for now.

# State and Environments
A quick example of how mutation can be used:
(define counter
(let ([counter (box 0)])
(lambda ()
(setbox! counter (+ 1 (unbox counter)))
(unbox counter))))
and compare that to:
(define (makecounter)
(let ([counter (box 0)])
(lambda ()
(setbox! counter (+ 1 (unbox counter)))
(unbox counter))))
It is a good idea if you follow the exact evaluation of
(define foo (makecounter))
(define bar (makecounter))
and see how both bindings have separate environment so each one gets its
own private state. The equivalent code in the homework interpreter
extended with `set!` doesn't need boxes:
{with {makecounter
{fun {}
{with {counter 0}
{fun {}
{set! counter {+ counter 1}}
counter}}}}
{with {foo {call makecounter}}
{with {bar {call makecounter}}
...}}}
To see multiple values from a single expression you can extend the
language with a `list` binding. As a temporary hack, we can use dummy
function inputs to cover for our lack of nullary functions, and use
`with` (with dummy bound ids) to sequence multiple expressions:
{with {makecounter
{fun {init}
{with {counter init}
{fun {_}
{with {_ {set! counter {+ counter 1}}}
counter}}}}}
{with {foo {call makecounter 0}}
{with {bar {call makecounter 1}}
{+ {+ {call foo 0} {+ {* 10 {call foo 0}}
{* 100 {call foo 0}}}}
{* 10000 {+ {call bar 0} {+ {* 10 {call bar 0}}
{* 100 {call bar 0}}}}}}}}}
Note that we cannot describe this behavior with substitution rules! We
now use the environments to make it possible to change bindings  so
finally an environment is actually an environment rather than a
substitution cache.
When you look at the above, note that we still use lexical scope  in
fact, the local binding is actually a private state that nobody can
access. For example, if we write this:
(define counter
(let ([counter (box 0)])
(lambda ()
(setbox! counter (+ 1 (unbox counter)))
(if (zero? (modulo (unbox counter) 4)) 'tock 'tick))))
then the resulting function that us bound to `counter` keeps a local
integer state which no other code can access  you cannot modify it,
reset it, or even know if it is really an integer that is used in there.

# Implementing Objects with State
We have already seen how several pieces of information can be
encapsulate in a Racket closure that keeps them all; now we can do a
little more  we can actually have mutable state, which leads to a
natural way to implement objects. For example:
(define (makepoint x y)
(let ([xb (box x)]
[yb (box y)])
(lambda (msg)
(match msg
['getx (unbox xb)]
['gety (unbox yb)]
['incx (setbox! xb (add1 (unbox xb)))]))))
implements a constructor for `point` objects which keep two values and
can move one of them. Note that the messages act as a form of methods,
and that the values themselves are hidden and are accessible only
through the interface that these messages make. For example, if these
points correspond to some graphic object on the screen, we can easily
incorporate a necessary screen update:
(define (makepoint x y)
(let ([xb (box x)]
[yb (box y)])
(lambda (msg)
(match msg
['getx (unbox xb)]
['gety (unbox yb)]
['incx (setbox! xb (add1 (unbox xb)))
(updatescreen)]))))
and be sure that this is always done when the value changes  since
there is no way to change the value except through this interface.
A more complete example would define functions that actually send these
messages  here is a better implementation of a point object and the
corresponding accessors and mutators:
(define (makepoint x y)
(let ([xb (box x)]
[yb (box y)])
(lambda (msg)
(match msg
['getx (unbox xb)]
['gety (unbox yb)]
[(list 'setx newx)
(setbox! xb newx)
(updatescreen)]
[(list 'sety newy)
(setbox! yb newy)
(updatescreen)]))))
(define (pointx p) (p 'getx))
(define (pointy p) (p 'gety))
(define (setpointx! p x) (p (list 'setx x)))
(define (setpointy! p y) (p (list 'sety y)))
And a quick imitation of inheritance can be achieved using delegation to
an instance of the superclass:
(define (makecoloredpoint x y color)
(let ([p (makepoint x y)])
(lambda (msg)
(match msg
['getcolor color]
[else (p msg)]))))
You can see how all of these could come from some preprocessing of a
more normallooking class definition form, like:
(defclass point (x y)
(public (getx) x)
(public (gety) y)
(public (setx new) (set! x newx))
(public (setx new) (set! x newx)))
(defclass coloredpoint point (c)
(public (getcolor) c))

# The Toy Language
> Not in PLAI
A quick note: from now on, we will work with a variation of our language
 it will change the syntax to look a little more like Racket, and we
will use Racket values for values in our language and Racket functions
for builtins in our language.
Main highlights:
* There can be multiple bindings in function arguments and local `bind`
forms  the names are required to be distinct.
* There are now a few keywords like `bind` that are parsed in a special
way. Other forms are taken as function application, which means that
there are no special parse rules (and AST entries) for arithmetic
functions. They're now bindings in the global environment, and
treated in the same way as all bindings. For example, `*` is an
expression that evaluates to the *primitive* multiplication function,
and `{bind {{+ *}} {+ 2 3}}` evaluates to `6`.
* Since function applications are now the same for primitive functions
and userbound functions, there is no need for a `call` keyword. Note
that the function call part of the parser must be last, since it
should apply only if the input is not some other known form.
* Note the use of `makeuntypedlistfunction`: it's a library function
(included in the course language) that can convert a few known Racket
functions to a function that consumes a list of *any* Racket values,
and returns the result of applying the given Racket function on these
values. For example:
(define add (makeuntypedlistfunction +))
(add (list 1 2 3 4))
evaluates to `10`.
* Another important aspect of this is its type  the type of `add` in
the previous example is `(List > Any)`, so the resulting function can
consume *any* input values. If it gets a bad value, it will throw an
appropriate error. This is a hack: it basically means that the
resulting `add` function has a very generic type (requiring just a
list), so errors can be thrown at runtime. However, in this case, a
better solution is not going to make these runtime errors go away
because the language that we're implementing is not statically typed.
* The benefit of this is that we can avoid the hassle of more verbose
code by letting these functions dynamically check the input values, so
we can use a single `RktV` variant in `VAL` which wraps any Racket
value. (Otherwise we'd need different wrappers for different types,
and implement these dynamic checks.)
The following is the complete implementation.
;;; <<>>
#lang pl
;;; 
;;; Syntax
# The BNF:
::=

 { bind {{ } ... } }
 { fun { ... } }
 { if }
 { ... }
#
;; A matching abstract syntax tree datatype:
(definetype TOY
[Num Number]
[Id Symbol]
[Bind (Listof Symbol) (Listof TOY) TOY]
[Fun (Listof Symbol) TOY]
[Call TOY (Listof TOY)]
[If TOY TOY TOY])
(: uniquelist? : (Listof Any) > Boolean)
;; Tests whether a list is unique, guards Bind and Fun values.
(define (uniquelist? xs)
(or (null? xs)
(and (not (member (first xs) (rest xs)))
(uniquelist? (rest xs)))))
(: parsesexpr : Sexpr > TOY)
;; parses sexpressions into TOYs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'bind more)
(match sexpr
[(list 'bind (list (list (symbol: names) (sexpr: nameds))
...)
body)
(if (uniquelist? names)
(Bind names (map parsesexpr nameds) (parsesexpr body))
(error 'parsesexpr "duplicate `bind' names: ~s" names))]
[else (error 'parsesexpr "bad `bind' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: names) ...) body)
(if (uniquelist? names)
(Fun names (parsesexpr body))
(error 'parsesexpr "duplicate `fun' names: ~s" names))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(cons 'if more)
(match sexpr
[(list 'if cond then else)
(If (parsesexpr cond)
(parsesexpr then)
(parsesexpr else))]
[else (error 'parsesexpr "bad `if' syntax in ~s" sexpr)])]
[(list fun args ...) ; other lists are applications
(Call (parsesexpr fun)
(map parsesexpr args))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > TOY)
;; Parses a string containing an TOY expression to a TOY AST.
(define (parse str)
(parsesexpr (string>sexpr str)))
;;; 
;;; Values and environments
(definetype ENV
[EmptyEnv]
[FrameEnv FRAME ENV])
;; a frame is an association list of names and values.
(definetype FRAME = (Listof (List Symbol VAL)))
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) TOY ENV]
[PrimV ((Listof VAL) > VAL)])
(: extend : (Listof Symbol) (Listof VAL) ENV > ENV)
;; extends an environment with a new frame.
(define (extend names values env)
(if (= (length names) (length values))
(FrameEnv (map (lambda ([name : Symbol] [val : VAL])
(list name val))
names values)
env)
(error 'extend "arity mismatch for names: ~s" names)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, frame by frame,
;; return its value or throw an error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(FrameEnv frame rest)
(let ([cell (assq name frame)])
(if cell
(second cell)
(lookup name rest)))]))
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': unwrap a RktV wrapper in
;; preparation to be sent to the primitive function
(define (unwraprktv x)
(cases x
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" x)]))
(: racketfunc>primval : Function > VAL)
;; converts a racket function to a primitive evaluator function
;; which is a PrimV holding a ((Listof VAL) > VAL) function.
;; (the resulting function will use the list function as is,
;; and it is the list function's responsibility to throw an error
;; if it's given a bad number of arguments or bad input types.)
(define (racketfunc>primval racketfunc)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(RktV (listfunc (map unwraprktv args))))))
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval +))
(list ' (racketfunc>primval ))
(list '* (racketfunc>primval *))
(list '/ (racketfunc>primval /))
(list '< (racketfunc>primval <))
(list '> (racketfunc>primval >))
(list '= (racketfunc>primval =))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f)))
(EmptyEnv)))
;;; 
;;; Evaluation
(: eval : TOY ENV > VAL)
;; evaluates TOY expressions
(define (eval expr env)
;; convenient helper
(: eval* : TOY > VAL)
(define (eval* expr) (eval expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (eval* funexpr)]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (eval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let ([result (eval (parse str) globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
;;; 
;;; Tests
(test (run "{{fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}} {add3 1}}")
=> 4)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}
{add1 {fun {x} {+ x 1}}}}
{bind {{x 3}} {add1 {add3 x}}}}")
=> 7)
(test (run "{bind {{identity {fun {x} x}}
{foo {fun {x} {+ x 1}}}}
{{identity foo} 123}}")
=> 124)
(test (run "{bind {{x 3}}
{bind {{f {fun {y} {+ x y}}}}
{bind {{x 5}}
{f 4}}}}")
=> 7)
(test (run "{{{fun {x} {x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)
;; More tests for complete coverage
(test (run "{bind x 5 x}") =error> "bad `bind' syntax")
(test (run "{fun x x}") =error> "bad `fun' syntax")
(test (run "{if x}") =error> "bad `if' syntax")
(test (run "{}") =error> "bad syntax")
(test (run "{bind {{x 5} {x 5}} x}") =error> "duplicate*bind*names")
(test (run "{fun {x x} x}") =error> "duplicate*fun*names")
(test (run "{+ x 1}") =error> "no binding for")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{1 2}") =error> "with a nonfunction")
(test (run "{{fun {x} x}}") =error> "arity mismatch")
(test (run "{if {< 4 5} 6 7}") => 6)
(test (run "{if {< 5 4} 6 7}") => 7)
(test (run "{if + 6 7}") => 6)
(test (run "{fun {x} x}") =error> "returned a bad value")
;;; 

# Compilation and Partial Evaluation
Instead of interpreting an expression, which is performing a full
evaluation, we can think about *compiling* it: translating it to a
different language which we can later run more easily, more efficiently,
on more platforms, etc. Another feature that is usually associated with
compilation is that a lot more work was done at the compilation stage,
making the actual running of the code faster.
For example, translating an AST into one that has deBruijn indexes
instead of identifier names is a form of compilation  not only is it
translating one language into another, it does the work involved in name
lookup before the program starts running.
This is something that we can experiment with now. An easy way to
achieve this is to start with our evaluation function:
(: eval : TOY ENV > VAL)
;; evaluates TOY expressions
(define (eval expr env)
;; convenient helper
(: eval* : TOY > VAL)
(define (eval* expr) (eval expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (eval* funexpr)]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (eval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
and change it so it compiles a given expression to a Racket function.
(This is, of course, just to demonstrate a conceptual point, it is only
the tip of what compilers actually do...) This means that we need to
turn it into a function that receives a TOY expression and compiles it.
In other words, `eval` no longer consumes and environment argument which
makes sense because the environment is a place to hold runtime values,
so it is a data structure that is not part of the compiler (it is
usually represented as the call stack).
So we split the two arguments into a compiletime and runtime, which
can be done by simply currying the `eval` function  here this is
done, and all calls to `eval` are also curried:
(: eval : TOY > ENV > VAL) ;*** note the curried type
;; evaluates TOY expressions
(define (eval expr)
(lambda (env)
;; convenient helper
(: eval* : TOY > VAL)
(define (eval* expr) ((eval expr) env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
((eval boundbody) (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (eval* funexpr)]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
((eval body) (extend names argvals funenv))]
[else (error 'eval
"function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (eval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))])))
We also need to change the `eval` call in the main `run` function:
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let ([result ((eval (parse str)) globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
Not much has changed so far.
Note that in the general case of a compiler we need to run a program
several times, so we'd want to avoid parsing it over and over again. We
can do that by keeping a single parsed AST of the input. Now we went
one step further by making it possible to do more work ahead and keep
the result of the first "stage" of eval around (except that "more work"
is really not saying much at the moment):
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let* ([compiled (eval (parse str))]
[result (compiled globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
At this point, even though our "compiler" is not much more than a
slightly different representation of the same functionality, we rename
`eval` to `compile` which is a more appropriate description of what we
intend it to do (so we change the purpose statement too):
(: compile : TOY > ENV > VAL)
;; compiles TOY expressions to Racket functions.
(define (compile expr)
(lambda (env)
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
((compile boundbody)
(extend names (map compile* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (compile* funexpr)]
[argvals (map compile* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
((compile body) (extend names argvals funenv))]
[else (error 'call ; this is *not* a compilation error
"function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(compile* (if (cases (compile* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))])))
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let* ([compiled (compile (parse str))]
[result (compiled globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))

Not much changed, still. We curried the `eval` function and renamed it
to `compile`. But when we actually call compile almost nothing happens
 all it does is create a Racket closure which will do the rest of the
work. (And this closure closes over the given expression.)
Running this "compiled" code is going to be very much like the previous
usage of `eval`, except a little *slower*, because now every recursive
call involves calling `compile` to generate a closure, which is then
immediately used  so we just added some allocations at the recursive
call points! (Actually, the extra cost is minimal because the Racket
compiler will optimize away such immediate closure applications.)
Another way to see how this is not really a compiler yet is to consider
*when* `compile` gets called. A proper compiler is something that does
all of its work *before* running the code, which means that once it
spits out the compiled code it shouldn't be used again (except for
compiling other code, of course). Our current code is not really a
compiler since it breaks this feature. (For example, if GCC behaved
this way, then it would "compile" files by producing code that invokes
GCC to compile the next step, which, when run, invokes GCC again, etc.)
However, the conceptual change is substantial  we now have a function
that does its work in two stages  the first part gets an expression
and *can* do some compiletime work, and the second part does the
runtime work, and includes anything inside the (lambda (env) ...). The
thing is that so far, the code does nothing at the compilation stage
(remember: only creates a closure). But because we have two stages, we
can now shift work from the second stage (the runtime) to the first
(the compiletime).
For example, consider the following simple example:
#lang pl
(: foo : Number Number > Number)
(define (foo x y)
(* x y))
(: bar : Number > Number)
(define (bar c)
(: loop : Number Number > Number)
(define (loop n acc)
(if (< 0 n)
(loop ( n 1) (+ (foo c n) acc))
acc))
(loop 40000000 0))
(time (bar 0))
We can do the same thing here  separate `foo` it into two stages
using currying, and modify `bar` appropriately:
#lang pl
(: foo : Number > Number > Number)
(define (foo x)
(lambda (y)
(* x y)))
(: bar : Number > Number)
(define (bar c)
(: loop : Number Number > Number)
(define (loop n acc)
(if (< 0 n)
(loop ( n 1) (+ ((foo c) n) acc))
acc))
(loop 40000000 0))
(time (bar 0))
Now instead of a simple multiplication, lets expand it a little, for
example, do a case split on common cases where `x` is `0`, `1`, or `2`:
(: foo : Number > Number > Number)
(define (foo x)
(lambda (y)
(cond [(= x 0) 0]
[(= x 1) y]
[(= x 2) (+ y y)] ; assume that this is faster
[else (* x y)])))
This is not much faster, since Racket already optimizes multiplication
in a similar way.
Now comes the real magic: deciding what branch of the cond to take
depends *only* on x, so we can `push` the lambda inside:
(: foo : Number > Number > Number)
(define (foo x)
(cond [(= x 0) (lambda (y) 0)]
[(= x 1) (lambda (y) y)]
[(= x 2) (lambda (y) (+ y y))]
[else (lambda (y) (* x y))]))
We just made an improvement  the comparisons for the common cases are
now done as soon as (foo x) is called, they're not delayed to when the
resulting function is used. Now go back to the way this is used in
`bar` and make it call `foo` once for the given `c`:
#lang pl
(: foo : Number > Number > Number)
(define (foo x)
(cond [(= x 0) (lambda (y) 0)]
[(= x 1) (lambda (y) y)]
[(= x 2) (lambda (y) (+ y y))]
[else (lambda (y) (* x y))]))
(: bar : Number > Number)
(define (bar c)
(define fooc (foo c))
(: loop : Number Number > Number)
(define (loop n acc)
(if (< 0 n)
(loop ( n 1) (+ (fooc n) acc))
acc))
(loop 40000000 0))
(time (bar 0))
Now fooc is generated once, and if `c` happens to be one of the three
common cases (as in the last expression), we can avoid doing any
multiplication. (And if we hit the default case, then we're doing the
same thing we did before.)
[However, the result runs a little slower! The reason is that dealing
with functions can have a higher cost when the compiler cannot "simplify
closures away"  and this is what happens in the last version. The
additional overhead is much higher than the multiplication we save (the
Racket compiler inlines multiplications, so their cost is close to just
executing a single machinecode instruction).]
Here is another useful example that demonstrates this:
(define (foo list)
(map (lambda (n) (if ...something... E1 E2))
list))
>
(define (foo list)
(map (if ...something...
(lambda (n) E1)
(lambda (n) E2))
list))
(Question: when can you do that?)
This is not unique to Racket, it can happen in any language. Racket (or
any language with first class function values) only makes it easy to
create a local function that is specialized for the flag.

Getting our thing closer to a compiler is done in a similar way  we
push the `(lambda (env) ...)` inside the various cases. (Note that
`compile*` depends on the `env` argument, so it also needs to move
inside  this is done for all cases that use it, and will eventually
go away.) We actually need to use `(lambda ([env : ENV]) ...)` though,
to avoid upsetting the type checker:
(: compile : TOY > ENV > VAL)
;; compiles TOY expressions to Racket functions.
(define (compile expr)
(cases expr
[(Num n) (lambda ([env : ENV]) (RktV n))]
[(Id name) (lambda ([env : ENV]) (lookup name env))]
[(Bind names exprs boundbody)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
((compile boundbody)
(extend names (map compile* exprs) env)))]
[(Fun names boundbody)
(lambda ([env : ENV]) (FunV names boundbody env))]
[(Call funexpr argexprs)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
(let ([fval (compile* funexpr)]
[argvals (map compile* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
((compile body) (extend names argvals funenv))]
[else (error 'call ; this is *not* a compilation error
"function call with a nonfunction: ~s"
fval)])))]
[(If condexpr thenexpr elseexpr)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
(compile* (if (cases (compile* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr)))]))
and with this we shifted a bit of actual work to compile time  the
code that checks what structure we have, and extracts its different
slots. But this is still not good enough  it's only the first
toplevel `cases` that is moved to compiletime  recursive calls to
`compile` are still there in the resulting closures. This can be seen
by the fact that we have those calls to `compile` in the Racket closures
that are the results of our compiler, which, as discussed above, mean
that it's not an actual compiler yet.
For example, consider the `Bind` case:
[(Bind names exprs boundbody)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
((compile boundbody)
(extend names (map compile* exprs) env)))]
At compiletime we identify and deconstruct the Bind structure, then
create a the runtime closure that will access these parts when the code
runs. But this closure will itself call `compile` on `boundbody` and
each of the expressions. Both of these calls can be done at compile
time, since they only need the expressions  they don't depend on the
environment. Note that `compile*` turns to `run` here, since all it
does is run a compiled expression on the current environment.
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(lambda ([env : ENV])
(: run : (ENV > VAL) > VAL)
(define (run compiledexpr) (compiledexpr env))
(compiledbody (extend names
(map run compiledexprs)
env))))]
We can move it back up, out of the resulting functions, by making it a
function that consumes an environment and returns a "caller" function:
(define (compile expr)
;; convenient helper
(: caller : ENV > (ENV > VAL) > VAL)
(define (caller env)
(lambda (compiled) (compiled env)))
(cases expr
...
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(lambda ([env : ENV])
(compiledbody (extend names
(map (caller env) compiledexprs)
env))))]
...))
Once this is done, we have a bunch of work that can happen at compile
time: we prescan the main "bind spine" of the code.
We can deal in a similar way with other occurrences of `compile` calls
in compiled code. The two branches that need to be fixed are:
1. In the `If` branch, there is not much to do. After we make it
precompile the `condexpr`, we also need to make it precompile both
the `thenexpr` and the `elseexpr`. This might seem like doing more
work (since before changing it only one would get compiled), but
since this is compiletime work, then it's not as important. Also,
`if` expressions are evaluated many times (being part of a loop, for
example), so overall we still win.
2. The `Call` branch is a little trickier: the problem here is that the
expressions that are compiled are coming from the closure that is
being applied. The solution for this is obvious: we need to change
the closure type so that it closes over *compiled* expressions
instead of over plain ones. This makes sense because closures are
runtime values, so they need to close over the compiled expressions
since this is what we use as "code" at runtime.
Again, the goal is to have no `compile` calls that happen at runtime:
they should all happen before that. This would allow, for example, to
obliterate the compiler once it has done its work, similar to how you
don't need GCC to run a C application. Yet another way to look at this
is that we shouldn't look at the AST at runtime  again, the analogy
to GCC is that the AST is a data structure that the compiler uses, and
it does not exist at runtime. Any runtime reference to the TOY AST is,
therefore, as bad as any runtime reference to `compile`.
When we're done with this process we'll have something that is an actual
compiler: translating TOY programs into Racket closures. To see how
this is an actual compiler consider the fact that Racket uses a JIT to
translate bytecode into machine code when it's running functions. This
means that the compiled version of our TOY programs are, in fact,
translated all the way down to machine code.
Yet another way to see this is to change the compiler code so instead of
producing a Racket closure it spits out the Racket code that makes up
these closures when evaluated. For example, change
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(lambda ([env : ENV])
(compiledbody (extend ...))))]
into
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(stringappend
"(lambda ([env : ENV]) ("
compiledbody
" (extend ...)))"))]
so we get a string that is a Racket program. But since we're using a
Lisp dialect, it's generally better to use Sexpressions instead:
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
`(lambda ([env : ENV])
(,compiledbody (extend ...))))]
(Later in the course we'll talk about these "`` ` ``"s and "`,`"s. For
now, it's enough to know that "`` ` ``" is kind of like a quote, and
"`,`" is an unquote.)

# Lazy Evaluation: Using a Lazy Racket
> [PLAI §7] (done with Haskell)
For this part, we will use a new language, Lazy Racket.
#lang pl lazy
As the name suggests, this is a version of the normal (untyped) Racket
language that is lazy.
First of all, let's verify that this is indeed a lazy language:
> (define (foo x) 3)
> (foo (+ 1 "2"))
3
That went without a problem  the argument expression was indeed not
evaluated. In this language, you can treat all expressions as future
`promises` to evaluate. There are certain points where such promises
are actually `forced`, all of these stem from some need to print a
resulting value, in our case, it's the REPL that prints such values:
> (+ 1 "2")
+: expects type as 2nd argument,
given: "2"; other arguments were: 1
The expression by itself only generates a promise, but when we want to
print it, this promise is forced to evaluate  this forces the
addition, which forces its arguments (plain values rather than
computation promises), and at this stage we get an error. (If we never
want to see any results, then the language will never do anything at
all.) So a promise is forced either when a value printout is needed, or
if it is needed to recursively compute a value to print:
> (* 1 (+ 2 "3"))
+: expects type as 2nd argument,
given: "3"; other arguments were: 2
Note that the error was raised by the internal expression: the outer
expression uses `*`, and `+` requires actual values not promises.
Another example, which is now obvious, is that we can now define an `if`
function:
> (define (myif x y z) (if x y z))
> (myif (< 1 2) 3 (+ 4 "5"))
3
Actually, in this language `if`, `and`, and `or` are all function values
instead of special forms:
> (list if and or)
(# # #)
> ((third (list if and or)) #t (+ 1 "two"))
#t
(By now, you should know that these have no value in Racket  using
them like this in plain will lead to syntax errors.) There are some
primitives that do not force their arguments. Constructors fall in this
category, for example `cons` and `list`:
> (define (fib n) (if (<= n 1) n (+ (fib ( n 1)) (fib ( n 2)))))
> (define a (list (+ 1 2) (+ 3 "4") (fib 30) (* 5 6)))
Nothing  the definition simply worked, but that's expected, since
nothing is printed. If we try to inspect this value, we can get some of
its parts, provided we do not force the bogus one:
> (first a)
3
> (fourth a)
30
> (third a)
196418
> (second a)
+: contract violation, expected: number?, given: "4" ...
The same holds for cons:
> (second (cons 1 (cons 2 (first null))))
2
Now if this is the case, then how about this:
> (define ones (cons 1 ones))
Everything is fine, as expected  but what is the value of `ones` now?
Clearly, it is a list that has 1 as its first element:
> (first ones)
1
But what do we have in the tail of this list? We have `ones` which we
already know is a list that has 1 in its first place  so following
Racket's usual rules, it means that the second element of `ones` is,
again, 1. If we continue this, we can see that `ones` is, in fact, an
*infinite* list of 1s:
> (second ones)
1
> (fifth ones)
1
In this sense, the way `define` behaves is that it defines a true
equation: if ones is defined as (cons 1 ones), then the real value does
satisfy
(equal? ones (cons 1 ones))
which means that the value is the fixpoint of the defined expression.
We can use `append` in a similar way:
> (define foo (append (list 1 2 3) foo))
> (fourth foo)
1
This looks like it has some common theme with the discussion of
implementing recursive environments  it actually demonstrates that in
this language, `letrec` can be used for *simple* values too. First of
all, a side note  here an expression that indicated a bug in our
substituting evaluator:
> (let ([x (list y)])
(let ([y 1])
x))
reference to undefined identifier: y
When our evaluator returned `1` for this, we noticed that this was a
bug: it does not obey the lexical scoping rules. As seen above, Lazy
Racket is correctly using lexical scope. Now we can go back to the use
of `letrec`  what do we get by this definition:
> (define twos (let ([xs (cons 2 xs)]) xs))
we get an error about `xs` being undefined.
`xs` is unbound because of the usual scope that `let` uses. How can we
make this work?  We simply use `letrec`:
> (define twos (letrec ([xs (cons 2 xs)]) xs))
> (first twos)
2
As expected, if we try to print an infinite list will cause an infinite
loop, which DrRacket catches and prints in that weird way:
> twos
#0=(2 . #0#)
How would we inspect an infinite list? We write a function that returns
part of it:
> (define (take n l)
(if (or (<= n 0) (null? l))
null
(cons (first l) (take (sub1 n) (rest l)))))
> (take 10 twos)
(2 2 2 2 2 2 2 2 2 2)
> (define foo (append (list 1 2 3) foo))
> (take 10 foo)
(1 2 3 1 2 3 1 2 3 1)
Dealing with infinite lists can lead to lots of interesting things, for
example:
> (define fibs (cons 1 (cons 1 (map + fibs (rest fibs)))))
> (take 10 fibs)
(1 1 2 3 5 8 13 21 34 55)
To see how it works, see what you know about `fibs[n]` which will be our
notation for the nth element of `fibs` (starting from `1`):
fibs[1] = 1 because of the first `cons'
fibs[2] = 1 because of the second `cons'
and for all `n>2`:
fibs[n] = (map + fibs (rest fibs))[n2]
= fibs[n2] + (rest fibs)[n2]
= fibs[n2] + fibs[n2+1]
= fibs[n2] + fibs[n1]
so it follows the exact definition of Fibonacci numbers.
Note that the list examples demonstrate that laziness applies to nested
values (actually, nested computations) too: a value that is not needed
is not computed, even if it is *contained* in a value that is needed.
For example, in:
(define x (/ 1 0))
(if (list (+ 1 x)) 1 2)
the `if` needs to know only whether its first argument (note: it *is* an
*argument*, since this `if` is a function) is `#f` or not. Once it is
determined that it is a pair (a `cons` cell), there is no need to
actually look at the values inside the pair, and therefore `(+ 1 x)`
(and more specifically, `x`) is never evaluated and we see no error.

# Lazy Evaluation: Some Issues
There are a few issues that we need to be aware of when we're dealing
with a lazy language. First of all, remember that our previous attempt
at lazy evaluation has made
{with {x y}
{with {y 1}
x}}
evaluate to 1, which does not follow the rules of lexical scope. This
is *not* a problem with lazy evaluation, but rather a problem with our
naive implementation. We will shortly see a way to resolve this
problem. In the meanwhile, remember that when we try the same in Lazy
Racket we do get the expected error:
> (let ([x y])
(let ([y 1])
x))
reference to undefined identifier: y
A second issue is a subtle point that you might have noticed when we
played with Lazy Racket: for some of the list values we have see a "`.`"
printed. This is part of the usual way Racket displays an *improper
list*  any list that does not terminate with a null value. For
example, in plain Racket:
> (cons 1 2)
(1 . 2)
> (cons 1 (cons 2 (cons 3 4)))
(1 2 3 . 4)
In the dialect that we're using in this course, this is not possible.
The secret is that the `cons` that we use first checks that its second
argument is a proper list, and it will raise an error if not. So how
come Lazy Racket's `cons` is not doing the same thing? The problem is
that to know that something is a proper list, we will have to force it,
which will make it not behave like a constructor.
> As a side note, we can achieve some of this protection if we don't
> insist on immediately checking the second argument completely, and
> instead we do the check when needed  lazily:
>
> (define (safecons x l)
> (cons x (if (pair? l) l (error "poof"))))
Finally, there are two consequences of using a lazy language that make
it more difficult to debug (or at lease take some time to get used to).
First of all, control tends to flow in surprising ways. For example,
enter the following into DrRacket, and run it in the normal language
level for the course:
(define (foo3 x)
(/ x "1"))
(define (foo2 x)
(foo3 x))
(define (foo1 x)
(list (foo2 x)))
(define (foo0 x)
(first (foo1 x)))
(+ 1 (foo0 3))
In the normal language level, we get an error, and red arrows that show
us how where in the computation the error was raised. The arrows are
all expected, except that `foo2` is not in the path  why is that?
Remember the discussion about tailcalls and how they are important in
Racket since they are the only tool to generate loops? This is what
we're seeing here: `foo2` calls `foo3` in a tail position, so there is
no need to keep the `foo2` context anymore  it is simply replaced by
`foo3`. (Incidentally, there is also no arrow that goes through `foo1`:
Racket does some smart inlining, and it figures out that `foo0`+`foo1`
are simply returning the same value, so it skips `foo1`.)
Now switch to Lazy Racket and rerun  you'll see no arrows at all.
What's the problem? The call of `foo0` creates a promise that is forced
in the toplevel expression, that simply returns the `first` of the
`list` that `foo1` created  and all of that can be done without
forcing the `foo2` call. Going this way, the computation is finally
running into an error *after* the calls to `foo0`, `foo1`, and `foo2`
are done  so we get the seemingly outofcontext error.
To follow what's happening here, we need to follow how promise are
forced: when we have code like
> (define (foo x) (/ x 0))
> (foo 9)
then the `foo` call is a *strict point*, since we need an actual value
to display on the REPL. Since it's in a strict position, we do the
call, but when we're in the function there is no need to compute the
division result  so it is returned as a lazy promise value back to
the toplevel. It is only then that we continue the process of getting
an actual value, which leads to trying to compute the division and get
the error.
Finally, there are also potential problems when you're not careful about
memory use. A common technique when using a lazy language is to
generate an infinite list and pull out its Nth element. For example, to
compute the Nth Fibonacci number, we've seen how we can do this:
(define fibs (cons 1 (cons 1 (map + fibs (rest fibs)))))
(define (fib n) (listref fibs n))
and we can also do this (reminder: `letrec` is the same as an internal
definition):
(define (fib n)
(letrec ([fibs (cons 1 (cons 1 (map + fibs (rest fibs))))])
(listref fibs n))) ; tailcall => no need to keep `fibs`
but the problem here is that when `listref` is making its way down the
list, it might still hold a reference to `fibs`, which means that as the
list is forced, all intermediate values are held in memory. In the
first of these two, this is guaranteed to happen since we have a binding
that points at the head of the `fibs` list. With the second form things
can be confusing: it might be that our language implementation is smart
enough to see that `fibs` is not really needed anymore and release the
offending reference. If it isn't, then we'd have to do something like
(define (fib n)
(listref
(letrec ([fibs (cons 1 (cons 1 (map + fibs (rest fibs))))])
fibs)
n))
to eliminate it. But even if the implementation does know that there is
no need for that reference, there are other tricky situations that are
hard to avoid.
Side note: Racket didn't use to do this optimization, but now it does,
and the lazy language helped in clarifying more cases where references
should be released. To see that, consider these two variants:
(define (nat1 n)
(define nats (cons 1 (map add1 nats)))
(if (number? (listref nats n))
"a number"
"not a number"))
;; we want to provide some information: show the first element
(define (nat2 n)
(define nats (cons 1 (map add1 nats)))
(if (number? (listref nats n))
"a number"
(error 'nat "the list starting with ~s is broken"
(first nats))))
If we try to use them with a big input:
(nat1 300000) ; or with nat2
then `nat1` would work fine, whereas `nat2` will likely run into
DrRacket's memory limit and the computation will be terminated. The
problem is that `nat2` uses the `nats` value *after* the `listref`
call, which will make a reference to the head of the list, preventing it
from being garbagecollected while `listref` is `cdr`ing down the list
and making more cons cells materialize.
It's still possible to show the extra information though  just save it:
;; we want to provide some information: show the first element
(define (nat3 n)
(define nats (cons 1 (map add1 nats)))
(define fst (first nats))
(if (number? (listref nats n))
"a number"
(error 'nat "the list starting with ~s is broken" fst)))
It looks like it's spending a redundant runtime cycle in the extra
computation, but it's a lazy language so this is not a problem.

# Lazy Evaluation: Shell Examples
There is a very simple and elegant principle in shell programming  we
get a single data type, a character stream, and many small functions,
each doing a single simple job. With these small building blocks, we
can construct more sequences that achieve more complex tasks, for
example  a sorted frequency table of lines in a file:
sort foo  uniq c  sort nr
This is very much like a programming language  we get small blocks,
and build stuff out of them. Of course there are swiss army knives like
awk that try to do a whole bunch of stuff, (the same attitude that
brought Perl to the world...) and even these respect the "stream" data
type. For example, a simple `{ print $1 }` statement will work over all
lines, one by one, making it a program over an infinite input stream,
which is what happens in reality in something like:
cat /dev/console  awk ...
But there is something else in shell programming that makes so
effective: it is implementing a sort of a lazy evaluation. For example,
compare this:
cat foo  awk '{ print $1+$2; }'  uniq
to:
cat foo  awk '{ print $1+$2; }'  uniq  head 5
Each element in the pipe is doing its own small job, and it is always
doing just enough to feed its output. Each basic block is designed to
work even on infinite inputs! (Even sort works on unlimited inputs...)
(Soon we will see a stronger connection with lazy evaluation.)
> Side note: (Alan Perlis) "It is better to have 100 functions operate
> on one data structure than 10 functions on 10 data structures"... But
> the uniformity comes at a price: the biggest problem shells have is in
> their lack of a recursive structure, contaminating the world with way
> too many hacked up solutions. More than that, it is extremely
> inefficient and usually leads to data being reparsed over and over
> and over  each small Unix command needs to always output stuff that
> is human readable, but the next command in the pipe will need to
> reparse that, eg, rereading decimal numbers. If you look at
> pipelines as composing functions, then a pipe of numeric commands
> translates to something like:
>
> itoa(baz(atoi(itoa(bar(atoi(itoa(foo(atoi(inp)))))))))
>
> and it is impossible to get rid of the redundant `atoi(itoa(...))`s.

# Lazy Evaluation: Programming Examples
We already know that when we use lazy evaluation, we are guaranteed to
have more robust programs. For example, a function like:
(define (myif x y z)
(if x y z))
is completely useless in Racket because all functions are eager, but in
a lazy language, it would behave exactly like the real if. Note that we
still need *some* primitive conditional, but this primitive can be a
function (and it is, in Lazy Racket).
But we get more than that. If we have a lazy language, then
*computations* are pushed around as if they were values (computations,
because these are expressions that are yet to be evaluated). In fact,
there is no distinction between computations and values, it just happens
that some values contain "computational promises", things that will do
something in the future.
To see how this happens, we write a simple program to compute the
(infinite) list of prime numbers using the sieve of Eratosthenes. To do
this, we begin by defining the list of all natural numbers:
(define nats (cons 1 (map add1 nats)))
And now define a `sift` function: it receives an integer `n` and an
infinite list of integers `l`, and returns a list without the numbers
that can be divided by `n`. This is simple to write using `filter`:
(define (sift n l)
(filter (lambda (x) (not (divides? n x))) l))
and it requires a definition for `divides?`  we use Racket's `modulo`
for this:
(define (divides? n m)
(zero? (modulo m n)))
Now, a `sieve` is a function that consumes a list that begins with a
prime number, and returns the prime numbers from this list. To do this,
it returns a list that has the same first number, and for its tail it
sifts out numbers that are divisible by the first from the original
list's tail, and calls itself recursively on the result:
(define (sieve l)
(cons (first l) (sieve (sift (first l) (rest l)))))
Finally, the list of prime numbers is the result of applying `sieve` on
the list of numbers from `2`. The whole program is now:
#lang pl lazy
(define nats (cons 1 (map add1 nats)))
(define (divides? n m)
(zero? (modulo m n)))
(define (sift n l)
(filter (lambda (x) (not (divides? n x))) l))
(define (sieve l)
(cons (first l) (sieve (sift (first l) (rest l)))))
(define primes (sieve (rest nats)))
To see how this runs, we trace `modulo` to see which tests are being
used. The effect of this is that each time `divides?` is actually
required to return a value, we will see a line with its inputs, and its
output. This output looks quite tricky  things are computed only on
a "need to know" basis, meaning that debugging lazy programs can be
difficult, since things happen when they are needed which takes time to
get used to. However, note that the program actually performs the same
tests that you'd do using any eagerlanguage implementation of the sieve
of Eratosthenes, and the advantage is that we don't need to decide in
advance how many values we want to compute  all values will be
computed when you want to see the corresponding result. Implementing
*this* behavior in an eager language is more difficult than a simple
program, yet we don't need such complex code when we use lazy
evaluation.
Note that if we trace `divides?` we see results that are some promise
struct  these are unevaluated expressions, and they point at the fact
that when `divides?` is used, it doesn't really force its arguments 
this happens later when these results are forced.
The analogy with shell programming using pipes should be clear now 
for example, we have seen this:
cat foo  awk '{ print $1+$2; }'  uniq  head 5
The last `head 5` means that no computation is done on parts of the
original file that are not needed. It is similar to a `(take 5 l)`
expression in Lazy Racket.

## Side Note: Similarity to Generators and Channels
Using infinite lists is similar to using channels  a tool for
synchronizing threads and (see a [Rob Pike's talk]), and
generators (as they exist in Python). Here are examples of both, note
how similar they both are, and how similar they are to the above
definition of `primes`. (But note that there is an important
difference, can you see it? It has to be with whether a stream is
reusable or not.)
[Rob Pike's talk]: https://www.youtube.com/watch?v=hB05UFqOtFA&t=1680s
First, the threads & channels version:
#lang racket
(definesyntaxrule (bg expr ...) (thread (lambda () expr ...)))
(define nats
(let ([out (makechannel)])
(define (loop i) (channelput out i) (loop (add1 i)))
(bg (loop 1))
out))
(define (divides? n m)
(zero? (modulo m n)))
(define (filter pred c)
(define out (makechannel))
(define (loop)
(let ([x (channelget c)])
(when (pred x) (channelput out x))
(loop)))
(bg (loop))
out)
(define (sift n c)
(filter (lambda (x) (not (divides? n x))) c))
(define (sieve c)
(define out (makechannel))
(define (loop c)
(define first (channelget c))
(channelput out first)
(loop (sift first c)))
(bg (loop c))
out)
(define primes
(begin (channelget nats) (sieve nats)))
(define (take n c)
(if (zero? n) null (cons (channelget c) (take (sub1 n) c))))
(take 10 primes)
And here is the generator version:
#lang racket
(require racket/generator)
(define nats
(generator ()
(define (loop i)
(yield i)
(loop (add1 i)))
(loop 1)))
(define (divides? n m)
(zero? (modulo m n)))
(define (filter pred g)
(generator ()
(define (loop)
(let ([x (g)])
(when (pred x) (yield x))
(loop)))
(loop)))
(define (sift n g)
(filter (lambda (x) (not (divides? n x))) g))
(define (sieve g)
(define (loop g)
(define first (g))
(yield first)
(loop (sift first g)))
(generator () (loop g)))
(define primes
(begin (nats) (sieve nats)))
(define (take n g)
(if (zero? n) null (cons (g) (take (sub1 n) g))))
(take 10 primes)

# Call by Need vs Call by Name
Finally, note that on requiring different parts of the `primes`, the
same calls are not repeated. This indicates that our language
implements "call by need" rather than "call by name": once an expression
is forced, its value is remembered, so subsequent usages of this value
do not require further computations.
Using "call by name" means that we actually use expressions which can
lead to confusing code. An old programming language that used this is
Algol. A confusing example that demonstrates this evaluation strategy
is:
#lang algol60
begin
integer procedure SIGMA(x, i, n);
value n;
integer x, i, n;
begin
integer sum;
sum := 0;
for i := 1 step 1 until n do
sum := sum + x;
SIGMA := sum;
end;
integer q;
printnln(SIGMA(q*21, q, 7));
end
`x` and `i` are arguments that are passed by name, which means that they
can use the same memory location. This is called *aliasing*, a problem
that happens when pointers are involved (eg, pointers in C and
`reference` arguments in C++). The code, BTW, is called "Jensen's
device".

# Example of Feature Embedding
Another interesting behavior that we can now observe, is that the TOY
evaluation rule for `with`:
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
is specifying an eager evaluator *only if* the language that this rule
is written in is itself eager. Indeed, if we run the TOY interpreter in
Lazy Racket (or other interpreters we have implemented), we can verify
that running:
(run "{bind {{x {/ 1 0}}} 1}")
is perfectly fine  the call to Racket's division is done in the
evaluation of the TOY division expression, but since Lazy Racket is
lazy, then if this value is never used then we never get to do this
division! On the other hand, if we evaluate
(run "{bind {{x {/ 1 0}}} {+ x 1}}")
we do get an error when DrRacket tries to display the result, which
forces strictness. Note how the arrows in DrRacket that show where the
computation is are quite confusing: the computation seem to go directly
to the point of the arithmetic operations (`arithop`) since the rest of
the evaluation that the evaluator performed was already done, and
succeeded. The actual failure happens when we try to force the
resulting promise which contains only the strict points in our code.

# Implementing Laziness (in plain Racket)
> [PLAI §8]
Generally, we know how lazy evaluation works when we use the
substitution model. We even know that if we have:
{bind {{x y}}
{bind {{y 2}}
{+ x y}}}
then the result should be an error because we cannot substitute the `y`
expression in because it will capture the `y`  changing the binding
structure. As an indication, the original expression contains a free
reference to `y`, which is exactly why we shouldn't substitute it. But
what about:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}}}
Evaluating this eagerly returns 18, we therefore expect any other
evaluation (eager or lazy, using substitutions or environments) to
return 18 too, because any of these options should not change the
meaning of numbers, of addition, *or* of the scoping rules. (And we
know that no matter what evaluation strategy we choose, if we get to a
value (no infinite loop or exception) then it'll always be the same
value.) For example, try using lazy evaluation with substitutions:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}}}
>
{bind {{y {+ {+ 4 5} {+ 4 5}}}}
{bind {{z y}}
{bind {{x 4}}
z}}}
>
{bind {{z {+ {+ 4 5} {+ 4 5}}}}
{bind {{x 4}}
z}}
>
{bind {{x 4}}
{+ {+ 4 5} {+ 4 5}}}
>
{+ {+ 4 5} {+ 4 5}}
>
{+ 9 9}
>
18
And what about lazy evaluation using environments:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}}} []
>
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}} [x:={+ 4 5}]
>
{bind {{z y}}
{bind {{x 4}}
z}} [x:={+ 4 5}, y:={+ x x}]
>
{bind {{x 4}}
z} [x:={+ 4 5}, y:={+ x x}, z:=y]
>
z [x:=4, y:={+ x x}, z:=y]
>
y [x:=4, y:={+ x x}, z:=y]
>
{+ x x} [x:=4, y:={+ x x}, z:=y]
>
{+ 4 4} [x:=4, y:={+ x x}, z:=y]
>
8 [x:=4, y:={+ x x}, z:=y]
We have a problem! This problem should be familiar now, it is very
similar to the problem that led us down the mistaken path of dynamic
scoping when we tried to have firstclass functions. In both cases,
substitution always worked, and it looks like in both cases the problem
is that we don't remember the environment of an expression: in the case
of functions, it is the environment at the time of creating the closure
that we want to capture and use when we go back later to evaluate the
body of the function. Here we have a similar situation, except that we
don't need a function to defer computation: *most* expressions get
evaluated at some time in the future, so every time we defer such a
computation we need to remember the lexical environment of the
expression.
This is the major point that will make things work again: every
expression creates something like a closure  an object that closes
over an expression and an environment at the (lexical) place where that
expression was used, and when we actually want to evaluate it later, we
need to do it in the right lexical context. So it is like a closure
except it doesn't need to be applied, and there are no arguments. In
fact it is also a form of a closure  instead of closing over a
function body and an environment, it closes over any expression and an
environment. (As we shall see, lazy evaluation is tightly related to
using nullary functions: *thunks*.)

# Sloth: A Lazy Evaluator
So we implement this by creating such closure values for all expressions
that are not evaluated right now. We begin with the Toy language, and
rename it to "Sloth". We then add one more case to the data type of
values which implements the new kind of expression closures, which
contains the expression and its environment:
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV] ;*** new: expression and scope
[PrimV ((Listof VAL) > VAL)])
(Intuition#1: `ExprV` is a delayed evaluation and therefore it has the
two values that are ultimately passed to `eval`. Intuition#2: laziness
can be implemented with thunks, so we hold the same information as a
`FunV` does, only there's no need for the argument names.)
Where should we use the new `ExprV`?  At any place where we want to
be lazy and defer evaluating an expression for later. The two places in
the interpreter where we want to delay evaluation are the named
expressions in a bind form and the argument expressions in a function
application. Both of these cases use the helper `eval*` function to do
their evaluations, for example:
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
To delay these evaluations, we need to change `eval*` so it returns an
expression closure instead of actually doing the evaluation  change:
(: eval* : SLOTH > VAL)
(define (eval* expr) (eval expr env))
to:
(: eval* : SLOTH > VAL)
(define (eval* expr) (ExprV expr env))
Note how simple this change is  instead of an `eval` function call,
we create a value that contains the parts that would have been used in
the `eval` function call. This value serves as a promise to do this
evaluation (the `eval` call) later, if needed. (This is exactly why a
Lazy Racket would make this a lazy evaluator: in it, *all* function
calls are promises.)

Side note: this can be used in any case when you're using an eager
language, and you want to delay some function call  all you need to
do is replace (using a Cish syntax)
int foo(int x, str y) {
...do some work...
}
with
// rename `foo':
int real_foo(int x, str y) {
...same work...
}
// `foo' is a delayed constructor, instead of a plain function
struct delayed_foo {
int x;
str y;
}
delayed_foo foo(int x, str y) {
return new delayed_foo(x, y);
}
now all calls to `foo` return a `delayed_foo` instance instead of an
integer. Whenever we want to force the delayed promise, we can use this
function:
int force_foo(delayed_foo promise) {
return real_foo(promise.x, promise.y);
}
You might even want to make sure that each such promise is evaluated
exactly once  this is simple to achieve by adding a cache field to
the struct:
int real_foo(int x, str y) {
...same work...
}
struct delayed_foo {
int x;
str y;
bool is_computed;
int result;
}
delayed_foo foo(int x, str y) {
return new delayed_foo(x, y, false, 0);
}
int force_foo(delayed_foo promise) {
if (!promise.is_computed) {
promise.result = real_foo(promise.x, promise.y);
promise.is_computed = true;
}
return promise.result;
}
As we will see shortly, this corresponds to switching from a
callbyname lazy language to a callbyneed one.

Back to our Sloth interpreter  given the `eval*` change, we expect
that `eval`uating:
{bind {{x 1}} x}
will return:
(ExprV (Num 1) ...theglobalenvironment...)
and the same goes for `eval`uating
{{fun {x} x} 1}
Similarly, evaluating
{bind {{x {+ 1 2}}} x}
should return
(ExprV (Call (Id +) (Num 1) (Num 2)) ...theglobalenvironment...)
But what about evaluating an expression like this one:
{bind {{x 2}}
{+ x x}}
?
Using what we have so far, we will get to evaluate the body, which is a
(Call ...) expression, but when we evaluate the arguments for this
function call, we will get `ExprV` values  so we will not be able to
perform the addition. Instead, we will get an error from the function
that `racketfunc>primval` creates, due to the value being an `ExprV`
instead of a `RktV`.
What we really want is to actually add two *values*, not promises. So
maybe distinguish the two applications  treat `PrimV` differently
from `FunV` closures?
(: eval* : SLOTH > VAL)
(define (eval* expr) (ExprV expr env))
(: realeval* : SLOTH > VAL)
(define (realeval* expr) (eval expr env))
(cases expr
...
[(Call funexpr argexprs)
(let ([fval (eval funexpr env)]
;; move: [argvals (map eval* argexprs)]
)
(cases fval
[(PrimV proc) (proc (map realeval* argexprs))] ; change
[(FunV names body funenv)
(eval body (extend names (map eval* argexprs) funenv))]
...))]
...)
This still doesn't work  the problem is that the function now gets a
bunch of values, where some of these can still be `ExprV`s because the
evaluation itself can return such values... Another way to see this
problem is to consider the code for evaluating an `If` conditional
expression:
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (realeval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]
...we need to take care of a possible `ExprV` here. What should we do?
The obvious solution is to use `eval` if we get an `ExprV` value:
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (realeval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[(ExprV expr env) (eval expr env)] ; force a promise
[else #t]) ; other values are always true
thenexpr
elseexpr))]
Note how this translates back the data structure that represents a
delayed `eval` promise back into a real `eval` call...
Going back to our code for `Call`, there is a problem with it  the
(define (realeval* expr) (eval expr env))
will indeed evaluate the expression instead of lazily deferring this to
the future, but this evaluation might itself return such lazy values.
So we need to inspect the resulting value again, forcing the promise if
needed:
(define (realeval* expr)
(let ([val (eval expr env)])
(cases val
[(ExprV expr env) (eval expr env)]
[else val])))
But we *still* have a problem  programs can get an arbitrarily long
nested chains of `ExprV`s that get forced to other `ExprV`s.
{bind {{x true}}
{bind {{y x}}
{bind {{z y}}
{if z
{foo}
{bar}}}}}
What we really need is to write a loop that keeps forcing promises over
and over until it gets a proper non`ExprV` value.
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise,
;; returns a VAL that is not an ExprV
(define (strict val)
(cases val
[(ExprV expr env) (strict (eval expr env))] ; loop back
[else val]))
Note that it's close to `realeval*`, but there's no need to mix it with
`eval`. The recursive call is important: we can never be sure that
`eval` didn't return an `ExprV` promise, so we have to keep looping
until we get a "real" value.
Now we can change the evaluation of function calls to something more
manageable:
[(Call funexpr argexprs)
(let ([fval (strict (eval* funexpr))] ;*** strict!
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc (map strict argvals))] ;*** strict!
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
The code is fairly similar to what we had previously  the only
difference is that we wrap a `strict` call where a proper value is
needed  the function value itself, and arguments to primitive
functions.
The `If` case is similar (note that it doesn't matter if `strict` is
used with the result of `eval` or `eval*` (which returns an `ExprV`)):
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (strict (eval* condexpr))
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]
Note that, like before, we always return `#t` for non`RktV` values 
this is because we know that the value there is never an `ExprV`. All
we need now to get a working evaluator, is one more strictness point:
the outermost point that starts our evaluation  `run`  needs to
use `strict` to get a proper result value.
(: run : String > Any)
;; evaluate a SLOTH program contained in a string
(define (run str)
(let ([result (strict (eval (parse str) globalenvironment))])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
With this, all of the tests that we took from the Toy evaluator run
successfully. To make sure that the interpreter is lazy, we can add a
test that will fail if the language is strict:
;; Test laziness
(test (run "{{fun {x} 1} {/ 9 0}}") => 1)
(test (run "{{fun {x} 1} {{fun {x} {x x}} {fun {x} {x x}}}}") => 1)
(test (run "{bind {{x {{fun {x} {x x}} {fun {x} {x x}}}}} 1}") => 1)
[In fact, we can continue and replace all `eval` calls with `ExprV`,
leaving only the one call in `strict`. This doesn't make any
difference, because the resulting promises will eventually be forced by
`strict` anyway.]

## Getting more from Sloth
As we've seen, using `strict` in places where we need an actual value
rather than a delayed promise is enough to get a working lazy
evaluator. Our current implementation assumes that all primitive
functions need strict values, therefore the argument values are all
passed through the `strict` function  but this is not always the
case. Specifically, if we have constructor functions, then we don't
need (and usually don't want) to force the promises. This is basically
what allows us to use infinite lists in Lazy Racket: the fact that
`list` and `cons` do not require forcing their arguments.
To allow some primitive functions to consume strict values and some to
leave them as is, we're going to change `racketfunc>primval` and add
a flag that indicates whether the primitive function is strict or not.
Obviously, we also need to move the `strict` call around arguments to a
primitive function application into the `racketfunc>primval`
generated function  which simplifies the `Call` case in `eval` (we go
from (proc (map strict argvals)) back to (proc argvals)). The new
code for `racketfunc>primval` and its helper is:
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function ...
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict?
(map unwraprktv args)
args)]) ;*** use values as is!
(RktV (listfunc args))))))
We now need to annotate the primitives in the global environment, as
well as add a few constructors:
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval + #t))
(list ' (racketfunc>primval  #t))
(list '* (racketfunc>primval * #t))
(list '/ (racketfunc>primval / #t))
(list '< (racketfunc>primval < #t))
(list '> (racketfunc>primval > #t))
(list '= (racketfunc>primval = #t))
;; note flags:
(list 'cons (racketfunc>primval cons #f))
(list 'list (racketfunc>primval list #f))
(list 'first (racketfunc>primval car #t)) ;**
(list 'rest (racketfunc>primval cdr #t)) ;**
(list 'null? (racketfunc>primval null? #t))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f))
(list 'null (RktV null)))
(EmptyEnv)))
Note that this last change raises a subtle type issue: we're actually
abusing the Racket `list` and `cons` constructors to hold Sloth values.
One way in which this becomes a problem is the current assumption that a
primitive function always returns a Racket value (it is always wrapped
in a `RktV`)  but this is no longer the case for `first` and `rest`:
when we use
{cons 1 null}
in Sloth, the resulting value will be
(RktV (cons (ExprV (Num 1) ...) (ExprV (Id null) ...)))
This leads to two problems: first, if we use Racket's `first` and
`rest`, they will complain (throw a runtime error) since the input value
is not a *proper* list (it's a pair that has a nonlist value in its
tail). To resolve that, we use the more primitive `car` and `cdr`
functions to implement Sloth's `first` and `rest`.
The second problem happens when we try and grab the first value of this
{first {cons 1 null}}
we will eventually get back the `ExprV` and wrap it in a `RktV`:
(RktV (ExprV (Num 1) ...))
and finally `run` will strip off the `RktV` and return the `ExprV`. A
solution to this is to make our `first` and `rest` functions return a
value *without* wrapping it in a `RktV`  we can identify this
situation by the fact that the returned value is already a VAL instead
of some other Racket value. We can identify such values with the `VAL?`
predicate that gets defined by our `definetype`, implemented by a new
`wrapinval` helper:
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: wrapinval : Any > VAL)
;; helper that ensures a VAL output using RktV wrapper when needed,
;; but leaving as is otherwise
(define (wrapinval x)
(if (VAL? x) x (RktV x)))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function ...
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict? (map unwraprktv args) args)])
(wrapinval (listfunc args))))))
Note that we don't need to worry about the result being an `ExprV` 
that will eventually be taken care of by `strict`.
The complete Sloth code follows. It can be used to do the same fun
things we did with Lazy Racket.
;;; <<>>
#lang pl
;;; 
;;; Syntax
# The BNF:
::=

 { bind {{ } ... } }
 { fun { ... } }
 { if }
 { ... }
#
;; A matching abstract syntax tree datatype:
(definetype SLOTH
[Num Number]
[Id Symbol]
[Bind (Listof Symbol) (Listof SLOTH) SLOTH]
[Fun (Listof Symbol) SLOTH]
[Call SLOTH (Listof SLOTH)]
[If SLOTH SLOTH SLOTH])
(: uniquelist? : (Listof Any) > Boolean)
;; Tests whether a list is unique, guards Bind and Fun values.
(define (uniquelist? xs)
(or (null? xs)
(and (not (member (first xs) (rest xs)))
(uniquelist? (rest xs)))))
(: parsesexpr : Sexpr > SLOTH)
;; parses sexpressions into SLOTHs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'bind more)
(match sexpr
[(list 'bind (list (list (symbol: names) (sexpr: nameds))
...)
body)
(if (uniquelist? names)
(Bind names (map parsesexpr nameds) (parsesexpr body))
(error 'parsesexpr "duplicate `bind' names: ~s" names))]
[else (error 'parsesexpr "bad `bind' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: names) ...) body)
(if (uniquelist? names)
(Fun names (parsesexpr body))
(error 'parsesexpr "duplicate `fun' names: ~s" names))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(cons 'if more)
(match sexpr
[(list 'if cond then else)
(If (parsesexpr cond)
(parsesexpr then)
(parsesexpr else))]
[else (error 'parsesexpr "bad `if' syntax in ~s" sexpr)])]
[(list fun args ...) ; other lists are applications
(Call (parsesexpr fun)
(map parsesexpr args))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > SLOTH)
;; Parses a string containing an SLOTH expression to a SLOTH AST.
(define (parse str)
(parsesexpr (string>sexpr str)))
;;; 
;;; Values and environments
(definetype ENV
[EmptyEnv]
[FrameEnv FRAME ENV])
;; a frame is an association list of names and values.
(definetype FRAME = (Listof (List Symbol VAL)))
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV]
[PrimV ((Listof VAL) > VAL)])
(: extend : (Listof Symbol) (Listof VAL) ENV > ENV)
;; extends an environment with a new frame.
(define (extend names values env)
(if (= (length names) (length values))
(FrameEnv (map (lambda ([name : Symbol] [val : VAL])
(list name val))
names values)
env)
(error 'extend "arity mismatch for names: ~s" names)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, frame by frame,
;; return its value or throw an error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(FrameEnv frame rest)
(let ([cell (assq name frame)])
(if cell
(second cell)
(lookup name rest)))]))
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: wrapinval : Any > VAL)
;; helper that ensures a VAL output using RktV wrapper when needed,
;; but leaving as is otherwise
(define (wrapinval x)
(if (VAL? x) x (RktV x)))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function
;; which is a PrimV holding a ((Listof VAL) > VAL) function.
;; (the resulting function will use the list function as is,
;; and it is the list function's responsibility to throw an error
;; if it's given a bad number of arguments or bad input types.)
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict? (map unwraprktv args) args)])
(wrapinval (listfunc args))))))
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval + #t))
(list ' (racketfunc>primval  #t))
(list '* (racketfunc>primval * #t))
(list '/ (racketfunc>primval / #t))
(list '< (racketfunc>primval < #t))
(list '> (racketfunc>primval > #t))
(list '= (racketfunc>primval = #t))
;; note flags:
(list 'cons (racketfunc>primval cons #f))
(list 'list (racketfunc>primval list #f))
(list 'first (racketfunc>primval car #t))
(list 'rest (racketfunc>primval cdr #t))
(list 'null? (racketfunc>primval null? #t))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f))
(list 'null (RktV null)))
(EmptyEnv)))
;;; 
;;; Evaluation
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise, returns a VAL that is
;; not an ExprV
(define (strict val)
(cases val
[(ExprV expr env) (strict (eval expr env))]
[else val]))
(: eval : SLOTH ENV > VAL)
;; evaluates SLOTH expressions
(define (eval expr env)
;; convenient helper
(: eval* : SLOTH > VAL)
(define (eval* expr) (ExprV expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (strict (eval* funexpr))]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (strict (eval* condexpr))
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
(: run : String > Any)
;; evaluate a SLOTH program contained in a string
(define (run str)
(let ([result (strict (eval (parse str) globalenvironment))])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
;;; 
;;; Tests
(test (run "{{fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}} {add3 1}}")
=> 4)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}
{add1 {fun {x} {+ x 1}}}}
{bind {{x 3}} {add1 {add3 x}}}}")
=> 7)
(test (run "{bind {{identity {fun {x} x}}
{foo {fun {x} {+ x 1}}}}
{{identity foo} 123}}")
=> 124)
(test (run "{bind {{x 3}}
{bind {{f {fun {y} {+ x y}}}}
{bind {{x 5}}
{f 4}}}}")
=> 7)
(test (run "{{{fun {x} {x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)
;; More tests for complete coverage
(test (run "{bind x 5 x}") =error> "bad `bind' syntax")
(test (run "{fun x x}") =error> "bad `fun' syntax")
(test (run "{if x}") =error> "bad `if' syntax")
(test (run "{}") =error> "bad syntax")
(test (run "{bind {{x 5} {x 5}} x}") =error> "duplicate*bind*names")
(test (run "{fun {x x} x}") =error> "duplicate*fun*names")
(test (run "{+ x 1}") =error> "no binding for")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{1 2}") =error> "with a nonfunction")
(test (run "{{fun {x} x}}") =error> "arity mismatch")
(test (run "{if {< 4 5} 6 7}") => 6)
(test (run "{if {< 5 4} 6 7}") => 7)
(test (run "{if + 6 7}") => 6)
(test (run "{fun {x} x}") =error> "returned a bad value")
;; Test laziness
(test (run "{{fun {x} 1} {/ 9 0}}") => 1)
(test (run "{{fun {x} 1} {{fun {x} {x x}} {fun {x} {x x}}}}") => 1)
(test (run "{bind {{x {{fun {x} {x x}} {fun {x} {x x}}}}} 1}") => 1)
;; Test lazy constructors
(test (run "{bind {{l {list 1 {/ 9 0} 3}}}
{+ {first l} {first {rest {rest l}}}}}")
=> 4)
;;; 

# Implementing Call by Need
As we have seen, there are a number of advantages for lazy evaluation,
but its main disadvantage is the fact that it is extremely inefficient,
to the point of rendering lots of programs impractical, for example, in:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
y}}
we end up adding 4 and 5 twice. In other words, we don't suffer from
textual redundancy (each expression is written once), but we don't avoid
dynamic redundancy. We can get it back by simply caching evaluation
results, using a box that will be used to remember the results. The box
will initially hold `#f`, and it will change to hold the VAL that
results from evaluation:
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV (Boxof (U #f VAL))] ;*** new: mutable cache field
[PrimV ((Listof VAL) > VAL)])
We need a utility function to create an evaluation promise, because when
an `ExprV` is created, its initial cache box needs to be initialized.
(: evalpromise : SLOTH ENV > VAL)
;; used instead of `eval' to create an evaluation promise
(define (evalpromise expr env)
(ExprV expr env (box #f)))
(And note that Typed Racket needs to figure out that the `#f` in this
definition has a type of `(U #f VAL)` and not just `#f`.)
This `evalpromise` is used instead of `ExprV` in eval. Finally,
whenever we force such an `ExprV` promise, we need to check if it was
already evaluated, otherwise force it and cache the result. This is
simple to do since there is a single field that is used both as a flag
and a cached value:
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise, returns a VAL that is
;; not an ExprV
(define (strict val)
(cases val
[(ExprV expr env cache)
(or (unbox cache)
(let ([val* (strict (eval expr env))])
(setbox! cache val*)
val*))]
[else val]))
But note that this makes using sideeffects in our interpreter even more
confusing. (It was true with callbyname too.)
The resulting code follows.
;;; <<>>
;; A callbyneed version of the SLOTH interpreter
#lang pl
;;; 
;;; Syntax
# The BNF:
::=

 { bind {{ } ... } }
 { fun { ... } }
 { if