Future notes, Wednesday, February 1st
=====================================
***This is provided for reference, and can change once we
go over this material in class.***


# Functions & Function Values
> [PLAI §4]
Now that we have a form for local bindings, which forced us to deal with
proper substitutions and everything that is related, we can get to
functions. The concept of a function is itself very close to
substitution, and to our `with` form. For example, when we write:
{with {x 5}
{* x x}}
then the `{* x x}` body is itself parametrized over some value for `x`.
If we take this expression and take out the `5`, we're left with
something that has all of the necessary ingredients of a function  a
bunch of code that is parameterized over some input identifier:
{with {x}
{* x x}}
We only need to replace `with` and use a proper name that indicates that
it's a function:
{fun {x}
{* x x}}
Now we have a new form in our language, one that should have a function
as its meaning. As we have seen in the case of `with` expressions, we
also need a new form to *use* these functions. We will use `call` for
this, so that
{call {fun {x} {* x x}}
5}
will be the same as the original `with` expression that we started with
 the `fun` expression is like the `with` expression with no value,
and applying it on `5` is providing that value back:
{with {x 5}
{* x x}}
Of course, this does not help much  all we get is a way to use local
bindings that is more verbose from what we started with. What we're
really missing is a way to *name* these functions. If we get the right
evaluation rules, we can evaluate a `fun` expression to some value 
which will allow us to bind it to a variable using `with`. Something
like this:
{with {sqr {fun {x} {* x x}}}
{+ {call sqr 5}
{call sqr 6}}}
In this expression, we say that `x` is the formal parameter (or
argument), and the `5` and `6` are actual parameters (sometimes
abbreviated as formals and actuals). Note that naming functions often
helps, but many times there are small functions that are fine to specify
without a name  for example, consider a twostage addition function,
where there is no apparent good name for the returned function:
{with {add {fun {x}
{fun {y}
{+ x y}}}}
{call {call add 8} 9}}

# Implementing First Class Functions
> [PLAI §6] (uses some stuff from [PLAI §5], which we do later)
This is a simple plan, but it is directly related to how functions are
going to be used in our language. We know that `{call {fun {x} E1} E2}`
is equivalent to a `with` expression, but the new thing here is that we
do allow writing just the `{fun ...}` expression by itself, and
therefore we need to have some meaning for it. The meaning, or the value
of this expression, should roughly be "an expression that needs a value
to be plugged in for `x`". In other words, our language will have these
new kinds of values that contain an expression to be evaluated later on.
There are three basic approaches that classify programming languages in
relation to how the deal with functions:
1. First order: functions are not real values. They cannot be used or
returned as values by other functions. This means that they cannot be
stored in data structures. This is what most "conventional" languages
used to have in the past. (You will be implementing such a language
in homework 4.)
An example of such a language is the Beginner Student language that
is used in HtDP, where the language is intentionally firstorder to
help students write correct code (at the early stages where using a
function as a value is usually an error). It's hard to find practical
modern languages that fall in this category.
2. Higher order: functions can receive and return other functions as
values. This is what you get with C and modern Fortran.
3. First class: functions are values with all the rights of other
values. In particular, they can be supplied to other functions,
returned from functions, stored in data structures, and new functions
can be created at runtime. (And most modern languages have first
class functions.)
The last category is the most interesting one. Back in the old days,
complex expressions were not firstclass in that they could not be
freely composed. This is still the case in machinecode: as we've seen
earlier, to compute an expression such as
(b + sqrt(b^2  4*a*c)) / 2a
you have to do something like this:
x = b * b
y = 4 * a
y = y * c
x = x  y
x = sqrt(x)
y = b
x = y + x
y = 2 * a
s = x / y
In other words, every intermediate value needs to have its own name. But
with proper ("highlevel") programming languages (at least most of
them...) you can just write the original expression, with no names for
these values.
With firstclass functions something similar happens  it is possible
to have complex expressions that consume and return functions, and they
do not need to be named.
What we get with our `fun` expression (if we can make it work) is
exactly this: it generates a function, and you can choose to either bind
it to a name, or not. The important thing is that the value exists
independently of a name.
This has a major effect on the "personality" of a programming language
as we will see. In fact, just adding this feature will make our language
much more advanced than languages with just higherorder or firstorder
functions.

Quick Example: the following is working JavaScript code, that uses first
class functions.
function foo(x) {
function bar(y) { return x + y; }
return bar;
}
function main() {
var f = foo(1);
var g = foo(10);
return [f(2), g(2)];
}
Note that the above definition of `foo` does *not* use an anonymous
"lambda expression"  in Racket terms, it's translated to
(define (foo x)
(define (bar y) (+ x y))
bar)
The returned function is not anonymous, but it's not really named
either: the `bar` name is bound only inside the body of `foo`, and
outside of it that name no longer exists since it's not its scope. It
gets used in the printed form if the function value is displayed, but
this is merely a debugging aid. The anonymous `lambda` version that is
common in Racket can be used in JavaScript too:
function foo(x) {
return function(y) { return x + y; }
}
> Sidenote: GCC includes extensions that allow internal function
> definitions, but it still does not have first class functions 
> trying to do the above is broken:
>
> #include
> typedef int(*int2int)(int);
> int2int foo(int x) {
> int bar(int y) { return x + y; }
> return bar;
> }
> int main() {
> int2int f = foo(1);
> int2int g = foo(10);
> printf(">> %d, %d\n", f(2), g(2));
> }

## Sidenote: how important is it to have *anonymous* functions?
You'll see many places where people refer to the feature of firstclass
functions as the ability to create *anonymous* functions, but this is a
confusion and it's not accurate. Whether a function has a name or not is
not the important question  instead, the important question is
whether functions can exist with no *bindings* that refers to them.
As a quick example in Racket:
(define (foo x)
(define (bar y) (+ x y))
bar)
in Javascript:
function foo(x) {
function bar(y) {
return x + y;
}
return bar;
}
and in Python:
def foo(x):
def bar(y):
return x + y
return bar
In all three of these, we have a `foo` function that returns a function
*named* `bar`  but the `bar` name, is only available in the scope of
`foo`. The fact that the name is displayed as part of the textual
rendering of the function value is merely a debugging feature.

# The FLANG Language
Now for the implementation  we call this new language FLANG.
First, the BNF:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
And the matching type definition:
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG] ; No namedexpression
[Call FLANG FLANG])
The parser for this grammar is, as usual, straightforward:
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
We also need to patch up the substitution function to deal with these
things. The scoping rule for the new function form is, unsurprisingly,
similar to the rule of `with`, except that there is no extra expression
now, and the scoping rule for `call` is the same as for the arithmetic
operators:
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]}
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
{call E1 E2}[v/x] = {call E1[v/x] E2[v/x]}
{fun {y} E}[v/x] = {fun {y} E[v/x]}
{fun {x} E}[v/x] = {fun {x} E}
And the matching code:
(: subst : FLANG Symbol FLANG > FLANG)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]
[(Call l r) (Call (subst l from to) (subst r from to))]
[(Fun boundid boundbody)
(if (eq? boundid from)
expr
(Fun boundid (subst boundbody from to)))]))

Now, before we start working on an evaluator, we need to decide on what
exactly do we use to represent values of this language. Before we had
functions, we had only number values and we used Racket numbers to
represent them. Now we have two kinds of values  numbers and
functions. It seems easy enough to continue using Racket numbers to
represent numbers, but what about functions? What should be the result
of evaluating
{fun {x} {+ x 1}}
? Well, this is the new toy we have: it should be a function value,
which is something that can be used just like numbers, but instead of
arithmetic operations, we can `call` these things. What we need is a way
to avoid evaluating the body expression of the function  *delay* it
 and instead use some value that will contain this delayed expression
in a way that can be used later.
To accommodate this, we will change our implementation strategy a
little: we will use our syntax objects for numbers (`(Num n)` instead of
just `n`), which will be a little inconvenient when we do the arithmetic
operations, but it will simplify life by making it possible to evaluate
functions in a similar way: simply return their own syntax object as
their values. The syntax object has what we need: the body expression
that needs to be evaluated later when the function is called, and it
also has the identifier name that should be replaced with the actual
input to the function call. This means that evaluating:
(Add (Num 1) (Num 2))
now yields
(Num 3)
and a number `(Num 5)` evaluates to `(Num 5)`.
In a similar way, `(Fun 'x (Num 2))` evaluates to `(Fun 'x (Num 2))`.
Why would this work? Well, because `call` will be very similar to `with`
 the only difference is that its arguments are ordered a little
differently, being retrieved from the function that is applied and the
argument.
The formal evaluation rules are therefore treating functions like
numbers, and use the syntax object to represent both values:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2)
eval({ E1 E2}) = eval(E1)  eval(E2)
eval({* E1 E2}) = eval(E1) * eval(E2)
eval({/ E1 E2}) = eval(E1) / eval(E2)
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(FUN) = FUN ; assuming FUN is a function expression
eval({call E1 E2})
= eval(Ef[eval(E2)/x]) if eval(E1) = {fun {x} Ef}
= error! otherwise
Note that the last rule could be written using a translation to a `with`
expression:
eval({call E1 E2})
= eval({with {x E2} Ef}) if eval(E1) = {fun {x} Ef}
= error! otherwise
And alternatively, we could specify `with` using `call` and `fun`:
eval({with {x E1} E2}) = eval({call {fun {x} E2} E1})
There is a small problem in these rules which is intuitively seen by the
fact that the evaluation rule for a `call` is expected to be very
similar to the one for arithmetic operations. We now have two kinds of
values, so we need to check the arithmetic operation's arguments too:
eval({+ E1 E2}) = N1 + N2
if eval(E1), eval(E2) evaluate to numbers N1, N2
otherwise error!
...
The corresponding code is:
(: eval : FLANG > FLANG) ;*** note return type
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr] ;*** change here
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))] ;*** no `(Num ...)'
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr] ;*** similar to `Num'
[(Call (Fun boundid boundbody) argexpr) ;*** nested pattern
(eval (subst boundbody ;*** just like `with'
boundid
(eval argexpr)))]
[(Call something argexpr)
(error 'eval "`call' expects a function, got: ~s" something)]))
Note that the `Call` case is doing the same thing we do in the `With`
case. In fact, we could have just *generated* a `With` expression and
evaluate that instead:
...
[(Call (Fun boundid boundbody) argexpr)
(eval (With boundid argexpr boundbody))]
...
The `arithop` function is in charge of checking that the input values
are numbers (represented as FLANG numbers), translating them to plain
numbers, performing the Racket operation, then rewrapping the result in
a `Num`. Note how its type indicates that it is a higherorder function.
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper (note the H.O. type, and note the hack of the `val`
;; name which is actually an AST that represents a runtime value)
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
It uses the following function to convert FLANG numbers to Racket
numbers. (Note that `else` is almost always a bad idea since it can
prevent the compiler from showing you places to edit code  but this
case is an exception since we never want to deal with anything other
than `Num`s.) The reason that this function is relatively trivial is
that we chose the easy way and represented numbers using Racket numbers,
but we could have used strings or anything else.
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
We can also make things a little easier to use if we make `run` convert
the result to a number:
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str))])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
Adding few simple tests we get:
;; The Flang interpreter
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
subst:
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]} ; if y =/= x
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
{call E1 E2}[v/x] = {call E1[v/x] E2[v/x]}
{fun {y} E}[v/x] = {fun {y} E[v/x]} ; if y =/= x
{fun {x} E}[v/x] = {fun {x} E}
eval:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2) \ if both E1 and E2
eval({ E1 E2}) = eval(E1)  eval(E2) \ evaluate to numbers
eval({* E1 E2}) = eval(E1) * eval(E2) / otherwise error!
eval({/ E1 E2}) = eval(E1) / eval(E2) /
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(FUN) = FUN ; assuming FUN is a function expression
eval({call E1 E2}) = eval(Ef[eval(E2)/x])
if eval(E1)={fun {x} Ef}, otherwise error!
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
(: subst : FLANG Symbol FLANG > FLANG)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]
[(Call l r) (Call (subst l from to) (subst r from to))]
[(Fun boundid boundbody)
(if (eq? boundid from)
expr
(Fun boundid (subst boundbody from to)))]))
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
(: eval : FLANG > FLANG)
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))]
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr]
[(Call (Fun boundid boundbody) argexpr)
(eval (subst boundbody
boundid
(eval argexpr)))]
[(Call something argexpr)
(error 'eval "`call' expects a function, got: ~s" something)]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str))])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)

There is still a problem with this version. First a question  if
`call` is similar to arithmetic operations (and to `with` in what it
actually does), then how come the code is different enough that it
doesn't even need an auxiliary function?
Second question: what *should* happen if we evaluate these code
snippets:
(run "{with {add {fun {x}
{fun {y}
{+ x y}}}}
{call {call add 8} 9}}")
(run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
(run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
Third question, what *will* happen if we do the above?
What we're missing is an evaluation of the function expression, in case
it's not a literal `fun` form. The following fixes this:
(: eval : FLANG > FLANG)
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))]
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr)]) ;*** need to evaluate this!
(cases fval
[(Fun boundid boundbody)
(eval (subst boundbody
boundid
(eval argexpr)))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
The complete code is:
;;; <<>>
;; The Flang interpreter
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
subst:
N[v/x] = N
{+ E1 E2}[v/x] = {+ E1[v/x] E2[v/x]}
{ E1 E2}[v/x] = { E1[v/x] E2[v/x]}
{* E1 E2}[v/x] = {* E1[v/x] E2[v/x]}
{/ E1 E2}[v/x] = {/ E1[v/x] E2[v/x]}
y[v/x] = y
x[v/x] = v
{with {y E1} E2}[v/x] = {with {y E1[v/x]} E2[v/x]} ; if y =/= x
{with {x E1} E2}[v/x] = {with {x E1[v/x]} E2}
{call E1 E2}[v/x] = {call E1[v/x] E2[v/x]}
{fun {y} E}[v/x] = {fun {y} E[v/x]} ; if y =/= x
{fun {x} E}[v/x] = {fun {x} E}
eval:
eval(N) = N
eval({+ E1 E2}) = eval(E1) + eval(E2) \ if both E1 and E2
eval({ E1 E2}) = eval(E1)  eval(E2) \ evaluate to numbers
eval({* E1 E2}) = eval(E1) * eval(E2) / otherwise error!
eval({/ E1 E2}) = eval(E1) / eval(E2) /
eval(id) = error!
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
eval(FUN) = FUN ; assuming FUN is a function expression
eval({call E1 E2}) = eval(Ef[eval(E2)/x])
if eval(E1)={fun {x} Ef}, otherwise error!
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
(: subst : FLANG Symbol FLANG > FLANG)
;; substitutes the second argument with the third argument in the
;; first argument, as per the rules of substitution; the resulting
;; expression contains no free instances of the second argument
(define (subst expr from to)
(cases expr
[(Num n) expr]
[(Add l r) (Add (subst l from to) (subst r from to))]
[(Sub l r) (Sub (subst l from to) (subst r from to))]
[(Mul l r) (Mul (subst l from to) (subst r from to))]
[(Div l r) (Div (subst l from to) (subst r from to))]
[(Id name) (if (eq? name from) to expr)]
[(With boundid namedexpr boundbody)
(With boundid
(subst namedexpr from to)
(if (eq? boundid from)
boundbody
(subst boundbody from to)))]
[(Call l r) (Call (subst l from to) (subst r from to))]
[(Fun boundid boundbody)
(if (eq? boundid from)
expr
(Fun boundid (subst boundbody from to)))]))
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
(: eval : FLANG > FLANG)
;; evaluates FLANG expressions by reducing them to *expressions* but
;; only expressions that stand for values: only `Fun`s and `Num`s
(define (eval expr)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l) (eval r))]
[(Sub l r) (arithop  (eval l) (eval r))]
[(Mul l r) (arithop * (eval l) (eval r))]
[(Div l r) (arithop / (eval l) (eval r))]
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(eval namedexpr)))]
[(Id name) (error 'eval "free identifier: ~s" name)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr)])
(cases fval
[(Fun boundid boundbody)
(eval (subst boundbody
boundid
(eval argexpr)))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str))])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {add {fun {x}
{fun {y}
{+ x y}}}}
{call {call add 8} 9}}")
=> 17)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Introducing Racket's `lambda`
> Quick laundry list of things to go over:
>  `fun` & `lambda`
>  difference between lambda and simple values
>  not being able to do recursive functions with `let`
>  let* as a derived form
>  let with lambda in Racket > can be a derived form
>  how `if` can be used to implement `and` and `or` as derived forms
>  Newtonian syntax vs. a lambda expression.
Almost all modern languages have this capability. For example, this:
(define (f g) (g 2 3))
(f +) ==> 5
(f *) ==> 6
(f (lambda (x y) (+ (square x) (square y)))) ==> 13
Can be written in JavaScript like this:
function f(g) { return g(2,3); }
function square(x) { return x*x; }
console.log(f(function (x,y) { return square(x) + square(y); }));
or in ES6 JavaScript:
let f = (g) => g(2,3);
let square = (x) => x*x;
console.log(f((x,y) => square(x) + square(y)));
In Perl:
sub f { my ($g) = @_; return $g>(2,3); }
sub square { my ($x) = @_; return $x * $x; }
print f(sub { my ($x, $y) = @_; return square($x) + square($y); });
In Ruby:
def f(g) g.call(2,3) end
def square(x) x*x end
puts f(lambda{x,y square(x) + square(y)})
etc. Even [Java has lambda expressions], and "recently" [C++ added them
too].
[Java has lambda expressions]:
http://www.drdobbs.com/jvm/lambdaexpressionsinjava8/240166764
[C++ added them too]:
http://www.cprogramming.com/c++11/c++11lambdaclosures.html

# Using Functions as Objects
A very important aspect of Racket  using "higher order" functions 
functions that get and return functions. Here is a very simple example:
(define (f x) (lambda () x))
(define a (f 2))
(a) > 2
(define b (f 3))
(b) > 3
Note: what we get is actually an object that remembers (by the
substitution we're doing) a number. How about:
(define aa (f a))
(aa) > # (this is a)
((aa)) > 2
Take this idea to the next level:
(define (kons x y)
(lambda (b)
(if b x y)))
(define (kar p) (p #t))
(define (kdr p) (p #f))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
Or, with types:
(: kons : (All (A B) A B > (Boolean > (U A B))))
(define (kons x y)
(lambda (b)
(if b x y)))
(: kar : (All (T) (Boolean > T) > T))
(define (kar p) (p #t))
(: kdr : (All (T) (Boolean > T) > T))
(define (kdr p) (p #f))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
Even more  why should the internal function expect a boolean and
choose what to return? We can simply expect a function that will take
the two values and return one:
(define (kons x y) (lambda (s) (s x y)))
(define (kar p) (p (lambda (x y) x)))
(define (kdr p) (p (lambda (x y) y)))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
And a typed version, using our own constructor to make it a little less
painful:
(definetype (Kons A B) = ((A B > (U A B)) > (U A B)))
(: kons : (All (A B) A B > (Kons A B)))
(define (kons x y) (lambda (s) (s x y)))
(: kar : (All (A B) (Kons A B) > (U A B)))
(define (kar p) (p (lambda (x y) x)))
(: kdr : (All (A B) (Kons A B) > (U A B)))
(define (kdr p) (p (lambda (x y) y)))
(define a (kons 1 2))
(define b (kons 3 4))
(list (kar a) (kdr a))
(list (kar b) (kdr b))
Note that the `Kons` type definition is the same as:
(definetype Kons = (All (A B) (A B > (U A B)) > (U A B)))
so `All` is to polymorphic type definitions what `lambda` is for
function definitions.
Finally in JavaScript:
function kons(x,y) { return function(s) { return s(x, y); } }
function kar(p) { return p(function(x,y){ return x; }); }
function kdr(p) { return p(function(x,y){ return y; }); }
a = kons(1,2);
b = kons(3,4);
console.log('a = <' + kar(a) + ',' + kdr(a) + '>' );
console.log('b = <' + kar(b) + ',' + kdr(b) + '>' );
Or with ES6 *arrow functions*, the function definitionss become:
const kons = (x,y) => s => s(x,y);
const kar = p => p((x,y) => x);
const kdr = p => p((x,y) => y);

# Using `definetype` for new "type aliases"
As seen in these examples, there is another way to use `definetype`,
using a `=` to create a new type name "alias" for an *existing* type.
For example:
(definetype Strings = (Listof String))
These uses of `definetype` do not define any new kind of type, they are
essentially a convenience tool for making code shorter and more
readable.
(definetype NumericFunction = Number > Number)
(: square : NumericFunction)
(define (square n) (* n n))
Note in particular that this can also be used to define "alias type
constructors" too: somewhat similar to creating new "type functions".
For example:
(definetype (BinaryFun In Out) = In In > Out)
(: diagonal : (BinaryFun Natural Number))
(define (diagonal width height)
(sqrt (+ (* width width) (* height height))))
This is something that we will only need in a few rare cases.

# Currying
A *curried* function is a function that, instead of accepting two (or
more) arguments, accepts only one and returns a function that accepts
the rest. For example:
(: plus : Number > (Number > Number))
(define (plus x)
(lambda (y)
(+ x y)))
It's easy to write functions for translating between normal and curried
versions.
(define (currify f)
(lambda (x)
(lambda (y)
(f x y))))
Typed version of that, with examples:
(: currify : (All (A B C) (A B > C) > (A > (B > C))))
;; convert a doubleargument function to a curried one
(define (currify f)
(lambda (x) (lambda (y) (f x y))))
(: add : Number Number > Number)
(define (add x y) (+ x y))
(: plus : Number > (Number > Number))
(define plus (currify add))
(test ((plus 1) 2) => 3)
(test (((currify add) 1) 2) => 3)
(test (map (plus 1) '(1 2 3)) => '(2 3 4))
(test (map ((currify add) 1) '(1 2 3)) => '(2 3 4))
(test (map ((currify +) 1) '(1 2 3)) => '(2 3 4))
Usages  common with H.O. functions like map, where we want to *fix*
one argument.
When dealing with such higherorder code, the types are very helpful,
since every arrow corresponds to a function:
(: currify : (All (A B C) (A B > C) > (A > (B > C))))
It is common to make the `>` function type associate to the right, so
you can find this type written as:
currify : (A B > C) > (A > B > C)
or even as
currify : (A B > C) > A > B > C
but that can be a little confusing...

# Using HigherOrder & Anonymous Functions
Say that we have a function for estimating derivatives of a function at
a specific point:
(define dx 0.01)
(: deriv : (Number > Number) Number > Number)
;; compute the derivative of `f' at the given point `x'
(define (deriv f x)
(/ ( (f (+ x dx)) (f x)) dx))
(: integrate : (Number > Number) Number > Number)
;; compute an integral of `f' at the given point `x'
(define (integrate f x)
(: loop : Number Number > Number)
(define (loop y acc)
(if (> y x)
(* acc dx)
(loop (+ y dx) (+ acc (f y)))))
(loop 0 0))
And say that we want to try out various functions given some `plot`
function that draws graphs of numeric functions, for example:
(plot sin)
The problem is that `plot` expects a single `(Number > Number)`
function  if we want to try it with a derivative, we can do this:
(: sinderiv : Number > Number)
;; the derivative of sin
(define sinderiv (lambda (x) (deriv sin x)))
(plot sinderiv)
But this will get very tedious very fast  it is much simpler to use
an anonymous function:
(plot (lambda (x) (deriv sin x)))
we can even verify that our derivative is correct by comparing a known
function to its derivative
(plot (lambda (x) ( (deriv sin x) (cos x))))
But it's still not completely natural to do these things  you need to
explicitly combine functions, which is not too convenient. Instead of
doing this, we can write H.O. functions that will work with functional
inputs and outputs. For example, we can write a function to subtract
functions:
(: fsub : (Number > Number) (Number > Number)
> (Number > Number))
;; subtracts two numeric 1argument functions
(define (fsub f g)
(lambda (x) ( (f x) (g x))))
and the same for the derivative:
(: fderiv : (Number > Number) > (Number > Number))
;; compute the derivative function of `f'
(define (fderiv f)
(lambda (x) (deriv f x)))
Now we can try the same in a much easier way:
(plot (fsub (fderiv sin) cos))
More than that  our `fderiv` could be created from `deriv`
automatically:
(: currify : (All (A B C) (A B > C) > (A > B > C)))
;; convert a doubleargument function to a curried one
(define (currify f)
(lambda (x) (lambda (y) (f x y))))
(: fderiv : (Number > Number) > (Number > Number))
;; compute the derivative function of `f'
(define fderiv (currify deriv))
Same principle with `fsub`: we can write a function that converts a
binary arithmetical function into a function that operates on unary
numeric function. But to make things more readable we can define new
types for unary and binary numeric functions:
(definetype UnaryFun = (Number > Number))
(definetype BinaryFun = (Number Number > Number))
(: binop>fbinop : BinaryFun > (UnaryFun UnaryFun > UnaryFun))
;; turns an arithmetic binary operator to a function operator
(define (binop>fbinop op)
(lambda (f g)
(lambda (x) (op (f x) (g x)))))
(: fsub : UnaryFun UnaryFun > UnaryFun)
;; functional pointwise subtraction
(define fsub (binop>fbinop ))
We can do this with anything  developing a rich library of functions
and functionals (functions over functions) is extremely easy... Here's a
pretty extensive yet very short library of functions:
#lang pl untyped
(define (currify f)
(lambda (x) (lambda (y) (f x y))))
(define (binop>fbinop op)
(lambda (f g)
(lambda (x) (op (f x) (g x)))))
(define (compose f g)
(lambda (x) (f (g x))))
(define dx 0.01)
(define (deriv f x)
(/ ( (f (+ x dx)) (f x)) dx))
(define (integrate f x)
(define over? (if (< x 0) < >))
(define step (if (< x 0)  +))
(define add (if (< x 0)  +))
(define (loop y acc)
(if (over? y x)
(* acc dx)
(loop (step y dx) (add acc (f y)))))
(loop 0 0))
(define fadd (binop>fbinop +))
(define fsub (binop>fbinop ))
(define fmul (binop>fbinop *))
(define fdiv (binop>fbinop /))
(define fderiv (currify deriv))
(define fintegrate (currify integrate))
;; ...
This is written in the "untyped dialect" of the class language, but it
should be easy now to add the types.
Examples:
;; want to verify that `integrate' is the opposite of `deriv':
;; take a function, subtract it from its derivative's integral
(plot (fsub sin (fintegrate (fderiv sin))))
;; want to magnify the errors?  here's how you magnify:
(plot (compose ((currify *) 5) sin))
;; so:
(plot (compose ((currify *) 20)
(fsub sin (fintegrate (fderiv sin)))))

## Sidenote: "PointFree" combinators
> Forming functions without using `lambda` (or an implicit `lambda`
> using a `define` syntactic sugar) is called *pointfree style*. It's
> especially popular in Haskell, where it is easier to form functions
> this way because of implicit currying and a large number of higher
> level function combinators. If used too much, it can easily lead to
> obfuscated code.

## This is not Runtime Code Generation
All of this is similar to runtime code generation, but not really. The
only thing that `fderiv` does is take a function and store it somewhere
in the returned function, then when that function receives a number, it
uses the stored function and send it to deriv with the number. We could
simply write deriv as what `fderiv` is  which is the *real*
derivative function:
(define (deriv f)
(lambda (x)
(/ ( (f (+ x dx)) (f x)) dx)))
but again, this is not faster or slower than the plain `deriv`. However,
there are some situations where we can do some of the computation on the
firststage argument, saving work from the second stage. Here is a
cookedtoexaggeration example  we want a function that receives two
inputs `x`, `y` and returns `fib(x)*y`, but we must use a stupid `fib`:
(define (fib n)
(if (<= n 1)
n
(+ (fib ( n 1)) (fib ( n 2)))))
The function we want is:
(define (bogus x y)
(* (fib x) y))
If we currify it as usual (or just use `currify`), we get:
(define (bogus x)
(lambda (y)
(* (fib x) y)))
And try this several times:
(define bogus36 (bogus 36))
(map bogus36 '(1 2 3 4 5))
But in the definition of `bogus`, notice that `(fib x)` does not depend
on `y`  so we can rewrite it a little differently:
(define (bogus x)
(let ([fibx (fib x)])
(lambda (y)
(* fibx y))))
and trying the above again is much faster now:
(define bogus36 (bogus 36))
(map bogus36 '(1 2 3 4 5))
This is therefore not doing any kind of runtime code generation, but it
*enables* doing similar optimizations in our code. A proper RTCG
facility would recompile the curried function for a given first input,
and (hopefully) automatically achieve the optimization that we did in a
manual way.

# Substitution Caches
> [PLAI §5] (called "deferred substitutions" there)
Evaluating using substitutions is very inefficient  at each scope, we
copy a piece of the program AST. This includes all function calls which
implies an impractical cost (function calls should be *cheap*!).
To get over this, we want to use a cache of substitutions.
Basic idea: we begin evaluating with no cached substitutions, then
collect them as we encounter bindings.
[Implies another change for our evaluator: we don't really substitute
identifiers until we get to them; when we reach an identifier, it is no
longer an error  we must consult the substitution cache.]

# Implementation of Cache Functionality
First, we need a type for a substitution cache. For this we will use a
list of lists of two elements each  a name and its value FLANG:
;; a type for substitution caches:
(definetype SubstCache = (Listof (List Symbol FLANG)))
We need to have an empty substitution cache, a way to extend it, and a
way to look things up:
(: emptysubst : SubstCache)
(define emptysubst null)
(: extend : Symbol FLANG SubstCache > SubstCache)
;; extend a given substitution cache with a new mapping
(define (extend id expr sc)
(cons (list id expr) sc))
(: lookup : Symbol SubstCache > FLANG)
;; lookup a symbol in a substitution cache, return the value it is
;; bound to (or throw an error if it isn't bound)
(define (lookup name sc)
(cond [(null? sc) (error 'lookup "no binding for ~s" name)]
[(eq? name (first (first sc))) (second (first sc))]
[else (lookup name (rest sc))]))
Actually, the reason to use such list of lists is that Racket has a
builtin function called `assq` that will do this kind of search (`assq`
is a search in an association list using `eq?` for the key comparison).
This is a version of `lookup` that uses `assq`:
(define (lookup name sc)
(let ([cell (assq name sc)])
(if cell
(second cell)
(error 'lookup "no binding for ~s" name))))

# Formal Rules for Cached Substitutions
The formal evaluation rules are now different. Evaluation carries along
a *substitution cache* that begins its life as empty: so `eval` needs an
extra argument. We begin by writing the rules that deal with the cache,
and use the above function names for simplicity  the behavior of the
three definitions can be summed up in a single rule for `lookup`:
lookup(x,emptysubst) = error!
lookup(x,extend(x,E,sc)) = E
lookup(x,extend(y,E,sc)) = lookup(x,sc) if `x' is not `y'
And now we can write the new rules for `eval`
eval(N,sc) = N
eval({+ E1 E2},sc) = eval(E1,sc) + eval(E2,sc)
eval({ E1 E2},sc) = eval(E1,sc)  eval(E2,sc)
eval({* E1 E2},sc) = eval(E1,sc) * eval(E2,sc)
eval({/ E1 E2},sc) = eval(E1,sc) / eval(E2,sc)
eval(x,sc) = lookup(x,sc)
eval({with {x E1} E2},sc) = eval(E2,extend(x,eval(E1,sc),sc))
eval({fun {x} E},sc) = {fun {x} E}
eval({call E1 E2},sc)
= eval(Ef,extend(x,eval(E2,sc),sc))
if eval(E1,sc) = {fun {x} Ef}
= error! otherwise
Note that there is no mention of `subst`  the whole point is that we
don't really do substitution, but use the cache instead. The `lookup`
rules, and the places where `extend` is used replaces `subst`, and
therefore specifies our scoping rules.
Also note that the rule for `call` is still very similar to the rule for
`with`, but it looks like we have lost something  the interesting bit
with substituting into `fun` expressions.

# Evaluating with Substitution Caches
Implementing the new `eval` is easy now  it is extended in the same
way that the formal `eval` rule is extended:
(: eval : FLANG SubstCache > FLANG)
;; evaluates FLANG expressions by reducing them to expressions
(define (eval expr sc)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l sc) (eval r sc))]
[(Sub l r) (arithop  (eval l sc) (eval r sc))]
[(Mul l r) (arithop * (eval l sc) (eval r sc))]
[(Div l r) (arithop / (eval l sc) (eval r sc))]
[(With boundid namedexpr boundbody)
(eval boundbody
(extend boundid (eval namedexpr sc) sc))]
[(Id name) (lookup name sc)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr sc)])
(cases fval
[(Fun boundid boundbody)
(eval boundbody
(extend boundid (eval argexpr sc) sc))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
Again, note that we don't need `subst` anymore, but the rest of the code
(the data type definition, parsing, and `arithop`) is exactly the same.
Finally, we need to make sure that `eval` is initially called with an
empty cache. This is easy to change in our main `run` entry point:
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) emptysubst)])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
The full code (including the same tests, but not including formal rules
for now) follows. Note that one test does not pass.
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; a type for substitution caches:
(definetype SubstCache = (Listof (List Symbol FLANG)))
(: emptysubst : SubstCache)
(define emptysubst null)
(: extend : Symbol FLANG SubstCache > SubstCache)
;; extend a given substitution cache with a new mapping
(define (extend name val sc)
(cons (list name val) sc))
(: lookup : Symbol SubstCache > FLANG)
;; lookup a symbol in a substitution cache, return the value it is
;; bound to (or throw an error if it isn't bound)
(define (lookup name sc)
(let ([cell (assq name sc)])
(if cell
(second cell)
(error 'lookup "no binding for ~s" name))))
(: Num>number : FLANG > Number)
;; convert a FLANG number to a Racket one
(define (Num>number e)
(cases e
[(Num n) n]
[else (error 'arithop "expected a number, got: ~s" e)]))
(: arithop : (Number Number > Number) FLANG FLANG > FLANG)
;; gets a Racket numeric binary operator, and uses it within a FLANG
;; `Num' wrapper
(define (arithop op val1 val2)
(Num (op (Num>number val1) (Num>number val2))))
(: eval : FLANG SubstCache > FLANG)
;; evaluates FLANG expressions by reducing them to expressions
(define (eval expr sc)
(cases expr
[(Num n) expr]
[(Add l r) (arithop + (eval l sc) (eval r sc))]
[(Sub l r) (arithop  (eval l sc) (eval r sc))]
[(Mul l r) (arithop * (eval l sc) (eval r sc))]
[(Div l r) (arithop / (eval l sc) (eval r sc))]
[(With boundid namedexpr boundbody)
(eval boundbody
(extend boundid (eval namedexpr sc) sc))]
[(Id name) (lookup name sc)]
[(Fun boundid boundbody) expr]
[(Call funexpr argexpr)
(let ([fval (eval funexpr sc)])
(cases fval
[(Fun boundid boundbody)
(eval boundbody
(extend boundid (eval argexpr sc) sc))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) emptysubst)])
(cases result
[(Num n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> "???")
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Dynamic and Lexical Scopes
This seems like it should work, and it even worked on a few examples,
except for one which was hard to follow. Seems like we have a bug...
Now we get to a tricky issue that managed to be a problem for *lots* of
language implementors, including the first version of Lisp. Lets try to
run the following expression  try to figure out what it will evaluate
to:
(run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
We expect it to return `7` (at least I do!), but we get `9` instead...
The question is  *should* it return `9`?
What we have arrived to is called *dynamic scope*. Scope is determined
by the dynamic runtime environment (which is represented by our
substitution cache). This is *almost always* undesirable, as I hope to
convince you.
Before we start, we define two scope options for a programming language:
* Static Scope (also called Lexical Scope): In a language with static
scope, each identifier gets its value from the scope of its
definition, not its use.
* Dynamic Scope: In a language with dynamic scope, each identifier gets
its value from the scope of its use, not its definition.
Racket uses lexical scope, our new evaluator uses dynamic, the old
substitutionbased evaluator was static etc.
As a sideremark, Lisp began its life as a dynamicallyscoped language.
The artifacts of this were (sortof) dismissed as an implementation bug.
When Scheme was introduced, it was the first Lisp dialect that used
strictly lexical scoping, and Racket is obviously doing the same. (Some
Lisp implementations used dynamic scope for interpreted code and lexical
scope for compiled code!) In fact, Emacs Lisp is the only *live*
dialects of Lisp that is still dynamically scoped by default. To see
this, compare a version of the above code in Racket:
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(f 4))))
and the Emacs Lisp version (which looks almost the same):
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(funcall f 4))))
which also happens when we use another function on the way:
(defun blah (func val)
(funcall func val))
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(blah f 4))))
and note that renaming identifiers can lead to different code  change
that `val` to `x`:
(defun blah (func x)
(funcall func x))
(let ((x 3))
(let ((f (lambda (y) (+ x y))))
(let ((x 5))
(blah f 4))))
and you get `8` because the argument name changed the `x` that the
internal function sees!
Consider also this Emacs Lisp function:
(defun returnx ()
x)
which has no meaning by itself (`x` is unbound),
(returnx)
but can be given a dynamic meaning using a `let`:
(let ((x 5)) (returnx))
or a function application:
(defun foo (x)
(returnx))
(foo 5)
There is also a dynamicallyscoped language in the course languages:
#lang pl dynamic
(define x 123)
(define (getx) x)
(define (bar1 x) (getx))
(define (bar2 y) (getx))
(test (getx) => 123)
(test (let ([x 456]) (getx)) => 456)
(test (getx) => 123)
(test (bar1 999) => 999)
(test (bar2 999) => 123)
(define (foo x) (define (helper) (+ x 1)) helper)
(test ((foo 0)) => 124)
;; and *much* worse:
(define (add x y) (+ x y))
(test (let ([+ *]) (add 6 7)) => 42)
Note how bad the last example gets: you basically cannot call any
function and know in advance what it will do.
There are some cases where dynamic scope can be useful in that it allows
you to "remotely" customize any piece of code. A good example of where
this is taken to an extreme is Emacs: originally, it was based on an
ancient Lisp dialect that was still dynamically scoped, but it retained
this feature even when practically all Lisp dialects moved on to having
lexical scope by default. The reason for this is that the danger of
dynamic scope is also a way to make a very open system where almost
anything can be customized by changing it "remotely". Here's a concrete
example for a similar kind of dynamic scope usage that makes a very
hackable and open system:
#lang pl dynamic
(define tax% 6.25)
(define (withtax n)
(+ n (* n (/ tax% 100))))
(withtax 10) ; how much do we pay?
(let ([tax% 17.0]) (withtax 10)) ; how much would we pay in Israel?
;; make that into a function
(define iltax% 17.0)
(define (maoverilsaving n)
( (let ([tax% iltax%]) (withtax n))
(withtax n)))
(maoverilsaving 10)
;; can even control that: how much would we save if
;; the tax in israel went down one percent?
(let ([iltax% ( iltax% 1)]) (maoverilsaving 10))
;; or change both: how much savings in NH instead of MA?
(let ((tax% 0.0) (iltax% tax%)) (maoverilsaving 1000))
Obviously, this power to customize everything is also the main source of
problems with getting no guarantees for code. A common way to get the
best of both worlds is to have *controllable* dynamic scope. For
example, Common Lisp also has lexical scope everywhere by default, but
some variables can be declared as *special*, which means that they are
dynamically scoped. The main problem with that is that you can't tell
when a variable is special by just looking at the code that uses it, so
a more popular approach is the one that is used in Racket: all bindings
are always lexically scoped, but there are *parameters* which are a kind
of dynamically scoped value containers  but they are bound to plain
(lexically scoped) identifiers. Here's the same code as above,
translated to Racket with parameters:
#lang racket
(define tax% (makeparameter 6.5)) ; create the dynamic container
(define (withtax n)
(+ n (* n (/ (tax%) 100)))) ; note how its value is accessed
(withtax 10) ; how much do we pay?
(parameterize ([tax% 17.0]) (withtax 10)) ; not a `let'
;; make that into a function
(define iltax% (makeparameter 17.0))
(define (maoverilsaving n)
( (parameterize ([tax% (iltax%)]) (withtax n))
(withtax n)))
(maoverilsaving 10)
(parameterize ([iltax% ( (iltax%) 1)]) (maoverilsaving 10))
The main point here is that the points where a dynamically scoped value
is used are under the programmer's control  you cannot "customize"
what `` is doing, for example. This gives us back the guarantees that
we like to have (= that code works), but of course these points are
predetermined, unlike an environment where everything can be customized
including things that are unexpectedly useful.
> As a sidenote, after many decades of debating this, Emacs has finally
> added lexical scope in its core language, but this is still determined
> by a flag  a global `lexicalbinding` variable.

# Dynamic versus Lexical Scope
And back to the discussion of whether we should use dynamic or lexical
scope:
* The most important fact is that we want to view programs as executed
by the normal substituting evaluator. Our original motivation was to
optimize evaluation only  not to *change* the semantics! It follows
that we want the result of this optimization to behave in the same
way. All we need is to evaluate:
(run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
in the original evaluator to get convinced that `7` should be the
correct result (note also that the same code, when translated into
Racket, evaluates to `7`).
(Yet, this is a very important optimization, which without it lots of
programs become too slow to be feasible, so you might claim that
you're fine with the modified semantics...)
* It does not allow using functions as objects, for example, we have
seen that we have a functional representation for pairs:
(define (kons x y)
(lambda (n)
(match n
['first x]
['second y]
[else (error ...)])))
(define mypair (kons 1 2))
If this is evaluated in a dynamicallyscoped language, we do get a
function as a result, but the values bound to `x` and `y` are now
gone! Using the substitution model we substituted these values in, but
now they were only held in a cache which no has no entries for them...
In the same way, currying would not work, our nice `deriv` function
would not work etc etc etc.
* Makes reasoning impossible, because any piece of code behaves in a way
that *cannot* be predicted until runtime. For example, if dynamic
scoping was used in Racket, then you wouldn't be able to know what
this function is doing:
(define (foo)
x)
As it is, it will cause a runtime error, but if you call it like
this:
(let ([x 1])
(foo))
then it will return `1`, and if you later do this:
(define (bar x)
(foo))
(let ([x 1])
(bar 2))
then you would get `2`!
These problems can be demonstrated in Emacs Lisp too, but Racket goes
one step further  it uses the same rule for evaluating a function
as well as its values (Lisp uses a different namespace for
functions). Because of this, you cannot even rely on the following
function:
(define (add x y)
(+ x y))
to always add `x` and `y`!  A similar example to the above:
(let ([+ ])
(add 1 2))
would return `1`!
* Many socalled "scripting" languages begin their lives with dynamic
scoping. The main reason, as we've seen, is that implementing it is
extremely simple (no, *nobody* does substitution in the real world!
(Well, *almost* nobody...)).
Another reason is that these problems make life impossible if you want
to use functions as object like you do in Racket, so you notice them
very fast  but in a `normal` language without firstclass
functions, problems are not as obvious.
* For example, bash has `local` variables, but they have dynamic scope:
x="the global x"
print_x() { echo "The current value of x is \"$x\""; }
foo() { local x="x from foo"; print_x; }
print_x; foo; print_x
Perl began its life with dynamic scope for variables that are declared
`local`:
$x="the global x";
sub print_x { print "The current value of x is \"$x\"\n"; }
sub foo { local($x); $x="x from foo"; print_x; }
print_x; foo; print_x;
When faced with this problem, "the Perl way" was, obviously, not to
remove or fix features, but to pile them up  so `local` *still*
behaves in this way, and now there is a `my` declaration which
achieves proper lexical scope (and every serious Perl programmer knows
that you should always use `my`)...
There are other examples of languages that changed, and languages that
want to change (e.g, nobody likes dynamic scope in Emacs Lisp, but
there's just too much code now).
* This is still a tricky issue, like any other issue with bindings. For
example, googling got me quickly to [a Python blog post] which is
confused about what "dynamic scoping" is... It claims that Python uses
dynamic scope (Search for "Python uses dynamic as opposed to lexical
scoping"), yet python always used lexical scope rules, as can be seen
by translating their code to Racket (ignore sideeffects in this
computation):
(define (orangejuice)
(* x 2))
(define x 3)
(define y (orangejuice)) ; y is now 6
(define x 1)
(define y (orangejuice)) ; y is now 2
or by trying this in Python:
def orange_juice():
return x*2
def foo(x):
return orange_juice()
foo(2)
The real problem of python (pre 2.1, and pre 2.2 without the funny
from __future__ import nested_scope
line) is that it didn't create closures, which we will talk about
shortly.
[a Python blog post]:
https://folk.idi.ntnu.no/mlh/hetland_org/writing/instantpython.html
* Another example, which is an indicator of how easy it is to mess up
your scope is the following Ruby bug  running in `irb`:
% irb
irb(main):001:0> x = 0
=> 0
irb(main):002:0> lambda{x x}.call(5)
=> 5
irb(main):003:0> x
=> 5
(This is a bug due to weird scoping rules for variables, which was
fixed in newer versions of Ruby. See [this Ruby rant] for details, or
read about [Ruby and the principle of unwelcome surprise] for
additional gems (the latter is gone, so you'll need the [web
archive](https://archive.org/) to read it).)
[this Ruby rant]:
http://innig.net/software/ruby/closuresinruby
[Ruby and the principle of unwelcome surprise]:
http://ceaude.twoticketsplease.de/articles/rubyandtheprincipleofunwelcomesurprise.html
* Another thing to consider is the fact that compilation is something
that you do based only on the lexical structure of programs, since
compilers never actually run code. This means that dynamic scope makes
compilation close to impossible.
* There are some advantages for dynamic scope too. Two notable ones are:
 Dynamic scope makes it easy to have a "configuration variable"
easily change for the extent of a calling piece of code (this is
used extensively in Emacs, for example). The thing is that usually
we want to control which variables are "configurable" in this way,
statically scoped languages like Racket often choose a separate
facility for these. To rephrase the problem of dynamic scoping, it's
that *all* variables are modifiable.
The same can be said about functions: it is sometimes desirable to
change a function dynamically (for example, see "Aspect Oriented
Programming"), but if there is no control and all functions can
change, we get a world where no code can every be reliable.
 It makes recursion immediately available  for example,
{with {f {fun {x} {call f x}}}
{call f 0}}
is an infinite loop with a dynamically scoped language. But in a
lexically scoped language we will need to do some more work to get
recursion going.

# Implementing Lexical Scope: Closures and Environments
So how do we fix this?
Lets go back to the root of the problem: the new evaluator does not
behave in the same way as the substituting evaluator. In the old
evaluator, it was easy to see how functions can behave as objects that
remember values. For example, when we do this:
{with {x 1}
{fun {y}
{+ x y}}}
the result was a function value, which actually was the syntax object
for this:
{fun {y} {+ 1 y}}
Now if we call this function from someplace else like:
{with {f {with {x 1} {fun {y} {+ x y}}}}
{with {x 2}
{call f 3}}}
it is clear what the result will be: f is bound to a function that adds
1 to its input, so in the above the later binding for `x` has no effect
at all.
But with the caching evaluator, the value of
{with {x 1}
{fun {y}
{+ x y}}}
is simply:
{fun {y} {+ x y}}
and there is no place where we save the 1  *that's* the root of our
problem. (That's also what makes people suspect that using `lambda` in
Racket and any other functional language involves some inefficient
coderecompiling magic.) In fact, we can verify that by inspecting the
returned value, and see that it does contain a free identifier.
Clearly, we need to create an object that contains the body and the
argument list, like the function syntax object  but we don't do any
substitution, so in addition to the body an argument name(s) we need to
remember that we still need to substitute `x` by `1` . This means that
the pieces of information we need to know are:
 formal argument(s): {y}
 body: {+ x y}
 pending substitutions: [1/x]
and that last bit has the missing `1`. The resulting object is called a
`closure` because it closes the function body over the substitutions
that are still pending (its environment).
So, the first change is in the value of functions which now need all
these pieces, unlike the `Fun` case for the syntax object.
A second place that needs changing is the when functions are called.
When we're done evaluating the `call` arguments (the function value and
the argument value) but before we apply the function we have two
*values*  there is no more use for the current substitution cache at
this point: we have finished dealing with all substitutions that were
necessary over the current expression  we now continue with
evaluating the body of the function, with the new substitutions for the
formal arguments and actual values given. But the body itself is the
same one we had before  which is the previous body with its suspended
substitutions that we *still* did not do.
Rewrite the evaluation rules  all are the same except for evaluating
a `fun` form and a `call` form:
eval(N,sc) = N
eval({+ E1 E2},sc) = eval(E1,sc) + eval(E2,sc)
eval({ E1 E2},sc) = eval(E1,sc)  eval(E2,sc)
eval({* E1 E2},sc) = eval(E1,sc) * eval(E2,sc)
eval({/ E1 E2},sc) = eval(E1,sc) / eval(E2,sc)
eval(x,sc) = lookup(x,sc)
eval({with {x E1} E2},sc) = eval(E2,extend(x,eval(E1,sc),sc))
eval({fun {x} E},sc) = <{fun {x} E}, sc>
eval({call E1 E2},sc1)
= eval(Ef,extend(x,eval(E2,sc1),sc2))
if eval(E1,sc1) = <{fun {x} Ef}, sc2>
= error! otherwise
As a side note, these substitution caches are a little more than "just a
cache" now  they actually hold an *environment* of substitutions in
which expression should be evaluated. So we will switch to the common
*environment* name now.:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2, env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
In case you find this easier to follow, the "flat algorithm" for
evaluating a `call` is:
1. f := evaluate E1 in env1
2. if f is not a <{fun ...},...> closure then error!
3. x := evaluate E2 in env1
4. new_env := extend env_of(f) by mapping arg_of(f) to x
5. evaluate (and return) body_of(f) in new_env
Note how the scoping rules that are implied by this definition match the
scoping rules that were implied by the substitutionbased rules. (It
should be possible to prove that they are the same.)
The changes to the code are almost trivial, except that we need a way to
represent `<{fun {x} Ef}, env>` pairs.

The implication of this change is that we now cannot use the same type
for function syntax and function values since function values have more
than just syntax. There is a simple solution to this  we never do any
substitutions now, so we don't need to translate values into expressions
 we can come up with a new type for values, separate from the type of
abstract syntax trees.
When we do this, we will also fix our hack of using FLANG as the type of
values: this was merely a convenience since the AST type had cases for
all kinds of values that we needed. (In fact, you should have noticed
that Racket does this too: numbers, strings, booleans, etc are all used
by both programs and syntax representation (sexpressions)  but note
that function values are *not* used in syntax.) We will now implement a
separate `VAL` type for runtime values.
First, we need now a type for such environments  we can use `Listof`
for this:
;; a type for environments:
(definetype ENV = (Listof (List Symbol VAL)))
but we can just as well define a new type for environment values:
(definetype ENV
[EmptyEnv]
[Extend Symbol VAL ENV])
Reimplementing `lookup` is now simple:
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id val restenv)
(if (eq? id name) val (lookup name restenv))]))
... we don't need `extend` because we get `Extend` from the type
definition, and we also get `(EmptyEnv)` instead of `emptysubst`.
We now use this with the new type for values  two variants of these:
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV]) ; argname, body, scope
And now the new implementation of `eval` which uses the new type and
implements lexical scope:
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
We also need to update `arithop` to use `VAL` objects. The full code
follows  it now passes all tests, including the example that we used
to find the problem.
;;; <<>>
;; The Flang interpreter, using environments
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2,env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype ENV
[EmptyEnv]
[Extend Symbol VAL ENV])
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id val restenv)
(if (eq? id name) val (lookup name restenv))]))
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Fixing an Overlooked Bug
Incidentally, this version fixes a bug we had previously in the
substitution version of FLANG:
(run "{with {f {fun {y} {+ x y}}}
{with {x 7}
{call f 1}}}")
This bug was due to our naive `subst`, which doesn't avoid capturing
renames. But note that since that version of the evaluator makes its way
from the outside in, there is no difference in semantics for *valid*
programs  ones that don't have free identifiers.
(Reminder: This was *not* a dynamically scoped language, just a bug that
happened when `x` wasn't substituted away before `f` was replaced with
something that refers to `x`.)

# Lexical Scope using Racket Closures
> [PLAI §11] (without the last part about recursion)
An alternative representation for an environment.
We've already seen how firstclass functions can be used to implement
"objects" that contain some information. We can use the same idea to
represent an environment. The basic intuition is  an environment is a
*mapping* (a function) between an identifier and some value. For
example, we can represent the environment that maps `'a` to `1` and `'b`
to `2` (using just numbers for simplicity) using this function:
(: mymap : Symbol > Number)
(define (mymap id)
(cond [(eq? 'a id) 1]
[(eq? 'b id) 2]
[else (error ...)]))
An empty mapping that is implemented in this way has the same type:
(: emptymapping : Symbol > Number)
(define (emptymapping id)
(error ...))
We can use this idea to implement our environments: we only need to
define three things  `EmptyEnv`, `Extend`, and `lookup`. If we manage
to keep the contract to these functions intact, we will be able to
simply plug it into the same evaluator code with no other changes. It
will also be more convenient to define `ENV` as the appropriate function
type for use in the `VAL` type definition instead of using the actual
type:
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
Now we get to `EmptyEnv`  this is expected to be a function that
expects no arguments and creates an empty environment, one that behaves
like the `emptymapping` function defined above. We could define it like
this (changing the `emptymapping` type to return a `VAL`):
(define (EmptyEnv) emptymapping)
but we can skip the need for an extra definition and simply return an
empty mapping function:
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error ...)))
(The unRackety name is to avoid replacing previous code that used the
`EmptyEnv` name for the constructor that was created by the type
definition.)
The next thing we tackle is `lookup`. The previous definition that was
used is:
(: lookup : Symbol ENV > VAL)
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id val restenv)
(if (eq? id name) val (lookup name restenv))]))
How should it be modified now? Easy  an environment is a mapping: a
Racket function that will do the searching job itself. We don't need to
modify the contract since we're still using `ENV`, except a different
implementation for it. The new definition is:
(: lookup : Symbol ENV > VAL)
(define (lookup name env)
(env name))
Note that `lookup` does almost nothing  it simply delegates the real
work to the `env` argument. This is a good hint for the error message
that empty mappings should throw 
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
Finally, `Extend`  this was previously created by the variant case of
the ENV type definition:
[Extend Symbol VAL ENV]
keeping the same type that is implied by this variant means that the new
`Extend` should look like this:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
...)
The question is  how do we extend a given environment? Well, first,
we know that the result should be mapping  a `symbol > VAL` function
that expects an identifier to look for:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
...))
Next, we know that in the generated mapping, if we look for `id` then
the result should be `val`:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
...)))
If the `name` that we're looking for is not the same as `id`, then we
need to search through the previous environment:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(lookup name restenv))))
But we know what `lookup` does  it simply delegates back to the
mapping function (which is our `rest` argument), so we can take a direct
route instead:
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name)))) ; same as (lookup name restenv)
To see how all this works, try out extending an empty environment a few
times and examine the result. For example, the environment that we began
with:
(define (mymap id)
(cond [(eq? 'a id) 1]
[(eq? 'b id) 2]
[else (error ...)]))
behaves in the same way (if the type of values is numbers) as
(Extend 'a 1 (Extend 'b 2 (EmptyEnv)))
The new code is now the same, except for the environment code:
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }

 { fun { } }
 { call }
Evaluation rules:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2,env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# More Closures (on both levels)
Racket closures (= functions) can be used in other places too, and as we
have seen, they can do more than encapsulate various values  they can
also hold the behavior that is expected of these values.
To demonstrate this we will deal with closures in our language. We
currently use a variant that holds the three pieces of relevant
information:
[FunV Symbol FLANG ENV]
We can replace this by a functional object, which will hold the three
values. First, change the `VAL` type to hold functions for `FunV`
values:
(definetype VAL
[NumV Number]
[FunV (? > ?)])
And note that the function should somehow encapsulate the same
information that was there previously, the question is *how* this
information is going to be done, and this will determine the actual
type. This information plays a role in two places in our evaluator 
generating a closure in the `Fun` case, and using it in the `Call` case:
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody ;***
(Extend boundid ;***
(eval argexpr env) ;***
fenv))] ;***
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]
we can simply fold the marked functionality bit of `Call` into a Racket
function that will be stored in a `FunV` object  this piece of
functionality takes an argument value, extends the closure's environment
with its value and the function's name, and continues to evaluate the
function body. Folding all of this into a function gives us:
(lambda (argval)
(eval boundbody (Extend boundid argval env)))
where the values of `boundbody`, `boundid`, and `val` are known at the
time that the `FunV` is *constructed*. Doing this gives us the following
code for the two cases:
[(Fun boundid boundbody)
(FunV (lambda (argval)
(eval boundbody (Extend boundid argval env))))]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV proc) (proc (eval argexpr env))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]
And now the type of the function is clear:
(definetype VAL
[NumV Number]
[FunV (VAL > VAL)])
And again, the rest of the code is unmodified:
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype VAL
[NumV Number]
[FunV (VAL > VAL)])
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV (lambda (argval)
(eval boundbody (Extend boundid argval env))))]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV proc) (proc (eval argexpr env))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Types of Evaluators
What we did just now is implement lexical environments and closures in
the language we implement using lexical environments and closures in our
own language (Racket)!
This is another example of embedding a feature of the host language in
the implemented language, an issue that we have already discussed.
There are many examples of this, even when the two languages involved
are different. For example, if we have this bit in the C implementation
of Racket:
// Disclaimer: not real Racket code
Racket_Object *eval_and(int argc, Racket_Object *argv[]) {
Racket_Object *tmp;
if ( argc != 2 )
signal_racket_error("bad number of arguments");
else if ( racket_eval(argv[0]) != racket_false &&
(tmp = racket_eval(argv[1])) != racket_false )
return tmp;
else
return racket_false;
}
then the special semantics of evaluating a Racket `and` form is being
inherited from C's special treatment of `&&`. You can see this by the
fact that if there is a bug in the C compiler, then it will propagate to
the resulting Racket implementation too. A different solution is to not
use `&&` at all:
// Disclaimer: not real Racket code
Racket_Object *eval_and(int argc, Racket_Object *argv[]) {
Racket_Object *tmp;
if ( argc != 2 )
signal_racket_error("bad number of arguments");
else if ( racket_eval(argv[0]) != racket_false )
return racket_eval(argv[1]);
else
return racket_false;
}
and we can say that this is even better since it evaluates the second
expression in tail position. But in this case we don't really get that
benefit, since C itself is not doing tailcall optimization as a
standard feature (though some compilers do so under some circumstances).
We have seen a few different implementations of evaluators that are
quite different in flavor. They suggest the following taxonomy.
* A ___syntactic evaluator___ is one that uses its own language to
represent expressions and semantic runtime values of the evaluated
language, implementing all the corresponding behavior explicitly.
* A ___meta evaluator___ is an evaluator that uses language features of
its own language to directly implement behavior of the evaluated
language.
While our substitutionbased FLANG evaluator was close to being a
syntactic evaluator, we haven't written any purely syntactic evaluators
so far: we still relied on things like Racket arithmetics etc. The most
recent evaluator that we have studied, is even more of a *meta*
evaluator than the preceding ones: it doesn't even implement closures
and lexical scope, and instead, it uses the fact that Racket itself has
them.
With a good match between the evaluated language and the implementation
language, writing a meta evaluator can be very easy. With a bad match,
though, it can be very hard. With a syntactic evaluator, implementing
each semantic feature will be somewhat hard, but in return you don't
have to worry as much about how well the implementation and the
evaluated languages match up. In particular, if there is a particularly
strong mismatch between the implementation and the evaluated language,
it may take less effort to write a syntactic evaluator than a meta
evaluator. As an exercise, we can build upon our latest evaluator to
remove the encapsulation of the evaluator's response in the VAL type.
The resulting evaluator is shown below. This is a true meta evaluator:
it uses Racket closures to implement FLANG closures, Racket function
application for FLANG function application, Racket numbers for FLANG
numbers, and Racket arithmetic for FLANG arithmetic. In fact, ignoring
some small syntactic differences between Racket and FLANG, this latest
evaluator can be classified as something more specific than a meta
evaluator:
* A ___metacircular evaluator___ is a meta evaluator in which the
implementation and the evaluated languages are the same.
(Put differently, the trivial nature of the evaluator clues us in to the
deep connection between the two languages, whatever their syntactic
differences may be.)

# Feature Embedding
We saw that the difference between lazy evaluation and eager evaluation
is in the evaluation rules for `with` forms, function applications, etc:
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
is eager, and
eval({with {x E1} E2}) = eval(E2[E1/x])
is lazy. But is the first rule *really* eager? The fact is that the only
thing that makes it eager is the fact that our understanding of the
mathematical notation is eager  if we were to take math as lazy, then
the description of the rule becomes a description of lazy evaluation.
Another way to look at this is  take the piece of code that
implements this evaluation:
(: eval : FLANG > Number)
;; evaluates FLANG expressions by reducing them to numbers
(define (eval expr)
(cases expr
...
[(With boundid namedexpr boundbody)
(eval (subst boundbody
boundid
(Num (eval namedexpr))))]
...))
and the same question applies: is this really implementing eager
evaluation? We know that this is indeed eager  we can simply try it
and check that it is, but it is only eager because we are using an eager
language for the implementation! If our own language was lazy, then the
evaluator's implementation would run lazily, which means that the above
applications of the `eval` and the `subst` functions would also be lazy,
making our evaluator lazy as well.
This is a general phenomena where some of the semantic features of the
language we use (math in the formal description, Racket in our code)
gets *embedded* into the language we implement.
Here's another example  consider the code that implements
arithmetics:
(: eval : FLANG > Number)
;; evaluates FLANG expressions by reducing them to numbers
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
...))
what if it was written like this:
FLANG eval(FLANG expr) {
if (is_Num(expr))
return num_of_Num(expr);
else if (is_Add(expr))
return eval(lhs_of_Add(expr)) + eval(rhs_of_Add(expr));
else if ...
...
}
Would it still implement unlimited integers and exact fractions? That
depends on the language that was used to implement it: the above syntax
suggests C, C++, Java, or some other relative, which usually come with
limited integers and no exact fractions. But this really depends on the
language  even our own code has unlimited integers and exact
rationals only because Racket has them. If we were using a language that
didn't have such features (there are such Scheme implementations), then
our implemented language would absorb these (lack of) features too, and
its own numbers would be limited in just the same way. (And this
includes the syntax for numbers, which we embedded intentionally, like
the syntax for identifiers).
The bottom line is that we should be aware of such issues, and be very
careful when we talk about semantics. Even the language that we use to
communicate (semiformal logic) can mean different things.

Aside: read "Reflections on Trusting Trust" by Ken Thompson (You can
skip to the "Stage II" part to get to the interesting stuff.)
(And when you're done, look for "XcodeGhost" to see a relevant example,
and don't miss the leaked document on the wikipedia page...)

Here is yet another variation of our evaluator that is even closer to a
metacircular evaluator. It uses Racket values directly to implement
values, so arithmetic operations become straightforward. Note especially
how the case for function application is similar to arithmetics: a FLANG
function application translates to a Racket function application. In
both cases (applications and arithmetics) we don't even check the
objects since they are simple Racket objects  if our language happens
to have some meaning for arithmetics with functions, or for applying
numbers, then we will inherit the same semantics in our language. This
means that we now specify less behavior and fall back more often on what
Racket does.
We use Racket values with this type definition:
(definetype VAL = (U Number (VAL > VAL)))
And the evaluation function can now be:
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) n] ;*** return the actual number
[(Add l r) (+ (eval l env) (eval r env))]
[(Sub l r) ( (eval l env) (eval r env))]
[(Mul l r) (* (eval l env) (eval r env))]
[(Div l r) (/ (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(lambda ([argval : VAL]) ;*** return the racket function
;; note that this requires input type specifications since
;; typed racket can't guess the right one
(eval boundbody (Extend boundid argval env)))]
[(Call funexpr argexpr)
((eval funexpr env) ;*** trivial like the arithmetics!
(eval argexpr env))]))
Note how the arithmetics implementation is simple  it's a direct
translation of the FLANG syntax to Racket operations, and since we don't
check the inputs to the Racket operations, we let Racket throw type
errors for us. Note also how function application is just like the
arithmetic operations: a FLANG application is directly translated to a
Racket application.
However, this does not work quite as simply in Typed Racket. The whole
point of typechecking is that we never run into type errors  so we
cannot throw back on Racket errors since code that might produce them is
forbidden! A way around this is to perform explicit checks that
guarantee that Racket cannot run into type errors. We do this with the
following two helpers that are defined inside `eval`:
(: evalN : FLANG > Number)
(define (evalN e)
(let ([n (eval e env)])
(if (number? n)
n
(error 'eval "got a nonnumber: ~s" n))))
(: evalF : FLANG > (VAL > VAL))
(define (evalF e)
(let ([f (eval e env)])
(if (function? f)
f
(error 'eval "got a nonfunction: ~s" f))))
Note that Typed Racket is "smart enough" to figure out that in `evalF`
the result of the recursive evaluation has to be either `Number` or
`(VAL > VAL)`; and since the `if` throws out on numbers, we're left
with `(VAL > VAL)` functions, not just any function.
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
;; Values are plain Racket values, no new VAL wrapper;
;; (but note that this is a recursive definition)
(definetype VAL = (U Number (VAL > VAL)))
;; Define a type for functional environments
(definetype ENV = (Symbol > VAL))
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(: evalN : FLANG > Number)
(define (evalN e)
(let ([n (eval e env)])
(if (number? n)
n
(error 'eval "got a nonnumber: ~s" n))))
(: evalF : FLANG > (VAL > VAL))
(define (evalF e)
(let ([f (eval e env)])
(if (function? f)
f
(error 'eval "got a nonfunction: ~s" f))))
(cases expr
[(Num n) n]
[(Add l r) (+ (evalN l) (evalN r))]
[(Sub l r) ( (evalN l) (evalN r))]
[(Mul l r) (* (evalN l) (evalN r))]
[(Div l r) (/ (evalN l) (evalN r))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(lambda ([argval : VAL])
(eval boundbody (Extend boundid argval env)))]
[(Call funexpr argexpr)
((evalF funexpr)
(eval argexpr env))]))
(: run : String > VAL) ; no need to convert VALs to numbers
;; evaluate a FLANG program contained in a string
(define (run str)
(eval (parse str) (EmptyEnv)))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Recursion, Recursion, Recursion
> [PLAI §9]
There is one major feature that is still missing from our language: we
have no way to perform recursion (therefore no kind of loops). So far,
we could only use recursion when we had *names*. In FLANG, the only way
we can have names is through `with` which not good enough for recursion.
To discuss the issue of recursion, we switch to a "broken" version of
(untyped) Racket  one where a `define` has a different scoping rules:
the scope of the defined name does *not* cover the defined expression.
Specifically, in this language, this doesn't work:
#lang pl broken
(define (fact n)
(if (zero? n) 1 (* n (fact ( n 1)))))
(fact 5)
In our language, this translation would also not work (assuming we have
`if` etc):
{with {fact {fun {n}
{if {= n 0} 1 {* n {call fact { n 1}}}}}}
{call fact 5}}
And similarly, in plain Racket this won't work if `let` is the only tool
you use to create bindings:
(let ([fact (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))])
(fact 5))
In the brokenscope language, the `define` form is more similar to a
mathematical definition. For example, when we write:
(define (F x) x)
(define (G y) (F y))
(G F)
it is actually shorthand for
(define F (lambda (x) x))
(define G (lambda (y) (F y)))
(G F)
we can then replace defined names with their definitions:
(define F (lambda (x) x))
(define G (lambda (y) (F y)))
((lambda (y) (F y)) (lambda (x) x))
and this can go on, until we get to the actual code that we wrote:
((lambda (y) ((lambda (x) x) y)) (lambda (x) x))
This means that the above `fact` definition is similar to writing:
fact := (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))
(fact 5)
which is not a wellformed definition  it is *meaningless* (this is a
formal use of the word "meaningless"). What we'd really want, is to take
the *equation* (using `=` instead of `:=`)
fact = (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))
and find a solution which will be a value for `fact` that makes this
true.
If you look at the Racket evaluation rules handout on the web page, you
will see that this problem is related to the way that we introduced the
Racket `define`: there is a handwavy explanation that talks about
*knowing* things.
The big question is: can we define recursive functions without Racket's
magical `define` form?
> Note: This question is a little different than the question of
> implementing recursion in our language  in the Racket case we have
> no control over the implementation of the language. As it will
> eventually turn out, implementing recursion in our own language will
> be quite easy when we use mutation in a specific way. So the question
> that we're now facing can be phrased as either "can we get recursion
> in Racket without Racket's magical definition forms?" or "can we get
> recursion in our interpreter without mutation?".

# Recursion without the Magic
> [PLAI §22.4] (we go much deeper)
> Note: This explanation is similar to the one you can find in "The Why
> of Y", by Richard Gabriel.
To implement recursion without the `define` magic, we first make an
observation: this problem does *not* come up in a dynamicallyscoped
language. Consider the `let`version of the problem:
#lang pl dynamic
(let ([fact (lambda (n)
(if (zero? n) 1 (* n (fact ( n 1)))))])
(fact 5))
This works fine  because by the time we get to evaluate the body of
the function, `fact` is already bound to itself in the current dynamic
scope. (This is another reason why dynamic scope is perceived as a
convenient approach in new languages.)
Regardless, the problem that we have with lexical scope is still there,
but the way things work in a dynamic scope suggest a solution that we
can use now. Just like in the dynamic scope case, when `fact` is called,
it does have a value  the only problem is that this value is
inaccessible in the lexical scope of its body.
Instead of trying to get the value in via lexical scope, we can imitate
what happens in the dynamically scoped language by passing the `fact`
value to itself so it can call itself (going back to the original code
in the brokenscope language):
(define (fact self n) ;***
(if (zero? n) 1 (* n (self ( n 1)))))
(fact fact 5) ;***
except that now the recursive call should still send itself along:
(define (fact self n)
(if (zero? n) 1 (* n (self self ( n 1))))) ;***
(fact fact 5)
The problem is that this required rewriting calls to `fact`  both
outside and recursive calls inside. To make this an acceptable solution,
calls from both places should not change. Eventually, we should be able
to get a working `fact` definition that uses just
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))
The first step in resolving this problem is to curry the `fact`
definition.
(define (fact self) ;***
(lambda (n) ;***
(if (zero? n)
1
(* n ((self self) ( n 1)))))) ;***
((fact fact) 5) ;***
Now `fact` is no longer our factorial function  it's a function that
constructs it. So call it `makefact`, and bind `fact` to the actual
factorial function.
(define (makefact self) ;***
(lambda (n)
(if (zero? n) 1 (* n ((self self) ( n 1))))))
(define fact (makefact makefact)) ;***
(fact 5) ;***
We can try to do the same thing in the body of the factorial function:
instead of calling `(self self)`, just bind `fact` to it:
(define (makefact self)
(lambda (n)
(let ([fact (self self)]) ;***
(if (zero? n)
1
(* n (fact ( n 1))))))) ;***
(define fact (makefact makefact))
(fact 5)
This works fine, but if we consider our original goal, we need to get
that local `fact` binding outside of the `(lambda (n) ...)`  so we're
left with a definition that uses the factorial expression as is. So,
swap the two lines:
(define (makefact self)
(let ([fact (self self)]) ;***
(lambda (n) ;***
(if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact (makefact makefact))
(fact 5)
But the problem is that this gets us into an infinite loop because we're
trying to evaluate `(self self)` too ea(ge)rly. In fact, if we ignore
the body of the `let` and other details, we basically do this:
(define (makefact self) (self self)) (makefact makefact)
reducesugar>
(define makefact (lambda (self) (self self))) (makefact makefact)
replacedefinition>
((lambda (self) (self self)) (lambda (self) (self self)))
renameidentifiers>
((lambda (x) (x x)) (lambda (x) (x x)))
And this expression has an interesting property: it reduces to itself,
so evaluating it gets stuck in an infinite loop.
So how do we solve this? Well, we know that `(self self)` *should* be
the same value that is the factorial function itself  so it must be a
oneargument function. If it's such a function, we can use a value that
is equivalent, except that it will not get evaluated until it is needed,
when the function is called. The trick here is the observation that
`(lambda (n) (add1 n))` is really the same as `add1` (provided that
`add1` is a oneargument function), except that the `add1` part doesn't
get evaluated until the function is called. Applying this trick to our
code produces a version that does not get stuck in the same infinite
loop:
(define (makefact self)
(let ([fact (lambda (n) ((self self) n))]) ;***
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact (makefact makefact))
(fact 5)
Continuing from here  we know that
(let ([x v]) e) is the same as ((lambda (x) e) v)
(remember how we derived `fun` from a `with`), so we can turn that `let`
into the equivalent function application form:
(define (makefact self)
((lambda (fact) ;***
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))
(lambda (n) ((self self) n)))) ;***
(define fact (makefact makefact))
(fact 5)
And note now that the (lambda (fact) ...) expression is everything that
we need for a recursive definition of `fact`  it has the proper
factorial body with a plain recursive call. It's almost like the usual
value that we'd want to define `fact` as, except that we still have to
abstract on the recursive value itself. So lets move this code into a
separate definition for `factstep`:
(define factstep ;***
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makefact self)
(factstep ;***
(lambda (n) ((self self) n))))
(define fact (makefact makefact))
(fact 5)
We can now proceed by moving the `(makefact makefact)` self
application into its own function which is what creates the real
factorial:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makefact self)
(factstep
(lambda (n) ((self self) n))))
(define (makerealfact) (makefact makefact)) ;***
(define fact (makerealfact)) ;***
(fact 5)
Rewrite the `makefact` definition using an explicit `lambda`:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define makefact ;***
(lambda (self) ;***
(factstep
(lambda (n) ((self self) n)))))
(define (makerealfact) (makefact makefact))
(define fact (makerealfact))
(fact 5)
and fold the functionality of `makefact` and `makerealfact` into a
single `makefact` function by just using the value of `makefact`
explicitly instead of through a definition:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerealfact)
(let ([make (lambda (self) ;***
(factstep ;***
(lambda (n) ((self self) n))))]) ;***
(make make)))
(define fact (makerealfact))
(fact 5)
We can now observe that `makerealfact` has nothing that is specific to
factorial  we can make it take a "core function" as an argument:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerealfact core) ;***
(let ([make (lambda (self)
(core ;***
(lambda (n) ((self self) n))))])
(make make)))
(define fact (makerealfact factstep)) ;***
(fact 5)
and call it `makerecursive`:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerecursive core) ;***
(let ([make (lambda (self)
(core
(lambda (n) ((self self) n))))])
(make make)))
(define fact (makerecursive factstep)) ;***
(fact 5)
We're almost done now  there's no real need for a separate
`factstep` definition, just use the value for the definition of `fact`:
(define (makerecursive core)
(let ([make (lambda (self)
(core
(lambda (n) ((self self) n))))])
(make make)))
(define fact
(makerecursive
(lambda (fact) ;***
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))) ;***
(fact 5)
turn the `let` into a function form:
(define (makerecursive core)
((lambda (make) (make make)) ;***
(lambda (self) ;***
(core (lambda (n) ((self self) n)))))) ;***
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
do some renamings to make things simpler  `make` and `self` turn to
`x`, and `core` to `f`:
(define (makerecursive f) ;***
((lambda (x) (x x)) ;***
(lambda (x) (f (lambda (n) ((x x) n)))))) ;***
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
or we can manually expand that first (lambda (x) (x x)) application to
make the symmetry more obvious (not really surprising because it started
with a `let` whose purpose was to do a selfapplication):
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n)))) ;***
(lambda (x) (f (lambda (n) ((x x) n)))))) ;***
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
And we finally got what we were looking for: a general way to define
*any* recursive function without any magical `define` tricks. This also
work for other recursive functions:
#lang pl broken
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n))))
(lambda (x) (f (lambda (n) ((x x) n))))))
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
(fact 5)
(define fib
(makerecursive
(lambda (fib)
(lambda (n) (if (<= n 1) n (+ (fib ( n 1)) (fib ( n 2))))))))
(fib 8)
(define length
(makerecursive
(lambda (length)
(lambda (l) (if (null? l) 0 (+ (length (rest l)) 1))))))
(length '(x y z))
A convenient tool that people often use on paper is to perform a kind of
a syntactic abstraction: "assume that whenever I write (twice foo) I
really meant to write (foo foo)". This can often be done as plain
abstractions (that is, using functions), but in some cases  for
example, if we want to abstract over definitions  we just want such a
rewrite rule. (More on this towards the end of the course.) The
brokenscope language does provide such a tool  `rewrite` extends the
language with a rewrite rule. Using this, and our `makerecursive`, we
can make up a recursive definition form:
(rewrite (define/rec (f x) E)
=> (define f (makerecursive (lambda (f) (lambda (x) E)))))
In other words, we've created our own "magical definition" form. The
above code can now be written in almost the same way it is written in
plain Racket:
#lang pl broken
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n))))
(lambda (x) (f (lambda (n) ((x x) n))))))
(rewrite (define/rec (f x) E)
=> (define f (makerecursive (lambda (f) (lambda (x) E)))))
;; examples
(define/rec (fact n) (if (zero? n) 1 (* n (fact ( n 1)))))
(fact 5)
(define/rec (fib n) (if (<= n 1) n (+ (fib ( n 1)) (fib ( n 2)))))
(fib 8)
(define/rec (length l) (if (null? l) 0 (+ (length (rest l)) 1)))
(length '(x y z))
Finally, note that makerecursive is limited to 1argument functions
only because of the protection from eager evaluation. In any case, it
can be used in any way you want, for example,
(makerecursive (lambda (f) (lambda (x) f)))
is a function that *returns itself* rather than calling itself. Using
the rewrite rule, this would be:
(define/rec (f x) f)
which is the same as:
(define (f x) f)
in plain Racket.

# The Core of `makerecursive`
As in Racket, being able to express recursive functions is a fundamental
property of the language. It means that we can have loops in our
language, and that's the essence of making a language powerful enough to
be TMequivalent  able to express undecidable problems, where we
don't know whether there is an answer or not.
The core of what makes this possible is the expression that we have seen
in our derivation:
((lambda (x) (x x)) (lambda (x) (x x)))
which reduces to itself, and therefore has no value: trying to evaluate
it gets stuck in an infinite loop. (This expression is often called
"Omega".)
This is the key for creating a loop  we use it to make recursion
possible. Looking at our final `makerecursive` definition and ignoring
for a moment the "protection" that we need against being stuck
prematurely in an infinite loop:
(define (makerecursive f)
((lambda (x) (x x)) (lambda (x) (f (x x)))))
we can see that this is almost the same as the Omega expression  the
only difference is that application of `f`. Indeed, this expression (the
result of (makerecursive F) for some `F`) reduces in a similar way to
Omega:
((lambda (x) (x x)) (lambda (x) (F (x x))))
((lambda (x) (F (x x))) (lambda (x) (F (x x))))
(F ((lambda (x) (F (x x))) (lambda (x) (F (x x)))))
(F (F ((lambda (x) (F (x x))) (lambda (x) (F (x x))))))
(F (F (F ((lambda (x) (F (x x))) (lambda (x) (F (x x)))))))
...
which means that the actual value of this expression is:
(F (F (F ...forever...)))
This definition would be sufficient if we had a lazy language, but to
get things working in a strict one we need to bring back the protection.
This makes things a little different  if we use `(protect f)` to be a
shorthand for the protection trick,
(rewrite (protect f) => (lambda (x) (f x)))
then we have:
(define (makerecursive f)
((lambda (x) (x x)) (lambda (x) (f (protect (x x))))))
which makes the (makerecursive F) evaluation reduce to
(F (protect (F (protect (F (protect (...forever...)))))))
and this is still the same result (as long as `F` is a singleargument
function).
(Note that `protect` cannot be implemented as a plain function!)

# Denotational Explanation of Recursion
> Note: This explanation is similar to the one you can find in "The
> Little Schemer" called "(Y Y) Works!", by Dan Friedman and Matthias
> Felleisen.
The explanation that we have now for how to derive the `makerecursive`
definition is fine  after all, we did manage to get it working. But
this explanation was done from a kind of an operational point of view:
we knew a certain trick that can make things work and we pushed things
around until we got it working like we wanted. Instead of doing this, we
can reapproach the problem from a more declarative point of view.
So, start again from the same broken code that we had (using the
brokenscope language):
(define fact
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))
This is as broken as it was when we started: the occurrence of `fact` in
the body of the function is free, which means that this code is
meaningless. To avoid the compilation error that we get when we run this
code, we can substitute *anything* for that `fact`  it's even better
to use a replacement that will lead to a runtime error:
(define fact
(lambda (n) (if (zero? n) 1 (* n (777 ( n 1)))))) ;***
This function will not work in a similar way to the original one  but
there is one case where it *does* work: when the input value is `0`
(since then we do not reach the bogus application). We note this by
calling this function `fact0`:
(define fact0 ;***
(lambda (n) (if (zero? n) 1 (* n (777 ( n 1))))))
Now that we have this function defined, we can use it to write `fact1`
which is the factorial function for arguments of `0` or `1`:
(define fact0
(lambda (n) (if (zero? n) 1 (* n (777 ( n 1))))))
(define fact1
(lambda (n) (if (zero? n) 1 (* n (fact0 ( n 1))))))
And remember that this is actually just shorthand for:
(define fact1
(lambda (n)
(if (zero? n)
1
(* n ((lambda (n)
(if (zero? n)
1
(* n (777 ( n 1)))))
( n 1))))))
We can continue in this way and write `fact2` that will work for n<=2:
(define fact2
(lambda (n) (if (zero? n) 1 (* n (fact1 ( n 1))))))
or, in full form:
(define fact2
(lambda (n)
(if (zero? n)
1
(* n ((lambda (n)
(if (zero? n)
1
(* n ((lambda (n)
(if (zero? n)
1
(* n (777 ( n 1)))))
( n 1)))))
( n 1))))))
If we continue this way, we *will* get the true factorial function, but
the problem is that to handle *any* possible integer argument, it will
have to be an infinite definition! Here is what it is supposed to look
like:
(define fact0 (lambda (n) (if (zero? n) 1 (* n (777 ( n 1))))))
(define fact1 (lambda (n) (if (zero? n) 1 (* n (fact0 ( n 1))))))
(define fact2 (lambda (n) (if (zero? n) 1 (* n (fact1 ( n 1))))))
(define fact3 (lambda (n) (if (zero? n) 1 (* n (fact2 ( n 1))))))
...
The true factorial function is `factinfinity`, with an infinite size.
So, we're back at the original problem...
To help make things more concise, we can observe the repeated pattern in
the above, and extract a function that abstracts this pattern. This
function is the same as the `factstep` that we have seen previously:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact0 (factstep 777))
(define fact1 (factstep fact0))
(define fact2 (factstep fact1))
(define fact3 (factstep fact2))
...
which is actually:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact0 (factstep 777))
(define fact1 (factstep (factstep 777)))
(define fact2 (factstep (factstep (factstep 777))))
...
(define fact
(factstep (factstep (factstep (... (factstep 777) ...)))))
Do this a little differently  rewrite `fact0` as:
(define fact0
((lambda (mk) (mk 777))
factstep))
Similarly, `fact1` is written as:
(define fact1
((lambda (mk) (mk (mk 777)))
factstep))
and so on, until the real factorial, which is still infinite at this
stage:
(define fact
((lambda (mk) (mk (mk (... (mk 777) ...))))
factstep))
Now, look at that `(lambda (mk) ...)`  it is an infinite expression,
but for every actual application of the resulting factorial function we
only need a finite number of `mk` applications. We can guess how many,
and as soon as we hit an application of `777` we know that our guess is
too small. So instead of `777`, we can try to use the maker function to
create and use the next.
To make things more explicit, here is the expression that is our
`fact0`, without the definition form:
((lambda (mk) (mk 777))
factstep)
This function has a very low guess  it works for 0, but with 1 it
will run into the `777` application. At this point, we want to somehow
invoke `mk` again to get the next level  and since `777` *does* get
applied, we can just replace it with `mk`:
((lambda (mk) (mk mk))
factstep)
The resulting function works just the same for an input of `0` because
it does not attempt a recursive call  but if we give it `1`, then
instead of running into the error of applying `777`:
(* n (777 ( n 1)))
we get to apply `factstep` there:
(* n (factstep ( n 1)))
and this is still wrong, because `factstep` expects a function as an
input. To see what happens more clearly, write `factstep` explicitly:
((lambda (mk) (mk mk))
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
The problem is in what we're going to pass into `factstep`  its
`fact` argument will not be the factorial function, but the `mk`
function constructor. Renaming the `fact` argument as `mk` will make
this more obvious (but not change the meaning):
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n (mk ( n 1)))))))
It should now be obvious that this application of `mk` will not work,
instead, we need to apply it on some function and *then* apply the
result on `( n 1)`. To get what we had before, we can use `777` as a
bogus function:
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n ((mk 777) ( n 1)))))))
This will allow one recursive call  so the definition works for both
inputs of `0` and `1`  but not more. But that `777` is used as a
maker function now, so instead, we can just use `mk` itself again:
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n ((mk mk) ( n 1)))))))
And this is a *working* version of the real factorial function, so make
it into a (nonmagical) definition:
(define fact
((lambda (mk) (mk mk))
(lambda (mk)
(lambda (n) (if (zero? n) 1 (* n ((mk mk) ( n 1))))))))
But we're not done  we "broke" into the factorial code to insert that
`(mk mk)` application  that's why we dragged in the actual value of
`factstep`. We now need to fix this. The expression on that last line
(lambda (n) (if (zero? n) 1 (* n ((mk mk) ( n 1)))))
is close enough  it is `(factstep (mk mk))`. So we can now try to
rewrite our `fact` as:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact
((lambda (mk) (mk mk))
(lambda (mk) (factstep (mk mk)))))
... and would fail in a familiar way! If it's not familiar enough, just
rename all those `mk`s as `x`s:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact
((lambda (x) (x x))
(lambda (x) (factstep (x x)))))
We've run into the eagerness of our language again, as we did before.
The solution is the same  the `(x x)` is the factorial function, so
protect it as we did before, and we have a working version:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define fact
((lambda (x) (x x))
(lambda (x) (factstep (lambda (n) ((x x) n))))))
The rest should not be surprising now... Abstract the recursive making
bit in a new `makerecursive` function:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
(define (makerecursive f)
((lambda (x) (x x))
(lambda (x) (f (lambda (n) ((x x) n))))))
(define fact (makerecursive factstep))
and now we can do the first reduction inside `makerecursive` and write
the `factstep` expression explicitly:
#lang pl broken
(define (makerecursive f)
((lambda (x) (f (lambda (n) ((x x) n))))
(lambda (x) (f (lambda (n) ((x x) n))))))
(define fact
(makerecursive
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1))))))))
and this is the same code we had before.

# The Y Combinator
Our `makerecursive` function is usually called the *fixpoint operator*
or the *Y combinator*.
It looks really simple when using the lazy version (remember: our
version is the eager one):
(define Y
(lambda (f)
((lambda (x) (f (x x)))
(lambda (x) (f (x x))))))
> Note that if we *do* allow a recursive definition for Y itself, then
> the definition can follow the definition that we've seen:
>
> (define (Y f) (f (Y f)))
And this all comes from the loop generated by:
((lambda (x) (x x)) (lambda (x) (x x)))
This expression, which is also called *Omega* (the `(lambda (x) (x x))`
part by itself is usually called *omega* and then `(omega omega)` is
*Omega*), is also the idea behind many deep mathematical facts. As an
example for what it does, follow the next rule:
I will say the next sentence twice:
"I will say the next sentence twice".
(Note the usage of colon for the first and quotes for the second 
what is the equivalent of that in the lambda expression?)
By itself, this just gets you stuck in an infinite loop, as Omega does,
and the Y combinator adds `F` to that to get an infinite chain of
applications  which is similar to:
I will say the next sentence twice:
"I will hop on one foot and then say the next sentence twice".
Sidenote:
[see this SO question](https://stackoverflow.com/q/25228394/128595)
and my answer, which came from the PLQ implementation.

## The main property of Y
`factstep` is a function that given any limited factorial, will
generate a factorial that is good for one more integer input. Start with
`777`, which is a factorial that is good for nothing (because it's not a
function), and you can get `fact0` as
fact0 == (factstep 777)
and that's a good factorial function only for an input of `0`. Use that
with `factstep` again, and you get
fact1 == (factstep fact0) == (factstep (factstep 777))
which is the factorial function when you only look at input values of
`0` or `1`. In a similar way
fact2 == (factstep fact1)
is good for `0`...`2`  and we can continue as much as we want, except
that we need to have an infinite number of applications  in the
general case, we have:
factn == (factstep (factstep (factstep ... 777)))
which is good for `0`...`n`. The *real* factorial would be the result of
running `factstep` on itself infinitely, it *is* `factinfinity`. In
other words (here `fact` is the *real* factorial):
fact = factinfinity == (factstep (factstep ...infinitely...))
but note that since this is really infinity, then
fact = (factstep (factstep ...infinitely...))
= (factstep fact)
so we get an equation:
fact = (factstep fact)
and a solution for this is going to be the real factorial. The solution
is the *fixedpoint* of the `factstep` function, in the same sense that
`0` is the fixed point of the `sin` function because
0 = (sin 0)
And the Y combinator does just that  it has this property:
(makerecursive f) = (f (makerecursive f))
or, using the more common name:
(Y f) = (f (Y f))
This property encapsulates the real magical power of Y. You can see how
it works: since `(Y f) = (f (Y f))`, we can add an `f` application to
both sides, giving us `(f (Y f)) = (f (f (Y f)))`, so we get:
(Y f) = (f (Y f)) = (f (f (Y f))) = (f (f (f (Y f)))) = ...
= (f (f (f ...)))
and we can conclude that
(Y factstep) = (factstep (factstep ...infinitely...))
= fact

# Yet another explanation for Y
Here's another explanation of how the Y combinator works. Remember that
our `factstep` function was actually a function that generates a
factorial function based on some input, which is supposed to be the
factorial function:
(define factstep
(lambda (fact)
(lambda (n) (if (zero? n) 1 (* n (fact ( n 1)))))))
As we've seen, you can apply this function on a version of factorial
that is good for inputs up to some n, and the result will be a factorial
that is good for those values up to n+1. The question is *what is the
fixpoint of `factstep`*? And the answer is that if it maps factₙ
factorial to factₙ₊₁, then the input will be equal to the output on the
*infinitieth* `fact`, which is the *actual* factorial. Since Y is a
fixpoint combinator, it gives us exactly that answer:
(define therealfactorial (Y factstep))

# Typing the Y Combinator
Typing the Y combinator is a tricky issue. For example, in standard ML
you must write a new type definition to do this:
datatype 'a t = T of 'a t > 'a
val y = fn f => (fn (T x) => (f (fn a => x (T x) a)))
(T (fn (T x) => (f (fn a => x (T x) a))))
> Can you find a pattern in the places where `T` is used?
>  Roughly speaking, that type definition is
>
> ;; `t' is the type name, `T' is the constructor (aka the variant)
> (definetype (RecTypeOf t)
> [T ((RecTypeOf t) > t)])
>
> First note that the two `fn a => ...` parts are the same as our
> protection, so ignoring that we get:
>
> val y = fn f => (fn (T x) => (f (x (T x))))
> (T (fn (T x) => (f (x (T x)))))
>
> if you now replace `T` with `Quote`, things make more sense:
>
> val y = fn f => (fn (Quote x) => (f (x (Quote x))))
> (Quote (fn (Quote x) => (f (x (Quote x)))))
>
> and with our syntax, this would be:
>
> (define (Y f)
> ((lambda (qx)
> (cases qx
> [(Quote x) (f (x qx))]))
> (Quote
> (lambda (qx)
> (cases qx
> [(Quote x) (f (x qx))])))))
>
> it's not really quotation  but the analogy should help: it uses
> `Quote` to distinguish functions as values that are applied (the `x`s)
> from functions that are passed as arguments.
In OCaml, this looks a little different:
# type 'a t = T of ('a t > 'a) ;;
type 'a t = T of ('a t > 'a)
# let y f = (fun (T x) > x (T x))
(T (fun (T x) > fun z > f (x (T x)) z)) ;;
val y : (('a > 'b) > 'a > 'b) > 'a > 'b =
# let fact = y (fun fact n > if n < 1 then 1 else n * fact(n1)) ;;
val fact : int > int =
# fact 5 ;;
 : int = 120
but OCaml has also a `rectypes` command line argument, which will make
it infer the type by itself:
# let y f = (fun x > x x) (fun x > fun z > f (x x) z) ;;
val y : (('a > 'b) > 'a > 'b) > 'a > 'b =
# let fact = y (fun fact n > if n < 1 then 1 else n * fact(n1)) ;;
val fact : int > int =
# fact 5 ;;
 : int = 120
The translation of this to `#lang pl` is a little verbose because we
don't have autocurrying, and because we need to declare input types to
functions, but it's essentially a direct translation of the above:
(definetype (RecTypeOf t)
[T ((RecTypeOf t) > t)])
(: Y : (All (A B) ((A > B) > (A > B)) > (A > B)))
(define (Y f)
((lambda ([x : (RecTypeOf (A > B))])
(cases x
[(T x) (x (T x))]))
(T (lambda ([x : (RecTypeOf (A > B))])
(cases x
[(T x) (lambda ([z : A])
((f (x (T x))) z))])))))
(define fact
(Y (lambda ([fact : (Integer > Integer)])
(lambda ([n : Integer])
(if (< n 1) 1 (* n (fact (sub1 n))))))))
(fact 5)
It is also possible to write this expression in "plain" Typed Racket,
without a userdefined type  and we need to start with a proper type
definition. First of all, the type of Y should be straightforward: it is
a fixpoint operation, so it takes a `T > T` function and produces its
fixpoint. The fixpoint itself is some `T` (such that applying the
function on it results in itself). So this gives us:
(: makerecursive : (T > T) > T)
However, in our case `makerecursive` computes a *functional* fixpoint,
for unary `S > T` functions, so we should narrow down the type
(: makerecursive : ((S > T) > (S > T)) > (S > T))
Now, in the body of `makerecursive` we need to add a type for the `x`
argument which is behaving in a weird way: it is used both as a function
and as its own argument. (Remember  I will say the next sentence
twice: "I will say the next sentence twice".) We need a recursive type
definition helper (not a new type) for that:
(definetype (Tau S T) = (Rec this (this > (S > T))))
This type is tailored for our use of `x`: it is a type for a function
that will *consume itself* (hence the `Rec`) and spit out the value that
the `f` argument consumes  an `S > T` function.
The resulting full version of the code:
(: makerecursive : (All (S T) ((S > T) > (S > T)) > (S > T)))
(definetype (Tau S T) = (Rec this (this > (S > T))))
(define (makerecursive f)
((lambda ([x : (Tau S T)]) (f (lambda (z) ((x x) z))))
(lambda ([x : (Tau S T)]) (f (lambda (z) ((x x) z))))))
(: fact : Number > Number)
(define fact (makerecursive
(lambda ([fact : (Number > Number)])
(lambda ([n : Number])
(if (zero? n)
1
(* n (fact ( n 1))))))))
(fact 5)

# Lambda Calculus  Schlac
> [PLAI §22] (we do much more)
We know that many constructs that are usually thought of as primitives
are not really needed  we can implement them ourselves given enough
tools. The question is how far can we go?
The answer: as far as we want. For example:
(define foo((lambda(f)((lambda(x)(x x))(lambda(x)(f(x x)))))(lambda(
f)(lambda(x)(((x(lambda(x)(lambda(x y)y))(lambda(x y)x))(x(lambda(x)
(lambda(x y)y))(lambda(x y)x))(((x(lambda (p)(lambda(s)(s(p(lambda(x
y)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(s) (s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x))(lambda(x)(lambda(x y)y))(lambda
(x y)x)))(lambda(f x)(f x))((f((x(lambda(p)(lambda(s)(s(p(lambda(x y
)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(y s)(s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x)))(lambda(n)(lambda(f x)(f(n f x)
)))(f((((x(lambda(p)(lambda(s)(s(p (lambda(x y)y))(lambda(f x)(f((p(
lambda(x y)y))f x))))))(lambda(s)(s(lambda(f x) x)(lambda(f x)x))))(
lambda(x y)x))(lambda(p)(lambda(s)(s(p(lambda(x y)y))(lambda(f x)(f(
(p(lambda(x y)y))f x))))))(lambda(s)(s(lambda(f x)x)(lambda(f x)x)))
)(lambda(x y)x)))))))))
We begin with a very minimal language, which is based on the Lambda
Calculus. In this language we get a very minimal set of constructs and
values.
In DrRacket, this we will use the Schlac language level (stands for
"SchemeRacket as Lambda Calculus"). This language has a
Racketlike syntax, but don't be confused  it is *very* different
from Racket. The only constructs that are available in this language
are: lambda expressions of at least one argument, function application
(again, at least one argument), and simple definition forms which are
similar to the ones in the "Broken define" language  definitions are
used as shorthand, and cannot be used for recursive function definition.
They're also only allowed at the toplevel  no local helpers, and a
definition is not an expression that can appear anywhere. The BNF is
therefore:
::= ...
::=
 (define )
::=
 (lambda ( ...) )
 ( ...)
Since this language has no primitive values (other than functions),
Racket numbers and booleans are also considered identifiers, and have no
builtin value that come with the language. In addition, all functions
and function calls are curried, so
(lambda (x y z) (z y x))
is actually shorthand for
(lambda (x) (lambda (y) (lambda (z) ((z y) x))))
The rules for evaluation are simple, there is one very important rule
for evaluation which is called "beta reduction":
((lambda (x) E1) E2) > E1[E2/x]
where substitution in this context requires being careful so you won't
capture names. This requires you to be able to do another kind of
transformation which is called "alpha conversion", which basically says
that you can rename identifiers as long as you keep the same binding
structure (eg, a valid renaming does not change the deBruijn form of
the expression). There is one more rule that can be used, *eta
conversion* which says that `(lambda (x) (f x))` is the same as `f` (we
used this rule above when deriving the Y combinator).
One last difference between Schlac and Racket is that Schlac is a *lazy*
language. This will be important since we do not have any builtin
special forms like `if`.
Here is a Schlac definition for the identity function:
(define identity (lambda (x) x))
and there is not much that we can do with this now:
> identity
#
> (identity identity)
#
> (identity identity identity)
#
(In the last expression, note that `(id id id)` is shorthand for `((id
id) id)`, and since `(id id)` is the identity, applying that on `id`
returns it again.)

# Church Numerals
So far, it seems like it is impossible to do anything useful in this
language, since all we have are functions and applications. We know how
to write the identity function, but what about other values? For
example, can you write code that evaluates to zero?
> What's zero? I only know how to write functions!
>
> (Turing Machine programmer: "What's a function?
>  I only know how to write 0s and 1s!")
The first thing we therefore need is to be able to *encode* numbers as
functions. For zero, we will use a function of two arguments that simply
returns its second value:
(define 0 (lambda (f) (lambda (x) x)))
or, more concisely
(define 0 (lambda (f x) x))
This is the first step in an encoding that is known as *Church
Numerals*: an encoding of natural numbers as functions. The number zero
is encoded as a function that takes in a function and a second value,
and applies the function zero times on the argument (which is really
what the above definition is doing). Following this view, the number one
is going to be a function of two arguments, that applies the first on
the second one time:
(define 1 (lambda (f x) (f x)))
and note that `1` is just like the identity function (as long as you
give it a function as its first input, but this is always the case in
Schlac). The next number on the list is two  which applies the first
argument on the second one twice:
(define 2 (lambda (f x) (f (f x))))
We can go on doing this, but what we really want is a way to perform
arbitrary arithmetic. The first requirement for that is an `add1`
function that increments its input (an encoded natural number) by one.
To do this, we write a function that expects an encoded number:
(define add1 (lambda (n) ...))
and this function is expected to return an encoded number, which is
always a function of `f` and `x`:
(define add1 (lambda (n) (lambda (f x) ...)))
Now, in the body, we need to apply `f` on `x` n+1 times  but remember
that `n` is a function that will do `n` applications of its first
argument on its second:
(define add1 (lambda (n) (lambda (f x) ... (n f x) ...)))
and all we have left to do now is to apply `f` one more time, yielding
this definition for `add1`:
(define add1 (lambda (n) (lambda (f x) (f (n f x)))))
Using this, we can define a few useful numbers:
(define 1 (add1 0))
(define 2 (add1 1))
(define 3 (add1 2))
(define 4 (add1 3))
(define 5 (add1 4))
This is all nice theoretically, but how can we make sure that it is
correct? Well, Schlac has a few additional builtin functions that
translate Church numerals into Racket numbers. To try our definitions we
use the `>nat` (read: to natural number):
(>nat 0)
(>nat 5)
(>nat (add1 (add1 5)))
You can now verify that the identity function is really the same as the
number 1:
(>nat identity)
We can even write a test case, since Schlac contains the `test` special
form, but we have to be careful in that  first of all, we cannot test
whether functions are equal (why?) so we must use `>nat`, but
(test (>nat (add1 (add1 5))) => 7)
will not work since `7` is undefined. To overcome this, Schlac has a
`backdoor` for primitive Racket values  just use a quote:
(test (>nat (add1 (add1 5))) => '7)
We can now define natural number addition  one simple idea is to get
two encoded numbers `m` and `n`, then start with `x`, apply `f` on it
`n` times by using it as a function, then apply `f` `m` more times on
the result in the same way:
(define + (lambda (m n) (lambda (f x) (m f (n f x)))))
or equivalently:
(define + (lambda (m n f x) (m f (n f x))))
Another idea is to use `add1` and increment `n` by `m` using `add1`:
(define + (lambda (m n) (m add1 n)))
(>nat (+ 4 5))
We can also define multiplication of `m` and `n` quite easily  begin
with addition  `(lambda (x) (+ n x))` is a function that expects an
`x` and returns `(+ x n)`  it's an incrementbyn function. But since
all functions and applications are curried, this is actually the same as
`(lambda (x) ((+ n) x))` which is the same as `(+ n)`. Now, what we want
to do is repeat this operation `m` times over zero, which will add `n`
to zero `m` times, resulting in `m` * `n`. The definition is therefore:
(define * (lambda (m n) (m (+ n) 0)))
(>nat (* 4 5))
(>nat (+ 4 (* (+ 2 5) 5)))
An alternative approach is to consider
(lambda (x) (n f x))
for some encoded number `n` and a function `f`  this function is like
`f`^`n` (f composed n times with itself). But remember that this is
shorthand for
(lambda (x) ((n f) x))
and we know that `(lambda (x) (foo x))` is just like `foo` (if it is a
function), so this is equivalent to just
(n f)
So `(n f)` is `f`^`n`, and in the same way `(m g)` is `g`^`m`  if we
use `(n f)` for `g`, we get `(m (n f))` which is n selfcompositions of
`f`, selfcomposed m times. In other words, `(m (n f))` is a function
that is like `m`*`n` applications of `f`, so we can define
multiplication as:
(define * (lambda (m n) (lambda (f) (m (n f)))))
which is the same as
(define * (lambda (m n f) (m (n f))))
The same principle can be used to define exponentiation (but now we have
to be careful with the order since exponentiation is not commutative):
(define ^ (lambda (m n) (n (* m) 1)))
(>nat (^ 3 4))
And there is a similar alternative here too 
* a Church numeral `m` is the mselfcomposition function,
* and `(1 m)` is just like `m`^`1` which is the same as `m`
(`1`=`identity`)
* and `(2 m)` is just like `m`^`2`  it takes a function `f`, self
composes it `m` times, and self composes the result `m` times  for
a total of `f`^`(m*m)`
* and `(3 m)` is similarly `f`^`(m*m*m)`
* so `(n m)` is `f`^`(m^n)` (note that the first `^` is
selfcompositions, and the second one is a mathematical exponent)
* so `(n m)` is a function that returns `m`^`n` selfcompositions of an
input function,
Which means that `(n m)` is the Church numeral for `m`^`n`, so we get:
(define ^ (lambda (m n) (n m)))
which basically says that any number encoding `n` is also the `?`^`n`
operation.
All of this is was not too complicated  but all so far all we did is
write functions that increment their inputs in various ways. What about
`sub1`? For that, we need to do some more work  we will need to
encode booleans.

# More Encodings
Our choice of encoding numbers makes sense  the idea is that the main
feature of a natural number is repeating something a number of times.
For booleans, the main property we're looking for is choosing between
two values. So we can encode true and false by functions of two
arguments that return either the first or the second argument:
(define #t (lambda (x y) x))
(define #f (lambda (x y) y))
Note that this encoding of `#f` is really the same as the encoding of
`0`, so we have to know what type to expect an use the proper operations
(this is similar to C, where everything is just integers). Now that we
have these two, we can define `if`:
(define if (lambda (c t e) (c t e)))
it expects a boolean which is a function of two arguments, and passes it
the two expressions. The `#t` boolean will simply return the first, and
the `#f` boolean will return the second. Strictly speaking, we don't
really need this definition, since instead of writing `(if c t e)`, we
can simply write `(c t e)`. In any case, we need the language to be lazy
for this to work. To demonstrate this, we'll intentionally use the quote
backdoor to use a nonfunctional value, using this will normally result
in an error:
(+ '1 '2)
But testing our `if` definition, things work just fine:
(if #t (+ 4 5) (+ 1 2))
and we see that DrRacket leaves the second addition expression in red,
which indicates that it was not executed. We can also make sure that
even when it is defined as a function, it is still working fine because
the language is lazy:
(if #f ((lambda (x) (x x)) (lambda (x) (x x))) 3)
What about `and` and `or`? Simple, `or` takes two arguments, and returns
either true or false if one of the inputs is true:
(define or (lambda (a b) (if a #t (if b #t #f))))
but `(if b #t #f)` is really the same as just `b` because it must be a
boolean (we cannot use more than one "truty" or "falsy" values):
(define or (lambda (a b) (if a #t b)))
also, if `a` is true, we want to return `#t`, but that is exactly the
value of `a`, so:
(define or (lambda (a b) (if a a b)))
and finally, we can get rid of the `if` (which is actually breaking the
`if` abstraction, if we encode booleans in some other way):
(define or (lambda (a b) (a a b)))
Similarly, you can convince yourself that the definition of `and` is:
(define and (lambda (a b) (a b a)))
Schlac has toRacket conversion functions for booleans too:
(>bool (or #f #f))
(>bool (or #f #t))
(>bool (or #t #f))
(>bool (or #t #t))
and
(>bool (and #f #f))
(>bool (and #f #t))
(>bool (and #t #f))
(>bool (and #t #t))
A `not` function is quite simple  one alternative is to choose from
true and false in the usual way:
(define not (lambda (a) (a #f #t)))
and another is to return a function that switches the inputs to an input
boolean:
(define not (lambda (a) (lambda (x y) (a y x))))
which is the same as
(define not (lambda (a x y) (a y x)))
We can now put numbers and booleans together: we define a `zero?`
function.
(define zero? (lambda (n) (n (lambda (x) #f) #t)))
(test (>bool (and (zero? 0) (not (zero? 3)))) => '#t)
(Good question: is this fast?)
(Note that it is better to test that the value is explicitly `#t`, if we
just use `(test (>bool ...))` then the test will work even if the
expression in question evaluated to some bogus value.)
The idea is simple  if `n` is the encoding of zero, it will return
it's second argument which is `#t`:
(zero? 0) > ((lambda (f n) n) (lambda (x) #f) #t) > #t
if `n` is an encoding of a bigger number, then it is a selfcomposition,
and the function that we give it is one that always returns `#f`, no
matter how many times it is selfcomposed. Try `2` for example:
(zero? 2) > ((lambda (f n) (f (f n))) (lambda (x) #f) #t)
> ((lambda (x) #f) ((lambda (x) #f) #t))
> #f
Now, how about an encoding for compound values? A minimal approach is
what we use in Racket  a way to generate pairs (`cons`), and encode
lists as chains of pairs with a special value at the end (`null`). There
is a natural encoding for pairs that we have previously seen  a pair
is a function that expects a selector, and will apply that on the two
values:
(define cons (lambda (x y) (lambda (s) (s x y))))
Or, equivalently:
(define cons (lambda (x y s) (s x y)))
To extract the two values from a pair, we need to pass a selector that
consumes two values and returns one of them. In our framework, this is
exactly what the two boolean values do, so we get:
(define car (lambda (x) (x #t)))
(define cdr (lambda (x) (x #f)))
(>nat (+ (car (cons 2 3)) (cdr (cons 2 3))))
We can even do this:
(define 1st car)
(define 2nd (lambda (l) (car (cdr l))))
(define 3rd (lambda (l) (car (cdr (cdr l)))))
(define 4th (lambda (l) (car (cdr (cdr (cdr l))))))
(define 5th (lambda (l) (car (cdr (cdr (cdr (cdr l)))))))
or write a `listref` function:
(define listref (lambda (l n) (car (n cdr l))))
Note that we don't need a recursive function for this: our encoding of
natural numbers makes it easy to "iterate N times". What we get with
this encoding is essentially free naturalnumber recursion.
We now need a special `null` value to mark list ends. This value should
have the same number of arguments as a `cons` value (one: a
selector/boolean function), and it should be possible to distinguish it
from other values. We choose
(define null (lambda (s) #t))
Testing the list encoding:
(define l123 (cons 1 (cons 2 (cons 3 null))))
(>nat (2nd l123))
And as with natural numbers and booleans, Schlac has builtin facility
to convert encoded lists to Racket values, except that this requires
specifying the type of values in a list so it's a higherorder function:
((>listof >nat) l123)
which ("as usual") can be written as
(>listof >nat l123)
We can even do this:
(>listof (>listof >nat) (cons l123 (cons l123 null)))
Defining `null?` is now relatively easy (and it's actually already used
by the above `>listof` conversion). The following definition
(define null? (lambda (x) (x (lambda (x y) #f))))
works because if `x` is null, then it simply ignores its argument and
returns `#t`, and if it's a pair, then it uses the input selector, which
always returns `#f` in its turn. Using some arbitrary `A` and `B`:
(null? (cons A B))
> ((lambda (x) (x (lambda (x y) #f))) (lambda (s) (s A B)))
> ((lambda (s) (s A B)) (lambda (x y) #f))
> ((lambda (x y) #f) A B)
> #f
(null? null)
> ((lambda (x) (x (lambda (x y) #f))) (lambda (s) #t))
> ((lambda (s) #t) (lambda (x y) #f))
> #t
We can use the Y combinator to create recursive functions  we can
even use the rewrite rules facility that Schlac contains (the same one
that we have previously seen):
(define Y
(lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x))))))
(rewrite (define/rec f E) => (define f (Y (lambda (f) E))))
and using it:
(define/rec length
(lambda (l)
(if (null? l)
0
(add1 (length (cdr l))))))
(>nat (length l123))
And to complete this, um, journey  we're still missing subtraction.
There are many ways to solve the problem of subtraction, and for a
challenge try to come up with a solution yourself. One of the clearer
solutions uses a simple idea  begin with a pair of two zeroes
`<0,0>`, and repeat this transformation `n` times: `` > ``.
After `n` steps, we will have ``  so we get:
(define inccons (lambda (p) (cons (cdr p) (add1 (cdr p)))))
(define sub1 (lambda (n) (car (n inccons (cons 0 0)))))
(>nat (sub1 5))
And from this the road is short to general subtraction, `m``n` is
simply `n` applications of `sub1` on `m`:
(define  (lambda (m n) (n sub1 m)))
(test (>nat ( 3 2)) => '1)
(test (>nat ( (* 4 (* 5 5)) 5)) => '95)
We now have a normallooking language, and we're ready to do anything we
want. Here are two popular examples:
(define/rec fact
(lambda (x)
(if (zero? x) 1 (* x (fact (sub1 x))))))
(test (>nat (fact 5)) => '120)
(define/rec fib
(lambda (x)
(if (or (zero? x) (zero? (sub1 x)))
1
(+ (fib ( x 1)) (fib ( x 2))))))
(test (>nat (fib (* 5 2))) => '89)
To get generalized arithmetic capability, Schlac has yet another
builtin facility for translating Racket natural numbers into Church
numerals:
(>nat (fib (nat> '10)))
... and to get to that frightening expression in the beginning, all you
need to do is replace all definitions in the `fib` definition over and
over again until you're left with nothing but lambda expressions and
applications, then reformat the result into some cute shape. For extra
fun, you can look for immediate applications of lambda expressions and
reduce them manually.
All of this is in the following code:
;;; <<>>
;; Making Schlac into a practical language (not an interpreter)
#lang pl schlac
(define identity (lambda (x) x))
;; Natural numbers
(define 0 (lambda (f x) x))
(define add1 (lambda (n) (lambda (f x) (f (n f x)))))
;; same as:
;; (define add1 (lambda (n) (lambda (f x) (n f (f x)))))
(define 1 (add1 0))
(define 2 (add1 1))
(define 3 (add1 2))
(define 4 (add1 3))
(define 5 (add1 4))
(test (>nat (add1 (add1 5))) => '7)
(define + (lambda (m n) (m add1 n)))
(test (>nat (+ 4 5)) => '9)
;; (define * (lambda (m n) (m (+ n) 0)))
(define * (lambda (m n f) (m (n f))))
(test (>nat (* 4 5)) => '20)
(test (>nat (+ 4 (* (+ 2 5) 5))) => '39)
;; (define ^ (lambda (m n) (n (* m) 1)))
(define ^ (lambda (m n) (n m)))
(test (>nat (^ 3 4)) => '81)
;; Booleans
(define #t (lambda (x y) x))
(define #f (lambda (x y) y))
(define if (lambda (c t e) (c t e))) ; not really needed
(test (>nat (if #t 1 2)) => '1)
(test (>nat (if #t (+ 4 5) (+ '1 '2))) => '9)
(define and (lambda (a b) (a b a)))
(define or (lambda (a b) (a a b)))
;; (define not (lambda (a) (a #f #t)))
(define not (lambda (a x y) (a y x)))
(test (>bool (and #f #f)) => '#f)
(test (>bool (and #t #f)) => '#f)
(test (>bool (and #f #t)) => '#f)
(test (>bool (and #t #t)) => '#t)
(test (>bool (or #f #f)) => '#f)
(test (>bool (or #t #f)) => '#t)
(test (>bool (or #f #t)) => '#t)
(test (>bool (or #t #t)) => '#t)
(test (>bool (not #f)) => '#t)
(test (>bool (not #t)) => '#f)
(define zero? (lambda (n) (n (lambda (x) #f) #t)))
(test (>bool (and (zero? 0) (not (zero? 3)))) => '#t)
;; Lists
(define cons (lambda (x y s) (s x y)))
(define car (lambda (x) (x #t)))
(define cdr (lambda (x) (x #f)))
(test (>nat (+ (car (cons 2 3)) (cdr (cons 2 3)))) => '5)
(define 1st car)
(define 2nd (lambda (l) (car (cdr l))))
(define 3rd (lambda (l) (car (cdr (cdr l)))))
(define 4th (lambda (l) (car (cdr (cdr (cdr l))))))
(define 5th (lambda (l) (car (cdr (cdr (cdr (cdr l)))))))
(define null (lambda (s) #t))
(define null? (lambda (x) (x (lambda (x y) #f))))
(define l123 (cons 1 (cons 2 (cons 3 null))))
;; Note that `>listof' is a H.O. converter
(test ((>listof >nat) l123) => '(1 2 3))
(test (>listof >nat l123) => '(1 2 3)) ; same as the above
(test (>listof (>listof >nat) (cons l123 (cons l123 null)))
=> '((1 2 3) (1 2 3)))
;; Subtraction is tricky
(define inccons (lambda (p) (cons (cdr p) (add1 (cdr p)))))
(define sub1 (lambda (n) (car (n inccons (cons 0 0)))))
(test (>nat (sub1 5)) => '4)
(define  (lambda (a b) (b sub1 a)))
(test (>nat ( 3 2)) => '1)
(test (>nat ( (* 4 (* 5 5)) 5)) => '95)
(test (>nat ( 2 4)) => '0) ; this is "natural subtraction"
;; Recursive functions
(define Y
(lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x))))))
(rewrite (define/rec f E) => (define f (Y (lambda (f) E))))
(define/rec length
(lambda (l)
(if (null? l)
0
(add1 (length (cdr l))))))
(test (>nat (length l123)) => '3)
(define/rec fact
(lambda (x)
(if (zero? x) 1 (* x (fact (sub1 x))))))
(test (>nat (fact 5)) => '120)
(define/rec fib
(lambda (x)
(if (or (zero? x) (zero? (sub1 x)))
1
(+ (fib (sub1 x)) (fib (sub1 (sub1 x)))))))
(test (>nat (fib (* 5 2))) => '89)
#
;; Fullyexpanded Fibonacci
(define fib
((lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x)))))
(lambda (f)
(lambda (x)
((lambda (c t e) (c t e))
((lambda (a b) (a a b))
((lambda (n)
(n (lambda (x) (lambda (x y) y)) (lambda (x y) x)))
x)
((lambda (n)
(n (lambda (x) (lambda (x y) y)) (lambda (x y) x)))
((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
x)))
((lambda (n) (lambda (f x) (f (n f x)))) (lambda (f x) x))
((lambda (x y)
(x (lambda (n) (lambda (f x) (f (n f x)))) y))
(f ((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
x))
(f ((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
((lambda (n)
((lambda (x) (x (lambda (x y) x)))
(n (lambda (p)
((lambda (x y s) (s x y))
((lambda (x) (x (lambda (x y) y))) p)
((lambda (n) (lambda (f x) (f (n f x))))
((lambda (x) (x (lambda (x y) y))) p))))
((lambda (x y s) (s x y))
(lambda (f x) x)
(lambda (f x) x)))))
x)))))))))
;; The same after reducing all immediate function applications
(define fib
((lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x)))))
(lambda (f)
(lambda (x)
(((x (lambda (x) (lambda (x y) y)) (lambda (x y) x))
(x (lambda (x) (lambda (x y) y)) (lambda (x y) x))
(((x (lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x))
(lambda (x) (lambda (x y) y))
(lambda (x y) x)))
(lambda (f x) (f x))
((f ((x (lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (y s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x)))
(lambda (n) (lambda (f x) (f (n f x))))
(f ((((x (lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x))
(lambda (p)
(lambda (s)
(s (p (lambda (x y) y))
(lambda (f x)
(f ((p (lambda (x y) y)) f x))))))
(lambda (s)
(s (lambda (f x) x) (lambda (f x) x))))
(lambda (x y) x)))))))))
;; Cute reformatting of the above:
(define fib((lambda(f)((lambda(x)(x x))(lambda(x)(f(x x)))))(lambda(
f)(lambda(x)(((x(lambda(x)(lambda(x y)y))(lambda(x y)x))(x(lambda(x)
(lambda(x y)y))(lambda(x y) x))(((x(lambda(p)(lambda(s)(s(p(lambda(x
y)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(s) (s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x))(lambda(x)(lambda(x y)y))(lambda
(x y)x)))(lambda(f x)(f x))((f((x(lambda(p)(lambda(s)(s(p(lambda(x y
)y))(lambda(f x)(f((p(lambda(x y)y))f x))))))(lambda(y s)(s(lambda(f
x)x)(lambda(f x)x))))(lambda(x y)x)))(lambda(n)(lambda(f x)(f(n f x)
)))(f((((x(lambda(p)(lambda(s)(s(p (lambda(x y)y))(lambda(f x)(f((p(
lambda(x y) y))f x))))))(lambda(s)(s(lambda(f x)x)(lambda(f x)x))))(
;; ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
;; `(cons 0 0)'
lambda(x y)x))(lambda(p)(lambda(s)(s(p(lambda(x y)y))(lambda(f x)(f(
(p(lambda(x y)y))f x))))))(lambda(s)(s(lambda(f x)x)(lambda(f x)x)))
)(lambda(x y)x)))))))))
;; And for extra fun:
(λ(f)(λ
(x)(((x(λ(
x)(λ(x y)y)
)(λ(x y)x))(
x(λ(x)(λ(x y)
y))(λ(x y
)x))(((
x(λ(p)(
λ(s)(s
(p (λ(
x y)y))
(λ(f x
)(f((p(
λ(x y)
y))f x
))))))(
λ(s)(s(
λ(f x)x)
(λ(f x)x)
)))(λ(x y)
x))(λ(x)(λ(
x y)y)) (λ(
x y) x)))(λ(
f x)(f x))((f
((x(λ(p )(λ (s
)(s(p( λ(x y)
y))(λ ( f x)(f(
(p (λ( x y)y)
)f x))) )))(λ(
y s)(s (λ (f x
)x)(λ( f x)x)
)))(λ( x y)x))
)(λ(n) (λ (f
x)(f (n f x)))
)(f((( (x(λ(p)
(λ(s)(s (p( λ(
x y )y ))(λ(f
x) (f(( p(λ(x y
)y)) f x)))))
)(λ(s)( s(λ(f x
)x)(λ( f x)x)
))) (λ (x y)x
))(λ(p )(λ(s)(
s(p(λ( x y)y)
)(λ (f x)(f((
p(λ (x y)y)) f
x)))))) (λ(s)(
s(λ (f x)x)(λ
(f x)x) )))(λ(
x y)x) ))))))
#

# Alternative Church Encoding
Finally, note that this is just one way to encode things  other
encodings are possible. One alternative encoding is in the following
code  it uses a list of `N` falses as the encoding for `N`. This
encoding makes it easier to `add1` (just `cons` another `#f`), and to
`sub1` (simply `cdr`). The tradeoff is that some arithmetics operations
becomes more complicated, for example, the definition of `+` requires
the fixpoint combinator. (As expected, some people want to see what can
we do with a language without recursion, so they don't like jumping to Y
too fast.)
;;; <<>>
;; An alternative "Church" encoding: use lists to encode numbers
#lang pl schlac
(define identity (lambda (x) x))
;; Booleans (same as before)
(define #t (lambda (x y) x))
(define #f (lambda (x y) y))
(define if (lambda (c t e) (c t e))) ; not really needed
(test (>bool (if #t #f #t))
=> '#f)
(test (>bool (if #f ((lambda (x) (x x)) (lambda (x) (x x))) #t))
=> '#t)
(define and (lambda (a b) (a b a)))
(define or (lambda (a b) (a a b)))
(define not (lambda (a x y) (a y x)))
(test (>bool (and #f #f)) => '#f)
(test (>bool (and #t #f)) => '#f)
(test (>bool (and #f #t)) => '#f)
(test (>bool (and #t #t)) => '#t)
(test (>bool (or #f #f)) => '#f)
(test (>bool (or #t #f)) => '#t)
(test (>bool (or #f #t)) => '#t)
(test (>bool (or #t #t)) => '#t)
(test (>bool (not #f)) => '#t)
(test (>bool (not #t)) => '#f)
;; Lists (same as before)
(define cons (lambda (x y s) (s x y)))
(define car (lambda (x) (x #t)))
(define cdr (lambda (x) (x #f)))
(define 1st car)
(define 2nd (lambda (l) (car (cdr l))))
(define 3rd (lambda (l) (car (cdr (cdr l)))))
(define 4th (lambda (l) (car (cdr (cdr (cdr l))))))
(define 5th (lambda (l) (car (cdr (cdr (cdr (cdr l)))))))
(define null (lambda (s) #t))
(define null? (lambda (x) (x (lambda (x y) #f))))
;; Natural numbers (alternate encoding)
(define 0 identity)
(define add1 (lambda (n) (cons #f n)))
(define zero? car) ; tricky
(define sub1 cdr) ; this becomes very simple
;; Note that we could have used something more straightforward:
;; (define 0 null)
;; (define add1 (lambda (n) (cons #t n))) ; cons anything
;; (define zero? null?)
;; (define sub1 (lambda (l) (if (zero? l) l (cdr l))))
(define 1 (add1 0))
(define 2 (add1 1))
(define 3 (add1 2))
(define 4 (add1 3))
(define 5 (add1 4))
(test (>nat* (add1 (add1 5))) => '7)
(test (>nat* (sub1 (sub1 (add1 (add1 5))))) => '5)
(test (>bool (and (zero? 0) (not (zero? 3)))) => '#t)
(test (>bool (zero? (sub1 (sub1 (sub1 3))))) => '#t)
;; listofnumbers tests
(define l123 (cons 1 (cons 2 (cons 3 null))))
(test (>listof >nat* l123) => '(1 2 3))
(test (>listof (>listof >nat*) (cons l123 (cons l123 null)))
=> '((1 2 3) (1 2 3)))
;; Recursive functions
(define Y
(lambda (f)
((lambda (x) (x x)) (lambda (x) (f (x x))))))
(rewrite (define/rec f E) => (define f (Y (lambda (f) E))))
;; note that this example is doing something silly now
(define/rec length
(lambda (l)
(if (null? l)
0
(add1 (length (cdr l))))))
(test (>nat* (length l123)) => '3)
;; addition becomes hard since it requires a recursive definition
;; (define/rec +
;; (lambda (m n) (if (zero? n) m (+ (add1 m) (sub1 n)))))
;; (test (>nat* (+ 4 5)) => '9)
;; faster alternative:
(define/rec +
(lambda (m n)
(if (zero? m) n
(if (zero? n) m
(add1 (add1 (+ (sub1 m) (sub1 n))))))))
(test (>nat* (+ 4 5)) => '9)
;; subtraction is similar to addition
;; (define/rec 
;; (lambda (m n) (if (zero? n) m ( (sub1 m) (sub1 n)))))
;; (test (>nat* ( (+ 4 5) 4)) => '5)
;; but this is not "natural subtraction": doesn't work when n>m,
;; because (sub1 0) does not return 0.
;; a solution is like alternative form of +:
(define/rec 
(lambda (m n)
(if (zero? m) 0
(if (zero? n) m
( (sub1 m) (sub1 n))))))
(test (>nat* ( (+ 4 5) 4)) => '5)
(test (>nat* ( 2 5)) => '0)
;; alternatively, could change sub1 above:
;; (define sub1 (lambda (n) (if (zero? n) n (cdr n))))
;; we can do multiplication in a similar way
(define/rec *
(lambda (m n)
(if (zero? m) 0
(+ n (* (sub1 m) n)))))
(test (>nat* (* 4 5)) => '20)
(test (>nat* (+ 4 (* (+ 2 5) 5))) => '39)
;; and the rest of the examples
(define/rec fact
(lambda (x)
(if (zero? x) 1 (* x (fact (sub1 x))))))
(test (>nat* (fact 5)) => '120)
(define/rec fib
(lambda (x)
(if (or (zero? x) (zero? (sub1 x)))
1
(+ (fib (sub1 x)) (fib (sub1 (sub1 x)))))))
(test (>nat* (fib (* 5 2))) => '89)
#
;; Fullyexpanded Fibonacci (note: much shorter than the previous
;; encoding, but see how Y appears twice  two "((lambda" pairs)
(define fib((lambda(f)((lambda(x)(x x))(lambda(x)(f(x x)))))(lambda(
f)(lambda(x)(((((x(lambda(x y)x))(x(lambda(x y)x)))((x(lambda(x y)y)
)(lambda(x y)x)))(lambda(s)(s(lambda(x y)y)(lambda(x)x))))((((lambda
(f)((lambda(x)(x x))(lambda(x)(f(x x))))) (lambda(f)(lambda(m n)((m(
lambda(x y)x))n (((n(lambda(x y)x)) m)(lambda(s)((s (lambda(x y)y))(
lambda(s)((s (lambda(x y)y))((f(m(lambda(x y)y)))(n(lambda(x y)y))))
))))))))(f(x(lambda(x y)y))))(f((x(lambda(x y)y))(lambda(x y)y))))))
)))
#

# Implementing `definetype` & `cases` in Schlac
Another interesting way to implement lists follows the pattern matching
approach, where both pairs and the null value are represented by a
function that serves as a kind of a `match` dispatcher. This function
takes in two inputs  if it is the representation of null then it will
return the first input, and if it is a pair, then it will apply the
second input on the two parts of the pair. This is implemented as
follows (with type comments to make it clear):
;; null : List
(define null
(lambda (n p)
n))
;; cons : A List > List
(define cons
(lambda (x y)
(lambda (n p)
(p x y))))
This might seem awkward, but it follows the intended use of pairs and
null as a matchlike construct. Here is an example, with the equivalent
Racket code on the side:
;; Sums up a list of numbers
(define (sum l)
(l ; (match l
0 ; ['() 0]
(lambda (x xs) ; [(cons x xs)
(+ x (sum xs))))) ; (+ x (sum xs))])
In fact, it's easy to implement our selectors and predicate using this:
(define null? (lambda (l) (l #t (lambda (x xs) #f))))
(define car (lambda (l) (l #f (lambda (x xs) x))))
(define cdr (lambda (l) (l #f (lambda (x xs) xs))))
;; in the above `#f' is really any value, since it
;; should be an error alternatively:
(define car (lambda (l)
(l ((lambda (x) (x x)) (lambda (x) (x x))) ; "error"
(lambda (x y) x))))
The same approach can be used to define any kind of new data type in a
way that looks like our own `definetype` definitions. For example,
consider a muchsimplified definition of the AE type we've seen early in
the semester, and a matching `eval` definition as an example for using
`cases`:
(definetype AE
[Num Number]
[Add AE AE])
(: eval : AE > Number)
(define (eval expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]))
We can follow the above approach now to write Schlac code that more than
being equivalent, is also very similar in nature. Note that the type
definition is replaced by two definitions for the two constructors:
(define Num (lambda (n) (lambda (num add) (num n ))))
(define Add (lambda (l r) (lambda (num add) (add l r))))
(define/rec eval
(lambda (expr) ; `expr` is always a (lambda (num add) ...), and it
; expects a unary `num` argument and a binary `add`
(expr (lambda (n) n)
(lambda (l r) (+ (eval l) (eval r))))))
(test (>nat (eval (Add (Num 1) (Num 2)))) => '3)
We can even take this further: the translations from `definetype` and
`cases` are mechanical enough that we could implement them almost
exactly via rewrites (there are a subtle change in that we're list field
names rather than types):
(rewrite (definetype ignoredType [Variant arg ...] ...)
=> (define Variant
(lambda (arg ...)
(lambda (Variant ...) (Variant arg ...))))
...)
(rewrite (cases value [(ignoredVariant arg ...) result] ...)
=> (value (lambda (arg ...) result) ...))
And using that, an evluator is simple:
(definetype AE [Num n] [Add l r] [Sub l r] [Mul l r])
(define/rec eval
(lambda (expr)
(cases expr
[(Num n) n]
[(Add l r) (+ (eval l) (eval r))]
[(Sub l r) ( (eval l) (eval r))]
[(Mul l r) (* (eval l) (eval r))])))
(test (>nat (eval (Mul (Add (Num 1) (Num 2))
(Sub (Num 4) (Num 2)))))
=> '6)

# Recursive Environments
> [PLAI §11.5]
What we really need for recursion, is a special kind of an environment,
one that can refer to itself. So instead of doing (note: `call`s removed
for readability):
{with {fact {fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}
{fact 5}}
which does not work for the usual reasons, we want to use some
{rec {fact {fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}
{fact 5}}
that will do the necessary magic.
One way to achieve this is using the Y combinator as we have seen  a
kind of a "constructor" for recursive functions. We can do that in a
similar way to the `rewrite` rule that we have seen in Schlac 
translate the above expression to:
{with {fact {makerec
{fun {fact}
{fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}}}
{fact 5}}
or even:
{with {fact {{fun {f} {{fun {x} {f {x x}}} {fun {x} {f {x x}}}}}
{fun {fact}
{fun {n}
{if {zero? n} 1 {* n {fact { n 1}}}}}}}}
{fact 5}}
Now, we will see how it can be used in *our* code to implement a
recursive environment.
If we look at what `with` does in
{with {fact {fun {n}
{if {zero? n} 1 {* n {call fact { n 1}}}}}}
{call fact 5}}
then we can say that to evaluate this expression, we evaluate the body
expression in an extended environment that contains `fact`, even if a
bogus one that is good for `0` only  the new environment is created
with something like this:
extend("fact", makefactclosure(), env)
so we can take this whole thing as an operation over `env`
addfact(env) := extend("fact", makefactclosure(), env)
This gives us the firstlevel fact. But `fact` itself is still undefined
in `env`, so it cannot call itself. We can try this:
addfact(addfact(env))
but that still doesn't work, and it will never work no matter how far we
go:
addfact(addfact(addfact(addfact(addfact(...env...)))))
What we really want is infinity: a place where addfact works and the
result is the same as what we've started with  we want to create a
"magical" environment that makes this possible:
let magicenv = ???
such that:
addfact(magicenv) = magicenv
which basically gives us the illusion of being at the infinity point.
This magicenv thing is exactly the *fixedpoint* of the `addfact`
operation. We can use:
magicenv = rec(addfact)
and following the main property of the Y combinator, we know that:
magicenv = rec(addfact) ; def. of magicenv
= addfact(rec(addfact)) ; Y(f) = f(Y(f))
= addfact(magicenv) ; def. of magicenv
What does all this mean? It means that if we have a fixedpoint operator
at the level of the implementation of our environments, then we can use
it to implement a recursive binder. In our case, this means that a
fixpoint in Racket can be used to implement a recursive language. But we
have that  Racket does have recursive functions, so we should be able
to use that to implement our recursive binder.
There are two ways that make it possible to write recursive functions in
Racket. One is to define a function, and use its name to do a recursive
call  using the Racket formal rules, we can see that we said that we
mark that we now *know* that a variable is bound to a value. This is
essentially a sideeffect  we modify what we know, which corresponds
to modifying the global environment. The second way is a new form:
`letrec`. This form is similar to `let`, except that the scope that is
established includes the named expressions  it is exactly what we
want `rec` to do. A third way is using recursive local definitions, but
that is equivalent to using `letrec`, more on this soon.

# Recursion: Racket's `letrec`
So we want to add recursion to our language, practically. We already
know that Racket makes it possible to write recursive functions, which
is possible because of the way it implements its "global environment":
our evaluator can only *extend* an environment, while Racket *modifies*
its global environment. This means that whenever a function is defined
in the global environment, the resulting closure will have it as its
environment "pointer", but the global environment was not extended 
it stays the same, and was just modified with one additional binding.
But Racket has another, a bit more organized way of using recursion:
there is a special localbinding construct that is similar to `let`, but
allows a function to refer to itself. It is called `letrec`:
(letrec ([fact (lambda (n)
(if (zero? n)
1
(* n (fact ( n 1)))))])
(fact 5))
Some people may remember that there was a third way for creating
recursive functions: using local definition in function bodies. For
example, we have seen things like:
(define (length list)
(define (helper list len)
(if (null? list)
len
(helper (rest list) (+ len 1))))
(helper list 0))
This looks like the same kind of environment magic that happens with a
global `define`  but actually, Racket defines the meaning of internal
definitions using `letrec`  so the above code is exactly the same as:
(define (length list)
(letrec ([helper (lambda (list len)
(if (null? list)
len
(helper (rest list) (+ len 1))))])
(helper list 0)))
The scoping rules for a `letrec` is that the scope of the bound name
covers both the body *and* the named expression. Furthermore, multiple
names can be bound to multiple expressions, and the scope of each name
covers all named expression as well as the body. This makes it easy to
define mutually recursive functions, such as:
(letrec ([even? (lambda (n) (if (zero? n) #t (odd? ( n 1))))]
[odd? (lambda (n) (if (zero? n) #f (even? ( n 1))))])
(even? 99))
But it is not a required functionality  it could be done with a
single recursive binding that contains several functions:
(letrec ([even+odd
(list (lambda (n)
(if (zero? n) #t ((second even+odd) ( n 1))))
(lambda (n)
(if (zero? n) #f ((first even+odd) ( n 1)))))])
((first even+odd) 99))
This is basically the same problem we face if we want to use the Y
combinator for mutually recursive bindings. The above solution is
inconvenient, but it can be improved using more `let`s to have easier
name access. For example:
(letrec ([even+odd
(list (lambda (n)
(let ([even? (first even+odd)]
[odd? (second even+odd)])
(if (zero? n) #t (odd? ( n 1)))))
(lambda (n)
(let ([even? (first even+odd)]
[odd? (second even+odd)])
(if (zero? n) #f (even? ( n 1))))))])
(let ([even? (first even+odd)]
[odd? (second even+odd)])
(even? 99)))

# Implementing Recursion using `letrec`
We will see how to add a similar construct to our language  for
simplicity, we will add a `rec` form that handles a single binding:
{rec {fact {fun {n}
{if {= 0 n}
1
{* n {fact { n 1}}}}}}
{fact 5}}
Using this, things can get a little tricky. What should we get if we do:
{rec {x x} x}
? Currently, it seems like there is no point in using any expression
except for a *function* expression in a `rec` expression, so we will
handle only these cases.
(BTW, under what circumstances would nonfunction values be useful in a
letrec?)

One way to achieve this is to use the same trick that we have recently
seen: instead of reimplementing language features, we can use existing
features in our own language, which hopefully has the right
functionality in a form that can be reused to in our evaluator.
Previously, we have seen a way to implement environments using Racket
closures:
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: lookup : Symbol ENV > VAL)
(define (lookup name env)
(env name))
(: Extend : Symbol VAL ENV > ENV)
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
We can use this implementation, and create circular environments using
Racket's `letrec`. The code for handling a `with` expressions is:
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
It looks like we should be able to handle `rec` in a similar way (the
AST constructor name is `WRec` ("withrec") so it doesn't collide with
TR's `Rec` constructor for recursive types):
[(WRec boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
but this won't work because the named expression is evaluated
prematurely, in the previous environment. Instead, we will move
everything that needs to be done, including evaluation, to a separate
`extendrec` function:
[(WRec boundid namedexpr boundbody)
(eval boundbody
(extendrec boundid namedexpr env))]
Now, the `extendrec` function needs to provide the new, "magically
circular" environment. Following what we know about the arguments to
`extendrec`, and the fact that it returns a new environment (= a lookup
function), we can sketch a rough definition:
(: extendrec : Symbol FLANG ENV > ENV) ; FLANG, not VAL!
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(lambda (name)
(if (eq? name id)
... something that uses expr to get a value ...
(restenv name))))
What should the missing expression be? It can simply evaluate the object
given itself:
(define (extendrec id expr restenv)
(lambda (name)
(if (eq? name id)
(eval expr ...this environment...)
(restenv name))))
But how do we get *this environment*, before it is defined? Well, the
environment is itself a Racket *function*, so we can use Racket's
`letrec` to make the function refer to itself recursively:
(define (extendrec id expr restenv)
(letrec ([recenv (lambda (name)
(if (eq? name id)
(eval expr recenv)
(restenv name)))])
recenv))
It's a little more convenient to use an internal definition, and add a
type for clarity:
(define (extendrec id expr restenv)
(: recenv : Symbol > VAL)
(define (recenv name)
(if (eq? name id)
(eval expr recenv)
(restenv name)))
recenv)
This works, but there are several problems:
1. First, we no longer do a simple lookup in the new environment.
Instead, we evaluate the expression on *every* such lookup. This
seems like a technical point, because we do not have sideeffects in
our language (also because we said that we want to handle only
function expressions). Still, it wastes space since each evaluation
will allocate a new closure.
2. Second, a related problem  what happens if we try to run this:
{rec {x x} x}
? Well, we do that stuff to extend the current environment, then
evaluate the body in the new environment, this body is a single
variable reference:
(eval (Id 'x) thenewenv)
so we look up the value:
(lookup 'x thenewenv)
which is:
(thenewenv 'x)
which goes into the function which implements this environment, there
we see that `name` is the same as `name1`, so we return:
(eval expr recenv)
but the `expr` here is the original namedexpression which is itself
`(Id 'x)`, and we're in an infinite loop.
We can try to get over these problems using another binding. Racket
allows several bindings in a single `letrec` expression or multiple
internal function definitions, so we change `extendrec` to use the
newlycreated environment:
(define (extendrec id expr restenv)
(: recenv : Symbol > VAL)
(define (recenv name)
(if (eq? name id)
val
(restenv name)))
(: val : VAL)
(define val (eval expr recenv))
recenv)
This runs into an interesting type error, which complains about possibly
getting some `Undefined` value. It does work if we switch to the untyped
language for now (using `#lang pl untyped`)  and it seems to run fine
too. But it raises more questions, beginning with: what is the meaning
of:
(letrec ([x ...]
[y ...x...])
...)
or equivalently, an internal block of
(define x ...)
(define y ...x...)
? Well, DrRacket seems to do the "right thing" in this case, but what
about:
(letrec ([y ...x...]
[x ...])
...)
? As a hint, see what happens when we now try to evaluate the
problematic
{rec {x x} x}
expression, and compare that with the result that you'd get from Racket.
This also clarifies the type error that we received.
It should be clear now why we want to restrict usage to just binding
recursive functions. There are no problems with such definitions because
when we evaluate a `fun` expression, there is no evaluation of the body,
which is the only place where there are potential references to the same
function that is defined  a function's body is *delayed*, and
executed only when the function is applied later.
But the biggest question that is still open: we just implemented a
circular environment using Racket's own circular environment
implementation, and that does not explain how they are actually
implemented. The cycle of pointers that we've implemented depends on the
cycle of pointers that Racket uses, and that is a black box we want to
open up.
For reference, the complete code is below.
#lang pl
#
The grammar:
::=
 { + }
 {  }
 { * }
 { / }
 { with { } }
 { rec { } }

 { fun { } }
 { call }
Evaluation rules:
eval(N,env) = N
eval({+ E1 E2},env) = eval(E1,env) + eval(E2,env)
eval({ E1 E2},env) = eval(E1,env)  eval(E2,env)
eval({* E1 E2},env) = eval(E1,env) * eval(E2,env)
eval({/ E1 E2},env) = eval(E1,env) / eval(E2,env)
eval(x,env) = lookup(x,env)
eval({with {x E1} E2},env) = eval(E2,extend(x,eval(E1,env),env))
eval({rec {x E1} E2},env) = ???
eval({fun {x} E},env) = <{fun {x} E}, env>
eval({call E1 E2},env1)
= eval(Ef,extend(x,eval(E2,env1),env2))
if eval(E1,env1) = <{fun {x} Ef}, env2>
= error! otherwise
#
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[WRec Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'with more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `with' syntax in ~s" sexpr)])]
[(cons 'rec more)
(match sexpr
[(list 'rec (list (symbol: name) named) body)
(WRec name (parsesexpr named) (parsesexpr body))]
[else (error 'parsesexpr "bad `rec' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
;; Define a type for functional environments
(definetype ENV = Symbol > VAL)
(: EmptyEnv : > ENV)
(define (EmptyEnv)
(lambda (id) (error 'lookup "no binding for ~s" id)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(env name))
(: Extend : Symbol VAL ENV > ENV)
;; extend a given environment cache with a new binding
(define (Extend id val restenv)
(lambda (name)
(if (eq? name id)
val
(restenv name))))
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(: recenv : Symbol > VAL)
(define (recenv name)
(if (eq? name id)
val
(restenv name)))
(: val : VAL)
(define val (eval expr recenv))
recenv)
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (eval namedexpr env) env))]
[(WRec boundid namedexpr boundbody)
(eval boundbody
(extendrec boundid namedexpr env))]
[(Id name) (lookup name env)]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (eval argexpr env) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Implementing `rec` Using Cyclic Structures
> [PLAI §10]
Looking at the arrows in the environment diagrams, what we're really
looking for is a closure that has an environment pointer which is the
same environment in which it was defined. This will make it possible for
`fact` to be bound to a closure that can refer to *itself* since its
environment is the same one in which it is defined. However, so far we
have no tools that makes it possible to do this.
What we need is to create a "cycle of pointers", and so far we do not
have a way of achieving that: when we create a closure, we begin with an
environment which is saved in the slot's environment slot, but we want
that closure to be the value of a binding in that same environment.

# Boxes and Mutation
To actually implement a circular structure, we will now use
*sideeffects*, using a new kind of Racket value which supports
mutation: a box. A box value is built with the `box` constructor:
(define mything (box 7))
the value is retrieved with the `unbox' function,
(* 6 (unbox mything))
and finally, the value can be changed with the `setbox!` function.
(setbox! mything 17)
(* 6 (unbox mything))
An important thing to note is that `setbox!` is much like `display`
etc, it returns a value that is not printed in the Racket REPL, because
there is no point in using the result of a `setbox!`, it is called for
the sideeffect it generates. (Languages like C blur this distinction
between returning a value and a sideeffect with its assignment
statement.)
As a side note, we now have side effects of two kinds: mutation of
state, and I/O (at least the O part). (Actually, there is also infinite
looping that can be viewed as another form of a side effect.) This means
that we're now in a completely different world, and lots of new things
can make sense now. A few things that you should know about:
* We never used more than one expression in a function body because
there was no point in it, but now there is. To evaluate a sequence of
Racket expressions, you wrap them in a `begin` expression.
* In most places you don't actually need to use `begin`  these are
places that are said to have an *implicit* `begin`: the body of a
function (or any lambda expression), the body of a `let` (and
`let`relatives), the consequence positions in `cond`, `match`, and
`cases` clauses and more. One of the common places where a `begin` is
used is in an `if` expression (and some people prefer using `cond`
instead when there is more than a single expression).
* `cond` without an `else` in the end can make sense, if all you're
using it it for is sideeffects.
* `if` could get a single expression which is executed when the
condition is true (and an unspecified value is used otherwise), but
our language (as well as the default Racket language) always forbids
this  there are convenient special forms for a onesided `if`s:
`when` & `unless`, and they can have any number of expressions (they
have an implicit `begin`). They have an advantage of saying "this code
does some sideeffects here" more explicit.
* There is a function called `foreach` which is just like `map`, except
that it doesn't collect the list of results, it is used only for
performing side effects.
When any one of these things is used (in Racket or other languages), you
can tell that sideeffects are involved, because there is no point in
any of them otherwise. In addition, any name that ends with a `!`
("bang") is used to mark a function that changes state (usually a
function that only changes state).
So how do we create a cycle? Simple, boxes can have any value, and they
can be put in other values like lists, so we can do this:
#lang pl untyped
(define foo (list 1 (box 3)))
(setbox! (second foo) foo)
and we get a circular value. (Note how it is printed.) And with types:
#lang pl
(: foo : (List Number (Boxof Any)))
(define foo (list 1 (box 3)))
(setbox! (second foo) foo)

# Types for Boxes
Obviously, `Any` is not too great  it is the most generic type, so it
provides the least information. For example, notice that
(unbox (second foo))
returns the right list, which is equal to `foo` itself  but if we try
to grab some part of the resulting list:
(second (unbox (second foo)))
we get a type error, because the result of the `unbox` is `Any`, so
Typed Racket knows nothing about it, and won't allow you to treat it as
a list. It is not too surprising that the type constructor that can help
in this case is `Rec` which we have already seen  it allows a type
that can refer to itself:
#lang pl
(: foo : (Rec this (List Number (Boxof (U #f this)))))
(define foo (list 1 (box #f)))
(setbox! (second foo) foo)
Note that either `foo` or the value in the box are both printed with a
`Rec` type  the value in the box can't just have a `(U #f this)`
type, since `this` doesn't mean anything in there, so the whole type
needs to still be present.
There is another issue to be aware of with `Boxof` types. For most type
constructors (like `Listof`), if `T1` is a subtype of `T2`, then we also
know that`(Listof T1)` is a subtype of `(Listof T2)`. This makes the
following code typecheck:
#lang pl
(: foo : (Listof Number) > Number)
(define (foo l)
(first l))
(: bar : Integer > Number)
(define (bar x)
(foo (list x)))
Since the `(Listof Integer)` is a subtype of the `(Listof Number)` input
for `foo`, the application typechecks. But this is *not* the same for
the output type, for example  if we change the `bar` type to:
(: bar : Integer > Integer)
we get a type error since `Number` is not a subtype of `Integer`. So
subtypes are required to "go up" on the input side and "down" on the
other. So, in a sense, the fact that boxes are mutable means that their
contents can be considered to be on the other side of the arrow, which
is why for such `T1` subtype of `T2`, it is `(Boxof T2)` that is a
subtype of `(Boxof T1)`, instead of the usual. For example, this doesn't
work:
#lang pl
(: foo : (Boxof Number) > Number)
(define (foo b)
(unbox b))
(: bar : Integer > Number)
(define (bar x)
(: b : (Boxof Integer))
(define b (box x))
(foo b))
And you can see why this is the case  the marked line is fine given a
`Number` contents, so if the type checker allows passing in a box
holding an integer, then that expression would mutate the contents and
make it an invalid value.
However, boxes are not only mutable, they hold a value that can be read
too, which means that they're on *both* sides of the arrow, and this
means that `(Boxof T1)` is a subtype of `(Boxof T2)` if `T2` is a
subtype of `T1` *and* `T1` is a subtype of `T2`  in other words, this
happens only when `T1` and `T2` are the same type. (See below for an
extended demonstration of all of this.)
Note also that this demonstration requires that extra `b` definition, if
it's skipped:
(define (bar x)
(foo (box x)))
then this will typecheck again  Typed Racket will just consider the
context that requires a box holding a `Number`, and it is still fine to
initialize such a box with an `Integer` value.
> As a side comment, this didn't always work. Earlier in its existence,
> Typed Racket would always choose a specific type for values, which
> would lead to confusing errors with boxes. For example, the above
> would need to be written as
>
> (define (bar x)
> (foo (box (ann x : Number))))
>
> to prevent Typed Racket from inferring a specific type. This is no
> longer the case, but there can still be some surprises. A similar
> annotation was needed in the case of a list holding a selfreferential
> box, to avoid the initial `#f` from getting a specificbutwrong type.

# `Boxof`'s Lack of Subtyping
The lack of any subtype relations between `(Boxof T)` and `(Boxof S)`
regardless of `S` and `T` can roughly be explained as follows.
First, a box is a container that you can pull a value out of  which
makes it similar to lists. In the case of lists, we have:
if: S subtypeof T
then: (Listof S) subtypeof (Listof T)
This is true for all such containers that you can pull a value out of:
if you expect to pull a `T` but you're given a container of a subtype
`S`, then things are still fine (you'll get an `S` which is also a `T`).
Such "containers" include functions that produce a value  for
example:
if: S subtypeof T
then: Q > S subtypeof Q > T
However, functions also have the other side, where things are different
 instead of a side of some *produced* value, it's the side of the
*consumed* value. We get the opposite rule there:
if: T subtypeof S
then: S > Q subtypeof T > Q
To see why this is right, use `Number` and `Integer` for `S` and `T`:
if: Integer subtypeof Number
then: Number > Q subtypeof Integer > Q
so  if you expect a function that takes an integer, a valid *subtype*
value that I can give you is a function that takes a number. In other
words, every function that takes a number is also a function that takes
an integer, but not the other way.
To summarize all of this, when you make the output type of a function
"smaller" (more constrained), the resulting type is smaller (a subset),
but on the input side things are flipped  a bigger input type means a
more constrained function.
> The technical names for these properties are: a "covariant" type is
> one that preserves the subtype relationship, and a "contravairant"
> type is one that reverses it. (Which is similar to how these terms are
> used in math.)
(Side note: this is related to the fact that in logic, `P => Q` is
roughly equivalent to `not(P) or Q`  the left side, `P`, is inside
negation. It also explains why in `((S > T) > Q)` the `S` obeys the
first rule, as if it was on the right side  because it's negated
twice.)
Now, a `(Boxof T)` is a producer of `T` when you pull a value out of the
box, but it's also a consumer of `T` when you put such a value in it.
This means that  using the above analogy  the `T` is on both sides
of the arrow. This means that
if: S subtypeof T *and* T subtypeof S
then: (Boxof S) subtypeof (Boxof T)
which is actually:
if: S isthesametypeas T
then: (Boxof S) isthesametypeas (Boxof T)
A different way to look at this conclusion is to consider the function
type of `(A > A)`: when is it a subtype of some other `(B > B)`? Only
when `A` is a subtype of `B` and `B` is a subtype of `A`, which means
that this happens only when `A` and `B` are the same type.
> The term for this is "nonvariant" (or "invariant"): `(A > A)` is
> unrelated to `(B > B)` regardless of how `A` and `B` are related. The
> only exception is, of course, when they are the same type. The
> Wikipedia entry about these puts the terms together nicely in the face
> of mutation:
>
> > Readonly data types (sources) can be covariant; writeonly data
> > types (sinks) can be contravariant. Mutable data types which act as
> > both sources and sinks should be invariant.
The following piece of code makes the analogy to function types more
formally. Boxes behave as if their contents is on both sides of a
function arrow  on the right because they're readable, and on the
left because they're writable, which the conclusion that a `(Boxof A)`
type is a subtype of itself and no other `(Boxof B)`.
#lang pl
;; a type for a "readonly" box
(definetype (Boxof/R A) = (> A))
;; Boxof/R constructor
(: box/r : (All (A) A > (Boxof/R A)))
(define (box/r x) (lambda () x))
;; we can see that (Boxof/R T1) is a subtype of (Boxof/R T2)
;; if T1 is a subtype of T2 (this is not surprising, since
;; these boxes are similar to any other container, like lists):
(: foo1 : Integer > (Boxof/R Integer))
(define (foo1 b) (box/r b))
(: bar1 : (Boxof/R Number) > Number)
(define (bar1 b) (b))
(test (bar1 (foo1 123)) => 123)
;; a type for a "writeonly" box
(definetype (Boxof/W A) = (A > Void))
;; Boxof/W constructor
(: box/w : (All (A) A > (Boxof/W A)))
(define (box/w x) (lambda (new) (set! x new)))
;; in contrast to the above, (Boxof/W T1) is a subtype of
;; (Boxof/W T2) if T2 is a subtype of T1, *not* the other way
;; (and note how this is related to A being on the *left* side
;; of the arrow in the `Boxof/W' type):
(: foo2 : Number > (Boxof/W Number))
(define (foo2 b) (box/w b))
(: bar2 : (Boxof/W Integer) Integer > Void)
(define (bar2 b new) (b new))
(test (bar2 (foo2 123) 456))
;; combining the above two into a type for a "read/write" box
(definetype (Boxof/RW A) = (A > A))
;; Boxof/RW constructor
(: box/rw : (All (A) A > (Boxof/RW A)))
(define (box/rw x) (lambda (new) (let ([old x]) (set! x new) old)))
;; this combines the above two: `A' appears on both sides of the
;; arrow, so (Boxof/RW T1) is a subtype of (Boxof/RW T2) if T1
;; is a subtype of T2 (because there's an A on the right) *and*
;; if T2 is a subtype of T1 (because there's another A on the
;; left)  and that can happen only when T1 and T2 are the same
;; type. So this is a type error:
;; (: foo3 : Integer > (Boxof/RW Integer))
;; (define (foo3 b) (box/rw b))
;; (: bar3 : (Boxof/RW Number) Number > Number)
;; (define (bar3 b new) (b new))
;; (test (bar3 (foo3 123) 456) => 123)
;; ** Expected (Number > Number), but got (Integer > Integer)
;; And this a type error too:
;; (: foo3 : Number > (Boxof/RW Number))
;; (define (foo3 b) (box/rw b))
;; (: bar3 : (Boxof/RW Integer) Integer > Integer)
;; (define (bar3 b new) (b new))
;; (test (bar3 (foo3 123) 456) => 123)
;; ** Expected (Integer > Integer), but got (Number > Number)
;; The two types must be the same for this to work:
(: foo3 : Integer > (Boxof/RW Integer))
(define (foo3 b) (box/rw b))
(: bar3 : (Boxof/RW Integer) Integer > Integer)
(define (bar3 b new) (b new))
(test (bar3 (foo3 123) 456) => 123)

# Implementing a Circular Environment
We now use this to implement `rec` in the following way:
1. Change environments so that instead of values they hold boxes of
values: `(Boxof VAL)` instead of `VAL`, and whenever `lookup` is
used, the resulting boxed value is unboxed,
2. In the `WRec` case, create the new environment with some temporary
binding for the identifier  any value will do since it should not
be used (when named expressions are always `fun` expressions),
3. Evaluate the expression in the new environment,
4. Change the binding of the identifier (the box) to the result of this
evaluation.
The resulting definition is:
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(let ([newcell (box (NumV 42))])
(let ([newenv (Extend id newcell restenv)])
(let ([value (eval expr newenv)])
(setbox! newcell value)
newenv))))
Racket has another `let` relative for such cases of multiplenested
`let`s  `let*`. This form is a derived form  it is defined as a
shorthand for using nested `let`s. The above is therefore exactly the
same as this code:
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(let* ([newcell (box (NumV 42))]
[newenv (Extend id newcell restenv)]
[value (eval expr newenv)])
(setbox! newcell value)
newenv))
This `let*` form can be read almost as a C/Javaish kind of code:
fun extend_rec(id, expr, rest_env) {
new_cell = new NumV(42);
new_env = Extend(id, new_cell, rest_env);
value = eval(expr, new_env);
*new_cell = value;
return new_env;
}
The code can be simpler if we fold the evaluation into the `setbox!`
(since `value` is used just there), and if use `lookup` to do the
mutation  since this way there is no need to hold onto the box. This
is a bit more expensive, but since the binding is guaranteed to be the
first one in the environment, the addition is just one quick step. The
only binding that we need is the one for the new environment, which we
can do as an internal definition, leaving us with:
(: extendrec : Symbol FLANG ENV > ENV)
(define (extendrec id expr restenv)
(define newenv (Extend id (box (NumV 42)) restenv))
(setbox! (lookup id newenv) (eval expr newenv))
newenv)
A complete rehacked version of FLANG with a `rec` binding follows. We
can't test `rec` easily since we have no conditionals, but you can at
least verify that
(run "{rec {f {fun {x} {call f x}}} {call f 0}}")
is an infinite loop.
;;; <<>>
#lang pl
(definetype FLANG
[Num Number]
[Add FLANG FLANG]
[Sub FLANG FLANG]
[Mul FLANG FLANG]
[Div FLANG FLANG]
[Id Symbol]
[With Symbol FLANG FLANG]
[WRec Symbol FLANG FLANG]
[Fun Symbol FLANG]
[Call FLANG FLANG])
(: parsesexpr : Sexpr > FLANG)
;; parses sexpressions into FLANGs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons (or 'with 'rec) more)
(match sexpr
[(list 'with (list (symbol: name) named) body)
(With name (parsesexpr named) (parsesexpr body))]
[(list 'rec (list (symbol: name) named) body)
(WRec name (parsesexpr named) (parsesexpr body))]
[(cons x more)
(error 'parsesexpr "bad `~s' syntax in ~s" x sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: name)) body)
(Fun name (parsesexpr body))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(list '+ lhs rhs) (Add (parsesexpr lhs) (parsesexpr rhs))]
[(list ' lhs rhs) (Sub (parsesexpr lhs) (parsesexpr rhs))]
[(list '* lhs rhs) (Mul (parsesexpr lhs) (parsesexpr rhs))]
[(list '/ lhs rhs) (Div (parsesexpr lhs) (parsesexpr rhs))]
[(list 'call fun arg)
(Call (parsesexpr fun) (parsesexpr arg))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > FLANG)
;; parses a string containing a FLANG expression to a FLANG AST
(define (parse str)
(parsesexpr (string>sexpr str)))
;; Types for environments, values, and a lookup function
(definetype ENV
[EmptyEnv]
[Extend Symbol (Boxof VAL) ENV])
(definetype VAL
[NumV Number]
[FunV Symbol FLANG ENV])
(: lookup : Symbol ENV > (Boxof VAL))
;; lookup a symbol in an environment, return its value or throw an
;; error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(Extend id boxedval restenv)
(if (eq? id name) boxedval (lookup name restenv))]))
(: extendrec : Symbol FLANG ENV > ENV)
;; extend an environment with a new binding that is the result of
;; evaluating an expression in the same environment as the extended
;; result
(define (extendrec id expr restenv)
(define newenv (Extend id (box (NumV 42)) restenv))
(setbox! (lookup id newenv) (eval expr newenv))
newenv)
(: NumV>number : VAL > Number)
;; convert a FLANG runtime numeric value to a Racket one
(define (NumV>number val)
(cases val
[(NumV n) n]
[else (error 'arithop "expected a number, got: ~s" val)]))
(: arithop : (Number Number > Number) VAL VAL > VAL)
;; gets a Racket numeric binary operator, and uses it within a NumV
;; wrapper
(define (arithop op val1 val2)
(NumV (op (NumV>number val1) (NumV>number val2))))
(: eval : FLANG ENV > VAL)
;; evaluates FLANG expressions by reducing them to values
(define (eval expr env)
(cases expr
[(Num n) (NumV n)]
[(Add l r) (arithop + (eval l env) (eval r env))]
[(Sub l r) (arithop  (eval l env) (eval r env))]
[(Mul l r) (arithop * (eval l env) (eval r env))]
[(Div l r) (arithop / (eval l env) (eval r env))]
[(With boundid namedexpr boundbody)
(eval boundbody
(Extend boundid (box (eval namedexpr env)) env))]
[(WRec boundid namedexpr boundbody)
(eval boundbody
(extendrec boundid namedexpr env))]
[(Id name) (unbox (lookup name env))]
[(Fun boundid boundbody)
(FunV boundid boundbody env)]
[(Call funexpr argexpr)
(let ([fval (eval funexpr env)])
(cases fval
[(FunV boundid boundbody fenv)
(eval boundbody
(Extend boundid (box (eval argexpr env)) fenv))]
[else (error 'eval "`call' expects a function, got: ~s"
fval)]))]))
(: run : String > Number)
;; evaluate a FLANG program contained in a string
(define (run str)
(let ([result (eval (parse str) (EmptyEnv))])
(cases result
[(NumV n) n]
[else (error 'run "evaluation returned a nonnumber: ~s"
result)])))
;; tests
(test (run "{call {fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{call add3 1}}")
=> 4)
(test (run "{with {add3 {fun {x} {+ x 3}}}
{with {add1 {fun {x} {+ x 1}}}
{with {x 3}
{call add1 {call add3 x}}}}}")
=> 7)
(test (run "{with {identity {fun {x} x}}
{with {foo {fun {x} {+ x 1}}}
{call {call identity foo} 123}}}")
=> 124)
(test (run "{with {x 3}
{with {f {fun {y} {+ x y}}}
{with {x 5}
{call f 4}}}}")
=> 7)
(test (run "{call {with {x 3}
{fun {y} {+ x y}}}
4}")
=> 7)
(test (run "{with {f {with {x 3} {fun {y} {+ x y}}}}
{with {x 100}
{call f 4}}}")
=> 7)
(test (run "{call {call {fun {x} {call x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)

# Variable Mutation
> [PLAI §12] and [PLAI §13] (different: adds boxes to the language)
>
> [PLAI §14] (that's what we do)
The code that we now have implements recursion by *changing* bindings,
and to make that possible we made environments hold boxes for all
bindings, therefore bindings are *all* mutable now. We can use this to
add more functionality to our evaluator, by allowing changing any
variable  we can add a `set!` form:
{set! }
to the evaluator that will modify the value of a variable. To implement
this functionality, all we need to do is to use `lookup` to retrieve
some box, then evaluate the expression and put the result in that box.
The actual implementation is left as a homework exercise.
One thing that should be considered here is  all of the expressions
in our language evaluate to some value, the question is what should be
the value of a `set!` expression? There are three obvious choices:
1. return some bogus value,
2. return the value that was assigned,
3. return the value that was previously in the box.
Each one of these has its own advantage  for example, C uses the
second option to `chain` assignments (eg, `x = y = 0`) and to allow side
effects where an expression is expected (eg, `while (x = x1) ...`).
The third one is useful in cases where you might use the old value that
is overwritten  for example, if C had this behavior, you could `pop`
a value from a linked list using something like:
first(stack = rest(stack));
because the argument to `first` will be the old value of `stack`, before
it changed to be its `rest`. You could also swap two variables in a
single expression: `x = y = x`.
(Note that the expression `x = x + 1` has the meaning of C's `++x` when
option (2) is used, and `x++` when option (3) is used.)
Racket chooses the first option, and we will do the same in our
language. The advantage here is that you get no discounts, therefore you
must be explicit about what values you want to return in situations
where there is no obvious choice. This leads to more robust programs
since you do not get other programmers that will rely on a feature of
your code that you did not plan on.
In any case, the modification that introduces mutation is small, but it
has a tremendous effect on our language: it was true for Racket, and it
is true for FLANG. We have seen how mutation affects the language subset
that we use, and in the extension of our FLANG the effect is even
stronger: since *any* variable can change (no need for explicit
`box`es). In other words, a binding is not always the same  in can
change as a result of a `set!` expression. Of course, we could extend
our language with boxes (using Racket boxes to implement FLANG boxes),
but that will be a little more verbose.
> Note that Racket does have a `set!` form, and in addition, fields in
> structs can be made modifiable. However, we do not use any of these.
> At least not for now.

# State and Environments
A quick example of how mutation can be used:
(define counter
(let ([counter (box 0)])
(lambda ()
(setbox! counter (+ 1 (unbox counter)))
(unbox counter))))
and compare that to:
(define (makecounter)
(let ([counter (box 0)])
(lambda ()
(setbox! counter (+ 1 (unbox counter)))
(unbox counter))))
It is a good idea if you follow the exact evaluation of
(define foo (makecounter))
(define bar (makecounter))
and see how both bindings have separate environment so each one gets its
own private state. The equivalent code in the homework interpreter
extended with `set!` doesn't need boxes:
{with {makecounter
{fun {}
{with {counter 0}
{fun {}
{set! counter {+ counter 1}}
counter}}}}
{with {foo {call makecounter}}
{with {bar {call makecounter}}
...}}}
To see multiple values from a single expression you can extend the
language with a `list` binding. As a temporary hack, we can use dummy
function inputs to cover for our lack of nullary functions, and use
`with` (with dummy bound ids) to sequence multiple expressions:
{with {makecounter
{fun {init}
{with {counter init}
{fun {_}
{with {_ {set! counter {+ counter 1}}}
counter}}}}}
{with {foo {call makecounter 0}}
{with {bar {call makecounter 1}}
{+ {+ {call foo 0} {+ {* 10 {call foo 0}}
{* 100 {call foo 0}}}}
{* 10000 {+ {call bar 0} {+ {* 10 {call bar 0}}
{* 100 {call bar 0}}}}}}}}}
Note that we cannot describe this behavior with substitution rules! We
now use the environments to make it possible to change bindings  so
finally an environment is actually an environment rather than a
substitution cache.
When you look at the above, note that we still use lexical scope  in
fact, the local binding is actually a private state that nobody can
access. For example, if we write this:
(define counter
(let ([counter (box 0)])
(lambda ()
(setbox! counter (+ 1 (unbox counter)))
(if (zero? (modulo (unbox counter) 4)) 'tock 'tick))))
then the resulting function that us bound to `counter` keeps a local
integer state which no other code can access  you cannot modify it,
reset it, or even know if it is really an integer that is used in there.

# Implementing Objects with State
We have already seen how several pieces of information can be
encapsulate in a Racket closure that keeps them all; now we can do a
little more  we can actually have mutable state, which leads to a
natural way to implement objects. For example:
(define (makepoint x y)
(let ([xb (box x)]
[yb (box y)])
(lambda (msg)
(match msg
['getx (unbox xb)]
['gety (unbox yb)]
['incx (setbox! xb (add1 (unbox xb)))]))))
implements a constructor for `point` objects which keep two values and
can move one of them. Note that the messages act as a form of methods,
and that the values themselves are hidden and are accessible only
through the interface that these messages make. For example, if these
points correspond to some graphic object on the screen, we can easily
incorporate a necessary screen update:
(define (makepoint x y)
(let ([xb (box x)]
[yb (box y)])
(lambda (msg)
(match msg
['getx (unbox xb)]
['gety (unbox yb)]
['incx (setbox! xb (add1 (unbox xb)))
(updatescreen)]))))
and be sure that this is always done when the value changes  since
there is no way to change the value except through this interface.
A more complete example would define functions that actually send these
messages  here is a better implementation of a point object and the
corresponding accessors and mutators:
(define (makepoint x y)
(let ([xb (box x)]
[yb (box y)])
(lambda (msg)
(match msg
['getx (unbox xb)]
['gety (unbox yb)]
[(list 'setx newx)
(setbox! xb newx)
(updatescreen)]
[(list 'sety newy)
(setbox! yb newy)
(updatescreen)]))))
(define (pointx p) (p 'getx))
(define (pointy p) (p 'gety))
(define (setpointx! p x) (p (list 'setx x)))
(define (setpointy! p y) (p (list 'sety y)))
And a quick imitation of inheritance can be achieved using delegation to
an instance of the superclass:
(define (makecoloredpoint x y color)
(let ([p (makepoint x y)])
(lambda (msg)
(match msg
['getcolor color]
[else (p msg)]))))
You can see how all of these could come from some preprocessing of a
more normallooking class definition form, like:
(defclass point (x y)
(public (getx) x)
(public (gety) y)
(public (setx new) (set! x newx))
(public (setx new) (set! x newx)))
(defclass coloredpoint point (c)
(public (getcolor) c))

# The Toy Language
> Not in PLAI
A quick note: from now on, we will work with a variation of our language
 it will change the syntax to look a little more like Racket, and we
will use Racket values for values in our language and Racket functions
for builtins in our language.
Main highlights:
* There can be multiple bindings in function arguments and local `bind`
forms  the names are required to be distinct.
* There are now a few keywords like `bind` that are parsed in a special
way. Other forms are taken as function application, which means that
there are no special parse rules (and AST entries) for arithmetic
functions. They're now bindings in the global environment, and treated
in the same way as all bindings. For example, `*` is an expression
that evaluates to the *primitive* multiplication function, and
`{bind {{+ *}} {+ 2 3}}` evaluates to `6`.
* Since function applications are now the same for primitive functions
and userbound functions, there is no need for a `call` keyword. Note
that the function call part of the parser must be last, since it
should apply only if the input is not some other known form.
* Note the use of `makeuntypedlistfunction`: it's a library function
(included in the course language) that can convert a few known Racket
functions to a function that consumes a list of *any* Racket values,
and returns the result of applying the given Racket function on these
values. For example:
(define add (makeuntypedlistfunction +))
(add (list 1 2 3 4))
evaluates to `10`.
* Another important aspect of this is its type  the type of `add` in
the previous example is `(List > Any)`, so the resulting function can
consume *any* input values. If it gets a bad value, it will throw an
appropriate error. This is a hack: it basically means that the
resulting `add` function has a very generic type (requiring just a
list), so errors can be thrown at runtime. However, in this case, a
better solution is not going to make these runtime errors go away
because the language that we're implementing is not statically typed.
* The benefit of this is that we can avoid the hassle of more verbose
code by letting these functions dynamically check the input values, so
we can use a single `RktV` variant in `VAL` which wraps any Racket
value. (Otherwise we'd need different wrappers for different types,
and implement these dynamic checks.)
The following is the complete implementation.
;;; <<>>
#lang pl
;;; 
;;; Syntax
# The BNF:
::=

 { bind {{ } ... } }
 { fun { ... } }
 { if }
 { ... }
#
;; A matching abstract syntax tree datatype:
(definetype TOY
[Num Number]
[Id Symbol]
[Bind (Listof Symbol) (Listof TOY) TOY]
[Fun (Listof Symbol) TOY]
[Call TOY (Listof TOY)]
[If TOY TOY TOY])
(: uniquelist? : (Listof Any) > Boolean)
;; Tests whether a list is unique, guards Bind and Fun values.
(define (uniquelist? xs)
(or (null? xs)
(and (not (member (first xs) (rest xs)))
(uniquelist? (rest xs)))))
(: parsesexpr : Sexpr > TOY)
;; parses sexpressions into TOYs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'bind more)
(match sexpr
[(list 'bind (list (list (symbol: names) (sexpr: nameds))
...)
body)
(if (uniquelist? names)
(Bind names (map parsesexpr nameds) (parsesexpr body))
(error 'parsesexpr "duplicate `bind' names: ~s" names))]
[else (error 'parsesexpr "bad `bind' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: names) ...) body)
(if (uniquelist? names)
(Fun names (parsesexpr body))
(error 'parsesexpr "duplicate `fun' names: ~s" names))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(cons 'if more)
(match sexpr
[(list 'if cond then else)
(If (parsesexpr cond)
(parsesexpr then)
(parsesexpr else))]
[else (error 'parsesexpr "bad `if' syntax in ~s" sexpr)])]
[(list fun args ...) ; other lists are applications
(Call (parsesexpr fun)
(map parsesexpr args))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > TOY)
;; Parses a string containing an TOY expression to a TOY AST.
(define (parse str)
(parsesexpr (string>sexpr str)))
;;; 
;;; Values and environments
(definetype ENV
[EmptyEnv]
[FrameEnv FRAME ENV])
;; a frame is an association list of names and values.
(definetype FRAME = (Listof (List Symbol VAL)))
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) TOY ENV]
[PrimV ((Listof VAL) > VAL)])
(: extend : (Listof Symbol) (Listof VAL) ENV > ENV)
;; extends an environment with a new frame.
(define (extend names values env)
(if (= (length names) (length values))
(FrameEnv (map (lambda ([name : Symbol] [val : VAL])
(list name val))
names values)
env)
(error 'extend "arity mismatch for names: ~s" names)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, frame by frame,
;; return its value or throw an error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(FrameEnv frame rest)
(let ([cell (assq name frame)])
(if cell
(second cell)
(lookup name rest)))]))
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': unwrap a RktV wrapper in
;; preparation to be sent to the primitive function
(define (unwraprktv x)
(cases x
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" x)]))
(: racketfunc>primval : Function > VAL)
;; converts a racket function to a primitive evaluator function
;; which is a PrimV holding a ((Listof VAL) > VAL) function.
;; (the resulting function will use the list function as is,
;; and it is the list function's responsibility to throw an error
;; if it's given a bad number of arguments or bad input types.)
(define (racketfunc>primval racketfunc)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(RktV (listfunc (map unwraprktv args))))))
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval +))
(list ' (racketfunc>primval ))
(list '* (racketfunc>primval *))
(list '/ (racketfunc>primval /))
(list '< (racketfunc>primval <))
(list '> (racketfunc>primval >))
(list '= (racketfunc>primval =))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f)))
(EmptyEnv)))
;;; 
;;; Evaluation
(: eval : TOY ENV > VAL)
;; evaluates TOY expressions
(define (eval expr env)
;; convenient helper
(: eval* : TOY > VAL)
(define (eval* expr) (eval expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (eval* funexpr)]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (eval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let ([result (eval (parse str) globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
;;; 
;;; Tests
(test (run "{{fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}} {add3 1}}")
=> 4)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}
{add1 {fun {x} {+ x 1}}}}
{bind {{x 3}} {add1 {add3 x}}}}")
=> 7)
(test (run "{bind {{identity {fun {x} x}}
{foo {fun {x} {+ x 1}}}}
{{identity foo} 123}}")
=> 124)
(test (run "{bind {{x 3}}
{bind {{f {fun {y} {+ x y}}}}
{bind {{x 5}}
{f 4}}}}")
=> 7)
(test (run "{{{fun {x} {x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)
;; More tests for complete coverage
(test (run "{bind x 5 x}") =error> "bad `bind' syntax")
(test (run "{fun x x}") =error> "bad `fun' syntax")
(test (run "{if x}") =error> "bad `if' syntax")
(test (run "{}") =error> "bad syntax")
(test (run "{bind {{x 5} {x 5}} x}") =error> "duplicate*bind*names")
(test (run "{fun {x x} x}") =error> "duplicate*fun*names")
(test (run "{+ x 1}") =error> "no binding for")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{1 2}") =error> "with a nonfunction")
(test (run "{{fun {x} x}}") =error> "arity mismatch")
(test (run "{if {< 4 5} 6 7}") => 6)
(test (run "{if {< 5 4} 6 7}") => 7)
(test (run "{if + 6 7}") => 6)
(test (run "{fun {x} x}") =error> "returned a bad value")
;;; 

# Compilation and Partial Evaluation
Instead of interpreting an expression, which is performing a full
evaluation, we can think about *compiling* it: translating it to a
different language which we can later run more easily, more efficiently,
on more platforms, etc. Another feature that is usually associated with
compilation is that a lot more work was done at the compilation stage,
making the actual running of the code faster.
For example, translating an AST into one that has deBruijn indexes
instead of identifier names is a form of compilation  not only is it
translating one language into another, it does the work involved in name
lookup before the program starts running.
This is something that we can experiment with now. An easy way to
achieve this is to start with our evaluation function:
(: eval : TOY ENV > VAL)
;; evaluates TOY expressions
(define (eval expr env)
;; convenient helper
(: eval* : TOY > VAL)
(define (eval* expr) (eval expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (eval* funexpr)]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (eval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
and change it so it compiles a given expression to a Racket function.
(This is, of course, just to demonstrate a conceptual point, it is only
the tip of what compilers actually do...) This means that we need to
turn it into a function that receives a TOY expression and compiles it.
In other words, `eval` no longer consumes and environment argument which
makes sense because the environment is a place to hold runtime values,
so it is a data structure that is not part of the compiler (it is
usually represented as the call stack).
So we split the two arguments into a compiletime and runtime, which
can be done by simply currying the `eval` function  here this is
done, and all calls to `eval` are also curried:
(: eval : TOY > ENV > VAL) ;*** note the curried type
;; evaluates TOY expressions
(define (eval expr)
(lambda (env)
;; convenient helper
(: eval* : TOY > VAL)
(define (eval* expr) ((eval expr) env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
((eval boundbody) (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (eval* funexpr)]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
((eval body) (extend names argvals funenv))]
[else (error 'eval
"function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (eval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))])))
We also need to change the `eval` call in the main `run` function:
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let ([result ((eval (parse str)) globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
Not much has changed so far.
Note that in the general case of a compiler we need to run a program
several times, so we'd want to avoid parsing it over and over again. We
can do that by keeping a single parsed AST of the input. Now we went one
step further by making it possible to do more work ahead and keep the
result of the first "stage" of eval around (except that "more work" is
really not saying much at the moment):
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let* ([compiled (eval (parse str))]
[result (compiled globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
At this point, even though our "compiler" is not much more than a
slightly different representation of the same functionality, we rename
`eval` to `compile` which is a more appropriate description of what we
intend it to do (so we change the purpose statement too):
(: compile : TOY > ENV > VAL)
;; compiles TOY expressions to Racket functions.
(define (compile expr)
(lambda (env)
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
((compile boundbody)
(extend names (map compile* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (compile* funexpr)]
[argvals (map compile* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
((compile body) (extend names argvals funenv))]
[else (error 'call ; this is *not* a compilation error
"function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(compile* (if (cases (compile* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))])))
(: run : String > Any)
;; evaluate a TOY program contained in a string
(define (run str)
(let* ([compiled (compile (parse str))]
[result (compiled globalenvironment)])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))

Not much changed, still. We curried the `eval` function and renamed it
to `compile`. But when we actually call compile almost nothing happens
 all it does is create a Racket closure which will do the rest of the
work. (And this closure closes over the given expression.)
Running this "compiled" code is going to be very much like the previous
usage of `eval`, except a little *slower*, because now every recursive
call involves calling `compile` to generate a closure, which is then
immediately used  so we just added some allocations at the recursive
call points! (Actually, the extra cost is minimal because the Racket
compiler will optimize away such immediate closure applications.)
Another way to see how this is not really a compiler yet is to consider
*when* `compile` gets called. A proper compiler is something that does
all of its work *before* running the code, which means that once it
spits out the compiled code it shouldn't be used again (except for
compiling other code, of course). Our current code is not really a
compiler since it breaks this feature. (For example, if GCC behaved this
way, then it would "compile" files by producing code that invokes GCC to
compile the next step, which, when run, invokes GCC again, etc.)
However, the conceptual change is substantial  we now have a function
that does its work in two stages  the first part gets an expression
and *can* do some compiletime work, and the second part does the
runtime work, and includes anything inside the (lambda (env) ...). The
thing is that so far, the code does nothing at the compilation stage
(remember: only creates a closure). But because we have two stages, we
can now shift work from the second stage (the runtime) to the first
(the compiletime).
For example, consider the following simple example:
#lang pl
(: foo : Number Number > Number)
(define (foo x y)
(* x y))
(: bar : Number > Number)
(define (bar c)
(: loop : Number Number > Number)
(define (loop n acc)
(if (< 0 n)
(loop ( n 1) (+ (foo c n) acc))
acc))
(loop 40000000 0))
(time (bar 0))
We can do the same thing here  separate `foo` it into two stages
using currying, and modify `bar` appropriately:
#lang pl
(: foo : Number > Number > Number)
(define (foo x)
(lambda (y)
(* x y)))
(: bar : Number > Number)
(define (bar c)
(: loop : Number Number > Number)
(define (loop n acc)
(if (< 0 n)
(loop ( n 1) (+ ((foo c) n) acc))
acc))
(loop 40000000 0))
(time (bar 0))
Now instead of a simple multiplication, lets expand it a little, for
example, do a case split on common cases where `x` is `0`, `1`, or `2`:
(: foo : Number > Number > Number)
(define (foo x)
(lambda (y)
(cond [(= x 0) 0]
[(= x 1) y]
[(= x 2) (+ y y)] ; assume that this is faster
[else (* x y)])))
This is not much faster, since Racket already optimizes multiplication
in a similar way.
Now comes the real magic: deciding what branch of the cond to take
depends *only* on x, so we can `push` the lambda inside:
(: foo : Number > Number > Number)
(define (foo x)
(cond [(= x 0) (lambda (y) 0)]
[(= x 1) (lambda (y) y)]
[(= x 2) (lambda (y) (+ y y))]
[else (lambda (y) (* x y))]))
We just made an improvement  the comparisons for the common cases are
now done as soon as (foo x) is called, they're not delayed to when the
resulting function is used. Now go back to the way this is used in `bar`
and make it call `foo` once for the given `c`:
#lang pl
(: foo : Number > Number > Number)
(define (foo x)
(cond [(= x 0) (lambda (y) 0)]
[(= x 1) (lambda (y) y)]
[(= x 2) (lambda (y) (+ y y))]
[else (lambda (y) (* x y))]))
(: bar : Number > Number)
(define (bar c)
(define fooc (foo c))
(: loop : Number Number > Number)
(define (loop n acc)
(if (< 0 n)
(loop ( n 1) (+ (fooc n) acc))
acc))
(loop 40000000 0))
(time (bar 0))
Now fooc is generated once, and if `c` happens to be one of the three
common cases (as in the last expression), we can avoid doing any
multiplication. (And if we hit the default case, then we're doing the
same thing we did before.)
[However, the result runs a little slower! The reason is that dealing
with functions can have a higher cost when the compiler cannot "simplify
closures away"  and this is what happens in the last version. The
additional overhead is much higher than the multiplication we save (the
Racket compiler inlines multiplications, so their cost is close to just
executing a single machinecode instruction).]
Here is another useful example that demonstrates this:
(define (foo list)
(map (lambda (n) (if ...something... E1 E2))
list))
>
(define (foo list)
(map (if ...something...
(lambda (n) E1)
(lambda (n) E2))
list))
(Question: when can you do that?)
This is not unique to Racket, it can happen in any language. Racket (or
any language with first class function values) only makes it easy to
create a local function that is specialized for the flag.

Getting our thing closer to a compiler is done in a similar way  we
push the `(lambda (env) ...)` inside the various cases. (Note that
`compile*` depends on the `env` argument, so it also needs to move
inside  this is done for all cases that use it, and will eventually
go away.) We actually need to use `(lambda ([env : ENV]) ...)` though,
to avoid upsetting the type checker:
(: compile : TOY > ENV > VAL)
;; compiles TOY expressions to Racket functions.
(define (compile expr)
(cases expr
[(Num n) (lambda ([env : ENV]) (RktV n))]
[(Id name) (lambda ([env : ENV]) (lookup name env))]
[(Bind names exprs boundbody)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
((compile boundbody)
(extend names (map compile* exprs) env)))]
[(Fun names boundbody)
(lambda ([env : ENV]) (FunV names boundbody env))]
[(Call funexpr argexprs)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
(let ([fval (compile* funexpr)]
[argvals (map compile* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
((compile body) (extend names argvals funenv))]
[else (error 'call ; this is *not* a compilation error
"function call with a nonfunction: ~s"
fval)])))]
[(If condexpr thenexpr elseexpr)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
(compile* (if (cases (compile* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr)))]))
and with this we shifted a bit of actual work to compile time  the
code that checks what structure we have, and extracts its different
slots. But this is still not good enough  it's only the first
toplevel `cases` that is moved to compiletime  recursive calls to
`compile` are still there in the resulting closures. This can be seen by
the fact that we have those calls to `compile` in the Racket closures
that are the results of our compiler, which, as discussed above, mean
that it's not an actual compiler yet.
For example, consider the `Bind` case:
[(Bind names exprs boundbody)
(lambda ([env : ENV])
(: compile* : TOY > VAL)
(define (compile* expr) ((compile expr) env))
((compile boundbody)
(extend names (map compile* exprs) env)))]
At compiletime we identify and deconstruct the Bind structure, then
create a the runtime closure that will access these parts when the code
runs. But this closure will itself call `compile` on `boundbody` and
each of the expressions. Both of these calls can be done at compile
time, since they only need the expressions  they don't depend on the
environment. Note that `compile*` turns to `run` here, since all it does
is run a compiled expression on the current environment.
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(lambda ([env : ENV])
(: run : (ENV > VAL) > VAL)
(define (run compiledexpr) (compiledexpr env))
(compiledbody (extend names
(map run compiledexprs)
env))))]
We can move it back up, out of the resulting functions, by making it a
function that consumes an environment and returns a "caller" function:
(define (compile expr)
;; convenient helper
(: caller : ENV > (ENV > VAL) > VAL)
(define (caller env)
(lambda (compiled) (compiled env)))
(cases expr
...
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(lambda ([env : ENV])
(compiledbody (extend names
(map (caller env) compiledexprs)
env))))]
...))
Once this is done, we have a bunch of work that can happen at compile
time: we prescan the main "bind spine" of the code.
We can deal in a similar way with other occurrences of `compile` calls
in compiled code. The two branches that need to be fixed are:
1. In the `If` branch, there is not much to do. After we make it
precompile the `condexpr`, we also need to make it precompile both
the `thenexpr` and the `elseexpr`. This might seem like doing more
work (since before changing it only one would get compiled), but
since this is compiletime work, then it's not as important. Also,
`if` expressions are evaluated many times (being part of a loop, for
example), so overall we still win.
2. The `Call` branch is a little trickier: the problem here is that the
expressions that are compiled are coming from the closure that is
being applied. The solution for this is obvious: we need to change
the closure type so that it closes over *compiled* expressions
instead of over plain ones. This makes sense because closures are
runtime values, so they need to close over the compiled expressions
since this is what we use as "code" at runtime.
Again, the goal is to have no `compile` calls that happen at runtime:
they should all happen before that. This would allow, for example, to
obliterate the compiler once it has done its work, similar to how you
don't need GCC to run a C application. Yet another way to look at this
is that we shouldn't look at the AST at runtime  again, the analogy
to GCC is that the AST is a data structure that the compiler uses, and
it does not exist at runtime. Any runtime reference to the TOY AST is,
therefore, as bad as any runtime reference to `compile`.
When we're done with this process we'll have something that is an actual
compiler: translating TOY programs into Racket closures. To see how this
is an actual compiler consider the fact that Racket uses a JIT to
translate bytecode into machine code when it's running functions. This
means that the compiled version of our TOY programs are, in fact,
translated all the way down to machine code.
Yet another way to see this is to change the compiler code so instead of
producing a Racket closure it spits out the Racket code that makes up
these closures when evaluated. For example, change
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(lambda ([env : ENV])
(compiledbody (extend ...))))]
into
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
(stringappend
"(lambda ([env : ENV]) ("
compiledbody
" (extend ...)))"))]
so we get a string that is a Racket program. But since we're using a
Lisp dialect, it's generally better to use Sexpressions instead:
[(Bind names exprs boundbody)
(let ([compiledbody (compile boundbody)]
[compiledexprs (map compile exprs)])
`(lambda ([env : ENV])
(,compiledbody (extend ...))))]
(Later in the course we'll talk about these "`` ` ``"s and "`,`"s. For
now, it's enough to know that "`` ` ``" is kind of like a quote, and
"`,`" is an unquote.)

# Lazy Evaluation: Using a Lazy Racket
> [PLAI §7] (done with Haskell)
For this part, we will use a new language, Lazy Racket.
#lang pl lazy
As the name suggests, this is a version of the normal (untyped) Racket
language that is lazy.
First of all, let's verify that this is indeed a lazy language:
> (define (foo x) 3)
> (foo (+ 1 "2"))
3
That went without a problem  the argument expression was indeed not
evaluated. In this language, you can treat all expressions as future
`promises` to evaluate. There are certain points where such promises are
actually `forced`, all of these stem from some need to print a resulting
value, in our case, it's the REPL that prints such values:
> (+ 1 "2")
+: expects type as 2nd argument,
given: "2"; other arguments were: 1
The expression by itself only generates a promise, but when we want to
print it, this promise is forced to evaluate  this forces the
addition, which forces its arguments (plain values rather than
computation promises), and at this stage we get an error. (If we never
want to see any results, then the language will never do anything at
all.) So a promise is forced either when a value printout is needed, or
if it is needed to recursively compute a value to print:
> (* 1 (+ 2 "3"))
+: expects type as 2nd argument,
given: "3"; other arguments were: 2
Note that the error was raised by the internal expression: the outer
expression uses `*`, and `+` requires actual values not promises.
Another example, which is now obvious, is that we can now define an `if`
function:
> (define (myif x y z) (if x y z))
> (myif (< 1 2) 3 (+ 4 "5"))
3
Actually, in this language `if`, `and`, and `or` are all function values
instead of special forms:
> (list if and or)
(# # #)
> ((third (list if and or)) #t (+ 1 "two"))
#t
(By now, you should know that these have no value in Racket  using
them like this in plain will lead to syntax errors.) There are some
primitives that do not force their arguments. Constructors fall in this
category, for example `cons` and `list`:
> (define (fib n) (if (<= n 1) n (+ (fib ( n 1)) (fib ( n 2)))))
> (define a (list (+ 1 2) (+ 3 "4") (fib 30) (* 5 6)))
Nothing  the definition simply worked, but that's expected, since
nothing is printed. If we try to inspect this value, we can get some of
its parts, provided we do not force the bogus one:
> (first a)
3
> (fourth a)
30
> (third a)
196418
> (second a)
+: contract violation, expected: number?, given: "4" ...
The same holds for cons:
> (second (cons 1 (cons 2 (first null))))
2
Now if this is the case, then how about this:
> (define ones (cons 1 ones))
Everything is fine, as expected  but what is the value of `ones` now?
Clearly, it is a list that has 1 as its first element:
> (first ones)
1
But what do we have in the tail of this list? We have `ones` which we
already know is a list that has 1 in its first place  so following
Racket's usual rules, it means that the second element of `ones` is,
again, 1. If we continue this, we can see that `ones` is, in fact, an
*infinite* list of 1s:
> (second ones)
1
> (fifth ones)
1
In this sense, the way `define` behaves is that it defines a true
equation: if ones is defined as (cons 1 ones), then the real value does
satisfy
(equal? ones (cons 1 ones))
which means that the value is the fixpoint of the defined expression.
We can use `append` in a similar way:
> (define foo (append (list 1 2 3) foo))
> (fourth foo)
1
This looks like it has some common theme with the discussion of
implementing recursive environments  it actually demonstrates that in
this language, `letrec` can be used for *simple* values too. First of
all, a side note  here an expression that indicated a bug in our
substituting evaluator:
> (let ([x (list y)])
(let ([y 1])
x))
reference to undefined identifier: y
When our evaluator returned `1` for this, we noticed that this was a
bug: it does not obey the lexical scoping rules. As seen above, Lazy
Racket is correctly using lexical scope. Now we can go back to the use
of `letrec`  what do we get by this definition:
> (define twos (let ([xs (cons 2 xs)]) xs))
we get an error about `xs` being undefined.
`xs` is unbound because of the usual scope that `let` uses. How can we
make this work?  We simply use `letrec`:
> (define twos (letrec ([xs (cons 2 xs)]) xs))
> (first twos)
2
As expected, if we try to print an infinite list will cause an infinite
loop, which DrRacket catches and prints in that weird way:
> twos
#0=(2 . #0#)
How would we inspect an infinite list? We write a function that returns
part of it:
> (define (take n l)
(if (or (<= n 0) (null? l))
null
(cons (first l) (take (sub1 n) (rest l)))))
> (take 10 twos)
(2 2 2 2 2 2 2 2 2 2)
> (define foo (append (list 1 2 3) foo))
> (take 10 foo)
(1 2 3 1 2 3 1 2 3 1)
Dealing with infinite lists can lead to lots of interesting things, for
example:
> (define fibs (cons 1 (cons 1 (map + fibs (rest fibs)))))
> (take 10 fibs)
(1 1 2 3 5 8 13 21 34 55)
To see how it works, see what you know about `fibs[n]` which will be our
notation for the nth element of `fibs` (starting from `1`):
fibs[1] = 1 because of the first `cons'
fibs[2] = 1 because of the second `cons'
and for all `n>2`:
fibs[n] = (map + fibs (rest fibs))[n2]
= fibs[n2] + (rest fibs)[n2]
= fibs[n2] + fibs[n2+1]
= fibs[n2] + fibs[n1]
so it follows the exact definition of Fibonacci numbers.
Note that the list examples demonstrate that laziness applies to nested
values (actually, nested computations) too: a value that is not needed
is not computed, even if it is *contained* in a value that is needed.
For example, in:
(define x (/ 1 0))
(if (list (+ 1 x)) 1 2)
the `if` needs to know only whether its first argument (note: it *is* an
*argument*, since this `if` is a function) is `#f` or not. Once it is
determined that it is a pair (a `cons` cell), there is no need to
actually look at the values inside the pair, and therefore `(+ 1 x)`
(and more specifically, `x`) is never evaluated and we see no error.

# Lazy Evaluation: Some Issues
There are a few issues that we need to be aware of when we're dealing
with a lazy language. First of all, remember that our previous attempt
at lazy evaluation has made
{with {x y}
{with {y 1}
x}}
evaluate to 1, which does not follow the rules of lexical scope. This is
*not* a problem with lazy evaluation, but rather a problem with our
naive implementation. We will shortly see a way to resolve this problem.
In the meanwhile, remember that when we try the same in Lazy Racket we
do get the expected error:
> (let ([x y])
(let ([y 1])
x))
reference to undefined identifier: y
A second issue is a subtle point that you might have noticed when we
played with Lazy Racket: for some of the list values we have see a "`.`"
printed. This is part of the usual way Racket displays an *improper
list*  any list that does not terminate with a null value. For
example, in plain Racket:
> (cons 1 2)
(1 . 2)
> (cons 1 (cons 2 (cons 3 4)))
(1 2 3 . 4)
In the dialect that we're using in this course, this is not possible.
The secret is that the `cons` that we use first checks that its second
argument is a proper list, and it will raise an error if not. So how
come Lazy Racket's `cons` is not doing the same thing? The problem is
that to know that something is a proper list, we will have to force it,
which will make it not behave like a constructor.
> As a side note, we can achieve some of this protection if we don't
> insist on immediately checking the second argument completely, and
> instead we do the check when needed  lazily:
>
> (define (safecons x l)
> (cons x (if (pair? l) l (error "poof"))))
Finally, there are two consequences of using a lazy language that make
it more difficult to debug (or at lease take some time to get used to).
First of all, control tends to flow in surprising ways. For example,
enter the following into DrRacket, and run it in the normal language
level for the course:
(define (foo3 x)
(/ x "1"))
(define (foo2 x)
(foo3 x))
(define (foo1 x)
(list (foo2 x)))
(define (foo0 x)
(first (foo1 x)))
(+ 1 (foo0 3))
In the normal language level, we get an error, and red arrows that show
us how where in the computation the error was raised. The arrows are all
expected, except that `foo2` is not in the path  why is that?
Remember the discussion about tailcalls and how they are important in
Racket since they are the only tool to generate loops? This is what
we're seeing here: `foo2` calls `foo3` in a tail position, so there is
no need to keep the `foo2` context anymore  it is simply replaced by
`foo3`. (Incidentally, there is also no arrow that goes through `foo1`:
Racket does some smart inlining, and it figures out that `foo0`+`foo1`
are simply returning the same value, so it skips `foo1`.)
Now switch to Lazy Racket and rerun  you'll see no arrows at all.
What's the problem? The call of `foo0` creates a promise that is forced
in the toplevel expression, that simply returns the `first` of the
`list` that `foo1` created  and all of that can be done without
forcing the `foo2` call. Going this way, the computation is finally
running into an error *after* the calls to `foo0`, `foo1`, and `foo2`
are done  so we get the seemingly outofcontext error.
To follow what's happening here, we need to follow how promise are
forced: when we have code like
> (define (foo x) (/ x 0))
> (foo 9)
then the `foo` call is a *strict point*, since we need an actual value
to display on the REPL. Since it's in a strict position, we do the call,
but when we're in the function there is no need to compute the division
result  so it is returned as a lazy promise value back to the
toplevel. It is only then that we continue the process of getting an
actual value, which leads to trying to compute the division and get the
error.
Finally, there are also potential problems when you're not careful about
memory use. A common technique when using a lazy language is to generate
an infinite list and pull out its Nth element. For example, to compute
the Nth Fibonacci number, we've seen how we can do this:
(define fibs (cons 1 (cons 1 (map + fibs (rest fibs)))))
(define (fib n) (listref fibs n))
and we can also do this (reminder: `letrec` is the same as an internal
definition):
(define (fib n)
(letrec ([fibs (cons 1 (cons 1 (map + fibs (rest fibs))))])
(listref fibs n))) ; tailcall => no need to keep `fibs`
but the problem here is that when `listref` is making its way down the
list, it might still hold a reference to `fibs`, which means that as the
list is forced, all intermediate values are held in memory. In the first
of these two, this is guaranteed to happen since we have a binding that
points at the head of the `fibs` list. With the second form things can
be confusing: it might be that our language implementation is smart
enough to see that `fibs` is not really needed anymore and release the
offending reference. If it isn't, then we'd have to do something like
(define (fib n)
(listref
(letrec ([fibs (cons 1 (cons 1 (map + fibs (rest fibs))))])
fibs)
n))
to eliminate it. But even if the implementation does know that there is
no need for that reference, there are other tricky situations that are
hard to avoid.
Side note: Racket didn't use to do this optimization, but now it does,
and the lazy language helped in clarifying more cases where references
should be released. To see that, consider these two variants:
(define (nat1 n)
(define nats (cons 1 (map add1 nats)))
(if (number? (listref nats n))
"a number"
"not a number"))
;; we want to provide some information: show the first element
(define (nat2 n)
(define nats (cons 1 (map add1 nats)))
(if (number? (listref nats n))
"a number"
(error 'nat "the list starting with ~s is broken"
(first nats))))
If we try to use them with a big input:
(nat1 300000) ; or with nat2
then `nat1` would work fine, whereas `nat2` will likely run into
DrRacket's memory limit and the computation will be terminated. The
problem is that `nat2` uses the `nats` value *after* the `listref`
call, which will make a reference to the head of the list, preventing it
from being garbagecollected while `listref` is `cdr`ing down the list
and making more cons cells materialize.
It's still possible to show the extra information though  just save
it:
;; we want to provide some information: show the first element
(define (nat3 n)
(define nats (cons 1 (map add1 nats)))
(define fst (first nats))
(if (number? (listref nats n))
"a number"
(error 'nat "the list starting with ~s is broken" fst)))
It looks like it's spending a redundant runtime cycle in the extra
computation, but it's a lazy language so this is not a problem.

# Lazy Evaluation: Shell Examples
There is a very simple and elegant principle in shell programming  we
get a single data type, a character stream, and many small functions,
each doing a single simple job. With these small building blocks, we can
construct more sequences that achieve more complex tasks, for example
 a sorted frequency table of lines in a file:
sort foo  uniq c  sort nr
This is very much like a programming language  we get small blocks,
and build stuff out of them. Of course there are swiss army knives like
awk that try to do a whole bunch of stuff, (the same attitude that
brought Perl to the world...) and even these respect the "stream" data
type. For example, a simple `{ print $1 }` statement will work over all
lines, one by one, making it a program over an infinite input stream,
which is what happens in reality in something like:
cat /dev/console  awk ...
But there is something else in shell programming that makes so
effective: it is implementing a sort of a lazy evaluation. For example,
compare this:
cat foo  awk '{ print $1+$2; }'  uniq
to:
cat foo  awk '{ print $1+$2; }'  uniq  head 5
Each element in the pipe is doing its own small job, and it is always
doing just enough to feed its output. Each basic block is designed to
work even on infinite inputs! (Even sort works on unlimited inputs...)
(Soon we will see a stronger connection with lazy evaluation.)
> Side note: (Alan Perlis) "It is better to have 100 functions operate
> on one data structure than 10 functions on 10 data structures"... But
> the uniformity comes at a price: the biggest problem shells have is in
> their lack of a recursive structure, contaminating the world with way
> too many hacked up solutions. More than that, it is extremely
> inefficient and usually leads to data being reparsed over and over
> and over  each small Unix command needs to always output stuff that
> is human readable, but the next command in the pipe will need to
> reparse that, eg, rereading decimal numbers. If you look at pipelines
> as composing functions, then a pipe of numeric commands translates to
> something like:
>
> itoa(baz(atoi(itoa(bar(atoi(itoa(foo(atoi(inp)))))))))
>
> and it is impossible to get rid of the redundant `atoi(itoa(...))`s.

# Lazy Evaluation: Programming Examples
We already know that when we use lazy evaluation, we are guaranteed to
have more robust programs. For example, a function like:
(define (myif x y z)
(if x y z))
is completely useless in Racket because all functions are eager, but in
a lazy language, it would behave exactly like the real if. Note that we
still need *some* primitive conditional, but this primitive can be a
function (and it is, in Lazy Racket).
But we get more than that. If we have a lazy language, then
*computations* are pushed around as if they were values (computations,
because these are expressions that are yet to be evaluated). In fact,
there is no distinction between computations and values, it just happens
that some values contain "computational promises", things that will do
something in the future.
To see how this happens, we write a simple program to compute the
(infinite) list of prime numbers using the sieve of Eratosthenes. To do
this, we begin by defining the list of all natural numbers:
(define nats (cons 1 (map add1 nats)))
And now define a `sift` function: it receives an integer `n` and an
infinite list of integers `l`, and returns a list without the numbers
that can be divided by `n`. This is simple to write using `filter`:
(define (sift n l)
(filter (lambda (x) (not (divides? n x))) l))
and it requires a definition for `divides?`  we use Racket's `modulo`
for this:
(define (divides? n m)
(zero? (modulo m n)))
Now, a `sieve` is a function that consumes a list that begins with a
prime number, and returns the prime numbers from this list. To do this,
it returns a list that has the same first number, and for its tail it
sifts out numbers that are divisible by the first from the original
list's tail, and calls itself recursively on the result:
(define (sieve l)
(cons (first l) (sieve (sift (first l) (rest l)))))
Finally, the list of prime numbers is the result of applying `sieve` on
the list of numbers from `2`. The whole program is now:
#lang pl lazy
(define nats (cons 1 (map add1 nats)))
(define (divides? n m)
(zero? (modulo m n)))
(define (sift n l)
(filter (lambda (x) (not (divides? n x))) l))
(define (sieve l)
(cons (first l) (sieve (sift (first l) (rest l)))))
(define primes (sieve (rest nats)))
To see how this runs, we trace `modulo` to see which tests are being
used. The effect of this is that each time `divides?` is actually
required to return a value, we will see a line with its inputs, and its
output. This output looks quite tricky  things are computed only on a
"need to know" basis, meaning that debugging lazy programs can be
difficult, since things happen when they are needed which takes time to
get used to. However, note that the program actually performs the same
tests that you'd do using any eagerlanguage implementation of the sieve
of Eratosthenes, and the advantage is that we don't need to decide in
advance how many values we want to compute  all values will be
computed when you want to see the corresponding result. Implementing
*this* behavior in an eager language is more difficult than a simple
program, yet we don't need such complex code when we use lazy
evaluation.
Note that if we trace `divides?` we see results that are some promise
struct  these are unevaluated expressions, and they point at the fact
that when `divides?` is used, it doesn't really force its arguments 
this happens later when these results are forced.
The analogy with shell programming using pipes should be clear now 
for example, we have seen this:
cat foo  awk '{ print $1+$2; }'  uniq  head 5
The last `head 5` means that no computation is done on parts of the
original file that are not needed. It is similar to a `(take 5 l)`
expression in Lazy Racket.

## Side Note: Similarity to Generators and Channels
Using infinite lists is similar to using channels  a tool for
synchronizing threads and (see a [Rob Pike's talk]), and generators (as
they exist in Python). Here are examples of both, note how similar they
both are, and how similar they are to the above definition of `primes`.
(But note that there is an important difference, can you see it? It has
to be with whether a stream is reusable or not.)
[Rob Pike's talk]: https://www.youtube.com/watch?v=hB05UFqOtFA&t=1680s
First, the threads & channels version:
#lang racket
(definesyntaxrule (bg expr ...) (thread (lambda () expr ...)))
(define nats
(let ([out (makechannel)])
(define (loop i) (channelput out i) (loop (add1 i)))
(bg (loop 1))
out))
(define (divides? n m)
(zero? (modulo m n)))
(define (filter pred c)
(define out (makechannel))
(define (loop)
(let ([x (channelget c)])
(when (pred x) (channelput out x))
(loop)))
(bg (loop))
out)
(define (sift n c)
(filter (lambda (x) (not (divides? n x))) c))
(define (sieve c)
(define out (makechannel))
(define (loop c)
(define first (channelget c))
(channelput out first)
(loop (sift first c)))
(bg (loop c))
out)
(define primes
(begin (channelget nats) (sieve nats)))
(define (take n c)
(if (zero? n) null (cons (channelget c) (take (sub1 n) c))))
(take 10 primes)
And here is the generator version:
#lang racket
(require racket/generator)
(define nats
(generator ()
(define (loop i)
(yield i)
(loop (add1 i)))
(loop 1)))
(define (divides? n m)
(zero? (modulo m n)))
(define (filter pred g)
(generator ()
(define (loop)
(let ([x (g)])
(when (pred x) (yield x))
(loop)))
(loop)))
(define (sift n g)
(filter (lambda (x) (not (divides? n x))) g))
(define (sieve g)
(define (loop g)
(define first (g))
(yield first)
(loop (sift first g)))
(generator () (loop g)))
(define primes
(begin (nats) (sieve nats)))
(define (take n g)
(if (zero? n) null (cons (g) (take (sub1 n) g))))
(take 10 primes)

# Call by Need vs Call by Name
Finally, note that on requiring different parts of the `primes`, the
same calls are not repeated. This indicates that our language implements
"call by need" rather than "call by name": once an expression is forced,
its value is remembered, so subsequent usages of this value do not
require further computations.
Using "call by name" means that we actually use expressions which can
lead to confusing code. An old programming language that used this is
Algol. A confusing example that demonstrates this evaluation strategy
is:
#lang algol60
begin
integer procedure SIGMA(x, i, n);
value n;
integer x, i, n;
begin
integer sum;
sum := 0;
for i := 1 step 1 until n do
sum := sum + x;
SIGMA := sum;
end;
integer q;
printnln(SIGMA(q*21, q, 7));
end
`x` and `i` are arguments that are passed by name, which means that they
can use the same memory location. This is called *aliasing*, a problem
that happens when pointers are involved (eg, pointers in C and
`reference` arguments in C++). The code, BTW, is called "Jensen's
device".

# Example of Feature Embedding
Another interesting behavior that we can now observe, is that the TOY
evaluation rule for `with`:
eval({with {x E1} E2}) = eval(E2[eval(E1)/x])
is specifying an eager evaluator *only if* the language that this rule
is written in is itself eager. Indeed, if we run the TOY interpreter in
Lazy Racket (or other interpreters we have implemented), we can verify
that running:
(run "{bind {{x {/ 1 0}}} 1}")
is perfectly fine  the call to Racket's division is done in the
evaluation of the TOY division expression, but since Lazy Racket is
lazy, then if this value is never used then we never get to do this
division! On the other hand, if we evaluate
(run "{bind {{x {/ 1 0}}} {+ x 1}}")
we do get an error when DrRacket tries to display the result, which
forces strictness. Note how the arrows in DrRacket that show where the
computation is are quite confusing: the computation seem to go directly
to the point of the arithmetic operations (`arithop`) since the rest of
the evaluation that the evaluator performed was already done, and
succeeded. The actual failure happens when we try to force the resulting
promise which contains only the strict points in our code.

# Implementing Laziness (in plain Racket)
> [PLAI §8]
Generally, we know how lazy evaluation works when we use the
substitution model. We even know that if we have:
{bind {{x y}}
{bind {{y 2}}
{+ x y}}}
then the result should be an error because we cannot substitute the `y`
expression in because it will capture the `y`  changing the binding
structure. As an indication, the original expression contains a free
reference to `y`, which is exactly why we shouldn't substitute it. But
what about:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}}}
Evaluating this eagerly returns 18, we therefore expect any other
evaluation (eager or lazy, using substitutions or environments) to
return 18 too, because any of these options should not change the
meaning of numbers, of addition, *or* of the scoping rules. (And we know
that no matter what evaluation strategy we choose, if we get to a value
(no infinite loop or exception) then it'll always be the same value.)
For example, try using lazy evaluation with substitutions:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}}}
>
{bind {{y {+ {+ 4 5} {+ 4 5}}}}
{bind {{z y}}
{bind {{x 4}}
z}}}
>
{bind {{z {+ {+ 4 5} {+ 4 5}}}}
{bind {{x 4}}
z}}
>
{bind {{x 4}}
{+ {+ 4 5} {+ 4 5}}}
>
{+ {+ 4 5} {+ 4 5}}
>
{+ 9 9}
>
18
And what about lazy evaluation using environments:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}}} []
>
{bind {{y {+ x x}}}
{bind {{z y}}
{bind {{x 4}}
z}}} [x:={+ 4 5}]
>
{bind {{z y}}
{bind {{x 4}}
z}} [x:={+ 4 5}, y:={+ x x}]
>
{bind {{x 4}}
z} [x:={+ 4 5}, y:={+ x x}, z:=y]
>
z [x:=4, y:={+ x x}, z:=y]
>
y [x:=4, y:={+ x x}, z:=y]
>
{+ x x} [x:=4, y:={+ x x}, z:=y]
>
{+ 4 4} [x:=4, y:={+ x x}, z:=y]
>
8 [x:=4, y:={+ x x}, z:=y]
We have a problem! This problem should be familiar now, it is very
similar to the problem that led us down the mistaken path of dynamic
scoping when we tried to have firstclass functions. In both cases,
substitution always worked, and it looks like in both cases the problem
is that we don't remember the environment of an expression: in the case
of functions, it is the environment at the time of creating the closure
that we want to capture and use when we go back later to evaluate the
body of the function. Here we have a similar situation, except that we
don't need a function to defer computation: *most* expressions get
evaluated at some time in the future, so every time we defer such a
computation we need to remember the lexical environment of the
expression.
This is the major point that will make things work again: every
expression creates something like a closure  an object that closes
over an expression and an environment at the (lexical) place where that
expression was used, and when we actually want to evaluate it later, we
need to do it in the right lexical context. So it is like a closure
except it doesn't need to be applied, and there are no arguments. In
fact it is also a form of a closure  instead of closing over a
function body and an environment, it closes over any expression and an
environment. (As we shall see, lazy evaluation is tightly related to
using nullary functions: *thunks*.)

# Sloth: A Lazy Evaluator
So we implement this by creating such closure values for all expressions
that are not evaluated right now. We begin with the Toy language, and
rename it to "Sloth". We then add one more case to the data type of
values which implements the new kind of expression closures, which
contains the expression and its environment:
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV] ;*** new: expression and scope
[PrimV ((Listof VAL) > VAL)])
(Intuition#1: `ExprV` is a delayed evaluation and therefore it has the
two values that are ultimately passed to `eval`. Intuition#2: laziness
can be implemented with thunks, so we hold the same information as a
`FunV` does, only there's no need for the argument names.)
Where should we use the new `ExprV`?  At any place where we want to
be lazy and defer evaluating an expression for later. The two places in
the interpreter where we want to delay evaluation are the named
expressions in a bind form and the argument expressions in a function
application. Both of these cases use the helper `eval*` function to do
their evaluations, for example:
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
To delay these evaluations, we need to change `eval*` so it returns an
expression closure instead of actually doing the evaluation  change:
(: eval* : SLOTH > VAL)
(define (eval* expr) (eval expr env))
to:
(: eval* : SLOTH > VAL)
(define (eval* expr) (ExprV expr env))
Note how simple this change is  instead of an `eval` function call,
we create a value that contains the parts that would have been used in
the `eval` function call. This value serves as a promise to do this
evaluation (the `eval` call) later, if needed. (This is exactly why a
Lazy Racket would make this a lazy evaluator: in it, *all* function
calls are promises.)

Side note: this can be used in any case when you're using an eager
language, and you want to delay some function call  all you need to
do is replace (using a Cish syntax)
int foo(int x, str y) {
...do some work...
}
with
// rename `foo':
int real_foo(int x, str y) {
...same work...
}
// `foo' is a delayed constructor, instead of a plain function
struct delayed_foo {
int x;
str y;
}
delayed_foo foo(int x, str y) {
return new delayed_foo(x, y);
}
now all calls to `foo` return a `delayed_foo` instance instead of an
integer. Whenever we want to force the delayed promise, we can use this
function:
int force_foo(delayed_foo promise) {
return real_foo(promise.x, promise.y);
}
You might even want to make sure that each such promise is evaluated
exactly once  this is simple to achieve by adding a cache field to
the struct:
int real_foo(int x, str y) {
...same work...
}
struct delayed_foo {
int x;
str y;
bool is_computed;
int result;
}
delayed_foo foo(int x, str y) {
return new delayed_foo(x, y, false, 0);
}
int force_foo(delayed_foo promise) {
if (!promise.is_computed) {
promise.result = real_foo(promise.x, promise.y);
promise.is_computed = true;
}
return promise.result;
}
As we will see shortly, this corresponds to switching from a
callbyname lazy language to a callbyneed one.

Back to our Sloth interpreter  given the `eval*` change, we expect
that `eval`uating:
{bind {{x 1}} x}
will return:
(ExprV (Num 1) ...theglobalenvironment...)
and the same goes for `eval`uating
{{fun {x} x} 1}
Similarly, evaluating
{bind {{x {+ 1 2}}} x}
should return
(ExprV (Call (Id +) (Num 1) (Num 2)) ...theglobalenvironment...)
But what about evaluating an expression like this one:
{bind {{x 2}}
{+ x x}}
?
Using what we have so far, we will get to evaluate the body, which is a
(Call ...) expression, but when we evaluate the arguments for this
function call, we will get `ExprV` values  so we will not be able to
perform the addition. Instead, we will get an error from the function
that `racketfunc>primval` creates, due to the value being an `ExprV`
instead of a `RktV`.
What we really want is to actually add two *values*, not promises. So
maybe distinguish the two applications  treat `PrimV` differently
from `FunV` closures?
(: eval* : SLOTH > VAL)
(define (eval* expr) (ExprV expr env))
(: realeval* : SLOTH > VAL)
(define (realeval* expr) (eval expr env))
(cases expr
...
[(Call funexpr argexprs)
(let ([fval (eval funexpr env)]
;; move: [argvals (map eval* argexprs)]
)
(cases fval
[(PrimV proc) (proc (map realeval* argexprs))] ; change
[(FunV names body funenv)
(eval body (extend names (map eval* argexprs) funenv))]
...))]
...)
This still doesn't work  the problem is that the function now gets a
bunch of values, where some of these can still be `ExprV`s because the
evaluation itself can return such values... Another way to see this
problem is to consider the code for evaluating an `If` conditional
expression:
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (realeval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]
...we need to take care of a possible `ExprV` here. What should we do?
The obvious solution is to use `eval` if we get an `ExprV` value:
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (realeval* condexpr)
[(RktV v) v] ; Racket value => use as boolean
[(ExprV expr env) (eval expr env)] ; force a promise
[else #t]) ; other values are always true
thenexpr
elseexpr))]
Note how this translates back the data structure that represents a
delayed `eval` promise back into a real `eval` call...
Going back to our code for `Call`, there is a problem with it  the
(define (realeval* expr) (eval expr env))
will indeed evaluate the expression instead of lazily deferring this to
the future, but this evaluation might itself return such lazy values. So
we need to inspect the resulting value again, forcing the promise if
needed:
(define (realeval* expr)
(let ([val (eval expr env)])
(cases val
[(ExprV expr env) (eval expr env)]
[else val])))
But we *still* have a problem  programs can get an arbitrarily long
nested chains of `ExprV`s that get forced to other `ExprV`s.
{bind {{x true}}
{bind {{y x}}
{bind {{z y}}
{if z
{foo}
{bar}}}}}
What we really need is to write a loop that keeps forcing promises over
and over until it gets a proper non`ExprV` value.
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise,
;; returns a VAL that is not an ExprV
(define (strict val)
(cases val
[(ExprV expr env) (strict (eval expr env))] ; loop back
[else val]))
Note that it's close to `realeval*`, but there's no need to mix it with
`eval`. The recursive call is important: we can never be sure that
`eval` didn't return an `ExprV` promise, so we have to keep looping
until we get a "real" value.
Now we can change the evaluation of function calls to something more
manageable:
[(Call funexpr argexprs)
(let ([fval (strict (eval* funexpr))] ;*** strict!
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc (map strict argvals))] ;*** strict!
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
The code is fairly similar to what we had previously  the only
difference is that we wrap a `strict` call where a proper value is
needed  the function value itself, and arguments to primitive
functions.
The `If` case is similar (note that it doesn't matter if `strict` is
used with the result of `eval` or `eval*` (which returns an `ExprV`)):
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (strict (eval* condexpr))
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]
Note that, like before, we always return `#t` for non`RktV` values 
this is because we know that the value there is never an `ExprV`. All we
need now to get a working evaluator, is one more strictness point: the
outermost point that starts our evaluation  `run`  needs to use
`strict` to get a proper result value.
(: run : String > Any)
;; evaluate a SLOTH program contained in a string
(define (run str)
(let ([result (strict (eval (parse str) globalenvironment))])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
With this, all of the tests that we took from the Toy evaluator run
successfully. To make sure that the interpreter is lazy, we can add a
test that will fail if the language is strict:
;; Test laziness
(test (run "{{fun {x} 1} {/ 9 0}}") => 1)
(test (run "{{fun {x} 1} {{fun {x} {x x}} {fun {x} {x x}}}}") => 1)
(test (run "{bind {{x {{fun {x} {x x}} {fun {x} {x x}}}}} 1}") => 1)
[In fact, we can continue and replace all `eval` calls with `ExprV`,
leaving only the one call in `strict`. This doesn't make any difference,
because the resulting promises will eventually be forced by `strict`
anyway.]

## Getting more from Sloth
As we've seen, using `strict` in places where we need an actual value
rather than a delayed promise is enough to get a working lazy evaluator.
Our current implementation assumes that all primitive functions need
strict values, therefore the argument values are all passed through the
`strict` function  but this is not always the case. Specifically, if
we have constructor functions, then we don't need (and usually don't
want) to force the promises. This is basically what allows us to use
infinite lists in Lazy Racket: the fact that `list` and `cons` do not
require forcing their arguments.
To allow some primitive functions to consume strict values and some to
leave them as is, we're going to change `racketfunc>primval` and add
a flag that indicates whether the primitive function is strict or not.
Obviously, we also need to move the `strict` call around arguments to a
primitive function application into the `racketfunc>primval`
generated function  which simplifies the `Call` case in `eval` (we go
from (proc (map strict argvals)) back to (proc argvals)). The new code
for `racketfunc>primval` and its helper is:
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function ...
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict?
(map unwraprktv args)
args)]) ;*** use values as is!
(RktV (listfunc args))))))
We now need to annotate the primitives in the global environment, as
well as add a few constructors:
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval + #t))
(list ' (racketfunc>primval  #t))
(list '* (racketfunc>primval * #t))
(list '/ (racketfunc>primval / #t))
(list '< (racketfunc>primval < #t))
(list '> (racketfunc>primval > #t))
(list '= (racketfunc>primval = #t))
;; note flags:
(list 'cons (racketfunc>primval cons #f))
(list 'list (racketfunc>primval list #f))
(list 'first (racketfunc>primval car #t)) ;**
(list 'rest (racketfunc>primval cdr #t)) ;**
(list 'null? (racketfunc>primval null? #t))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f))
(list 'null (RktV null)))
(EmptyEnv)))
Note that this last change raises a subtle type issue: we're actually
abusing the Racket `list` and `cons` constructors to hold Sloth values.
One way in which this becomes a problem is the current assumption that a
primitive function always returns a Racket value (it is always wrapped
in a `RktV`)  but this is no longer the case for `first` and `rest`:
when we use
{cons 1 null}
in Sloth, the resulting value will be
(RktV (cons (ExprV (Num 1) ...) (ExprV (Id null) ...)))
This leads to two problems: first, if we use Racket's `first` and
`rest`, they will complain (throw a runtime error) since the input value
is not a *proper* list (it's a pair that has a nonlist value in its
tail). To resolve that, we use the more primitive `car` and `cdr`
functions to implement Sloth's `first` and `rest`.
The second problem happens when we try and grab the first value of this
{first {cons 1 null}}
we will eventually get back the `ExprV` and wrap it in a `RktV`:
(RktV (ExprV (Num 1) ...))
and finally `run` will strip off the `RktV` and return the `ExprV`. A
solution to this is to make our `first` and `rest` functions return a
value *without* wrapping it in a `RktV`  we can identify this
situation by the fact that the returned value is already a VAL instead
of some other Racket value. We can identify such values with the `VAL?`
predicate that gets defined by our `definetype`, implemented by a new
`wrapinval` helper:
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: wrapinval : Any > VAL)
;; helper that ensures a VAL output using RktV wrapper when needed,
;; but leaving as is otherwise
(define (wrapinval x)
(if (VAL? x) x (RktV x)))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function ...
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict? (map unwraprktv args) args)])
(wrapinval (listfunc args))))))
Note that we don't need to worry about the result being an `ExprV` 
that will eventually be taken care of by `strict`.

## The Sloth Implementation
The complete Sloth code follows. It can be used to do the same fun
things we did with Lazy Racket.
;;; <<>>
#lang pl
;;; 
;;; Syntax
# The BNF:
::=

 { bind {{ } ... } }
 { fun { ... } }
 { if }
 { ... }
#
;; A matching abstract syntax tree datatype:
(definetype SLOTH
[Num Number]
[Id Symbol]
[Bind (Listof Symbol) (Listof SLOTH) SLOTH]
[Fun (Listof Symbol) SLOTH]
[Call SLOTH (Listof SLOTH)]
[If SLOTH SLOTH SLOTH])
(: uniquelist? : (Listof Any) > Boolean)
;; Tests whether a list is unique, guards Bind and Fun values.
(define (uniquelist? xs)
(or (null? xs)
(and (not (member (first xs) (rest xs)))
(uniquelist? (rest xs)))))
(: parsesexpr : Sexpr > SLOTH)
;; parses sexpressions into SLOTHs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'bind more)
(match sexpr
[(list 'bind (list (list (symbol: names) (sexpr: nameds))
...)
body)
(if (uniquelist? names)
(Bind names (map parsesexpr nameds) (parsesexpr body))
(error 'parsesexpr "duplicate `bind' names: ~s" names))]
[else (error 'parsesexpr "bad `bind' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: names) ...) body)
(if (uniquelist? names)
(Fun names (parsesexpr body))
(error 'parsesexpr "duplicate `fun' names: ~s" names))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(cons 'if more)
(match sexpr
[(list 'if cond then else)
(If (parsesexpr cond)
(parsesexpr then)
(parsesexpr else))]
[else (error 'parsesexpr "bad `if' syntax in ~s" sexpr)])]
[(list fun args ...) ; other lists are applications
(Call (parsesexpr fun)
(map parsesexpr args))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > SLOTH)
;; Parses a string containing an SLOTH expression to a SLOTH AST.
(define (parse str)
(parsesexpr (string>sexpr str)))
;;; 
;;; Values and environments
(definetype ENV
[EmptyEnv]
[FrameEnv FRAME ENV])
;; a frame is an association list of names and values.
(definetype FRAME = (Listof (List Symbol VAL)))
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV]
[PrimV ((Listof VAL) > VAL)])
(: extend : (Listof Symbol) (Listof VAL) ENV > ENV)
;; extends an environment with a new frame.
(define (extend names values env)
(if (= (length names) (length values))
(FrameEnv (map (lambda ([name : Symbol] [val : VAL])
(list name val))
names values)
env)
(error 'extend "arity mismatch for names: ~s" names)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, frame by frame,
;; return its value or throw an error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(FrameEnv frame rest)
(let ([cell (assq name frame)])
(if cell
(second cell)
(lookup name rest)))]))
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: wrapinval : Any > VAL)
;; helper that ensures a VAL output using RktV wrapper when needed,
;; but leaving as is otherwise
(define (wrapinval x)
(if (VAL? x) x (RktV x)))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function
;; which is a PrimV holding a ((Listof VAL) > VAL) function.
;; (the resulting function will use the list function as is,
;; and it is the list function's responsibility to throw an error
;; if it's given a bad number of arguments or bad input types.)
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict? (map unwraprktv args) args)])
(wrapinval (listfunc args))))))
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval + #t))
(list ' (racketfunc>primval  #t))
(list '* (racketfunc>primval * #t))
(list '/ (racketfunc>primval / #t))
(list '< (racketfunc>primval < #t))
(list '> (racketfunc>primval > #t))
(list '= (racketfunc>primval = #t))
;; note flags:
(list 'cons (racketfunc>primval cons #f))
(list 'list (racketfunc>primval list #f))
(list 'first (racketfunc>primval car #t))
(list 'rest (racketfunc>primval cdr #t))
(list 'null? (racketfunc>primval null? #t))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f))
(list 'null (RktV null)))
(EmptyEnv)))
;;; 
;;; Evaluation
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise, returns a VAL that is
;; not an ExprV
(define (strict val)
(cases val
[(ExprV expr env) (strict (eval expr env))]
[else val]))
(: eval : SLOTH ENV > VAL)
;; evaluates SLOTH expressions
(define (eval expr env)
;; convenient helper
(: eval* : SLOTH > VAL)
(define (eval* expr) (ExprV expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (strict (eval* funexpr))]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (strict (eval* condexpr))
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
(: run : String > Any)
;; evaluate a SLOTH program contained in a string
(define (run str)
(let ([result (strict (eval (parse str) globalenvironment))])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
;;; 
;;; Tests
(test (run "{{fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}} {add3 1}}")
=> 4)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}
{add1 {fun {x} {+ x 1}}}}
{bind {{x 3}} {add1 {add3 x}}}}")
=> 7)
(test (run "{bind {{identity {fun {x} x}}
{foo {fun {x} {+ x 1}}}}
{{identity foo} 123}}")
=> 124)
(test (run "{bind {{x 3}}
{bind {{f {fun {y} {+ x y}}}}
{bind {{x 5}}
{f 4}}}}")
=> 7)
(test (run "{{{fun {x} {x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)
;; More tests for complete coverage
(test (run "{bind x 5 x}") =error> "bad `bind' syntax")
(test (run "{fun x x}") =error> "bad `fun' syntax")
(test (run "{if x}") =error> "bad `if' syntax")
(test (run "{}") =error> "bad syntax")
(test (run "{bind {{x 5} {x 5}} x}") =error> "duplicate*bind*names")
(test (run "{fun {x x} x}") =error> "duplicate*fun*names")
(test (run "{+ x 1}") =error> "no binding for")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{1 2}") =error> "with a nonfunction")
(test (run "{{fun {x} x}}") =error> "arity mismatch")
(test (run "{if {< 4 5} 6 7}") => 6)
(test (run "{if {< 5 4} 6 7}") => 7)
(test (run "{if + 6 7}") => 6)
(test (run "{fun {x} x}") =error> "returned a bad value")
;; Test laziness
(test (run "{{fun {x} 1} {/ 9 0}}") => 1)
(test (run "{{fun {x} 1} {{fun {x} {x x}} {fun {x} {x x}}}}") => 1)
(test (run "{bind {{x {{fun {x} {x x}} {fun {x} {x x}}}}} 1}") => 1)
;; Test lazy constructors
(test (run "{bind {{l {list 1 {/ 9 0} 3}}}
{+ {first l} {first {rest {rest l}}}}}")
=> 4)
;;; 

## Shouldn't there be more `ExprV` promises?
You might notice that there are some apparently missing promises. For
example, consider our evaluation of `Bind` forms:
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
The named expressions are turned into expression promises via `eval*`,
but shouldn't we change the first `eval` (the one that evaluates the
body) into a promise too? This is a confusing point, and the bottom line
is that there is no need to create a promise there. The main idea is
that the `eval` function is actually called from contexts that actually
*need* to be evaluated. One example is when we force a promise via
`strict`, and another one is when `run` calls `eval`. Note that in both
of these cases, we actuallly need a (forced) value, so creating a
promise in there doesn't make any difference.
To see this differently, consider how `bind` might be used *within* the
language. The first case is when `bind` is the topmost expression, or
part of a `bind` "spine":
{bind {{x ...}}
{bind {{y ...}}
...}}
In these cases we evaluate the `bind` expression when we need to return
a result for the whole run, so adding an `ExprV` is not going to make a
difference. The second case is when `bind` is used in an expression line
a function argument:
{foo {bind {{x ...}} ...}}
Here there is also no point in adding an `ExprV` to the `Bind` case,
since the evaluation of the whole argument (the `Bind` value) will be
wrapped in an `ExprV`, so it is already delayed. (And when it get
forced, we will need to do the `bind` evaluation anyway, so again, it
adds no value.)
A generalization of this is that when we actually call `eval` (either
directly or via `strict`), there is never any point in making the result
that it returns a promise.
(And if you'll follow this carefully and look at all of the `eval`
calls, you will see that this means that *neither* of the `eval*`s in
the `If` case are needed!)

# Implementing Call by Need
As we have seen, there are a number of advantages for lazy evaluation,
but its main disadvantage is the fact that it is extremely inefficient,
to the point of rendering lots of programs impractical, for example, in:
{bind {{x {+ 4 5}}}
{bind {{y {+ x x}}}
y}}
we end up adding 4 and 5 twice. In other words, we don't suffer from
textual redundancy (each expression is written once), but we don't avoid
dynamic redundancy. We can get it back by simply caching evaluation
results, using a box that will be used to remember the results. The box
will initially hold `#f`, and it will change to hold the VAL that
results from evaluation:
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV (Boxof (U #f VAL))] ;*** new: mutable cache field
[PrimV ((Listof VAL) > VAL)])
We need a utility function to create an evaluation promise, because when
an `ExprV` is created, its initial cache box needs to be initialized.
(: evalpromise : SLOTH ENV > VAL)
;; used instead of `eval' to create an evaluation promise
(define (evalpromise expr env)
(ExprV expr env (box #f)))
(And note that Typed Racket needs to figure out that the `#f` in this
definition has a type of `(U #f VAL)` and not just `#f`.)
This `evalpromise` is used instead of `ExprV` in eval. Finally,
whenever we force such an `ExprV` promise, we need to check if it was
already evaluated, otherwise force it and cache the result. This is
simple to do since there is a single field that is used both as a flag
and a cached value:
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise, returns a VAL that is
;; not an ExprV
(define (strict val)
(cases val
[(ExprV expr env cache)
(or (unbox cache)
(let ([val* (strict (eval expr env))])
(setbox! cache val*)
val*))]
[else val]))
But note that this makes using sideeffects in our interpreter even more
confusing. (It was true with callbyname too.)
The resulting code follows.
;;; <<>>
;; A callbyneed version of the SLOTH interpreter
#lang pl
;;; 
;;; Syntax
# The BNF:
::=

 { bind {{ } ... } }
 { fun { ... } }
 { if }
 { ... }
#
;; A matching abstract syntax tree datatype:
(definetype SLOTH
[Num Number]
[Id Symbol]
[Bind (Listof Symbol) (Listof SLOTH) SLOTH]
[Fun (Listof Symbol) SLOTH]
[Call SLOTH (Listof SLOTH)]
[If SLOTH SLOTH SLOTH])
(: uniquelist? : (Listof Any) > Boolean)
;; Tests whether a list is unique, guards Bind and Fun values.
(define (uniquelist? xs)
(or (null? xs)
(and (not (member (first xs) (rest xs)))
(uniquelist? (rest xs)))))
(: parsesexpr : Sexpr > SLOTH)
;; parses sexpressions into SLOTHs
(define (parsesexpr sexpr)
(match sexpr
[(number: n) (Num n)]
[(symbol: name) (Id name)]
[(cons 'bind more)
(match sexpr
[(list 'bind (list (list (symbol: names) (sexpr: nameds))
...)
body)
(if (uniquelist? names)
(Bind names (map parsesexpr nameds) (parsesexpr body))
(error 'parsesexpr "duplicate `bind' names: ~s" names))]
[else (error 'parsesexpr "bad `bind' syntax in ~s" sexpr)])]
[(cons 'fun more)
(match sexpr
[(list 'fun (list (symbol: names) ...) body)
(if (uniquelist? names)
(Fun names (parsesexpr body))
(error 'parsesexpr "duplicate `fun' names: ~s" names))]
[else (error 'parsesexpr "bad `fun' syntax in ~s" sexpr)])]
[(cons 'if more)
(match sexpr
[(list 'if cond then else)
(If (parsesexpr cond)
(parsesexpr then)
(parsesexpr else))]
[else (error 'parsesexpr "bad `if' syntax in ~s" sexpr)])]
[(list fun args ...) ; other lists are applications
(Call (parsesexpr fun)
(map parsesexpr args))]
[else (error 'parsesexpr "bad syntax in ~s" sexpr)]))
(: parse : String > SLOTH)
;; Parses a string containing an SLOTH expression to a SLOTH AST.
(define (parse str)
(parsesexpr (string>sexpr str)))
;;; 
;;; Values and environments
(definetype ENV
[EmptyEnv]
[FrameEnv FRAME ENV])
;; a frame is an association list of names and values.
(definetype FRAME = (Listof (List Symbol VAL)))
(definetype VAL
[RktV Any]
[FunV (Listof Symbol) SLOTH ENV]
[ExprV SLOTH ENV (Boxof (U #f VAL))]
[PrimV ((Listof VAL) > VAL)])
(: extend : (Listof Symbol) (Listof VAL) ENV > ENV)
;; extends an environment with a new frame.
(define (extend names values env)
(if (= (length names) (length values))
(FrameEnv (map (lambda ([name : Symbol] [val : VAL])
(list name val))
names values)
env)
(error 'extend "arity mismatch for names: ~s" names)))
(: lookup : Symbol ENV > VAL)
;; lookup a symbol in an environment, frame by frame,
;; return its value or throw an error if it isn't bound
(define (lookup name env)
(cases env
[(EmptyEnv) (error 'lookup "no binding for ~s" name)]
[(FrameEnv frame rest)
(let ([cell (assq name frame)])
(if cell
(second cell)
(lookup name rest)))]))
(: unwraprktv : VAL > Any)
;; helper for `racketfunc>primval': strict and unwrap a RktV
;; wrapper in preparation to be sent to the primitive function
(define (unwraprktv x)
(let ([s (strict x)])
(cases s
[(RktV v) v]
[else (error 'racketfunc "bad input: ~s" s)])))
(: wrapinval : Any > VAL)
;; helper that ensures a VAL output using RktV wrapper when needed,
;; but leaving as is otherwise
(define (wrapinval x)
(if (VAL? x) x (RktV x)))
(: racketfunc>primval : Function Boolean > VAL)
;; converts a racket function to a primitive evaluator function
;; which is a PrimV holding a ((Listof VAL) > VAL) function.
;; (the resulting function will use the list function as is,
;; and it is the list function's responsibility to throw an error
;; if it's given a bad number of arguments or bad input types.)
(define (racketfunc>primval racketfunc strict?)
(define listfunc (makeuntypedlistfunction racketfunc))
(PrimV (lambda (args)
(let ([args (if strict? (map unwraprktv args) args)])
(wrapinval (listfunc args))))))
;; The global environment has a few primitives:
(: globalenvironment : ENV)
(define globalenvironment
(FrameEnv (list (list '+ (racketfunc>primval + #t))
(list ' (racketfunc>primval  #t))
(list '* (racketfunc>primval * #t))
(list '/ (racketfunc>primval / #t))
(list '< (racketfunc>primval < #t))
(list '> (racketfunc>primval > #t))
(list '= (racketfunc>primval = #t))
;; note flags:
(list 'cons (racketfunc>primval cons #f))
(list 'list (racketfunc>primval list #f))
(list 'first (racketfunc>primval car #t))
(list 'rest (racketfunc>primval cdr #t))
(list 'null? (racketfunc>primval null? #t))
;; values
(list 'true (RktV #t))
(list 'false (RktV #f))
(list 'null (RktV null)))
(EmptyEnv)))
;;; 
;;; Evaluation
(: evalpromise : SLOTH ENV > VAL)
;; used instead of `eval' to create an evaluation promise
(define (evalpromise expr env)
(ExprV expr env (box #f)))
(: strict : VAL > VAL)
;; forces a (possibly nested) ExprV promise, returns a VAL that is
;; not an ExprV
(define (strict val)
(cases val
[(ExprV expr env cache)
(or (unbox cache)
(let ([val* (strict (eval expr env))])
(setbox! cache val*)
val*))]
[else val]))
(: eval : SLOTH ENV > VAL)
;; evaluates SLOTH expressions
(define (eval expr env)
;; convenient helper
(: eval* : SLOTH > VAL)
(define (eval* expr) (evalpromise expr env))
(cases expr
[(Num n) (RktV n)]
[(Id name) (lookup name env)]
[(Bind names exprs boundbody)
(eval boundbody (extend names (map eval* exprs) env))]
[(Fun names boundbody)
(FunV names boundbody env)]
[(Call funexpr argexprs)
(let ([fval (strict (eval* funexpr))]
[argvals (map eval* argexprs)])
(cases fval
[(PrimV proc) (proc argvals)]
[(FunV names body funenv)
(eval body (extend names argvals funenv))]
[else (error 'eval "function call with a nonfunction: ~s"
fval)]))]
[(If condexpr thenexpr elseexpr)
(eval* (if (cases (strict (eval* condexpr))
[(RktV v) v] ; Racket value => use as boolean
[else #t]) ; other values are always true
thenexpr
elseexpr))]))
(: run : String > Any)
;; evaluate a SLOTH program contained in a string
(define (run str)
(let ([result (strict (eval (parse str) globalenvironment))])
(cases result
[(RktV v) v]
[else (error 'run "evaluation returned a bad value: ~s"
result)])))
;;; 
;;; Tests
(test (run "{{fun {x} {+ x 1}} 4}")
=> 5)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}} {add3 1}}")
=> 4)
(test (run "{bind {{add3 {fun {x} {+ x 3}}}
{add1 {fun {x} {+ x 1}}}}
{bind {{x 3}} {add1 {add3 x}}}}")
=> 7)
(test (run "{bind {{identity {fun {x} x}}
{foo {fun {x} {+ x 1}}}}
{{identity foo} 123}}")
=> 124)
(test (run "{bind {{x 3}}
{bind {{f {fun {y} {+ x y}}}}
{bind {{x 5}}
{f 4}}}}")
=> 7)
(test (run "{{{fun {x} {x 1}}
{fun {x} {fun {y} {+ x y}}}}
123}")
=> 124)
;; More tests for complete coverage
(test (run "{bind x 5 x}") =error> "bad `bind' syntax")
(test (run "{fun x x}") =error> "bad `fun' syntax")
(test (run "{if x}") =error> "bad `if' syntax")
(test (run "{}") =error> "bad syntax")
(test (run "{bind {{x 5} {x 5}} x}") =error> "duplicate*bind*names")
(test (run "{fun {x x} x}") =error> "duplicate*fun*names")
(test (run "{+ x 1}") =error> "no binding for")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{+ 1 {fun {x} x}}") =error> "bad input")
(test (run "{1 2}") =error> "with a nonfunction")
(test (run "{{fun {x} x}}") =error> "arity mismatch")
(test (run "{if {< 4 5} 6 7}") => 6)
(test (run "{if {< 5 4} 6 7}") => 7)
(test (run "{if + 6 7}") => 6)
(test (run "{fun {x} x}") =error> "returned a bad value")
;; Test laziness
(test (run "{{fun {x} 1} {/ 9 0}}") => 1)
(test (run "{{fun {x} 1} {{fun {x} {x x}} {fun {x} {x x}}}}") => 1)
(test (run "{bind {{x {{fun {x} {x x}} {fun {x} {x x}}}}} 1}") => 1)
;; Test lazy constructors
(test (run "{bind {{l {list 1 {/ 9 0} 3}}}
{+ {first l} {first {rest {rest l}}}}}")
=> 4)
;;; 

# Side Effects in a Lazy Language
We've seen that a lazy language without the callbyneed optimization is
too slow to be practical, but the optimization makes using sideeffects
extremely confusing. Specifically, when we deal with sideeffects (I/O,
mutation, errors, etc) the order of evaluation matters, but in our
interpreter expressions are getting evaluated as needed. (Remember
tracing the primenumbers code in Lazy Racket  numbers are tested as
needed, not in order.) If we can't do these things, the question is
whether there is any point in using a purely functional lazy language at
all  since computer programs often interact with an imperative world.
There is a solution for this: the lazy language does not have any (sane)
facilities for *doing* things (like `printf` that prints something in
plain Racket), but it can use a data structure that *describes* such
operations. For example, in Lazy Racket we cannot print stuff sanely
using `printf`, but we can construct a string using `format` (which is
just like `printf`, except that it returns the formatted string instead
of printing it). So (assuming Racket syntax for simplicity), instead of:
(define (foo n)
(printf "~s + 1 = ~s\n" n (+ n 1)))
we will write:
(define (foo n)
(format "~s + 1 = ~s\n" n (+ n 1)))
and get back a string. We can now change the way that our interpreter
deals with the output value that it receives after evaluating a lazy
expression: if it receives a string, then it can take that string as
denoting a request for printout, and simply print it. Such an evaluator
will do the printout when the lazy evaluation is done, and everything
works fine because we don't try to use any sideeffects in the lazy
language  we just describe the desired sideeffects, and constructing
such a description does not require *performing* sideeffects.
But this only solves printing a single string, and nothing else. If we
want to print two strings, then the only thing we can do is concatenate
the two strings  but that is not only inefficient, it cannot describe
infinite output (since we will not be able to construct the infinite
string in memory). So we need a better way to chain several printout
representations. One way to do so is to use a list of strings, but to
make things a little easier to manage, we will create a type for I/O
descriptions  and populate it with one variant holding a string (for
plain printout) and one for holding a chain of two descriptions (which
can be used to construct an arbitrarily long sequence of descriptions):
(definetype IO
[Print String]
[Begin2 IO IO])
Now we can use this to chain any number of printout representations by
turning them into a single `Begin2` request, which is very similar to
simply using a loop to print the list. For example, the eager printout
code:
(: printlist : (Listof A) > Void)
(define (printlist l)
(if (null? l)
(printf "\n")
(begin (printf "~s " (first l))
(printlist (rest l)))))
turns to the following code:
(: printlist : (Listof A) > IO)
(define (printlist l)
(if (null? l)
(Print "\n")
(Begin2 (Print (format "~s " (first l)))
(printlist (rest l)))))
This will basically scan an input list like the eager version, but
instead of printing the list, it will convert it into a single output
request that forms a recipe for this printout. Note that within the lazy
world, the result of `printlist` is just a value, there are no side
effects involved. Turning this value into the actual printout is
something that needs to be done on the eager side, which must be part of
the implementation. In the case of Lazy Racket, we have no access to the
implementation, but we can do so in our Sloth implementation: again,
`run` will inspect the result and either print a given string (if it
gets a `Print` value), or print two things recursively (if it gets a
`Begin2` value). (To implement this, we will add an `IOV` variant to the
`VAL` type definition, and have it contain an `IO` description of the
above type.)
Because the sequence is constructed in the lazy world, it will not
require allocating the whole sequence in memory  it can be forced
bits by bits (using `strict`) as the imperative backend (the `run` part
of the implementation) follows the instructions in the resulting IO
description. More concretely, it will also work on an infinite list: the
translation of an infiniteloop printout function will be one that
returns an infinite IO description tree of `Begin2` values. This loop
will also force only what it needs to print and will go on recursively
printing the whole sequence (possibly not terminating). For example
(again, using Racket syntax), the infinite printout loop
(: printloop : > Void)
(define (printloop)
(printf "foo\n")
(printloop))
is translated into a function that returns an infinite tree of print
operations:
(: printloop : > IO)
(define (printloop)
(Begin2 (Print "foo\n")
(printloop)))
When this tree is converted to actions, it will result in an infinite
loop that produces the same output  it is essentially the same
infinite loop, only now it's derived by an infinite description rather
than an infinite process.
Finally, how should we deal with inputs? We can add another variant to
our type definition that represents a `readline` operation, assuming
that like `readline` it does not require any arguments:
(definetype IO
[Print String]
[ReadLine ]
[Begin2 IO IO])
Now the eager implementation can invoke `readline` when it encounters a
`ReadLine` value  but what should it do with the resulting string?
Even worse, naively binding a value to `ReadLine`
(let ([name (ReadLine)])
(Print (format "Your name is ~a" name)))
doesn't get us the string that is read  instead, the value is a
*description* of a read operation, which is very different from the
actual string value that we want in the binding.
The solution is to take the "code that acts on the string value" and
make *it* be the argument to `ReadLine`. In the above example, that
could would be the `let` expression without the `(ReadLine)` part 
and as you rememebr from the time we introduced `fun` into `WAE`, taking
away a named expression from a binding expression leads to a function.
With this in mind, it makes sense to make `ReadLine` take a function
value that represents what to do in the future, once the reading is
actually done.
(ReadLine (lambda (name)
(Print (format "Your name is ~a" name))))
This receiver value is a kind of a *continuation* of the computation,
provided as a callback value  it will get the string that was read on
the terminal, and will return a new description of sideeffects that
represents the rest of the process:
(definetype IO
[Print String]
[ReadLine (String > IO)]
[Begin2 IO IO])
Now, when the eager side sees a `ReadLine` value, it will read a line,
and invoke the callback function with the string that it has read. By
doing this, the control goes back to the lazy world to process the value
and get back another IO value to continue the processing. This results
in a process where the lazy code generates some IO descriptions, then
the imperative side will execute it and control goes back to the lazy
code, then back to the imperative side, etc.
As a more verbose example of all of the above, this silly loop:
(: sillyloop : > Void)
(define (sillyloop)
(printf "What is your name? ")
(let ([name (readline)])
(if (equal? name "quit")
(printf "bye\n")
(begin (printf "Your name is ~s\n" name)
(sillyloop)))))
is now translated to:
(: sillyloop : > IO)
(define (sillyloop)
(Begin2 (Print "What is your name? ")
(ReadLine
(lambda (name)
(if (equal? name "quit")
(Print "bye\n")
(Begin2 (Print (format "Your name is ~s\n" name))
(sillyloop)))))))
Using this strategy to implement sideeffects is possible, and you will
do that in the homework  some technical details are going to be
different but the principle is the same as discussed above. The last
problem is that the above code is difficult to work with  in the
homework you will see how to use syntactic abstractions to make things
much simpler.

# Designing Domain Specific Languages (DSLs)
> [PLAI §35]
Programming languages differ in numerous ways:
1. Each uses different notations for writing down programs. As we've
observed, however, syntax is only partially interesting. (This is,
however, less true of languages that are trying to mirror the
notation of a particular domain.)
2. Control constructs: for instance, early languages didn't even support
recursion, while most modern languages still don't have
continuations.
3. The kinds of data they support. Indeed, sophisticated languages like
Racket blur the distinction between control and data by making
fragments of control into data values (such as firstclass functions
and continuations).
4. The means of organizing programs: do they have functions, modules,
classes, namespaces, ...?
5. Automation such as memory management, runtime safety checks, and so
on.
Each of these items suggests natural questions to ask when you design
your own languages in particular domains.
Obviously, there are a lot of domain specific languages these days 
and that's not new. For example, four of the oldest languages were
conceived as domain specific languages:
* **Fortran**  *Formula Translator*
* **Algol**  *Algorithmic Language*
* **Cobol**  *Common Business Oriented Language*
* **Lisp**  *List Processing*
Only in the late 60s / early 70s languages began to get free from their
special purpose domain and become *general purpose* languages (GPLs).
These days, we usually use some GPL for our programs and often come up
with small *domain specific* languages (DSLs) for specific jobs. The
problem is designing such a specific language. There are lots of
decisions to make, and as should be clear now, many ways of shooting
your self in the foot. You need to know:
* What is your domain?
* What are the common notations in this domain (need to be convenient
both for the machine and for humans)?
* What do you expect to get from your DSL? (eg, performance gains when
you know that you're dealing with a certain limited kind of
functionality like arithmetics.)
* Do you have any semantic reason for a new language? (For example,
using special scoping rules, or a mixture of lazy and eager
evaluation, maybe a completely different way of evaluation (eg,
makefiles).)
* Is your language expected to envelope other functionality (eg, shell
scripts, TCL), perhaps throwing some functionality on a different
language (makefiles and shell scripts), or is it going to be embedded
in a bigger application (eg, PHP), or embedded in a way that exposes
parts of an application to user automation (Emacs Lisp, Word Basic,
Visual Basic for Office Application or Some Other Long List of
Buzzwords).
* If you have one language embedded in another enveloping language 
how do you handle syntax? How can they communicate (eg, share
variables)?
And very important:
* Is there a benefit for implementing a DSL over using a GPL  how
much will your DSL grow (usually more than you think)? Will it get to
a point where it will need the power of a full GPL? Do you want to
risk doing this just to end up admitting that you need a "Real
Language" and dump your solution for "Visual Basic for Applications"?
(It might be useful to think ahead about things that you know you
don't need, rather than things you need.)
To clarify why this can be applicable in more situations than you think,
consider what programming languages are used for. One example that
should not be ignored is using a programming language to implement a
programming language  for example, what we did so far (or any other
interpreter or compiler). In the same way that some piece of code in a
PL represent functions about the "real world", there are other programs
that represent things in a language  possibly even the same one. To
make a sideeffectfull example, the meaning of `onebrick` might
abstract over laying a brick when making a wall  it abstracts all the
little details into a function:
(define (onebrick wall brickpile)
(moveeye (location brickpile))
(let ([pos (findavailablebrickposition brickpile)])
(movehand pos)
(grabobject))
(moveeye wall)
(let ([pos (findnextbrickposition wall)])
(movehand pos)
(dropobject)))
and we can now write
(onebrick mywall mybrickpile)
instead of all of the above. We might use that in a loop:
(define (buildwall wall pile)
(define (loop n)
(when (< n 500)
(onebrick wall pile)
(loop (add1 n))))
(loop 0))
This is a common piece of looping code that we've seen in many forms,
and a common complaint of newcomers to functional languages is the lack
of some kind of a loop. But once you know the template, writing such
loops is easy  and in fact, you can write code that would take
something like:
(define (buildwall wall pile)
(loopfor i from 1 to 500
(onebrick wall pile)))
and produce the previous code. Note the main point here: we switch from
code that deals with bricks to code that deals with code.
Now, a viable option for implementing a new DSL is to do so by
transforming it into an existing language. Such a process is usually
tedious and error prone  tedious because you need to deal with the
boring parts of a language (making a parser etc), and error prone
because it's easy to generate bad code (especially when you're dealing
with strings) and you get bad errors in terms of the translated code
instead of the actual code, resorting to debugging the intermediate
generated programs. Lisp languages traditionally have taken this idea
one level further than other languages: instead of writing a new
transformer for your language, you use the host language, but you extend
and customize it by adding you own forms.

# Syntax Transformations: Macros
> [PLAI §36]
Macros are one of the biggest advantages of all Lisps, and specifically
even more so an advantage of Scheme implementations, and yet more
specifically, it is a major Racket feature: this section is therefore
specific to Racket (which has this unique feature), although most of
this is the same in most Schemes.
As we have previously seen, it is possible to implement one language
construct using another. What we did could be described as bits of a
compiler, since they translate one language to another.
We will see how this can be done *in* Racket: implementing some new
linguistic forms in terms of ones that are already known. In essence, we
will be translating Racket constructs to other Racket constructs  and
all that is done *in Racket*, no need to go back to the language Racket
was implemented in (C).
This is possible with a simple "trick": the Racket implementation uses
some syntax objects. These objects are implemented somehow inside
Racket's own source code. But these objects are also directly available
for our use  part of the implementation is exposed to our level. This
is quite similar to the way we have implemented pairs in our language
 a TOY or a SLOTH pair is implemented using a Racket pair, so the
*same* data object is available at both levels.
This is the big idea in Lisp, which Scheme (and therefore Racket)
inherited from (to some extent): programs are made of numbers, strings,
symbols and lists of these  and these are all used both at the
metalevel as well as the user level. This means that instead of having
no metalanguage at all (locking away a lot of useful stuff), and
instead of having some crippled halfbaked meta language (CPP being both
the most obvious (as well as the most popular) example for such a meta
language), instead of all this we get exactly the same language at both
levels.
How is this used? Well, the principle is simple. For example, say we
want to write a macro that will evaluate two forms in sequence, but if
the first one returns a result that is not false then it returns it
instead of evaluating the second one too. This is exactly how `or`
behaves, so pretend we don't have it  call our version `orelse`:
(orelse )
in effect, we add a new *special* form to our language, with its own
evaluation rule, only we know how to express this evaluation rule by
translating it to things that are already part of our language.
We could do this as a simple function  only if we're willing to
explicitly delay the arguments with a `lambda`, and use the thunks in
the function:
(define (orelse thunk1 thunk2)
(if (thunk1)
(thunk1) ; ignore the double evaluation for now
(thunk2)))
or:
(define (orelse thunk1 thunk2)
((if (thunk1)
thunk1
thunk2)))
and using it like this:
(orelse (lambda () 1) (lambda () (error "boom")))
But this is clearly not the right way to do this: whoever uses this code
must be aware of the need to send us thunks, and it's verbose and
inconvenient.
> Note that this could be a feasible solution if there was a uniform way
> to have an easy syntactic way to say "a chunk of code" instead of
> immediately execute it  this is exactly what `(lambda () ...)`
> does. So we could, for example, make `{...}` be a shorthand for that,
> which is what Perl6 is doing. However, we will soon see examples
> where we want more than just delay the evaluation of some code.
We want to translate
(orelse )
to>
(if
)
If we look at the code as an sexpression, then we can write the
following function:
(define (translateorelse l)
(if (and (list? l)
(= 3 (length l))
(eq? 'orelse (first l)))
(list 'if (second l) (second l) (third l))
(error 'translateorelse "bad input: ~s" l)))
We can now try it with a simple list:
(translateorelse '(orelse foo1 foo2))
and note that the result is correct.
How is this used? Well, all we need is to hook *our* function into our
implementation's evaluator. In Lisp, we get a `defmacro` form for this,
and many Schemes inherited it or something similar. In Racket, we need
to
(require compatibility/defmacro)
but it requires the transformation to be a little different in a way
that makes life easier: the above contains a lot of boilerplate code.
Usually, we will require the input to be a list of some known length,
the first element to be a symbol that specifies our form, and then do
something with the other arguments. So we'd want to always follow a
template that looks like:
(define (translate??? exprs)
(if (and (list? exprs)
(= N (length exprs))
(eq? '??? (car exprs)))
(let ([E1 (cadr exprs)]
[E2 (caddr exprs)]
...)
...make result expression...)
(error ...)))
But this looks very similar to making sure that a function call is a
specific function call (and for a good reason  macro usages look just
like function calls). So make the translation function get a number of
arguments one each for each part of the input, an sexpression. For
example, the above translation and test become:
(define (translateorelse )
(list 'if ))
(translateorelse 'foo1 'foo2)
The number of arguments is used to check the input (turning an arity
error for the macro to an arity error for the translator function call),
and we don't need to "caddr our way" to arguments.
This gives us the simple definition  but what about the promised
hook?  All we need is to use `definemacro` instead of `define`, and
change the name to the name that will trigger this translation
(providing the last missing test of the input):
(definemacro (orelse )
(list 'if ))
and test it:
(orelse 1 (error "boom"))
Note that this is basically a (usually purely functional) lazy language
of transformations which is built on top of Racket. It is possible for
macros to generate pieces of code that contain references to these same
macros, and they will be used to expand those instances again.
Now we start with the problems, one by one.

# Macro Problems
There is an inherent problem when macros are being used, in any form and
any language (even in CPP): you must remember that you are playing with
*expressions*, not with values  which is why this is problematic:
(define (foo x) (printf "foo ~s!\n" x) x)
(or (foo 1) (foo 2))
(orelse (foo 1) (foo 2))
And the reason for this should be clear. The standard solution for this
is to save the value as a binding  so back to the drawing board, we
want this transformation instead:
(orelse )
>
(let ((val ))
(if val
val
))
(Note that we would have the same problem in the version that used
simple functions and thunks.)
And to write the new code:
(definemacro (orelse )
(list 'let (list (list 'val ))
(list 'if 'val
'val
)))
(orelse (foo 1) (foo 2))
and this works like we want it to.

## Complexity of Sexpression transformations
As can be seen, writing a simple macro doesn't look too good  what if
we want to write a more complicated macro? A solution to this is to look
at the above macro and realize that it *almost* looks like the code we
want  we basically want to return a list of a certain fixed shape, we
just want some parts to be filled in by the given arguments. Something
like:
(definemacro (orelse )
'(let ((val ))
(if val
val
)))
if we had a way to make the `<...>`s not be a fixed part of the result,
but we actually want the values that the transformation function
received. (Remember that the `<` and `>` are just a part of the name, no
magic, just something to make these names stand out.) This is related to
notational problems that logicians and philosophers had problems with
for centuries. One solution that Lisp uses for this is: instead of a
quote, use backquote (called `quasiquote` in Racket) which is almost
like quote, except that you can `unquote` parts of the value inside.
This is done with a "`,`" comma. Using this, the above code can be
written like this:
(definemacro (orelse )
`(let ((val ,))
(if val
val
,)))

## Scoping problems
You should be able to guess what's this problem about. The basic problem
of these macros is that they cannot be used reliably  the name that
is produced by the macro can shadow a name that is in a completely
different place, therefore destroying lexical scope. For example, in:
(let ((val 4))
(orelse #f val))
the `val` in the macro shadows the use of this name in the above. One
way to solve this is to write macros that look like this:
(definemacro (orelse )
`(let ((%%!my*internal*vardonotuse!%% ,))
(if %%!my*internal*vardonotuse!%%
%%!my*internal*vardonotuse!%%
,)))
or:
(definemacro (orelse )
`(let ((iamusingorelsesoishouldnotusethisname ,))
(if iamusingorelsesoishouldnotusethisname
iamusingorelsesoishouldnotusethisname
,)))
or (this is actually similar to using UUIDs):
(definemacro (orelse )
`(let ((eli@barzilay.org/foo/bar/2002020210:22:22 ,))
(if eli@barzilay.org/foo/bar/2002020210:22:22
eli@barzilay.org/foo/bar/2002020210:22:22
,)))
Which is really not too good because such obscure variables tend to
clobber each other too, in all kinds of unexpected ways.
Another way is to have a function that gives you a different variable
name every time you call it:
(definemacro (orelse )
(let ((temp (gensym)))
`(let ((,temp ,))
(if ,temp
,temp
,))))
but this is not safe either since there might still be clashes of these
names (eg, if they're using a counter that is specific to the current
process, and you start a new process and load code that was generated
earlier). The Lisp solution for this (which Racket's `gensym` function
implements as well) is to use *uninterned* symbols  symbols that have
their own identity, much like strings, and even if two have the same
name, they are not `eq?`.
Note also that there is the mirror side of this problem  what happens
if we try this:
(let ([if 123]) (orelse #f #f))
? This leads to capture in the other direction  the code above
shadows the `if` binding that the macro produces.
Some Schemes will allow something like
(definemacro (foo x)
`(,mullist ,x))
but this is a hack since the macro outputs something that is not a pure
sexpression (and it cannot work for a syntactic keyword like `if`).
Specifically, it is not possible to write the resulting expression (to a
compiled file, for example).
We will ignore this for a moment.

Another problem  manageability of these transformations.
Quasiquotes gets us a long way, but it is still insufficient.
For example, lets write a Racket `bind` that uses `lambda` for binding.
The transformation we now want is:
(bind ((var expr) ...)
body)
>
((lambda (var ...) body)
expr ...)
The code for this looks like this:
(definemacro (bind varexprlist body)
(cons (list 'lambda (map car varexprlist) body)
(map cadr varexprlist)))
This already has a lot more pitfalls. There are `list`s and `cons`es
that you should be careful of, there are `map`s and there are `cadr`s
that would be catastrophic if you use `car`s instead. The quasiquote
syntax is a little more capable  you can write this:
(definemacro (bind varexprlist body)
`((lambda ,(map car varexprlist) ,body)
,@(map cadr varexprlist)))
where "`,@`" is similar to "`,`" but the unquoted expression should
evaluate to a list that is *spliced* into its surrounding list (that is,
its own parens are removed and it's made into elements in the containing
list). But this is still not as readable as the transformation you
actually want, and worse, it is not checking that the input syntax is
valid, which can lead to very confusing errors.
This is yet another problem  if there is an error in the resulting
syntax, the error will be reported in terms of this result rather than
the syntax of the code. There is no easy way to tell where these errors
are coming from. For example, say that we make a common mistake: forget
the "`@`" character in the above macro:
(definemacro (bind varexprlist body)
`((lambda ,(map car varexprlist) ,body)
,(map cadr varexprlist)))
Now, someone else (the client of this macro), tries to use it:
> (bind ((x 1) (y 2)) (+ x y))
procedure application: expected procedure,
given: 1; arguments were: 2
Yes? Now what? Debugging this is difficult, since in most cases it is
not even clear that you were using a macro, and in any case the macro
comes from code that you have no knowledge of and no control over. [The
problem in this specific case is that the macro expands the code to:
((lambda (x y) (+ x y))
(1 2))
so Racket will to use `1` as a function and throw a runtime error.]
Adding error checking to the macro results in this code:
(definemacro (bind varexprlist body)
(if (andmap (lambda (varexpr)
(and (list? varexpr)
(= 2 (length varexpr))
(symbol? (car varexpr))))
varexprlist)
`((lambda ,(map car varexprlist) ,body)
,@(map cadr varexprlist))
(error 'bind "bad syntax whaaaa!")))
Such checks are very important, yet writing this is extremely tedious.

# Scheme (and Racket) Macros
Scheme, Racket included (and much extended), has a solution that is
better than `defmacro`: it has `definesyntax` and `syntaxrules`. First
of all, `definesyntax` is used to create the "magical connection"
between user code and some macro transformation code that does some
rewriting. This definition:
(definesyntax foo
...something...)
makes `foo` be a special syntax that, when used in the head of an
expression, will lead to transforming the expression itself, where the
result of this transformation is what gets used instead of the original
expression. The "`...something...`" in this code fragment should be a
transformation function  one that consumes the expression that is to
be transformed, and returns the new expression to run.
Next, `syntaxrules` is used to create such a transformation in an easy
way. The idea is that what we thought to be an informal specification of
such rewrites, for example:
`let' can be defined as the following transformation:
(let ((x v) ...) body ...)
> ((lambda (x ...) body ...) v ...)
and
`let*' is defined with two transformation rules:
1. (let* () body ...)
> (let () body ...)
2. (let* ((x1 v1) (x2 v2) ...) body ...)
> (let ((x1 v1)) (let* ((x2 v2) ...) body ...))
can actually be formalized by automatically creating a syntax
transformation function from these rule specifications. (Note that this
example has round parentheses so we don't fall into the illusion that
square brackets are different: the resulting transformation would be the
same.) The main point is to view the left hand side as a *pattern* that
can match some forms of syntax, and the right hand side as producing an
output that can use some matched patterns.
`syntaxrules` is used with such rewrite specifications, and it produces
the corresponding transformation function. For example, this:
(syntaxrules () ;*** ignore this "()" for now
[(x y) (y x)])
evaluates to a function that is somewhat similar to:
(lambda (expr)
(if (and (list? expr) (= 2 (length expr)))
(list (second expr) (first expr))
(error "bad syntax")))
but `match` is a little closer, since it uses similar input patterns:
(lambda (expr)
(match expr
[(list x y) (list y x)]
[else (error "bad syntax")]))
Such transformations are used in a `definesyntax` expression to tie the
transformer back into the compiler by hooking it on a specific keyword.
You can now appreciate how all this work when you see how easy it is to
define macros that are very tedious with `definemacro`. For example,
the above `bind`:
(definesyntax bind
(syntaxrules ()
[(bind ((x v) ...) body ...)
((lambda (x ...) body ...) v ...)]))
and `let*` with its two rules:
(definesyntax let*
(syntaxrules ()
[(let* () body ...)
(let () body ...)]
[(let* ((x v) (xs vs) ...) body ...)
(let ((x v)) (let* ((xs vs) ...) body ...))]))
These transformations are so convenient to follow, that Scheme
specifications (and reference manuals) describe forms by specifying
their definition. For example, the Scheme report, specifies `let*` as a
"derived form", and explains its semantics via this transformation.
The input patterns in these rules are similar to `match` patterns, and
the output patterns assemble an sexpression using the matched parts in
the input. For example:
(x y) > (y x)
does the thing you expect it to do  matches a parenthesized form with
two subforms, and produces a form with the two subforms swapped. The
rules for "`...`" on the left side are similar to `match`, as we have
seen many times, and on the right side it is used to *splice* a matched
sequence into the resulting expression and it is required to use the
`...` for sequencematched pattern variables. For example, here is a
list of some patterns, and a description of how they match an input when
used on the left side of a transformation rule and how they produce an
output expression when they appear on the right side:
* `(x ...)`
> **LHS:** matches a parenthesized sequence of zero or more
> expressions, and the `x` pattern variable is bound to this whole
> sequence; `match` analogy: `(match ? [(list x ...) ?])`
>
> **RHS:** when `x` is bound to a sequence, this will produce a
> parenthesized expression containing this sequence; `match` analogy:
> `(match ? [(list x ...) x])`
* `(x1 x2 ...)`
> **LHS:** matches a parenthesized sequence of one or more
> expressions, the first is bound to `x1` and the rest of the sequence
> is bound to `x2`;
>
> `match` analogy: `(match ? [(list x1 x2 ...) ?])`
>
> **RHS:** produces a parenthesized expression that contains the
> expression bound to `x1` first, then all of the expressions in the
> sequence that `x2` is bound to;
>
> `match` analogy: `(match ? [(list x1 x2 ...) (cons x1 x2)])`
* `((x y) ...)`
> **LHS:** matches a parenthesized sequence of 2form parenthesized
> sequences, binding `x` to all the first forms of these, and `y` to
> all the seconds of these (so they will both have the same number of
> items);
>
> `match` analogy: `(match ? [(list (list x y) ...) ?])`
>
> **RHS:** produces a list of forms where each one is made of
> consecutive forms in the `x` sequence and consecutive forms in the
> `y` sequence (both sequences should have the same number of
> elements);
>
> `match` analogy:
>
> (match ? [(list (list x y) ...)
> (map (lambda (x y) (list x y)) x y)])
Some examples of transformations that would be very tedious to write
code manually for:
* `((x y) ...) > ((y x) ...)`
Matches a sequence of 2item sequences, produces a similar sequence
with all of the nested 2item sequences flipped.
* `((x y) ...) > ((x ...) (y ...))`
Matches a similar sequence, and produces a sequence of two sequences,
one of all the first items, and one of the second ones.
* `((x y ...) ...) > ((y ... x) ...)`
Similar to the first example, but the nested sequences can have 1 or
more items in them, and the nested sequences in the result have the
first element moved to the end. Note how the `...` are nested: the
rule is that for each pattern variable you count how many `...`s apply
to it, and that tells you what it holds  and you have to use the
same `...` nestedness for it in the output template.

This is solving the problems of easy code  no need for `list`, `cons`
etc, not even for quasiquotes and tedious syntax massaging. But there
were other problems. First, there was a problem of bad scope, one that
was previously solved with a `gensym`:
(definemacro (orelse )
(let ((temp (gensym)))
`(let ((,temp ,))
(if ,temp ,temp ,))))
Translating this to `definesyntax` and `syntaxrules` we get something
simpler:
(definesyntax orelse
(syntaxrules ()
[(orelse )
(let ((temp ))
(if temp temp ))]))
Even simpler, Racket has a macro called `definesyntaxrule` that
expands to a `definesyntax` combined with a `syntaxrules`  using
it, we can write:
(definesyntaxrule (orelse )
(let ((temp ))
(if temp temp )))
This looks like like a function  but you must remember that it is a
transformation rule specification which is a *very* different beast, as
we'll see.
The main thing here is that Racket takes care of making bindings follow
the lexical scope rules:
(let ([temp 4])
(orelse #f temp))
works fine. In fact, it fully respects the scoping rules: there is no
confusion between bindings that the macro introduces and bindings that
are introduced where the macro is used. (Think about different colors
for bindings introduced by the macro and other bindings.) It's also fine
with many cases that are much harder to cope with otherwise (eg, cases
where there is no `gensym` magic solution):
(let ([if +])
(orelse 1 1))
(let ([if +])
(if (orelse 1 1) 10 100)) ; two different `if's here
or combining both:
(let ([if #f] [temp 4])
(orelse if temp))
(You can try DrRacket's macro debugger to see how the various bindings
get colored differently.)
`definemacro` advocates will claim that it is difficult to make a macro
that intentionally plants a *known* identifier. Think about a `loop`
macro that has an `abort` that can be used inside its body. Or an
`ifit` form that is like `if`, but makes it possible to use the
condition's value in the "then" branch as an `it` binding. It is
possible with all Scheme macro systems to "break hygiene" in such ways,
and we will later see how to do this in Racket. However, Racket also
provides a better way to deal with such problems (think about `it` being
always "bound to a syntax error", but locally rebound in an `ifit`
form).
Scheme macros are said to be *hygienic*  a term used to specify that
they respect lexical scope. There are several implementations of
hygienic macro systems across Scheme implementations, Racket uses the
one known as "syntaxcase system", named after the `syntaxcase`
construct that we discuss below.
> All of this can get much more important in the presence of a module
> system, since you can write a module that provides transformations
> rules, not just values and functions. This means that the concept of
> "a library" in Racket is something that doesn't exist in other
> languages: it's a library that has values, functions, as well as
> macros (or, "compiler plugins").

The way that Scheme implementations achieve hygiene in a macro system is
by making it deal with more than just raw Sexpressions. Roughly
speaking, it deals with syntax objects that are sort of a wrapper
structure around Sexpression, carrying additional information. The
important part of this information when it gets to dealing with hygiene
is the "lexical scope"  which can roughly be described as having
identifiers be represented as symbols plus a "color" which represents
the scope. This way such systems can properly avoid confusing
identifiers with the same name that come from different scopes.
There was also the problem of making debugging difficult, because a
macro can introduce errors that are "coming out of nowhere". In the
implementation that we work with, this is solved by adding yet more
information to these syntax objects  in addition to the underlying
Sexpression and the lexical scope, they also contain source location
information. This allows Racket (and DrRacket) to locate the source of a
specific syntax error, so locating the offending code is easy.
DrRacket's macro debugger heavily relies on this information to provide
a very useful tool  since writing macros can easily become a hard
job.
Finally, there was the problem of writing bad macros. For example, it is
easy to forget that you're dealing with a macro definition and write:
(definesyntaxrule (twice x) (+ x x))
just because you want to inline the addition  but in this case you
end up duplicating the input expression which can have a disastrous
effect. For example:
(twice (twice (twice (twice (twice (twice (twice (twice 1))))))))
expands to a *lot* of code to compile.
Another example is:
(definesyntaxrule (withincrement var expr)
(let ([var (add1 var)]) expr))
...
(withincrement (* foo 2)
...code...)
the problem here is that (* foo 2) will be used as an identifier to be
bound by the `let` expression  which can lead to a confusing syntax
error.
Racket provides many tools to help macro programmers  in addition to
a userinterface tool like the macro debugger there are also
programmerlevel tools where you can reject an input if it doesn't
contain an identifier at a certain place etc. Still, writing macros is
much harder than writing functions  some of these problems are
inherent to the problem that macros solve; for example, you may *want* a
`twice` macro that replicates an expression. By specifying a
transformation to the core language, a macro writer has full control
over which expressions get evaluated and how, which identifiers are
binding instances, and how is the scope of the given expression is
shaped.

# Meta Macros
One of the nice results of `syntaxrules` dealing with the subtle points
of identifiers and scope is that things works fine even when we "go up a
level". For example, the short `definesyntaxrule` form that we've seen
is *itself* a defined as a simple macro:
(definesyntax definesyntaxrule
(syntaxrules ()
[(definesyntaxrule (name P ...) B)
(definesyntax name
(syntaxrules ()
[(name P ...) B]))]))
In fact, this is very similar to something that we have already seen:
the `rewrite` form that we have used in Schlac is implemented in just
this way. The only difference is that `rewrite` requires an actual `=>`
token to separate the input pattern from the output template. If we just
use it in a syntax rule:
(definesyntax rewrite
(syntaxrules ()
[(rewrite (name P ...) => B)
(definesyntax name
(syntaxrules ()
[(name P ...) B]))]))
it won't work. Racket treats the above `=>` just like any identifier,
which in this case acts as a pattern variable which matches anything.
The solution to this is to list the `=>` as a keyword which is expected
to appear in the macro use asis  and that's what the mysterious `()`
of `syntaxrules` is used for: any identifier listed there is taken to
be such a keyword. This makes the following version
(definesyntax rewrite
(syntaxrules (=>)
[(rewrite (name P ...) => B)
(definesyntax name
(syntaxrules ()
[(name P ...) B]))]))
do what we want and throw a syntax error unless `rewrite` is used with
an actual `=>` in the proper place.

# Lazy Constructions in an Eager Language
> [PLAI §37] (has some examples)
This is not really lazy evaluation, but it gets close, and provides the
core useful property of easytouse infinite lists.
(definesyntaxrule (consstream x y)
(cons x (lambda () y)))
(define stream? pair?)
(define nullstream null)
(define nullstream? null?)
;; note that there are not proper lists in racket,
;; so we use car and cdr here
(define streamfirst car)
(define (streamrest s) ((cdr s)))
Using it:
(define ones (consstream 1 ones))
(define (streammap f s)
(if (nullstream? s)
nullstream
(consstream (f (streamfirst s))
(streammap f (streamrest s)))))
(define (streammap2 f s1 s2)
(if (nullstream? s1)
nullstream
(consstream (f (streamfirst s1) (streamfirst s2))
(streammap2 f (streamrest s1)
(streamrest s2)))))
(define ints (consstream 0 (streammap2 + ones ints)))
Actually, all Scheme implementations come with a generalized tool for
(local) laziness: a `delay` form that delays computation of its body
expression, and a `force` function that forces such promises. Here is a
naive implementation of this:
(definetype promise
[makepromise (> Any)])
(definesyntaxrule (delay expr)
(makepromise (lambda () expr)))
(define (force p)
(cases p [(makepromise thunk) (thunk)]))
Proper definitions of `delay`/`force` cache the result  and practical
ones can get pretty complex, for example, in order to allow tail calls
via promises.

# Recursive Macros
Syntax transformations can be recursive. For example, we have seen how
`let*` can be implemented by a transformation that uses two rules, one
of which expands to another use of `let*`:
(definesyntax let*
(syntaxrules ()
[(let* () body ...)
(let () body ...)]
[(let* ((x v) (xs vs) ...) body ...)
(let ((x v)) (let* ((xs vs) ...) body ...))]))
When Racket expands a `let*` expression, the result may contain a new
`let*` which needs extending as well. An important implication of this
is that recursive macros are fine, as long as the recursive case is
using a *smaller* expression. This is just like any form of recursion
(or loop), where you need to be looping over a `wellfounded` set of
values  where each iteration uses a new value that is closer to some
base case.
For example, consider the following macro:
(definesyntaxrule (while condition body ...)
(when condition
body ...
(while condition body ...)))
It seems like this is a good implementation of a `while` loop  after
all, if you were to implement it as a function using thunks, you'd write
very similar code:
(define (while conditionthunk bodythunk)
(when (conditionthunk)
(bodythunk)
(while conditionthunk bodythunk)))
But if you look at the nested `while` form in the transformation rule,
you'll see that it is exactly the same as the input form. This means
that this macro can never be completely expanded  it specifies
infinite code! In practice, this makes the (Racket) compiler loop
forever, consuming more and more memory. This is unlike, for example,
the recursive `let*` rule which uses one less bindingvalue pair than
specified as its input.
The reason that the function version of `while` is fine is that it
iterates using the *same* code, and the condition thunk will depend on
some state that converges to a base case (usually the body thunk will
perform some sideeffects that makes the loop converge). But in the
macro case there is *no* evaluation happening, if the transformed syntax
contains the same input pattern, we end up having a macro that expands
infinitely.
The correct solution for a `while` macro is therefore to use plain
recursion using a local recursive function:
(definesyntaxrule (while condition body ...)
(letrec ([loop (lambda ()
(when condition
body ...
(loop)))])
(loop)))
A popular way to deal with macros like this that revolve around a
specific control flow is to separate them into a function that uses
thunks, and a macro that does nothing except wrap input expressions as
thunks. In this case, we get this solution:
(define (while/proc conditionthunk bodythunk)
(when (conditionthunk)
(bodythunk)
(while/proc conditionthunk bodythunk)))
(definesyntaxrule (while condition body ...)
(while/proc (lambda () condition)
(lambda () body ...)))

## Another example: a simple loop
Here is an implementation of a macro that does a simple arithmetic loop:
(definesyntax for
(syntaxrules (= to do)
[(for x = m to n do body ...)
(letrec ([loop (lambda (x)
(when (<= x n)
body ...
(loop (+ x 1))))])
(loop m))]))
(Note that this is not complete code: it suffers from the usual problem
of multiple evaluations of the `n` expression. We'll deal with it soon.)
This macro combines both control flow and lexical scope. Control flow is
specified by the loop (done, as usual in Racket, as a tailrecursive
function)  for example, it determines how code is iterated, and it
also determines what the `for` form will evaluate to (it evaluates to
whatever `when` evaluates to, the void value in this case). Scope is
also specified here, by translating the code to a function  this code
makes `x` have a scope that covers the body so this is valid:
(for i = 1 to 3 do (printf "i = ~s\n" i))
but it also makes the boundary expression `n` be in this scope, making
this:
(for i = 1 to (if (even? i) 10 20) do (printf "i = ~s\n" i))
valid. In addition, while evaluating the condition on each iteration
might be desirable, in most cases it's not  consider this example:
(for i = 1 to (read) do (printf "i = ~s\n" i))
This is easily solved by using a `let` to make the expression evaluate
just once:
(definesyntax for
(syntaxrules (= to do)
[(for x = m to n do body ...)
(let ([m* m] ; execution order
[n* n])
(letrec ([loop (lambda (x)
(when (<= x n*)
body ...
(loop (+ x 1))))])
(loop m*)))]))
which makes the previous use result in a "`reference to undefined
identifier: i`" error.
Furthermore, the fact that we have a hygienic macro system means that it
is perfectly fine to use nested `for` expressions:
(for a = 1 to 9 do
(for b = 1 to 9 do (printf "~s,~s " a b))
(newline))
The transformation is, therefore, completely specifying the semantics of
this new form.
Extending this syntax is easy using multiple transformation rules 
for example, say that we want to extend it to have a `step` optional
keyword. The standard idiom is to have the stepless pattern translated
into one that uses `step 1`:
(for x = m to n do body ...)
> (for x = m to n step 1 do body ...)
Usually, you should remember that `syntaxrules` tries the patterns one
by one until a match is found, but in this case there is no problems
because the keywords make the choice unambiguous:
(definesyntax for
(syntaxrules (= to do step)
[(for x = m to n do body ...)
(for x = m to n step 1 do body ...)]
[(for x = m to n step d do body ...)
(let ([m* m]
[n* n]
[d* d])
(letrec ([loop (lambda (x)
(when (<= x n*)
body ...
(loop (+ x d*))))])
(loop m*)))]))
(for i = 1 to 10 do (printf "i = ~s\n" i))
(for i = 1 to 10 step 2 do (printf "i = ~s\n" i))
We can even extend it to do a different kind of iteration, for example,
iterate over list:
(definesyntax for
(syntaxrules (= to do step in)
[(for x = m to n do body ...)
(for x = m to n step 1 do body ...)]
[(for x = m to n step d do body ...)
(let ([m* m]
[n* n]
[d* d])
(letrec ([loop (lambda (x)
(when (<= x n*)
body ...
(loop (+ x d*))))])
(loop m*)))]
;; list
[(for x in l do body ...)
(foreach (lambda (x) body ...) l)]))
(for i in (list 1 2 3 4) do (printf "i = ~s\n" i))
(for i in (list 1 2 3 4) do
(for i = 0 to i do (printf "i = ~s " i))
(newline))

## Yet Another: List Comprehension
At this point it's clear that macros are a powerful language feature
that makes it relatively easy to implement new features, making it a
language that is easy to use as a tool for quick experimentation with
new language features. As an example of a practical feature rather than
a toy, let's see how we can implement [Python's list comprehenions].
These are expressions that conveniently combine `map`, `filter`, and
nested uses of both.
First, a simple implementation that uses only the `map` feature:
(definesyntax listof
(syntaxrules (for in)
[(listof EXPR for ID in LIST)
(map (lambda (ID) EXPR)
LIST)]))
(listof (* x x) for x in (range 10))
It is a good exercise to see how everything that we've seen above plays
a role here. For example, how we get the `ID` to be bound in `EXPR`.
Next, add a condition expression with an `if` keyword, and implemented
using a `filter`:
(definesyntax listof
(syntaxrules (for in if)
[(listof EXPR for ID in LIST if COND)
(map (lambda (ID) EXPR)
(filter (lambda (ID) COND) LIST))]
[(listof EXPR for ID in LIST)
(listof EXPR for ID in LIST if #t)]))
(listof (* x x) for x in (range 10) if (odd? x))
Again, go over it and see how the binding structure makes the identifier
available in both expressions. Note that since we're just playing around
we're not paying too much attention to performance etc. (For example, if
we cared, we could have implemented the `if`less case by not using
`filter` at all, or we could implement a `filter` that accepts `#t` as a
predicate and in that case just returns the list, or even implementing
it as a macro that identifies a `(lambda (_) #t)` pattern and expands to
just the list (a bad idea in general).)
The last step: Python's comprehension accepts multiple `for``in`s for
nested loops, possibly with `if` filters at each level:
(definesyntax listof
(syntaxrules (for in if)
[(listof EXPR for ID in LIST if COND)
(map (lambda (ID) EXPR)
(filter (lambda (ID) COND) LIST))]
[(listof EXPR for ID in LIST)
(listof EXPR for ID in LIST if #t)]
[(listof EXPR for ID in LIST for MORE ...)
(listof EXPR for ID in LIST if #t for MORE ...)]
[(listof EXPR for ID in LIST if COND for MORE ...)
(apply append (map (lambda (ID) (listof EXPR for MORE ...))
(filter (lambda (ID) COND) LIST)))]))
A collection of examples that I found in the Python docs and elsewhere,
demonstrating all of these:
;; [x**2 for x in range(10)]
(listof (* x x) for x in (range 10))
;; [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
(listof (list x y) for x in '(1 2 3) for y in '(3 1 4)
if (not (= x y)))
(define (roundn x n) ; pythonlike round to n digits
(define 10^n (expt 10 n))
(/ (round (* x 10^n)) 10^n))
;; [str(round(pi, i)) for i in range(1, 6)]
(listof (number>string (roundn pi i)) for i in (range 1 6))
(define matrix
'((1 2 3 4)
(5 6 7 8)
(9 10 11 12)))
;; [[row[i] for row in matrix] for i in range(4)]
(listof (listof (listref row i) for row in matrix)
for i in (range 4))
(define text '(("bar" "foo" "fooba")
("Rome" "Madrid" "Houston")
("aa" "bb" "cc" "dd")))
;; [y for x in text if len(x)>3 for y in x]
(listof y for x in text if (> (length x) 3) for y in x)
;; [y for x in text for y in x if len(y)>4]
(listof y for x in text for y in x if (> (stringlength y) 4))
;; [y.upper() for x in text if len(x) == 3
;; for y in x if y.startswith('f')]
(listof (stringupcase y) for x in text if (= (length x) 3)
for y in x if (regexpmatch? #rx"^f" y))
[Python's list comprehenions]:
https://docs.python.org/3/tutorial/datastructures.html#listcomprehensions

# Problems of `syntaxrules` Macros
As we've seen, using `syntaxrules` solves many of the problems of
macros, but it comes with a high price tag: the macros are "just"
rewrite rules. As rewrite rules they're pretty sophisticated, but it
still loses a huge advantage of what we had with `definemacro`  the
macro code is no longer Racket code but a simple language of rewrite
rules.
There are two big problems with this which we will look into now.
(DrRacket's macro stepper tool can be very useful in clarifying these
examples.) The first problem is that in some cases we want to perform
computations at the macro level  for example, consider a `repeat`
macro that needs to expand like this:
(repeat 1 E) > (begin E)
(repeat 2 E) > (begin E E)
(repeat 3 E) > (begin E E E)
...
With a `syntaxrules` macro we can match over specific integers, but we
just cannot do this with *any* integer. Note that this specific case can
be done better via a function  better by not replicating the
expression:
(define (repeat/proc n thunk)
(when (> n 0) (thunk) (repeat/proc (sub1 n) thunk)))
(definesyntaxrule (repeat N E)
(repeat/proc N (lambda () E)))
or even better, assuming the above `for` is already implemented:
(definesyntaxrule (repeat N E)
(for i = 1 to N do E))
But still, we want to have the ability to do such computation. A
similar, and perhaps better example, is better error reporting. For
example, the above `for` implementation blindly expands its input, so:
> (for 1 = 1 to 3 do (printf "i = ~s\n" i))
lambda: not an identifier in: 1
we get a bad error message in terms of `lambda`, which is breaking
abstraction (it comes from the expansion of `for`, which is an
implementation detail), and worse  it is an error about something
that the user didn't write.
Yet another aspect of this problem is that sometimes we need to get
creative solutions where it would be very simple to write the
corresponding Racket code. For example, consider the problem of writing
a `revapp` macro  (revapp F E ...) should evaluate to a function
similar to (F E ...), except that we want the evaluation to go from
right to left instead of the usual lefttoright that Racket does. This
code is obviously very broken:
(definesyntaxrule (revapp F E ...)
(let (reverse ([x E] ...))
(F x ...)))
because it *generates* a malformed `let` form  there is no way for
the macro expander to somehow know that the `reverse` should happen at
the transformation level. In this case, we can actually solve this using
a helper macro to do the reversing:
(definesyntaxrule (revapp F E ...)
(revapphelper F (E ...) ()))
(definesyntax revapphelper
(syntaxrules ()
;; this rule does the reversing, collecting the reversed
;; sequence in the last part
[(revapphelper F (E0 E ...) (E* ...))
(revapphelper F (E ...) (E0 E* ...))]
;; and this rule fires up when we're done with the reversal
[(revapphelper F () (E ...))
(let ([x E] ...)
(F x ...))]))
There are still problems with this  it complains about `x ...`
because there is a single `x` there rather than a sequence of them; and
even if it did somehow work, we also need the `x`s in that last line in
the original order rather than the reversed one. So the solution is
complicated by collecting new `x`s while reversing  and since we need
them in both orders, we're going to collect both orders:
(definesyntaxrule (revapp F E ...)
(revapphelper F (E ...) () () ()))
(definesyntax revapphelper
(syntaxrules ()
;; this rule does the reversing, collecting the reversed
;; sequence in the last part  also make up new identifiers
;; and collect them in *both* directions (`X' is the straight
;; sequence of identifiers, `X*' is the reversed one, and `E*'
;; is the reversed expression sequence); note that each
;; iteration introduces a new identifier called `t'
[(revapphelper F (E0 E ...) (X ... ) ( X* ...) ( E* ...))
(revapphelper F ( E ...) (X ... t) (t X* ...) (E0 E* ...))]
;; and this rule fires up when we're done with the reversal and
;; the generation
[(revapphelper F () (x ...) (x* ...) (E* ...))
(let ([x* E*] ...)
(F x ...))]))
;; see that it works
(define (show x) (printf ">>> ~s\n" x) x)
(revapp list (show 1) (show 2) (show 3))
So, this worked, but in this case the simplicity of the `syntaxrules`
rewrite language worked against us, and made a very inconvenient
solution. This could have been much easier if we could just write a
"metalevel" reverse, and a use of `map` to generate the names.
... And all of that was just the first problem. The second one is even
harder: `syntaxrules` is *designed* to avoid all name captures, but
what if we *want* to break hygiene? There are some cases where you want
a macro that "injects" a uservisible identifier into its result. The
most common (and therefore the classic) example of this is an anaphoric
`if` macro, that binds `it` to the result of the test (which can be any
value, not just a boolean):
;; find the element of `l' that is immediately following `x'
;; (assumes that if `x' is found, it is not the last one)
(define (after x l)
(let ([m (member x l)])
(if m
(second m)
(error 'after "~s not found in ~s" x l))))
which we want to turn to:
;; find the element of `l' that is immediately following `x'
;; (assumes that if `x' is found, it is not the last one)
(define (after x l)
(if (member x l)
(second it)
(error 'after "~s not found in ~s" x l)))
The obvious definition of `ifit' doesn't work:
(definesyntaxrule (ifit E1 E2 E3)
(let ([it E1]) (if it E2 E3)))
The reason it doesn't work should be obvious now  it is *designed* to
avoid the `it` that the macro introduced from interfering with the `it`
that the user code uses.
Next, we'll see Racket's "low level" macro system, which can later be
used to solve these problems.

# Racket's "LowLevel" Macros
As we've previously seen, `syntaxrules` *creates* transformation
functions  but there are other more direct ways to write these
functions. These involve writing a function directly rather than
creating one with `syntaxrules`  and because this is a more
lowlevel approach than using `syntaxrules` to generate a transformer,
it is called a "low level macro system". All Scheme implementations have
such lowlevel systems, and these systems vary from one to the other.
They all involve some particular type that is used as "syntax"
 this type is always related to Sexpressions, but it cannot be the
simple `definemacro` tool that we've seen earlier if we want to avoid
the problems of capturing identifiers.
Historical note: For a very long time the Scheme standard had avoided a
concrete specification of this lowlevel system, leaving `syntaxrules`
as the only way to write portable Scheme code. This had lead some people
to explore more thoroughly the things that can be done with just
`syntaxrules` rewrites, even beyond the examples we've seen. As it
turns out, there's a lot that can be done with it  in fact, it is
possible to write rewrite rules that implement a lambda calculus, making
it possible to write things that look close to "real code". This is,
however, awkward to get used to, and redundant with a macro system that
can use the full language for arbitrary computations. It has also became
less popular recently, since R6RS dictates something that is known as a
"syntaxcase macro system" (not really a good name, since `syntaxcase`
is just a tool in this system).
Racket uses an extended version of this `syntaxcase` system, which is
what we will discuss now. In the Racket macro system, "syntax" is a new
type, not just Sexpressions as is the case with `definemacro`. The way
to think about this type is as a wrapper around Sexpressions, where the
Sexpression is the "raw" symbolic form of the syntax, and a bunch of
"stuff" is added. Two important bits of this "stuff" are the source
location information for the syntax, and its lexical scope. The source
location is what you'd expect: the file name for the syntax (if it was
read from a file), its position in the file, and its line and column
numbers; this information is mostly useful for reporting errors. The
lexical scope information is used in a somewhat peculiar way: there is
no direct way to access it, since usually you don't need to do so 
instead, for the rare cases where you do need to manipulate it, you
*copy* the lexical scope of an existing syntax to create another. This
allows the macro interface to be usable without specification of a
specific representation for the scope.
The main piece of functionality in this system is `syntaxcase` (which
has lead to its common name)  a form that is used to deconstruct the
input via patternmatching similar to `syntaxrules`. In fact, the
syntax of `syntaxcase` looks very similar to the syntax of
`syntaxrules`  there are zero or more parenthesized keywords, and
then clauses that begin with the same kind of patterns to match the
syntax against. The first obvious difference is that the syntax to be
matched is given explicitly:
(syntaxcase ()
[