PL: Lecture #4  Tuesday, September 18th
(text file)

BNF, Grammars, the AE Language

Getting back to the theme of the course: we want to investigate programming languages, and we want to do that using a programming language.

The first thing when we design a language is to specify the language. For this we use BNF (Backus-Naur Form). For example, here is the definition of a simple arithmetic language:

<AE> ::= <num>
      | <AE> + <AE>
      | <AE> - <AE>

Explain the different parts. Specifically, this is a mixture of low-level (concrete) syntax definition with parsing.

We use this to derive expressions in some language. We start with <AE>, which should be one of these:

<num> is a terminal: when we reach it in the derivation, we’re done. <AE> is a non-terminal: when we reach it, we have to continue with one of the options. It should be clear that the + and the - are things we expect to find in the input — because they are not wrapped in <>s.

We could specify what <num> is (turning it into a <NUM> non-terminal):

<AE> ::= <NUM>
      | <AE> + <AE>
      | <AE> - <AE>

<NUM> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
        | <NUM> <NUM>

But we don’t — why? Because in Racket we have numbers as primitives and we want to use Racket to implement our languages. This makes life a lot easier, and we get free stuff like floats, rationals etc.

To use a BNF formally, for example, to prove that 1-2+3 is a valid <AE> expression, we first label the rules:

<AE> ::= <num>        (1)
      | <AE> + <AE>  (2)
      | <AE> - <AE>  (3)

and then we can use them as formal justifications for each derivation step:

<AE>
<AE> + <AE>        ; (2)
<AE> + <num>        ; (1)
<AE> - <AE> + <num> ; (3)
<AE> - <AE> + 3    ; (num)
<num> - <AE> + 3    ; (1)
<num> - <num> + 3  ; (1)
1 - <num> + 3      ; (num)
1 - 2 + 3          ; (num)

This would be one way of doing this. Alternatively, we can can visualize the derivation using a tree, with the rules used at the nodes.

These specifications suffer from being ambiguous: an expression can be derived in multiple ways. Even the little syntax for a number is ambiguous — a number like 123 can be derived in two ways that result in trees that look different. This ambiguity is not a “real” problem now, but it will become one very soon. We want to get rid of this ambiguity, so that there is a single (= deterministic) way to derive all expressions.

There is a standard way to resolve that — we add another non-terminal to the definition, and make it so that each rule can continue to exactly one of its alternatives. For example, this is what we can do with numbers:

<NUM>  ::= <DIGIT> | <DIGIT> <NUM>

<DIGIT> ::= 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

Similar solutions can be applied to the <AE> BNF — we either restrict the way derivations can happen or we come up with new non-terminals to force a deterministic derivation trees.

As an example of restricting derivations, we look at the current grammar:

<AE> ::= <num>
      | <AE> + <AE>
      | <AE> - <AE>

and instead of allowing an <AE> on both sides of the operation, we force one to be a number:

<AE> ::= <num>
      | <num> + <AE>
      | <num> - <AE>

Now there is a single way to derive any expression, and it is always associating operations to the right: an expression like 1+2+3 can only be derived as 1+(2+3). To change this to left-association, we would use this:

<AE> ::= <num>
      | <AE> + <num>
      | <AE> - <num>

But what if we want to force precedence? Say that our AE syntax has addition and multiplication:

<AE> ::= <num>
      | <AE> + <AE>
      | <AE> * <AE>

We can do that same thing as above and add new non-terminals — say one for “products”:

<AE>  ::= <num>
        | <AE> + <AE>
        | <PROD>

<PROD> ::= <num>
        | <PROD> * <PROD>

Now we must parse any AE expression as additions of multiplications (or numbers). First, note that if <AE> goes to <PROD> and that goes to <num>, then there is no need for an <AE> to go to a <num>, so this is the same syntax:

<AE>  ::= <AE> + <AE>
        | <PROD>

<PROD> ::= <num>
        | <PROD> * <PROD>

Now, if we want to still be able to multiply additions, we can force them to appear in parentheses:

<AE>  ::= <AE> + <AE>
        | <PROD>

<PROD> ::= <num>
        | <PROD> * <PROD>
        | ( <AE> )

Next, note that <AE> is still ambiguous about additions, which can be fixed by forcing the left hand side of an addition to be a factor:

<AE>  ::= <PROD> + <AE>
        | <PROD>

<PROD> ::= <num>
        | <PROD> * <PROD>
        | ( <AE> )

We still have an ambiguity for multiplications, so we do the same thing and add another non-terminal for “atoms”:

<AE>  ::= <PROD> + <AE>
        | <PROD>

<PROD> ::= <ATOM> * <PROD>
        | <ATOM>

<ATOM> ::= <num>
        | ( <AE> )

And you can try to derive several expressions to be convinced that derivation is always deterministic now.

But as you can see, this is exactly the cosmetics that we want to avoid — it will lead us to things that might be interesting, but unrelated to the principles behind programming languages. It will also become much much worse when we have a real language rather such a tiny one.

Is there a good solution? — It is right in our face: do what Racket does — always use fully parenthesized expressions:

<AE> ::= <num>
      | ( <AE> + <AE> )
      | ( <AE> - <AE> )

To prevent confusing Racket code with code in our language(s), we also change the parentheses to curly ones:

<AE> ::= <num>
      | { <AE> + <AE> }
      | { <AE> - <AE> }

But in Racket everything has a value — including those +s and -s, which makes this extremely convenient with future operations that might have either more or less arguments than 2 as well as treating these arithmetic operators as plain functions. In our toy language we will not do this initially (that is, + and - are second order operators: they cannot be used as values). But since we will get to it later, we’ll adopt the Racket solution and use a fully-parenthesized prefix notation:

<AE> ::= <num>
      | { + <AE> <AE> }
      | { - <AE> <AE> }

(Remember that in a sense, Racket code is written in a form of already-parsed syntax…)

Simple Parsing

On to an implementation of a “parser”:

Unrelated to what the syntax actually looks like, we want to parse it as soon as possible — converting the concrete syntax to an abstract syntax tree.

No matter how we write our syntax:

we always mean the same abstract thing — adding the number 3 and the number 4. The essence of this is basically a tree structure with an addition operation as the root and two leaves holding the two numerals.

With the right data definition, we can describe this in Racket as the expression (Add (Num 3) (Num 4)) where Add and Num are constructors of a tree type for syntax, or in a C-like language, it could be something like Add(Num(3),Num(4)).

Similarly, the expression (3-4)+7 will be described in Racket as the expression:

(Add (Sub (Num 3) (Num 4)) (Num 7))

Important note: “expression” was used in two different ways in the above — each way corresponds to a different language, and the result of evaluating the second “expression” is a Racket value that represents the first expression.

To define the data type and the necessary constructors we will use this:

(define-type AE
  [Num Number]
  [Add AE AE]
  [Sub AE AE])

To make things very simple, we will use the above fact through a double-level approach:

This is achieved by the following simple recursive function:

(: parse-sexpr : Sexpr -> AE)
;; parses s-expressions into AEs
(define (parse-sexpr sexpr)
  (cond [(number? sexpr) (Num sexpr)]
        [(and (list? sexpr) (= 3 (length sexpr)))
        (let ([make-node
                (match (first sexpr)
                  ['+ Add]
                  ['- Sub]
                  [else (error 'parse-sexpr "unknown op: ~s"
                              (first sexpr))])
                #| the above is the same as:
                (cond [(equal? '+ (first sexpr)) Add]
                      [(equal? '- (first sexpr)) Sub]
                      [else (error 'parse-sexpr "unknown op: ~s"
                                  (first sexpr))])
                |#])
          (make-node (parse-sexpr (second sexpr))
                      (parse-sexpr (third sexpr))))]
        [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

This function is pretty simple, but as our languages grow, they will become more verbose and more difficult to write. So, instead, we use a new special form: match, which is matching a value and binds new identifiers to different parts (try it with “Check Syntax”). Re-writing the above code using match:

(: parse-sexpr : Sexpr -> AE)
;; parses s-expressions into AEs
(define (parse-sexpr sexpr)
  (match sexpr
    [(number: n) (Num n)]
    [(list '+ left right)
    (Add (parse-sexpr left) (parse-sexpr right))]
    [(list '- left right)
    (Sub (parse-sexpr left) (parse-sexpr right))]
    [else (error 'parse-sexpr "bad syntax in ~s" sexpr)]))

And finally, to make it more uniform, we will combine this with the function that parses a string into a sexpr so we can use strings to represent our programs:

(: parse : String -> AE)
;; parses a string containing an AE expression to an AE
(define (parse str)
  (parse-sexpr (string->sexpr str)))