PL: Lecture #20  Tuesday, November 13th
(text file)

Designing Domain Specific Languages (DSLs)

PLAI §35

Programming languages differ in numerous ways:

  1. Each uses different notations for writing down programs. As we’ve observed, however, syntax is only partially interesting. (This is, however, less true of languages that are trying to mirror the notation of a particular domain.)

  2. Control constructs: for instance, early languages didn’t even support recursion, while most modern languages still don’t have continuations.

  3. The kinds of data they support. Indeed, sophisticated languages like Racket blur the distinction between control and data by making fragments of control into data values (such as first-class functions and continuations).

  4. The means of organizing programs: do they have functions, modules, classes, namespaces, …?

  5. Automation such as memory management, run-time safety checks, and so on.

Each of these items suggests natural questions to ask when you design your own languages in particular domains.

Obviously, there are a lot of domain specific languages these days — and that’s not new. For example, four of the oldest languages were conceived as domain specific languages:

Only in the late 60s / early 70s languages began to get free from their special purpose domain and become general purpose languages (GPLs). These days, we usually use some GPL for our programs and often come up with small domain specific languages (DSLs) for specific jobs. The problem is designing such a specific language. There are lots of decisions to make, and as should be clear now, many ways of shooting your self in the foot. You need to know:

And very important:

To clarify why this can be applicable in more situations than you think, consider what programming languages are used for. One example that should not be ignored is using a programming language to implement a programming language — for example, what we did so far (or any other interpreter or compiler). In the same way that some piece of code in a PL represent functions about the “real world”, there are other programs that represent things in a language — possibly even the same one. To make a side-effect-full example, the meaning of one-brick might abstract over laying a brick when making a wall — it abstracts all the little details into a function:

(define (one-brick wall brick-pile)
  (move-eye (location brick-pile))
  (let ([pos (find-available-brick-position brick-pile)])
    (move-hand pos)
    (grab-object))
  (move-eye wall)
  (let ([pos (find-next-brick-position wall)])
    (move-hand pos)
    (drop-object)))

and we can now write

(one-brick my-wall my-brick-pile)

instead of all of the above. We might use that in a loop:

(define (build-wall wall pile)
  (define (loop n)
    (when (< n 500)
      (one-brick wall pile)
      (loop (add1 n))))
  (loop 0))

This is a common piece of looping code that we’ve seen in many forms, and a common complaint of newcomers to functional languages is the lack of some kind of a loop. But once you know the template, writing such loops is easy — and in fact, you can write code that would take something like:

(define (build-wall wall pile)
  (loop-for i from 1 to 500
    (one-brick wall pile)))

and produce the previous code. Note the main point here: we switch from code that deals with bricks to code that deals with code.

Now, a viable option for implementing a new DSL is to do so by transforming it into an existing language. Such a process is usually tedious and error prone — tedious because you need to deal with the boring parts of a language (making a parser etc), and error prone because it’s easy to generate bad code (especially when you’re dealing with strings) and you get bad errors in terms of the translated code instead of the actual code, resorting to debugging the intermediate generated programs. Lisp languages traditionally have taken this idea one level further than other languages: instead of writing a new transformer for your language, you use the host language, but you extend and customize it by adding you own forms.

Side-note: WSJ on the proliferation of PLs

“Computer Languages Multiply, Pleasing Many–But Not All” Wall Street Journal (12/14/05) P. B1; Gomes, Lee

While the proliferation of languages has been a boon to software programmers, the extensive variety often frustrates their bosses and confounds the larger software companies. C and the subsequent C++ may be the most popular languages in use today, but any programmer working on the Web must also include languages such as Perl, Python, PHP, and TCL in his resume. The explosion has been partially fueled by the ability of an individual programmer or a small group to create and market a language, as was the case with Ruby on Rails, which became an overnight sensation thanks to a 15-minute demonstration video the Danish programmer David Hansson circulated over the Web. Once a language has gained a core following, blogs and Web sites appear to track its developments. Many languages owe their origins to small design firms trying to make a commercial success of themselves, while others are labors of love, as is the case with many open source projects. As new languages continue to emerge, however, more programmers are defecting from mainstream systems such as .NET and Java in favor of niche offerings that are more tailored to a specific project. CIOs are often assailed by complaints from their programmers when they try to impose restrictions on the number of languages that are permissible. While it has been demonstrated theoretically that each language is the rough equivalent of any other, it is no more likely for a consensus to appear within the programming community than it is for a single car to be met with a universal embrace from the entire fleet of motorists.

Syntax Transformations: Macros

PLAI §36

Macros are one of the biggest advantages of all Lisps, and specifically even more so an advantage of Scheme implementations, and yet more specifically, it is a major Racket feature: this section is therefore specific to Racket (which has this unique feature), although most of this is the same in most Schemes.

As we have previously seen, it is possible to implement one language construct using another. What we did could be described as bits of a compiler, since they translate one language to another.

We will see how this can be done in Racket: implementing some new linguistic forms in terms of ones that are already known. In essence, we will be translating Racket constructs to other Racket constructs — and all that is done in Racket, no need to go back to the language Racket was implemented in (C).

This is possible with a simple “trick”: the Racket implementation uses some syntax objects. These objects are implemented somehow inside Racket’s own source code. But these objects are also directly available for our use — part of the implementation is exposed to our level. This is quite similar to the way we have implemented pairs in our language — a TOY or a SLOTH pair is implemented using a Racket pair, so the same data object is available at both levels.

This is the big idea in Lisp, which Scheme (and therefore Racket) inherited from (to some extent): programs are made of numbers, strings, symbols and lists of these — and these are all used both at the meta-level as well as the user level. This means that instead of having no meta-language at all (locking away a lot of useful stuff), and instead of having some crippled half-baked meta language (CPP being both the most obvious (as well as the most popular) example for such a meta language), instead of all this we get exactly the same language at both levels.

How is this used? Well, the principle is simple. For example, say we want to write a macro that will evaluate two forms in sequence, but if the first one returns a result that is not false then it returns it instead of evaluating the second one too. This is exactly how or behaves, so pretend we don’t have it — call our version orelse:

(orelse <expr1> <expr2>)

in effect, we add a new special form to our language, with its own evaluation rule, only we know how to express this evaluation rule by translating it to things that are already part of our language.

We could do this as a simple function — only if we’re willing to explicitly delay the arguments with a lambda, and use the thunks in the function:

(define (orelse thunk1 thunk2)
  (if (thunk1)
    (thunk1) ; ignore the double evaluation for now
    (thunk2)))

or:

(define (orelse thunk1 thunk2)
  ((if (thunk1)
    thunk1
    thunk2)))

and using it like this:

(orelse (lambda () 1) (lambda () (error "boom")))

But this is clearly not the right way to do this: whoever uses this code must be aware of the need to send us thunks, and it’s verbose and inconvenient.

Note that this could be a feasible solution if there was a uniform way to have an easy syntactic way to say “a chunk of code” instead of immediately execute it — this is exactly what (lambda () ...) does. So we could, for example, make {...} be a shorthand for that, which is what Perl-6 is doing. However, we will soon see examples where we want more than just delay the evaluation of some code.

We want to translate

(orelse <expr1> <expr2>)

--to-->

(if <expr1>
  <expr1>
  <expr2>)

If we look at the code as an s-expression, then we can write the following function:

(define (translate-orelse l)
  (if (and (list? l)
          (= 3 (length l))
          (eq? 'orelse (first l)))
    (list 'if (second l) (second l) (third l))
    (error 'translate-orelse "bad input: ~s" l)))

We can now try it with a simple list:

(translate-orelse '(orelse foo1 foo2))

and note that the result is correct.

How is this used? Well, all we need is to hook our function into our implementation’s evaluator. In Lisp, we get a defmacro form for this, and many Schemes inherited it or something similar. In Racket, we need to

(require compatibility/defmacro)

but it requires the transformation to be a little different in a way that makes life easier: the above contains a lot of boilerplate code. Usually, we will require the input to be a list of some known length, the first element to be a symbol that specifies our form, and then do something with the other arguments. So we’d want to always follow a template that looks like:

(define (translate-??? exprs)
  (if (and (list? exprs)
          (= N (length exprs))
          (eq? '??? (car exprs)))
    (let ([E1 (cadr exprs)]
          [E2 (caddr exprs)]
          ...)
      ...make result expression...)
    (error ...)))

But this looks very similar to making sure that a function call is a specific function call (and for a good reason — macro usages look just like function calls). So make the translation function get a number of arguments one each for each part of the input, an s-expression. For example, the above translation and test become:

(define (translate-orelse <expr1> <expr2>)
  (list 'if <expr1> <expr1> <expr2>))

(translate-orelse 'foo1 'foo2)

The number of arguments is used to check the input (turning an arity error for the macro to an arity error for the translator function call), and we don’t need to “caddr our way” to arguments.

This gives us the simple definition — but what about the promised hook? — All we need is to use define-macro instead of define, and change the name to the name that will trigger this translation (providing the last missing test of the input):

(define-macro (orelse <expr1> <expr2>)
  (list 'if <expr1> <expr1> <expr2>))

and test it:

(orelse 1 (error "boom"))

Note that this is basically a (usually purely functional) lazy language of transformations which is built on top of Racket. It is possible for macros to generate pieces of code that contain references to these same macros, and they will be used to expand those instances again.

Now we start with the problems, one by one.

Macro Problems

There is an inherent problem when macros are being used, in any form and any language (even in CPP): you must remember that you are playing with expressions, not with values — which is why this is problematic:

(define (foo x) (printf "foo ~s!\n" x) x)

(or (foo 1) (foo 2))

(orelse (foo 1) (foo 2))

And the reason for this should be clear. The standard solution for this is to save the value as a binding — so back to the drawing board, we want this transformation instead:

(orelse <expr1> <expr2>)
-->
(let ((val <expr1>))
  (if val
    val
    <expr2>))

(Note that we would have the same problem in the version that used simple functions and thunks.)

And to write the new code:

(define-macro (orelse <expr1> <expr2>)
  (list 'let (list (list 'val <expr1>))
    (list 'if 'val
      'val
      <expr2>)))

(orelse (foo 1) (foo 2))

and this works like we want it to.

Complexity of S-expression transformations

As can be seen, writing a simple macro doesn’t look too good — what if we want to write a more complicated macro? A solution to this is to look at the above macro and realize that it almost looks like the code we want — we basically want to return a list of a certain fixed shape, we just want some parts to be filled in by the given arguments. Something like:

(define-macro (orelse <expr1> <expr2>)
  '(let ((val <expr1>))
    (if val
      val
      <expr2>)))

if we had a way to make the <...>s not be a fixed part of the result, but we actually want the values that the transformation function received. (Remember that the < and > are just a part of the name, no magic, just something to make these names stand out.) This is related to notational problems that logicians and philosophers had problems with for centuries. One solution that Lisp uses for this is: instead of a quote, use backquote (called quasiquote in Racket) which is almost like quote, except that you can unquote parts of the value inside. This is done with a “,” comma. Using this, the above code can be written like this:

(define-macro (orelse <expr1> <expr2>)
  `(let ((val ,<expr1>))
    (if val
      val
      ,<expr2>)))