FOO Special Operators

You could stop there; certainly the FOO language is expressive enough to generate nearly any HTML you’d care to. However, you can add two features to the language, with just a bit more code, that will make it quite a bit more powerful: special operators and macros.

Special operators in FOO are analogous to special operators in Common Lisp. Special operators provide ways to express things in the language that can’t be expressed in the language supported by the basic evaluation rule. Or, another way to look at it is that special operators provide access to the primitive mechanisms used by the language evaluator.1

To take a simple example, in the FOO compiler, the language evaluator uses the embed-value function to generate code that will embed the value of a variable in the output HTML. However, because only symbols are passed to embed-value, there’s no way, in the language I’ve described so far, to embed the value of an arbitrary Common Lisp expression; the process function passes cons cells to embed-code rather than embed-value, so the values returned are ignored. Typically this is what you’d want, since the main reason to embed Lisp code in a FOO program is to use Lisp control constructs. However, sometimes you’d like to embed computed values in the generated HTML. For example, you might like this FOO program to generate a paragraph tag containing a random number:

(:p (random 10))

But that doesn’t work because the code is run and its value discarded.

HTML> (html (:p (random 10)))
<p></p>
NIL

In the language, as you’ve implemented it so far, you could work around this limitation by computing the value outside the call to html and then embedding it via a variable.

HTML> (let ((x (random 10))) (html (:p x)))
<p>1</p>
NIL

But that’s sort of annoying, particularly when you consider that if you could arrange for the form (random 10) to be passed to embed-value instead of embed-code, it’d do exactly what you want. So, you can define a special operator, :print, that’s processed by the FOO language processor according to a different rule than a normal FOO expression. Namely, instead of generating a <print> element, it passes the form in its body to embed-value. Thus, you can generate a paragraph containing a random number like this:

HTML> (html (:p (:print (random 10))))
<p>9</p>
NIL

Obviously, this special operator is useful only in compiled FOO code since embed-value doesn’t work in the interpreter. Another special operator that can be used in both interpreted and compiled FOO code is :format, which lets you generate output using the **FORMAT** function. The arguments to the :format special operator are a string used as a format control string and then any arguments to be interpolated. When all the arguments to :format are self-evaluating objects, a string is generated by passing them to **FORMAT**, and that string is then emitted like any other string. This allows such :format forms to be used in FOO passed to emit-html. In compiled FOO, the arguments to :format can be any Lisp expressions.

Other special operators provide control over what characters are automatically escaped and to explicitly emit newline characters: the :noescape special operator causes all the forms in its body to be evaluated as regular FOO forms but with *escapes* bound to **NIL**, while :attribute evaluates the forms in its body with *escapes* bound to *attribute-escapes*. And :newline is translated into code to emit an explicit newline.

So, how do you define special operators? There are two aspects to processing special operators: how does the language processor recognize forms that use special operators, and how does it know what code to run to process each special operator?

You could hack process-sexp-html to recognize each special operator and handle it in the appropriate manner—special operators are, logically, part of the implementation of the language, and there aren’t going to be that many of them. However, it’d be nice to have a slightly more modular way to add new special operators—not because users of FOO will be able to but just for your own sanity.

Define a special form as any list whose **CAR** is a symbol that’s the name of a special operator. You can mark the names of special operators by adding a non-**NIL** value to the symbol’s property list under the key html-special-operator. So, you can define a function that tests whether a given form is a special form like this:

(defun special-form-p (form)
  (and (consp form) (symbolp (car form)) (get (car form) 'html-special-operator)))

The code that implements each special operator is responsible for taking apart the rest of the list however it sees fit and doing whatever the semantics of the special operator require. Assuming you’ll also define a function process-special-form, which will take the language processor and a special form and run the appropriate code to generate a sequence of calls on the processor object, you can augment the top-level process function to handle special forms like this:

(defun process (processor form)
  (cond
    ((special-form-p form) (process-special-form processor form))
    ((sexp-html-p form)    (process-sexp-html processor form))
    ((consp form)          (embed-code processor form))
    (t                     (embed-value processor form))))

You must add the special-form-p clause first because special forms can look, syntactically, like regular FOO expressions just the way Common Lisp’s special forms can look like regular function calls.

Now you just need to implement process-special-form. Rather than define a single monolithic function that implements all the special operators, you should define a macro that allows you to define special operators much like regular functions and that also takes care of adding the html-special-operator entry to the property list of the special operator’s name. In fact, the value you store in the property list can be a function that implements the special operator. Here’s the macro:

(defmacro define-html-special-operator (name (processor &rest other-parameters) &body body)
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (setf (get ',name 'html-special-operator)
           (lambda (,processor ,@other-parameters) ,@body))))

This is a fairly advanced type of macro, but if you take it one line at a time, there’s nothing all that tricky about it. To see how it works, take a simple use of the macro, the definition of the special operator :noescape, and look at the macro expansion. If you write this:

(define-html-special-operator :noescape (processor &rest body)
  (let ((*escapes* nil))
    (loop for exp in body do (process processor exp))))

it’s as if you had written this:

(eval-when (:compile-toplevel :load-toplevel :execute)
  (setf (get ':noescape 'html-special-operator)
        (lambda (processor &rest body)
          (let ((*escapes* nil))
            (loop for exp in body do (process processor exp))))))

The **EVAL-WHEN** special operator, as I discussed in Chapter 20, ensures that the effects of code in its body will be made visible during compilation when you compile with **COMPILE-FILE**. This matters if you want to use define-html-special-operator in a file and then use the just-defined special operator in that same file.

Then the **SETF** expression sets the property html-special-operator on the symbol :noescape to an anonymous function with the same parameter list as was specified in define-html-special-operator. By defining define-html-special-operator to split the parameter list in two parts, processor and everything else, you ensure that all special operators accept at least one argument.

The body of the anonymous function is then the body provided to define-html-special-operator. The job of the anonymous function is to implement the special operator by making the appropriate calls on the backend interface to generate the correct HTML or the code that will generate it. It can also use process to evaluate an expression as a FOO form.

The :noescape special operator is particularly simple—all it does is pass the forms in its body to process with *escapes* bound to **NIL**. In other words, this special operator disables the normal character escaping preformed by process-sexp-html.

With special operators defined this way, all process-special-form has to do is look up the anonymous function in the property list of the special operator’s name and **APPLY** it to the processor and rest of the form.

(defun process-special-form (processor form)
  (apply (get (car form) 'html-special-operator) processor (rest form)))

Now you’re ready to define the five remaining FOO special operators. Similar to :noescape is :attribute, which evaluates the forms in its body with *escapes* bound to *attribute-escapes*. This special operator is useful if you want to write helper functions that output attribute values. If you write a function like this:

(defun foo-value (something)
  (html (:print (frob something))))

the html macro is going to generate code that escapes the characters in *element-escapes*. But if you’re planning to use foo-value like this:

(html (:p :style (foo-value 42) "Foo"))

then you want it to generate code that uses *attribute-escapes*. So, instead, you can write it like this:2

(defun foo-value (something)
  (html (:attribute (:print (frob something)))))

The definition of :attribute looks like this:

(define-html-special-operator :attribute (processor &rest body)
  (let ((*escapes* *attribute-escapes*))
    (loop for exp in body do (process processor exp))))

The next two special operators, :print and :format, are used to output values. The :print special operator, as I discussed earlier, is used in compiled FOO programs to embed the value of an arbitrary Lisp expression. The :format special operator is more or less equivalent to generating a string with (format nil ...) and then embedding it. The primary reason to define :format as a special operator is for convenience. This:

(:format "Foo: ~d" x)

is nicer than this:

(:print (format nil "Foo: ~d" x))

It also has the slight advantage that if you use :format with arguments that are all self-evaluating, FOO can evaluate the :format at compile time rather than waiting until runtime. The definitions of :print and :format are as follows:

(define-html-special-operator :print (processor form)
  (cond
    ((self-evaluating-p form)
     (warn "Redundant :print of self-evaluating form ~s" form)
     (process-sexp-html processor form))
    (t
     (embed-value processor form))))
(define-html-special-operator :format (processor &rest args)
  (if (every #'self-evaluating-p args)
    (process-sexp-html processor (apply #'format nil args))
    (embed-value processor `(format nil ,@args))))

The :newline special operator forces an output of a literal newline, which is occasionally handy.

(define-html-special-operator :newline (processor)
  (newline processor))

Finally, the :progn special operator is analogous to the **PROGN** special operator in Common Lisp. It simply processes the forms in its body in sequence.

(define-html-special-operator :progn (processor &rest body)
  (loop for exp in body do (process processor exp)))

In other words, the following:

(html (:p (:progn "Foo " (:i "bar") " baz")))

will generate the same code as this:

(html (:p "Foo " (:i "bar") " baz"))

This might seem like a strange thing to need since normal FOO expressions can have any number of forms in their body. However, this special operator will come in quite handy in one situation—when writing FOO macros, which brings you to the last language feature you need to implement.