What can you do with Lisp macros that you can't do with first-class functions?

First of all Lisp has first-class functions too, so you could as well ask: "Why do I need macros in Lisp if I already have first-class functions". The answer to that is that first-class functions don't allow you to play with syntax.

On a cosmetic level, first-class functions allow you to write f(filename, some_function) or f(filename, lambda fh: fh.whatever(x)), but not f(filename, fh, fh.whatever(x)). Though arguably that's a good thing because in that last case it is a lot less clear where fh suddenly comes from.

More importantly functions can only contain code that is valid. So you can't write a higher-order function reverse_function that takes a function as an argument and executes it "in reverse", so that reverse_function(lambda: "hello world" print) would execute print "hello world". With a macro you can do this. Of course this particular example is quite silly, but this ability is enormously useful when embedding domain specific languages.

For example you couldn't implement common lisp's loop construct in python. Hell, you couldn't even implement python's for ... in construct in python if it wasn't really built-in - at least not with that syntax. Sure you could implement something like for(collection, function), but that's a lot less pretty.

Here's Matthias Felleisen's answer from 2002 (via http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg01539.html):

I'd like to propose that there are three disciplined uses of macros:

1. data sublanguages: I can write simple looking expressions and create complex nested lists/arrays/tables with quote, unquote etc neatly dressed up with macros.

2. binding constructs: I can introduce new binding constructs with macros. That helps me get rid of lambda's and with placing things closer together that belong together. For example, one of our teachpacks contains a form
   (web-query ([last-name (string-append "Hello " first-name " what's your last name?"]) ... last-name ... first-name ...) with the obvious interaction between a program and a Web consumer implied.
   [Note: In ML you could write web-query(fn last-name => ...)string_append(...) but by golly that's a pain and an unnecessary pattern.]

3. evaluation reordering: I can introduce constructs that delay/postpone the evaluation of expressions as needed. Think of loops, new conditionals, delay/force, etc.
   [Note: In Haskell, you don't need that one.]

I understand that Lispers use macros for other reasons. In all honesty, I believe that this is partly due to compiler deficiencies, and partly due to "semantic" irregularities in the target language.

I challenge people to address all three issues when they say language X can do what macros can do.

-- Matthias

Felleisen is one of the most influential macro researchers in the field. (I don't know whether he would still agree with this message, though.)

More reading: Paul Graham's On Lisp (http://www.paulgraham.com/onlisp.html; Graham definitely doesn't agree with Felleisen that these are the only useful uses of macros), and Shriram Krishnamurthi's paper "Automata via Macros" (http://www.cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros/).

Macros do code transformations

The macro transforms source code. A lazy evaluation does not. Imagine that you can now write functions which transform arbitrary code to arbitrary different code.

Very simple code transformations

The creation of simple language constructs is also only a very simple example. Consider your example of opening a file:

(with-open-file (stream file :direction :input)
  (do-something stream))

vs.

(call-with-stream (function do-something)
                  file
                  :direction :input)

What the macro gives me is a slightly different syntax and code structure.

Embedded language: advanced iteration constructs

Next consider a slightly different example:

(loop for i from 10 below 20 collect (sqr i))

vs.

(collect-for 10 20 (function sqr))

We can define a function COLLECT-FOR which does the same for a simple loop and has variables for start, end and a step function.

But LOOP provides a new language. The LOOP macro is a compiler for this language. This compiler can do LOOP specific optimizations and can also check the syntax at compile time for this new language. An even more powerful loop macro is ITERATE. These powerful tools on the language level now can be written as libraries without any special compiler support.

Walking the code tree in a macro and making changes

Next another simple example:

(with-slots (age name) some-person
  (print name)
  (princ " "
  (princ age))

vs. something similar:

(flet ((age (person) (slot-value person 'age))
       (name (person) (slot-value person 'name)))
   (print (name))
   (princ " ")
   (princ (age)))

The WITH-SLOTS macro causes the complete walk of the enclosed source tree and replaces the variable name with a call to (SLOT-VALUE SOME-PERSON 'name):

(progn
  (print (slot-value some-person 'name))
  (princ " "
  (princ (slot-value some-person 'age)))

In this case the macro can rewrite selected parts of the code. It understands the structure of the Lisp language and knows that name and age are variables. It also understands that in some situations name and age might not be variables and should not be rewritten. This is an application of a so-called Code Walker, a tool that can walk code trees and make changes to the code tree.

Macros can modify the compile-time environment

Another simple example, the contents of a small file:

(defmacro oneplus (x)
  (print (list 'expanding 'oneplus 'with x))
  `(1+ ,x))

(defun example (a b)
   (+ (oneplus a) (oneplus (* a b))))

In this example we are not interested in the macro ONEPLUS, but in the macro DEFMACRO itself.

What is interesting about it? In Lisp you can have a file with above contents and use the file compiler to compile that file.

;;; Compiling file /private/tmp/test.lisp ...
;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 1
;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3
;;; Source level debugging is on
;;; Source file recording is  on
;;; Cross referencing is on
; (TOP-LEVEL-FORM 0)
; ONEPLUS

(EXPANDING ONEPLUS SOURCE A) 
(EXPANDING ONEPLUS SOURCE (* A B)) 
; EXAMPLE
;; Processing Cross Reference Information

So we see, that the file compiler expands the use of the ONEPLUS macro.

What is special about that? There is a macro definition in the file and in the next form we already use that new macro ONEPLUS. We have never loaded the macro definition into Lisp. Somehow the compiler knows and registers the defined macro ONEPLUS and is then able to use it.

So the macro DEFMACRO registers the newly defined macro ONEPLUS in the compile-time environment, so that the compiler knows about this macro - without loading the code. The macro then can be executed at compile-time during macro expansion.

With a function we can't do that. The compiler creates code for function calls, but does not run them. But a macro can be run at compile time and add 'knowledge' to the compiler. This knowledge then is valid during the run of the compiler and partially forgotten later. DEFMACRO is a macro which executes at compile time and then informs the compile-time environment of a new macro.

Note also that the macro ONEPLUS is also run twice, since it is used twice in the file. The side effect is that it prints something. But ONEPLUS could have also other arbitrary side effects. For example it could check the enclosed source against a rule base and alert you if for example the enclosed code violates some rules (think of a style checker).

That means, that a macro, here DEFMACRO, can change the language and its environment during compilation of a file. In other languages the compiler might provide special compiler directives which will be recognized during compilation. There are many examples for such defining macros influencing the compiler: DEFUN, DEFCLASS, DEFMETHOD, ...

Macros can make the user code shorter

A typical example is the DEFSTRUCT macro for defining record-like data structures.

(defstruct person name age salary)

Above defstruct macro creates code for

• a new structure type person with three slots
• slot accessors for reading and writing the values
• a predicate to check if some object is of class person
• a make-person function to create structure objects
• a printed representation

Additionally it may:

• record the source code
• record the origin of the source code (file, editor buffer, REPL, ...)
• cross-reference the source code

The original code to define the structure is a short line. The expanded code is much longer.

The DEFSTRUCT macro does not need access to a meta-level of the language to create these various things. It just transforms a compact piece of descriptive code into the, typically longer, defining code using the typical language constructs.