Dynamic and Lexical variables in Common Lisp
Maybe this example will help.
;; the lexical version
(let ((x 10))
(defun lex-foo ()
(format t "Before assignment~18tX: ~d~%" x)
(setf x (+ 1 x))
(format t "After assignment~18tX: ~d~%" x)))
(defun lex-bar ()
(lex-foo)
(let ((x 20)) ;; does not do anything
(lex-foo))
(lex-foo))
;; CL-USER> (lex-bar)
;; Before assignment X: 10
;; After assignment X: 11
;; Before assignment X: 11
;; After assignment X: 12
;; Before assignment X: 12
;; After assignment X: 13
;; the dynamic version
(defvar *x* 10)
(defun dyn-foo ()
(format t "Before assignment~18tX: ~d~%" *x*)
(setf *x* (+ 1 *x*))
(format t "After assignment~18tX: ~d~%" *x*))
(defun dyn-bar()
(dyn-foo)
(let ((*x* 20))
(dyn-foo))
(dyn-foo))
;; CL-USER> (dyn-bar)
;; Before assignment X: 10
;; After assignment X: 11
;; Before assignment X: 20
;; After assignment X: 21
;; Before assignment X: 11
;; After assignment X: 12
;; the special version
(defun special-foo ()
(declare (special *y*))
(format t "Before assignment~18tX: ~d~%" *y*)
(setf *y* (+ 1 *y*))
(format t "After assignment~18tX: ~d~%" *y*))
(defun special-bar ()
(let ((*y* 10))
(declare (special *y*))
(special-foo)
(let ((*y* 20))
(declare (special *y*))
(special-foo))
(special-foo)))
;; CL-USER> (special-bar)
;; Before assignment X: 10
;; After assignment X: 11
;; Before assignment X: 20
;; After assignment X: 21
;; Before assignment X: 11
;; After assignment X: 12
What's going on?
You say: feel like there is nothing special is going on here. The outer foo in bar increments the global x, and foo surrounded by let in bar increments the shadowed x. What's the big deal?
The special that's going on here is that LET can shadow the value of *x*. With lexical variables that's not possible.
The code declares *x* to be special via the DEFVAR.
In FOO now the value of *x* is looked up dynamic. FOO will take the current dynamic binding of *x* or, if there is none, the symbol value of the symbol *x*. A new dynamic binding can, for example, be introduced with LET.
A lexical variable on the other hand has to be present in the lexical environment somewhere. LET, LAMBDA, DEFUN and others can introduce such lexical variables. See here the lexical variable x introduced in three different ways:
(let ((x 3))
(* (sin x) (cos x)))
(lambda (x)
(* (sin x) (cos x)))
(defun baz (x)
(* (sin x) (cos x)))
If our code were:
(defvar x 0)
(let ((x 3))
(* (sin x) (cos x)))
(lambda (x)
(* (sin x) (cos x)))
(defun baz (x)
(* (sin x) (cos x)))
Then X were special in all three above cases, because of the DEFVAR declaration, which declares X to be special - globally for all levels. Because of this, there is the convention to declare special variables as *X*. Thus only variables with stars around them are special - by convention. That's a useful convention.
In your code you have then:
(defun bar ()
(foo)
(let ((*x* 20))
(foo))
(foo))
Since *x* has be declared special via the DEFVAR above in your code, the LET construct introduces a new dynamic binding for *x*. FOO is then called. Since inside FOO the *x* uses dynamic binding, it looks up the current one and finds that *x* is dynamically bound to 20.
The value of a special variable is found in the current dynamic binding.
Local SPECIAL declarations
There are also local special declarations:
(defun foo-s ()
(declare (special *x*))
(+ *x* 1))
If the variable had been declared special by a DEFVAR or DEFPARAMETER, then the local special declaration can be omitted.
A lexical variable directly references the variable binding:
(defun foo-l (x)
(+ x 1))
Let's see it in practice:
(let ((f (let ((x 10))
(lambda ()
(setq x (+ x 1))))))
(print (funcall f)) ; form 1
(let ((x 20)) ; form 2
(print (funcall f))))
Here all variables are lexical. In form 2 the LET will not shadow the X in our function f. It can't. The function uses the lexical bound variable, introduced by the LET ((X 10). Surrounding the call with another lexically bound X in form 2 has no effect on our function.
Let's try special variables:
(let ((f (let ((x 10))
(declare (special x))
(lambda ()
(setq x (+ x 1))))))
(print (funcall f)) ; form 1
(let ((x 20)) ; form 2
(declare (special x))
(print (funcall f))))
What now? Does that work?
It does not!
The first form calls the function and it tries to look up the dynamic value of X and there is none. We get an error in form 1: X is unbound, because there is no dynamic binding in effect.
Form 2 would work, since the LET with the special declaration introduces a dynamic binding for X.
When a variable is lexically scoped, the system looks to where the function is defined to find the value for a free variable. When a variable is dynamically scoped, the system looks to where the function is called to find the value for the free variable. Variables in Common Lisp are all lexical by default; however, dynamically scoped variables can be defined at the top level using defvar or defparameter.
A simpler example
lexical scoping (with setq):
(setq x 3)
(defun foo () x)
(let ((x 4)) (foo)) ; returns 3
dynamic scoping (with defvar):
(defvar x 3)
(defun foo () x)
(let ((x 4)) (foo)) ; returns 4
How does the let know if a variable is lexical or dynamic? It doesn't. On the other hand, when foo goes to find the value of X, it will initially find the lexical value defined at the top level. It then checks to see if the variable is supposed to be dynamic. If it is, then foo looks to the calling environment, which, in this case, uses let to overshadow the value of X to be 4.
(note: this is an oversimplification, but it will help to visualize the difference between the different scoping rules)