What is open recursion?
This paper analyzes the possibility of adding OO to ML, with regards to expressivity and complexity. It has the following excerpt on objects, which seems to make this term relatively clear –
3.3. Objects
The simplest form of object is just a record of functions that share a common closure environment that carries the object state (we can call these simple objects). The function members of the record may or may not be defined as mutually recursive. However, if one wants to support inheritance with overriding, the structure of objects becomes more complicated. To enable open recursion, the call-graph of the method functions cannot be hard-wired, but needs to be implemented indirectly, via object self-reference. Object self-reference can be achieved either by construction, making each object a recursive, self-referential value (the fixed-point model), or dynamically, by passing the object as an extra argument on each method call (the self-application or self-passing model).5 In either case, we will call these self-referential objects.
just copying http://www.comlab.ox.ac.uk/people/ralf.hinze/talks/Open.pdf: "Open recursion Another handy feature offered by most languages with objects and classes is the ability for one method body to invoke another method of the same object via a special variable called self or, in some langauges, this. The special behavior of self is that it is late-bound, allowing a method defined in one class to invoke another method that is defined later, in some subclass of the first. "