The call order of python3 metaclass
The trick, identified
Update 2: Based on behavior, the fact that M0.__call__
is called below must be a side effect of this line in builtin__build_class
in the CPython source (Python/bltinmodule.c
).
In order to define a class that has a metaclass, we call the metaclass's __prepare__
, __new__
, and __init__
as usual. This creates a class—in the example below, Meta
—that is callable, but its internal PyFunction_GET_CODE
slot points not to its own __call__
but rather to its metaclass's __call__
. Hence if we call Meta()
(the metaclass object), we invoke M0.__call__
:
print("call Meta")
print("Meta returns:", Meta('name', (), {}))
print("finished calling Meta")
produces:
call Meta
M0 __call__: mmcls=<class '__main__.Meta'>, args=('name', (), {}), kwargs={}
Meta __new__: mcs=<class '__main__.Meta'>, name='name', bases=(), attrs={}, kwargs={}
Meta __init__: mcs=<class '__main__.name'>, name='name', bases=(), attrs={}, kwargs={}
Meta returns: <class '__main__.name'>
finished calling Meta
In other words, we see that Meta
acts like type
, but it (rather magically and not very well documented) invokes M0.__call__
. This is no doubt due to looking up __call__
in the class's type, rather than in an instance of the class (and indeed there is no instance except for the one we're creating). This is in fact the general case: it falls out of the fact that we call __call__
on the type of Meta
, and the type of Meta
is M0
:
print("type(Meta) =", type(Meta))
prints:
type(Meta) = <class '__main__.M0'>
which explains where this comes from. (I still think this should be emphasized in the documentation, which also should describe the constraints on metaclass typing—these are enforced in _calculate_winner
in Lib/types.py
and, as C code, in _PyType_CalculateMetaclass
in Objects/typeobject.c.)
Updated original answer
I don't know where your diagram came from, but it's wrong. UPDATE: You can in fact have a metaclass for your metaclass; see jsbueno's answer, and I've updated the example below. New sentences / text are in bold, except for the final section describing my puzzlement at the apparent lack of documentation.
Your existing metaclass code has at least one error. Most significantly, its __prepare__
needs to be a class-method. See also Using the __call__ method of a metaclass instead of __new__? and PEP 3115. And, to use a meta-meta-class, your metaclass needs to have a metaclass of its own, not a base class.
Chris's answer contains correct definitions. But there are some unfortunate asymmetries between metaclass method arguments and class method arguments, which I'll illustrate below.
One other thing that may help: note that the metaclass __prepare__
method is called before creating any instances of class B
: it is called when class B
itself is being defined. To show this, here is a corrected metaclass-and-class. I have also added a few more illustrators. I've added a meta-metaclass as well, based on jsbueno's answer. I cannot find formal Python documentation on this, but I've updated the output below.
class M0(type):
def __call__(mmcls, *args, **kwargs):
print("M0 __call__: mmcls={!r}, "
"args={!r}, kwargs={!r}".format(mmcls, args, kwargs))
return super().__call__(*args, **kwargs)
class Meta(type, metaclass=M0):
def __call__(cls, *args, **kwargs):
print("Meta __call__: cls={!r}, "
"args={!r}, kwargs={!r}".format(cls, args, kwargs))
return super().__call__(*args, **kwargs)
def __new__(mcs, name, bases, attrs, **kwargs):
print("Meta __new__: mcs={!r}, name={!r}, bases={!r}, "
"attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
return super().__new__(mcs, name, bases, attrs)
def __init__(mcs, name, bases, attrs, **kwargs):
print("Meta __init__: mcs={!r}, name={!r}, bases={!r}, "
"attrs={!r}, kwargs={!r}".format(mcs, name, bases, attrs, kwargs))
super().__init__(name, bases, attrs, **kwargs)
@classmethod
def __prepare__(cls, name, bases, **kwargs):
print("Meta __prepare__: name={!r}, "
"bases={!r}, kwargs={!r}".format(name, bases, kwargs))
return {}
print("about to create class A")
class A(metaclass=Meta): pass
print("finished creating class A")
print("about to create class B")
class B(A, metaclass=Meta, foo=3):
@staticmethod
def __new__(cls, *args, **kwargs):
print("B __new__: cls={!r}, "
"args={!r}, kwargs={!r}".format(cls, args, kwargs))
return super().__new__(cls)
def __init__(self, *args, **kwargs):
print("B __init__: args={!r}, kwargs={!r}, ".format(args, kwargs))
print("finished creating class B")
print("about to create instance b = B()")
b = B('hello', bar=7)
print("finished creating instance b")
Now, let's observe what happens when I run this, and take each piece apart:
$ python3.6 meta.py
about to create class A
Meta __prepare__: name='A', bases=(), kwargs={}
M0 __call__: mmcls=<class '__main__.Meta'>, args=('A', (), {'__module__': '__main__', '__qualname__': 'A'}), kwargs={}
Meta __new__: mcs=<class '__main__.Meta'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
Meta __init__: mcs=<class '__main__.A'>, name='A', bases=(), attrs={'__module__': '__main__', '__qualname__': 'A'}, kwargs={}
finished creating class A
To create class A
itself, Python first calls the metaclass's __prepare__
, delivering it the name of the class (A
), the list of base classes (an empty tuple—it's called a list but is actually a tuple), and any keyword arguments (none). As PEP 3115 notes, the metaclass needs to return a dictionary or dict
-like object; this one does by just returning an empty dictionary, so we are good here.
(I don't print cls
itself here, but if you do, you will see it is just <class '__main__.Meta'>
.)
Next, having gotten a dictionary from __prepare__
, Python first calls the meta-meta __call__
, i.e., M0.__call__
, passing the entire set of arguments as the args
tuple. It then populates the __prepare__
-supplied dictionary with all the attributes for the class, passing this as the attrs
to the metaclass __new__
and __init__
. If you print the id
of the dictionary returned from __prepare__
and passed to __new__
and __init__
you will see they all match.
Since class A
has no methods or data members, we see only the magic __module__
and __qualname__
attributes here. We also see no keyword arguments, so now let's move on to creating class B
:
about to create class B
Meta __prepare__: name='B', bases=(<class '__main__.A'>,), kwargs={'foo': 3}
M0 __call__: mmcls=<class '__main__.Meta'>, args=('B', (<class '__main__.A'>,), {'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0a58>, '__init__': <function B.__init__ at 0x800ad2840>, '__classcell__': <cell at 0x800a749d8: empty>}), kwargs={'foo': 3}
Meta __new__: mcs=<class '__main__.Meta'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: empty>}, kwargs={'foo': 3}
Meta __init__: mcs=<class '__main__.B'>, name='B', bases=(<class '__main__.A'>,), attrs={'__module__': '__main__', '__qualname__': 'B', '__new__': <staticmethod object at 0x800ad0940>, '__init__': <function B.__init__ at 0x800ad27b8>, '__classcell__': <cell at 0x800a745b8: Meta object at 0x802047018>}, kwargs={'foo': 3}
finished creating class B
This one is rather more interesting. Now we have one base class, namely __main__.A
. Class B
also defines several methods (__new__
and __init__
) and we see them in the attrs
dictionaries passed to the metaclass __new__
and __init__
methods (which, remember, are just the now-populated dictionary returned by the metaclass's __prepare__
). As before, the passing-on happens through the meta-meta-class M0.__call__
. We also see one keyword argument throughout, {'foo': 3}
. In the attribute dictionary, we can also observe the magic __classcell__
entry: see Provide __classcell__ example for Python 3.6 metaclass for a short description as to what this is about, but to be, er, super-short, it's for making super()
work.
The keyword argument is passed to all three metaclass methods, plus that of the meta-meta-class. (I'm not quite sure why. Note that modifying the dictionary in any metaclass method does not affect it in any other, as it's a copy each time of the original keyword arguments. However, we can modify it in the meta-meta-class: add kwargs.pop('foo', None)
to M0.__call__
to observe this.)
Now that we have our classes A
and B
, we can get on to the process of creating an actual instance of class B
. Now we see the metaclass's __call__
invoked (not the meta-meta-class's):
about to create instance b = B()
Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
It's possible to change the args
or kwargs
passed on, but I don't; the sample code above winds up calling type.__call__(cls, *args, **kwargs)
(through the magic of super().__call__
). This in turn calls B.__new__
and B.__init__
:
B __new__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
B __init__: args=('hello',), kwargs={'bar': 7},
finished creating instance b
which finishes the realization of the new instance of class B
, which we then bind to the name b
.
Note that B.__new__
says:
return super().__new__(cls)
so we invoke object.__new__
to create the instance—this is more or less a requirement of all versions of Python; you can only "cheat" when you return a singleton instance (ideally, one that's non-modifiable). It's type.__call__
that calls B.__init__
on this object, passing the arguments and keyword-arguments we passed it. If we replace Meta
's __call__
with:
def __call__(cls, *args, **kwargs):
print("Meta __call__: cls={!r}, "
"args={!r}, kwargs={!r}".format(cls, args, kwargs))
return object.__new__(cls)
we will see that B.__new__
and B.__init__
are never called:
about to create instance b = B()
Meta __call__: cls=<class '__main__.B'>, args=('hello',), kwargs={'bar': 7}
finished creating instance b
This would, in effect, create a useless/uninitialized instance b
. It's therefore critical that the metaclass __call__
method call the underlying class's __init__
, usually by invoking type.__call__
via super().__call__
. If the underlying class has a __new__
, the metaclass should call that first, again usually by invoking type.__call__
.
Side note: what the documentation says
To quote section 3.3.3.6:
Once the class namespace has been populated by executing the class body, the class object is created by calling
metaclass(name, bases, namespace, **kwds)
(the additional keywords passed here are the same as those passed to__prepare__
).
This explains the call to Meta.__call__
when creating b
as an instance of class B
, but not the fact that Python first calls M0.__call__
before calling Meta.__new__
and Meta.__init__
when creating classes A
and B
themselves.
The next paragraph mentions the __classcell__
entry; the one after that goes on to describe the use of __set_name__
and __init_subclass__
hooks. Nothing here tells us how or why Python calls M0.__call__
at this point.
Earlier, in sections 3.3.3.3 through 3.3.3.5, the documentation describes the process of determining the metaclass, preparing the class namespace, and executing the class body. This is where the meta-metaclass action should be described, but isn't.
Several additional sections describe a few additional constraints. One important one is 3.3.10, which talks about how special methods are found via the object type, bypassing both regular member attribute lookups and even (sometimes) a metaclass getattribute, saying:
Bypassing the
__getattribute__()
machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).
Update 2: This is really the secret of the trick: the special __call__
method is found via the type's type. If the metaclass has a metaclass, the meta-meta-class provides the __call__
slot; otherwise the type of the metaclass is type
, so that the __call__
slot is type.__call__
.
Despite @torek's lenghty answer, with a lot of other details on class creation, what you brought together to this question is mostly correct.
The only thing that is wrong in your code, which propably puzzled you is that te class you call Meta
have to be itself the metaclass from SubMeta
and not its parent.
Simply change Submeta
declaration to:
class SubMeta(type, metaclass=Meta):
...
(No need for it to inherit from "Meta" as well - it can derive only from type
. It is otherwise though to think of a customization to type.__call__
that would be usefull at the same time for creating instances of your classes (that is when SubMeta.__call__
is called), and your classes themselves (Meta.__call__
called))
Here is another, shorter example I just typed at the terminal. Sorry for the naming inconsistencies, and for being less complete - but it shows the main point:
class M(type):
def __call__(mmcls, *args, **kwargs):
print("M's call", args, kwargs)
return super().__call__(*args, **kwargs)
class MM(type, metaclass=M):
def __prepare__(cls, *args, **kw):
print("MM Prepare")
return {}
def __new__(mcls, *args, **kw):
print("MM __new__")
return super().__new__(mcls, *args, **kw)
class klass(metaclass=MM):
pass
Upon processing the klass
body, Python output was:
MM Prepare
M's call ('klass', (), {'__module__': '__main__', '__qualname__': 'klass'}) {}
MM __new__
Moreover
As you can see from this, with a meta-meta class it is possible to customize the call order and parameters to the metaclass __init__
and __new__
, but there are still steps that can't be customized from pure-Python code, and would require native calls to API's (and possibly raw object structure manipulation) - that are:
- One can't control the call to
__prepare__
- One can't control the call to
__init_subclass__
on the created classes - One can control when descriptors'
__set_name__
are called
The last two items take place after meta-meta's __call__
return, and before resuming the flow to the module where the class module is.