What is a "runtime context"?

The with context takes care that on entry, the __enter__ method is called and the given var is set to whatever __enter__ returns.

In most cases, that is the object which is worked on previously - in the file case, it is - but e.g. on a database, not the connection object, but a cursor object is returned.

The file example can be extended like this:

f1 = open("debuglog","a")
with f1 as f2:
    print f1 is f2

which will print True as here, the file object is returned by __enter__. (From its point of view, self.)

A database works like

d = connect(...)
with d as c:
    print d is c # False
    print d, c

Here, d and c are completely different: d is the connection to the database, c is a cursor used for one transaction.

The with clause is terminated by a call to __exit__() which is given the state of execution of the clause - either success or failure. In this case, the __exit__() method can act appropriately.

In the file example, the file is closed no matter if there was an error or not.

In the database example, normally the transaction is committed on success and rolled back on failure.

The context manager is for easy initialisation and cleanup of things like exactly these - files, databases etc.

There is no direct correspondence in C or C++ that I am aware of.

C knows no concept of exception, so none can be caught in a __exit__(). C++ knows exceptions, and there seems to be ways to do soo (look below at the comments).

The with statement was introduced in PEP 343. This PEP also introduced a new term, "context manager", and defined what that term means.

Briefly, a "context manager" is an object that has special method functions .__enter__() and .__exit__(). The with statement guarantees that the .__enter__() method will be called to set up the block of code indented under the with statement, and also guarantees that the .__exit__() method function will be called at the time of exit from the block of code (no matter how the block is exited; for example, if the code raises an exception, .__exit__() will still be called).

http://www.python.org/dev/peps/pep-0343/

http://docs.python.org/2/reference/datamodel.html?highlight=context%20manager#with-statement-context-managers

The with statement is now the preferred way to handle any task that has a well-defined setup and teardown. Working with a file, for example:

with open(file_name) as f:
    # do something with file

You know the file will be properly closed when you are done.

Another great example is a resource lock:

with acquire_lock(my_lock):
    # do something

You know the code won't run until you get the lock, and as soon as the code is done the lock will be released. I don't often do multithreaded coding in Python, but when I did, this statement made sure that the lock was always released, even in the face of an exception.

P.S. I did a Google search online for examples of context managers and I found this nifty one: a context manager that executes a Python block in a specific directory.

http://ralsina.me/weblog/posts/BB963.html

EDIT:

The runtime context is the environment that is set up by the call to .__enter__() and torn down by the call to .__exit__(). In my example of acquiring a lock, the block of code runs in the context of having a lock available. In the example of reading a file, the block of code runs in the context of the file being open.

There isn't any secret magic inside Python for this. There is no special scoping, no internal stack, and nothing special in the parser. You simply write two method functions, .__enter__() and .__exit__() and Python calls them at specific points for your with statement.

Look again at this section from the PEP:

Remember, PEP 310 proposes roughly this syntax (the "VAR =" part is optional):

    with VAR = EXPR:
        BLOCK

which roughly translates into this:

    VAR = EXPR
    VAR.__enter__()
    try:
        BLOCK
    finally:
        VAR.__exit__()

In both examples, BLOCK is a block of code that runs in a specific runtime context that is set up by the call to VAR.__enter__() and torn down by VAR.__exit__().

There are two main benefits to the with statement and the way it is all set up.

The more concrete benefit is that it's "syntactic sugar". I would much rather write a two-line with statement than a six-line sequence of statements; it's easier two write the shorter one, it looks nicer and is easier to understand, and it is easier to get right. Six lines versus two means more chances to screw things up. (And before the with statement, I was usually sloppy about wrapping file I/O in a try block; I only did it sometimes. Now I always use with and always get the exception handling.)

The more abstract benefit is that this gives us a new way to think about designing our programs. Raymond Hettinger, in a talk at PyCon 2013, put it this way: when we are writing programs we look for common parts that we can factor out into functions. If we have code like this:

A
B
C
D
E

F
B
C
D
G

we can easily make a function:

def BCD():
    B
    C
    D

A
BCD()
E

F
BCD()
G

But we have never had a really clean way to do this with setup/teardown. When we have a lot of code like this:

A
BCD()
E

A
XYZ()
E

A
PDQ()
E

Now we can define a context manager and rewrite the above:

with contextA:
    BCD()

with contextA:
    XYZ()

with contextA:
    PDQ()

So now we can think about our programs and look for setup/teardown that can be abstracted into a "context manager". Raymond Hettinger showed several new "context manager" recipes he had invented (and I'm racking my brain trying to remember an example or two for you).

EDIT: Okay, I just remembered one. Raymond Hettinger showed a recipe, that will be built in to Python 3.4, for using a with statement to ignore an exception within a block. See it here: https://stackoverflow.com/a/15566001/166949

P.S. I've done my best to give the sense of what he was saying... if I have made any mistake or misstated anything, it's on me and not on him. (And he posts on StackOverflow sometimes so he might just see this and correct me if I've messed anything up.)

EDIT: You've updated the question with more text. I'll answer it specifically as well.

is this what Beazley means when he talks about 'runtime context', that f is scoped only within the block and looses all significance outside the with-block?? Why does he say that the statements "execute inside a runtime context"??? Is this like an "eval"??

Actually, f is not scoped only within the block. When you bind a name using the as keyword in a with statement, the name remains bound after the block.

The "runtime context" is an informal concept and it means "the state set up by the .__enter__() method function call and torn down by the .__exit__() method function call." Again, I think the best example is the one about getting a lock before the code runs. The block of code runs in the "context" of having the lock.

I understand that open returns an object that is "not ... assigned to var"?? Why isn't it assigned to var? What does Beazley mean by making a statement like that?

Okay, suppose we have an object, let's call it k. k implements a "context manager", which means that it has method functions k.__enter__() and k.__exit__(). Now we do this:

with k as x:
    # do something

What David Beazley wants you to know is that x will not necessarily be bound to k. x will be bound to whatever k.__enter__() returns. k.__enter__() is free to return a reference to k itself, but is also free to return something else. In this case:

with open(some_file) as f:
    # do something

The call to open() returns an open file object, which works as a context manager, and its .__enter__() method function really does just return a reference to itself.

I think most context managers return a reference to self. Since it's an object it can have any number of member variables, so it can return any number of values in a convenient way. But it isn't required.

For example, there could be a context manager that starts a daemon running in the .__enter__() function, and returns the process ID number of the daemon from the .__enter__() function. Then the .__exit__() function would shut down the daemon. Usage:

with start_daemon("parrot") as pid:
    print("Parrot daemon running as PID {}".format(pid))
    daemon = lookup_daemon_by_pid(pid)
    daemon.send_message("test")

But you could just as well return the context manager object itself with any values you need tucked inside:

with start_daemon("parrot") as daemon:
    print("Parrot daemon running as PID {}".format(daemon.pid))
    daemon.send_message("test")

If we need the PID of the daemon, we can just put it in a .pid member of the object. And later if we need something else we can just tuck that in there as well.