Cartesian Theater Cartesian Theater - 1 year ago 41
Python Question

"with", context manager, python: What's going on in simple terms?

Novice Python coder here coming from a Java background. I'm still puzzled by this:

with open(...) as f:

even after Googling and reading some of the answers here (I just couldn't get my head around them).

My understanding is that there is this thing called a context manager that is some sort of wrapper that contains a reference to a file that is created. Regarding

as f:

the 'as' above like the 'as' below

import numpy as np

It's just an alias. 'f' doesn't refer to a file, but to the context manager. The context manager, using the decorator pattern, implements all the methods the file that is opened does, so that I can treat it like a file object (and get at the file object by calling the appropriate methods, which will be called on the file inside the context manager). And, of course, the file is closed when the block completes (the whole point of this).

This begs the question: Does open() in general return a file (or a reference to a file), or a context manager? Does it return context managers in general, and that's what we've been using all the time without knowing it? Or does it return file types except in this special context when returns something different like a context manager.

Is this anywhere near right? Would anyone like to clarify?

Answer Source

File objects are themselves context managers, in that they have __enter__ and __exit__ methods. with notifies the file object when the context is entered and exited (by calling __enter__ and __exit__, respectively), and this is how a file object "knows" to close the file. There is no wrapper object involved here; file objects provide those two methods (in Java terms you could say that file objects implement the context manager interface).

Note that as is not an alias just like import module as altname; instead, the return value of contextmanager.__enter__() is assigned to the target. The fileobject.__enter__() method returns self (so the file object itself), to make it easier to use the syntax:

with open(...) as fileobj:

If fileobject.__enter__() did not do this but either returned None or another object, you couldn't inline the open() call; to keep a reference to the returned file object you'd have to assign the result of open() to a variable first before using it as a context manager:

fileobj = open(...)
with fileobj as something_enter_returned:


fileobj = open(...)
with fileobj:  # no as, ignore whatever fileobj.__enter__() produced 

Note that nothing stops you from using the latter pattern in your own code; you don't have to use an as target part here if you already have another reference to the file object, or simply don't need to even access the file object further.

However, other context managers could return something different. Some database connectors return a database cursor:

conn = database.connect(....)
with conn as cursor:

and exiting the context causes the transaction to be committed or rolled back (depending on wether or not there was an exception).