Monstrous polymorphism and a Python post-import hook decorator
Wednesday, April 15th, 2009I queue up a few thousand things to do before I get on an airplane: synchronize two-thousand Google Reader entries, load up a bunch of websites I’ve been meaning to read, and make sure for-fun projects are pulled from their most updated branches.
Then, once I get up in the air, I realize that I don’t really want to do 90% of those things crammed into a seat with no elbow room. I end up doing one or two. Along with reading Stevey’s Drunken Blog Rant: When Polymorphism Fails, this entry is all the productivity I can claim. The full code repository for this entry is online if you’d like to follow along.
Polymorphism Recap
The word “polymorphic” comes from Greek roots meaning “many shaped.” (Or they lied to me in school — one of those.) From a worldly perspective I can see this meaning two things:
- A single object can take on many shapes, or
- Requirements for a general “shape” can be satisfied by different categories of objects.
As it turns out, both of these concepts apply to the Object-Oriented programming, but the canonical meaning is the latter. [*] As Yegge says:
If you have a bunch of similar objects [...], and they’re all supposed to respond differently to some situation, then you add a virtual method to them and implement it differently for each object.
(If you don’t know what a virtual method is, the Wikipedia page has an alternate explanation.)
Yegge’s Example
Yegge demonstrates that strictly adhering to the principles of polymorphism does not always produce the best design:
Let’s say you’ve got a big installed base of monsters. [...] Now let’s say one of your users wants to come in and write a little OpinionatedElf monster. [...] Let’s say the OpinionatedElf’s sole purpose in life is to proclaim whether it likes other monsters or not. It sits on your shoulder, and whenever you run into, say, an Orc, it screams bloodthirstily: “I hate Orcs!!! Aaaaaargh!!!” (This, incidentally, is how I feel about C++.)
The polymorphic approach to this problem is simple: go through every one of your 150 monsters and add a doesMrOpinionatedElfHateYou() method.
This is a great counterexample — it induces an instant recognition of absurdity.
He then touches on the fact that dynamic languages allow you to do neat things consistent with polymorphism due to the flexibility of the object structure (which is typically just a hash map from identifiers to arbitrary object values):
I guess if you could somehow enumerate all the classes in the system, and check if they derive from Monster, then you could do this whole thing in a few lines of code. In Ruby, I bet you can… but only for the already-loaded classes. It doesn’t work for classes still sitting on disk! You could solve that, but then there’s the network…
This is clearly impractical, but I figured there was some exploratory value to implementing this challenge in Python. This entry is a small walk-through for code to detect interface conformity by inspection, enumerate the classes in the environment, manipulate classes in place, and add an import hook to manipulate classes loaded from future modules.
The Antagonist
Double entendre intended. :-)
class OpinionatedElf(object): is_liked_by_class_name = { 'OpinionatedElf': True, 'Orc': False, 'Troll': False} def __init__(self, name): self.name = name def be_scary(self): print("I'm small, ugly, and don't like the cut of your jib!") def proclaim_penchance(self, other): if not IMonster.is_conforming(other): print("I can't even tell what that is!") return is_liked = other.is_liked_by_elf() class_name = other.__class__.__name__ if is_liked is None: print("I'm not sure how I feel about %s" % class_name) return emotion = 'love' if is_liked else 'hate' print('I %s %s!!! Aaaaaargh!!!' % (emotion, other.__class__.__name__))
Determining which Classes are Monsters
First of all, Python doesn’t require (nor does it encourage) a rigid type hierarchy. Python’s all about the interfaces, which are often implicit. Step one is to create a way to recognize classes that implement the monster interface: [%]
class IMonster(object): required_methods = ['be_scary'] def be_scary(self): raise NotImplementedError @classmethod def is_conforming(cls, object): result = all(callable(getattr(object, attr_name, None)) for attr_name in cls.required_methods) logging.debug('%s conforms? %s', object, result) return result assert IMonster.is_conforming(IMonster)
This is a simple little class — there are better third party libraries to use if you want real interface functionality (i.e. more generic conformity testing and Design By Contract).
Enumerating the Classes in the Environment
All of the modules that have been loaded into the Python environment are placed into sys.modules. By inspecting each of these modules, we can manipulate the classes contained inside if they conform to our monster interface.
for name, module in sys.modules.iteritems(): extend_monsters(module)
The extend_monsters function is a bit nuanced because immutable modules also live in sys.modules. We skip those, along with abstract base classes, which have trouble with inspect.getmembers():
def extend_monsters(module, extension_tag='_opinionated_extended'): """Extend monsters in the module's top-level namespace to tell if they are liked by the :class:`OpinionatedElf`. and tag it with the :param:`extension_tag` as a flag name. Do not attempt to extend already-flagged modules. Do not clobber existing methods with the extension method name. Warning: swallows exceptional cases where :param:`module` is builtin, frozen, or None. """ name = module.__name__ if module else None logging.info('Instrumenting module %s', name) if not module or imp.is_builtin(name) or imp.is_frozen(name) \ or getattr(module, extension_tag, False): logging.info('Skipping module: %s', name) return module._opinionated_instrumented = True classes = inspect.getmembers(module, inspect.isclass) for name, cls in classes: logging.debug('%s: %s', name, cls) try: conforming = IMonster.is_conforming(cls) except AttributeError, e: if '__abstractmethods__' in str(e): # Abstract class. continue raise if not conforming: continue class_name = cls.__name__ logging.debug('Instrumenting class %s', class_name) attr_name = 'is_liked_by_elf' if hasattr(cls, attr_name): # Don't clobber existing methods. logging.warn('Method already exists: %s', cls) continue logging.info('Setting %s on %s', attr_name, class_name) setattr(cls, attr_name, lambda self: OpinionatedElf.is_liked_by_class_name.get( self.__class__.__name__, None))
If we were going to be thorough, we would recurse on the members of the class to see if the class scope was enclosing any more IMonster classes, but you’re never really going to find them all: if a module defines a monster class in a function-local scope, there’s no good way to get the local class statement and modify it through inspection.
In any case, we’re at the point where we can modify all monsters in the top-level namespace of already-loaded modules. What about modules that we have yet to load?
Post-import Hook
There is no standard post-import hook (that I know of) in Python. PEP 369 looks promising, but I couldn’t find any record of additional work being done on it. The current import hooks, described in PEP 302, are all pre-import hooks. As such, you have to decorate the __import__ builtin, wrapping the original with your intended post-input functionality, like so: [^]
def import_decorator(old_import, post_processor): """ :param old_import: The import function to decorate, most likely ``__builtin__.__import__``. :param post_processor: Function of the form `post_processor(module) -> module`. :return: A new import function, most likely to be assigned to ``__builtin__.__import__``. """ assert all(callable(fun) for fun in (old_import, post_processor)) def new_import(*args, **kwargs): module = old_import(*args, **kwargs) return post_processor(module) return new_import
After which we can replace the old __import__ with its decorated counterpart:
__builtin__.__import__ = import_decorator(__builtin__.__import__, extend_monsters)
The Network
Yegge brings up the issue of dynamically generated classes by mentioning network communications, calling to mind examples such as Java’s RMI and CORBA. This is a scary place to go, even just conceptualizing. If metaclasses are used, I don’t see any difficulty in decorating __new__ with the same kind of inspection we employed above; however, code generation presents potentially insurmountable problems.
Decorating the eval family of functions to modify new classes created seems possible, but it would be challenging and requires additional research on my part. exec is a keyword/statement, which I would think is a hopeless cause.
Footnotes
[*] In accordance with the former, an object can implement many interfaces.
[^] This function is actually a generic decorate-with-post-processing closure, but I added the references to import for more explicit documentation.