blog banner alt

Dynamic Import in Python 3

Last updated Feb 10, 2018


A couple of month ago, I made a post about Factory Patten in Python3. If you’ve looked in the source code on Github, you probably have noticed a few changes, especially some weird looking code in animals/fish/__init__.py.

 

Yes… I’m talking about this..

for (_, name, _) in pkgutil.iter_modules([os.path.dirname(__file__)]):
    imported_module = import_module('.' + name, package='animals.fish')

    class_name = list(filter(lambda x: x != 'AnimalBaseClass' and not x.startswith('__'), 
    				  dir(imported_module)))

    fish_class = getattr(imported_module, class_name[0])

    if issubclass(fish_class, AnimalBaseClass):
        setattr(sys.modules[__name__], name, fish_class)

 

The simple answer is: this is for the automatic import of AnimalBaseClass in the sub-packages. So when you add new fish into the category, you don’t have to keep having to add an import statement in the __init__.py file, to be able to use this nice syntax:

fish_instance = animals.grab('fish.goldfish', name='fishy')

Nice, isn’t it? However, this code is not exactly ideal; as it definitely wouldn’t survive many scenarios, like if there are more modules in the package that are not any animal classes. For example, if I have a utils.py; or if I have other classes or functions within the animal file.

How about I show you how this works, and how we can make this better?

Interested?

Okay, read on!

 

The ingredient - Necessary Imports

You have probably seen a few imports: os, importlib.import_module, pkgutil, os, sys. This time, I’m going to change the import a bit, like so:

import sys
import inspect
import pkgutil
from pathlib import Path
from importlib import import_module

Let’s walk through each of them.

sys: We will only be using this to retrieve the module object of the current package with sys.modules.

inspect: This is a very handy builtin python module to check the nature of the object. This time, we will be using it to check if the target we try to import is a class.

pkgutil: The only method we are interested in from this package is ‘iter_modules’. By passing in a package path(as a list, even if you have only one item!), we get back a generator method that yields pkgnutil.ModuleInfo namedtuple object, which consist of the module finder(of the package path), module name, and a Boolean value of whether it’s a package. Here we only need the module name.

pathlib.Path:  I’m using pathlib here as an alternative to os.path, as this is a more object-oriented approach, and it’s also more readable.

importlib.import_module: This is the core packaging for our dynamic import. It allows you to import a module or class by passing a string, and assign the imported object to a variable. Remember, everything in python is an object, including functions and modules.

 

Walkthrough

We start this with a loop, iterating through the modules in the current package path with pkgutil.iter_modules, like so:

for (_, name, _) in pkgutil.iter_modules([Path(__file__).parent]):

Here we used Path(__file__).parent instead of os.path.dirname(__file__) to get the directory path of the current module, in this case, it’s the  __init__.py file.

Next up is importing of the module itself.

imported_module = import_module('.' + name, package=__name__)

We take the name, with a dot in front for relative import of the package argument. Alternatively, you can also pass in the absolute import path:

imported_module = import_module(f"{__name__}.{name}")

We also take the package name with __name__ as the current package. In our animal.fish package, this would return exactly ‘animal.fish’ as a string value.

Moving on, we will add a for loop to iterated through the attribute names of imported_module.

for i in dir(imported_module):
    attribute = getattr(imported_module, i)

    if inspect.isclass(attribute) and issubclass(attribute, AnimalBaseClass):
        setattr(sys.modules[__name__], name, attribute)

Right! I know there is quite a bit going on, let me explain all the components:

  • dir(imported_module) will get us a list of attribute names (as string values) in the imported_module;
  • Then we get the attribute as an object with getattr(imported_module, i);
  • We check if the attribute we get back is a class and if it’s a subclass of our base class;
  • The last line, setattr() function takes an object, the name and the value, and it assigns the object an attribute with the name and value you define. Here, set the import class as the name of the imported module, and set the attribute on the current package

 

So when we bring it all together, this is how our animals/fish/__init__.py  would look like:

from ..base_animal import AnimalBaseClass
from pathlib import Path
import sys
import inspect
import pkgutil
from importlib import import_module

from .goldfish import GoldFish as goldfish

for (_, name, _) in pkgutil.iter_modules([Path(__file__).parent]):

    imported_module = import_module('.' + name, package=__name__)

    for i in dir(imported_module):
        attribute = getattr(imported_module, i)

        if inspect.isclass(attribute) and issubclass(attribute, AnimalBaseClass):
            setattr(sys.modules[__name__], name, attribute)

if we are importing the goldfish module. The above would be the same as

from .goldfish import GoldFish as goldfish

The advantage here is: this code will be in charge of importing any subclasses of AnimalBaseClass we may add in the future without having to add the import line every time!

And after this, you can forget about the __init__.py all together and still get the nice syntax I mentioned in the beginning!

 

Key Takeaways - More on Dynamic Imports

  • import_module() from importlib is very important if you want to import any kind of modules dynamically, simply by passing a string value.
  • After you have gotten the imported module as an object, you can use getattr() function, passing in the module object, and a string value of a class/variable/function you want.
  •  

    Now, some extra.

    If you write a dynamic import function to import modules and classes by passing string values, here is an example:

    def dynamic_import(abs_module_path, class_name):
        module_object = import_module(abs_module_path)
    
        target_class = getattr(module_object, class_name)
    
        return target_class
    

    Tags