Modules, Packages, and Libraries

Python allows you to organize code into modules and packages. Using this system, we can share and use each other’s code.

In this video, I’m going to cover virtual environments and the import statement. I’ll also give some very basic directions on how to create your own packages.

Virtualenv

If you plan on importing anything, let alone installing anything, you should create a virtualenv.

A virtualenv (venv for short) will isolate your packages and modules from the system packages and modules. What someone else installs in their virtualenv will not affect yours. And you can have different packages than the system packages.

Creating a virtualenv is ridiculously easy.

$ python -m venv <path>

<path> is the path you want to create your virtualenv inside. Typically, I name it venv and I put it at the top-level of whatever project I am working on. For instance:

$ cd Projects/calculator
$ python -m venv venv

Note that you should be invoking venv with your Python 3.7 installation.

Once you have created your virtualenv, it’s time to activate it.

$ . venv/bin/activate # Linux/MacOS
$ venv/Scripts/activate # Windows

Once you’ve activated your virtualenv, all the packages you install with pip will be installed in your virtualenv and be immediately accessible. See https://PyPI.org/ for a list of publicly available packages. You may also find packages you want to install at your favorite project’s website.

import statement

The simplest form of the import statement is:

import <identifier>

This will import the module named <identifier> and assign it to a variable of the same name.

As an example:

import sys

The exact steps Python follows are:

  • Check sys.modules to see if the module has already been loaded. If so, just assign the module to a variable of the same name in your namespace.

  • If the module has not been loaded, search for a file named <identifier>.py or <identifier>/__init__.py in the virtualenv. (Note: This is not the whole story. It’s quite a bit more complicated than this.)

  • Once the file is found, run it in its own namespace. (The module’s global namespace is not the same as your global namespace. You can access its global namespace through the module attributes.)

  • Registers the module in sys.modules and assign the module to a variable of the same name in your namespace.

Note

You can import at any time. People typically like to import before running code, but this is not necessary.

If you execute the import statement in a function call, then a local variable will be used for the module name, rather than a global.

Accessing the Contents of a Module

Once you’ve imported a module, you can access objects within its namespace via the attribute access . syntax:

import sys
sys.modules

You should read the documentation on the module if you want to know what is in it.

If you don’t have the documentation available, or you just want a quick reminder, then use help() or dir()

>>> import sys
>>> help(sys) # Documentation of all the objects
>>> dir(sys)  # List of all the objects

ImportError

If anything goes wrong during the import process, an ImportError is raised. You can catch this error and handle it:

try:
    import foo
except ImportError:
    print("Error!")

Typically, ImportError isn’t caught and handled. The only case where I have done this is when I have code that needs to run in Python 2 and 3, and the module is named differently.

ImportErrors could arise for any one of these problems:

  • There is no module with that name.

  • The module has a SyntaxError that prevents Python from compiling and running it.

  • While running the module, an exception was raised.

There isn’t much you can do except resolve the underlying problems. Maybe you misspelled the module name, or you forgot to install it with pip, or you wrote the module yourself and created a SyntaxError or some other bug in it.

Importing multiple modules

You can import multiple modules at the same time:

import os, sys

This is the same as importing each of the modules one at a time.

import os
import sys

Importing Modules in Packages

Some modules are organized in a package.

Note

  • Modules are files in Python.

  • Packages are paths to the files in Python.

For instance, foo/bar/baz.py would be the module foo.bar.baz and the package foo.bar.

This is an important distinction in Python, as we’ll see when we write our own packages and modules.

import <package>.<module>

This will assign the top-level package to a variable of the same name. You can access the module with the . syntax.

You may have packages nested in packages nested in packages. Use as many . as needed.

There aren’t many nested packages in the Python library. Here’s an example, though:

import os.path

os.path.join(...)

Importing with a different name

Sometimes you want to import a module, but store it under a different name.

import sys as s
s.modules

This will import sys but assign it to s.

This works for package imports as well:

import os.path as p
p.join

The benefit of this is the module, not the package, is assigned to the identifier.

Importing specific objects

You can import individual objects from a module:

from <module> import <id>, <id>, <id>

If you have a lot of things, using parentheses is OK:

from <module> import (
  <id>,
  <id>,
  <id>,
)

Note that you can also import from modules in a package this way.

This will import the module, and then assign the objects to variables of the same name.

If you want a different name, use as:

from <module> import (
  x as a,
  y as b,
  z as c,
)

Module Special Attributes

Modules have a few special attributes.

  • __name__: The name of the module.

  • __package__: The name of the package of the module.

  • __file__: The location of the file the module came from.

  • __doc__: The doc string.

Running a module as a script

You can invoke any module as a script from the commandline:

$ python -m <module name> <script parameters>

We did this with the venv module when creating a virtual environment:

$ python -m venv venv

if __name__ == '__main__':

Sometimes you want to write a module that also functions as a script.

When a python file is loaded as a script, Python treats it as a module named __main__. You can check the module’s __name__ from within the module itself.

This means you can hide the code that should only be run as a script behind this if statement:

... module code ...

if __name__ == '__main__':
    ... script code ...

There is a better way to do this using setuptools that I’ll cover in a later video.

Creating your own Packages and Modules

Creating your own Packages and Modules is a fairly involved process that involves writing a setup.py file. There isn’t a lot of consensus in the Python community how to do this properly, so there is a lot of confusion around it.

I recommend something simple and robust.

mycode/ # Project directory

  setup.py

  venv/ # For your virtual environment

  mycode/  # If a package

    __init__.py  # Empty

    mycode.py

    ...

  mycode.py # If a single module

To structure your code, first create a directory. You’ll likely want to manage this directory with git or a similar tool. (I’ll create a video on what you really need to know about git later.)

You need a package or module name. I’m using mycode as a stand-in for that.

The package would live under a directory called mycode. In that directory, you need a __init__.py file, which I recommend leaving blank. If you have any subdirectories, they each need their own __init__.py, which I recommend leaving blank. Any modules you want you just drop in the appropriate directory with the appropriate name. Let’s say you want a mycode.foo.bar module. In that case, create a directory called mycode/foo and drop an __init__.py there, then add in mycode/foo/bar.py for the module file.

If you just want a single module with no packages, then just create mycode.py in your project directory.

Next, you need a setup.py file. People try to get really fancy with their setup.py files, but I strongly discourage this.

Here’s bare minimum of what I would put in my setup.py file:

from setuptools import setup, find_packages

setup(
    name='mycode',
    version='0.0.1',

    python_requires='>=3.7',

    # If a package, then...
    packages=find_packages(include=['mycode', 'mycode.*']),

    # If a module, then...
    py_modules=['mycode'],
)

If you are serious about creating packages and modules, then I highly recommend reading and understanding the content at https://packaging.python.org/. I really can’t do a better job than what they have done.

That said, every project is different, and you’re likely going to run into cases where you’re at a loss for how to properly package it. I can help. Find me on discord.

Installing your Own Code for Testing

Once you get your setup.py written, then install your code with pip install -e <project directory>. That’s all you need. With the venv activated, you can import your package or module however you see fit.

The -e option tells pip to not copy the files over, but to link them. This means that when you change one of the files, those changes will appear when someone imports your modules and packages again.

Note that if you add directories, you’ll have to pip install again, otherwise they won’t be linked in to the venv.

Distributing your Code

Before you go launching your code to PyPI, talk to me first, please. I can help you get your package production ready so that you won’t make some rookie mistakes when deploying packages.

Future Videos

  • Going into setuptools.setup in depth

  • Conflict resolution

  • git