The asyncio Module

In this section, we are going to study what asynchronous programming is as well as how the asyncio module in Python allows you to do asynchronous programming. We’ll also cover the meaning of the “async” and “await” keywords.

Event Loops

The first critical concept you must understand in order to fully understand asynchronous programming is the concept of the event loop.

Most programs have some sort of event loop within them. Event loops make it possible to write code that responds to events. For instance, in a video game, if you press a key it might make your character move forward. The program needs to wait for the key press, and then react to the key press.

A typical event loop looks something like this:

while True:
    for event in events_that_are_happening()
          event.take_action()

The exact nature of these commands depends entirely on how you can check for active events, how you respond to them, and how you limit the number of events you want to look for.

Typically, you need to do something like the following:

1. Register events you are interested in.

2. Identify the code you want to run when the event occurs.

3. Figure out a way to efficiently check for events.

4. Run the code when the event occurs.

Now, I can’t possibly address all of these issue right now. We will, however, visit these topics as we look into some specific details of how asyncio works.

Callbacks

Typically, when you register an event, you also register the callable that you want to be called when the event occurs. This is what I mean be a “callback”: A callable that is meant to be called when an event occurs.

The arguments to the callback typically involve some that are part of the event, and some that are part of the context of when the event was registered.

Although callbacks are a simple concept, in practice, it makes messy code. The reason for this is it is difficult for the reader of the code to understand when certain callables are called because there is no direct call of those callables in your code! The calls actually occur within the code you are using to manage your event loop.

For this reason, I typically name my callbacks with the _callback suffix. When you see that, don’t expect anyone to call your code. It is also common practice to name the callbacks for when they are called, not based on what they do. For instance, on_keydown_callback is a callback that is called when a key on the key board is pushed down. What does it do? I don’t know, but I know when it is called.

Because callbacks typically don’t have any information in the name about what they actually do, we write other callables that do things, appropriately named, and use the callbacks to dispatch code flow to those callables. For instance, when the w key is pressed, the character will move forward, so the callback will call move_character_forward rather than moving the character itself.

This combination makes testing a really simple matter. The callbacks tend to grow as more behaviors are added to the program, so moving the behavior in other functions keeps the callbacks simple and allows testing each behavior independently.

There really isn’t much more to say about callbacks, except that you typically wrap them in lambda statements. You may also want to look at functional programming methods like partial and such.

Coroutines

Coroutines, in my mind, fall into the same class of functions as callbacks. Before I continue, I want to impress upon your mind the true nature of coroutines.

In order to define a coroutine, simple prepend async to the def line of the function definition:

async def foo(): ...

Just like generators, when you call a coroutine, nothing happens. Instead of running the code in the coroutine, you get a coroutine object back.

What can you do with that coroutine? If you read the Python documentation, they give you three things you can do with a coroutine:

1. Add it to the event loop with asyncio.create_task or event_loop.create_task.

2. Run it directly with asyncio.run.

3. Use it in an await expression.

The third option – the await expression – is a little weird. Note that you can only use await expressions inside of a coroutine.

What the await expression does is schedule the other coroutine to be run in the same loop as the coroutine is currently running. Then it waits for the coroutine to fully complete, and it returns the result.

It sounds simple, but it’s actually quite weird.

In order to understand what is really going on, you must first understand that inside of the python program itself there is an event loop. The python program that runs your program is actually listening for events and has a sort of event loop. When those events are triggered, it interrupts whatever code is running, handles the event, and then continues onward.

How does it do this? It’s simple, really. Python keeps track of where it is as it executes the code. Not just what line of code it is working on, but all the code that called this code – the call stack. It also remembers the context, the values of the variables in the global and local scope and such. When you interrupt the python program, it just simply sets that all aside, handles the interrupt, and then restores it all as if nothing happened.

If python can do this for regular code just to handle events, why can’t we have it do it for everything? The answer is, indeed, you can, and you should look into something called “greenlets” for how you can modify python to make everything a coroutine.

The key fact of a coroutine is that the “magic” that allows you to run your code, oblivious to what is actually going on behind the scenes is revealed. You are no longer in the audience observing the magician plying the trade, you get to set foot back behind the curtain, where all of his helpers are moving things around to make the next trick appear to work flawlessly.

See, when you write a coroutine, you give python everything it needs to run a function. When you create the coroutine, however, what python does is say, “OK, I’m going to create a function call, but I’m not going to execute it. You have to tell me when to run it.” What you can then do is put that coroutine in an event loop, and when certain events occur (or even, don’t occur) it will run that code. It will keep running that code until the code is done, and then the event loop will process the next event.

Well, it will run the coroutine until it hits an await expression, at which point, it will freeze the running function, thaw out the awaited coroutine, and run that instead. And when that coroutine is done it will then continue the original coroutine.

Let’s look at a concrete example:

import asyncio

async def a():

print(“Starting a”) print(await b()) print(“Ending a”) return 9

async def b():

print(“Starting b”) print(“Ending b”) return 7

print(asyncio.run(a()))

If you run it you should see:

Starting a
Starting b
Ending b
7
Ending a
9

Let’s walk through what happens.

• When you define a and b nothing happens. Just like regular function definitions, Python simply remembers the code and assigns it to a and b.

• When you call a(), a coroutine object is created. a is not yet run!

• When you call asyncio.run(), it will actually start running a.

• Now that a() has actually started running, it will print “Starting a”.

• Next, we create a coroutine for b.

• And then we await for that coroutine to run. So a is now suspended. b is run.

• Nothing weird is happening in b. It just prints “Starting b” and then “Ending b”. Finally, it returns 7.

• Now that b is done, a continues where it left of. Since b returned 7, the result of await b() is 7. This is printed.

• Finally, it prints “Ending a” and returns 9. Execution complete.

• asyncio.run() sees that a is done, and so it checks its result – 9. That is printed.

With that, you know basically everything there is to know about coroutines.