Coroutines In Python

A coroutine is a generalization of the subroutine. Forget for a moment everything you've been taught about calling functions, and stacks, etc... Think back to BASIC, and the evil GOTO statement. Imagine a more powerful GOTO that could jump back and forth between functions, passing arguments, and you begin to get the idea

Coroutines are a bit of ancient forgotten computer-science lore; stamped out of the collective memory by the hegemony of C. But they are useful in a wide variety of situations that can only be clumsily solved using 'standard' tools like threads and processes.

Coroutines can be used to simplify just about any difficult state-machine programming problem. Any time you find yourself tempted to build a complex state machine to solve what should be a simple problem, you've actually been pining for coroutines. With them, you can usually turn an 'inside-out' problem into a 'right-side-out' problem, and replace a few pages of hairy code with a single function.

A popular application of coroutines is in the construction of lightweight threading libraries (in fact, on Win32 coroutines are called 'fibers'). Coroutine threads are more scalable than OS threads - a desktop machine can easily juggle tens of thousands of them without gobbling up the entire virtual memory space.

Efficient coroutines hold out the possibility of building powerful, stateful internet servers that are massively scalable.

python-coro eGroup

Stackless Python

• Christian Tismer's "stackless" Python
  Coroutines and Continuations done the Right Way.
• Python-Dev Mailing List
  Pythoneers discuss a Mad Scheme

Coroutines in a Virgin Python

• coro-1.0.3.tar.gz
  Edgar Toernig's coroutine library in C. [http://user.exit.de/froese/]
• coroutinemodule.c
  A Python Module interfacing to it.
• eGroups coroutine-based I/O system
  This is an open source (see LICENSE) library developed by eGroups engineers for internal use.
  Includes a web server, a web crawler, a DNS resolver, remote interpreter, and other goodies.
• A demo system for browsing digicam jpeg's built around a stateful http server.

Combining I/O Event-Delivery with Coroutines

If you can convince your OS to deliver events, you can avoid the overhead of select() and poll() entirely...

• Linux

  In the latest versions of Linux (2.3.x) there are kernel tweaks that allow posix realtime signals to be sent to indicate I/O events. This is an improvement over SIGIO because realtime signals are queued and can be fetched in batches, so the whole mechanism should scale much better.

  Anyway, here are the beginnings of my attempt to build an I/O scheduler around linux's realtime signals.

  • coro_rtsig_module.c
  • test_coro_rtsig.py

  Soon I hope to plug this into our existing coroutine scheduler infrastructure.

• FreeBSD

  FreeBSD-5.0 (-CURRENT) and possible 4.1-RELEASE include a new feature for kernel event delivery called kqueue() . Doug White is working on a Python interface, which I have hacked and plugged into egp-coro.

  Note: The kqueue() interface just changed, so this code will require some rearrangement...

• Win32

  Win32 includes a robust event-delivery API, even supporting async file I/O. If you're interested in working on this, please contact me so we can coordinate effort and hopefully produce a portable high-level framework.