Recently, I’ve been experimenting somewhat with Celery, Django and Fabric.

These fit really nicely together if you want stick a web interface on system management tasks, such as deploying some code.

There’s a couple of gotchas with the way Celery uses fork to run a task though, as it removes the environment that the Celery task is running in.

1. RNG must be re-initialized after fork(). Hint: Try Random.atfork()

If you do this:

@task
def update():
    run('ls')

It’ll fail with the above error. The fix is to do what the error says:

from Crypto import Random

@task
def update():
    Random.atfork()
    run('ls')

This will initialise the Random Number Generator sufficiently for the paramiko ssh library that fabric is based on to connect.

1. raw_input(“No hosts found. Please specify (single) host string for connection: ”)

You’re using fabric as an api, rather than using the fabric runner. This means the usual technique of doing:

from fabric.api import *

 env.hosts = ['192.168.0.1']

Won’t work, as there is no processing done on the env. The fix looks like this:

from fabric.api import *

@task
def update():
    Random.atfork()
    env.host_string = 'ubuntu@192.168.0.1'
    run('ls')

If you need to connect to multiple hosts, you’ll have to do your own flow control.

1. disable_known_hosts

This is more of a tip, and not a brilliant one at that. If you’re doing a lot of work with pregenerated VMs (such as the cloud providers), then destroying a VM on an IP, then rebuilding it, will cause the host key to change. Fabric uses your local .ssh/known_hosts for it’s connections, so this will probably cause the connection to fail as the keys will not match your previous one.

from fabric.api import *

@task
def update():
    Random.atfork()
    env.host_string = 'ubuntu@192.168.0.1'
    env.disable_known_hosts = True
    run('ls')

disable_known_hosts will disable this behaviour, at the cost of some security, as you are no longer verifying that you are connecting to exactly the same machine. The current alternative seems to be that every time you rebuild a VM, you must remove it’s line from your known_hosts.

Each and every Celery task you use with Fabric, must set these variables, or it’ll fail. I’d recommend either inheriting your tasks, or creating a helper method to pop them all in, and call that at the start of your task.

Hope this helps someone.