Stefan Seelmann

Python/Django Continuous Deployment

This page describes a setup for (continuous) deployment of a Python/Django application.

Preface

An existing Python/Django application with the following structure:

myproject
|-- dependencies.pip
|-- static
|-- myproject
|   |-- manage.py
|   `-- ...
`-- venv
    |-- bin
    |-- include
    `-- lib

One-time setup (server-side)

Python and virtualenv

$ apt-get install python python-pip python-virtualenv

Gitolite

See Gitolite.

The following two scripts are used to deploy and restart the python application after a git push.

The script /srv/www/bin/deploy-python-app.sh updates the application and its dependencies: 

#!/bin/bash
set -e
APPNAME=$1
APPDIR=/srv/www/$APPNAME
cd $APPDIR
source venv/bin/activate
git reset --hard || true
git pull
pip install -r dependencies.pip

The script /srv/www/bin/restart-python-app.sh restarts the python application:

#!/bin/bash
set -e
APPNAME=$1
/usr/bin/supervisorctl restart $APPNAME

In order to run the scripts the git user need sudoers rights. Add the follwing via visudo:

git ALL=(root) NOPASSWD: /srv/www/bin/restart-python-app.sh, (www-data) NOPASSWD: /srv/www/bin/deploy-python-app.sh

Supervisor

$ apt-get install supervisor

The script /srv/www/bin/run-gunicorn-django.sh is used by supervisor to start and watch the Django application.

#!/bin/bash
set -e
APPNAME=$1
PORT=$2
APPDIR=/srv/www/$APPNAME
LOGDIR=/var/log/gunicorn
LOGFILE=$LOGDIR/$APPNAME.log
USER=www-data
GROUP=www-data
NUM_WORKERS=3
cd $APPDIR
source $APPDIR/venv/bin/activate
test -d $LOGDIR || mkdir -p $LOGDIR
exec gunicorn_django $APPNAME \
  -w $NUM_WORKERS -b 127.0.0.1:$PORT \
  --user=$USER --group=$GROUP --log-level=debug \
  --log-file=$LOGFILE 2>>$LOGFILE
exit 0

Per-project setup (server-side)

Create gitolite project

See also Gitolite.

Add to gitolite-admin/conf/gitolite.conf:

    repo    myproject
            RW+     =   user1, user2, ...
            R       =   www-data

Make sure the public key of www-data is in gitolite-admin/keydir/.

Create post-receive script /srv/gitolite/repositories/myproject.git/hooks/post-receive:

#!/bin/sh
sudo -u www-data /srv/www/bin/deploy-python-app.sh myproject
sudo -u root /srv/www/bin/restart-python-app.sh myproject

And set execute bit:

chown git:git /srv/gitolite/repositories/myproject.git/hooks/post-receive
chmod 700 /srv/gitolite/repositories/myproject.git/hooks/post-receive

Setup web project

$ sudo su - www-data
$ cd /srv/www
$ git clone git@localhost:myproject
$ cd myproject
$ virtualenv --no-site-packages venv

Setup supervisor

Create /etc/supervisor/conf.d/myproject.conf, :

[program:myproject]
directory = /srv/www/myproject
user = root
command = /srv/www/bin/run-gunicorn-django.sh myproject 8000
stdout_logfile = /var/log/supervisor/myproject.log
stderr_logfile = /var/log/supervisor/myproject.log

Reload configuration:

$ supervisorctl reload

Setup nginx

Create a config file /etc/nginx/sites-available/www-example-com and create a symlink to /etc/nginx/sites-enabled:

server {
  server_name www.example.com;
  server_name_in_redirect off;
  access_log   /var/log/nginx/www-example-com.access.log;
  error_log   /var/log/nginx/www-example-com.error.log;

  location /static/ {
      alias  /srv/www/www-example-com/static/;
  }

  location / {
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Scheme $scheme;
    proxy_connect_timeout 10;
    proxy_read_timeout 10;
    proxy_pass http://127.0.0.1:8000/;
  }
}

TODO: static

Push project:

$ git remote add prod git@server.example.com:myproject
$ git push prod master

Per-project setup (developer-side)

Install python and virtualenv

$ apt-get install python python-pip python-virtualenv

Setup a new project and virtualenv

Setup virtualenv:

$ mkdir myproject && cd myproject
$ virtualenv venv
$ source venv/bin/activate

Install dependencies:

(venv)$ pip install django
(venv)$ pip install gunicorn
(venv)$ pip freeze > dependencies.pip

Setup Django project:

(venv)$ django_admin.py startproject myproject myproject

Test Django server:

(venv)$ python myproject/manage.py runserver

Test Gunicorn server:

(venv)$ gunicorn_django myproject

From another console:

$ curl -i localhost:8000
HTTP/1.1 200 OK
Server: gunicorn/0.14.2
...

Setup git repo and push to github

Create a .gitignore file:

venv
*.pyc

Create git repo, commit and push

$ git init
$ git add .
$ git commit -m "Initial commit"
$ git remote add origin git@github.com:...
$ git push origin master

Developer workflow

Clone the project and setup virtualenv

$ git clone git@github.com:...
$ cd myproject

Setup virtualenv:

$ virtualenv venv
$ source venv/bin/activate

Install dependencies:

(venv)$ pip install -r dependencies.pip
(venv)$ python manage.py runserver

Update dependencies

When new dependencies were added:

$ pip freeze > dependencies.pip

Deployment

$ git remote add prod git@server.example.com:myproject
$ git push prod master

Diagnostics

Test HTTP request to gunicorn:

$ curl -i locahost:8000
HTTP/1.1 200 OK
Server: gunicorn/0.14.2
...

Test HTTP request to nginx:

$ curl -i locahost:80
HTTP/1.1 200 OK
Server: nginx
...

Gunicorn should only listen on local interface:

$ netstat -tan | grep 8000
tcp  0  0  127.0.0.1:8000  0.0.0.0:*  LISTEN

Only the gunicorn parent process runs as root, workers run as www-data:

$ ps auxwww | grep gunicorn
root     ...
www-data ...
www-data ...
www-data ...

TODOs:

Sources: