Gianluca Pacchiella (Posts about sysadmin)

Let's encrypt

Gianluca Pacchiella — Sat, 16 Jan 2016 00:00:00 GMT

Let's encrypt is the new thing in town: allows a seamless procedure for obtaining TLS certificates; and it's free ;)

Roughly speaking, it's a certification authority, capable of generating certificates accepted from any major browser; it has appositely created an (open source) client to do that without human intervention.

The protocol used by the client is ACME (stands for Automatic Certificate Management Environment);

First of all, install the client (in the future will exist a maintained package) in the server (all the operations must be done as root, I know, sucks)

# git clone https://github.com/letsencrypt/letsencrypt && cd letsencrypt
# ./letsencrypt-auto
[... installing packages...]
Creating virtual environment...
Updating letsencrypt and virtual environment dependencies.......
Running with virtualenv: /root/.local/share/letsencrypt/bin/letsencrypt
No installers seem to be present and working on your system; fix that or try running letsencrypt with the "certonly" command

This creates in the $HOME/.local/share/letsencrypt a virtualenv with the client, letsencrypt-auto should be a wrapper to the main executable named letsencrypt, that checks everytime if updates are available. If you want to use letsencrypt directly you have to activate the virtualenv.

There are several different ways to obtain a certificate and to deploy it, I choose a manual method, since I usually I have nginx that is not officially supported. If you have apache all should be completely automated. Exist also other methods, if you want to improve your knowledge, read the documentation.

From this post I stole the configuration for nginx (to place in /etc/nginx/snippets/letsencryptauth.conf)

location /.well-known/acme-challenge {
    alias /etc/letsencrypt/webrootauth/.well-known/acme-challenge;
    location ~ /.well-known/acme-challenge/(.*) {
        add_header Content-Type application/jose+json;
    }
}

then in the server block serving the domain for which you want to issue the certificate you can include this snippet

server {

        # the include must be placed before any location directive
        include snippets/letsencryptauth.conf;

        # other location directives
}

Finally we have to create the authentication directory

# mkdir /etc/letsencrypt/webrootauth

and execute the last step

# ./letsencrypt-auto  \
    --webroot-path /etc/letsencrypt/webrootauth \
    --domain yourdomain.com  \
    -a webroot certonly
[... wait a little bit ...]
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem. Your cert will expire
   on 2016-03-03. To obtain a new version of the certificate in the
   future, simply run Let's Encrypt again.
 - If like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Remember that the certificate generated will expire after just three months.

Security

All is working but someone has (rightly) rised some concerns about security since all the scripts are autoupdating and running as root. An alternative way is to install and run it using docker with the following steps: first of all, pull the image

$ docker pull quay.io/letsencrypt/letsencrypt:latest

and then run it, mounting the path used to store configuration and certificates by letsencrypt

$ docker run \
    -v "/etc/letsencrypt:/etc/letsencrypt" \
    -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
    --entrypoint=/bin/bash \
    -it quay.io/letsencrypt/letsencrypt
root@d24cd7b4b487:/opt/letsencrypt#

I warn you that docker works only on 64bit machines.

Restore backup and move mail server

Gianluca Pacchiella — Sun, 10 Jan 2016 00:00:00 GMT

We all know that backup are essential, but your backup procedure is tested? I mean, you have ever tried to restore a service?

In this post I try to explain the procedure that I used to restore and move my mail service to another server: it's a pretty basic setup, consisting of postfix as service SMTP, dovecot as IMAPS service and having as pool directory the path /var/mail/.

The backup is done using rsnapshot via my easy-backup package: from the snapshot I created a tar archive with the configuration files for the services of interest in the /etc directory and the mail pool.

Restore

First of all install the necessary packages: to obtain the list of packages installed with the actual version on the src server you can use

$ /usr/bin/aptitude -q -F "%?p=%?V %M" --disable-columns search \~i

(this is already generated if you install my package ;)); now, depending how different is the destination system, you have to check if the versions that it finds make sense or a major version change happened (like dovecot in my case) and in such case google for problems.

In any case, if something goes wrong you can apt-get remove --purge <packages> and restart from beginning.

After that you can copy the backuped data into the new machine: from the machine where you have the backup, create an archive containing all the needed

$ tar \
    -c \
    -C <root path of the backup> \
    etc/dovecot etc/postfix etc/aliases etc/aliases.db var/mail /home/gipi/mail/ \
    > archive-`date --iso`.tar

As double check, look at the configuration files and try to find some reference to files in /etc that can be needed (for example, in my case, some certificates). Also, remember that is possible that the two systems can have the uid and gid of the corresponding users not equal causing permission issues ( I would like to extend easy-backup to handle these cases).

Finally, compress the archive and unarchive to the final server

cat archive-2016-01-01.tar | gzip -9 | ssh dest tar -C / -xzv
etc/postfix/
etc/postfix/postfix-script
etc/postfix/main.cf
etc/postfix/sasl/
etc/postfix/master.cf
etc/postfix/virtual
etc/postfix/post-install
etc/postfix/postfix-files
etc/postfix/dynamicmaps.cf
etc/postfix/virtual.db
etc/dovecot/
etc/dovecot/dovecot-sql.conf
etc/dovecot/dovecot.conf
etc/dovecot/dovecot-ldap.conf
etc/dovecot/dovecot-db-example.conf
etc/dovecot/dovecot.conf.bak
etc/dovecot/dovecot-dict-sql-example.conf
etc/aliases
etc/aliases.db
var/mail/
var/mail/postgres
var/mail/gipi

This step can be time expensive (in my case the archive was like 80MB).

After all, restart the services and tail -f /var/log/syslog to watch any problem that can arise.

Test

After all the procedure we can test if the new installation is working correctly, but since this want to be a test, without interrupting the normal mail server, we can use SWAKS and its option --server to direct the connections to the new server, otherwise it looks for the MX DNS's entry of the recipient (i.e. the email address indicated in the to field); in the following example I used as the domain yourdomain.com

$ swaks \
    --server mail.yourdomain.com \
    --to user@yourdomain.com \
    --from test@example.com

Meanwhile you can look at the syslog on the server: in my case the first time I've done this I forgot to add /etc/aliases.db into the backup and this below is what the server told me

Jan  3 12:30:13 miao postfix/smtpd[7337]: error: open database /etc/aliases.db: No such file or directory

Obviously, we care to have the TLS available, so we can test that also with autentication

$ swaks \
    --server mail.yourdomain.com \
    -tls --tls-verify --tls-protocol tlsv1_2 \
    --auth plain \
    --from user@yourdomain.com \
    --to uptoyou@example.com

At this point you can also use some online tool to check you mail server, like starttls. It's also possible to check for blacklist.

Conclusion

If all it's ok, you are ready to switch your mail server: my procedure was to add a MX record with lower precedence to the one pre-existing, but lowering the time-to-live of both the entries, so to have less time to wait in order to adjust the values. Once the new entry was available I swapped the precedence so to have the new entry to be used and not the old one.

At this point I tried the normal access with my email client so to assure the IMAP worked and all the folders was there.

Finally, activated the backup for mail on the new server.

I advise you to try this, or in general, backup procedures, as probably you are not aware exactly of what you need to restore a system: myself I missed for years the backup of the mail folders in the home of my user.

Manage processes in a web application

Gianluca Pacchiella — Wed, 30 Dec 2015 00:00:00 GMT

Suppose we have the following architecture for our web application: a standard python web application (can be anything, Django, Flask or a custom WSGI code) where nginx communicates with it using an unix socket by the uwsgi protocol as indicated in the diagram below

                      socket
browser <---> nginx <----------> uwsgi <---> django/flask/kebab
                                |           |__ database
                                |__ celery

This is a pretty standard way to deploy python web applications, but I haven't found any complete description on how manage this configuration.

I want

Security
each web application must be isolated from any other service on the server
obviously the code must run with the minimal permissions possible
the developer don't have to be a system administrator, must have the same permission of the running web application
Control
a single point for controlling the status of the web application
the web application can have more than one process, not directly web related (celery anyone)
the developer can add processes
when the processes die must be restarted in correct order
updating must not cause sudden kill of processes (a long running celery task should finish)
restarting on updating must be explicit
the service must restart automagically at reboot

Bad enough, not all of these points will be addressed in the first revision of this post.

Security

First of all, the channel of communication between nginx and uwsgi is an unix socket (i.e. a particular type of file), this allows a more granular access control and is more customizable (the alternative is to use an internet socket that can be only be customized as port number, less human readable and not access controllable).

In order to be functional (i.e. the user requesting the page would actually see the page) the socket should be readable/writable by the nginx process and its own uwsgi process; since I'm interested in securing the socket from unwanted interaction, I want it to be not readable from other possible uwsgi instances related to other projects (obviously a vulnerability in nginx could allow an attacker to access all of them but this is unavoidable and I think less probable). Obviously I want to avoid access from random processes as well.

This means that we have to create a separate user for each project and that the socket must be chowned to www-data:user with permissions srw-rw---- (i.e. 660).

The order of the owner seems weird but take into account that chmod(2) can change permission only by who owns the file, or quoting the documentation:

The effective UID of the calling process must match the owner of the file, or the process must be privileged

(i.e. the attacker could change the permissions' bits if owns the file) so, at the end of the day, to avoid escalation we need to set the owner of the files of the web application to not be the user of under which is running the web application.

Since the socket is created by the uwsgi instance and you cannot chown to a group you don't belong, uwsgi must be started as superuser and then it must downgrade its own capabilities (this is a standard behaviour for daemon); in order to do so exist some flags

--uid: the user id of the running process
--gid: the group id of the running process
--chmod-socket: the permissions on the socket, 660 should be just fine (here we must use = just after to avoid strange errors)
--chown-socket: who owns the socket, like www-data:user

I know that I missing a lot of stuffs in securing a web application, like filesystem permission for the web root, but that will be argument of a future post. Indeed if I want to make a configuration I need to determine what secure means by the definition of a threat model for the web application, i.e. what your adversary is capable of and from what you want to defend (for example: your developers are a possible threat? should be them allowed to change configuration files?).

In the following I assume that the developer can edit the uwsgi.ini configuration file and he/she can add processes.

Control

In my specific case, usually I deploy untarring the archive with the code in a new directory, executing some operations (like database migrations, backup, etc...); the tricky thing is to substitute the running processes in the way described in the list above.

The entry point for the request at the web application is uwsgi and the idea that I want to follow is that uwsgi controls the part after it, in this way if some process is added we don't have to touch with sysadmin power the installation, i.e. we want the developer to be able to add processes using the uwsgi.ini configuration file.

Now I need a process supervisor for uwsgi, that starts it and manages its lifecycle. There are some choices available for process managers, like supervisord, upstart and many others (read this post about process managers for more informations).

Could be possible to use stuffs like Procfile (inspired from heroku) to manage multiple processes, or another option could be using container-like technology (Docker or lxc) but in my opinion are a little bit immature technologies.

The choice is to use supervisord, it will start the uwsgi process; since supervisor need you to be a superuser in order to interat with, I will configure sudo in order to allow execution of commands like

$ sudo /usr/bin/supervisorctl restart uwsgi_example

By the way this kind of configuration with sudo can be used with whatever process manager you want.

The tricky step is to configure correctly uwsgi to manage external processes, generally the right way to do that is to avoid daemonizing them and undertstand what signals kill/restart them; for example celery uses the TERM signal as stated here).

Configuration

After all this, let see how to configure all the things: these are the parameters

webuser is the UNIX user under which the web application run
www-data is the identity of the web server
/var/www/ is the webroot
/var/www/.virtualenv/ is the virtualenv used

Supervisor

[program:uwsgi_example]
command=/var/www/.virtualenv/bin/uwsgi
    --ini /var/www/app/uwsgi.ini
    --processes 1
    --need-app
    --uid webuser
    --gid webuser
    --chown-socket www-data:webuser
    --chmod-socket=660
redirect_stderr=true
stdout_logfile=/var/www/logs/uwsgi.log
stderr_logfile=/var/www/logs/uwsgi_error.log
autostart=true
autorestart=true
stopsignal=QUIT

Note that if we use --ini as the first option for the supervisor's uwsgi option, then any value will be overwritten by the flags passed (the main concern here are the uid and gid flags); in this way the developer cannot override these values from the uwsgi.ini file.

Sudo

With the configuration below, the webuser can issue command to the uwsgi_example configuration of the supervisor daemon, without requiring a password

%webuser ALL = (root) NOPASSWD:/usr/bin/supervisorctl [a-z]\* uwsgi_example

UWSGI

Finally the uwsgi.ini file can contain the following entries

[uwsgi]
module=.wsgi:application
chdir=/var/www/
socket=/tmp/uwsgi_example.sock
pidfile=/tmp/project-master_example.pid
vacuum=True
max-requests=5000
harakiri=30
stats=/tmp/stats_example.sock
# https://uwsgi-docs.readthedocs.org/en/latest/AttachingDaemons.html#examples
# smart-attach-daemon = /tmp/celery_example.pid .virtualenv/bin/celery -A app.tasks worker --pidfile=/tmp/celery_example.pid --loglevel=debug --logfile logs/celery.log
attach-daemon2 = stopsignal=15,reloadsignal=15,cmd=.virtualenv/bin/celery -A app.tasks worker --pidfile=/tmp/celery_example.pid --loglevel=debug --logfile logs/celery.log --concurrency=5

In my model this file is versioned together with the code; in this specific example is enabled the control of a celery's worker. For more information there is a page in the celery's documentation.

Testing

If seems to you that all of this is tricky to build you are right, but you are lucky: I prepared a cookiecutter template for provisioning a web app project, with a script aimed to configure all the necessary in order to test what I described here: simply do

$ git clone https://github.com/gipi/cookiecutter-eep-provisioning.git
$ cd cookiecutter-eep-provisioning
$ ./tests/tests.sh

 now you are into the provisioning directory, you can use "sshme" to enter
 as the web application user.

 At the end remember to destroy the vagrant instance with "destroy_provision".

[/tmp/tmp.PjjmjCkjtC/provision] $

In order to use the test script you need vagrant and ansible. All it's tested for a linux system so be aware the in other OSes may be not working correctly.

For now it's all, let me know if all this seems reasonable to you.