Running a web server on FreeBSD inside a jail Jun 27 2020

Creating a jailed web server is a fun exercise to learn how about jails. If you are reading this post, you are probably already convinced of the benefits of running your services inside jails. A jailed service provides additional security by restricting the jailed environment to access only to its perceived root directory. We can run many services in the same host, and we could isolate them to their own jailed environment. Another useful trait of jails is dependency segregation. We can run different jails that depend on different versions of the same libraries or programs, without causing problems between them. Each jail will have its own userland. In this post, we are going to explore how to run a service inside a jail. We are going to use Nginx as an example, but you can take what you learned in the post and apply it to your specific case.

NOTE: If you are searching for a more complex setup where you use virtual networks on your jails you can check the Jails and VNET guide (the guide is free, just put $0.00 on the value).

Let's start with a few basics.

Creating a jail's userland

Jails don't run their own kernel. They defer to using the kernel provided by the host. What a jail need is a base system, also know as userland. The base system includes the applications and libraries that make FreeBSD more than just a kernel. If you want to check what's inside the base system, you can check the source¹. You'll see directories like /bin, /etc, /lib, /libexec, /sbin, and many more that define the base system.

We are going to get the base system using the bsdinstall(8) command. If you are using ZFS as your filesystem, you can create a dataset for all of your jails. Or a dataset for each independent jails, that is up to you. I'll be using UFS so I'll just create a directory under root called /jail and put my jail there.

1
# mkdir -p /jail/webserver/

If we want to use the base system for the same release of our host we could use the following command:

1
# bsdinstall jail /jail/webserver

This command will prompt you to select a mirror and the packages you want your jail to have. I don't need any additional packages so I'll just pick the primary mirror and deselect everything in the additional packages.

If you want to get the base system for a specific FreeBSD release, we need to set the following environment variables:

1
2
3
DISTRIBUTIONS
BSDINSTALL_DISTDIR
BSDINSTALL_DISTSITE

From the bsdinstall(8) man page, we get the meaning of each environment variable:

• DISTRIBUTIONS - The set of distributions to install (e.g. "base kernel ports"). Default: none
• BSDINSTALL_DISTDIR The directory in which the distribution files can be found (or to which they should be downloaded). Default: /usr/freebsd-dist
• BSDINSTALL_DISTSITE - URL from which the distribution files should be downloaded if they are not already present in the directory defined by BSDINSTALL_DISTDIR. This should be a full path to the files, including architecture and release names. Most targets (e.g. auto and jail) that prompt for a FreeBSD mirror will skip that step if this variable is already defined in the environment. Example: ftp://ftp.freebsd.org/pub/FreeBSD/releases/powerpc/powerpc64/9.1-RELEASE

For example, to set it using bash(1):

1
2
3
export DISTRIBUTIONS="base.txz"
export BSDINSTALL_DISTDIR="/jail/webserver/"
export BSDINSTALL_DISTSITE="ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE"

Or for tcsh(1):

1
2
3
setenv DISTRIBUTIONS "base.txz"
setenv BSDINSTALL_DISTDIR "/jail/webserver/"
setenv BSDINSTALL_DISTSITE "ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/12.0-RELEASE"

To explore the ftp.freebsd.org server on your browser, you can use the following URL:

1
http://ftp.freebsd.org/pub/FreeBSD/releases

After setting the environment variables, we can fetch the base system using the following command:

1
# bsdinstall distfetch

We can then verify that we have the files in the directory we defined in BSDINSTALL_DISTDIR.

1
# ls -l ${BSDINSTALL_DISTDIR}

And extract the base.txz if we want to:

1
2
# cd ${BSDINSTALL_DISTDIR}
# tar -xvpf base.txz

However you managed to get the base system, we are now ready to continue.

Once we have the base system ready, we can proceed to enable jails on our host and creating the jail.conf file.

Setting up the host and jail configuration

First, we need to enable jails on our host server:

1
# sysrc jail_enable="YES"

Setting up that variable will start our jails at boot time. If you don't want all your jails to run at boot, specify only the ones you want using the following variable:

1
# sysrc jail_list="webserver"

With that out of the way, we are now ready to configure our jail. All of our jail's configuration will be in the file /etc/jail.conf, and it'll include the configuration for any jails we create. Let's create it:

1
# touch /etc/jail.conf

The configuration file could have the following sections:

1. Definition of variables that we'll use through the config file
2. Default configuration for all jails.
3. Definition of specific jails and their configuration.

With that in mind, let's begin by defining some variables that could be useful in the future when we want to run multiple jails:

1
2
3
# 1. definition of variables that we'll use through the config file
$jail_path="/jail";
path="$jail_path/$name";

We can now easily reference the path to our jails throughout the configuration file. Let's set the default configuration that will apply to all the jails. Keep in mind that you can overwrite the default values when you define your specific jail.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 2. begin - default configuration for all jails 

# Some applications might need access to devfs
mount.devfs;

# Clear environment variables
exec.clean;

# Use the host's network stack for all jails
ip4=inherit;
ip6=inherit;

# Initialisation scripts
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";

As you can see, we are going to use the host's network stack in our jail. If you want to learn how to do more advanced networking for your jails using virtual networks, check the Jails and VNET guide.

We are using full jails, that means we'll use the base system initialisation scripts. If you were running a thin jail, one that only runs a specific process, you would have to provide your own initialisation script.

Ok, let's now define our jail, webserver, and add its specific configuration:

1
2
# specific jail configuration
webserver {}

All the defaults work perfectly for our jail, so we don't have to add any specific configuration. The following is the complete jail.conf file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 1. definition of variables that we'll use through the config file
$jail_path="/jail";
path="$jail_path/$name";

# 2. begin - default configuration for all jails 

# 3. Some applications might need access to devfs
mount.devfs;

# 4. Clear environment variables
exec.clean;

# 5. Use the host's network stack for all jails
ip4=inherit;
ip6=inherit;

# 6. Initialisation scripts
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";

# 7. specific jail configuration
webserver {}

With the configuration in place, we can now create and run the jail.

Running our base jail

With our configuration done, running our jails is as simple as:

1
2
# service jail start webserver
Starting jails: webserver.

If we list our jails with jls(8), we'll see that our jail is running (Your JID might be different).

1
2
3
4
# jls
   JID  IP Address      Hostname                      Path
     1                                                /jail/webserver

We can start a shell in our jail and play around using the jexec(8) command:

1
# jexec webserver /bin/sh

You can run any commands provided by the base system. When you finish working with your jail, you can stop it as easily as you started it:

1
# service jail stop webserver

Our jail is ready. We can now begin to install the specific packages we'll need for our web server.

Installing packages

We are going to manage all the packages form the host. We could manage the installation of packages from inside the jail, running a shell or using SSH to log in to the jail and use pkg(8). This is not necessary, pkg(8) is jail aware so we can just pass the jail name from the host.

This part will be very similar to installing a package in a non jailed system, aside from passing a different flag to pkg(8) and the installation location, not much changes. So let's get started.

Make sure our jail is running:

1
# jls 

If it's not running, start it up:

1
2
# service jail start webserver
Starting jails: webserver.

Check that your jail can access the internet:

1
2
3
4
(on host)# jexec webserver sh
(on jail)# host pkg.FreeBSD.org
pkg.FreeBSD.org is an alias for pkgmir.geo.FreeBSD.org.
pkgmir.geo.FreeBSD.org has address 96.47.72.71

If your jail can't access the internet, check its resolv.conf. You could also copy the resolv.conf from the host file:

1
(from host)# cp /etc/resolv.conf /jail/webserver/etc/

Once the jail is running and has access to the internet, we are ready to install the package:

1
(from host)# pkg -j webserver install nginx

Notice that we use the -j flag to tell pkg(8) to do the installation on our jail webserver. That should install everything we need, let's configure our jail to run Nginx and test it.

Nginx initial set up

We need to first, enable nginx_enable on the jail using sysrc(8). The sysrc(8) command, as the pkg(8) command, is also jail aware. That means that we just need to pass the -j flag with our jail name and the sysrc(8) will work on our jail.

1
# sysrc -j webserver nginx_enable="YES"

Once Nginx is enabled, we can start the service:

1
# jexec webserver service nginx start

Check the IP address of our host:

1
2
3
4
5
6
# ifconfig
em0: 
...
        inet 10.211.55.8 netmask 0xffffff00 broadcast 10.211.55.255
...
#

My host's IP is 10.211.55.8 we can visit it on our web browser:

1
http://10.211.55.8

Great! We can see Nginx default page.

We have our web server running in a Jail. You can now configure it as you need.

Using this technique, we can configure other jails to provide services on our host. If we run our jails in this manner, all of the services will run on the host's network stack. This is ok for a simple setup, but we might want to do something more complex. We might want to isolate jails from each other or have each jail with their own loopback interface, or maybe we would like to replicate a network setup for a testing server. You could use virtual networks to accomplish that. But that is beyond the scope of this post.

Final thoughts

This post showed a quick and simple example of how can we manage services using jails. You will configure and manage your jails in a very similar manner to how you would handle any regular host. Just take into account that the jail assumes its root directory is where you specified it.

If you know how to do something on a typical server installation, you'll be able to do the same on a complete jail. Because the focus of this post was only to show you the process of setting a service on a jail, I skipped some maintenance steps. For example, patching and keeping your jail base-system up to date. Keep that in mind. You still need to maintain your jail's base system.

There are much more topics covered on the Jails and VNET guide, you should check it. The guide is free, but if you find it useful, you can later buy it and pay whatever you feel would be a reasonable price.

Ok, that's it for this post. I hope this was useful.