Multi-Purpose Raspberry Pi 3 Home Server with Docker [Nextcloud, Gogs, Home Assistant etc.] Running from HDD, with Reverse Proxy [Traefik] and Auto DNS Updates [via ddclient/Cloudflare]

Panos Mazarakis -

Here is the setup I'm using on my Raspberry Pi 3 server, compiled from different guides across the internet. Raspbian is running from an HDD for better performance, with most of the services running on Docker. This offers great maintainability, as all services start with a single docker-compose up. By having a reverse-proxy you don't need to expose various ports on your system, only 80 and 443. Plus Traefik auto-manages Let’s Encrypt SSL certificates, with auto renewal.

Raspbian Setup

If you don't have Raspbian installed, grab the official Stretch image and flash it with Etcher.

After logging in update your distro:

$ sudo apt-get update && sudo apt-get upgrade -y
$ sudo apt-get dist-upgrade
$ sudo apt-get autoremove
$ sudo reboot

Running Raspbian from HDD

Make sure you have rsync installed:

$ sudo apt-get install rsync

You can follow part 4 of this excellent guide. To summarize it:

First unmount your sda:

 $ sudo umount /dev/sda1

Then prepare the disk. This example makes a 10GB ext4 partition and another ext4 with the rest of the disk space:

$ sudo parted /dev/sda

(parted) mktable msdos
Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to continue?
Yes/No? Yes
(parted) mkpart primary ext4 0% 10000M
(parted) mkpart primary ext4 10000M 100%

Format boot and root file systems:

$ sudo mkfs.ext4 /dev/sda1
$ sudo mkfs.ext4 /dev/sda2

Mount your sda1 into mnt:

$ sudo mount /dev/sda1 /mnt

Copy the root to the mounted hdd:

$ sudo rsync -axv / /mnt

Now get the list os UUIDs and PARIDS of your disk:

$ sudo blkid | grep sda

/dev/sda1: UUID="uuid" TYPE="ext4" PARTUUID="partuuid-01"
/dev/sda2: UUID="uuid" TYPE="ext4" PARTUUID="partuuid-02"

You need to add your sda1's PARTUUID to root and add rootdelay=5 at the end. Change the boot file from default SD to HDD:

$ sudo nano /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=partuuid-01 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait rootdelay=5

Create the mount target to your sda2:

$ sudo mkdir /mnt/mnt/storage

Now will need to change fstab on the HD to auto mount everything. DO NOT CHANGE ON THE SD:

$ sudo nano /mnt/etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  none            ext4    ro,noauto         0       1
/dev/disk/by-uuid/uuid /            ext4 defaults,noatime 0 1
/dev/disk/by-uuid/uuid /mnt/storage ext4 defaults,noatime 0 0

$ sudo cp /mnt/etc/fstab /boot/fstab_usb
$ sudo reboot

Setup ddclient for Auto DNS Updates

I'm using Cloudflare as my DNS. Other services work too, but they're not covered by this guide. Create the A records on Cloudflare for the subdomains you want to use.

You're going to need git, perl plus some perl packages:

$ sudo apt-get install git perl libdata-validate-ip-perl libjson-perl

First clone ddclient to your home dir:

$ cd ~
$ git clone https://github.com/ddclient/ddclient
$ cd ddclient

Then install it:

$ sudo cp ddclient /usr/sbin/
$ sudo mkdir /etc/ddclient
$ sudo mkdir /var/cache/ddclient
$ sudo cp sample-etc_ddclient.conf /etc/ddclient/ddclient.conf
$ sudo nano /etc/ddclient/ddclient.conf

ddclient.conf:

daemon=300				# check every 300 seconds
syslog=yes				# log update msgs to syslog
pid=/var/run/ddclient.pid		# record PID in file.
ssl=yes					# use ssl-support.
use=web					# via web

##
## CloudFlare (www.cloudflare.com)
##
protocol=cloudflare,        \
zone=yourdomain.com,            \
ttl=1,                      \
login=yourcloudflaremail@sample.com,     \
password=your_cloudflare_api_key             \
subdomain1.yourdomain.com,subdomain2.yourdomain.com

$ sudo cp sample-etc_rc.d_init.d_ddclient.ubuntu /etc/init.d/ddclient
$ sudo update-rc.d ddclient defaults

Start the first time by hand:

$ sudo service ddclient start

To debug it:

$ sudo ddclient -daemon=0 -debug -verbose -noquiet

Docker with Traefik Proxy Setup

Here are the services we'll be using:

Install Docker & Docker Compose

Install Docker using the convenience script from Docker Documentation:

$ curl -fsSL get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh

Install pip if you don't have it already:

$ sudo apt-get install python-pip

Then install docker-compose via pip:

$ sudo pip install docker-compose

Add your user to the docker group

$ sudo usermod -aG docker ${USER}

Logout and login again to apply changes.

Docker Folders and Permissions

$ mkdir ~/docker
$ sudo chmod -R 775 ~/docker

.htpasswd File

This will be used to password protect plain HTTP pages on your server (like the traefik dashboard), via an .htpasswd file. Use this HTPASSWD Generator to create a combination with this structure:

sample_username:sample_password

Make a shared folder to use for docker containers, with a .htpasswd file inside, then paste the generated password:

$ mkdir ~/docker/shared
$ nano ~/docker/shared/.htpasswd

Docker environment variables

Make an .env file in your docker folder to keep all of the environment variables you'll be using:

$ touch ~/docker/.env

Before writing the .env file, get the PUID(uid) and PGID(gid) of your user with the id command:

$ id

Edit your .env file similar to this:

$ cd ~/docker
$ nano .env

.env:

    PUID=1004
    PGID=1004
    TZ=Europe/Athens
    USERDIR=/home/pi
    MYSQL_ROOT_PASSWORD=your_secure_password_here
    HTTP_USERNAME=htpasswd_file_username
    HTTP_PASSWORD=htpasswd_file_password
    DOMAINNAME=cloud.sample.com
    CLOUDFLARE_EMAIL=sample@sample.com
    CLOUDFLARE_API_KEY=your_cloudflare_api_key

Prepare the Traefik configs

Create new folders for Traefik and ACME:

$ mkdir ~/docker/traefik
$ mkdir ~/docker/traefik/acme

Docker cannot create missing files (only directories). So we will need to create an empty file for Traefik docker container to use. You also need to set the proper permissions:

$ touch ~/docker/traefik/acme/acme.json
$ chmod 600 ~/docker/traefik/acme/acme.json

Next we move on to the traefik.toml file:

$ touch ~/docker/traefik/traefik.toml
$ nano ~/docker/traefik/traefik.toml

traefik.toml:

#debug = true

logLevel = "ERROR" #DEBUG, INFO, WARN, ERROR, FATAL, PANIC
InsecureSkipVerify = true 
defaultEntryPoints = ["https", "http"]

# WEB interface of Traefik - it will show web page with overview of frontend and backend configurations 
[web]
address = ":8080"
  [web.auth.basic]
  usersFile = "/shared/.htpasswd"

# Force HTTPS
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]

# Let's encrypt configuration
[acme]
email = "your@email.com"
storage="/etc/traefik/acme/acme.json"
entryPoint = "https"
acmeLogging=true 
onDemand = false #create certificate when container is created
[acme.dnsChallenge]
  provider = "cloudflare"
  delayBeforeCheck = 0
[[acme.domains]]
   main = "CLOUD.YOURDOMAIN.COM"
[[acme.domains]]
   main = "*.CLOUD.YOURDOMAIN.COM"
[[acme.domains]]
   main = "OTHERSUBDOMAIN.YOURDOMAIN.COM"
   
# Connection to docker host system (docker.sock), pointing to your raspberry
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "CLOUD.YOURDOMAIN.COM"
watch = true
# This will hide all docker containers that don't have explicitly  
# set label to "enable"
exposedbydefault = false

Create the proxy network for docker

After this you'll have an internal docker network (default) and an external, used by Traefik (traefik_proxy):

$ docker network create traefik_proxy

The docker-compose file

This will start all services, along with the reverse-proxy.

Create the file:

$ touch ~/docker/docker-compose.yml
$ nano ~/docker/docker-compose.yml

docker-compose.yml:

version: "3.6"
services:

    traefik:
        hostname: traefik
        image: traefik:latest
        container_name: traefik
        restart: always
        domainname: ${DOMAINNAME}
        networks:
          - default
          - traefik_proxy
        ports:
          - "80:80"
          - "443:443"
          - "8080:8080"
        environment:
          - CLOUDFLARE_EMAIL=${CLOUDFLARE_EMAIL}
          - CLOUDFLARE_API_KEY=${CLOUDFLARE_API_KEY}
        labels:
          - "traefik.enable=true"
          - "traefik.backend=traefik"
          - "traefik.frontend.rule=Host:traefik.${DOMAINNAME}"  
          - "traefik.port=8080"
          - "traefik.docker.network=traefik_proxy"
        volumes:
          - /var/run/docker.sock:/var/run/docker.sock:ro
          - ${USERDIR}/docker/traefik:/etc/traefik
          - ${USERDIR}/docker/shared:/shared

    portainer:
      image: portainer/portainer:arm #use the arm image for this
      container_name: portainer
      restart: always
      ports:
        - "9000:9000"
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock
        - ${USERDIR}/docker/portainer/data:/data
        - ${USERDIR}/docker/shared:/shared
      environment:
        - TZ=${TZ}

    homeassistant:
      container_name: homeassistant
      restart: always
      image: homeassistant/raspberrypi3-homeassistant:0.73.0
      volumes:
        - ${USERDIR}/docker/homeassistant:/config
        - /etc/localtime:/etc/localtime:ro
        - ${USERDIR}/docker/shared:/shared
      privileged: true
      environment:
        - PUID=${PUID}
        - PGID=${PGID}
        - TZ=${TZ}
      labels:
        - "traefik.enable=true"
        - "traefik.backend=homeassistant"
        - "traefik.frontend.rule=Host:homeassistant.${DOMAINNAME}"
        - "traefik.port=8123"
        - "traefik.docker.network=traefik_proxy"

    nextcloud:
      container_name: nextcloud
      restart: always
      image: nextcloud:latest
      links:
        - mariadb
      volumes:
        - /mnt/nextcloud:/var/www/html
      environment:
        - PUID=${PUID}
        - PGID=${PGID}
        - TZ=${TZ}
      networks:
        - traefik_proxy
        - default
      labels:
        - "traefik.enable=true"
        - "traefik.backend=nextcloud"
        - "traefik.frontend.rule=Host:nextcloud.${DOMAINNAME}"
        - "traefik.docker.network=traefik_proxy"
        - "traefik.port=80"

    gogs:
        container_name: gogs
        restart: always
        image: gogs/gogs-rpi
        links:
            - mariadb
        ports:
            - "10022:22"
        volumes:
          - /mnt/gogs:/data
        labels:
          - "traefik.enable=true"
          - "traefik.backend=gogs"
          - "traefik.frontend.rule=Host:git.${DOMAINNAME}"
          - "traefik.port=3000"
          - "traefik.docker.network=traefik_proxy"
          
    mariadb:
       image: hypriot/rpi-mysql:latest
       container_name: "mariadb"
       hostname: mariadb
       volumes:
           - ${USERDIR}/docker/mariadb:/var/lib/mysql
       ports:
         - target: 3306
           published: 3306
           protocol: tcp
           mode: host
       restart: always
       environment:
         - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
         - PUID=${PUID}
         - PGID=${PGID}
         - TZ=${TZ}

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  default:
    driver: bridge

If you want to use Owncloud instead of Nextcloud, use the arm32v7 image:

image: arm32v7/owncloud:latest

Start the server

$ docker-compose -f ~/docker/docker-compose.yml up -d

Make sure you have forwarded ports 80 & 443 on your router! Docker will download the images you specified and start. Traefik will then generate the certificates from Let's Encrypt and store them to the acme.json file. The first time you run it it will take some minutes. You can monitor the process with this command:

$ docker stats

Maintenance

Stopping/restarting containers. For single containers use:

$ docker-compose stop CONTAINER-NAME

To stop all the containers spawned by docker-compose:

$ docker-compose -f ~/docker/docker-compose.yml down

Cleaning up docker from leftover images:

$ docker system prune
$ docker image prune
$ docker volume prune

Backups:

To take backups the only thing you need to do is copy the docker-mounted folders to another drive (~/docker, /mnt/nextcloud, /mnt/gogs).

Updates:

The update process is simple:

$ docker-compose -f ~/docker/docker-compose.yml down && docker-compose -f ~/docker/docker-compose.yml up -d

Docker will download images with the latest tag if newer versions are found. If you're not using latest tags, just edit your docker-compose.yml before restarting.

Sources: