Chapter 1: Initial Setup and Key Creation

This stage establishes your key pair and trust level.

Create Master Key Pair incl. encryption subkey

gpg --full-gen-key

On the following dialog you use ECC algorithms (ed25519 for signing, cv25519 for encryption) and chose option (9) ECC (sign, encryption) standard. You set an expiration date (e.g., 5 years) for forward compatibility. Finally you choose a strong, unique passphrase!

Create signing Subkey

Do not use the Master Key for signing in your daily work. Create an additional separate subkey for signing (S).

gpg --edit-key <KEY-ID>

gpg\> addkey

You type addkey and then option (10) ECC (only sign).

Then you check the generated Master Key including the subkeys

gpg --list-secret-keys

sec   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

sec: The secret master key (sec) is for Signing and Certification (SC).

KEY-ID: C957EA3BD909D1D3BA594D2CE14085D8197A0018.

ssb: The first subkey (ssb) is for encryption (E).

ssb: The second subkey (ssb) is for signing (S).

All Keys expire at 2030-12-24.

Set Trust Level

As you can see above my trust level is already ultimate. So here just for completion the command how to set the trust level on your keys.

Mark your own key as ultimately trusted so GnuPG relies on your certifications (key separation, subkey creation).

gpg --edit-key <KEY-ID>

gpg\> trust

Then type trust and select option 5 (ultimate).

Chapter 2: Security and Hardening

This stage prepares the necessary backups and secures the master key.

Backup the Master Key and generate Revocation Certificate

Do this immediately and store the fullback-up file and the revocation certificate file offline.

gpg --export-secret-keys --armor <KEY-ID> > fullbackup.asc	

gpg --output revocation.asc --gen-revoke <KEY-ID>

Implement Key Separation

Check the key setup again to identify clearly your 3 Key-IDs. Here we use the long format command.

gpg --list-secret-keys --keyid-format long

sec   ed25519/E14085D8197A0018 2025-12-25 \[SC] \[verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid              [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)

ssb   cv25519/AF0CB4F423538B80 2025-12-25 [E] [verfällt: 2030-12-24]

ssb   ed25519/56025D0BF8BF5B2F 2025-12-25 [S] [verfällt: 2030-12-24]

Master Key-ID (SC): E14085D8197A0018

Subkey 1-Key-ID (E): AF0CB4F423538B80

Subkey 2-Key-ID (S): 56025D0BF8BF5B2F

The goal is: Remove the Master Secret Key (sec) from your local machine using the delete command, turning it into a stub (sec#). This protects the core of your identity from malware. Keep both Subkeys for daily encryption (E) and signing (S).

Before you run the delete command, you must be 100% certain your backup file is valid. If you delete the master key now without a verified backup, you are back to the fatal „lost key“ situation. Run the following 2 checks.

Check 1:

gpg --list-packets /path/to/fullbackup.asc | grep "secret key packet"

:secret key packet:

The output must! show:

:secret key packet:

Check 2:

gpg --list-packets /path/to/fullbackup.asc

off=0 ctb=94 tag=5 hlen=2 plen=134
:secret key packet:
	version 4, algo 22, created 1766645842, expires 0
	pkey[0]: [80 bits] ed25519 (1.3.6.1.4.1.11591.15.1)
	pkey[1]: [263 bits]
	iter+salt S2K, algo: 7, SHA1 protection, hash: 2, salt: E5DD0BCA22F746A5
	protect count: 65011712 (255)
	protect IV:  d6 b2 c9 5d e2 3f 05 d3 35 fb cc 7b 87 41 cf f5
	skey[2]: [v4 protected]
	keyid: E14085D8197A0018
# off=136 ctb=b4 tag=13 hlen=2 plen=44
:user ID packet: "Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)"
# off=182 ctb=88 tag=2 hlen=2 plen=153
:signature packet: algo 22, keyid E14085D8197A0018
	version 4, created 1766645842, md5len 0, sigclass 0x13
	digest algo 10, begin of digest f0 54
	hashed subpkt 33 len 21 (issuer fpr v4 C957EA3BD909D1D3BA594D2CE14085D8197A0018)
	hashed subpkt 2 len 4 (sig created 2025-12-25)
	hashed subpkt 27 len 1 (key flags: 03)
	hashed subpkt 9 len 4 (key expires after 5y0d0h0m)
	hashed subpkt 11 len 4 (pref-sym-algos: 9 8 7 2)
	hashed subpkt 34 len 1 (pref-aead-algos: 2)
	hashed subpkt 21 len 5 (pref-hash-algos: 10 9 8 11 2)
	hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
	hashed subpkt 30 len 1 (features: 07)
	hashed subpkt 23 len 1 (keyserver preferences: 80)
	subpkt 16 len 8 (issuer key ID E14085D8197A0018)
	data: [256 bits]
	data: [255 bits]
# off=337 ctb=9c tag=7 hlen=2 plen=139
:secret sub key packet:
	version 4, algo 18, created 1766645842, expires 0
	pkey[0]: [88 bits] cv25519 (1.3.6.1.4.1.3029.1.5.1)
	pkey[1]: [263 bits]
	pkey[2]: [32 bits]
	iter+salt S2K, algo: 7, SHA1 protection, hash: 2, salt: 2DE27015BEF91567
	protect count: 65011712 (255)
	protect IV:  f8 13 05 2b 8d a0 97 0d 22 99 3b b5 c3 62 43 be
	skey[3]: [v4 protected]
	keyid: AF0CB4F423538B80
# off=478 ctb=88 tag=2 hlen=2 plen=126
:signature packet: algo 22, keyid E14085D8197A0018
	version 4, created 1766645842, md5len 0, sigclass 0x18
	digest algo 10, begin of digest fc 2e
	hashed subpkt 33 len 21 (issuer fpr v4 C957EA3BD909D1D3BA594D2CE14085D8197A0018)
	hashed subpkt 2 len 4 (sig created 2025-12-25)
	hashed subpkt 27 len 1 (key flags: 0C)
	hashed subpkt 9 len 4 (key expires after 5y0d0h0m)
	subpkt 16 len 8 (issuer key ID E14085D8197A0018)
	data: [254 bits]
	data: [256 bits]
# off=606 ctb=9c tag=7 hlen=2 plen=134
:secret sub key packet:
	version 4, algo 22, created 1766645997, expires 0
	pkey[0]: [80 bits] ed25519 (1.3.6.1.4.1.11591.15.1)
	pkey[1]: [263 bits]
	iter+salt S2K, algo: 7, SHA1 protection, hash: 2, salt: 09FD8782B037DC77
	protect count: 65011712 (255)
	protect IV:  e3 fc 51 3f 7d 35 f8 78 a9 43 72 fe 73 f9 82 16
	skey[2]: [v4 protected]
	keyid: 56025D0BF8BF5B2F
# off=742 ctb=88 tag=2 hlen=2 plen=245
:signature packet: algo 22, keyid E14085D8197A0018
	version 4, created 1766645997, md5len 0, sigclass 0x18
	digest algo 10, begin of digest fe 58
	hashed subpkt 33 len 21 (issuer fpr v4 C957EA3BD909D1D3BA594D2CE14085D8197A0018)
	hashed subpkt 2 len 4 (sig created 2025-12-25)
	hashed subpkt 27 len 1 (key flags: 02)
	hashed subpkt 9 len 4 (key expires after 5y0d0h0m)
	subpkt 16 len 8 (issuer key ID E14085D8197A0018)
	subpkt 32 len 117 (signature: v4, class 0x19, algo 22, digest algo 10)
	data: [256 bits]
	data: [256 bits]

This is the expected output: You expect 3 packets most important is the :secret key packet:

Master Key package:

:secret key packet: secret master key packet with KeyID: E14085D8197A0018

:signature packet: This show the signature packet belonging to your secret key packet and issuer key ID E14085D8197A0018

Subkey 1 Key package:

:secret sub key packet: secret subkey packet with KeyID: AF0CB4F423538B80

:signature packet: This show the signature packet belonging to your secret sub key packet and issuer key ID E14085D8197A0018

Subkey 2 Key package:

:secret sub key packet: secret subkey packet with keyid: 56025D0BF8BF5B2F

:signature packet: This show the signature packet belonging to your secret sub key packet and issuer key ID E14085D8197A0018

Only in case :secret key packet: is available proceed!

GnuPG does not have a single command like —delete-only-master key. Instead, the standard „clean“ way to do this is a three-step process.

Step 1: Export ONLY your Secret Subkeys in a temporary subkeys-only file

gpg --export-secret-subkeys --armor E14085D8197A0018 > subkeys-only.asc

Step 2: Delete the current Secret Key (complete wipe) from your MacBook

gpg --delete-secret-keys E14085D8197A0018

Step 3: Import the Secret Subkeys

gpg --import subkeys-only.asc

Step 4: Remove the temporary subkeys-only file

rm subkeys-only.asc

Then check your local setup.

gpg --list-secret-keys

sec#  ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
      C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender <patrick@rottlaender.eu>
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

Here you see your Master Secret Key (sec) has been removed from your local machine and turning it into a stub (sec#).

Chapter 3: Reset and Restore (Scenario: Start From Scratch)

These steps show how to completely wipe your local key and restore it using the backup.

Delete Complete Key Set (Local Wipe)

Use both commands to ensure both the remaining subkeys and the Public Key are completely removed from your local keyring before restoring.

gpg --delete-secret-keys <KEY-ID> (Deletes subkeys/stubs)
gpg --delete-keys <KEY-ID> (Deletes public key)

Import Complete Key Set (Restore)

Use the verified backup file created in Step 2 to fully restore your Master Key and Subkeys back to the pre-separation state.

First you check the fullback file before you run the import.

#Check the backupfile
gpg --show-keys /path/to/fullbackup.asc

sec   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

#run the import
gpg --import /path/to/fullbackup.asc

Chapter 4: Useful commands

List public keys

gpg –list-keys

pub   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
sub   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
sub   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

List secret keys

gpg –list-secret-keys

sec   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

List public keys showing all KeyIDs

gpg --list-keys --keyid-format long

pub   ed25519/E14085D8197A0018 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid              [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
sub   cv25519/AF0CB4F423538B80 2025-12-25 [E] [verfällt: 2030-12-24]
sub   ed25519/56025D0BF8BF5B2F 2025-12-25 [S] [verfällt: 2030-12-24]

List secret keys showing all KeyIDs

gpg --list-secret-keys --keyid-format long

sec   ed25519/E14085D8197A0018 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid              [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519/AF0CB4F423538B80 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519/56025D0BF8BF5B2F 2025-12-25 [S] [verfällt: 2030-12-24]

Master Key (SC): E14085D8197A0018

Subkey 1 (E): AF0CB4F423538B80

Subkey 2 (S): 56025D0BF8BF5B2F

Encrypt

gpg -e -r patrick@rottlaender.eu file.txt

Decrypt

gpg -d file.txt.gpg > file.txt

Sign & Encrypt

gpg -se -r recipient@email.com file.txt

Decrypt & Verify

#shows the signature status automatically
gpg -d file.txt.gpg

Chapter 5: General note

ed25519 and cv25519 are algorithms based on Curve25519

ed25519 (Master Key & Signing Subkey)

Feature Description

Full Name: Edwards-curve Digital Signature Algorithm over Curve25519.

Cryptographic Role: Digital Signing and Verification. It is designed to create a mathematical signature that proves the origin (authenticity) and integrity of a message or file.

GnuPG Use: Used for your Master Key ([C]) to certify other keys, and your Signing Subkey ([S]) to sign emails or commits.

Security Property: It is highly resistant to various side-channel attacks and has a very reliable, straightforward implementation.

cv25519 (Encryption Subkey)

Feature Description

Full Name: Curve25519 variant used for Key Exchange.

Cryptographic Role: Key Exchange and Encryption. It is designed for the Diffie-Hellman (DH) key exchange process, which allows two parties to securely establish a shared secret (like a session key) over an insecure channel.

GnuPG Use: Used for your Encryption Subkey ([E]). It enables others to encrypt data to you, and you to decrypt it using the established session key.

Security Property: Highly efficient and designed specifically for confidential (private) communication.

Modern GnuPG key management strictly separates cryptographic roles signing and encryption for security reasons:

• You sign with ed25519: To prove you are the one you are (in my case I am Patrick!).
• You encrypt with cv25519: To ensure privacy.

This separation of duties means if one subkey were ever theoretically broken, it wouldn’t automatically compromise the function of the other key. For example, if your signing key (ed25519) was compromised, your ability to decrypt past files or conversations (protected by cv25519) would remain secure.

Prepare your Server

Before we start we need to prepare the server. In this step we create the environment to manage the vaultwarden instance. Login to your server using your standard User (not root).

Basic requirements

Login to your server with SSH and authenticate with keys. Never use simple password authentication. You must create a private and a public SSH key-pair on your Host Machine and copy only the public SSH key to your Remote Server. Keep your private key safe on your Host machine. Then copy your public key to your Remote Server and configure your ssh-deamon on your Remote Server. Disable password authentication and root login in your configuration on your Remote Server. How all this works is very good explained on Linuxize.com.

You must ensure that SSL certificates for your server are installed. I use free LetsEncrypt certificates and certbot to install and renew my certificates on the system. Therefore pls. read on my Digitaldocblog in the Article SSL Certificates with Lets Encrypt and certbot on a Linux Server . Here you find a very detailed description of how you can do this .

You must ensure that docker and docker-compose is installed on your system. Therefore pls. read on my Digitaldocblog a very detailed description of how you can do this in the Article Containerize a nodejs app with nginx. You should read the following chapters:

• Prepare the System for secure Downloads from Docker
• Install Docker from Docker Resources
• Install standalone docker-compose from Docker Resources
  

Make sure that your server is running behind a firewall. In my case I have a virtual server and I am responsible for server security. Therefore I install and configure a firewall on my system.

Before you configure the firewall be sure that your ssh service is running and on which port your ssh service is running. This is important to know because you don’t want to lock yourself out. First you check if the ssh service is running with systemctl and then on which port ssh is running using netstat. If netstat is not installed on your system you can do this with apt.

#control ssh status
sudo systemctl status ssh

#check ssh port
sudo netstat -tulnp | grep ssh

#check if net-tools (include netstat) are installed
which netstat

#install net-tools only in case not installed
sudo apt install net-tools

I install ufw (uncomplicated firewall) on my server and configure it to provide only SSH and HTTPS to the outside world.

# check if ufw is installed
ufw version
which ufw

#install ufw if not installed
sudo apt install ufw

#open SSH and HTTPS
sudo ufw allow OpenSSH
sudo ufw allow 443
sudo ufw allow 80/tcp

#Default rules
sudo ufw default deny incoming
sudo ufw default allow outgoing

#Start the firewall
sudo ufw enable

#Check the firewall status
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
80/tcp                     ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)             
80/tcp (v6)                ALLOW IN    Anywhere (v6)

Port 443 must run TCP and UDP

In the Docker environment, we will later configure Caddy as a reverse proxy for the vaultwarden service. Caddy requires both TCP and UDP on port 443. This is due to a modern protocol called HTTP/3 (QUIC).

Traditionally, the web (HTTP/1.1 and HTTP/2) ran exclusively over the TCP protocol. TCP is very reliable, but sometimes a bit slow when establishing a connection. Google and others subsequently developed QUIC, on which the new standard HTTP/3 is based. QUIC uses UDP instead of TCP.

HTTP/3 QUIC is significantly faster when establishing a connection (handshake). It handles packet loss better, which is especially important for mobile connections on smartphones, for example, when using the Bitwarden app.

Caddy is a very modern reverse proxy that has HTTP/3 enabled by default. The „normal“ HTTPS connection (HTTP/1.1 or HTTP/2) is established over TCP port 443. Caddy offers clients (Browsers or the Bitwarden app) the option to switch to HTTP/3 via UDP port 443.

With vaultwarden, users often access their vaults via smartphones using LTE or Wi-Fi connections. When UDP port 443 is open, the app uses HTTP/3. This results in a more stable connection and improved vault synchronization due to reduced latency.

Why must Port 80 be open

When you follow the instructions in the Article SSL Certificates with Lets Encrypt and certbot on a Linux Server then you are using certbot and install your LetsEncrypt certificates on your local server with the following certbot command :

sudo certbot certonly --standalone -d <yourDomain>

Here you use the method standalone. Whenever you renew your certificates certbot starts a temporary web server to prove ownership of your domain to LetsEncrypt.

When renewing your certificates certbot attempts to start a temporary web server on port 80. Therefore, port 80 must be open on your system via your firewall rules.

Important: If another web server (such as Nginx or Apache) is already running permanently on port 80, this step will fail unless the service is stopped. So, if a web server is running on port 80, this service must be permanently stopped.

After Certbot successfully starts the web server on port 80, the following happens:

• ACME Challenge: Certbot contacts the Let’s Encrypt servers. These provide Certbot with a random string (token).
• Deployment: Certbot creates a small file on the spot at the URL http://yourDomain/.well-known/acme-challenge/token
• Verification: The Let’s Encrypt servers now attempt to access this exact URL via the public internet. If they find the file (and the correct token), it proves that you own the server under this domain.
  

Once the verification is successful, the temporary standalone web server is immediately shut down and port 80 is opened. Certbot generates a new private key (if configured) and signs the new certificate. The new certificate files are stored in the directory /etc/letsencrypt/archive/ and the symbolic links in the directory /etc/letsencrypt/live/ are updated.

Create new user vaultwarden

You are logged in with your standarduser. From the home directory of the standarduser you create the user vaultwarden and create the hidden directory /home/vaultwarden/.ssh. Then copy the authorized_keys file from your .ssh directory into the new created .ssh directory of the new user vaultwarden and set the owner and permissions.

The new user vaultwarden should be in sudo group to perform commands under root using sudo. And the user vaultwarden should be in docker group to perform docker commands without using sudo.

#logged-in with standard user and create new user 
sudo adduser vaultwarden

#create hidden .ssh directory in new users home directory
sudo mkdir /home/vaultwarden/.ssh

#copy authorized_keys file to enable ssh key login for new user
cd /home/standarduser/.ssh
sudo cp authorized_keys /home/vaultwarden/.ssh

#set the owner vaultwarden and permissions
sudo chown -R vaultwarden:vaultwarden /home/vaultwarden/.ssh
sudo chmod 700 /home/vaultwarden/.ssh
sudo chmod 600 /home/vaultwarden/.ssh/authorized_keys

#check permissions
ls -al /home/vaultwarden
drwx-- vaultwarden vaultwarden 4096 May 11 14:20 .ssh

ls -l /home/vaultwarden/.ssh
-rw-- vaultwarden vaultwarden 400 May 11 14:20 authorized_keys

#add user vaultwarden to sudo- and docker group
sudo usermod -aG sudo vaultwarden
sudo usermod -aG docker vaultwarden

#check vaultwarden groups (3 groups: vaultwarden sudo docker)
sudo groups vaultwarden
vaultwarden : vaultwarden sudo docker

Create /opt/vaultwarden directory

Login with the new user vaultwarden. Stay logged in as vaultwarden for the next steps. Do not perform the next steps or the installation of the vaultwarden server with root or any other user.

After you are logged in with vaultwarden you create a new directory /opt/vaultwarden. This is the runtime directory of your vaultwarden application and the place from where the docker containers will be started.

sudo mkdir /opt/vaultwarden
sudo chmod -R 700 /opt/vaultwarden

ls -l /opt/vaultwarden
drwx--vaultwarden vaultwarden 4096 Mai 16 07:06 vaultwarden
 

Then change into /opt/vaultwarden. You create the /opt/vaultwarden/vw-data directory which is the host directory for the docker containers. One of these containers will be started under the container name vaultwarden. This container vaultwarden run with root privileges and write into this host directory /opt/vaultwarden/vw-data.

mkdir /opt/vaultwarden/vw-data

ls -l /opt/vaultwarden
drwxrwxr-x 6 vaultwarden vaultwarden 4096 Mai 15 15:54 vw-data
`

Create /opt/vaultwarden/certs and copy your SSL certificates

The container vaultwarden is the web application where you can log-in and manage your passwords. As we will se below the container will be started under the user vaultwarden in /opt/vaultwarden and run with root privileges behind a reverse proxy server. As reverse proxy I will use Caddy which is a powerful platform. Caddy manages the requests from the outside world and forward requests via an internal docker network to the vaultwarden server.

Caddy must accept only HTTPS connections and use the standard directory for certificates /opt/vaultwarden/certs. Therefore we create this directory.

I use Letsencrypt certificates which are managed automatically by certbot and stored in the directory /etc/letsencrypt on my host server. The keys are set up the following structure:

• In /etc/letsencrypt/live/<domain> there are only symlinks that point
• to the real files in /etc/letsencrypt/archive/<domain>.
  

Copy the fullchain.pem and privkey.pem files from /etc/letsencrypt/live/<domain> to /opt/vaultwarden/certs and set the permissions accordingly.

#create certs directory
mkdir /opt/vaultwarden/certs

#change into certs directory
cd /opt/vaultwarden/certs

#copy the files
cp /etc/letsencrypt/live/<domain>/fullchain.pem fullchain.pem
cp /etc/letsencrypt/live/<domain>/privkey.pem privkey.pem

#set owner and group vaultwarden
chown vaultwarden:vaultwarden fullchain.pem
chown vaultwarden:vaultwarden privkey.pem

#set access (read write only vaultwarden)
chmod 600 privkey.pem
chmod 600 fullchain.pem

note: The cp command will place the actual files (not the symlinks) into the directory /opt/vaultwarden/certs unless you specify a -P or -d option. So the cp command follows the symlinks and grab the actual files behind. This is important when the certificates have been renewed and new certificate files are stored behind the symlinks.

Run Vaultwarden and Caddy as docker containers

The following setup creates a secure, email-enabled Vaultwarden instance behind a Caddy reverse proxy with HTTPS and admin access, running entirely via Docker.

Create docker-compose.yml

This docker-compose.yml file sets up two services Vaultwarden and Caddy to host a self-hosted password manager with HTTPS support.

#docker-compose.yml

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      DOMAIN: "<yourDomain>"
      SIGNUPS_ALLOWED: "false"
      SMTP_HOST: "<yourSmtpServer>"
      SMTP_FROM: "<yourEmail>"
      SMTP_FROM_NAME: "<yourName>"
      SMTP_USERNAME: "<yourEmail>"
      SMTP_PASSWORD: "<yourSmtpPasswd>"
      SMTP_SECURITY: "force_tls"
      SMTP_PORT: "465"
      ADMIN_TOKEN: '<yourAdminToken>'
    volumes:
      - ./vw-data:/data

  caddy:
    image: caddy:2
    container_name: caddy
    restart: always
    ports:
      - 443:443
      - 443:443/udp 
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
      - ./certs:/certs:ro
    environment:
      DOMAIN: "<yourDomain>"
      LOG_FILE: "/data/access.log"

vaultwarden service:

Runs the Vaultwarden server (a lightweight Bitwarden-compatible backend).

• Disables user signups (SIGNUPS_ALLOWED: „false“).
• Configures SMTP settings for sending emails (e.g. for password resets).
• Sets a secure admin token (using Argon2 hash) to access the /admin interface.
• Persists Vaultwarden data to ./vw-data on the host.
  

Enable admin page access:

With the ADMIN_TOKEN set we enable login to the admin site via <yourDomain>/admin. First create a secure admin password. The <yourAdminToken> value can be created with your admin password piped into argon2. The result is a hash value that must be a little modified and then inserted as <yourAdminToken>. It is important that you use single quotes in docker-compose.yml when you insert <yourAdminToken>.

sudo apt install -y argon2
echo -n '<yourAdminPassword>' | argon2 somesalt -e

#This is the result of the argon2 hashing
$argon2i$v=19$m=4096,t=3,p=1$c29tZXNhbHQ$D...

Then you modify the hash by adding a $ sign in front of each $ sign in the hash. In this case we add 5 $ signs.

#original value
$argon2i$v=19$m=4096,t=3,p=1$c29thbHQ$D...

#modified value
$$argon2i$$v=19$$m=4096,t=3,p=1$$c29thbHQ$$D...

Then you put the modified value in single quotes into docker-compose.yml.

#docker-compose.yml

.....

ADMIN_TOKEN: '$$argon2i$$v=19$$m=4096,t=3,p=1$$c29thbHQ$$D...'

Now you can access the admin page with <yourDomain>/admin and login with your admin password.

caddy service:

Uses Caddy web server to reverse proxy to Vaultwarden.

• Handles HTTPS using custom certificates from ./certs.
• Binds to port 443 for secure access.
• Reads its configuration from ./Caddyfile.
• Logs access to /data/access.log (mapped from ./caddy-data on the host).
  

Create Caddyfile

Caddy is a modern, powerful web server that automatically handles HTTPS, reverse proxying, and more. Caddy acts as a secure HTTPS reverse proxy, forwarding external requests to the Vaultwarden Docker container running on internal port 80.

This Caddyfile defines how Caddy should serve and protect your vaultwarden instance over HTTPS.

#Caddyfile
https://<domain> {
  log {
    level INFO
    output file /data/access.log {
      roll_size 10MB
      roll_keep 10
    }
  }

  # Use custom certificate and key
  tls /certs/fullchain.pem /certs/privkey.pem

  # This setting may have compatibility issues with some browsers
  # (e.g., attachment downloading on Firefox). Try disabling this
  # if you encounter issues.
  encode zstd gzip

  # Admin path matcher
  @adminPath path /admin*
  
  # Basic Auth for admin access
  handle @adminPath {
    # If admin path require basic auth
    basicauth {
      superadmin <passwdhash>
    }

    reverse_proxy vaultwarden:80 {
      header_up X-Real-IP {remote_host}
    }
  }

  # Everything else
  reverse_proxy vaultwarden:80 {
    header_up X-Real-IP {remote_host}
  }
}

Domain

https://<domain>

This defines the domain name Caddy listens on (e.g. https://yourinstance.example.com).

Logging

log {
  level INFO
  output file /data/access.log {
    roll_size 10MB
    roll_keep 10
  }
}

Logs all access to a file inside the container (/data/access.log), with log rotation.

TLS with Custom Certificates

tls /certs/fullchain.pem /certs/privkey.pem

Use your own Let’s Encrypt certificates from mounted files rather than auto-generating them.

Compression

encode zstd gzip

Enables modern compression methods to improve performance, though may cause issues with attachments on some browsers.

Admin Area Protection

@adminPath path /admin*

Matches all requests to /admin paths.

handle @adminPath {
  basicauth {
    superadmin <passwdhash>
  }

  reverse_proxy vaultwarden:80 {
    header_up X-Real-IP {remote_host}
  }
}

• Requires HTTP Basic Auth for access to /admin.
• Proxies/Forwards authenticated admin requests to the Vaultwarden container.
• Ensures the backend sees the original client IP address.
  

All Other Requests

reverse_proxy vaultwarden:80 {
  header_up X-Real-IP {remote_host}
}

• Proxies/Forwards all non-/admin traffic directly to Vaultwarden container.
• Ensures the backend sees the original client IP address.
  

Protect your admin page

To protect your admin page we can use a HTTP Basic Auth. This mean whenever you access <yourDomain>/admin a login window will pop up in your browser and ask you to provide a user name and a password. We use htpasswd which is part of the apache2-utils.

#install apache2-utils if not available
sudo apt install apache2-utils

#Create a hash for the user admin (you can use any user-name) 
htpasswd -nB admin

New password: 
Re-type new password: 
admin:$2y$05$HZukVJWhWMrT7qMO2n65bm/5JYlt5tO...

Then you insert only the hash (without admin: ….) into the Caddyfile.

#Caddyfile

.....

handle @adminPath 
bash
  basicauth {
    admin $2y$05$HZukVJWhWMrT7qMO2n65bm/5JYlt5tO...
  }

Create ssl certificates with Letsencrypt and certbot

You can follow the instructions in my post on digitaldocblog.com. When you followed these instructions your ssl certificates are installed on your server in standalone mode. Whenever you renew your certificates certbot initiates the domain validation challenge and a temporary server will be started trying to listen on port 80. Because we run a web server this port is blocked and the web server must be stoped before we can initiate the renewal process.

To avoid this we should change the certbot renewal from standalone mode to webroot. Webroot is a method that leverages your existing web server to handle the domain validation challenge process without the need to stop the web server. Certbot places a temporary file with a unique token in a specific directory within your web servers public „webroot“ directory (e.g., /var/www/html/.well-known/acme-challenge/). Let’s Encrypt then sends a request to your domain to retrieve this file. Since your web server is already running, it can serve the file without any interruption to your website’s availability.

In our docker.compose.yml we defined the backend service vaultwarden. This service is the backend service listening to port 80. We also defined a web server service caddy listening to port 443.

In our Caddyfile we specified only a web server only for https://<Domain> working as a reverse proxy. Any request for https://<Domain> will be forwarded to the backend service vaultwarden listening on port 80 vaultwarden:80.

To enable webroot for certbot certificate renewal we must change the docker.compose.yml file. For the caddy service and in the ports section we must allow port 80 and expose a webroot directory via port 80 only for serving the domain challenge validation file. Therefore we create on the local host directory /opt/vaultwarden/caddyacme. In the volumes section of the caddy service we map this local directory 1:1 into the caddy container. Note: the notation with the dot like ./caddyacme:/caddyacme requires that the actual Caddyfile is in the same directory /opt/vaultwarden/Caddyfile than /opt/vaultwarden/caddyacme.

#docker-compose.yml

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      DOMAIN: "https://bitwarden.rottlaender.eu"
      SIGNUPS_ALLOWED: "false"
      SMTP_HOST: "smtp.strato.de"
      SMTP_FROM: "bw@bitwarden.rottlaender.eu"
      SMTP_FROM_NAME: "Vaultwarden"
      SMTP_USERNAME: "bw@bitwarden.rottlaender.eu"
      SMTP_PASSWORD: "ZZqrmQEvydkxY3E2u8Y.KEbKNwgkTV"
      SMTP_SECURITY: "force_tls"
      SMTP_PORT: "465"
      ADMIN_TOKEN: '$$argon2i$$v=19$$m=4096,t=3,p=1$$c29tZXNhbHQ$$D/yu7vPhcpPz8Kk7G/R34YSO+NgtLzai0wVGSGL0RDE'
    volumes:
      - ./vw-data:/data

  caddy:
    image: caddy:2
    container_name: caddy
    restart: always
    ports:
      - 80:80
      - 80:80/udp
      - 443:443
      - 443:443/udp
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
      - ./certs:/certs:ro
      - ./caddy_acme:/caddy_acme
    environment:
      DOMAIN: "https://bitwarden.rottlaender.eu"
      LOG_FILE: "/data/access.log"

With this configuration we configure caddy to listen on port 80 and on port 443. In the current ufo firewall configuration we only allow the ports 22 and 443.

#Check the firewall status
sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)

We must open port 80 to enable the certbot webroot certificate renewal.

# open port 80
sudo ufw allow 80/tcp

# Check the firewall status
sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
80/tcp                     ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)             
80/tcp (v6)                ALLOW IN    Anywhere (v6)

Then we must also change the Caddyfile. We must insert a http://.. block to enable caddy to serve the ACME challenge.

#Caddyfile

http://bitwarden.rottlaender.eu {
    # Serve ACME Challenge at Port 80
    handle_path /.well-known/acme-challenge/* {
        root * /caddy_acme
        file_server
    }

    # Any other request redirect to HTTPS
    @notACME not path /.well-known/acme-challenge/*
    handle @notACME {
        redir https://bitwarden.rottlaender.eu{uri} permanent
    }
}

https://bitwarden.rottlaender.eu {
  log {
    level INFO
    output file /data/access.log {
      roll_size 10MB
      roll_keep 10
    }
  }

  # Use custom certificate and key
  tls /certs/fullchain.pem /certs/privkey.pem

  # ACME Challenge for Certbot Webroot
  handle_path /.well-known/acme-challenge/* {
    root * /caddy_acme
    file_server
  }

  # This setting may have compatibility issues with some browsers (e.g., attachment downloading on Firefox). Try disabling this if you encounter issues.
  encode zstd gzip

  # Admin path matcher
  @adminPath path /admin*
  
  # Basic Auth for admin access
  handle @adminPath {
    # If admin path require basic auth
    basicauth {
      superadmin $2y$05$HZukVJWhWMrT7qMOIenLkuf2n65bm/6n260TPXKb4Wn825JYlt5tO  # Password Hash
    }

    reverse_proxy vaultwarden:80 {
      header_up X-Real-IP {remote_host}
    }
  }

  # Everything else
  reverse_proxy vaultwarden:80 {
    header_up X-Real-IP {remote_host}
  }
}

Then we must also change the /etc/letsencrypt/renewal/bitwarden.rottlaender.eu.conf to configure certbot to use webroot instead of standalone. Here change the authenticator directive to webroot and we add the webrootpath. Everything else keep the same.

# renew_before_expiry = 30 days
version = 1.21.0
archive_dir = /etc/letsencrypt/archive/bitwarden.rottlaender.eu
cert = /etc/letsencrypt/live/bitwarden.rottlaender.eu/cert.pem
privkey = /etc/letsencrypt/live/bitwarden.rottlaender.eu/privkey.pem
chain = /etc/letsencrypt/live/bitwarden.rottlaender.eu/chain.pem
fullchain = /etc/letsencrypt/live/bitwarden.rottlaender.eu/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 7844ed1ad487ff139e3adaa80aa7bbab
authenticator = webroot
webroot_path = /opt/vaultwarden/caddy_acme
server = https://acme-v02.api.letsencrypt.org/directory
renew_hook = /usr/local/bin/sync-certs.sh

And finally we must change the entry in the crontab. Before the depoly-hook we change the code to –webroot -w /opt/vaultwarden/caddyacme. This renewal will be executed daily at 03:30h. In case the certificates are still valid (no renewal required) then the deploy-hook will be skipped. Only in case a renewal must be executed the script /usr/local/bin/sync-certs.sh will be executed.

#crontab

30 3 * * * sh -c '/usr/bin/certbot renew --webroot -w /opt/vaultwarden/caddy_acme --deploy-hook /usr/local/bin/sync-certs.sh >> /var/log/certbot-renew.log 2>&1'

Create sync-certs.sh script and root crontab

This script copies renewed Let’s Encrypt certificates from the standard location to a custom destination /opt/vaultwarden/certs, sets strict permissions, assigns correct ownership, and restarts the Caddy container to apply the new certificates. At the end it reloads the Caddy container (via docker-compose restart) to apply new certs.

#!/bin/bash

# Variables
DOMAIN="<Domain>"
SRC="/etc/letsencrypt/live/$DOMAIN"
DEST="/opt/vaultwarden/certs"

# Check Source Directory
if [ ! -d "$SRC" ]; then
    echo "Certificate Path $SRC not found"
    exit 1
fi

# Check Destination Directory
if [ ! -d "$DEST" ]; then
    mkdir -p "$DEST"
    chown vaultwarden:vaultwarden "$DEST"
    chmod 700 "$DEST"
    echo "Target Path $DEST created"
fi

# Copy files (overwrite)
cp "$SRC/fullchain.pem" "$DEST/fullchain.pem"
cp "$SRC/privkey.pem" "$DEST/privkey.pem"

# set owner:group vaultwarden
chown vaultwarden:vaultwarden "$DEST/fullchain.pem"
chown vaultwarden:vaultwarden "$DEST/privkey.pem"

# set access (read write only vaultwarden)
chmod 600 "$DEST/privkey.pem"
chmod 600 "$DEST/fullchain.pem"

echo "[sync-certs] Certificates for $DOMAIN synced"

# successful sync of certificates – caddy re-start
echo "[sync-certs] re-start caddy ..."
cd /opt/vaultwarden
/usr/local/bin/docker-compose restart caddy

echo "[sync-certs] caddy reloaded new certificates"

Check Source Directory

if [ ! -d "$SRC" ]; then
    echo "Certificate Path $SRC not found"
    exit 1
fi

• What it checks: Whether the Let’s Encrypt certificate source directory for the domain exists.
• Why it’s needed:
  
  • Let’s Encrypt stores certificates as symlinks in /etc/letsencrypt/live/<domain>.
  • If this folder doesn’t exist, the script stops immediately to avoid copying from a missing or invalid source.
• Fail-safe: Prevents copying non-existent files, which would cause later commands to fail.
  

Check and Create Destination Directory

if [ ! -d "$DEST" ]; then
    mkdir -p "$DEST"
    chown vaultwarden:vaultwarden "$DEST"
    chmod 700 "$DEST"
    echo "Target Path $DEST created"
fi

• What it checks: Whether the destination directory for the copied certificates exists.
• If not:
  
  • It creates the directory (mkdir -p ensures parent paths are created if missing).
  • Sets secure permissions:
    
    • Owner: vaultwarden
    • Permissions: 700 – only the vaultwarden user can access the directory.
• Why this matters:
  
  • The vaultwarden container or process needs access to the certificates.
  • These permissions ensure only vaultwarden can read the certs, improving security.
    

Start, Stop and Check containers

Here are the most important docker-compose commands. Run these commands when you are in the directory where the docker-compose.yml file is and be sure that the logged in user is in the docker group (otherwise these commands work with sudo).

Here art the most important docker commands to check the status of containers.

Backup the vaultwarden data

To backup your database you must login to your remote server and navigate to the directory /opt/vaulwarden/vw-data.

cd /opt/vaultwarden/vw-data
ls -l /opt/vaultwarden/vw-data

drwxr-xr-x 2 root root    4096 Mai 14 07:29 attachments
-rw-r--r-- 1 root root 1437696 Mai 29 10:03 db_20250529_080308.sqlite3
-rw-r--r-- 1 root root 1437696 Mai 29 10:04 db_20250529_080448.sqlite3
-rw-r--r-- 1 root root 1445888 Aug  9 08:31 db_20250809_063113.sqlite3
-rw-r--r-- 1 root root 1552384 Aug 10 07:26 db.sqlite3
-rw-r--r-- 1 root root   32768 Sep  3 08:27 db.sqlite3-shm
-rw-r--r-- 1 root root  135992 Sep  3 08:27 db.sqlite3-wal
drwxr-xr-x 2 root root   16384 Sep  3 07:57 icon_cache
-rw-r--r-- 1 root root    1675 Mai 14 07:29 rsa_key.pem
drwxr-xr-x 2 root root    4096 Mai 14 07:29 sends
drwxr-xr-x 2 root root    4096 Mai 14 07:29 tmp

Before we run the backup here are some background infos.

Note: Pls. Be aware that you are currently navigating within your remote server not within the container vaultwarden. Under the service vaultwarden and volumes we defined in docker-compose.yml that the directory ./vw-data should be mounted into the container under the directory of /data. This mean that these directories are synchronized and you can check this by navigating within the container as follows.

To navigate within the container vaultwarden you can run the following commands.

#list your containers home directory
docker exec vaultwarden ls -l /

#list the data directory within your container
docker exec vaultwarden ls -l /data

With docker exec vaultwarden ls -l / you list your container root directory. Here you find among others the file vaultwarden which is basically the main program. With docker exec vaultwarden ls -l /data you list the files in /data on container side. You see exact the same files than in ./vw-data on your remote server.

All data of your vaultwarden service are stored in /data on container side. Here you have the database db.sqlite3 and various directories and other files.

To backup these data you run the program vaultwarden with the option backup.

#interactive mode
docker exec -it vaultwarden /vaultwarden backup

#standard mode
docker exec vaultwarden /vaultwarden backup
Backup to 'data/db_20250905_045937.sqlite3' was successful

The option exec -it is not necessary. It runs the command in interactive mode within the terminal.

This script:

1. Reads the database location and configurations on container side in /data.
2. Creates a backup of the SQLite database db.sqlite3, attachments, icons, etc.
3. Compresses everything as a .tar.gz archive.
4. Stores the archive in the /data directory on container side.
   

After executing this command the backup will be also be present in ./vw-data on your remote server.

Then you can download the backup from your remote server to your local computer. Note: Your local computer is in my case a Mac where I sit in front of and from where I connect to my remote server via ssh.

To download the backup you navigate on your local computer to the directory where you want to store your backup files. Then you run the scp command with the . at the end and download the backup files from your remote server into the current directory which should be your backup directory on your local computer.

#on your local computer
cd /Users/patrick/Software/docker/vaultwarden/backup

scp user@remote-host:/opt/vaultwarden/vw-data/<backupfile> .

sudo apt update
sudo apt install certbot

Then we run certbot.

sudo certbot certonly --standalone -d <yourDomain>

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): <yourEmail>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Requesting a certificate for <yourDomain>

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/<yourDomain>/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/<yourDomain>/privkey.pem
This certificate expires on 2025-07-25.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The Letsencrypt certificates are automatically stored by certbot in the directory /etc/letsencrypt. Pls. Check the permissions accordingly.

#700 for letsencrypt and owned by root
sudo ls -l /etc
drwx------ 9 root  root    4096 Mai 22 04:49 letsencrypt

#750 for archive and live and owned by root 
ls -l /etc/letsencrypt
drwx------ 3 root root 4096 Apr 26 10:16 accounts
drwxr-x--- 3 root root 4096 Apr 26 10:18 archive
-rw-r--r-- 1 root root  207 Nov 12  2021 cli.ini
drwx------ 2 root root 4096 Mai 16 07:15 csr
drwx------ 2 root root 4096 Mai 16 07:15 keys
drwxr-x--- 3 root root 4096 Apr 26 10:18 live
drwxr-x--- 2 root root 4096 Mai 16 07:15 renewal
drwxr-xr-x 5 root root 4096 Apr 26 10:16 renewal-hooks

#700 for the directory with the real certificates 
sudo ls -l /etc/letsencrypt/archive
drwx------ 2 root root 4096 Mai 16 07:15 <domain>

#750 for the directory with the symlinks
sudo ls -l /etc/letsencrypt/live
drwxr-x--- 2 root root 4096 Mai 16 07:15 <domain>

note: Permissions in Linux are set using the following scheme.

The following permissions are recommended:

The keys are set up the following structure:

• In /etc/letsencrypt/live/<domain> there are only symlinks that point to the real key files
• the real files are stored in /etc/letsencrypt/archive/<domain>
  

A symlink itself in /etc/letsencrypt/live/<domain> has no real permissions. It effectively inherits the access rights of the file it points to. However: ls -l shows symbolic permissions which, for symlinks, are usually lrwxrwxrwx. This isn’t problematic in itself, since access is always controlled by the target files.

sudo ls -l /etc/letsencrypt/live/<domain>
total 4
lrwxrwxrwx 1 root root  48 Mai 16 07:15 cert.pem -> ../../
lrwxrwxrwx 1 root root  49 Mai 16 07:15 chain.pem -> ../../
lrwxrwxrwx 1 root root  53 Mai 16 07:15 fullchain.pem -> ../../
lrwxrwxrwx 1 root root  51 Mai 16 07:15 privkey.pem -> ../../

Important are the permissions of the actual certificate files in the directory /etc/letsencrypt/archive/<domain>. The permissions should be set as follows.

sudo ls -l /etc/letsencrypt/archive/<domain>
total 32
-rw-r--r-- 1 root root 1858 Apr 26 10:18 cert1.pem
-rw-r--r-- 1 root root 1801 Apr 26 10:18 chain1.pem
-rw-r--r-- 1 root root 3659 Apr 26 10:18 fullchain1.pem
-rw------- 1 root root 1704 Apr 26 10:18 privkey1.pem

You should pay special attention to some other files.

/etc/letsencrypt/accounts/acme-<v02.api>/directory/<accountID>

The files meta.json, regr.json and private_key.json in the accounts directory should be owned by root and only root should have access. These files contain your Let’s Encrypt account credentials, especially the private account key, and are therefore highly sensitive.

sudo ls -l /etc/letsencrypt/accounts/acme-<v02.api>/directory/<accountID>
total 12
-rw------- 1 root root   85 Apr 26 10:18 meta.json
-r-------- 1 root root 1632 Apr 26 10:17 private_key.json
-rw------- 1 root root   80 Apr 26 10:17 regr.json

private_key.json: Contains your ACME account key – this is like a password for your certificate management.

• Read-only: The file must be readable by root so that Let’s Encrypt (or other tools like certbot) can communicate with this key to renew or issue certificates.
• Write-protected: Making the file unreadable even by root prevents it from being modified accidentally or through an unsafe operation. The private key should not be changed, as this could prevent access to your Let’s Encrypt account.

meta.json: Contains information such as the registered email address.

regr.json: Let’s Encrypt registration details.

A leak of these files would allow attackers to:

• Hijack your Let’s Encrypt account
• Issue certificates for any domain (if DNS access is compromised)
• Tamper or delete existing certificates
  

/etc/letsencrypt/csr and /etc/letsencrypt/keys directory

The files in the /etc/letsencrypt/csr and /etc/letsencrypt/keys directories should have the following permissions to ensure security.

Directory /etc/letsencrypt/csr stores Certificate Signing Requests (CSRs) generated by Certbot during certificate issuance or renewal. These are not used after issuance — they are kept for reference or debugging, not for active use.

sudo ls -l /etc/letsencrypt/csr
total 8
-rw------- 1 root root 936 Apr 26 10:18 0000_csr-certbot.pem
-rw-r--r-- 1 root root 936 Mai 16 07:15 0001_csr-certbot.pem

CSR files contain requests for certificate issuance and may contain sensitive information, such as the public key associated with a private key. Although the CSR file itself does not contain the private key, it can still pose informational risks because it represents an association with a private key.

Security measure: Access should be restricted to root to prevent an attacker from tampering with or accessing the CSR files.

Issue:After a renewal of the certificates on Mai 16 you can see above that the permissions to the file 0001_csr-certbot.pem has been set by certbot to 644. You can correct this with the following command.

sudo find /etc/letsencrypt/csr -name '*.pem' -exec chmod 600 {} +

sudo ls -l /etc/letsencrypt/csr
total 8
-rw------- 1 root root 936 Apr 26 10:18 0000_csr-certbot.pem
-rw------- 1 root root 936 Mai 16 07:15 0001_csr-certbot.pem

Directory /etc/letsencrypt/keys holds actual private keys used to sign CSRs and serve HTTPS. They must never be world-readable.

sudo ls -l /etc/letsencrypt/keys
total 8
-rw------- 1 root root 1704 Apr 26 10:18 0000_key-certbot.pem
-rw------- 1 root root 1704 Mai 16 07:15 0001_key-certbot.pem

Private keys are critical for security because they are used for authentication during certificate requests and for encrypting and decrypting data. The private key should be readable and writable only by root to prevent unauthorized access. Important: Any user with access to these key files could perform encryption operations or misuse certificates.

/etc/letsencrypt/renewal/<yourDomain>.conf

The file /etc/letsencrypt/renewal/<yourDomain>.conf contains the configuration for automatic certificate renewal and can contain sensitive data, such as the paths to private keys and certificates. Therefore, it is important to secure this file accordingly. The file should only be readable and writable by root, as it contains potentially sensitive information about the certificate configuration and the paths to private keys.

sudo ls -l /etc/letsencrypt/renewal
total 4
-rw------- 1 root root 606 Mai 16 07:15 <yourDomain>.conf

1. docker: Docker is a tool for building, running, and managing containers. A container is a lightweight, isolated environment that packages an application and all its dependencies (Docker application).
2. docker-compose: Docker Compose is a tool for defining and running multi-container Docker applications using a single file called docker-compose.yml.
3. node.js: Node.js is a JavaScript runtime that lets you run JavaScript outside the browser, typically on the server. It’s built on Chrome’s V8 engine, and it’s great for building fast, scalable network applications, like APIs or web servers.
4. npm: npm stands for Node Package Manager. It’s a tool that comes with Node.js and is used to Install packages (libraries, tools, frameworks), manage project dependencies and share your own packages with others.
5. curl: curl is a command-line tool used to send requests to URLs. It lets you interact with APIs or download content from the internet right from your terminal.
6. gnupg: GnuPG (or GPG, short for Gnu Privacy Guard) is a tool for encryption and signing data and communications. It uses public-key cryptography to encrypt, decrypt, sign, and verify files or messages.
7. ca-certificates: A collection of trusted root certificates used to validate HTTPS connections.
   

We must check if these packages are already installed on our system. Therefore we use the following commands in the terminal and see if there is any output. If there is no output the packages are not installed and we continue with the installation as described below.

# check the docker components
docker --version
docker-compose --version

#check the node and npm components
node --version
npm --version

#check if required dependencies are already installed
curl --version
gpg --version

In case some of these packages are already installed on your system you need to reduce the installation scope of the packages accordingly.

We expect to have no output after typing the above commands and go through the installation process step-by-step.

Step 1: Install node.js and npm from the standard Ubuntu resources.

Step 2:

• Prepare the system for secure downloads from Docker resources and
• install ca-certificates, curl and gnupg from standard Ubuntu resources

Step 3: Install Docker from Docker resources.

Step 4: Install docker-compose standalone from Docker resources.

Install Node.js and npm from Ubuntu Resources

Before we start with the installation we update and upgrade all packages. Node.js is available in Ubuntu’s repositories, so you can install it with the following commands.

sudo apt update
sudo apt upgrade

sudo apt install -y nodejs npm

Verify the installation:

node -v
npm -v

Prepare the System for secure Downloads from Docker

To prepare our system we ensure that ca-certificates, curl and gnupg are available on our system.

To be able to install the Docker packages from the external or not Ubuntu Docker repository apt must know where these resources are otherwise apt would install the packages from the Ubuntu repositories which would be the standard but not what we want. Therefore we must add the Docker repository to the apt tool. The complete process can be followed on the Docker Manual Pages.

When we add the Docker repository, the packages from that repository are digitally signed to ensure that these packages really come from docker. gnupg is a tool that allows our system to check these signatures against a trusted Docker GPG key. Therefore gnupg must be available on our system.

To make sure that the GPG key is available we must download the key from the Docker site. For the download we use the curl tool. Therefore curl must be available on our system.

We access the Docker site via HTTPS. Here ca-certificates comes into play. ca-certificates is a collection of trusted root certificates used to validate HTTPS connections. When downloading the Docker GPG key or accessing the Docker apt repository via HTTPS, Ubuntu checks the site’s SSL certificate against the collection of trusted root certificates. Therefore ca-certificates must be available on our system.

To check if ca-certificates is already installed we run the following command:

dpkg -l | grep ca-certificates

Note: The dpkg command stands for Debian Package Manager and is used to manage .deb packages on Debian-based systems like Ubuntu. It works at a lower level than apt, which is a higher-level package management tool that uses dpkg under the hood.

If it’s installed, you’ll see output like this:

ii ca-certificates 20230311ubuntu0.22.04.1 all Common CA..

In this case you do not need to install ca-certificates as describes below but it is highly recommended to update the collection of trusted root certificates before you continue.

sudo update-ca-certificates

We assume that we must install ca-certificates, curl and gnupg. First, we update the system package list to ensure everything is up-to-date on our system. This checks the installed packages for updates and upgrade the packages if required in one process. Then we install ca-certificates, curl and gnupg.

sudo apt update && sudo apt upgrade -y

sudo apt install -y ca-certificates curl gnupg

Add Docker GPG key:

To install the GPG keys from Docker we run the following command.

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null
sudo chmod a+r /etc/apt/keyrings/docker.asc

Let’s break down the full set of commands step by step:

sudo install -m 0755 -d /etc/apt/keyrings ….

• sudo runs the command with superuser (root) privileges.
• install is used for copying files and setting permissions.

note: This command is only sudo install and not sudo apt install . If we use apt install then this installs software packages from Ubuntu’s package repositories. Example: sudo apt install docker-ce. It’s used to install applications.

Only install is a Unix command (part of coreutils) used to create directories, copy files, and set permissions in a single step. Example: sudo install -m 0755 -d /etc/apt/keyrings. It’s used to prepare the system, not install software. So this command creates the /etc/apt/keyrings folder with secure permissions, which is later used to store GPG keyring files (such as Docker’s signing key).

• -m 0755 sets file permissions:
  
  • 0755 means:
  • 1st (0) is just a numerical representation
  • 2nd (7) is for the Owner (root) having 1 x read (4) + 1 x write (2) + 1 x execute (1) = 7 permissions.
  • 3rd (5) is for the Group (root) having 1 x read (4) + 0 x write (2) + 1 x execute (1) = 5 permissions (no write).
  • 4th (5) is for the Others having 1 x read (4) + 0 x write (2) + 1 x execute (1) = 5 permissions (no write).
• -d tells install to create a directory
• /etc/apt/keyrings is the target directory where the Docker GPG key will be stored.

What it does?

• Ensures that the /etc/apt/keyrings directory exists.
• Sets the correct permissions (readable but not writable by non-root users).
• This is a security best practice to keep GPG keys safe from tampering.
  

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null …

• curl a command-line tool to fetch files from a URL (we have installed it before)
• -fsSL flags to control curl behavior:
  
  • -f (fail silently on server http errors like 404 – site or resource not found).
  • -s (silent mode, no progress output).
  • -S (shows error messages if -s is used).
  • -L (follows redirects if the URL points elsewhere).
• https://download.docker.com/linux/ubuntu/gpg the URL for Docker’s GPG key file (the file name on the docker site is gpg).
• | (pipe) passes the downloaded data (the gpg file) to another command. In this case the data will be passed to the following sudo command.
• sudo tee /etc/apt/keyrings/docker.asc writes the key to before created directory /etc/apt/keyrings/docker.asc:
  
  • tee writes the output to a file (here it is docker.asc) while also displaying it in the terminal.
  • sudo ensures that the file is written with root permissions.
• > /dev/null redirects standard output to /dev/null to suppress unnecessary output. The tee command can also display and write at the same time, unless you silence it with > /dev/null.
  

note: sudo tee… runs with root permissions, so the file can be written even to protected directories such as /etc/apt/keyrings/ (we have set the permission to 0755. See above). You can also run the curl command with root permissions (sudo curl …) and then directly pass the output into a file with the -o option: sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc. This is suggested on the Docker Manaual Page but you can do it in both ways.

What it does?

• Downloads Docker’s official GPG key.
• Saves it securely in /etc/apt/keyrings/docker.asc.
• Ensures the key isn’t printed to the terminal.
  

sudo chmod a+r /etc/apt/keyrings/docker.asc

• sudo runs the command (in this case the chmod command) as root.
• chmod modifies file permissions.
  
  • a+r grants read (r) permission to all users (a).
• /etc/apt/keyrings/docker.asc the file whose permissions are being modified.

What it does?

• Ensures that all users (including apt processes) can read the GPG key.
• This is necessary so that apt can verify Docker package signatures when installing updates.
  

Previously, GPG files were stored in /etc/apt/trusted.gpg. This has changed.

Why Is This Necessary?

1. Security:

• Storing GPG keys in /etc/apt/keyrings/ instead of /etc/apt/trusted.gpg is a best practice.
• Prevents malicious modifications to package signatures.

1. Package Verification:

• The GPG key allows Ubuntu’s package manager (apt) to verify that Docker packages are genuine and not tampered with.

1. Future-proofing:

• Newer versions of Ubuntu prefer keys in /etc/apt/keyrings/ instead of the older /etc/apt/trusted.gpg.
  

Final Summary:

Add the Docker repository:

To install the docker resources to the apt sources list we run the following command.

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Let’s break down the command step by step:

echo „deb….“

The echo command output the text between the “….“. This is the APT repository entry for Docker. Let’s analyze its components:

• deb → Indicates that this is a Debian-based software repository.
• arch=$(dpkg –print-architecture):
  
  • dpkg –print-architecture dynamically retrieves the system architecture from your system (e.g., amd64, arm64).
  • This ensures that the correct package version for your system’s architecture is used.
• signed-by=/etc/apt/keyrings/docker.asc specifies the location of the docker GPG key docker.asc (we installed it before), which is used to verify the authenticity of packages downloaded from the repository.
• https://download.docker.com/linux/ubuntu the URL of Docker’s official repository.
• $(lsb_release -cs) dynamically fetches the codename of the Ubuntu version (e.g., jammy for Ubuntu 22.04).
  
  • This ensures that the correct repository for the current Ubuntu version is used.
• stable specifies that we are using the stable release channel of Docker.

| sudo tee /etc/apt/sources.list.d/docker.list

• The | (pipe) takes the output of echo and passes it to the tee command.
• sudo tee /etc/apt/sources.list.d/docker.list does the following:
  
  • tee writes the output to a file (/etc/apt/sources.list.d/docker.list).
  • sudo is required because writing to /etc/apt/sources.list.d/ requires root privileges.

> /dev/null

• The > /dev/null part discards the standard output of the tee command.
  
  • This prevents unnecessary output from being displayed in the terminal.
  • Without this, tee would both write to the file and display the text on the screen.
    

Install Docker from Docker Resources

Now, update the package list again and install Docker.

sudo apt update

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin

This command installs the following Docker key components using the apt package manager on Ubuntu (the -y option automatically answers „yes“ to all prompts, so the install runs without asking for confirmation).

docker-ce

• Docker Community Edition
• This is the core Docker Engine, the daemon that runs containers.
• Installs the Docker server that manages images, containers, volumes, and networks.

docker-ce-cli

• Docker Command-Line Interface
• This is the docker command you use in your terminal (e.g., docker run, docker ps, etc.).
• Separates the CLI from the engine so they can be updated independently.

containerd.io

• Container runtime
• A lightweight, powerful runtime for containers, used internally by Docker.
• Handles the actual low-level execution of containers.

docker-buildx-plugin

• BuildKit-powered Docker build plugin
• Adds docker buildx functionality for advanced builds, multi-arch images, and caching strategies.
• Useful when building complex container images.
  

Note: In some documentation you will see that the sudo apt install… command will include also the docker-compose-plugin. The docker-compose-plugin is not required here because we are using the docker-compose stand alone packed (see below). The docker-compose-plugin is integrated into the Docker CLI and can replace the docker-compose standalone binary. But we use the standalone version because of the lightweight minimal install, the backward compatibility and the easy and independent manual version control.

It is highly recommended to omit the docker-compose-plugin from your apt install command if you plan to install the standalone version of Docker Compose binary manually as we will do later. If you have both versions installed this can cause confusion, especially if scripts assume one or the other. Also, Docker might prioritize the plugin version in newer setups which might cause conflicts in our preferred standalone Docker Compose setup. The following table illustrates the problem because the command styles differ only very little.

In case the docker-compose-plugin has been installed on your system you can remove it with the following command:

sudo apt remove docker-compose-plugin

This removes the plugin version that integrates into the docker compose command. Later when we installed the standalone version of docker compose we use the commands instead of the space with a dash like docker-compose.

Verify that Docker is installed correctly:

sudo docker --version

Enable and start the Docker service:

sudo systemctl enable docker
sudo systemctl start docker

Test Docker by running the hello-world image.

sudo docker run hello-world

This command is a quick test to verify that Docker is installed and working correctly.

sudo

• Runs the command with superuser (root) privileges.
• Required unless your user is in the docker group.
• Docker needs elevated permissions to communicate with the Docker daemon (which runs as root).

docker

• The main Docker CLI (Command-Line Interface) tool.
• Used to interact with Docker Engine to manage containers, images, networks, volumes, etc.

run

• Tells Docker to create a new container and start it based on the hello-world image you specify.
  It does the following:
  
  • Pulls the image (if it’s not already downloaded).
  • Creates a new container from that image.
  • Starts and runs the container.
  • Outputs the result and then exits (for short-lived containers like hello-world).

hello-world

• This is the name of the Docker image.
• It’s an official image maintained by Docker, specifically designed to test Docker installations.
  

Install standalone docker-compose from Docker Resources

Installing the standalone docker-compose is useful when you:

• Need compatibility with legacy tools or scripts
• Want to control the exact version
• Prefer a lightweight, portable binary
  

The following command downloads the latest standalone docker-compose binary and saves it to a system-wide location.

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Let’s break down the command step by step:

sudo

• Runs the command with root privileges.
  
  • Required because /usr/local/bin is a protected directory that only root can write to.

curl

• A command-line tool used to download files from the internet.

-L

• Tells curl to follow redirects.
• GitHub uses redirects for release URLs, so this flag ensures the final binary is actually downloaded.

„https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)“

URL with Substitution. This ensures the correct binary is downloaded for your system. This is the dynamic download URL for the latest docker-compose release.

• $(uname -s) returns the operating system name (e.g., Linux, Darwin).
• $(uname -m) returns the architecture (e.g., x8664, arm64).

Example Output: https://github.com/docker/compose/releases/latest/download/docker-compose-Linux-x8664

-o /usr/local/bin/docker-compose

• Tells curl to write (-o option; output) the downloaded file to /usr/local/bin/docker-compose
• This is a standard location for user-installed binaries that are globally available in the system PATH.
  

After you run the command above you must do this. Give execution permissions. You’ll need to make the binary executable:

sudo chmod +x /usr/local/bin/docker-compose

And then check the version to confirm it worked:

docker-compose --version

Note: In some cases it might be necessary to switch to the docker compose plugin. This mean you must remove the standalone version from your system and install the docker compose plugin instead. Here is how you should proceed in such a scenario:

#find our where docker-compose has been installed
which docker-compose
/usr/bin/docker-compose

#remove file docker-compose 
sudo rm /usr/bin/docker-compose

#intstall docker compose plugin via docker
sudo apt install docker-compose-plugin

Final Verification

Check if everything is installed correctly:

docker --version
docker-compose --version
node -v
npm -v

Host-Docker-Setup for nodejs app behind nginx

The code of the nodejs app is explained in detail in the article Bootstrap Website running as nodejs app.

We have a simple website funtrails build in a one-page index.html with two sections: one section showing pictures of a Paterlini bike and one for showing pictures of a Gianni Motta bike. Each section containing an image gallery and text. The images are stored in an images directory.

  ├── .
	├── images
	├── index.html

Now we want to make this funtrails website run as a nodejs app behind a nginx reverse proxy. Both, the funtrails nodejs app and nginx should run in docker containers composed to work together. Therefore we crate the following file structure on the Host machine:

 ├── node
	├── funtrails
        │    ├─ views    
        │       ├── images
	│	├── index.html
	├── nginx

We copy all our web-content into the views directory under funtrails. The nodejs app is build in the mainfile app.js. All Dependencies for the nodejs app are defined the file package.json.

 ├── node
	├── funtrails
        │    ├─ app.js
    	│    ├─ package.json
        │    ├─ views    
        │       ├── images
	│	├── index.html
	├── nginx

In the Terminal go into the node/funtrails directory. Install the dependencies.

npm install

Then we have the following structure.

funtrails
├── app.js
├── node_modules
├── package.json
├── package-lock.json
└── views
    ├── images
    └── index.html

Go to node/funtrails. Run a test with the following command.

node app.js

nodejs funtrails demo app listening on port 8080!

Switch to your browser and hit http://localhost:8080 to see if all is working as expected. With Ctrl c in the terminal you can stop the app.

The nginx server will be started with the configuration of a nginx.conf file. The file nginx.conf will be created under nginx.

Note: This file nginx.conf is only for testing and will be changed when we add the SSL/TLS certificates. In this configuration below the nginx server will listen on port 80 and pass requests received from port 80 to the nodejs app listening on port 8080. Processing requests from port 80 is not state of the art as these connections are not encrypted. In production we need a port 443 connection with SSL/TLS for encrypted connections.

#nginx.conf

events {}

http {
	#Service node-app from docker-compose.yml
    upstream node_app {
        server node-app:8080;  
    }

    server {
        listen 80;

        location / {
            proxy_pass http://node_app;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

events {}

• This is a required block in NGINX configuration, even if empty.
• It handles connection-related events (like concurrent connections), but you don’t need to configure it unless you have advanced use cases.

http {

• Starts the HTTP configuration block — this is where you define web servers, upstreams, headers, etc.

upstream

• Defines a group of backend servers (can be one or many).
• node_app is just a name.
• Inside: server node-app:8080; means:
  
  • Forward traffic to the container with hostname node-app
  • Use port 8080 (that’s where your nodejs app listens)
  • node-app should match the docker-compose service name (node-app must be declared in your docker-compose.yml which will be explained below).
  • This lets you use proxy_pass http://nodeapp later, instead of hardcoding an IP or port.

server

• Defines a virtual server (a website or domain).
• listen 80; tells NGINX to listen for HTTP (port 80) traffic.

location

• Defines a rule for requests to / (the root URL of your site).
• You could add more location blocks for /api, /images, etc. if needed.
• Inside:
  
  • proxy_pass:
    
    • proxy_pass http://node_app; Tells NGINX to forward requests to the backend defined in upstream nodeapp
    • So: if you go to http://yourdomain.com/ NGINX proxies that to http://node-app:8080
• proxysetheader (see table)

The $variables in nginx.conf are built-in variables that NGINX provides automatically. They are dynamically set based on the incoming HTTP request. So these variables come from the NGINX core HTTP module and you don’t need to define them or import anything. They are always available in the config.

Here’s what each one is and where it comes from:

$host

• The value of the Host header in the original HTTP request.
• Example: If the user visits http://example.com, then $host is example.com.
• Use case: Tells the backend app what domain the client used — useful for apps serving multiple domains.

$remote_addr

• The IP address of the client making the request.
• Example: If someone from IP 203.0.113.45 visits your site, this variable is set to 203.0.113.45.
• Use case: Useful for logging, rate limiting, or geolocation in the backend app.

$proxy_add_x_forwarded_for

• A composite header that appends the client’s IP to the existing X-Forwarded-For header.
• Use case: Maintains a full list of proxy hops (useful if your request goes through multiple reverse proxies).
• If X-Forwarded-For is already set (by another proxy), it appends $remote_addr to it; otherwise, it sets it to $remote_addr.

$scheme

• The protocol used by the client to connect to NGINX — either http or https.
• Example: If the user visits https://example.com, then $scheme is https.
• Use case: Lets your backend know whether the original request was secure or not.
  

Then we have the following structure.

 ├── node
	├── funtrails
	│    ├─ app.js
    	│    ├─ package.json
        │    ├─ views    
        │       ├── images
	│	├── index.html
	├── nginx
            ├── nginx.conf

Create Docker Image and Container for nodejs app

This is to create the Docker Image with docker build. Then we run the Container from the image with docker run to test if everything is working as expected. If everything goes well we can go ahead with the nginx configuration and then with the composition of all together with docker-compose.

The dockerization process of the nodejs app in node/funtrails directory is controlled by the Dockerfile which will be created in node/funtrails. The dockerization process has the following steps.

1. Image creation
2. Container creation from the image
   

The Container can then be started, stopped and removed using terminal commands.

Go into in node/funtrails.

To get an Overview and check the Docker status of the system.

List all images. No images on your system.

sudo docker image ls

REPOSITORY   TAG       IMAGE ID   CREATED   SIZE  

List all Containers. As expected no Containers on the system.

sudo docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

List an Overview about Docker Images and Containers on your system. As expected no Containers and no Images on the system.

sudo docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     16        0         47.39MB   47.39MB

Create a Dockerfile in node/funtrails/Dockerfile. This file is required to build the image and run the Container.

#Dockerfile

#Image Build
#Install nodejs 12.22.9 for the Container
FROM node:12.22.9

#Set workdirectory for the Container
WORKDIR /home/node/app

#Change Owner and Group for Container Workdirectory to node
RUN chown node:node /home/node/app

#Run the Container with User node
USER node

#Copy all files from HOST Dir to the Container workdirectory 
COPY --chown=node:node . .

#(After COPY) Run the command to create Image for the Container
RUN npm install

#Container Start
#Open Port 8080 when the Container starts
EXPOSE 8080

#RUN the command when the Container starts
CMD [ "node", "app.js" ]
 

Create a .dockerignore file in node/funtrails/.dockerignore. The hidden dockerignore file exclude only files from the Host that will be copied into the image for the Container with the command COPY . .

#.dockerignore
node_modules

Note: In case you copy files from the Host into the image directly with I.e. COPY <file.1> <file.2> then file.1 and file.2 would be copies even if they would be listed in dockerignore.

We have the following structure on the Host machine.

funtrails
├─ Dockerfile
├─ .dockerignore
├── app.js
├── node_modules
├── package.json
├── package-lock.json
└── views
    ├── images
    └── index.html

Still be in node/funtrails.

Build the Docker image from the Dockerfile with the image name node-demo. The dot (.) at the end set the current directory on the Host machine for the docker command. This is location where docker is looking for the Dockerfile to build the image.

sudo docker build -t node-demo .

List all Docker images. 1 images just created.

sudo docker image ls

REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
node-demo    latest    c353353f045e   22 seconds ago   944MB

Run the Container from the image with the name node-demo and give the Container the name funtrails-solo-demo.

sudo docker run -d -p 8080:8080 --name funtrails-solo-demo node-demo

List all Docker Containers with the option -a. 1 Container with the name funtrails-solo-demo running from the image node-demo.

sudo docker ps -a

Access the running app on port 8080.

sudo curl http://localhost:8080

If everything went well you get a feedback in the terminal showing the HTML code. In this case the Test was successful.

Stop the running Docker Container with the name funtrails-solo-demo.

sudo docker stop funtrails-solo-demo

List all Containers with the option -a. 1 Container from the image node-demo with the name funtrails-solo-demo is EXITED.

sudo docker ps -a

List all images. Still 1 image available.

sudo docker image ls

REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
node-demo    latest    c353353f045e   36 hours ago   944MB

List an Overview about Docker Images and Containers on the system. 1 active Image and 1 not Active Container. Status of the Container is EXITED as we have seen above.

sudo docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         944MB     0B (0%)
Containers      1         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     18        0         47.39MB   47.39MB

To clean up your system use the following commands.

The full clean up (be careful).

sudo docker system prune -a

sudo docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

Configure docker-compose

Go back to the node directory and create a docker-compose.yml file there.

Then we have the following structure.

 ├── node
     ├── docker-compose.yml
     ├── funtrails
     │	   ├─ Dockerfile
     │	   ├─ .dockerignore
     │     ├─ app.js
     │     ├─ node_modules
     │	   ├─ package.json
     │	   ├─ package-lock.json
     │     ├─ views    
     │         ├── images
     │         ├── index.html
     ├── nginx
           ├── nginx.conf

docker-compose is a tool that helps you define and run multi-container Docker applications using a YAML file. Instead of running multiple docker run commands, you describe everything in one file and start it all with the command docker-compose up.

Create docker-compose.yml with the following content.

#docker-compose.yml

services:
  funtrails:
    build: ./funtrails
    container_name: funtrails 
    networks:
      - funtrails-network

  nginx:
    image: nginx:latest
    container_name: nginx-proxy
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - funtrails
    networks:
      - funtrails-network

networks:
  funtrails-network:
    driver: bridge

services

This section defines containers that make up your app

• funtrails
  
  • build: ./funtrails
    Builds the image from the Dockerfile inside the ./funtrails directory.
  • container_name: funtrails
    Names the container funtrails instead of a random name.
  • networks: funtrails-network
    Connects the container to a custom user-defined network.
• nginx
• image: nginx:latest
  Uses the official latest NGINX image.
• container_name: nginx-proxy
  Container will be named nginx-proxy.
• ports: „80:80“
• Exposes Host port 80 to Container port 80.
• volumes:
  Mounts your local nginx.conf into the container, read-only (:ro).
• depends_on: funtrails
  Ensures funtrails is started before nginx.
• networks: funtrails-network
  Both services are in the same network, so they can communicate by name.

networks

• Creates a custom bridge network named funtrails-network.
• Ensures containers can resolve each other by name (funtrails, nginx).
  

Note: We are using the official NGINX image directly (image: nginx:latest). This image is prebuilt and includes everything NGINX needs to run.

We don’t need to write a custom Dockerfile because we don’t want to:

• Add extra modules
• Customize the image beyond just the config
• Install additional tools
• Include SSL certs directly, etc.

Instead, we simply mount our own nginx.conf into the container using a volume. This tells Docker Use the official NGINX image, but replace its config file with mine. We would use a Dockerfile in the nginx directory if we need to build a custom NGINX image, for example to copy SSL certs directly into the image.

Example:

FROM nginx:latest
COPY ./nginx.conf /etc/nginx/nginx.conf
COPY ./certs /etc/nginx/certs

But for most use cases like reverse proxying a nodejs app, just mounting your own config file is perfectly sufficient and simpler.

Note:We integrate SSL in the next chapter using free Lets Encrypt certificates.

Integrate SSL certificates – free Lets Encrypt

To integrate SSL we need to do the following steps:

1. Prepare your Domain
2. Install certbot
3. Create Lets Encrypt SSL certificates
4. Adapt your node/docker-compose.yml
5. Adapt your node/nginx/nginx.conf
6. Create a cron-Job to renew SSL certificates
   

prepare the domain

You must own a domain like example.com and you must have access to your DNS-servers to adapt the A-Record. Here in my example I create a subdomain funtrails.example.com and create on my DNS an A-Record for funtrails.example.com that point to the servers IP-Adress.

install certbot

To install our certificates for SSL we use a tool called certbot. We install certbot with apt on our Linux machine.

sudo apt update
sudo apt install certbot

create Lets Encrypt SSL certificates

We create the SSL certificates with certbot.

sudo certbot certonly --standalone -d funtrails.example.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): <your-email>@funtrails.example.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Requesting a certificate for funtrails.example.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/funtrails.example.com/fullchain.pem

Key is saved at:         /etc/letsencrypt/live/funtrails.example.com/privkey.pem

This certificate expires on 2025-07-25.

These files will be updated when the certificate renews.

Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

To use SSL you need

• a server certificate (e.g. certificate.crt)
• a private key (e.g. private.key) and
• the CA certificate (e.g. ca.crt).

certbot create these file in the following directory on your Host server.

/etc/letsencrypt/live/funtrails.example.com

sudo ls -l /etc/letsencrypt/live/funtrails.example.com

cert.pem -> ../../archive/funtrails.example.com/cert1.pem
chain.pem -> ../../archive/funtrails.example.com/chain1.pem
fullchain.pem -> ../../archive/funtrails.example.com/fullchain1.pem
privkey.pem -> ../../archive/funtrails.example.com/privkey1.pem

The translation to the standard is as follows.

adapt node/docker-compose.yml

The docker-compose.yml will be adapted as follows.

services:
  funtrails:
    build: ./funtrails
    container_name: funtrails
    networks:
      - funtrails-network

  nginx:
    image: nginx:latest
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    depends_on:
      - funtrails
    networks:
      - funtrails-network

networks:
  funtrails-network:
    driver: bridge

We create a bridge network with the name funtrails-network and both services are running in this network. This is important to reach all services by their container name(s).

The funtrails service will rebuild from the Dockerfile in ./funtrails. For the nginx service the nginx-image will be loaded from the Docker resources in the latest version. For the nginx image it is defined that the Host ports 80 and 443 will be mapped into the Container ports 80 and 443. When the Container is started we mount the Host files ./nginx/nginx.conf and the SSL certificates unter /etc/letsencrypt into the Container. Both will be loaded with read only!

...
volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
...

With the dependson directive we declare that first the funtrails service must be started and then nginx.

adapt node/nginx/nginx.conf

The file will be adapted as follows.

events {
  worker_connections 1024; 
}

http {

    server {
      listen 80;
      server_name funtrails.example.com;

      # Redirect HTTP -> HTTPS
      return 301 https://$host$request_uri;
    }

   server {
    listen 443 ssl;
    server_name funtrails.example.com;

    ssl_certificate /etc/letsencrypt/live/funtrails.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/funtrails.example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://funtrails:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
  }
}

The events{} Block is a required block in NGINX configuration, even if empty. It handles connection-related events (like concurrent connections), but you don’t need to configure it unless you have advanced use cases. Here I configured 1024 concurrent connections.

Within the http block we have 2 virtual server-blocks. The first server-block define that server funtrails.example.com is listening to port 80 (HTTP) but all requests to this port will immediately be redirected to port 443 (HTTPS). The second server-block define that server funtrails.example.com is listening also to port 443 (HTTPS) followed by the location of the SSL certificate and SSL key on our local Host and the protocol definition.

The location-block define a rule for requests to / (the root URL of your site). You could add more location blocks i.e. for /api, /images, etc. if needed. In this config, we are skipping the upstream block and directly writing proxypass. The proxypass tells NGINX to forward requests to port 8080 of the backend service defined in docker-compose.yml. This backend service in docker-compose.yml is defined with the container_name directive which is set to funtrails.

...
services:
  funtrails:
	build: ./funtrails
    container_name: funtrails
	networks:
	   - funtrails-network
... 

Docker Compose creates an internal Docker network funtrails-network, and all services can reach each other by their service name(s) as hostname(s). So nginx can resolve funtrails because it’s part of the same Docker network (no need for a manual upstream block).

These other $variables come from NGINX’s core HTTP module, so we don’t need to define them. They are always available in the config.

$host

• What it is: The value of the Host header in the original HTTP request.
• Example: If the user visits http://example.com, then $host is example.com.
• Use case: Tells the backend app what domain the client used — useful for apps serving multiple domains.

$remoteaddr

• What it is: The IP address of the client making the request.
• Example: If someone from IP 203.0.113.45 visits your site, this variable is set to 203.0.113.45.
• Use case: Useful for logging, rate limiting, or geolocation in the backend app.

$proxyaddxforwardedfor

• What it is: A composite header that appends the client’s IP to the existing X-Forwarded-For header.
• Use case: Maintains a full list of proxy hops (useful if your request goes through multiple reverse proxies).
• How it works: If X-Forwarded-For is already set (by another proxy), it appends $remoteaddr to it; otherwise, it sets it to $remoteaddr.

$scheme

• What it is: The protocol used by the client to connect to NGINX — either http or https.
• Example: If the user visits https://example.com, then $scheme is https.
• Use case: Lets your backend know whether the original request was secure or not.
  

Create a cron-Job to renew Lets Encrypt SSL certificates

Lets Encrypt SSL Certificates must be renewed after 90 days. certbot can renew your certificates. To automize the renewal you can create a cronjob on your Host machine.

Note: Sometimes cron doesn’t know where docker-compose is located (because the environment variables are missing). Therefore, it’s safer to use the full paths in crontab. You check the relevant paths as follows:

which docker-compose
/usr/bin/docker-compose

which certbot
/usr/bin/certbot

The create a cronjob in crontab of the user root (use sudo):

sudo crontab -e 

0 3 * * * 	/usr/bin/certbot renew --quiet && /usr/bin/docker-compose restart nginx

With sudo crontab -e you create a crontab for the user root. All commands within the root crontab will be executed with root privileges.

certbot renew checks all certificates for expiration dates and automatically renews them.

docker-compose restart nginx ensures that nginx is reloaded so that it can apply the new certificates. Otherwise, nginx would still be using old certificates even though new ones are available. With the command you call docker-compose restart <service-name>. Here you specify the service name from docker-compose.yml not the container name.

Note: In case you would call crontab -e (without sudo) you would edit your own user crontab. This crontab then runs under your user, not as root and the tasks in crontab run unter this normal user. When you renew SSL certificates using certbot renew this job must write into the directories under /etc/letsencrypt/ on your machine. But these directories are owned by root. So you cannot write to the directories when the crontab runs under a normal user. One might then think that the commands in the crontab of a normal user should be executed with sudo. If a crontab job of your normal user (i.e. patrick) is running and sudo is used in the command, then sudo will attempt to prompt for the password. But there is no terminal in crontab where you can enter a password. Therefore, the command will fail (error in the log, nothing happens). Therefore it is essential here to edit the crontab of the user root with sudo crontab -e.

Finally you can check the crontab of your own or the crontab as root as follows.

crontab -l
no crontab for patrick

sudo crontab -l
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

0 3 * * * /usr/bin/certbot renew --quiet && /usr/bin/docker-compose restart nginx
 

You can check the logs for the renewal process using the following command.

sudo cat /var/log/letsencrypt/letsencrypt.log

Start the Containers with docker-compose

Navigate to the directory with docker-compose.yml. Then use the following commands.

sudo docker-compose build

sudo docker-compose up -d

The command docker-compose build reads the Dockerfile for each service defined in docker-compose.yml and builds the Docker image accordingly. The command docker-compose up -d run the container(s) and the network. This starts all services defined in the docker-compose.yml and links them via the defined docker network. The -d flag runs the containers in the background (detached mode).

Then you can check the status using the following commands.

sudo docker-compose ps

sudo docker ps

Here is an overview of the most important commands.

How to manage Changes made to the application code

When we make changes to the app code i.e. in node/funtrails/app.js or in node/funtrails/Dockerfile we need to rebuild the image for the funtrails service defined in node/docker-compose.yml. In such a change scenario it is not necessary to stop the containers with docker-compose down before you rebuild the image with docker-compose build.

You can rebuild and restart only the funtrails service with the following commands.

docker-compose build funtrails

docker-compose up -d funtrails

This will:

• Rebuild the funtrails image
• Stop the old funtrails container (if running)
• Start a new container using the updated image
• Without affecting other services like nginx
  

.
├── images
│   ├── 2022-05-28-1-Paterlini-Umbau-80s-Ansicht.jpeg
│   ├── 2022-05-28-2-Paterlini-Umbau-80s-Ansicht.jpeg
│   ├── 2022-05-28-3-Paterlini-Umbau-80s-Ansicht.jpeg
│   ├── 2022-05-28-4-Paterlini-Umbau-80s-Ansicht.jpeg
│   ├── 2022-07-09-Gianni-Motta-1.jpeg
│   ├── 2022-07-09-Gianni-Motta-2.jpeg
│   └── 2022-07-09-Gianni-Motta-3.jpeg
└── index.html

The index.html contain the HTML and include a navbar, a hero section, and a footer. The index.html looks like the following.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Funtrails - Nodejs Demo</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet">
</head>
<body>
    <!-- Navbar -->
    <nav class="navbar navbar-expand-lg navbar-dark bg-dark">
        <div class="container">
            <a class="navbar-brand" href="#">Funtrails - Demo</a>
            <button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav">
                <span class="navbar-toggler-icon"></span>
            </button>
            <div class="collapse navbar-collapse" id="navbarNav">
                <ul class="navbar-nav ms-auto">
                    <li class="nav-item"><a class="nav-link" href="#paterlini">Paterlini</a></li>
                    <li class="nav-item"><a class="nav-link" href="#motta">Gianni Motta</a></li>
                </ul>
            </div>
        </div>
    </nav>
    
    <!-- Hero Section -->
    <header class="bg-primary text-white text-center py-5">
        <div class="container">
            <h1>Funtrails</h1>
            <p class="lead">History of Bikes and Components</p>
        </div>
    </header>
    
    <!-- Patrlini Bike Section -->
    <section id="paterlini" class="py-5">
        <div class="container">
            <h2 class="text-center">Paterlini</h2>
            <p class="text-center">Paterlini Road Steel Bike from 1982</p>
            <div id="paterliniCarousel" class="carousel slide" data-bs-ride="carousel">
                <div class="carousel-inner">
                    <div class="carousel-item active">
                        <img src="images/2022-05-28-1-Paterlini-Umbau-80s-Ansicht.jpeg" class="d-block w-100" alt="Mountain Bike 1">
                    </div>
                    <div class="carousel-item">
                        <img src="images/2022-05-28-4-Paterlini-Umbau-80s-Ansicht.jpeg" class="d-block w-100" alt="Mountain Bike 2">
                    </div>
                    <div class="carousel-item">
                        <img src="images/2022-05-28-3-Paterlini-Umbau-80s-Ansicht.jpeg" class="d-block w-100" alt="Mountain Bike 3">
                    </div>
                </div>
                <button class="carousel-control-prev" type="button" data-bs-target="#paterliniCarousel" data-bs-slide="prev">
                    <span class="carousel-control-prev-icon" aria-hidden="true"></span>
                </button>
                <button class="carousel-control-next" type="button" data-bs-target="#paterliniCarousel" data-bs-slide="next">
                    <span class="carousel-control-next-icon" aria-hidden="true"></span>
                </button>
            </div>
        </div>
    </section>
    
    <!-- Gianni Motta Bike Section -->
    <section id="motta" class="py-5">
        <div class="container">
            <h2 class="text-center">Gianni Motta Personal 2001R</h2>
            <p class="text-center">Gianni Motta Road Steel Bike from 1989</p>
            <div id="mottaCarousel" class="carousel slide" data-bs-ride="carousel">
                <div class="carousel-inner">
                    <div class="carousel-item active">
                        <img src="images/2022-07-09-Gianni-Motta-1.jpeg" class="d-block w-100" alt="Road Bike 1">
                    </div>
                    <div class="carousel-item">
                        <img src="images/2022-07-09-Gianni-Motta-2.jpeg" class="d-block w-100" alt="Road Bike 2">
                    </div>
                    <div class="carousel-item">
                        <img src="images/2022-07-09-Gianni-Motta-3.jpeg" class="d-block w-100" alt="Road Bike 3">
                    </div>
                </div>
                <button class="carousel-control-prev" type="button" data-bs-target="#mottaCarousel" data-bs-slide="prev">
                    <span class="carousel-control-prev-icon" aria-hidden="true"></span>
                </button>
                <button class="carousel-control-next" type="button" data-bs-target="#mottaCarousel" data-bs-slide="next">
                    <span class="carousel-control-next-icon" aria-hidden="true"></span>
                </button>
            </div>
        </div>
    </section>
    
    <!-- Footer -->
    <footer class="bg-dark text-white text-center py-3">
        <p>&copy; 2025 Funtrails. All rights reserved.</p>
    </footer>
    
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js"></script>
</body>
</html>

This website is a one-page bicycle showcase with two sections: one for my Paterlini bike and one for my Gianni Motta bike, each containing a gallery and text. The image galleries in the Bike sections are Bootstrap carousels that allow users to slide through images.

Then I create a new file structure for my node app. Within the funtrails directory I create a views directory. This is the location where my HTML part will be located. Then I create a package.json file just under funtrails. I copy the images folder and the index.html file from the HTML part above into the views folder of my new structure. The new structure looks like this.

funtrails
├── package.json
└── views
    ├── images
    │   ├── 2022-05-28-1-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-2-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-3-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-4-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-07-09-Gianni-Motta-1.jpeg
    │   ├── 2022-07-09-Gianni-Motta-2.jpeg
    │   └── 2022-07-09-Gianni-Motta-3.jpeg
    └── index.html

The package.json file has the following code.

{
  "name": "funrails-demo",
  "version": "1.0.0",
  "description": "nodejs funtrails website demo",
  "author": "Patrick Rottlaender <p.rottlaendere@icloud.com>",
  "license": "MIT",
  "main": "app.js",
  "keywords": [
    "nodejs",
    "bootstrap",
    "express"
  ],
  "dependencies": {
    "express": "^4.16.4"
  }
}

For my node app I use express framework in version 4.16.4 or above. The code of my app will be in the main file app.js. Then I run npm install to install all dependencies.

npm install

This create a new folder for my node modules and a package-lock.json file.

funtrails
├── node_modules
├── package.json
├── package-lock.json
└── views
    ├── images
    │   ├── 2022-05-28-1-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-2-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-3-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-4-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-07-09-Gianni-Motta-1.jpeg
    │   ├── 2022-07-09-Gianni-Motta-2.jpeg
    │   └── 2022-07-09-Gianni-Motta-3.jpeg
    └── index.html

I create the main app.js file with the following code.

//Load the express framework
const express = require('express');

//create express app
const app = express();

//create Router instance
const router = express.Router();

//path for static content 
const path = __dirname + '/views/';

//set port 
const port = process.env.PORT || 8080;

//create Middleware
router.use(function (req,res,next) {
    console.log('/' + req.method);
    next();
  });

//create the route handlet for get
  router.get('/', function(req,res){
    res.sendFile(path + 'index.html');
  });

//app will use the static path and the router 
app.use(express.static(path));
app.use('/', router);

//app listens on localhost port 8080
app.listen(port, function () {
  console.log('nodejs funtrails demo app listening on port 8080!')
})                        

Let’s go through this Node.js code line by line.

• require(‚express‘) loads the Express module. This allows us to use express functionalities in our application.
• Calls the express() function to create an Express application. The app object will be used to define routes, middleware, and configure the server.
• Creates a new router instance using express.Router(). The router allows grouping route handlers separately from the main app.
• dirname refers to the directory of the current JavaScript file. Concatenating ‚/views/‘ results in a full path to the views directory. This will be used to serve static HTML files and images.
• process.env.PORT checks if a port number is set in the environment variables. If not set, it defaults to 8080. This allows the application to use a dynamic port in cloud environments while using 8080 locally.
• This middleware logs the HTTP request method (e.g., GET, POST). req.method returns the request type. next() passes control to the next middleware or route handler. Example logs when requests are made: /GET, /POST
• Defines a route handler for GET / (the home page). Uses res.sendFile(path + ‚index.html‘) to send index.html from the views directory.
• express.static(path): Serves static files (CSS, JS, images) from the views directory.
• app.use(‚/‘, router): Registers the router for handling requests.
• Starts the server on the specified port. Logs a message indicating that the server is running.
  

The final file structure is as follows.

funtrails
├── app.js
├── node_modules
├── package.json
├── package-lock.json
└── views
    ├── images
    │   ├── 2022-05-28-1-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-2-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-3-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-05-28-4-Paterlini-Umbau-80s-Ansicht.jpeg
    │   ├── 2022-07-09-Gianni-Motta-1.jpeg
    │   ├── 2022-07-09-Gianni-Motta-2.jpeg
    │   └── 2022-07-09-Gianni-Motta-3.jpeg
    └── index.html

Run the app with node app.js.

node app.js

Under no circumstances should you start attacks on systems located on the internet, even if the systems belong to you, because from a legal point of view it is always a risk and you can get into trouble. You should build a game environment in your local network in which you install attacker and victim system. It is important that you only use private IP addresses and that the game environment does not route any data traffic to and from the Internet. This gaming environment is then your Hacking Lab in which you can try out different attack scenarios safely and securely.

The idea is to install a virtualization software on the operating system of the host computer which is in my case an M2 Apple Mac with Ventura 13.3. The virtualization software I use is UTM. UTM create the virtual guest machines in a virtual Lan (Vlan) isolated from the host operating system and the local network. The virtual guest machines can talk to each other but can not access the local network or the internet via the host. The host can talk to the virtual guest machines. This network mode is defined as host only mode.

Note: UTM provide various network operating modes. Details can be read in the UTM documentation.

The installation of UTM can be done in two ways:

1. Payed version in the Apple App Store
2. Free download from the UTM site

I recommend to install the payed version from the App Store. You will get all updates whenever there are updates available and you support the development of this fantastic tool. At the time of writing this article the costs for the payed version of UTM were 11.99 Euro.

Once UTM has been installed we need to create the attacker and the victim as virtual guest machines in UTM. The attacker is a Kali Linux Image the victim will be a host prepared with vulnarabilities from vulnerable hub.

On vulnerable hub you find vulnerable virtual machines. These machines are prepared from cyber security enthusiasts for other security enthusiasts. These machines are available with vulnerabilities and created specifically for the purpose of hacking them.

Create a vulnerable virtual machine in UTM

To create a victim in UTM you search for a machine which best suits your needs and download the virtual machine from vulnerable hub which is available there in the .ova format. OVA is basically a tar based archive that contains among other things, an .ovf file with the specification of the virtual machine and in most cases disk image files in the format .vdi and .vmdk. Other files contained in the .ova file we don’t consider.

On vulnerable hub we search for Robot and find 3 search results. We look for Mr-Robot: 1, click on it and find the download link to download the .ova file for the vulnerable virtual machine.

To install Mr-Robot: 1 as virtual machine in UTM we first must unpack the .ova file.

patrick % ls
total 2884016
mrRobot.ova

patrick % tar -xvf mrRobot.ova 
x mrRobot.ovf
x mrRobot.mf
x mrRobot-disk1.vmdk

patrick % ls
total 2884016
mrRobot-disk1.vmdk mrRobot.mf mrRobot.ova mrRobot.ovf

Then we convert the disk image file mrRobot-disk1.vmdk into the .qcow2 format which is the supported format by UTM. If you need some more detailed information there is a very good explanation on Xmodulo.

To convert the disk image from into .qcow2 we need the utility qemu-img. I install the complete qemu emulator from homebrew as the qemu utilities are not available as single option. If you don’t know homebrew or in case homebrew is not installed on your Mac just go to my blog site digitaldocblog and read the article how to intstall and use homebrew on your Mac.

patrick %  brew install qemu

patrick % ls
total 2884016
mrRobot-disk1.vmdk mrRobot.mf mrRobot.ova mrRobot.ovf

patrick % qemu-img convert -O \
qcow2 mrRobot-disk1.vmdk mrRobot.qcow2

patrick % ls
total 2884016
mrRobot-disk1.vmdk mrRobot.mf mrRobot.ova mrRobot.ovf
mrRobot.qcow2

Then you start UTM from the main menue and click create new virtual machine.

Then you click on Emulate. You click on Emulate because the virtual machine you are going to install is compiled for the x86 64 Intel CPU Architechture. This Intel CPU must be emulated by UTM to provide the basis that your host can run as guest on your Apple ARM Silicon Mac (M1 and M2). In the next chapter when we install the attacker Linux we choose a virtual machine that is compiled for the Apple ARM 64 Silicon architechture. Then we click Virtualize because this guest machine is compiled for your host architecture (M1 or M2).

After Emulate you click on Custom. Then you skip ISO boot, accept in the following all default values ​​and assign a name fitting your needs for your new virtual host. At the end, the configuration is completed with safe.

In the left panel you see the virtual machine you just created. Mark it and click on the settings menue at the top right side.

In the left panel of the settings go to the Drives section and delete the IDE drive. Stay in the Drives section and create a new drive. Here you click on Import and select the .qcow2 file on your hard disk (the one you created as you converted the .vmdk file). After this step go to the QEMU section where you unselect UEFI Boot. To set the correct network settings go to Network section and select Host only. Finally you click on safe and end the configuration.

Create the attacker virtual machine in UTM

To launch attacks against a vulnerable virtual host I recommend installing a standard Kali Linux machine in UTM. Most tools are already pre-installed here and you can start immediately.

We go on the Kali.org website an there into the section get Kali. I choose installer images and then we select the ARM Silicon (ARM64) for download the Kali especially compiled for our Apple ARM Silicon M2 architecture (if you have a M1 this is also working fine). Click on the recommended installer and download the iso image to your hard drive.

After you have downloaded the iso image the installation will be performed in 2 steps. First you create a new virtual machine in virtualization mode and then in step 2 you install the Kali Linux on it.

Click on the + sign in the top menue and then select Virtualize.

After that you can select the OS you want to install. Here you click on Linux and here you select the iso image you downloaded from Kali in the Boot-Iso-Image section.

Then you click continue and keep the default value for the RAM but set the CPU cores to 4.

In the following you can keep all default values i.e. for storage and always click on continue until you come to the Summary section. Here you can give you new virtual machine a name according to your needs. Finally you click on safe.

Then click on the settings to open the settings menue. Go to the Devices section and click on new and add a new serial device emulation. This is very important for the installation of Kali Linux on your virtual machine otherwise the installer will not start.

Run the virtual machine to start the installation. Here you see that 2 windows will be opened. Choose the window that says terminal one and start the installation. Don’t choose Graphical installation. Then follow the installation steps. You can see a very good video on YouTube that guide you through the standard installation.

Once you see the finish installation screen then use the navigation bar of this window terminal 1 and shut down the virtual machine. Close the second window that has been opened when you started with the installation.

Then you go to the main window, select the virtual machine you just installed, go to the main window and unmount the iso at the bottom of the page.

Open the settings and go to the device sections and remove the serial device you created before. To set the correct network settings go to Network section and select Host only. Finally you click on safe and end the configuration.

Check the network configuration

Run both virtual machines in UTM, Robot1VulnerableHost and the Attacker machine KaliLinux. When you run Robot1VulnerableHost you see the logon screen after the machine has beefed. Of course you don’t have login credentials because you want to hack into it instead of logging in. This is different on your KaliLinux. Here you created an account during the installation.

Log into KaliLinux and open a Terminal. Check the ip address from KaliLinux. The ip address is 192.168.128.5.

Then discover the network 192.168.128.0/24 to find other hosts in the same subnet. Type the following command.

kaliLinux % netdiscover -r 192.168.128.0/24

You see 2 hosts in your subnet 192.168.128.1 which is the bridge between your local host network and your vlan and you see 192.168.128.6 which is Robot1VulnerableHost on the same subnet.

You can ping Robot1VulnerableHost but you cannot ping the host computer which is my Mac on subnet 192.168.0.0/24 with the ip address 192.168.0.38. So the bridge does not route traffic from your vlan into the local network which is what we need. The guests can talk to each other but they can not talk to the outside world.

Then I ping the virtual guests from my Mac. Here you see the host can talk to them. So the network setup of the Hacking Lab is correct.

patrick % ping 192.168.128.5
PING 192.168.128.5 (192.168.128.5): 56 data bytes
64 bytes from 192.168.128.5: icmp_seq=0 ttl=64 time=0.979 ms
64 bytes from 192.168.128.5: icmp_seq=1 ttl=64 time=0.568 ms
64 bytes from 192.168.128.5: icmp_seq=2 ttl=64 time=0.811 ms
64 bytes from 192.168.128.5: icmp_seq=3 ttl=64 time=0.718 ms
^C

patrick % ping 192.168.128.6
PING 192.168.128.6 (192.168.128.6): 56 data bytes
64 bytes from 192.168.128.6: icmp_seq=0 ttl=64 time=2.663 ms
64 bytes from 192.168.128.6: icmp_seq=1 ttl=64 time=2.298 ms
64 bytes from 192.168.128.6: icmp_seq=2 ttl=64 time=2.330 ms
64 bytes from 192.168.128.6: icmp_seq=3 ttl=64 time=2.310 ms
64 bytes from 192.168.128.6: icmp_seq=4 ttl=64 time=2.318 ms
^C

patrick %

Installation

So lets install nginx with homebrew. In case you dont know what homebrew is and need to install it on your Mac read my article on Digitaldocblog.

brew install nginx

brew services start nginx

brew services stop nginx

• nginx has been installed in /usr/local/Cellar/nginx/1.17.7
• nginx configuration file is /usr/local/etc/nginx/nginx.conf
  

Configuration

Setup the core nginx configuration file nginx.conf in your environment:

# /usr/local/etc/nginx/nginx.conf 

# Main Context
worker_processes  auto;
error_log  /usr/local/etc/nginx/logs/error.log;
pid /usr/local/etc/nginx/run/nginx.pid;  

# Events Context
events {
    worker_connections  1024;
}

# HTTP Context
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    access_log  /usr/local/etc/nginx/logs/access.log;

    include servers/*;
}

The Main Context is the most general context and is the only context that is not surrounded by curly braces. The Main Context is placed at the beginning of the core nginx configuration file. The directives of the Main Context cannot be inherited in any other context and can not be overridden.

The Main Context is used to configure the basic details of an application. Some common details that are configured in the Main Context are:

• The directive worker_processes defines the number of worker processes. The number depend on the number of CPU cores and the default value is 1. The value auto will autodetect the number of worker processes. The directive tell the virtual servers defined in the servers folder (see below) know how many workers can be created once they are bound to the IP(s) and port(s). It is common practice to run 1 worker process per core. Anything above this won’t hurt your system, but it will leave idle processes usually just lying about
• The directive error_log define the default error file for the entire application
• The directive pid define the file to save the main process ID.
  

The Event Context as also the HTTP-Context are childs of the Main Context and therefore lower contexts. Lower Contexts handle the request, and the directives at this level and control the defined defaults for every virtual server.

The Events Context define global options for connection processing and is contained within the main context. There can be only one Event Context defined within Nginx configuration. Within the Events Context we define the worker_connections to tells our worker processes how many connections can simultaneously be served by Nginx. The default value is 768. Considering that every browser usually opens up at least 2 connections per server, this is why we need to adjust our worker connections to its full potential. Therefore we define 1024 worker connections.

The HTTP Context is used to hold the directives for handling HTTP or HTTPS traffic.

• with the include directive we includes another file, or files matching the specified mask, into configuration.
• with the default_type directive we define the default MIME type of a response. When a web server sends HTTP traffic back to the client (response), it usually adds an IANA media type (formerly known as MIME types), or content type, to the packet header that shows what kind of content is in the packet. The HTTP header on the data stream contains this content type. It is added by the web server before the data is sent.
• using the directive sendfile enables or disables the use of the function sendfile(). By default, NGINX handles file transmission itself and copies the file into the buffer before sending it. Enabling the sendfile directive eliminates the step of copying the data into the buffer and enables direct file transmission.
• the directive keepalive_timeout sets a timeout during which a keep-alive client connection will stay open on the server side.
  

Finally the include directive within the HTTP Context tell nginx to load all virtual servers from the files in /usr/local/etc/nginx/servers in the HTTP Context.

Then I create 2 serverfiles in /usr/local/etc/nginx/servers.

1. A serverfile to run a virtual http server on localhost port 7445
2. A serverfile to run a virtual http reverse proxy server on localhost port 7444
   

Patricks-Macbook Pro:servers patrick$ ls -l

total 24
-rw-r--r--@ 1 patrick  admin  188 15 Nov 06:12 patrick_7445
-rw-r--r--@ 1 patrick  admin  452 15 Nov 07:39 reverse_7444

Patricks-Macbook Pro:servers patrick$ 

The Server Context is inside these serverfiles. In general the Server Context is defined within the HTTP Context.

The Server Context define the virtual host settings. There can be multiple Server Context definitions inside the HTTP Context. The directives inside the Server Context handle the processing of requests for resources associated with a particular domain or IP address.

The directives in this Server Context can override many of the directives that may be defined in the HTTP Context, including the document root, logging, compression, etc. In addition to the directives that are taken from the HTTP Context, we can also configure files to try to respond to requests, issue redirects, and rewrites, and set arbitrary variables.

# /usr/local/etc/nginx/servers/reverse_7444

server {

    listen 7444;
    server_name patrick.local;

    location / {
       proxy_pass http://localhost:7445;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}

To connect to your localhost using patrick.local the dns settings in etc/host must be changed accordingly.

# /private/etc/hosts

192.168.178.20 servtest.rottlaender.lan

# /usr/local/etc/nginx/servers/patrick_7445

server {
     listen       7445;
     server_name  localhost;

     location / {
         root   /Users/patrick/Sites/www/nginx/patrick_7445;
         index  index.html index.htm;
    }
}

TLS/SSL for nginx

SSL/TLS works by using the combination of a public certificate and a private key.

The SSL key (private key) is kept secret on the server. It is used to encrypt content sent to clients.

The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

create private key

Patricks-MBP:express-security patrick$ cd /usr/local/etc/nginx

Patricks-MBP:nginx patrick$ ls -l
total 144
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  8 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  8 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types.default
-rw-r--r--  1 patrick  admin  3106 15 Mai 11:19 nginx.conf
-rw-r--r--  1 patrick  admin  2680  8 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params.default
drwxr-xr-x  3 patrick  admin    96 21 Jan 06:02 servers
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  8 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ mkdir ssl

Patricks-MBP:nginx patrick$ ls -l
total 152
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  8 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  8 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types.default
-rw-r--r--@ 1 patrick  admin   373 18 Mai 13:38 nginx.conf
-rw-r--r--  1 patrick  admin  2680  8 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--@ 1 patrick  admin  1390 17 Mai 09:19 nginx_old.conf
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params.default
drwxr-xr-x  5 patrick  admin   160 18 Mai 13:20 servers
drwxr-xr-x  4 patrick  admin   128 16 Mai 07:41 ssl
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  8 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ cd ssl

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl genrsa -out privateKey.pem 4096

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem

create certificate signing request (CSR)

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl req -new -key privateKey.pem -out csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  1740 16 Mai 07:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem

In case I would like to request an official certificate I must send this csr to the certificate authority. This authority would then create an authority signed certificate from the CSR and send it back to me.

This step is done by ourselves and this is the reason why we create a self signed certificate. This self signed certificate is not an official certificate and not trusted by any browser. It is not useful to use a self signed certificate in production because it produces error messages in the browsers. But for local development a self signed certificate is ok.

So create the self signed certificate. The csr file can then be removed.

create the self signed certificate

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in csr.pem -out selfsignedcertificate.pem -req -signkey privateKey.pem -days 365

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  1740 16 Mai 07:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 07:39 selfsignedcertificate.pem

Patricks-MBP:ssl patrick$ rm csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 07:39 selfsignedcertificate.pem
 

show certificate details

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in selfsignedcertificate.pem -text -noout

renew self signed certificate

So let’s assume 1 year has passed by and now we want to renew this certificate selfsignedcertificate.pem. Now you may either generate a new CSR or export the CSR from the existing expired self-signed certificate.

First let’s check the validity of the existing certificate.

patrick@MacBookPro-Patrick ssl % ls -l                                                          
total 32
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % openssl x509 -noout -text -in selfsignedcertificate.pem | grep -i -A2 validity
        Validity
            Not Before: May 16 05:39:28 2020 GMT
            Not After : May 16 05:39:28 2021 GMT

patrick@MacBookPro-Patrick ssl %

So our certificate selfsignedcertificate.pem expired on 16 Mai 2021 and today is 12 Mar 2022.

It is recommended to export the CSR as you don’t have to worry about giving the same Distinguished Name information. Although this information can be collected from the certificate itself. For us, we will export the CSR from the certificate itself.

patrick@MacBookPro-Patrick ssl % ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % openssl x509 -x509toreq -in selfsignedcertificate.pem  -signkey privateKey.pem -out new-csr.pem          
Getting request Private Key
Generating certificate request

patrick@MacBookPro-Patrick ssl % ls -l
total 32
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl %

Now that we have our private key privateKey.pem and the new CSR new-csr.pem, we can renew our certificate selfsignedcertificate.pem. You should not get confused with the term „renew“, we are not extending the expiry of the certificate, instead we are just creating a new self signed certificate using the CSR new-csr.pem from the existing private key privateKey.pem. The Signature Modules are the same and our application considers the certificate as same instead of new.

patrick@MacBookPro-Patrick ssl % ls -l
total 32
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % openssl x509 -req -days 365 -in new-csr.pem -signkey privateKey.pem -out new-selfsignedcertificate.pem
Signature ok
subject=/C=DE/ST=Bayern/L=Munich/O=Digitaldocblog/CN=localhost/emailAddress=p.rottlaender@icloud.com
Getting Private key

patrick@MacBookPro-Patrick ssl % ls -l
total 40
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  1980 12 Mär 08:30 new-selfsignedcertificate.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % 

Lets check the validity of the new certificate new-selfsignedcertificate.pem.

patrick@MacBookPro-Patrick ssl % openssl x509 -noout -text -in selfsignedcertificate.pem | grep -i -A2 validity
        Validity
            Not Before: May 16 05:39:28 2020 GMT
            Not After : May 16 05:39:28 2021 GMT

patrick@MacBookPro-Patrick ssl % openssl x509 -noout -text -in new-selfsignedcertificate.pem | grep -i -A2 validity
        Validity
            Not Before: Mar 12 07:30:52 2022 GMT
            Not After : Mar 12 07:30:52 2023 GMT

patrick@MacBookPro-Patrick ssl % 

While our certificate selfsignedcertificate.pem expired on 16 Mai 2021 the new certificate new-selfsignedcertificate.pem expires on 12 Mar 2023.

The we rename the new-selfsignedcertificate.pem to selfsignedcertificate.pem to ensure

patrick@MacBookPro-Patrick ssl % mv selfsignedcertificate.pem old-selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % mv new-selfsignedcertificate.pem selfsignedcertificate.pem
patrick@MacBookPro-Patrick ssl % ls -l
total 40
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 old-selfsignedcertificate.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 12 Mär 08:30 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl %

Finally we restart nginx service.

patrick@MacBookPro-Patrick ssl % brew services list                                        
Name                  Status  User    File
mongodb-community@4.4 started patrick ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community@4.4.plist
nginx                 started patrick ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
patrick@MacBookPro-Patrick ssl % brew services stop nginx
Stopping `nginx`... (might take a while)
==> Successfully stopped `nginx` (label: homebrew.mxcl.nginx)
patrick@MacBookPro-Patrick ssl % brew services start nginx
==> Successfully started `nginx` (label: homebrew.mxcl.nginx)
patrick@MacBookPro-Patrick ssl % brew services list       
Name                  Status  User    File
mongodb-community@4.4 started patrick ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community@4.4.plist
nginx                 started patrick ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
patrick@MacBookPro-Patrick ssl %

configure nginx with SSL

Any default configurations are done in nginx.conf. In this configuration we define a local webserver and a reverse proxy server.

The ip 127.0.0.1 is resolved in hosts file to localhost.

In the default configuration we define that any request to localhost on port 80 is redirected to https/ssl (443 ssl). This is a typical production configuration to enforce htps/ssl.

// /usr/local/etc/nginx.conf
// default configuration

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    
	# default server configuration
    server {
      	listen 80 default_server;
    	listen [::]:80 default_server;
    	
    	# default server your.domain.com
    	server_name localhost;
    	
    	# enforce 301 redirect to ssl port on your default server
    	return 301 https://$host$request_uri;
    }
    
	# In servers directory we define the servers running on this machine
    include servers/*;
}

The local webserver is bound to 127.0.0.1 and this ip is resolved to localhost. localhost is listening on port 443. This local webserver is using https/ssl. Any ssl configuration is done via the ssl_* directives in the server block.

In the location block we define the root directory of the local webserver and the index files. Here the local webserver process https client requests and send responses back to the client. So when a client request the route / then the local webserver render index.htm or index.html and respond to the client.

// /usr/local/etc/servers/localweb

server {
     listen       443 ssl;
     # here you would define in production your.domain.com
     server_name  localhost;

         ssl_certificate      ssl/selfsignedcertificate.pem;
         ssl_certificate_key  ssl/privateKey.pem;
         ssl_session_cache    shared:SSL:1m;
         ssl_session_timeout  5m;
         ssl_ciphers  HIGH:!aNULL:!MD5;
         ssl_prefer_server_ciphers  on;

     location / {
         root   /Users/patrick/Sites/prod/digitaldocblog-V2/public;
         index  index.html index.htm;
    }
}

The reverse proxy server is bound to ip 192.168.178.20. The ip 192.168.178.20 is resolved to servtest.rottlaender.lan in hosts file. This reverse proxy server is using https/ssl. Any ssl configuration is done via the ssl_* directives in the server block.

The node express server is listening on localhost port 3300. This express server is using http.

Any request to 192.168.178.20 or servtest.rottlaender.lan to port 3000 will be passed to application server listening on localhost port 3300.

So any client https request for servtest.rottlaender.lan on route / will be directly handed over to the express server. The express server will then process the request and send the response back to servtest.rottlaender.lan which will then respond 1:1 to the client.

// /usr/local/etc/servers/reverse

server {
    listen      3000 ssl;
    
	# /private/etc/hosts
	# 192.168.178.20	servtest.rottlaender.lan
	
    server_name servtest.rottlaender.lan;

    ssl_certificate      ssl/selfsignedcertificate.pem;
    ssl_certificate_key  ssl/privateKey.pem;
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
       proxy_pass http://localhost:3300;
    }
}

In Classes functions are called methods. The init-method __init__() in a Class is called whenever a Class is called to create an Object instance using Class(). The init-method define attributes, variables and set the Object instance to a defined initial state. Therefore the init-Method is called constructor.

The Parameter self

Let’s imagine a Class Car that has attributes like brand, color and horsepower. We define the init-method using def __init__(self) and define all attributes using self.<attribut> = <value> in the init-method. A peculiarity of Python is that the init-method has at least one parameter self which is always passed by default.

The parameter self ensure that each attribute and its values will be linked to the correct Object instance stored in a certain memory address space.

 class Car:
	def __init__(self):
		self.brand = None
        self.color = None
        self.horsepower = None
	
car1 = Car()
print(car1)

#Print-Output
#<__main__.Car object at 0x10e0ffc40>

When an Object instance car1 of the Class Car will be created using car1 = Car() the init-method within the Class-Definition will be called and the self parameter will be passed to the new Object instance car1. If we print the Object instance car1 on the console using print(car1) we don’t see a list of attributes or something similar, but we see the specific location in memory where the Object instance car1 has been stored. This memory address is stored in the self parameter.

To show the peculiarity of Python with the self parameter more clearly, please take a look at the following notation.

class Car:
    def __init__(self):
        self.brand = None
        self.color = None
        self.horsepower = None
        self.self = self #reference is stored in self


car1 = Car()
print(car1)
print(car1.self)
print(car1.brand)

#Print-Output
#<__main__.Car object at 0x10e0ffc40>
#<__main__.Car object at 0x10e0ffc40>
#None

Within the init-method the value of the self parameter is assigned to the attribute self.self using self.self = self. The output for print(car1.brand) is None as expected. print(car1) and print(car1.self) show exactly the same print output. The memory address ist stored in the parameter self and the Object instance car1 is a reference to this memory address.

The dot notation

We can assign values to each attribute, access each value and print the value to the console using the dot notation print(<object-instance>.<attribute-name>).

class Car:
	def __init__(self):
		self.brand = None
        self.color = None
        self.horsepower = None
        
car1 = Car()
car1.brand = "BMW"
car1.color = "red"
car1.horsepower = 120

car2 = Car()
car2.brand = "Audi"
car2.color = "blue"
car2.horsepower = 150

print(car1.brand)
print(car2.brand)

#Print-Output
#BMW
#Audi

We create the Object car1 as an instance of the Class Car. car1 is a BMW, is red, has 120 PS. And we create another Object car2. car2 is an Audi, is blue, has 150 PS. car1 and car2 are Object Instances of the Class Car. It is also often said that car1 and car2 are Objects of type Car. We access the individual attributes of the objects with the dot notation and get a print output of the brand using print(car1.brand) and print(car2.brand).

Working with Object instances

To make the code clearer the attribute values ​​can also be passed as parameters directly during object instantiation.

class Car:
	def __init__(self, brand, color, horsepower):
		self.brand = brand
        self.color = color
        self.horsepower = horsepower
        
car1 = Car("BMW", "red", 120)
car2 = Car("Audi", "blue", 150)

print(car1.brand)
print(car2.brand)

#Print-Output
#BMW
#Audi

Therefore the init-method must have beside the parameter self also the parameters brand, color and horsepower. When we create the Object instances car1 and car2 of the Class Car we pass the parameters in the brackets during the call of the Class Car .

As mentioned above, functions in Classes are called methods. We have already got to know one method, the init method __init__() that will be executed when an Object instance will be created.

Classes can have also methods designed by the developer himself as well. In our example we add the drive-Method to the Class Car as follows.

class Car:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.x = 0
        self.y = 0

    def drive(self):
        self.x += 10
        self.y += 15
		print("X and Y have been increased by 10 and 15")

car1 = Car("BMW", "red", 120)

print(car1.x)
print(car1.y)

car1.drive()

print(car1.x)
print(car1.y)

#Print-Output
#0
#0
#X and Y have been increased by 10 and 15
#10
#15	

Here we add 2 additional attributes to the constructor self.x and self.y and set both attributes initially to the value 0. The drive-method drive() is defined to increase the values self.x by 10 and self.y by 15 and to print a string to the console. In the above example you can see that the values of self.x and self.y are 0 when we print them to the console directly after the Object car1 has been created. Then we call the drive-method on the Object instance car1 using car1.drive() and self.x and self.y have been increase by 10 or 15 plus the string has been printed to the console.

Inheritance

There can be several classes in programs, including classes that are very similar and only differ in details. This is where the concept of inheritance comes into play. We want to write a program in which there are 3 classes of vehicles: there are cars, trucks and motorcycles.

We assume all vehicles have the attributes brand, color, and horsepower, but cars should have two drive wheels, trucks four, and motorcycles should have only one drive wheel. In addition, the vehicles have different drive-methods because they can move at different speeds.

class Car:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.drivewheels = 2
        self.x = 0
        self.y = 0

    def drive(self):
        self.x += 10
        self.y += 15
        print("Car accelerates by 10 and 15")

class Truck:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.drivewheels = 4
        self.x = 0
        self.y = 0

    def drive(self):
        self.x += 5
        self.y += 10
        print("Truck accelerates by 5 and 10")

class Motorcycle:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.drivewheels = 1
        self.x = 0
        self.y = 0

    def drive(self):
        self.x += 30
        self.y += 40
        print("Motorcycle accelerates by 30 and 40")

car1 = Car("BMW", "blue", 120)
truck1 = Truck("Volvo", "black", 270)
moto1 = Motorcycle("Yamaha", "green", 70)

print(car1.brand, car1.x, car1.y)
car1.drive()
print(car1.brand, car1.x, car1.y)

print(truck1.brand, truck1.y, truck1.y)
truck1.drive()
print(truck1.brand, truck1.y, truck1.y)

print(moto1.brand, moto1.x, moto1.y)
moto1.drive()
print(moto1.brand, moto1.x, moto1.y)

#Print-Output
#BMW 0 0
#Car accelerates by 10 and 15
#BMW 10 15
#Volvo 0 0
#Truck accelerates by 5 and 10
#Volvo 10 10
#Yamaha 0 0
#Motorcycle accelerates by 30 and 40
#Yamaha 30 40

We create 3 Object instances from Car, Truck and Motorcycle and have access to their attributes and to their individual drive-methods. Everything is working fine but most of the code is redundant.

Instead of duplicating the code in each Class, we can define a parent Class from which all Child classes inherit their properties. The parent Class in our example will be the Class Vehicle.

class Vehicle:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.x = 0
        self.y = 0

	def inheritedmethod(self):
		print("this method was inherited")

class Car(Vehicle):
    drivewheels = 2

    def drive(self):
        self.x += 10
        self.y += 15
        print("Car accelerates by 10 and 15")

class Truck(Vehicle):
    drivewheels = 4

    def drive(self):
        self.x += 5
        self.y += 10
        print("Truck accelerates by 5 and 10")

class Motorcycle(Vehicle):
    drivewheels = 1

    def drive(self):
        self.x += 30
        self.y += 40
        print("Motorcycle accelerates by 30 and 40")


car1 = Car("BMW", "blue", 120)
truck1 = Truck("Volvo", "black", 270)
moto1 = Motorcycle("Yamaha", "green", 70)

print(car1.brand, car1.x, car1.y, car1.drivewheels)
car1.drive()
print(car1.brand, car1.x, car1.y, car1.drivewheels)

print(truck1.brand, truck1.y, truck1.y, truck1.drivewheels)
truck1.drive()
print(truck1.brand, truck1.y, truck1.y, truck1.drivewheels)

print(moto1.brand, moto1.x, moto1.y, moto1.drivewheels)
moto1.drive()
print(moto1.brand, moto1.x, moto1.y, moto1.drivewheels)
moto1.inheritedmethod()

#Print-Output
#BMW 0 0 2
#Car accelerates by 10 and 15
#BMW 10 15 2
#Volvo 0 0 4
#Truck accelerates by 5 and 10
#Volvo 10 10 4
#Yamaha 0 0 1
#Motorcycle accelerates by 30 and 40
#Yamaha 30 40 1
#this method was inherited

The constructor __init__() of the parent-Class Vehicle contain the initial setup that is inherited to the child-Classes. The parent-Class Vehicle also inherit the method inheritedmethod() to each child-Class. Each Object instance created from a child-Class will have this initial setup. The child-Classes itself have no __init__() constructor in these examples. Each Object instance such as car1, truck1 and moto1 created from their child-Classes Car, Truck and Motorcycle can access all inherited properties from their common parent-Class Vehicle and their individual Class attributes drivewheels and the individual methods drive().

note: I will explain Class-attributes later in the text

Inherited methods from the parent-Class can be overwritten in the child-Class.

class Vehicle:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.x = 0
        self.y = 0

    def inheritedmethod(self):
        print("this method was inherited")

class Car(Vehicle):
    drivewheels = 2

    def drive(self):
        self.x += 10
        self.y += 15
        print("Car accelerates by 10 and 15")

    def inheritedmethod(self):
        print("this method was inherited and! overwritten")

car1 = Car("BMW", "blue", 120)

print(car1.brand, car1.x, car1.y, car1.drivewheels)
car1.drive()
print(car1.brand, car1.x, car1.y, car1.drivewheels)
car1.inheritedmethod()

#Print-Output
#BMW 0 0 2
#Car accelerates by 10 and 15
#BMW 10 15 2
#this method was inherited and! overwritten

In order to overwrite an inherited method, the same method head is simply written down again in the child-Class and the code of the method body is then adapted accordingly.

The super() method

A reference to the parent-Class can be established using the super() method. This allows us to take methods from the parent-Class and add additional properties in the code of the method in the child-Class.

class Vehicle:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower
        self.x = 0
        self.y = 0

    def inheritedmethod(self):
        print("this method was inherited")

class Car(Vehicle):
    def __init__(self, brand, color, horsepower, drivewheels):
        super().__init__(brand, color, horsepower)
        self.drivewheels = drivewheels

    def drive(self):
        self.x += 10
        self.y += 15
        print("Car accelerates by 10 and 15")

    def inheritedmethod(self):
        super().inheritedmethod()
        print("this method was inherited and! overwritten")

class Truck(Vehicle):
    def __init__(self, brand, color, horsepower, drivewheels):
        super().__init__(brand, color, horsepower)
        self.drivewheels = drivewheels

    def drive(self):
        self.x += 5
        self.y += 10
        print("Truck accelerates by 5 and 10")
        
car1 = Car("BMW", "blue", 120, 2)
truck1 = Truck("Magirus Deuz", "grey", 177, 4)

print(car1.brand, car1.x, car1.y, car1.drivewheels)
car1.drive()
print(car1.brand, car1.x, car1.y, car1.drivewheels)
car1.inheritedmethod()

print(truck1.brand, truck1.y, truck1.y, truck1.drivewheels)
truck1.drive()
print(truck1.brand, truck1.y, truck1.y, truck1.drivewheels)
truck1.inheritedmethod()

#Print-Output
#BMW 0 0 2
#Car accelerates by 10 and 15
#BMW 10 15 2
#this method was inherited
#this method was inherited and! overwritten
#Magirus Deuz 0 0 4
#Truck accelerates by 5 and 10
#Magirus Deuz 10 10 4
#this method was inherited

We had no __init__() method so far in our child-Classes but we defined for each child-Class the specific Class attribute drivewheels. This approach works fine. To understand how the super() method works we code for each child-Class a specific __init__() method and provide all parameters plus the parameter drivewheels. This means the value for drivewheels will be passed when the Object-instance from the child-Class is created and we have more flexibility there. Finally, it is possible that cars for example can have 4 drive wheels instead of 2. Then we use the super() method to refer to the __init__() method in the parent-Class and pass the properties required there and create the additional property self.drivewheels = drivewheels . In the child-Class Car we also use the super() method in the method inheritedmethod and add additional functionality (simple print command). We don’t do this in the child-Class Truck. Then we create the Object instances car1 and truck1 and access the properties and methods.

Special attributes

The attributes used so far are all 100 percent usable outside of the respective class definition. In the code below I have simplified the classes considerably to focus on the essentials.

class Vehicle:
    def __init__(self, brand, color, horsepower):
        self.brand = brand
        self.color = color
        self.horsepower = horsepower

class Car(Vehicle):
    def __init__(self, brand, color, horsepower, drivewheels):
        super().__init__(brand, color, horsepower)
        self.drivewheels = drivewheels

car1 = Car("BMW", "blue", 120, 2)

print(car1.brand)
car1.brand = "Audi"
print(car1.brand)

#Print-Output
#BMW
#Audi

We access the attribute brand and print the value BMW to the console. Then we change the value from BMW to Audi using the dot notation and overwrite the existing value.

Sometimes developers of a Class want to signal developers that attributes should only be used inside the Class and not outside. Developers can mark attributes with one underscore _ or with two underscores __ to show that attributes should not or can not be easily overridden.

class Vehicle:
    def __init__(self, brand, color, vmax):
        self._brand = brand
        self.__color = color
        self.vmax = vmax

class Car(Vehicle):
    def __init__(self, brand, color, vmax, drivewheels):
        super().__init__(brand, color, vmax)
        self.drivewheels = drivewheels

car1 = Car("BMW", "blue", 100, 2)

print(car1._brand)
car1._brand = "Audi"
print(car1._brand)

print(car1._Vehicle__color)
car1._Vehicle__color = "black"
print(car1._Vehicle__color)

#Print-Output
#BMW
#Audi
#blue
#black

In the example above we have the attribute _brand with one underscore. This shows the developer that _brand is a special attribute which is defined in the parent-Class Vehicle and inherited to the child-Class Car and should not be overwritten. However, overwriting is possible without any problems using the dot notation. The attribute __color is also defined in parent-Class Vehicle and inherited to child-Class Car but can not be overwritten using the standard dot notation. The value of the attribute __color can only be accessed using a different attribute name _Vehicle__color. Obviously the name of the attribute __color can only be accessed if we extend the attribute name at the beginning with <_parent-Classname><__attribute-name>. This is called Name Mangling.

Multiple inheritance

So far we had a parent-Class and single inheritance of properties to a child-Class. But it is also possible to define more parent-Classes and inherit their properties to a child-Class. This is called multiple inheritance. With multiple inheritance we can create complex data structures. In the following code I define 3 parent-Classes Person, Organisation, Building and 2 child-Classes Employee and Location.

class Person:
    def __init__(self, userid, name, lastname, age):
        self.userid = userid
        self.name = name
        self.lastname = lastname
        self.age = age

class Organisation:
    def __init__(self, company, orgunit):
        self.company = company
        self.orgunit = orgunit

class Building:
    def __init__(self, country, postalcode, street, city):
        self.country = country
        self.postalcode = postalcode
        self.street = street
        self.city = city

class Employee(Person, Organisation):
    def __init__(self, userid, name, lastname, age, company, orgunit):
        Person.__init__(self, userid, name, lastname, age)
        Organisation.__init__(self,  company, orgunit)

class Location(Building, Organisation):
    def __init__(self, country, postalcode, street, city, company, orgunit):
        Building.__init__(self, country, postalcode, street, city)
        Organisation.__init__(self, company, orgunit)


employee1 = Employee("12345", "Parick", "Rottländer", 54, "Company 1", "OE123456")
employee2 = Employee("67890", "Carol", "Meier", 32, "Company 2", "OE678901")
location1 = Location("Germany", "81234", "First Street 20", "Munich", "Company 1", "OE123456")
location2 = Location("Italy", "I23345", "Second Street 100", "Verona", "Company 2", "OE678901")

print(employee1.name, employee1.orgunit)
print(employee2.name, employee2.orgunit)
print(location1.country, location1.city)
print(location2.country, location2.city)

#Print-Output
#Parick OE123456
#Carol OE678901
#Germany Munich
#Italy Verona

In the code example above we define the child-Class Employee with properties from the parent-Classes Person and Organisation. Child-Class Building is defined with properties from parent-Classes Location and Organisation. Therefore the child-Class Employee inherits properties from parent-Classes Person and Organization and the child-Class Location inherit properties from parent-Classes Building and Organisation. In each child-Class the first __init__() constructor define the complete setup of an Object instance. Then each inherited parent-Class __init()__ constructor is overwritten using the dot notation <parent-Class>.__init()__. Then we create the object instances employee1, employee2 and location1 and location2 passing all required attributes and have access to each of those attributes using the dot notation.

Instance- and Class-attributes

If attributes of a Class are defined within the __init()__ method, these are Instance-attributes and are available as initial setup in each Object instance created from that Class. If we define attributes in a Class before the __init()__ method then these are Class-attributes.

class Person:
    counter = 0
    def __init__(self, name, lastname, age):
        self.userid = "P" +"-" +str(Person.counter)
        self.name = name
        self.lastname = lastname
        self.age = age
        Person.counter += 1

class Organisation:
    def __init__(self, company, orgunit):
        self.company = company
        self.orgunit = orgunit

class Employee(Person, Organisation):
    def __init__(self, name, lastname, age, company, orgunit):
        Person.__init__(self, name, lastname, age)
        Organisation.__init__(self,  company, orgunit)

print(Person.counter)
employee1 = Employee("Parick", "Rottländer", 54, "Company 1", "OE123456")
print(employee1.userid)
print(Person.counter)
employee2 = Employee("Carol", "Meier", 32, "Company 2", "OE678901")
print(employee2.userid)
print(Person.counter)

#Print-Output
#0
#P-0
#1
#P-1
#2

In the Class Person I define the Class-attribute counter and set its value to 0. Within the __init__() method of the Class Person I access the Class-attribute counter using str(Person.counter) to create a unique userid. Whenever an Object instance of Person is created Person.counter increase by 1 (see the respective print-output).

Modules

In the examples above we write the code in one Python file with the ending <filename>.py. Each Python file is a Module itself and can be executed independently. But a developer must not code any method on his own he or she can use existing code from third parties by loading other modules into his Python file. After modules have been loaded methods and properties defined in those modules can be user in the code.

Python has numerous built-in modules. All of them can be used for free by developers in the code. In the following examples I will use the build-in module math because math contain the parameter pi that we need for the calculation of the area of a circle for a given radius.

#objectMain.py

import math

class Circle:
    def __init__(self, radius):
        self.radius = radius

    #calculate area from given radius
    def area(self):
        return math.pi * self.radius ** 2

c1 = Circle(5)
print(c1.area())

#Print-Output
#78.53981633974483

In the file objectMain.py we first import math and define the Class Circle. Then we pass the parameter radius in the __init()__ constructor. The instance attribute self.radius receive the given value radius when an Object instance is created with a given radius value. The area()method use the parameter pi from the math Module and return the calculated area for the given radius.

But we can also develop methods on our own separately in separate Python files and make them available as modules. We can then either import the entire module or just individual methods from the module. I define a method radius() in a separate module objectModuleCalc.py to calculate the radius from a given area.

#objectModuleCalc.py

import math

#calculate radius from given area
def radius(area):
    return math.sqrt(area/math.pi)

To use the method radius() in my objectMain.py file I must import the module from the objectModuleCalc.py file using import <filename>.

#objectMain.py

import math
import objectModuleCalc

class Circle:
    def __init__(self, radius):
        self.radius = radius

    #calculate area from given radius
    def area(self):
        return math.pi * self.radius ** 2

c1 = Circle(5)
print(c1.area())

print(objectModuleCalc.radius(78.54))

#Print-Output
#78.53981633974483
#5.000005846084075

After the import I can access the method radius() in the imported module using the dot notation<filename>.<method>.

Another way to import the Module from the objectModuleCalc.py file is using an alias import <filename> as <alias>.

#objectMain.py

import math
import objectModuleCalc as calc

class Circle:
    def __init__(self, radius):
        self.radius = radius

    #calculate area from given radius
    def area(self):
        return math.pi * self.radius ** 2

c1 = Circle(5)
print(c1.area())

print(calc.radius(78.54))

#Print-Output
#78.53981633974483
#5.000005846084075

After the import I can access the code in the imported module using the dot notation<alias>.<method>.

In the example above you see that the method area() can only be used when an object instance has been created. The method is used on the Object instance using c1.area().

Static Methods

With static methods we can use a method defined in a class without having created an Object instance. We have no Class reference and no Object reference. We use the Function Decorator @staticmathod before the method definition.

#objectMain.py

import math

class Circle:
    def __init__(self, radius):
        self.radius = radius

    #calculate area from given radius
    @staticmethod
    def area(radius):
        return math.pi * radius ** 2

print(Circle.area(5))

#Print-Output
#78.53981633974483

Static methods have no reference to the Class and no reference to the Object instance once it has been created. Therefore we can not access any attributes defined in the Class like Class-attributes or the Instance-attribute self.radius. This is the reason why we must pass a separate parameter radius into the method area(). The value of the parameter radius can then be used by the variable radius within the method area() when we call the method using <Classname>.<method>(<parameter>).

Class methods

With Class methods we can use a method defined in a Class also without having created an Object instance. We use the Function Decorator @classmethod before the method definition.

#objectMain.py

import math

class Circle:
    counter = 0
    def __init__(self, radius):
        self.radius = radius
        Circle.counter += 1

    #calculate area from given radius
    @staticmethod
    def area(radius):
        return math.pi * radius ** 2

    #print the current counter value
    @classmethod
    def printcounter(cls):
        print("Actual value of counter is: " +str(cls.counter))

Circle.printcounter()
c1 = Circle(5)
Circle.printcounter()

#Print-Output
#Actual value of counter is: 0
#Actual value of counter is: 1

I define the Class method printcounter(cls). The parameter cls is the Class reference to ensure that also Class-attributes like counter can be used within the Class-method. The Class-method is called using <Classname>.<class-method-name>. In the Class Circle we define the Class-attribute counter ant set it to the value of 0. The __init__() constructor create an Object instance of type Circle when we call Circle() . Each time we create an Object instance, the counter value of the Class Circle is incremented by 1. We define the Class-method printcounter(cls) and pass the parameter cls to establish the Class reference. Then we can access the Class attribute counter in the printcounter(cls) method using cls.counter. We call the Class method using <Classname>.<classmethod> and see in the print output that the counter value increased by 1 after we created the Object instance c1.

note: In the __init__() constructor above we still use the parameter radius and we pass a value for radius when we create an Object instance from Class Circle. This does not really make sense in this example because we do not use it as we the method area() is defined as static method and there we pass a separate radius value with the method call. Because we will need passing the radius in the text below I decided to keep the __init__() constructor as it is to avoid confusion.

The Class method printcounter(cls) above return a print() function. So when we call the method printcounter(cls) we get a print output at the console.

A Class-method can also return an Object instance. Therefore I define another Class-method radius(cls, area) in the code below. This Class-method has 2 parameters: cls is the Class parameter and stand for the Class reference to ensure that also Class-attributes can be used within the Class method and area to pass the given area value to calculate the radius.

#objectMain.py

import math

class Circle:
    counter = 0
    def __init__(self, radius):
        self.radius = radius
        Circle.counter += 1

    #calculate area from given radius
    @staticmethod
    def area(radius):
        return math.pi * radius ** 2

    #print the current counter value
    @classmethod
    def printcounter(cls):
        print("Actual value of counter is: " +str(cls.counter))

    #calculate radius from given area
    @classmethod
    def radius(cls, area):
        r = math.sqrt(area / math.pi)
        return Circle(r)

Circle.printcounter()
c1 = Circle(5)
print(type(c1))
print(c1.radius)
Circle.printcounter()
c2 = Circle.radius(78)
print(type(c2))
print(c2.radius)
Circle.printcounter()

#Print-Output
#Actual value of counter is: 0
#<class '__main__.Circle'>
#5
#Actual value of counter is: 1
#<class '__main__.Circle'>
#4.982787485166879
#Actual value of counter is: 2

The Class method radius()here is the same method we defined above in a separate module. Here we use radius() to calculate the radius r for a given area value that we pass with the method call. The Class method radius() return an Object instance of type Circle passing the calculated radius using return Circle(r). After we create the Object instance c1 using c1 = Circle(5) we see in the print output above that c1 is of type Circle class '__main__.Circle . We can access the passed radius value using the dot notation c1.radius. Then we call the Class-method Circle.radius(cls, area) with the area parameter 78 and assign the return value to the variable c2 using c2 = Circle.radius(78). As we expected c2 is also of type class '__main__.Circle . We can access the calculated radius value using c2.radius. In the print output you also see that the counter increased from 0 to 2 meaning that 2 Object instances c1 and c2 both of type Circle have been created.

A Class-method can return an Object instance of type of the calling class. Therefore I define a new child-Class Rim. Rim inherits all properties of the parent-Class Circle using Rim(Circle) and Rim has no __init__() constructor. Instead we use pass so that we can create an Object instance of Rim and all properties come from the parent-Class Circle.

#objectMain.py

import math

class Circle:
    counter = 0
    def __init__(self, radius):
        self.radius = radius
        Circle.counter += 1

    #calculate area from given radius
    @staticmethod
    def area(radius):
        return math.pi * radius ** 2

    #print the current counter value
    @classmethod
    def printcounter(cls):
        print("Actual value of counter is: " +str(cls.counter))

    #calculate radius from given area
    @classmethod
    def radius(cls, area):
        r = math.sqrt(area / math.pi)
        return cls(r)

class Rim(Circle):
    pass

Circle.printcounter()
c2 = Circle.radius(78)
print(type(c2))
print(c2.radius)

Circle.printcounter()
r1 = Rim.radius(120)
print(type(r1))
print(r1.radius)

Circle.printcounter()

#Print-Output
#Actual value of counter is: 0
#<class '__main__.Circle'>
#4.982787485166879
#Actual value of counter is: 1
#<class '__main__.Rim'>
#6.180387232371033
#Actual value of counter is: 2

The Class method radius() in the parent-Class Circle has changed a bit: The method return not necessarily an Object instance of type Circle but an Object instance of the calling Class return cls(r). Then we call the Class method Circle.radius(cls, area) with the area parameter 78 and assign the return value to the variable c2 using c2 = Circle.radius(78). As expected c2 is of type class '__main__.Circle . When we call the Class method using Rim.radius(cls, area) with the area parameter 120 and assign the return value to the variable r1 using r1 = Rim.radius(120) we see that r1 is of type class '__main__.Rim .

Property attributes and setters

In the text above we have already dealt with attributes that begin with one underscore _ or double underscores __ and thus indicate that these attributes should be touched carefully from the outside or even cannot be changed at all in the case when we use double underscores.

Here we want to deal again with accessing attributes and changing their values. To show the problem I use the Class Circle and the method area() that is defined within the Class Circle to calculate the area value for the given radius that we pass when we create an Object instance c1using c1 = Circle(<radius>).

import math

class Circle:
    def __init__(self, radius):
        self.radius = radius

    # calculate area from given radius
    def area(self):
        return math.pi * self.radius ** 2

c1 = Circle(5)
print(c1.radius)
print(c1.area())
c1.radius = -10
print(c1.radius)

#Print-Output
#5
#78.53981633974483
#-10
#print(c1.area())
#314.1592653589793

In the code above you see that we are able to change the value of radius from outside to -10 using c1.radius = -10. This doesn’t make much sense, even if the value of the area is calculated correctly because self.radius ** 2 in the area() method.

Then we define the instance attribute self._radius in the __init__() constructor. The radius value passed when we create the Object instance will be assigned to self_radius using self_radius = radius . The underscore shows that the attribute should be touched carefully from the outside.

import math

class Circle:
    def __init__(self, radius):
        self._radius = radius

    # calculate area from given radius
    def area(self):
        return math.pi * self._radius ** 2

    @property
    def radius(self):
        return self._radius

    @radius.setter
    def radius(self, radius):
        if radius >= 0:
            self._radius = radius

c1 = Circle(5)

print(c1.radius)
print(c1.area())

c1.radius = -10
print(c1.radius)
print(c1.area())

c1.radius = 10
print(c1.radius)
print(c1.area())

c1._radius = -10
print(c1.radius)
print(c1.area())

#Print-Output
#5
#78.53981633974483
#5
#78.53981633974483
#10
#314.1592653589793
#-10
#314.1592653589793

First we define a Property attribute using the Property Decorator @Property. The Property attribute definition is a method that return self_radius when we call radius. This mean when we can call <Object-Instance>.radius instead of <Object-Instance>._radius when we want to access the Property attribute. Then we define a Property setter using @<Property-attribute>.setter, in this case we use @radius.setter. The Property setter definition is a method that set self._radius = radius only in case radius is greater or equal to 0. In the following we create the Object instance c1 with given radius of 5 and access the given radius using c1.radius. As you see in the print output we are able to change the value to 10. Even if we are not able to change the value to -10 when we use c1.radius = -10 we are still able to change the value to -10 using c1._radius = -10.

We define the instance attribute self.__radius in the __init__() constructor. The radius value passed when we create the Object instance will be assigned to self__radius using self__radius = radius. The double underscore __ shows that the attribute can not be touched from the outside.

class Circle:
    def __init__(self, radius):
        self.__radius = radius

    # calculate area from given radius
    def area(self):
        return math.pi * self.__radius ** 2

    @property
    def radius(self):
        return self.__radius

    @radius.setter
    def radius(self, radius):
        if radius >= 0:
            self.__radius = radius

c1 = Circle(5)
print(c1.radius)
print(c1.area())

c1.radius = -10
print(c1.radius)
print(c1.area())

c1.radius = 10
print(c1.radius)
print(c1.area())

c1.__radius = -10
print(c1.radius)
print(c1.area())

#Print-Output
#5
#78.53981633974483
#5
#78.53981633974483
#10
#314.1592653589793
#10
#314.1592653589793

The Property attribute definition is a method that now return self__radius when we call radius. The Property setter definition is now a method that set self.__radius = radius only in case radius is greater or equal to 0. We create the Object instance c1 with given radius of 5 and access the given radius using c1.radius. As you see in the print output we are able to change the value to 10 but we are not able to change the value to -10 when we use c1.radius = -10 and c1.__radius = -10.

System Python is installed in /usr/bin directory which is owned by root. This mean all files and symlinks in /usr/bin are system owned which means managed by the system and not managed by the user. In /usr/bin you find python symlinks pointing to the Python binaries in /System/Library/Frameworks/Python.framework/Versions/2.7/bin directory.

patrick@PatrickMBNeu ~ % ls -l /usr/bin/python* 

lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

....

patrick@PatrickMBNeu ~ %

The symlinks python, python2 and python2.7 in /usr/bin each point to the same binary in /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7. So when you type one of these as shell command in your shell terminal the binary in /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 will be executed.

patrick@PatrickMBNeu ~ % python --version
Python 2.7.16

patrick@PatrickMBNeu ~ % python2 --version
Python 2.7.16

patrick@PatrickMBNeu ~ % python2.7 --version
Python 2.7.16

patrick@PatrickMBNeu

Type in the command without using the path like /usr/bin/python works because /usr/bin is configured in the shell PATH variable.

patrick@PatrickMBNeu ~ % echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

patrick@PatrickMBNeu ~ %  

The shell lookup for a command in the paths defined in the PATH variable and starts on the left side of the PATH. So python, python2 and python2.7 do not exist in /usr/local/bin but in /usr/bin.

When you came from Catalina and upgraded to BigSur you realized that System Python has not been upgraded from Python 2 to Python 3 for example. So obviously BigSur still relies on Python 2 which is ok for your Mac but maybe not for you as a Developer. You might need newer or different versions of Python on your system. You must install this them separately.

Many people use Homebrew to install software packages on their MacOS. Homebrew can also be used to install a Python version in addition to the System Python. You can check if Python is installed via Homebrew on your system using the command brew info <formula>.

patrick@PatrickMBNeu ~ % brew info python
python@3.9: stable 3.9.7 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@3.9/3.9.7_1 (3,191 files, 56MB) *
  Poured from bottle on 2021-10-15 at 07:09:27
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.9.rb
License: Python-2.0
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, mpdecimal ✔, openssl@1.1 ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.9/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.9/site-packages

tkinter is no longer included with this formula, but it is available separately:
  brew install python-tk@3.9

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 717,857 (30 days), 1,988,212 (90 days), 8,421,529 (365 days)
install-on-request: 322,364 (30 days), 784,390 (90 days), 2,811,664 (365 days)
build-error: 0 (30 days)

patrick@PatrickMBNeu ~ %

Here on my system Python has been installed with Homebrew in the directory /usr/local/bin which is owned by the user. That means all files and symlinks in this directory can be managed by the user. In /usr/local/bin you find the python symlinks python3 and python3.9 pointing to further symlinks in the Homebrew Cellar in /usr/local/Cellar.

patrick@PatrickMBNeu ~ % ls -l /usr/local/bin/python*

lrwxr-xr-x  1 patrick  admin  40 15 Okt 07:09 /usr/local/bin/python3 -> ../Cellar/python@3.9/3.9.7_1/bin/python3

lrwxr-xr-x  1 patrick  admin  42 15 Okt 07:09 /usr/local/bin/python3.9 -> ../Cellar/python@3.9/3.9.7_1/bin/python3.9

...

patrick@PatrickMBNeu ~ %

The python3 symlink in the Homebrew Cellar point again to another python3 symlink in /usr/local/Frameworks/Python.framework/Versions/3.9/bin and this symlink point finally to python3.9 binary /usr/local/Frameworks/Python.framework/Versions/3.9/bin. The python3.9 symlink from the Homebrew Cellar point directly to the python3.9 binary in /usr/local/Frameworks/Python.framework/Versions/3.9/bin.

patrick@PatrickMBNeu ~ % ls -l /usr/local/Frameworks/Python.framework/Versions/3.9/bin
total 136

lrwxr-xr-x  1 patrick  admin      9 30 Aug 21:19 python3 -> python3.9

-rwxr-xr-x  1 patrick  admin  50472 15 Okt 07:09 python3.9

...

patrick@PatrickMBNeu ~ % 

So If you enter python3 or python3.9 as a command in the terminal the shell lookup these symlinks in directory /usr/local/bin and find a match. In any case you always end up with the python3.9 binary in /usr/local/Frameworks/Python.framework/Versions/3.9/bin and execute the same binary.

patrick@PatrickMBNeu ~ % python3 --version
Python 3.9.7

patrick@PatrickMBNeu ~ % python3.9 --version
Python 3.9.7

patrick@PatrickMBNeu ~ %

It is not a problem to install other Python versions with Homebrew in addition to the System Python, but the Homebrew ecosystem has a special feature: dependencies. Python versions that are installed in the Homebrew perimeter can also represent a dependency on other applications. node, for example, needs Python 3.9. If we uninstall Python 3.9, we have a problem with node. Therefore, Python can never be seen completely isolated in the Homebrew ecosystem and this is exactly the problem we solve with pyenv.

patrick@PatrickMBNeu ~ % brew info node   
                                                 
node: stable 16.11.1 (bottled), HEAD
Platform built on V8 to build network applications
https://nodejs.org/
/usr/local/Cellar/node/16.11.1 (1,980 files, 45.4MB) *
  Poured from bottle on 2021-10-15 at 07:09:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/node.rb
License: MIT
==> Dependencies
Build: pkg-config ✔
Required: brotli ✔, c-ares ✔, icu4c ✔, libnghttp2 ✔, libuv ✔, openssl@1.1 ✔, python@3.9 ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 501,912 (30 days), 1,345,537 (90 days), 4,731,822 (365 days)
install-on-request: 404,509 (30 days), 1,070,393 (90 days), 3,650,272 (365 days)
build-error: 0 (30 days)

patrick@PatrickMBNeu ~ %

pyenv create a complete isolated environment for your Python versions. You are able to install, use and uninstall Python versions and you can switch between them. pyenv can be easily installed with Homebrew. pyenv is therefore installed in the /usr/local/bin directory.

patrick@PatrickMBNeu ~ % which pyenv
/usr/local/bin/pyenv

patrick@PatrickMBNeu ~ % pyenv --version
pyenv 2.1.0

patrick@PatrickMBNeu ~ % 

We must adapt the PATH variable for the standard shell. Here in my example this is configured in the .zshrc file in my Home-Directory.

patrick@PatrickMBNeu ~ % cat .zshrc

# ~/.pyenv provides Python before others of the same name
export PYENV_ROOT=$(pyenv root)
export PATH="$PYENV_ROOT/shims:$PATH"

patrick@PatrickMBNeu ~ %

You check the versions managed by pyenv with the following command. As you see pyenv realize the System Python. With the command pyenv install 3.9.7 you install Python 3.9.7 in your pyenv environment. You can switch to your desired Python version using the command pyenv global <version> .

When you then run the command python you are directly connected to global python version you selected.

patrick@PatrickMBNeu ~ % pyenv versions 
*  system

patrick@PatrickMBNeu ~ % pyenv install 3.9.7

.....

patrick@PatrickMBNeu ~ % pyenv versions 
* system
  3.9.7 (set by /Users/patrick/.pyenv/version)

patrick@PatrickMBNeu ~ % pyenv global 3.9.7

patrick@PatrickMBNeu ~ % pyenv versions 
  system
* 3.9.7 (set by /Users/patrick/.pyenv/version)

patrick@PatrickMBNeu ~ % python
Python 3.9.7 (default, Oct 19 2021, 16:02:07) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

patrick@PatrickMBNeu ~ % pyenv global system

patrick@PatrickMBNeu ~ % pyenv versions     
* system (set by /Users/patrick/.pyenv/version)
  3.9.7

patrick@PatrickMBNeu ~ % python             

WARNING: Python 2.7 is not recommended. 
This version is included in macOS for compatibility with legacy software. 
Future versions of macOS will not include Python 2.7. 
Instead, it is recommended that you transition to using 'python3' from within Terminal.

Python 2.7.16 (default, Aug 30 2021, 14:43:11) 
[GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

patrick@PatrickMBNeu ~ %

Finally: To avoid a warning message from brew when you run brew doctor you must create a brew alias in your .zshrc file.

patrick@PatrickMBNeu ~ % cat .zshrc

alias brew='PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin brew'

# ~/.pyenv provides Python before others of the same name
export PYENV_ROOT=$(pyenv root)
export PATH="$PYENV_ROOT/shims:$PATH"

patrick@PatrickMBNeu ~ %

In my booking system I give users different roles in my app and depending on their role, the users have different authorizations. An admin for example is able to access more sensitive data and functionalities than a normal player or a coach. So my app must know the role of a user to assign different authorizations to the particular user.

Clients, usually browsers send requests the app. The app responds to requests and is solely responsible for ensuring that the client only has access to the data that are intended for it. This request and response game is based on the HTTP protocol. HTTP is a stateless network protocol and requests cannot be related to each other. Each request is isolated and unrelated to previous requests and the server has no chance to recognize clients and does therefore not know their role.

This problem can be solved with sessions and cookies and means that session management must be implemented in the application. The application creates a session and stores session data such as the role of a requestor in this session. The session has a unique ID and the app saves only this ID in a cookie. The cookie is transferred to the browser and stored locally there. From now on, the browser always sends this cookie with the HTTP request and thus identifies itself to the application. The application can check the role of the requestor in the stored session data and control the appropriate access.

Basic setup of the server

First we need a working Server OS. I run Linux Ubuntu in production and have written an article about the basic setup of a production Linux server on my blog site Digitaldocblog. Since I am going to store the sessions in a MongoDB, MongoDB must be installed on the Linux server. I use MongoDB Community Edition but you can also install or upgrade to the MongoDB Enterprise Server version. In the lower part of the article you find the instructions how to install and setup your MongoDB Community Edition on your Linux System. In case you want to read the original documentation go on the MongoDB site and read how to install the MongoDB Community Edition for your OS.

In my express application I use a number of external modules or dependencies that have to be installed for the application in order for the application to run. In the repository of the bookingsystemon my GitHub account you find the package.json file which contains all the necessary dependencies. In principle, it is sufficient if you put this package.json file in your application main directory and install all dependencies with npm install.

Alternatively, of course, all modules can also be installed individually with

npm install <module> --save

Session Management

I discuss in the first part different code snippets in my application main file booking.js. The goal here is that you understand how session management is basically implemented.

// booking.js

// Load express module and create app
const express = require('express');
const app = express();
// Trust the first Proxy
app.set('trust proxy', 1);
// Load HTTP response header security module
const helmet = require('helmet');
// use secure HTTP headers using helmet with every request
app.use(
  helmet({
      frameguard: {
        action: "deny",
      },
      referrerPolicy: {
        policy: "no-referrer",
    },
    })
  );
// Load envy module to manage environment variables
const envy = require('envy');
const env = envy();

// Set environment variables
const port = env.port
const host = env.host
const mongodbpath = env.mongodbpath
const sessionsecret = env.sessionsecret
const sessioncookiename = env.sessioncookiename
const sessioncookiecollection = env.sessioncookiecollection

// Load server side session and cookie module
const session = require('express-session');
// Load mongodb session storage module
const connectMdbSession = require('connect-mongodb-session');
// Create MongoDB session store object
const MongoDBStore = connectMdbSession(session)
// Create new session store in mongodb
const store = new MongoDBStore({
  uri: mongodbpath,
  collection: sessioncookiecollection
});
// Catch errors in case session store creation fails
store.on('error', function(error) {
  console.log(`error store session in session store: ${error.message}`);
});
// Use session to create session and session cookie
app.use(session({
  secret: sessionsecret,
  name: sessioncookiename,
  store: store,
  resave: false,
  saveUninitialized: false,
  // set cookie to 1 week maxAge
  cookie: {
    maxAge: 1000 * 60 * 60 * 24 * 7,
    sameSite: true
  },
}));

... //further code not taken into account at this point

I create a server application using the Express-js  Web Application Framework. Therefore I load the Express-js module with the require() function and store the express() function in the constant app. Because my app is running behind a reverse proxy server I set the app to trust the first proxy server. Then I load the helmet module to use secure response headers in my app. I configure that all browsers should deny iFrames and that my app will set no referrer in the response header.

I use the envy module in my application to manage environment variables. Therefore I load the module with require() and store the envy() function in the constant env. With envy you can define your environment variables in your .env and .env.example files. These files must be stored in the application main directory as explained in the envy documentation.

Since my booking app is a real web application running on a web server in production I can not discuss the real environment variables because of security reasons. Therefore let us see how this work and make an example .env file.

// .env

port=myport
host=myhost
mongodbpath=myexamplemongodbpath
sessionsecret=myexamplesecret
sessioncookiename=booking
sessioncookiecollection=col_sessions

These variables have different values in my .env file. In the booking.jsfile above I define the constant env for the envy function with env = envy() . Then I have access to the environment variables defined in my .env file with env.<variable>. I define constants for each variable and assign the variable from the .env file with env.<variable>. These constants can now be used as values in the code.

I load the express-session module and the connect-mongodb-session module with the require() function. The session module stored in the constant session takes over the entire control of the session and cookie management.

The connect-mongodb-session stored in the constant connectMdbSession module is basically responsible for storing the session in the database. That is why we pass session as a parameter in the code and assign the constant MongoDBstore.

const MongoDBstore = connectMdbSession(session)

With new MongoDBStore I create a new store object. Here I pass the uri of the mongodb path and the collection where sessions should be stored.

// booking.js
...

const store = new MongoDBStore({
  uri: mongodbpath,
  collection: sessioncookiecollection
});

...

The store object initialized in this way contains all necessary parameters to successfully store a session object in my MongoDB database.

After we have defined the storage of the session object, we take care of the session object itself.

With app.use(session( {... cookie: {...} })) I create a session object with various options. The session object will be created with each request and also contains a cookie object. I pass the values for cookie: {...} and then other options likesecret: sessionsecret, the session object name with name: sessioncookiename as well as the location where the session object should be stored with store: store. Furthermore the session object has the option saveUninitialized: false and resave: false .

When the saveUninitialized option is set to false the session object is not stored into the store as long as the session is un-initialized. The option resave: false enforce that a session will not be saved back to the store even if the session is initialized. So we must understand what initialized and un-initialized mean. This must be explained.

A browser send a request to the app. More precisely, the browser sends the request to a defined endpoint in the app. An endpoint defines a path within the app that reacts to HTTP requests and executes code. Depending on the HTTP method GET or POST, the endpoint expects that the requestor requires a document back (GET) or that the requestor wants to send data to the app (POST).

In the example below the browser should send a GET request to GET home endpoint. The endpoint render the HTML template index and send the HTML back to the browser. Then the request is finished. So this process, which starts with the request and ends with the response is the runtime of the request.

In the code snippet below you see 2 GET endpoints in the booking.js file one for the home route and another one for the register route.

// booking.js

... //further code not taken into account at this point

// GET home route only for anonym users. Authenticated users redirected to dashboard
app.get('/', redirectDashboard, (req, res) => {
	
  console.log(req.url);
  console.log(req.session.id);
  console.log(req.session);

  res.render('index', {
      title: 'User Login Page',
    });

});

// GET register route only for anonym users. Authenticated users redirected to dashboard
app.get('/register', redirectDashboard, (req, res) => {
  
  console.log(req.url);
  console.log(req.session.id);
  console.log(req.session);

  res.render('register', {
      title: 'User Registration Page',
    });
});

... //further code not taken into account at this point

The code with app.use (session({ ... })) in my booking.js file ensures that a session object is always generated with each request. As long as a session object is not changed during the runtime of a request a separate session is created for each request and has its own session ID. The option saveUninitialized: false ensure that the session object will not be stored into the database. Each session object created in this way is un-initialized.

You can see the following output on the console for the home route and for the register route when we log the path, the session-ID and the session object on the console for each route.

/
BmbE8RVoTRcPP9nUnBm5JLE1w1mQiNyt
Session {
  cookie: {
    path: '/',
    _expires: 2021-04-24T04:27:04.265Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/register
awlPO-KpyVM51Gp6UAoeXGGmRWo-QFtP
Session {
  cookie: {
    path: '/',
    _expires: 2021-04-24T05:54:57.439Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}
  

The code of my app changes a session object during the runtime of a request by adding a data object when a user has successfully logged in. I will explain the code in detail in the next chapter but at the moment it is enough to know that. Therefore we play through the login of a user as follows.

The browser send a GET request to the home route as explained above, then the index template is rendered and the HTML page with the login form is sent back to the browser. During the runtime of this GET request a session object is created but the session object has not changed. We have already seen this above.

Then the user enters email and password in the login form and click submit. With this submit the browser send a POST request to the POST endpoint /loginusers and again a session object is generated for this POST request. During the runtime of the POST request, the code checks whether the transferred credentials are correct. If the credentials are correct, a data object with user data is generated and attached to the session object. Here the session object is changed during the runtime of the POST request. The existing session created with the POST request is now initialized at that moment. Because of the option saveUninitialized: false this session object is stored into the database store. When we look into the database store using the tool MongoDB Compass we see that the entire session object has been saved into the colsessions collection including the data object containing the required data of the user.

After the session initialization the code called by the POST endpoint redirect the request and send a new GET request to the /dashboard route. The code with app.use(session({ ... })) is called again but now there is an initialized session existing in the store. Because of the option resave: false the existing session object will not be updated and dragged along unchanged with every further request.

You see this in the output on the console when we log the path, the session-ID and the session object on the console for each route. The first output on the console is created when the GET request is sent to the home route. Then, the second output, after the user clicked submit the POST route /loginusers is called and a new session object is created. You see this from the different session IDs. During the runtime of this POST request the data object is added to the session object which initializes the session. Then, the third output, the GET route /dashboard is called and we see the same session object ID but the session object now contain the data object with the user data.

/
TEAZITdX7nLWBDc8uOk2HhXIiMZO7W-4
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:13.236Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/loginusers
gVlKut3bdEMiDHnK455FGjCi6YbPTBuZ
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:35.202Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/dashboard
gVlKut3bdEMiDHnK455FGjCi6YbPTBuZ
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:35.468Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    secure: null,
    domain: null,
    sameSite: true
  },
  data: {
    userId: 5f716b7439777365c18639f1,
    status: 'active',
    name: 'Oskar David',
    lastname: 'Rottländer',
    email: 'oskar@test.com',
    role: 'player',
    age: 17,
    cat: 'youth'
  }
}

In summary, session management works as follows: A session object will be created with each request and the session object is only saved in the database when the user is logged in (saveUninitialized: false). As long as the user is logged in, the session object is not changed and the data of the session object in the database are not updated (resave: false).

But what happens to the cookie ? This will be explained in the next chapter.

User login

When the session has been initialized the cookie containing the session ID is stored in the browser of the requestor. With every request the browser provide the cookie to authenticate the requestor. To authenticate the requestor the code app.use(session({...})) is called and compare the session ID sent by the browser with the session IDs stored in the session store. If a session ID matches, the session object including the data object is attached to the request object to give the app access to the data object. Within the app we now have access to any attribute of the data object with req.session.data.<attribute>. Therefore we can now implement role based authorization by accessing the role of the requestor with req.session.data.role and use this information in conditions in the code to control access depending on the role of the requestor.

But lets start from the beginning with the login of the requestor or user as I call the requestor from now on. In order for a user to be able to login, he or she must first call the login page which can be displayed by calling up the home endpoint.

// booking.js

... // Code not discussed here

// Redirect GET requests from authenticated users to dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.data) {
    res.redirect('/dashboard')

  } else {
    next()

  }
}

... // Code not discussed here

// GET home route only for anonym users. Authenticated users redirected to dashboard

app.get('/', redirectDashboard, (req, res) => {

  res.render('index', {
      title: 'User Login Page',
    });

});

... // Code not discussed here

As you see above in the code I have first defined the middleware function redirectDashboard. This middleware ensure that only users who are not logged in see the login page. If we look at the code of the middleware function we can see that req.session.data is used in an if-condition to check whether a data object is attached to the current session object. In case the if-condition is true, the user is logged in and the request is redirected to the dashboard, but in case the if-condition is false, the user is not logged in and the next() function is called.

The GET endpoint has the routingPath to the home route. When a user visits the homepage of my booking application, the GET HTTP request ask for the home routingPath. The middleware function redirectDashboard is put in front of the routingHandler function. If the user is not logged in the routingHandler function render the HTML template index.pugand send the HTML back to the user or more precisely to the user’s browser.

So far so good. We now imagine a not logged in user who sees the index page in front of him or her now wants to login using his or her email and password.

As described above, the index page is nothing more than a login form for entering an email address and a password. When we look at the index.pug file we see that the form action attribute define that the form data email and password will be sent to the form handler /loginusers using the POST method when the Submit button is clicked.

...

form#loginForm.col.s12(
		method='post', 
		action='/loginusers'
		)

		input.validate(
			type='email', 
			name='email', 
			autocomplete='username' 
			required
			)
		...

		input.validate(
			type='password', 
			name='password', 
			autocomplete='current-password' 
			required
			)
		...

button.btn.waves-effect.waves-light(
		type='submit', 
		form='loginForm'
		)
...

Note: To understand the autocomplete attributes of the input tags I recommend reading the documentation of the Chromium Project. Most browsers have password management functionalities and automatically fill in the credentials after you provide a master password to unlock your local password store. By using these autocomplete attributes in login forms but also in user registration forms or change password forms you help browsers by using these autocomplete functions to better identify these forms.

When the user has entered his or her email and password in the HTML form and clicked the Submit button, the request body contain the Form Data attributes email and password. Then a POST HTTP request is sent via HTTPS to the POST endpoint /loginusers defined in my booking.js file (see above).

In the picture below you can see the output of the network analysis in the developer tool of the chrome browser. Here you can see that the Form Data are not encrypted on browser side but you also see that the POST request URL /loginusers is HTTPS. This mean that when the browser sent the POST request to the server these data are encrypted with SSL/TLS in transit from the browser to the server.

On the server side we have the web application behind a proxy server listening to HTTP requests addressed to the POST endpoint /loginusers. This POST endpoint is an anonym POST Route which means that the routingHandler controller function is restricted to not logged-in users only. This makes sense because a login function must not be used by already logged in users. So already logged in users can not send data to this POST endpoint. This check is controlled by the middleware function verifyAnonym which is put in front of the routingHandler.

So lets look at the relevant code snippets in booking.js.

// booking.js

...

// Load db controllers and db models
const userController = require('./database/controllers/userC');

...

// Verify POST requests only for anonym users
const verifyAnonym = (req, res, next) => {

  if (req.session.data) {
    var message = 'You are not authorized to perform this request because you are already logged-in !';
    res.status(400).redirect('/400badRequest?message='+message);

  } else {
    next()

  }
}

...

// Anonym POST Route
// Login user available for anonym only
app.post('/loginusers', verifyAnonym, userController.loginUser)

...

// GET bad request route render 400badRequest
app.get('/400badRequest', (req, res) => {
 
  res.status(400).render('400badRequest', {
    title: 'Bad Request',
    code: 400,
    status: 'Bad Request',
    message: req.query.message,
  })
})

...

At the beginning of the code I refer the constant userController to the user controller file userC.js using the require method. In userC.js all user functions are defined to control user related operations.

Note: When you look into the userC.js file you see that we export modules using module.exports = {...}. Using this directive we export in fact an object with various attributes and the values of these attributes are functions. So with module.exports = { loginUser: function(...) ...} we export the object including the attribute loginUser which contains a function as value. So when we refer the constant userController using the require() function in the booking.js file we store the complete exported object with all its attributes to the userController constant. Now we have access to any attribute of the exported object from userC.js file with userController.<attribute>. Because the attributes are in fact functions we call these functions with this statement.

In the verifyAnonym function req.session.data is used in the if-condition to check whether a data object is attached to the current session object. In case the if-condition is true, the user is already logged-in and is redirected to the Bad Request GET endpoint /400badRequest which is the standard route in my application to show the user that something went wrong. The user can see what went wrong from a message that has been attached to the request using the request parameter ?message=+message. In case the if-condition is false, the user is not logged-in and the next() function forwards the request to the routingHandler controller function that call the loginUser function using userController.loginUser. This function has access to the attributes email and password of the request body with req.body.email and req.body.password.

So lets look at the relevant code snippets in userC.js file.

// database/controllers/userC.js

// load the bcryptjs module
const bcrypt = require('bcryptjs');
// define hash saltrounds for password hashing
const saltRounds = 10;
// load the relevant Prototype Objects (exported from the models)
...

const User = require('../models/userM');

...

loginUser: function (req, res) {

    const inputemail = req.body.email
    const email = inputemail.toLowerCase()

    console.log(req.url);
    console.log(req.session.id);
    console.log(req.session);

    try {

      User.findOne({ email: email }, async function(error, user) {
        if (!user) {
          var message = 'User not found. Login not possible';
          res.status(400).redirect('/400badRequest?message='+message);

        } else {
          if (user._status !== 'active') {
            var message = 'Login not possible. Await User to be activated';
            res.status(400).redirect('/400badRequest?message='+message);

          } else {
              if (bcrypt.compareSync(req.body.password, user.password)) {

                var yearInMs = 3.15576e+10;
                var currentDate = new Date ()
                var currentDateMs = currentDate.getTime()
                var birthDateMs = user.birthdate.getTime()
                var age = Math.floor((currentDateMs - birthDateMs) / yearInMs)

                if (age < 18) {
                  var cat = 'youth'
                } else {
                  var cat = 'adult'
                };

                var userData = {
                  userId: user._id,
                  status: user._status,
                  name: user.name,
                  lastname: user.lastname,
                  email: user.email,
                  role: user.role,
                  age: age,
                  cat: cat,
                }

                req.session.data = userData

                res.status(200).redirect('/dashboard')

              } else {
                var message = 'Login not possible. Wrong User password';
                res.status(400).redirect('/400badRequest?message='+message);
              }
          }
        }
      })

    } catch (error) {
      // if user query fail call default error function
      next(error)

    }
  // End Module
  },

...

In order to authenticate a user, the loginUser function must find a user in the user database with the same email address as the one that was sent by the browser and attached to the request body by the app. If a user was found with the email, the function must check whether the transmitted password matches the password that is stored in the database for this user. If the email and password match the user is authenticated and the login is successful, if not, the login fails.

Passwords are never saved in plain text. Therefore I use the bcryptjs module to hash passwords. The bcryptjs module is loaded into the code with the require() function and assigned to the constant bcrypt. We set the constant saltRounds to the value of 10. This is the so called cost factor in the bcrypt hashing function and controls how much time bcrypt need to calculate a single bcrypt hash. Increasing the cost factor by 1 doubles the time and the more time bycrypt need to hash the more difficult it is to brute force stored passwords.

Then I load the user model from userM.js using the require() function and assign the constant User. Here at this point I have to explain the background. To do this, we also need a look at the userM.js file.

Note: I use MongoDB as the database and Mongoose to model the data. If you look in the file userM.js you see that a user object is created with the function new Schema() and saved in the variable userSchema. This userSchema object describes a user with all its attributes. At the end of the file, the mongoose.model() function is used to reference the userSchema to the collection colusers in my MongoDB. This reference is assigned to the variable User and exported using the function module.exports(). With User I have access to the user model meaning to all user objects and attributes in my database that are stored in the colusers collection. So that I can use this access in the code of my userC.js file I load userM.js with the require() function and assign the constant User. I can now use Mongoose functions for example to query user data from my colusers collection. This is exactly what we do with User.findOne() when we try to find a user with a certain email.

The actual user authentication now takes place in the userFindOne() function.

When we run User.findOne() we check the criteria that do not lead to a successful authentication.

1. No user found: We are looking for a user object that matches the email that has been submitted. If no user object is found with that email or the user found is not active, the request is redirected to the 400badRequest route. If we have found an active user, the submitted password string is hashed with bcrypt and compared with the saved password.
2. Wrong password: If the password comparison is not successful, the submitted password was wrong and the request is also redirected to the 400badRequest route.
   

Note: User.findOne() has a query object {email: email} and a callback function async function(error, user {...}) as parameters. When the async function find a user with the email in the database, this async function returns a user object with all the user attributes and store this object into the user parameter. Within the scope of the async function I have now access to the user attributes using user.<attribute>.

Only in case the user with the email is found and the submitted password is correct the authentication is successful.

If the user is successfully authenticated, the category of the user is calculated based on the current date and the user’s birth date. Then a userData object is created in which various user attributes are stored. The data of the userData object are then attached to the session. More precisely, the object data is attached to the session with req.session.data and the value userData is assigned. Now the session is initialized and the session object is stored in the colsessions collection of the MongoDB.

Then the response is sent back to the browser.

In this response, the browser is instructed to call up a GET request to the GET endpoint /dashboard. The response is sent using res.status(200).redirect('/dashboard'). In the Response Header you see that the cookie with the name booking is set in the users browser using the set-cookie directive. The cookie only contain the session ID which has been signed and encrypted with the secret we provided in app.use(session( {... cookie: {...} })).

Then the browser send the GET request to the endpoint /dashboard. Lets have a look into the booking.js file again.

// booking.js 

...

// Redirect GET requests from not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.data) {
    res.redirect('/')

  } else {
    next()

  }
}

...

// GET dashboard route only for authenticated users. Anonym users redirected to home
app.get('/dashboard', redirectLogin, async (req, res) => {

  // Check admin authorization and render admin_dashboard
  if (req.session.data.role == 'admin') {

    const user_query = User.find( {} ).sort({lastname: 1, name: 1});
    var users = await user_query.exec();

    const training_query = Training.find( {} ).sort({date: 'desc'});
    var trainings = await training_query.exec();

    const location_query = Location.find( {} ).sort({location: 'desc'});
    var locations = await location_query.exec();

    const booking_query = Booking.find( {} ).sort({_booktrainingdate: 'desc'});
    var bookings = await booking_query.exec();

    const invoice_query = Invoice.find( {} ).sort({invoicedate: 'desc'});
    var invoices = await invoice_query.exec();

    res.status(200).render('admin_dashboard', {
      title: 'Admin Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      data_users: users,
      data_trainings: trainings,
      data_locations: locations,
      data_bookings: bookings,
      data_invoices: invoices,

      });

  // Check player authorization and render player_dashboard
  } else if (req.session.data.role == 'player') {

    var currentDate = new Date();
    console.log('current date: ' +currentDate);

    const availabletraining_query = Training.find( { _status: 'active', date: { $gte: currentDate } } ).sort({ date: 'desc' });
    var availabletrainings = await availabletraining_query.exec();

    const booking_query = Booking.find( { _bookuseremail: req.session.data.email, _bookparticipation: { $ne: 'invoice' } } ).sort({ _booktrainingdate: 'desc' });
    var bookings = await booking_query.exec();

    const myuser_query = User.findOne( { email: req.session.data.email } );
    var myuser = await myuser_query.exec();

    const invoice_query = Invoice.find( {invoiceemail: req.session.data.email} ).sort({invoicedate: 'desc'});
    var invoices = await invoice_query .exec();

    res.status(200).render('player_dashboard', {
      title: 'Player Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      email: req.session.data.email,
      data_availabletrainings: availabletrainings,
      data_bookings: bookings,
      data_myuser: myuser,
      data_myinvoices: invoices,
      });

  // Check coach authorization and render coach_dashboard
  } else if (req.session.data.role == 'coach') {
   
    var currentDate = new Date().setHours(00, 00, 00);
    console.log('currentDate: ' +currentDate);

    const training_query = Training.find( { _status: 'active', date: { $gte: currentDate } } ).sort({ date: 'asc' });
    var trainings = await training_query.exec();

    res.status(200).render('coach_dashboard', {
      title: 'Coach Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      data_trainings: trainings,
      });

  } else {
    // if user not authorized as admin, player or coach end request and send response
    var message = 'You are not authorized. Access prohibited';
    res.status(400).redirect('/400badRequest?message='+message);
  }

});

...

As you see above in the code we have first defined the middleware function redirectLogin. This middleware ensure that only users who are logged in see the dashboard page. In case the if-condition is true, the user is not logged in and the request is redirected to the home route, but in case the if-condition is false, the user is logged in and the next() function is called.

The GET HTTP request ask for the dashboard routingPath. The middleware function  redirectLogin is put in front of the routingHandler function. If the user is not logged in the  redirectLogin middleware redirect the request to the home route. In case the user is logged-in the routingHandler function is called using the request object and the response object as parameters.

app.get('/dashboard', redirectLogin, async (req, res) => {...})

If we look at the Request Header of this new GET request in the browser we can see that the cookie is dragged along unchanged with the GET request to the endpoint /dashboard. This happens from now on with every request until the session expires or until the user logout.

And now within the routingHandler function we do the user authorization check. The if condition check the users role using req.session.data.role. Depending on the role of the user different <role>dashboard HTML templates are rendered and for each role different HTML is sent back to the user’s browser. Various queries are executed beforehand because we need role specific data within each <role>dashboard HTML template. The return values ​​of the different queries find() and findOne() are only executed in case one of the if conditions become true. Then in each case the return values of the queries are stored in variables. In case all if conditions are false, meaning we cannot find a user with a role like admin, player or coach in the database for some reason the request is redirected to the Bad Request GET endpoint /400badRequest using the message as request parameter that this user is not authorized.

Within each if condition and for each role admin, player or coach, we create the response object by first setting the HTTP status to the value of 200 and then using the render method to render the respective HTML template.

res.status(200).render('<role>_dashboard', {...})

Within the render method, we now have the option of transferring a data object with different attributes to the HTML template. Later we can access these data in the respective HTML template and use it within the HTML template. How this works is not part of this article. But of course you can take a closer look at the templates admin dashboard, player dashboardand coach dashboardon my GitHub repository and you will immediately see how this works.

Create Authorizations

As I have already shown in the upper part, I work with middleware functions to control access to GET and POST endpoints in my app. Therefore these middleware functions are the Authorizations and you can find them in the code of my booking.js file.

// booking.js

...

// Authorizations
// Redirect GET requests from not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.data) {
    res.redirect('/')

  } else {
    next()

  }
}

// Redirect GET requests from authenticated users to dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.data) {
    res.redirect('/dashboard')

  } else {
    next()

  }
}

// Authorize POST requests only for not authenticated users
const verifyAnonym = (req, res, next) => {
  if (!req.session.data) {
    next()

  } else {
    var message = 'You are already logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for anonym and admin users
const verifyAnonymAndAdmin = (req, res, next) => {
  if (!req.session.data) {
    next()

  } else {

    if (req.session.data.role == 'admin') {
      next()

    } else {
      var message = 'You are no Admin. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);

    }

  }
}

// Authorize POST requests only for admin and player users
const verifyAdminAndPlayer = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'admin') {
      next()

    } else if (req.session.data.role == 'player') {
      next()

    } else {
      var message = 'You are no Admin, no Player. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);
  }

}

// Authorize POST requests only for admin users
const verifyAdmin = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'admin') {
      next()

    } else {
      var message = 'You are no Admin. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for player users
const verifyPlayer = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'player') {
      next()

    } else {
      var message = 'You are no Player. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for coach users
const verifyCoach = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'coach') {
      next()

    } else {
      var message = 'You are no Coach. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);

    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

...

As I have already explained above I use redirect functions as a middleware to control access to the GET endpoints home, register and dashboard. These middleware functions basically control access based on whether a user is logged-in or not. The redirect function redirectDashboard allow only not logged-in users access to the home endpoint and to the register endpoint, while users who are already logged-in have no access and would be redirected directly to the dashboard route if they try to access here. The redirectLogin middleware function allow only logged-in users access to the dashboard route while not logged-in users are redirected to the login or better to the home endpoint.

In addition to redirect functions I use verify functions as a middleware to control access to the POST endpoints. With the help of POST requests, data are sent via POST endpoints to the app. That is why it is particularly important to control who is allowed to send data and who is not. I use basically 5 types of POST endpoints.

Anonym POST endpoint. I only have one endpoint here. The loginusers endpoint can only be called by not logged-in users. Therefore the verifyAnonym  middleware is set before the routingHandler function to verify if the user is not logged-in.

// booking.js
...

// Anonym POST endpoint
// Login user available for anonym only
app.post('/loginusers', verifyAnonym, userController.loginUser)

...

Shared POST endpoints. The createusers endpoint can be called by not logged-in users and Admin users. The  verifyAnonymAndAdmin  middleware is set before the routingHandler function to verify if the user is not logged-in or if the user that is logged-in has the role admin. The  updateuseremail and setnewuserpassword endpoints can be called only by Admin and Player users. Therefore the  verifyAdminAndPlayer  middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is admin or player.

// booking.js
...

// Shared POST endpoints
// Create Users available for anonym and admin
app.post('/createusers', verifyAnonymAndAdmin, birthdateFormatValidation, userController.createUser)

// Update User-Email available for admin and player
app.post('/updateuseremail', verifyAdminAndPlayer, userController.updateUserEmail)

// Update User-Password available for admin and player
app.post('/setnewuserpassword', verifyAdminAndPlayer, userController.setNewUserPassword)

...

Admin POST endpoints. I have 19 endpoints here and each of these endpoint can only be called by Admin users. The  verifyAdmin middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is admin.

// booking.js
...

// Admin POST endpoints
// Admin User Management
app.post('/callupdateusers', verifyAdmin, userController.callUpdateUsers)

app.post('/updateuser', verifyAdmin, birthdateFormatValidation, userController.updateUser)

app.post('/terminateusers', verifyAdmin, userController.terminateUser)

app.post('/activateusers', verifyAdmin, userController.activateUser)

app.post('/removeusers', verifyAdmin, userController.removeUser)

// Admin Update Training
app.post('/callupdatetrainings', verifyAdmin, trainingController.callUpdateTrainings)

app.post('/updatetraining', verifyAdmin, trainingController.updateTraining)

// Admin Location Management
app.post('/createlocations', verifyAdmin, locationController.createLocation)

app.post('/callupdatelocations', verifyAdmin, locationController.callUpdateLocations)

app.post('/updatelocation', verifyAdmin, locationController.updateLocation)

app.post('/callcreatetrainings', verifyAdmin, trainingController.callCreateTrainings)

app.post('/createtraining', verifyAdmin, trainingController.createTraining)

// Admin Invoice Management
app.post('/createinvoice', verifyAdmin, invoiceController.createInvoiceUser)

app.post('/callcancelinvoice', verifyAdmin, invoiceController.callCancelInvoice)

app.post('/cancelinvoice', verifyAdmin, invoiceController.cancelInvoice)

app.post('/callpayinvoice', verifyAdmin, invoiceController.callPayInvoice)

app.post('/payinvoice', verifyAdmin, invoiceController.payInvoice)

app.post('/callrepayinvoice', verifyAdmin, invoiceController.callRePayInvoice)

app.post('/repayinvoice', verifyAdmin, invoiceController.rePayInvoice)

...

Player POST endpoints. I have 7 endpoints here and each of these endpoint can only be called by Player users. The  verifyPlayer middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is player.

// booking.js
...

// Player POST endpoints
// Player Booking Management
app.post('/callbooktrainings', verifyPlayer, bookingController.callBookTrainings)

app.post('/booktrainings', verifyPlayer, bookingController.bookTraining)

app.post('/bookingreactivate', verifyPlayer, bookingController.bookingReactivate)

app.post('/callcancelbookings', verifyPlayer, bookingController.callCancelBooking)

app.post('/cancelbookings', verifyPlayer, bookingController.cancelBooking)

// Player User Management
app.post('/callupdatemyuserdata', verifyPlayer, userController.callUpdateMyUserData)

app.post('/updatemyuserdata', verifyPlayer, birthdateFormatValidation, userController.updateMyUserData)

...

Coach POST endpoints. I have 2 endpoints here and each of these endpoint can only be called by Coach users. The  verifyCoach middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is coach.

// booking.js
...

// Coach POST endpoints
// Coach Confirmation Management
app.post('/callparticipants', verifyCoach, bookingController.callParticipants)

app.post('/callconfirmpatricipants', verifyCoach, bookingController.callConfirmPatricipants)

...

User logout

The user initiates a logout himself by clicking on the logout link in the navigation of the application. This sends a GET request to the /logout endpoint of the application. In this routing definition, the session is first deleted from the database using req.session.destroy() and then the cookie is removed from the browser and the user is redirected to the 200success site using res.status(200).clearCookie('booking').redirect().

// booking.js

...

// GET logout route only for authenticated users. Anonym users redirected to home
app.get('/logout', redirectLogin, (req, res) => {
  req.session.destroy(function(err) {
    if (err) {
      res.send('An err occured: ' +err.message);
    } else {
      var message = 'You have been successfully logged out';
      res.status(200).clearCookie('booking').redirect('/200success?message='+message)
    }
  });
})

...