Prepare your Server

Before we start we need to prepare the server. In this step we create the environment to manage the vaultwarden instance. Login to your server using your standard User (not root).

Basic requirements

Login to your server with SSH and authenticate with keys. Never use simple password authentication. You must create a private and a public SSH key-pair on your Host Machine and copy only the public SSH key to your Remote Server. Keep your private key safe on your Host machine. Then copy your public key to your Remote Server and configure your ssh-deamon on your Remote Server. Disable password authentication and root login in your configuration on your Remote Server. How all this works is very good explained on Linuxize.com.

You must ensure that SSL certificates for your server are installed. I use free LetsEncrypt certificates and certbot to install and renew my certificates on the system. Therefore pls. read on my Digitaldocblog in the Article SSL Certificates with Lets Encrypt and certbot on a Linux Server . Here you find a very detailed description of how you can do this .

You must ensure that docker and docker-compose is installed on your system. Therefore pls. read on my Digitaldocblog a very detailed description of how you can do this in the Article Containerize a nodejs app with nginx. You should read the following chapters:

• Prepare the System for secure Downloads from Docker
• Install Docker from Docker Resources
• Install standalone docker-compose from Docker Resources
  

Make sure that your server is running behind a firewall. In my case I have a virtual server and I am responsible for server security. Therefore I install and configure a firewall on my system.

Before you configure the firewall be sure that your ssh service is running and on which port your ssh service is running. This is important to know because you don’t want to lock yourself out. First you check if the ssh service is running with systemctl and then on which port ssh is running using netstat. If netstat is not installed on your system you can do this with apt.

#control ssh status
sudo systemctl status ssh

#check ssh port
sudo netstat -tulnp | grep ssh

#check if net-tools (include netstat) are installed
which netstat

#install net-tools only in case not installed
sudo apt install net-tools

I install ufw (uncomplicated firewall) on my server and configure it to provide only SSH and HTTPS to the outside world.

# check if ufw is installed
ufw version
which ufw

#install ufw if not installed
sudo apt install ufw

#open SSH and HTTPS
sudo ufw allow OpenSSH
sudo ufw allow 443
sudo ufw allow 80/tcp

#Default rules
sudo ufw default deny incoming
sudo ufw default allow outgoing

#Start the firewall
sudo ufw enable

#Check the firewall status
sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
80/tcp                     ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)             
80/tcp (v6)                ALLOW IN    Anywhere (v6)

Port 443 must run TCP and UDP

In the Docker environment, we will later configure Caddy as a reverse proxy for the vaultwarden service. Caddy requires both TCP and UDP on port 443. This is due to a modern protocol called HTTP/3 (QUIC).

Traditionally, the web (HTTP/1.1 and HTTP/2) ran exclusively over the TCP protocol. TCP is very reliable, but sometimes a bit slow when establishing a connection. Google and others subsequently developed QUIC, on which the new standard HTTP/3 is based. QUIC uses UDP instead of TCP.

HTTP/3 QUIC is significantly faster when establishing a connection (handshake). It handles packet loss better, which is especially important for mobile connections on smartphones, for example, when using the Bitwarden app.

Caddy is a very modern reverse proxy that has HTTP/3 enabled by default. The „normal“ HTTPS connection (HTTP/1.1 or HTTP/2) is established over TCP port 443. Caddy offers clients (Browsers or the Bitwarden app) the option to switch to HTTP/3 via UDP port 443.

With vaultwarden, users often access their vaults via smartphones using LTE or Wi-Fi connections. When UDP port 443 is open, the app uses HTTP/3. This results in a more stable connection and improved vault synchronization due to reduced latency.

Why must Port 80 be open

When you follow the instructions in the Article SSL Certificates with Lets Encrypt and certbot on a Linux Server then you are using certbot and install your LetsEncrypt certificates on your local server with the following certbot command :

sudo certbot certonly --standalone -d <yourDomain>

Here you use the method standalone. Whenever you renew your certificates certbot starts a temporary web server to prove ownership of your domain to LetsEncrypt.

When renewing your certificates certbot attempts to start a temporary web server on port 80. Therefore, port 80 must be open on your system via your firewall rules.

Important: If another web server (such as Nginx or Apache) is already running permanently on port 80, this step will fail unless the service is stopped. So, if a web server is running on port 80, this service must be permanently stopped.

After Certbot successfully starts the web server on port 80, the following happens:

• ACME Challenge: Certbot contacts the Let’s Encrypt servers. These provide Certbot with a random string (token).
• Deployment: Certbot creates a small file on the spot at the URL http://yourDomain/.well-known/acme-challenge/token
• Verification: The Let’s Encrypt servers now attempt to access this exact URL via the public internet. If they find the file (and the correct token), it proves that you own the server under this domain.
  

Once the verification is successful, the temporary standalone web server is immediately shut down and port 80 is opened. Certbot generates a new private key (if configured) and signs the new certificate. The new certificate files are stored in the directory /etc/letsencrypt/archive/ and the symbolic links in the directory /etc/letsencrypt/live/ are updated.

Create new user vaultwarden

You are logged in with your standarduser. From the home directory of the standarduser you create the user vaultwarden and create the hidden directory /home/vaultwarden/.ssh. Then copy the authorized_keys file from your .ssh directory into the new created .ssh directory of the new user vaultwarden and set the owner and permissions.

The new user vaultwarden should be in sudo group to perform commands under root using sudo. And the user vaultwarden should be in docker group to perform docker commands without using sudo.

#logged-in with standard user and create new user 
sudo adduser vaultwarden

#create hidden .ssh directory in new users home directory
sudo mkdir /home/vaultwarden/.ssh

#copy authorized_keys file to enable ssh key login for new user
cd /home/standarduser/.ssh
sudo cp authorized_keys /home/vaultwarden/.ssh

#set the owner vaultwarden and permissions
sudo chown -R vaultwarden:vaultwarden /home/vaultwarden/.ssh
sudo chmod 700 /home/vaultwarden/.ssh
sudo chmod 600 /home/vaultwarden/.ssh/authorized_keys

#check permissions
ls -al /home/vaultwarden
drwx-- vaultwarden vaultwarden 4096 May 11 14:20 .ssh

ls -l /home/vaultwarden/.ssh
-rw-- vaultwarden vaultwarden 400 May 11 14:20 authorized_keys

#add user vaultwarden to sudo- and docker group
sudo usermod -aG sudo vaultwarden
sudo usermod -aG docker vaultwarden

#check vaultwarden groups (3 groups: vaultwarden sudo docker)
sudo groups vaultwarden
vaultwarden : vaultwarden sudo docker

Create /opt/vaultwarden directory

Login with the new user vaultwarden. Stay logged in as vaultwarden for the next steps. Do not perform the next steps or the installation of the vaultwarden server with root or any other user.

After you are logged in with vaultwarden you create a new directory /opt/vaultwarden. This is the runtime directory of your vaultwarden application and the place from where the docker containers will be started.

sudo mkdir /opt/vaultwarden
sudo chmod -R 700 /opt/vaultwarden

ls -l /opt/vaultwarden
drwx--vaultwarden vaultwarden 4096 Mai 16 07:06 vaultwarden
 

Then change into /opt/vaultwarden. You create the /opt/vaultwarden/vw-data directory which is the host directory for the docker containers. One of these containers will be started under the container name vaultwarden. This container vaultwarden run with root privileges and write into this host directory /opt/vaultwarden/vw-data.

mkdir /opt/vaultwarden/vw-data

ls -l /opt/vaultwarden
drwxrwxr-x 6 vaultwarden vaultwarden 4096 Mai 15 15:54 vw-data
`

Create /opt/vaultwarden/certs and copy your SSL certificates

The container vaultwarden is the web application where you can log-in and manage your passwords. As we will se below the container will be started under the user vaultwarden in /opt/vaultwarden and run with root privileges behind a reverse proxy server. As reverse proxy I will use Caddy which is a powerful platform. Caddy manages the requests from the outside world and forward requests via an internal docker network to the vaultwarden server.

Caddy must accept only HTTPS connections and use the standard directory for certificates /opt/vaultwarden/certs. Therefore we create this directory.

I use Letsencrypt certificates which are managed automatically by certbot and stored in the directory /etc/letsencrypt on my host server. The keys are set up the following structure:

• In /etc/letsencrypt/live/<domain> there are only symlinks that point
• to the real files in /etc/letsencrypt/archive/<domain>.
  

Copy the fullchain.pem and privkey.pem files from /etc/letsencrypt/live/<domain> to /opt/vaultwarden/certs and set the permissions accordingly.

#create certs directory
mkdir /opt/vaultwarden/certs

#change into certs directory
cd /opt/vaultwarden/certs

#copy the files
cp /etc/letsencrypt/live/<domain>/fullchain.pem fullchain.pem
cp /etc/letsencrypt/live/<domain>/privkey.pem privkey.pem

#set owner and group vaultwarden
chown vaultwarden:vaultwarden fullchain.pem
chown vaultwarden:vaultwarden privkey.pem

#set access (read write only vaultwarden)
chmod 600 privkey.pem
chmod 600 fullchain.pem

note: The cp command will place the actual files (not the symlinks) into the directory /opt/vaultwarden/certs unless you specify a -P or -d option. So the cp command follows the symlinks and grab the actual files behind. This is important when the certificates have been renewed and new certificate files are stored behind the symlinks.

Run Vaultwarden and Caddy as docker containers

The following setup creates a secure, email-enabled Vaultwarden instance behind a Caddy reverse proxy with HTTPS and admin access, running entirely via Docker.

Create docker-compose.yml

This docker-compose.yml file sets up two services Vaultwarden and Caddy to host a self-hosted password manager with HTTPS support.

#docker-compose.yml

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      DOMAIN: "<yourDomain>"
      SIGNUPS_ALLOWED: "false"
      SMTP_HOST: "<yourSmtpServer>"
      SMTP_FROM: "<yourEmail>"
      SMTP_FROM_NAME: "<yourName>"
      SMTP_USERNAME: "<yourEmail>"
      SMTP_PASSWORD: "<yourSmtpPasswd>"
      SMTP_SECURITY: "force_tls"
      SMTP_PORT: "465"
      ADMIN_TOKEN: '<yourAdminToken>'
    volumes:
      - ./vw-data:/data

  caddy:
    image: caddy:2
    container_name: caddy
    restart: always
    ports:
      - 443:443
      - 443:443/udp 
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
      - ./certs:/certs:ro
    environment:
      DOMAIN: "<yourDomain>"
      LOG_FILE: "/data/access.log"

vaultwarden service:

Runs the Vaultwarden server (a lightweight Bitwarden-compatible backend).

• Disables user signups (SIGNUPS_ALLOWED: „false“).
• Configures SMTP settings for sending emails (e.g. for password resets).
• Sets a secure admin token (using Argon2 hash) to access the /admin interface.
• Persists Vaultwarden data to ./vw-data on the host.
  

Enable admin page access:

With the ADMIN_TOKEN set we enable login to the admin site via <yourDomain>/admin. First create a secure admin password. The <yourAdminToken> value can be created with your admin password piped into argon2. The result is a hash value that must be a little modified and then inserted as <yourAdminToken>. It is important that you use single quotes in docker-compose.yml when you insert <yourAdminToken>.

sudo apt install -y argon2
echo -n '<yourAdminPassword>' | argon2 somesalt -e

#This is the result of the argon2 hashing
$argon2i$v=19$m=4096,t=3,p=1$c29tZXNhbHQ$D...

Then you modify the hash by adding a $ sign in front of each $ sign in the hash. In this case we add 5 $ signs.

#original value
$argon2i$v=19$m=4096,t=3,p=1$c29thbHQ$D...

#modified value
$$argon2i$$v=19$$m=4096,t=3,p=1$$c29thbHQ$$D...

Then you put the modified value in single quotes into docker-compose.yml.

#docker-compose.yml

.....

ADMIN_TOKEN: '$$argon2i$$v=19$$m=4096,t=3,p=1$$c29thbHQ$$D...'

Now you can access the admin page with <yourDomain>/admin and login with your admin password.

caddy service:

Uses Caddy web server to reverse proxy to Vaultwarden.

• Handles HTTPS using custom certificates from ./certs.
• Binds to port 443 for secure access.
• Reads its configuration from ./Caddyfile.
• Logs access to /data/access.log (mapped from ./caddy-data on the host).
  

Create Caddyfile

Caddy is a modern, powerful web server that automatically handles HTTPS, reverse proxying, and more. Caddy acts as a secure HTTPS reverse proxy, forwarding external requests to the Vaultwarden Docker container running on internal port 80.

This Caddyfile defines how Caddy should serve and protect your vaultwarden instance over HTTPS.

#Caddyfile
https://<domain> {
  log {
    level INFO
    output file /data/access.log {
      roll_size 10MB
      roll_keep 10
    }
  }

  # Use custom certificate and key
  tls /certs/fullchain.pem /certs/privkey.pem

  # This setting may have compatibility issues with some browsers
  # (e.g., attachment downloading on Firefox). Try disabling this
  # if you encounter issues.
  encode zstd gzip

  # Admin path matcher
  @adminPath path /admin*
  
  # Basic Auth for admin access
  handle @adminPath {
    # If admin path require basic auth
    basicauth {
      superadmin <passwdhash>
    }

    reverse_proxy vaultwarden:80 {
      header_up X-Real-IP {remote_host}
    }
  }

  # Everything else
  reverse_proxy vaultwarden:80 {
    header_up X-Real-IP {remote_host}
  }
}

Domain

https://<domain>

This defines the domain name Caddy listens on (e.g. https://yourinstance.example.com).

Logging

log {
  level INFO
  output file /data/access.log {
    roll_size 10MB
    roll_keep 10
  }
}

Logs all access to a file inside the container (/data/access.log), with log rotation.

TLS with Custom Certificates

tls /certs/fullchain.pem /certs/privkey.pem

Use your own Let’s Encrypt certificates from mounted files rather than auto-generating them.

Compression

encode zstd gzip

Enables modern compression methods to improve performance, though may cause issues with attachments on some browsers.

Admin Area Protection

@adminPath path /admin*

Matches all requests to /admin paths.

handle @adminPath {
  basicauth {
    superadmin <passwdhash>
  }

  reverse_proxy vaultwarden:80 {
    header_up X-Real-IP {remote_host}
  }
}

• Requires HTTP Basic Auth for access to /admin.
• Proxies/Forwards authenticated admin requests to the Vaultwarden container.
• Ensures the backend sees the original client IP address.
  

All Other Requests

reverse_proxy vaultwarden:80 {
  header_up X-Real-IP {remote_host}
}

• Proxies/Forwards all non-/admin traffic directly to Vaultwarden container.
• Ensures the backend sees the original client IP address.
  

Protect your admin page

To protect your admin page we can use a HTTP Basic Auth. This mean whenever you access <yourDomain>/admin a login window will pop up in your browser and ask you to provide a user name and a password. We use htpasswd which is part of the apache2-utils.

#install apache2-utils if not available
sudo apt install apache2-utils

#Create a hash for the user admin (you can use any user-name) 
htpasswd -nB admin

New password: 
Re-type new password: 
admin:$2y$05$HZukVJWhWMrT7qMO2n65bm/5JYlt5tO...

Then you insert only the hash (without admin: ….) into the Caddyfile.

#Caddyfile

.....

handle @adminPath 
bash
  basicauth {
    admin $2y$05$HZukVJWhWMrT7qMO2n65bm/5JYlt5tO...
  }

Create ssl certificates with Letsencrypt and certbot

You can follow the instructions in my post on digitaldocblog.com. When you followed these instructions your ssl certificates are installed on your server in standalone mode. Whenever you renew your certificates certbot initiates the domain validation challenge and a temporary server will be started trying to listen on port 80. Because we run a web server this port is blocked and the web server must be stoped before we can initiate the renewal process.

To avoid this we should change the certbot renewal from standalone mode to webroot. Webroot is a method that leverages your existing web server to handle the domain validation challenge process without the need to stop the web server. Certbot places a temporary file with a unique token in a specific directory within your web servers public „webroot“ directory (e.g., /var/www/html/.well-known/acme-challenge/). Let’s Encrypt then sends a request to your domain to retrieve this file. Since your web server is already running, it can serve the file without any interruption to your website’s availability.

In our docker.compose.yml we defined the backend service vaultwarden. This service is the backend service listening to port 80. We also defined a web server service caddy listening to port 443.

In our Caddyfile we specified only a web server only for https://<Domain> working as a reverse proxy. Any request for https://<Domain> will be forwarded to the backend service vaultwarden listening on port 80 vaultwarden:80.

To enable webroot for certbot certificate renewal we must change the docker.compose.yml file. For the caddy service and in the ports section we must allow port 80 and expose a webroot directory via port 80 only for serving the domain challenge validation file. Therefore we create on the local host directory /opt/vaultwarden/caddyacme. In the volumes section of the caddy service we map this local directory 1:1 into the caddy container. Note: the notation with the dot like ./caddyacme:/caddyacme requires that the actual Caddyfile is in the same directory /opt/vaultwarden/Caddyfile than /opt/vaultwarden/caddyacme.

#docker-compose.yml

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      DOMAIN: "https://bitwarden.rottlaender.eu"
      SIGNUPS_ALLOWED: "false"
      SMTP_HOST: "smtp.strato.de"
      SMTP_FROM: "bw@bitwarden.rottlaender.eu"
      SMTP_FROM_NAME: "Vaultwarden"
      SMTP_USERNAME: "bw@bitwarden.rottlaender.eu"
      SMTP_PASSWORD: "ZZqrmQEvydkxY3E2u8Y.KEbKNwgkTV"
      SMTP_SECURITY: "force_tls"
      SMTP_PORT: "465"
      ADMIN_TOKEN: '$$argon2i$$v=19$$m=4096,t=3,p=1$$c29tZXNhbHQ$$D/yu7vPhcpPz8Kk7G/R34YSO+NgtLzai0wVGSGL0RDE'
    volumes:
      - ./vw-data:/data

  caddy:
    image: caddy:2
    container_name: caddy
    restart: always
    ports:
      - 80:80
      - 80:80/udp
      - 443:443
      - 443:443/udp
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy-config:/config
      - ./caddy-data:/data
      - ./certs:/certs:ro
      - ./caddy_acme:/caddy_acme
    environment:
      DOMAIN: "https://bitwarden.rottlaender.eu"
      LOG_FILE: "/data/access.log"

With this configuration we configure caddy to listen on port 80 and on port 443. In the current ufo firewall configuration we only allow the ports 22 and 443.

#Check the firewall status
sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)

We must open port 80 to enable the certbot webroot certificate renewal.

# open port 80
sudo ufw allow 80/tcp

# Check the firewall status
sudo ufw status verbose

Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp (OpenSSH)           ALLOW IN    Anywhere                  
443                        ALLOW IN    Anywhere                  
80/tcp                     ALLOW IN    Anywhere                  
22/tcp (OpenSSH (v6))      ALLOW IN    Anywhere (v6)             
443 (v6)                   ALLOW IN    Anywhere (v6)             
80/tcp (v6)                ALLOW IN    Anywhere (v6)

Then we must also change the Caddyfile. We must insert a http://.. block to enable caddy to serve the ACME challenge.

#Caddyfile

http://bitwarden.rottlaender.eu {
    # Serve ACME Challenge at Port 80
    handle_path /.well-known/acme-challenge/* {
        root * /caddy_acme
        file_server
    }

    # Any other request redirect to HTTPS
    @notACME not path /.well-known/acme-challenge/*
    handle @notACME {
        redir https://bitwarden.rottlaender.eu{uri} permanent
    }
}

https://bitwarden.rottlaender.eu {
  log {
    level INFO
    output file /data/access.log {
      roll_size 10MB
      roll_keep 10
    }
  }

  # Use custom certificate and key
  tls /certs/fullchain.pem /certs/privkey.pem

  # ACME Challenge for Certbot Webroot
  handle_path /.well-known/acme-challenge/* {
    root * /caddy_acme
    file_server
  }

  # This setting may have compatibility issues with some browsers (e.g., attachment downloading on Firefox). Try disabling this if you encounter issues.
  encode zstd gzip

  # Admin path matcher
  @adminPath path /admin*
  
  # Basic Auth for admin access
  handle @adminPath {
    # If admin path require basic auth
    basicauth {
      superadmin $2y$05$HZukVJWhWMrT7qMOIenLkuf2n65bm/6n260TPXKb4Wn825JYlt5tO  # Password Hash
    }

    reverse_proxy vaultwarden:80 {
      header_up X-Real-IP {remote_host}
    }
  }

  # Everything else
  reverse_proxy vaultwarden:80 {
    header_up X-Real-IP {remote_host}
  }
}

Then we must also change the /etc/letsencrypt/renewal/bitwarden.rottlaender.eu.conf to configure certbot to use webroot instead of standalone. Here change the authenticator directive to webroot and we add the webrootpath. Everything else keep the same.

# renew_before_expiry = 30 days
version = 1.21.0
archive_dir = /etc/letsencrypt/archive/bitwarden.rottlaender.eu
cert = /etc/letsencrypt/live/bitwarden.rottlaender.eu/cert.pem
privkey = /etc/letsencrypt/live/bitwarden.rottlaender.eu/privkey.pem
chain = /etc/letsencrypt/live/bitwarden.rottlaender.eu/chain.pem
fullchain = /etc/letsencrypt/live/bitwarden.rottlaender.eu/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = 7844ed1ad487ff139e3adaa80aa7bbab
authenticator = webroot
webroot_path = /opt/vaultwarden/caddy_acme
server = https://acme-v02.api.letsencrypt.org/directory
renew_hook = /usr/local/bin/sync-certs.sh

And finally we must change the entry in the crontab. Before the depoly-hook we change the code to –webroot -w /opt/vaultwarden/caddyacme. This renewal will be executed daily at 03:30h. In case the certificates are still valid (no renewal required) then the deploy-hook will be skipped. Only in case a renewal must be executed the script /usr/local/bin/sync-certs.sh will be executed.

#crontab

30 3 * * * sh -c '/usr/bin/certbot renew --webroot -w /opt/vaultwarden/caddy_acme --deploy-hook /usr/local/bin/sync-certs.sh >> /var/log/certbot-renew.log 2>&1'

Create sync-certs.sh script and root crontab

This script copies renewed Let’s Encrypt certificates from the standard location to a custom destination /opt/vaultwarden/certs, sets strict permissions, assigns correct ownership, and restarts the Caddy container to apply the new certificates. At the end it reloads the Caddy container (via docker-compose restart) to apply new certs.

#!/bin/bash

# Variables
DOMAIN="<Domain>"
SRC="/etc/letsencrypt/live/$DOMAIN"
DEST="/opt/vaultwarden/certs"

# Check Source Directory
if [ ! -d "$SRC" ]; then
    echo "Certificate Path $SRC not found"
    exit 1
fi

# Check Destination Directory
if [ ! -d "$DEST" ]; then
    mkdir -p "$DEST"
    chown vaultwarden:vaultwarden "$DEST"
    chmod 700 "$DEST"
    echo "Target Path $DEST created"
fi

# Copy files (overwrite)
cp "$SRC/fullchain.pem" "$DEST/fullchain.pem"
cp "$SRC/privkey.pem" "$DEST/privkey.pem"

# set owner:group vaultwarden
chown vaultwarden:vaultwarden "$DEST/fullchain.pem"
chown vaultwarden:vaultwarden "$DEST/privkey.pem"

# set access (read write only vaultwarden)
chmod 600 "$DEST/privkey.pem"
chmod 600 "$DEST/fullchain.pem"

echo "[sync-certs] Certificates for $DOMAIN synced"

# successful sync of certificates – caddy re-start
echo "[sync-certs] re-start caddy ..."
cd /opt/vaultwarden
/usr/local/bin/docker-compose restart caddy

echo "[sync-certs] caddy reloaded new certificates"

Check Source Directory

if [ ! -d "$SRC" ]; then
    echo "Certificate Path $SRC not found"
    exit 1
fi

• What it checks: Whether the Let’s Encrypt certificate source directory for the domain exists.
• Why it’s needed:
  
  • Let’s Encrypt stores certificates as symlinks in /etc/letsencrypt/live/<domain>.
  • If this folder doesn’t exist, the script stops immediately to avoid copying from a missing or invalid source.
• Fail-safe: Prevents copying non-existent files, which would cause later commands to fail.
  

Check and Create Destination Directory

if [ ! -d "$DEST" ]; then
    mkdir -p "$DEST"
    chown vaultwarden:vaultwarden "$DEST"
    chmod 700 "$DEST"
    echo "Target Path $DEST created"
fi

• What it checks: Whether the destination directory for the copied certificates exists.
• If not:
  
  • It creates the directory (mkdir -p ensures parent paths are created if missing).
  • Sets secure permissions:
    
    • Owner: vaultwarden
    • Permissions: 700 – only the vaultwarden user can access the directory.
• Why this matters:
  
  • The vaultwarden container or process needs access to the certificates.
  • These permissions ensure only vaultwarden can read the certs, improving security.
    

Start, Stop and Check containers

Here are the most important docker-compose commands. Run these commands when you are in the directory where the docker-compose.yml file is and be sure that the logged in user is in the docker group (otherwise these commands work with sudo).

Here art the most important docker commands to check the status of containers.

Backup the vaultwarden data

To backup your database you must login to your remote server and navigate to the directory /opt/vaulwarden/vw-data.

cd /opt/vaultwarden/vw-data
ls -l /opt/vaultwarden/vw-data

drwxr-xr-x 2 root root    4096 Mai 14 07:29 attachments
-rw-r--r-- 1 root root 1437696 Mai 29 10:03 db_20250529_080308.sqlite3
-rw-r--r-- 1 root root 1437696 Mai 29 10:04 db_20250529_080448.sqlite3
-rw-r--r-- 1 root root 1445888 Aug  9 08:31 db_20250809_063113.sqlite3
-rw-r--r-- 1 root root 1552384 Aug 10 07:26 db.sqlite3
-rw-r--r-- 1 root root   32768 Sep  3 08:27 db.sqlite3-shm
-rw-r--r-- 1 root root  135992 Sep  3 08:27 db.sqlite3-wal
drwxr-xr-x 2 root root   16384 Sep  3 07:57 icon_cache
-rw-r--r-- 1 root root    1675 Mai 14 07:29 rsa_key.pem
drwxr-xr-x 2 root root    4096 Mai 14 07:29 sends
drwxr-xr-x 2 root root    4096 Mai 14 07:29 tmp

Before we run the backup here are some background infos.

Note: Pls. Be aware that you are currently navigating within your remote server not within the container vaultwarden. Under the service vaultwarden and volumes we defined in docker-compose.yml that the directory ./vw-data should be mounted into the container under the directory of /data. This mean that these directories are synchronized and you can check this by navigating within the container as follows.

To navigate within the container vaultwarden you can run the following commands.

#list your containers home directory
docker exec vaultwarden ls -l /

#list the data directory within your container
docker exec vaultwarden ls -l /data

With docker exec vaultwarden ls -l / you list your container root directory. Here you find among others the file vaultwarden which is basically the main program. With docker exec vaultwarden ls -l /data you list the files in /data on container side. You see exact the same files than in ./vw-data on your remote server.

All data of your vaultwarden service are stored in /data on container side. Here you have the database db.sqlite3 and various directories and other files.

To backup these data you run the program vaultwarden with the option backup.

#interactive mode
docker exec -it vaultwarden /vaultwarden backup

#standard mode
docker exec vaultwarden /vaultwarden backup
Backup to 'data/db_20250905_045937.sqlite3' was successful

The option exec -it is not necessary. It runs the command in interactive mode within the terminal.

This script:

1. Reads the database location and configurations on container side in /data.
2. Creates a backup of the SQLite database db.sqlite3, attachments, icons, etc.
3. Compresses everything as a .tar.gz archive.
4. Stores the archive in the /data directory on container side.
   

After executing this command the backup will be also be present in ./vw-data on your remote server.

Then you can download the backup from your remote server to your local computer. Note: Your local computer is in my case a Mac where I sit in front of and from where I connect to my remote server via ssh.

To download the backup you navigate on your local computer to the directory where you want to store your backup files. Then you run the scp command with the . at the end and download the backup files from your remote server into the current directory which should be your backup directory on your local computer.

#on your local computer
cd /Users/patrick/Software/docker/vaultwarden/backup

scp user@remote-host:/opt/vaultwarden/vw-data/<backupfile> .

sudo apt update
sudo apt install certbot

Then we run certbot.

sudo certbot certonly --standalone -d <yourDomain>

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): <yourEmail>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Requesting a certificate for <yourDomain>

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/<yourDomain>/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/<yourDomain>/privkey.pem
This certificate expires on 2025-07-25.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The Letsencrypt certificates are automatically stored by certbot in the directory /etc/letsencrypt. Pls. Check the permissions accordingly.

#700 for letsencrypt and owned by root
sudo ls -l /etc
drwx------ 9 root  root    4096 Mai 22 04:49 letsencrypt

#750 for archive and live and owned by root 
ls -l /etc/letsencrypt
drwx------ 3 root root 4096 Apr 26 10:16 accounts
drwxr-x--- 3 root root 4096 Apr 26 10:18 archive
-rw-r--r-- 1 root root  207 Nov 12  2021 cli.ini
drwx------ 2 root root 4096 Mai 16 07:15 csr
drwx------ 2 root root 4096 Mai 16 07:15 keys
drwxr-x--- 3 root root 4096 Apr 26 10:18 live
drwxr-x--- 2 root root 4096 Mai 16 07:15 renewal
drwxr-xr-x 5 root root 4096 Apr 26 10:16 renewal-hooks

#700 for the directory with the real certificates 
sudo ls -l /etc/letsencrypt/archive
drwx------ 2 root root 4096 Mai 16 07:15 <domain>

#750 for the directory with the symlinks
sudo ls -l /etc/letsencrypt/live
drwxr-x--- 2 root root 4096 Mai 16 07:15 <domain>

note: Permissions in Linux are set using the following scheme.

The following permissions are recommended:

The keys are set up the following structure:

• In /etc/letsencrypt/live/<domain> there are only symlinks that point to the real key files
• the real files are stored in /etc/letsencrypt/archive/<domain>
  

A symlink itself in /etc/letsencrypt/live/<domain> has no real permissions. It effectively inherits the access rights of the file it points to. However: ls -l shows symbolic permissions which, for symlinks, are usually lrwxrwxrwx. This isn’t problematic in itself, since access is always controlled by the target files.

sudo ls -l /etc/letsencrypt/live/<domain>
total 4
lrwxrwxrwx 1 root root  48 Mai 16 07:15 cert.pem -> ../../
lrwxrwxrwx 1 root root  49 Mai 16 07:15 chain.pem -> ../../
lrwxrwxrwx 1 root root  53 Mai 16 07:15 fullchain.pem -> ../../
lrwxrwxrwx 1 root root  51 Mai 16 07:15 privkey.pem -> ../../

Important are the permissions of the actual certificate files in the directory /etc/letsencrypt/archive/<domain>. The permissions should be set as follows.

sudo ls -l /etc/letsencrypt/archive/<domain>
total 32
-rw-r--r-- 1 root root 1858 Apr 26 10:18 cert1.pem
-rw-r--r-- 1 root root 1801 Apr 26 10:18 chain1.pem
-rw-r--r-- 1 root root 3659 Apr 26 10:18 fullchain1.pem
-rw------- 1 root root 1704 Apr 26 10:18 privkey1.pem

You should pay special attention to some other files.

/etc/letsencrypt/accounts/acme-<v02.api>/directory/<accountID>

The files meta.json, regr.json and private_key.json in the accounts directory should be owned by root and only root should have access. These files contain your Let’s Encrypt account credentials, especially the private account key, and are therefore highly sensitive.

sudo ls -l /etc/letsencrypt/accounts/acme-<v02.api>/directory/<accountID>
total 12
-rw------- 1 root root   85 Apr 26 10:18 meta.json
-r-------- 1 root root 1632 Apr 26 10:17 private_key.json
-rw------- 1 root root   80 Apr 26 10:17 regr.json

private_key.json: Contains your ACME account key – this is like a password for your certificate management.

• Read-only: The file must be readable by root so that Let’s Encrypt (or other tools like certbot) can communicate with this key to renew or issue certificates.
• Write-protected: Making the file unreadable even by root prevents it from being modified accidentally or through an unsafe operation. The private key should not be changed, as this could prevent access to your Let’s Encrypt account.

meta.json: Contains information such as the registered email address.

regr.json: Let’s Encrypt registration details.

A leak of these files would allow attackers to:

• Hijack your Let’s Encrypt account
• Issue certificates for any domain (if DNS access is compromised)
• Tamper or delete existing certificates
  

/etc/letsencrypt/csr and /etc/letsencrypt/keys directory

The files in the /etc/letsencrypt/csr and /etc/letsencrypt/keys directories should have the following permissions to ensure security.

Directory /etc/letsencrypt/csr stores Certificate Signing Requests (CSRs) generated by Certbot during certificate issuance or renewal. These are not used after issuance — they are kept for reference or debugging, not for active use.

sudo ls -l /etc/letsencrypt/csr
total 8
-rw------- 1 root root 936 Apr 26 10:18 0000_csr-certbot.pem
-rw-r--r-- 1 root root 936 Mai 16 07:15 0001_csr-certbot.pem

CSR files contain requests for certificate issuance and may contain sensitive information, such as the public key associated with a private key. Although the CSR file itself does not contain the private key, it can still pose informational risks because it represents an association with a private key.

Security measure: Access should be restricted to root to prevent an attacker from tampering with or accessing the CSR files.

Issue:After a renewal of the certificates on Mai 16 you can see above that the permissions to the file 0001_csr-certbot.pem has been set by certbot to 644. You can correct this with the following command.

sudo find /etc/letsencrypt/csr -name '*.pem' -exec chmod 600 {} +

sudo ls -l /etc/letsencrypt/csr
total 8
-rw------- 1 root root 936 Apr 26 10:18 0000_csr-certbot.pem
-rw------- 1 root root 936 Mai 16 07:15 0001_csr-certbot.pem

Directory /etc/letsencrypt/keys holds actual private keys used to sign CSRs and serve HTTPS. They must never be world-readable.

sudo ls -l /etc/letsencrypt/keys
total 8
-rw------- 1 root root 1704 Apr 26 10:18 0000_key-certbot.pem
-rw------- 1 root root 1704 Mai 16 07:15 0001_key-certbot.pem

Private keys are critical for security because they are used for authentication during certificate requests and for encrypting and decrypting data. The private key should be readable and writable only by root to prevent unauthorized access. Important: Any user with access to these key files could perform encryption operations or misuse certificates.

/etc/letsencrypt/renewal/<yourDomain>.conf

The file /etc/letsencrypt/renewal/<yourDomain>.conf contains the configuration for automatic certificate renewal and can contain sensitive data, such as the paths to private keys and certificates. Therefore, it is important to secure this file accordingly. The file should only be readable and writable by root, as it contains potentially sensitive information about the certificate configuration and the paths to private keys.

sudo ls -l /etc/letsencrypt/renewal
total 4
-rw------- 1 root root 606 Mai 16 07:15 <yourDomain>.conf

1. docker: Docker is a tool for building, running, and managing containers. A container is a lightweight, isolated environment that packages an application and all its dependencies (Docker application).
2. docker-compose: Docker Compose is a tool for defining and running multi-container Docker applications using a single file called docker-compose.yml.
3. node.js: Node.js is a JavaScript runtime that lets you run JavaScript outside the browser, typically on the server. It’s built on Chrome’s V8 engine, and it’s great for building fast, scalable network applications, like APIs or web servers.
4. npm: npm stands for Node Package Manager. It’s a tool that comes with Node.js and is used to Install packages (libraries, tools, frameworks), manage project dependencies and share your own packages with others.
5. curl: curl is a command-line tool used to send requests to URLs. It lets you interact with APIs or download content from the internet right from your terminal.
6. gnupg: GnuPG (or GPG, short for Gnu Privacy Guard) is a tool for encryption and signing data and communications. It uses public-key cryptography to encrypt, decrypt, sign, and verify files or messages.
7. ca-certificates: A collection of trusted root certificates used to validate HTTPS connections.
   

We must check if these packages are already installed on our system. Therefore we use the following commands in the terminal and see if there is any output. If there is no output the packages are not installed and we continue with the installation as described below.

# check the docker components
docker --version
docker-compose --version

#check the node and npm components
node --version
npm --version

#check if required dependencies are already installed
curl --version
gpg --version

In case some of these packages are already installed on your system you need to reduce the installation scope of the packages accordingly.

We expect to have no output after typing the above commands and go through the installation process step-by-step.

Step 1: Install node.js and npm from the standard Ubuntu resources.

Step 2:

• Prepare the system for secure downloads from Docker resources and
• install ca-certificates, curl and gnupg from standard Ubuntu resources

Step 3: Install Docker from Docker resources.

Step 4: Install docker-compose standalone from Docker resources.

Install Node.js and npm from Ubuntu Resources

Before we start with the installation we update and upgrade all packages. Node.js is available in Ubuntu’s repositories, so you can install it with the following commands.

sudo apt update
sudo apt upgrade

sudo apt install -y nodejs npm

Verify the installation:

node -v
npm -v

Prepare the System for secure Downloads from Docker

To prepare our system we ensure that ca-certificates, curl and gnupg are available on our system.

To be able to install the Docker packages from the external or not Ubuntu Docker repository apt must know where these resources are otherwise apt would install the packages from the Ubuntu repositories which would be the standard but not what we want. Therefore we must add the Docker repository to the apt tool. The complete process can be followed on the Docker Manual Pages.

When we add the Docker repository, the packages from that repository are digitally signed to ensure that these packages really come from docker. gnupg is a tool that allows our system to check these signatures against a trusted Docker GPG key. Therefore gnupg must be available on our system.

To make sure that the GPG key is available we must download the key from the Docker site. For the download we use the curl tool. Therefore curl must be available on our system.

We access the Docker site via HTTPS. Here ca-certificates comes into play. ca-certificates is a collection of trusted root certificates used to validate HTTPS connections. When downloading the Docker GPG key or accessing the Docker apt repository via HTTPS, Ubuntu checks the site’s SSL certificate against the collection of trusted root certificates. Therefore ca-certificates must be available on our system.

To check if ca-certificates is already installed we run the following command:

dpkg -l | grep ca-certificates

Note: The dpkg command stands for Debian Package Manager and is used to manage .deb packages on Debian-based systems like Ubuntu. It works at a lower level than apt, which is a higher-level package management tool that uses dpkg under the hood.

If it’s installed, you’ll see output like this:

ii ca-certificates 20230311ubuntu0.22.04.1 all Common CA..

In this case you do not need to install ca-certificates as describes below but it is highly recommended to update the collection of trusted root certificates before you continue.

sudo update-ca-certificates

We assume that we must install ca-certificates, curl and gnupg. First, we update the system package list to ensure everything is up-to-date on our system. This checks the installed packages for updates and upgrade the packages if required in one process. Then we install ca-certificates, curl and gnupg.

sudo apt update && sudo apt upgrade -y

sudo apt install -y ca-certificates curl gnupg

Add Docker GPG key:

To install the GPG keys from Docker we run the following command.

sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null
sudo chmod a+r /etc/apt/keyrings/docker.asc

Let’s break down the full set of commands step by step:

sudo install -m 0755 -d /etc/apt/keyrings ….

• sudo runs the command with superuser (root) privileges.
• install is used for copying files and setting permissions.

note: This command is only sudo install and not sudo apt install . If we use apt install then this installs software packages from Ubuntu’s package repositories. Example: sudo apt install docker-ce. It’s used to install applications.

Only install is a Unix command (part of coreutils) used to create directories, copy files, and set permissions in a single step. Example: sudo install -m 0755 -d /etc/apt/keyrings. It’s used to prepare the system, not install software. So this command creates the /etc/apt/keyrings folder with secure permissions, which is later used to store GPG keyring files (such as Docker’s signing key).

• -m 0755 sets file permissions:
  
  • 0755 means:
  • 1st (0) is just a numerical representation
  • 2nd (7) is for the Owner (root) having 1 x read (4) + 1 x write (2) + 1 x execute (1) = 7 permissions.
  • 3rd (5) is for the Group (root) having 1 x read (4) + 0 x write (2) + 1 x execute (1) = 5 permissions (no write).
  • 4th (5) is for the Others having 1 x read (4) + 0 x write (2) + 1 x execute (1) = 5 permissions (no write).
• -d tells install to create a directory
• /etc/apt/keyrings is the target directory where the Docker GPG key will be stored.

What it does?

• Ensures that the /etc/apt/keyrings directory exists.
• Sets the correct permissions (readable but not writable by non-root users).
• This is a security best practice to keep GPG keys safe from tampering.
  

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo tee /etc/apt/keyrings/docker.asc > /dev/null …

• curl a command-line tool to fetch files from a URL (we have installed it before)
• -fsSL flags to control curl behavior:
  
  • -f (fail silently on server http errors like 404 – site or resource not found).
  • -s (silent mode, no progress output).
  • -S (shows error messages if -s is used).
  • -L (follows redirects if the URL points elsewhere).
• https://download.docker.com/linux/ubuntu/gpg the URL for Docker’s GPG key file (the file name on the docker site is gpg).
• | (pipe) passes the downloaded data (the gpg file) to another command. In this case the data will be passed to the following sudo command.
• sudo tee /etc/apt/keyrings/docker.asc writes the key to before created directory /etc/apt/keyrings/docker.asc:
  
  • tee writes the output to a file (here it is docker.asc) while also displaying it in the terminal.
  • sudo ensures that the file is written with root permissions.
• > /dev/null redirects standard output to /dev/null to suppress unnecessary output. The tee command can also display and write at the same time, unless you silence it with > /dev/null.
  

note: sudo tee… runs with root permissions, so the file can be written even to protected directories such as /etc/apt/keyrings/ (we have set the permission to 0755. See above). You can also run the curl command with root permissions (sudo curl …) and then directly pass the output into a file with the -o option: sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc. This is suggested on the Docker Manaual Page but you can do it in both ways.

What it does?

• Downloads Docker’s official GPG key.
• Saves it securely in /etc/apt/keyrings/docker.asc.
• Ensures the key isn’t printed to the terminal.
  

sudo chmod a+r /etc/apt/keyrings/docker.asc

• sudo runs the command (in this case the chmod command) as root.
• chmod modifies file permissions.
  
  • a+r grants read (r) permission to all users (a).
• /etc/apt/keyrings/docker.asc the file whose permissions are being modified.

What it does?

• Ensures that all users (including apt processes) can read the GPG key.
• This is necessary so that apt can verify Docker package signatures when installing updates.
  

Previously, GPG files were stored in /etc/apt/trusted.gpg. This has changed.

Why Is This Necessary?

1. Security:

• Storing GPG keys in /etc/apt/keyrings/ instead of /etc/apt/trusted.gpg is a best practice.
• Prevents malicious modifications to package signatures.

1. Package Verification:

• The GPG key allows Ubuntu’s package manager (apt) to verify that Docker packages are genuine and not tampered with.

1. Future-proofing:

• Newer versions of Ubuntu prefer keys in /etc/apt/keyrings/ instead of the older /etc/apt/trusted.gpg.
  

Final Summary:

Add the Docker repository:

To install the docker resources to the apt sources list we run the following command.

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Let’s break down the command step by step:

echo „deb….“

The echo command output the text between the “….“. This is the APT repository entry for Docker. Let’s analyze its components:

• deb → Indicates that this is a Debian-based software repository.
• arch=$(dpkg –print-architecture):
  
  • dpkg –print-architecture dynamically retrieves the system architecture from your system (e.g., amd64, arm64).
  • This ensures that the correct package version for your system’s architecture is used.
• signed-by=/etc/apt/keyrings/docker.asc specifies the location of the docker GPG key docker.asc (we installed it before), which is used to verify the authenticity of packages downloaded from the repository.
• https://download.docker.com/linux/ubuntu the URL of Docker’s official repository.
• $(lsb_release -cs) dynamically fetches the codename of the Ubuntu version (e.g., jammy for Ubuntu 22.04).
  
  • This ensures that the correct repository for the current Ubuntu version is used.
• stable specifies that we are using the stable release channel of Docker.

| sudo tee /etc/apt/sources.list.d/docker.list

• The | (pipe) takes the output of echo and passes it to the tee command.
• sudo tee /etc/apt/sources.list.d/docker.list does the following:
  
  • tee writes the output to a file (/etc/apt/sources.list.d/docker.list).
  • sudo is required because writing to /etc/apt/sources.list.d/ requires root privileges.

> /dev/null

• The > /dev/null part discards the standard output of the tee command.
  
  • This prevents unnecessary output from being displayed in the terminal.
  • Without this, tee would both write to the file and display the text on the screen.
    

Install Docker from Docker Resources

Now, update the package list again and install Docker.

sudo apt update

sudo apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin

This command installs the following Docker key components using the apt package manager on Ubuntu (the -y option automatically answers „yes“ to all prompts, so the install runs without asking for confirmation).

docker-ce

• Docker Community Edition
• This is the core Docker Engine, the daemon that runs containers.
• Installs the Docker server that manages images, containers, volumes, and networks.

docker-ce-cli

• Docker Command-Line Interface
• This is the docker command you use in your terminal (e.g., docker run, docker ps, etc.).
• Separates the CLI from the engine so they can be updated independently.

containerd.io

• Container runtime
• A lightweight, powerful runtime for containers, used internally by Docker.
• Handles the actual low-level execution of containers.

docker-buildx-plugin

• BuildKit-powered Docker build plugin
• Adds docker buildx functionality for advanced builds, multi-arch images, and caching strategies.
• Useful when building complex container images.
  

Note: In some documentation you will see that the sudo apt install… command will include also the docker-compose-plugin. The docker-compose-plugin is not required here because we are using the docker-compose stand alone packed (see below). The docker-compose-plugin is integrated into the Docker CLI and can replace the docker-compose standalone binary. But we use the standalone version because of the lightweight minimal install, the backward compatibility and the easy and independent manual version control.

It is highly recommended to omit the docker-compose-plugin from your apt install command if you plan to install the standalone version of Docker Compose binary manually as we will do later. If you have both versions installed this can cause confusion, especially if scripts assume one or the other. Also, Docker might prioritize the plugin version in newer setups which might cause conflicts in our preferred standalone Docker Compose setup. The following table illustrates the problem because the command styles differ only very little.

In case the docker-compose-plugin has been installed on your system you can remove it with the following command:

sudo apt remove docker-compose-plugin

This removes the plugin version that integrates into the docker compose command. Later when we installed the standalone version of docker compose we use the commands instead of the space with a dash like docker-compose.

Verify that Docker is installed correctly:

sudo docker --version

Enable and start the Docker service:

sudo systemctl enable docker
sudo systemctl start docker

Test Docker by running the hello-world image.

sudo docker run hello-world

This command is a quick test to verify that Docker is installed and working correctly.

sudo

• Runs the command with superuser (root) privileges.
• Required unless your user is in the docker group.
• Docker needs elevated permissions to communicate with the Docker daemon (which runs as root).

docker

• The main Docker CLI (Command-Line Interface) tool.
• Used to interact with Docker Engine to manage containers, images, networks, volumes, etc.

run

• Tells Docker to create a new container and start it based on the hello-world image you specify.
  It does the following:
  
  • Pulls the image (if it’s not already downloaded).
  • Creates a new container from that image.
  • Starts and runs the container.
  • Outputs the result and then exits (for short-lived containers like hello-world).

hello-world

• This is the name of the Docker image.
• It’s an official image maintained by Docker, specifically designed to test Docker installations.
  

Install standalone docker-compose from Docker Resources

Installing the standalone docker-compose is useful when you:

• Need compatibility with legacy tools or scripts
• Want to control the exact version
• Prefer a lightweight, portable binary
  

The following command downloads the latest standalone docker-compose binary and saves it to a system-wide location.

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Let’s break down the command step by step:

sudo

• Runs the command with root privileges.
  
  • Required because /usr/local/bin is a protected directory that only root can write to.

curl

• A command-line tool used to download files from the internet.

-L

• Tells curl to follow redirects.
• GitHub uses redirects for release URLs, so this flag ensures the final binary is actually downloaded.

„https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)“

URL with Substitution. This ensures the correct binary is downloaded for your system. This is the dynamic download URL for the latest docker-compose release.

• $(uname -s) returns the operating system name (e.g., Linux, Darwin).
• $(uname -m) returns the architecture (e.g., x8664, arm64).

Example Output: https://github.com/docker/compose/releases/latest/download/docker-compose-Linux-x8664

-o /usr/local/bin/docker-compose

• Tells curl to write (-o option; output) the downloaded file to /usr/local/bin/docker-compose
• This is a standard location for user-installed binaries that are globally available in the system PATH.
  

After you run the command above you must do this. Give execution permissions. You’ll need to make the binary executable:

sudo chmod +x /usr/local/bin/docker-compose

And then check the version to confirm it worked:

docker-compose --version

Note: In some cases it might be necessary to switch to the docker compose plugin. This mean you must remove the standalone version from your system and install the docker compose plugin instead. Here is how you should proceed in such a scenario:

#find our where docker-compose has been installed
which docker-compose
/usr/bin/docker-compose

#remove file docker-compose 
sudo rm /usr/bin/docker-compose

#intstall docker compose plugin via docker
sudo apt install docker-compose-plugin

Final Verification

Check if everything is installed correctly:

docker --version
docker-compose --version
node -v
npm -v

Host-Docker-Setup for nodejs app behind nginx

The code of the nodejs app is explained in detail in the article Bootstrap Website running as nodejs app.

We have a simple website funtrails build in a one-page index.html with two sections: one section showing pictures of a Paterlini bike and one for showing pictures of a Gianni Motta bike. Each section containing an image gallery and text. The images are stored in an images directory.

  ├── .
	├── images
	├── index.html

Now we want to make this funtrails website run as a nodejs app behind a nginx reverse proxy. Both, the funtrails nodejs app and nginx should run in docker containers composed to work together. Therefore we crate the following file structure on the Host machine:

 ├── node
	├── funtrails
        │    ├─ views    
        │       ├── images
	│	├── index.html
	├── nginx

We copy all our web-content into the views directory under funtrails. The nodejs app is build in the mainfile app.js. All Dependencies for the nodejs app are defined the file package.json.

 ├── node
	├── funtrails
        │    ├─ app.js
    	│    ├─ package.json
        │    ├─ views    
        │       ├── images
	│	├── index.html
	├── nginx

In the Terminal go into the node/funtrails directory. Install the dependencies.

npm install

Then we have the following structure.

funtrails
├── app.js
├── node_modules
├── package.json
├── package-lock.json
└── views
    ├── images
    └── index.html

Go to node/funtrails. Run a test with the following command.

node app.js

nodejs funtrails demo app listening on port 8080!

Switch to your browser and hit http://localhost:8080 to see if all is working as expected. With Ctrl c in the terminal you can stop the app.

The nginx server will be started with the configuration of a nginx.conf file. The file nginx.conf will be created under nginx.

Note: This file nginx.conf is only for testing and will be changed when we add the SSL/TLS certificates. In this configuration below the nginx server will listen on port 80 and pass requests received from port 80 to the nodejs app listening on port 8080. Processing requests from port 80 is not state of the art as these connections are not encrypted. In production we need a port 443 connection with SSL/TLS for encrypted connections.

#nginx.conf

events {}

http {
	#Service node-app from docker-compose.yml
    upstream node_app {
        server node-app:8080;  
    }

    server {
        listen 80;

        location / {
            proxy_pass http://node_app;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

events {}

• This is a required block in NGINX configuration, even if empty.
• It handles connection-related events (like concurrent connections), but you don’t need to configure it unless you have advanced use cases.

http {

• Starts the HTTP configuration block — this is where you define web servers, upstreams, headers, etc.

upstream

• Defines a group of backend servers (can be one or many).
• node_app is just a name.
• Inside: server node-app:8080; means:
  
  • Forward traffic to the container with hostname node-app
  • Use port 8080 (that’s where your nodejs app listens)
  • node-app should match the docker-compose service name (node-app must be declared in your docker-compose.yml which will be explained below).
  • This lets you use proxy_pass http://nodeapp later, instead of hardcoding an IP or port.

server

• Defines a virtual server (a website or domain).
• listen 80; tells NGINX to listen for HTTP (port 80) traffic.

location

• Defines a rule for requests to / (the root URL of your site).
• You could add more location blocks for /api, /images, etc. if needed.
• Inside:
  
  • proxy_pass:
    
    • proxy_pass http://node_app; Tells NGINX to forward requests to the backend defined in upstream nodeapp
    • So: if you go to http://yourdomain.com/ NGINX proxies that to http://node-app:8080
• proxysetheader (see table)

The $variables in nginx.conf are built-in variables that NGINX provides automatically. They are dynamically set based on the incoming HTTP request. So these variables come from the NGINX core HTTP module and you don’t need to define them or import anything. They are always available in the config.

Here’s what each one is and where it comes from:

$host

• The value of the Host header in the original HTTP request.
• Example: If the user visits http://example.com, then $host is example.com.
• Use case: Tells the backend app what domain the client used — useful for apps serving multiple domains.

$remote_addr

• The IP address of the client making the request.
• Example: If someone from IP 203.0.113.45 visits your site, this variable is set to 203.0.113.45.
• Use case: Useful for logging, rate limiting, or geolocation in the backend app.

$proxy_add_x_forwarded_for

• A composite header that appends the client’s IP to the existing X-Forwarded-For header.
• Use case: Maintains a full list of proxy hops (useful if your request goes through multiple reverse proxies).
• If X-Forwarded-For is already set (by another proxy), it appends $remote_addr to it; otherwise, it sets it to $remote_addr.

$scheme

• The protocol used by the client to connect to NGINX — either http or https.
• Example: If the user visits https://example.com, then $scheme is https.
• Use case: Lets your backend know whether the original request was secure or not.
  

Then we have the following structure.

 ├── node
	├── funtrails
	│    ├─ app.js
    	│    ├─ package.json
        │    ├─ views    
        │       ├── images
	│	├── index.html
	├── nginx
            ├── nginx.conf

Create Docker Image and Container for nodejs app

This is to create the Docker Image with docker build. Then we run the Container from the image with docker run to test if everything is working as expected. If everything goes well we can go ahead with the nginx configuration and then with the composition of all together with docker-compose.

The dockerization process of the nodejs app in node/funtrails directory is controlled by the Dockerfile which will be created in node/funtrails. The dockerization process has the following steps.

1. Image creation
2. Container creation from the image
   

The Container can then be started, stopped and removed using terminal commands.

Go into in node/funtrails.

To get an Overview and check the Docker status of the system.

List all images. No images on your system.

sudo docker image ls

REPOSITORY   TAG       IMAGE ID   CREATED   SIZE  

List all Containers. As expected no Containers on the system.

sudo docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

List an Overview about Docker Images and Containers on your system. As expected no Containers and no Images on the system.

sudo docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     16        0         47.39MB   47.39MB

Create a Dockerfile in node/funtrails/Dockerfile. This file is required to build the image and run the Container.

#Dockerfile

#Image Build
#Install nodejs 12.22.9 for the Container
FROM node:12.22.9

#Set workdirectory for the Container
WORKDIR /home/node/app

#Change Owner and Group for Container Workdirectory to node
RUN chown node:node /home/node/app

#Run the Container with User node
USER node

#Copy all files from HOST Dir to the Container workdirectory 
COPY --chown=node:node . .

#(After COPY) Run the command to create Image for the Container
RUN npm install

#Container Start
#Open Port 8080 when the Container starts
EXPOSE 8080

#RUN the command when the Container starts
CMD [ "node", "app.js" ]
 

Create a .dockerignore file in node/funtrails/.dockerignore. The hidden dockerignore file exclude only files from the Host that will be copied into the image for the Container with the command COPY . .

#.dockerignore
node_modules

Note: In case you copy files from the Host into the image directly with I.e. COPY <file.1> <file.2> then file.1 and file.2 would be copies even if they would be listed in dockerignore.

We have the following structure on the Host machine.

funtrails
├─ Dockerfile
├─ .dockerignore
├── app.js
├── node_modules
├── package.json
├── package-lock.json
└── views
    ├── images
    └── index.html

Still be in node/funtrails.

Build the Docker image from the Dockerfile with the image name node-demo. The dot (.) at the end set the current directory on the Host machine for the docker command. This is location where docker is looking for the Dockerfile to build the image.

sudo docker build -t node-demo .

List all Docker images. 1 images just created.

sudo docker image ls

REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
node-demo    latest    c353353f045e   22 seconds ago   944MB

Run the Container from the image with the name node-demo and give the Container the name funtrails-solo-demo.

sudo docker run -d -p 8080:8080 --name funtrails-solo-demo node-demo

List all Docker Containers with the option -a. 1 Container with the name funtrails-solo-demo running from the image node-demo.

sudo docker ps -a

Access the running app on port 8080.

sudo curl http://localhost:8080

If everything went well you get a feedback in the terminal showing the HTML code. In this case the Test was successful.

Stop the running Docker Container with the name funtrails-solo-demo.

sudo docker stop funtrails-solo-demo

List all Containers with the option -a. 1 Container from the image node-demo with the name funtrails-solo-demo is EXITED.

sudo docker ps -a

List all images. Still 1 image available.

sudo docker image ls

REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
node-demo    latest    c353353f045e   36 hours ago   944MB

List an Overview about Docker Images and Containers on the system. 1 active Image and 1 not Active Container. Status of the Container is EXITED as we have seen above.

sudo docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          1         1         944MB     0B (0%)
Containers      1         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     18        0         47.39MB   47.39MB

To clean up your system use the following commands.

The full clean up (be careful).

sudo docker system prune -a

sudo docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          0         0         0B        0B
Containers      0         0         0B        0B
Local Volumes   0         0         0B        0B
Build Cache     0         0         0B        0B

Configure docker-compose

Go back to the node directory and create a docker-compose.yml file there.

Then we have the following structure.

 ├── node
     ├── docker-compose.yml
     ├── funtrails
     │	   ├─ Dockerfile
     │	   ├─ .dockerignore
     │     ├─ app.js
     │     ├─ node_modules
     │	   ├─ package.json
     │	   ├─ package-lock.json
     │     ├─ views    
     │         ├── images
     │         ├── index.html
     ├── nginx
           ├── nginx.conf

docker-compose is a tool that helps you define and run multi-container Docker applications using a YAML file. Instead of running multiple docker run commands, you describe everything in one file and start it all with the command docker-compose up.

Create docker-compose.yml with the following content.

#docker-compose.yml

services:
  funtrails:
    build: ./funtrails
    container_name: funtrails 
    networks:
      - funtrails-network

  nginx:
    image: nginx:latest
    container_name: nginx-proxy
    ports:
      - "80:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - funtrails
    networks:
      - funtrails-network

networks:
  funtrails-network:
    driver: bridge

services

This section defines containers that make up your app

• funtrails
  
  • build: ./funtrails
    Builds the image from the Dockerfile inside the ./funtrails directory.
  • container_name: funtrails
    Names the container funtrails instead of a random name.
  • networks: funtrails-network
    Connects the container to a custom user-defined network.
• nginx
• image: nginx:latest
  Uses the official latest NGINX image.
• container_name: nginx-proxy
  Container will be named nginx-proxy.
• ports: „80:80“
• Exposes Host port 80 to Container port 80.
• volumes:
  Mounts your local nginx.conf into the container, read-only (:ro).
• depends_on: funtrails
  Ensures funtrails is started before nginx.
• networks: funtrails-network
  Both services are in the same network, so they can communicate by name.

networks

• Creates a custom bridge network named funtrails-network.
• Ensures containers can resolve each other by name (funtrails, nginx).
  

Note: We are using the official NGINX image directly (image: nginx:latest). This image is prebuilt and includes everything NGINX needs to run.

We don’t need to write a custom Dockerfile because we don’t want to:

• Add extra modules
• Customize the image beyond just the config
• Install additional tools
• Include SSL certs directly, etc.

Instead, we simply mount our own nginx.conf into the container using a volume. This tells Docker Use the official NGINX image, but replace its config file with mine. We would use a Dockerfile in the nginx directory if we need to build a custom NGINX image, for example to copy SSL certs directly into the image.

Example:

FROM nginx:latest
COPY ./nginx.conf /etc/nginx/nginx.conf
COPY ./certs /etc/nginx/certs

But for most use cases like reverse proxying a nodejs app, just mounting your own config file is perfectly sufficient and simpler.

Note:We integrate SSL in the next chapter using free Lets Encrypt certificates.

Integrate SSL certificates – free Lets Encrypt

To integrate SSL we need to do the following steps:

1. Prepare your Domain
2. Install certbot
3. Create Lets Encrypt SSL certificates
4. Adapt your node/docker-compose.yml
5. Adapt your node/nginx/nginx.conf
6. Create a cron-Job to renew SSL certificates
   

prepare the domain

You must own a domain like example.com and you must have access to your DNS-servers to adapt the A-Record. Here in my example I create a subdomain funtrails.example.com and create on my DNS an A-Record for funtrails.example.com that point to the servers IP-Adress.

install certbot

To install our certificates for SSL we use a tool called certbot. We install certbot with apt on our Linux machine.

sudo apt update
sudo apt install certbot

create Lets Encrypt SSL certificates

We create the SSL certificates with certbot.

sudo certbot certonly --standalone -d funtrails.example.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices)
 (Enter 'c' to cancel): <your-email>@funtrails.example.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf. You must
agree in order to register with the ACME server. Do you agree?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
(Y)es/(N)o: Y

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Account registered.
Requesting a certificate for funtrails.example.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/funtrails.example.com/fullchain.pem

Key is saved at:         /etc/letsencrypt/live/funtrails.example.com/privkey.pem

This certificate expires on 2025-07-25.

These files will be updated when the certificate renews.

Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

To use SSL you need

• a server certificate (e.g. certificate.crt)
• a private key (e.g. private.key) and
• the CA certificate (e.g. ca.crt).

certbot create these file in the following directory on your Host server.

/etc/letsencrypt/live/funtrails.example.com

sudo ls -l /etc/letsencrypt/live/funtrails.example.com

cert.pem -> ../../archive/funtrails.example.com/cert1.pem
chain.pem -> ../../archive/funtrails.example.com/chain1.pem
fullchain.pem -> ../../archive/funtrails.example.com/fullchain1.pem
privkey.pem -> ../../archive/funtrails.example.com/privkey1.pem

The translation to the standard is as follows.

adapt node/docker-compose.yml

The docker-compose.yml will be adapted as follows.

services:
  funtrails:
    build: ./funtrails
    container_name: funtrails
    networks:
      - funtrails-network

  nginx:
    image: nginx:latest
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    depends_on:
      - funtrails
    networks:
      - funtrails-network

networks:
  funtrails-network:
    driver: bridge

We create a bridge network with the name funtrails-network and both services are running in this network. This is important to reach all services by their container name(s).

The funtrails service will rebuild from the Dockerfile in ./funtrails. For the nginx service the nginx-image will be loaded from the Docker resources in the latest version. For the nginx image it is defined that the Host ports 80 and 443 will be mapped into the Container ports 80 and 443. When the Container is started we mount the Host files ./nginx/nginx.conf and the SSL certificates unter /etc/letsencrypt into the Container. Both will be loaded with read only!

...
volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
...

With the dependson directive we declare that first the funtrails service must be started and then nginx.

adapt node/nginx/nginx.conf

The file will be adapted as follows.

events {
  worker_connections 1024; 
}

http {

    server {
      listen 80;
      server_name funtrails.example.com;

      # Redirect HTTP -> HTTPS
      return 301 https://$host$request_uri;
    }

   server {
    listen 443 ssl;
    server_name funtrails.example.com;

    ssl_certificate /etc/letsencrypt/live/funtrails.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/funtrails.example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://funtrails:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
  }
}

The events{} Block is a required block in NGINX configuration, even if empty. It handles connection-related events (like concurrent connections), but you don’t need to configure it unless you have advanced use cases. Here I configured 1024 concurrent connections.

Within the http block we have 2 virtual server-blocks. The first server-block define that server funtrails.example.com is listening to port 80 (HTTP) but all requests to this port will immediately be redirected to port 443 (HTTPS). The second server-block define that server funtrails.example.com is listening also to port 443 (HTTPS) followed by the location of the SSL certificate and SSL key on our local Host and the protocol definition.

The location-block define a rule for requests to / (the root URL of your site). You could add more location blocks i.e. for /api, /images, etc. if needed. In this config, we are skipping the upstream block and directly writing proxypass. The proxypass tells NGINX to forward requests to port 8080 of the backend service defined in docker-compose.yml. This backend service in docker-compose.yml is defined with the container_name directive which is set to funtrails.

...
services:
  funtrails:
	build: ./funtrails
    container_name: funtrails
	networks:
	   - funtrails-network
... 

Docker Compose creates an internal Docker network funtrails-network, and all services can reach each other by their service name(s) as hostname(s). So nginx can resolve funtrails because it’s part of the same Docker network (no need for a manual upstream block).

These other $variables come from NGINX’s core HTTP module, so we don’t need to define them. They are always available in the config.

$host

• What it is: The value of the Host header in the original HTTP request.
• Example: If the user visits http://example.com, then $host is example.com.
• Use case: Tells the backend app what domain the client used — useful for apps serving multiple domains.

$remoteaddr

• What it is: The IP address of the client making the request.
• Example: If someone from IP 203.0.113.45 visits your site, this variable is set to 203.0.113.45.
• Use case: Useful for logging, rate limiting, or geolocation in the backend app.

$proxyaddxforwardedfor

• What it is: A composite header that appends the client’s IP to the existing X-Forwarded-For header.
• Use case: Maintains a full list of proxy hops (useful if your request goes through multiple reverse proxies).
• How it works: If X-Forwarded-For is already set (by another proxy), it appends $remoteaddr to it; otherwise, it sets it to $remoteaddr.

$scheme

• What it is: The protocol used by the client to connect to NGINX — either http or https.
• Example: If the user visits https://example.com, then $scheme is https.
• Use case: Lets your backend know whether the original request was secure or not.
  

Create a cron-Job to renew Lets Encrypt SSL certificates

Lets Encrypt SSL Certificates must be renewed after 90 days. certbot can renew your certificates. To automize the renewal you can create a cronjob on your Host machine.

Note: Sometimes cron doesn’t know where docker-compose is located (because the environment variables are missing). Therefore, it’s safer to use the full paths in crontab. You check the relevant paths as follows:

which docker-compose
/usr/bin/docker-compose

which certbot
/usr/bin/certbot

The create a cronjob in crontab of the user root (use sudo):

sudo crontab -e 

0 3 * * * 	/usr/bin/certbot renew --quiet && /usr/bin/docker-compose restart nginx

With sudo crontab -e you create a crontab for the user root. All commands within the root crontab will be executed with root privileges.

certbot renew checks all certificates for expiration dates and automatically renews them.

docker-compose restart nginx ensures that nginx is reloaded so that it can apply the new certificates. Otherwise, nginx would still be using old certificates even though new ones are available. With the command you call docker-compose restart <service-name>. Here you specify the service name from docker-compose.yml not the container name.

Note: In case you would call crontab -e (without sudo) you would edit your own user crontab. This crontab then runs under your user, not as root and the tasks in crontab run unter this normal user. When you renew SSL certificates using certbot renew this job must write into the directories under /etc/letsencrypt/ on your machine. But these directories are owned by root. So you cannot write to the directories when the crontab runs under a normal user. One might then think that the commands in the crontab of a normal user should be executed with sudo. If a crontab job of your normal user (i.e. patrick) is running and sudo is used in the command, then sudo will attempt to prompt for the password. But there is no terminal in crontab where you can enter a password. Therefore, the command will fail (error in the log, nothing happens). Therefore it is essential here to edit the crontab of the user root with sudo crontab -e.

Finally you can check the crontab of your own or the crontab as root as follows.

crontab -l
no crontab for patrick

sudo crontab -l
# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').
# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

0 3 * * * /usr/bin/certbot renew --quiet && /usr/bin/docker-compose restart nginx
 

You can check the logs for the renewal process using the following command.

sudo cat /var/log/letsencrypt/letsencrypt.log

Start the Containers with docker-compose

Navigate to the directory with docker-compose.yml. Then use the following commands.

sudo docker-compose build

sudo docker-compose up -d

The command docker-compose build reads the Dockerfile for each service defined in docker-compose.yml and builds the Docker image accordingly. The command docker-compose up -d run the container(s) and the network. This starts all services defined in the docker-compose.yml and links them via the defined docker network. The -d flag runs the containers in the background (detached mode).

Then you can check the status using the following commands.

sudo docker-compose ps

sudo docker ps

Here is an overview of the most important commands.

How to manage Changes made to the application code

When we make changes to the app code i.e. in node/funtrails/app.js or in node/funtrails/Dockerfile we need to rebuild the image for the funtrails service defined in node/docker-compose.yml. In such a change scenario it is not necessary to stop the containers with docker-compose down before you rebuild the image with docker-compose build.

You can rebuild and restart only the funtrails service with the following commands.

docker-compose build funtrails

docker-compose up -d funtrails

This will:

• Rebuild the funtrails image
• Stop the old funtrails container (if running)
• Start a new container using the updated image
• Without affecting other services like nginx
  

Installation

So lets install nginx with homebrew. In case you dont know what homebrew is and need to install it on your Mac read my article on Digitaldocblog.

brew install nginx

brew services start nginx

brew services stop nginx

• nginx has been installed in /usr/local/Cellar/nginx/1.17.7
• nginx configuration file is /usr/local/etc/nginx/nginx.conf
  

Configuration

Setup the core nginx configuration file nginx.conf in your environment:

# /usr/local/etc/nginx/nginx.conf 

# Main Context
worker_processes  auto;
error_log  /usr/local/etc/nginx/logs/error.log;
pid /usr/local/etc/nginx/run/nginx.pid;  

# Events Context
events {
    worker_connections  1024;
}

# HTTP Context
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    access_log  /usr/local/etc/nginx/logs/access.log;

    include servers/*;
}

The Main Context is the most general context and is the only context that is not surrounded by curly braces. The Main Context is placed at the beginning of the core nginx configuration file. The directives of the Main Context cannot be inherited in any other context and can not be overridden.

The Main Context is used to configure the basic details of an application. Some common details that are configured in the Main Context are:

• The directive worker_processes defines the number of worker processes. The number depend on the number of CPU cores and the default value is 1. The value auto will autodetect the number of worker processes. The directive tell the virtual servers defined in the servers folder (see below) know how many workers can be created once they are bound to the IP(s) and port(s). It is common practice to run 1 worker process per core. Anything above this won’t hurt your system, but it will leave idle processes usually just lying about
• The directive error_log define the default error file for the entire application
• The directive pid define the file to save the main process ID.
  

The Event Context as also the HTTP-Context are childs of the Main Context and therefore lower contexts. Lower Contexts handle the request, and the directives at this level and control the defined defaults for every virtual server.

The Events Context define global options for connection processing and is contained within the main context. There can be only one Event Context defined within Nginx configuration. Within the Events Context we define the worker_connections to tells our worker processes how many connections can simultaneously be served by Nginx. The default value is 768. Considering that every browser usually opens up at least 2 connections per server, this is why we need to adjust our worker connections to its full potential. Therefore we define 1024 worker connections.

The HTTP Context is used to hold the directives for handling HTTP or HTTPS traffic.

• with the include directive we includes another file, or files matching the specified mask, into configuration.
• with the default_type directive we define the default MIME type of a response. When a web server sends HTTP traffic back to the client (response), it usually adds an IANA media type (formerly known as MIME types), or content type, to the packet header that shows what kind of content is in the packet. The HTTP header on the data stream contains this content type. It is added by the web server before the data is sent.
• using the directive sendfile enables or disables the use of the function sendfile(). By default, NGINX handles file transmission itself and copies the file into the buffer before sending it. Enabling the sendfile directive eliminates the step of copying the data into the buffer and enables direct file transmission.
• the directive keepalive_timeout sets a timeout during which a keep-alive client connection will stay open on the server side.
  

Finally the include directive within the HTTP Context tell nginx to load all virtual servers from the files in /usr/local/etc/nginx/servers in the HTTP Context.

Then I create 2 serverfiles in /usr/local/etc/nginx/servers.

1. A serverfile to run a virtual http server on localhost port 7445
2. A serverfile to run a virtual http reverse proxy server on localhost port 7444
   

Patricks-Macbook Pro:servers patrick$ ls -l

total 24
-rw-r--r--@ 1 patrick  admin  188 15 Nov 06:12 patrick_7445
-rw-r--r--@ 1 patrick  admin  452 15 Nov 07:39 reverse_7444

Patricks-Macbook Pro:servers patrick$ 

The Server Context is inside these serverfiles. In general the Server Context is defined within the HTTP Context.

The Server Context define the virtual host settings. There can be multiple Server Context definitions inside the HTTP Context. The directives inside the Server Context handle the processing of requests for resources associated with a particular domain or IP address.

The directives in this Server Context can override many of the directives that may be defined in the HTTP Context, including the document root, logging, compression, etc. In addition to the directives that are taken from the HTTP Context, we can also configure files to try to respond to requests, issue redirects, and rewrites, and set arbitrary variables.

# /usr/local/etc/nginx/servers/reverse_7444

server {

    listen 7444;
    server_name patrick.local;

    location / {
       proxy_pass http://localhost:7445;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}

To connect to your localhost using patrick.local the dns settings in etc/host must be changed accordingly.

# /private/etc/hosts

192.168.178.20 servtest.rottlaender.lan

# /usr/local/etc/nginx/servers/patrick_7445

server {
     listen       7445;
     server_name  localhost;

     location / {
         root   /Users/patrick/Sites/www/nginx/patrick_7445;
         index  index.html index.htm;
    }
}

TLS/SSL for nginx

SSL/TLS works by using the combination of a public certificate and a private key.

The SSL key (private key) is kept secret on the server. It is used to encrypt content sent to clients.

The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

create private key

Patricks-MBP:express-security patrick$ cd /usr/local/etc/nginx

Patricks-MBP:nginx patrick$ ls -l
total 144
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  8 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  8 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types.default
-rw-r--r--  1 patrick  admin  3106 15 Mai 11:19 nginx.conf
-rw-r--r--  1 patrick  admin  2680  8 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params.default
drwxr-xr-x  3 patrick  admin    96 21 Jan 06:02 servers
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  8 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ mkdir ssl

Patricks-MBP:nginx patrick$ ls -l
total 152
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  8 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  8 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  8 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  8 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  8 Apr 13:18 mime.types.default
-rw-r--r--@ 1 patrick  admin   373 18 Mai 13:38 nginx.conf
-rw-r--r--  1 patrick  admin  2680  8 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--@ 1 patrick  admin  1390 17 Mai 09:19 nginx_old.conf
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  8 Apr 13:18 scgi_params.default
drwxr-xr-x  5 patrick  admin   160 18 Mai 13:20 servers
drwxr-xr-x  4 patrick  admin   128 16 Mai 07:41 ssl
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  8 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  8 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ cd ssl

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl genrsa -out privateKey.pem 4096

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem

create certificate signing request (CSR)

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl req -new -key privateKey.pem -out csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  1740 16 Mai 07:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem

In case I would like to request an official certificate I must send this csr to the certificate authority. This authority would then create an authority signed certificate from the CSR and send it back to me.

This step is done by ourselves and this is the reason why we create a self signed certificate. This self signed certificate is not an official certificate and not trusted by any browser. It is not useful to use a self signed certificate in production because it produces error messages in the browsers. But for local development a self signed certificate is ok.

So create the self signed certificate. The csr file can then be removed.

create the self signed certificate

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in csr.pem -out selfsignedcertificate.pem -req -signkey privateKey.pem -days 365

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  1740 16 Mai 07:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 07:39 selfsignedcertificate.pem

Patricks-MBP:ssl patrick$ rm csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  3247 16 Mai 07:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 07:39 selfsignedcertificate.pem
 

show certificate details

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in selfsignedcertificate.pem -text -noout

renew self signed certificate

So let’s assume 1 year has passed by and now we want to renew this certificate selfsignedcertificate.pem. Now you may either generate a new CSR or export the CSR from the existing expired self-signed certificate.

First let’s check the validity of the existing certificate.

patrick@MacBookPro-Patrick ssl % ls -l                                                          
total 32
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % openssl x509 -noout -text -in selfsignedcertificate.pem | grep -i -A2 validity
        Validity
            Not Before: May 16 05:39:28 2020 GMT
            Not After : May 16 05:39:28 2021 GMT

patrick@MacBookPro-Patrick ssl %

So our certificate selfsignedcertificate.pem expired on 16 Mai 2021 and today is 12 Mar 2022.

It is recommended to export the CSR as you don’t have to worry about giving the same Distinguished Name information. Although this information can be collected from the certificate itself. For us, we will export the CSR from the certificate itself.

patrick@MacBookPro-Patrick ssl % ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % openssl x509 -x509toreq -in selfsignedcertificate.pem  -signkey privateKey.pem -out new-csr.pem          
Getting request Private Key
Generating certificate request

patrick@MacBookPro-Patrick ssl % ls -l
total 32
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl %

Now that we have our private key privateKey.pem and the new CSR new-csr.pem, we can renew our certificate selfsignedcertificate.pem. You should not get confused with the term „renew“, we are not extending the expiry of the certificate, instead we are just creating a new self signed certificate using the CSR new-csr.pem from the existing private key privateKey.pem. The Signature Modules are the same and our application considers the certificate as same instead of new.

patrick@MacBookPro-Patrick ssl % ls -l
total 32
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % openssl x509 -req -days 365 -in new-csr.pem -signkey privateKey.pem -out new-selfsignedcertificate.pem
Signature ok
subject=/C=DE/ST=Bayern/L=Munich/O=Digitaldocblog/CN=localhost/emailAddress=p.rottlaender@icloud.com
Getting Private key

patrick@MacBookPro-Patrick ssl % ls -l
total 40
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  1980 12 Mär 08:30 new-selfsignedcertificate.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % 

Lets check the validity of the new certificate new-selfsignedcertificate.pem.

patrick@MacBookPro-Patrick ssl % openssl x509 -noout -text -in selfsignedcertificate.pem | grep -i -A2 validity
        Validity
            Not Before: May 16 05:39:28 2020 GMT
            Not After : May 16 05:39:28 2021 GMT

patrick@MacBookPro-Patrick ssl % openssl x509 -noout -text -in new-selfsignedcertificate.pem | grep -i -A2 validity
        Validity
            Not Before: Mar 12 07:30:52 2022 GMT
            Not After : Mar 12 07:30:52 2023 GMT

patrick@MacBookPro-Patrick ssl % 

While our certificate selfsignedcertificate.pem expired on 16 Mai 2021 the new certificate new-selfsignedcertificate.pem expires on 12 Mar 2023.

The we rename the new-selfsignedcertificate.pem to selfsignedcertificate.pem to ensure

patrick@MacBookPro-Patrick ssl % mv selfsignedcertificate.pem old-selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl % mv new-selfsignedcertificate.pem selfsignedcertificate.pem
patrick@MacBookPro-Patrick ssl % ls -l
total 40
-rw-r--r--  1 patrick  admin  6279 12 Mär 08:09 new-csr.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai  2020 old-selfsignedcertificate.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai  2020 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 12 Mär 08:30 selfsignedcertificate.pem

patrick@MacBookPro-Patrick ssl %

Finally we restart nginx service.

patrick@MacBookPro-Patrick ssl % brew services list                                        
Name                  Status  User    File
mongodb-community@4.4 started patrick ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community@4.4.plist
nginx                 started patrick ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
patrick@MacBookPro-Patrick ssl % brew services stop nginx
Stopping `nginx`... (might take a while)
==> Successfully stopped `nginx` (label: homebrew.mxcl.nginx)
patrick@MacBookPro-Patrick ssl % brew services start nginx
==> Successfully started `nginx` (label: homebrew.mxcl.nginx)
patrick@MacBookPro-Patrick ssl % brew services list       
Name                  Status  User    File
mongodb-community@4.4 started patrick ~/Library/LaunchAgents/homebrew.mxcl.mongodb-community@4.4.plist
nginx                 started patrick ~/Library/LaunchAgents/homebrew.mxcl.nginx.plist
patrick@MacBookPro-Patrick ssl %

configure nginx with SSL

Any default configurations are done in nginx.conf. In this configuration we define a local webserver and a reverse proxy server.

The ip 127.0.0.1 is resolved in hosts file to localhost.

In the default configuration we define that any request to localhost on port 80 is redirected to https/ssl (443 ssl). This is a typical production configuration to enforce htps/ssl.

// /usr/local/etc/nginx.conf
// default configuration

worker_processes  1;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    
	# default server configuration
    server {
      	listen 80 default_server;
    	listen [::]:80 default_server;
    	
    	# default server your.domain.com
    	server_name localhost;
    	
    	# enforce 301 redirect to ssl port on your default server
    	return 301 https://$host$request_uri;
    }
    
	# In servers directory we define the servers running on this machine
    include servers/*;
}

The local webserver is bound to 127.0.0.1 and this ip is resolved to localhost. localhost is listening on port 443. This local webserver is using https/ssl. Any ssl configuration is done via the ssl_* directives in the server block.

In the location block we define the root directory of the local webserver and the index files. Here the local webserver process https client requests and send responses back to the client. So when a client request the route / then the local webserver render index.htm or index.html and respond to the client.

// /usr/local/etc/servers/localweb

server {
     listen       443 ssl;
     # here you would define in production your.domain.com
     server_name  localhost;

         ssl_certificate      ssl/selfsignedcertificate.pem;
         ssl_certificate_key  ssl/privateKey.pem;
         ssl_session_cache    shared:SSL:1m;
         ssl_session_timeout  5m;
         ssl_ciphers  HIGH:!aNULL:!MD5;
         ssl_prefer_server_ciphers  on;

     location / {
         root   /Users/patrick/Sites/prod/digitaldocblog-V2/public;
         index  index.html index.htm;
    }
}

The reverse proxy server is bound to ip 192.168.178.20. The ip 192.168.178.20 is resolved to servtest.rottlaender.lan in hosts file. This reverse proxy server is using https/ssl. Any ssl configuration is done via the ssl_* directives in the server block.

The node express server is listening on localhost port 3300. This express server is using http.

Any request to 192.168.178.20 or servtest.rottlaender.lan to port 3000 will be passed to application server listening on localhost port 3300.

So any client https request for servtest.rottlaender.lan on route / will be directly handed over to the express server. The express server will then process the request and send the response back to servtest.rottlaender.lan which will then respond 1:1 to the client.

// /usr/local/etc/servers/reverse

server {
    listen      3000 ssl;
    
	# /private/etc/hosts
	# 192.168.178.20	servtest.rottlaender.lan
	
    server_name servtest.rottlaender.lan;

    ssl_certificate      ssl/selfsignedcertificate.pem;
    ssl_certificate_key  ssl/privateKey.pem;
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
       proxy_pass http://localhost:3300;
    }
}

In my booking system I give users different roles in my app and depending on their role, the users have different authorizations. An admin for example is able to access more sensitive data and functionalities than a normal player or a coach. So my app must know the role of a user to assign different authorizations to the particular user.

Clients, usually browsers send requests the app. The app responds to requests and is solely responsible for ensuring that the client only has access to the data that are intended for it. This request and response game is based on the HTTP protocol. HTTP is a stateless network protocol and requests cannot be related to each other. Each request is isolated and unrelated to previous requests and the server has no chance to recognize clients and does therefore not know their role.

This problem can be solved with sessions and cookies and means that session management must be implemented in the application. The application creates a session and stores session data such as the role of a requestor in this session. The session has a unique ID and the app saves only this ID in a cookie. The cookie is transferred to the browser and stored locally there. From now on, the browser always sends this cookie with the HTTP request and thus identifies itself to the application. The application can check the role of the requestor in the stored session data and control the appropriate access.

Basic setup of the server

First we need a working Server OS. I run Linux Ubuntu in production and have written an article about the basic setup of a production Linux server on my blog site Digitaldocblog. Since I am going to store the sessions in a MongoDB, MongoDB must be installed on the Linux server. I use MongoDB Community Edition but you can also install or upgrade to the MongoDB Enterprise Server version. In the lower part of the article you find the instructions how to install and setup your MongoDB Community Edition on your Linux System. In case you want to read the original documentation go on the MongoDB site and read how to install the MongoDB Community Edition for your OS.

In my express application I use a number of external modules or dependencies that have to be installed for the application in order for the application to run. In the repository of the bookingsystemon my GitHub account you find the package.json file which contains all the necessary dependencies. In principle, it is sufficient if you put this package.json file in your application main directory and install all dependencies with npm install.

Alternatively, of course, all modules can also be installed individually with

npm install <module> --save

Session Management

I discuss in the first part different code snippets in my application main file booking.js. The goal here is that you understand how session management is basically implemented.

// booking.js

// Load express module and create app
const express = require('express');
const app = express();
// Trust the first Proxy
app.set('trust proxy', 1);
// Load HTTP response header security module
const helmet = require('helmet');
// use secure HTTP headers using helmet with every request
app.use(
  helmet({
      frameguard: {
        action: "deny",
      },
      referrerPolicy: {
        policy: "no-referrer",
    },
    })
  );
// Load envy module to manage environment variables
const envy = require('envy');
const env = envy();

// Set environment variables
const port = env.port
const host = env.host
const mongodbpath = env.mongodbpath
const sessionsecret = env.sessionsecret
const sessioncookiename = env.sessioncookiename
const sessioncookiecollection = env.sessioncookiecollection

// Load server side session and cookie module
const session = require('express-session');
// Load mongodb session storage module
const connectMdbSession = require('connect-mongodb-session');
// Create MongoDB session store object
const MongoDBStore = connectMdbSession(session)
// Create new session store in mongodb
const store = new MongoDBStore({
  uri: mongodbpath,
  collection: sessioncookiecollection
});
// Catch errors in case session store creation fails
store.on('error', function(error) {
  console.log(`error store session in session store: ${error.message}`);
});
// Use session to create session and session cookie
app.use(session({
  secret: sessionsecret,
  name: sessioncookiename,
  store: store,
  resave: false,
  saveUninitialized: false,
  // set cookie to 1 week maxAge
  cookie: {
    maxAge: 1000 * 60 * 60 * 24 * 7,
    sameSite: true
  },
}));

... //further code not taken into account at this point

I create a server application using the Express-js  Web Application Framework. Therefore I load the Express-js module with the require() function and store the express() function in the constant app. Because my app is running behind a reverse proxy server I set the app to trust the first proxy server. Then I load the helmet module to use secure response headers in my app. I configure that all browsers should deny iFrames and that my app will set no referrer in the response header.

I use the envy module in my application to manage environment variables. Therefore I load the module with require() and store the envy() function in the constant env. With envy you can define your environment variables in your .env and .env.example files. These files must be stored in the application main directory as explained in the envy documentation.

Since my booking app is a real web application running on a web server in production I can not discuss the real environment variables because of security reasons. Therefore let us see how this work and make an example .env file.

// .env

port=myport
host=myhost
mongodbpath=myexamplemongodbpath
sessionsecret=myexamplesecret
sessioncookiename=booking
sessioncookiecollection=col_sessions

These variables have different values in my .env file. In the booking.jsfile above I define the constant env for the envy function with env = envy() . Then I have access to the environment variables defined in my .env file with env.<variable>. I define constants for each variable and assign the variable from the .env file with env.<variable>. These constants can now be used as values in the code.

I load the express-session module and the connect-mongodb-session module with the require() function. The session module stored in the constant session takes over the entire control of the session and cookie management.

The connect-mongodb-session stored in the constant connectMdbSession module is basically responsible for storing the session in the database. That is why we pass session as a parameter in the code and assign the constant MongoDBstore.

const MongoDBstore = connectMdbSession(session)

With new MongoDBStore I create a new store object. Here I pass the uri of the mongodb path and the collection where sessions should be stored.

// booking.js
...

const store = new MongoDBStore({
  uri: mongodbpath,
  collection: sessioncookiecollection
});

...

The store object initialized in this way contains all necessary parameters to successfully store a session object in my MongoDB database.

After we have defined the storage of the session object, we take care of the session object itself.

With app.use(session( {... cookie: {...} })) I create a session object with various options. The session object will be created with each request and also contains a cookie object. I pass the values for cookie: {...} and then other options likesecret: sessionsecret, the session object name with name: sessioncookiename as well as the location where the session object should be stored with store: store. Furthermore the session object has the option saveUninitialized: false and resave: false .

When the saveUninitialized option is set to false the session object is not stored into the store as long as the session is un-initialized. The option resave: false enforce that a session will not be saved back to the store even if the session is initialized. So we must understand what initialized and un-initialized mean. This must be explained.

A browser send a request to the app. More precisely, the browser sends the request to a defined endpoint in the app. An endpoint defines a path within the app that reacts to HTTP requests and executes code. Depending on the HTTP method GET or POST, the endpoint expects that the requestor requires a document back (GET) or that the requestor wants to send data to the app (POST).

In the example below the browser should send a GET request to GET home endpoint. The endpoint render the HTML template index and send the HTML back to the browser. Then the request is finished. So this process, which starts with the request and ends with the response is the runtime of the request.

In the code snippet below you see 2 GET endpoints in the booking.js file one for the home route and another one for the register route.

// booking.js

... //further code not taken into account at this point

// GET home route only for anonym users. Authenticated users redirected to dashboard
app.get('/', redirectDashboard, (req, res) => {
	
  console.log(req.url);
  console.log(req.session.id);
  console.log(req.session);

  res.render('index', {
      title: 'User Login Page',
    });

});

// GET register route only for anonym users. Authenticated users redirected to dashboard
app.get('/register', redirectDashboard, (req, res) => {
  
  console.log(req.url);
  console.log(req.session.id);
  console.log(req.session);

  res.render('register', {
      title: 'User Registration Page',
    });
});

... //further code not taken into account at this point

The code with app.use (session({ ... })) in my booking.js file ensures that a session object is always generated with each request. As long as a session object is not changed during the runtime of a request a separate session is created for each request and has its own session ID. The option saveUninitialized: false ensure that the session object will not be stored into the database. Each session object created in this way is un-initialized.

You can see the following output on the console for the home route and for the register route when we log the path, the session-ID and the session object on the console for each route.

/
BmbE8RVoTRcPP9nUnBm5JLE1w1mQiNyt
Session {
  cookie: {
    path: '/',
    _expires: 2021-04-24T04:27:04.265Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/register
awlPO-KpyVM51Gp6UAoeXGGmRWo-QFtP
Session {
  cookie: {
    path: '/',
    _expires: 2021-04-24T05:54:57.439Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}
  

The code of my app changes a session object during the runtime of a request by adding a data object when a user has successfully logged in. I will explain the code in detail in the next chapter but at the moment it is enough to know that. Therefore we play through the login of a user as follows.

The browser send a GET request to the home route as explained above, then the index template is rendered and the HTML page with the login form is sent back to the browser. During the runtime of this GET request a session object is created but the session object has not changed. We have already seen this above.

Then the user enters email and password in the login form and click submit. With this submit the browser send a POST request to the POST endpoint /loginusers and again a session object is generated for this POST request. During the runtime of the POST request, the code checks whether the transferred credentials are correct. If the credentials are correct, a data object with user data is generated and attached to the session object. Here the session object is changed during the runtime of the POST request. The existing session created with the POST request is now initialized at that moment. Because of the option saveUninitialized: false this session object is stored into the database store. When we look into the database store using the tool MongoDB Compass we see that the entire session object has been saved into the colsessions collection including the data object containing the required data of the user.

After the session initialization the code called by the POST endpoint redirect the request and send a new GET request to the /dashboard route. The code with app.use(session({ ... })) is called again but now there is an initialized session existing in the store. Because of the option resave: false the existing session object will not be updated and dragged along unchanged with every further request.

You see this in the output on the console when we log the path, the session-ID and the session object on the console for each route. The first output on the console is created when the GET request is sent to the home route. Then, the second output, after the user clicked submit the POST route /loginusers is called and a new session object is created. You see this from the different session IDs. During the runtime of this POST request the data object is added to the session object which initializes the session. Then, the third output, the GET route /dashboard is called and we see the same session object ID but the session object now contain the data object with the user data.

/
TEAZITdX7nLWBDc8uOk2HhXIiMZO7W-4
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:13.236Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/loginusers
gVlKut3bdEMiDHnK455FGjCi6YbPTBuZ
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:35.202Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/dashboard
gVlKut3bdEMiDHnK455FGjCi6YbPTBuZ
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:35.468Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    secure: null,
    domain: null,
    sameSite: true
  },
  data: {
    userId: 5f716b7439777365c18639f1,
    status: 'active',
    name: 'Oskar David',
    lastname: 'Rottländer',
    email: 'oskar@test.com',
    role: 'player',
    age: 17,
    cat: 'youth'
  }
}

In summary, session management works as follows: A session object will be created with each request and the session object is only saved in the database when the user is logged in (saveUninitialized: false). As long as the user is logged in, the session object is not changed and the data of the session object in the database are not updated (resave: false).

But what happens to the cookie ? This will be explained in the next chapter.

User login

When the session has been initialized the cookie containing the session ID is stored in the browser of the requestor. With every request the browser provide the cookie to authenticate the requestor. To authenticate the requestor the code app.use(session({...})) is called and compare the session ID sent by the browser with the session IDs stored in the session store. If a session ID matches, the session object including the data object is attached to the request object to give the app access to the data object. Within the app we now have access to any attribute of the data object with req.session.data.<attribute>. Therefore we can now implement role based authorization by accessing the role of the requestor with req.session.data.role and use this information in conditions in the code to control access depending on the role of the requestor.

But lets start from the beginning with the login of the requestor or user as I call the requestor from now on. In order for a user to be able to login, he or she must first call the login page which can be displayed by calling up the home endpoint.

// booking.js

... // Code not discussed here

// Redirect GET requests from authenticated users to dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.data) {
    res.redirect('/dashboard')

  } else {
    next()

  }
}

... // Code not discussed here

// GET home route only for anonym users. Authenticated users redirected to dashboard

app.get('/', redirectDashboard, (req, res) => {

  res.render('index', {
      title: 'User Login Page',
    });

});

... // Code not discussed here

As you see above in the code I have first defined the middleware function redirectDashboard. This middleware ensure that only users who are not logged in see the login page. If we look at the code of the middleware function we can see that req.session.data is used in an if-condition to check whether a data object is attached to the current session object. In case the if-condition is true, the user is logged in and the request is redirected to the dashboard, but in case the if-condition is false, the user is not logged in and the next() function is called.

The GET endpoint has the routingPath to the home route. When a user visits the homepage of my booking application, the GET HTTP request ask for the home routingPath. The middleware function redirectDashboard is put in front of the routingHandler function. If the user is not logged in the routingHandler function render the HTML template index.pugand send the HTML back to the user or more precisely to the user’s browser.

So far so good. We now imagine a not logged in user who sees the index page in front of him or her now wants to login using his or her email and password.

As described above, the index page is nothing more than a login form for entering an email address and a password. When we look at the index.pug file we see that the form action attribute define that the form data email and password will be sent to the form handler /loginusers using the POST method when the Submit button is clicked.

...

form#loginForm.col.s12(
		method='post', 
		action='/loginusers'
		)

		input.validate(
			type='email', 
			name='email', 
			autocomplete='username' 
			required
			)
		...

		input.validate(
			type='password', 
			name='password', 
			autocomplete='current-password' 
			required
			)
		...

button.btn.waves-effect.waves-light(
		type='submit', 
		form='loginForm'
		)
...

Note: To understand the autocomplete attributes of the input tags I recommend reading the documentation of the Chromium Project. Most browsers have password management functionalities and automatically fill in the credentials after you provide a master password to unlock your local password store. By using these autocomplete attributes in login forms but also in user registration forms or change password forms you help browsers by using these autocomplete functions to better identify these forms.

When the user has entered his or her email and password in the HTML form and clicked the Submit button, the request body contain the Form Data attributes email and password. Then a POST HTTP request is sent via HTTPS to the POST endpoint /loginusers defined in my booking.js file (see above).

In the picture below you can see the output of the network analysis in the developer tool of the chrome browser. Here you can see that the Form Data are not encrypted on browser side but you also see that the POST request URL /loginusers is HTTPS. This mean that when the browser sent the POST request to the server these data are encrypted with SSL/TLS in transit from the browser to the server.

On the server side we have the web application behind a proxy server listening to HTTP requests addressed to the POST endpoint /loginusers. This POST endpoint is an anonym POST Route which means that the routingHandler controller function is restricted to not logged-in users only. This makes sense because a login function must not be used by already logged in users. So already logged in users can not send data to this POST endpoint. This check is controlled by the middleware function verifyAnonym which is put in front of the routingHandler.

So lets look at the relevant code snippets in booking.js.

// booking.js

...

// Load db controllers and db models
const userController = require('./database/controllers/userC');

...

// Verify POST requests only for anonym users
const verifyAnonym = (req, res, next) => {

  if (req.session.data) {
    var message = 'You are not authorized to perform this request because you are already logged-in !';
    res.status(400).redirect('/400badRequest?message='+message);

  } else {
    next()

  }
}

...

// Anonym POST Route
// Login user available for anonym only
app.post('/loginusers', verifyAnonym, userController.loginUser)

...

// GET bad request route render 400badRequest
app.get('/400badRequest', (req, res) => {
 
  res.status(400).render('400badRequest', {
    title: 'Bad Request',
    code: 400,
    status: 'Bad Request',
    message: req.query.message,
  })
})

...

At the beginning of the code I refer the constant userController to the user controller file userC.js using the require method. In userC.js all user functions are defined to control user related operations.

Note: When you look into the userC.js file you see that we export modules using module.exports = {...}. Using this directive we export in fact an object with various attributes and the values of these attributes are functions. So with module.exports = { loginUser: function(...) ...} we export the object including the attribute loginUser which contains a function as value. So when we refer the constant userController using the require() function in the booking.js file we store the complete exported object with all its attributes to the userController constant. Now we have access to any attribute of the exported object from userC.js file with userController.<attribute>. Because the attributes are in fact functions we call these functions with this statement.

In the verifyAnonym function req.session.data is used in the if-condition to check whether a data object is attached to the current session object. In case the if-condition is true, the user is already logged-in and is redirected to the Bad Request GET endpoint /400badRequest which is the standard route in my application to show the user that something went wrong. The user can see what went wrong from a message that has been attached to the request using the request parameter ?message=+message. In case the if-condition is false, the user is not logged-in and the next() function forwards the request to the routingHandler controller function that call the loginUser function using userController.loginUser. This function has access to the attributes email and password of the request body with req.body.email and req.body.password.

So lets look at the relevant code snippets in userC.js file.

// database/controllers/userC.js

// load the bcryptjs module
const bcrypt = require('bcryptjs');
// define hash saltrounds for password hashing
const saltRounds = 10;
// load the relevant Prototype Objects (exported from the models)
...

const User = require('../models/userM');

...

loginUser: function (req, res) {

    const inputemail = req.body.email
    const email = inputemail.toLowerCase()

    console.log(req.url);
    console.log(req.session.id);
    console.log(req.session);

    try {

      User.findOne({ email: email }, async function(error, user) {
        if (!user) {
          var message = 'User not found. Login not possible';
          res.status(400).redirect('/400badRequest?message='+message);

        } else {
          if (user._status !== 'active') {
            var message = 'Login not possible. Await User to be activated';
            res.status(400).redirect('/400badRequest?message='+message);

          } else {
              if (bcrypt.compareSync(req.body.password, user.password)) {

                var yearInMs = 3.15576e+10;
                var currentDate = new Date ()
                var currentDateMs = currentDate.getTime()
                var birthDateMs = user.birthdate.getTime()
                var age = Math.floor((currentDateMs - birthDateMs) / yearInMs)

                if (age < 18) {
                  var cat = 'youth'
                } else {
                  var cat = 'adult'
                };

                var userData = {
                  userId: user._id,
                  status: user._status,
                  name: user.name,
                  lastname: user.lastname,
                  email: user.email,
                  role: user.role,
                  age: age,
                  cat: cat,
                }

                req.session.data = userData

                res.status(200).redirect('/dashboard')

              } else {
                var message = 'Login not possible. Wrong User password';
                res.status(400).redirect('/400badRequest?message='+message);
              }
          }
        }
      })

    } catch (error) {
      // if user query fail call default error function
      next(error)

    }
  // End Module
  },

...

In order to authenticate a user, the loginUser function must find a user in the user database with the same email address as the one that was sent by the browser and attached to the request body by the app. If a user was found with the email, the function must check whether the transmitted password matches the password that is stored in the database for this user. If the email and password match the user is authenticated and the login is successful, if not, the login fails.

Passwords are never saved in plain text. Therefore I use the bcryptjs module to hash passwords. The bcryptjs module is loaded into the code with the require() function and assigned to the constant bcrypt. We set the constant saltRounds to the value of 10. This is the so called cost factor in the bcrypt hashing function and controls how much time bcrypt need to calculate a single bcrypt hash. Increasing the cost factor by 1 doubles the time and the more time bycrypt need to hash the more difficult it is to brute force stored passwords.

Then I load the user model from userM.js using the require() function and assign the constant User. Here at this point I have to explain the background. To do this, we also need a look at the userM.js file.

Note: I use MongoDB as the database and Mongoose to model the data. If you look in the file userM.js you see that a user object is created with the function new Schema() and saved in the variable userSchema. This userSchema object describes a user with all its attributes. At the end of the file, the mongoose.model() function is used to reference the userSchema to the collection colusers in my MongoDB. This reference is assigned to the variable User and exported using the function module.exports(). With User I have access to the user model meaning to all user objects and attributes in my database that are stored in the colusers collection. So that I can use this access in the code of my userC.js file I load userM.js with the require() function and assign the constant User. I can now use Mongoose functions for example to query user data from my colusers collection. This is exactly what we do with User.findOne() when we try to find a user with a certain email.

The actual user authentication now takes place in the userFindOne() function.

When we run User.findOne() we check the criteria that do not lead to a successful authentication.

1. No user found: We are looking for a user object that matches the email that has been submitted. If no user object is found with that email or the user found is not active, the request is redirected to the 400badRequest route. If we have found an active user, the submitted password string is hashed with bcrypt and compared with the saved password.
2. Wrong password: If the password comparison is not successful, the submitted password was wrong and the request is also redirected to the 400badRequest route.
   

Note: User.findOne() has a query object {email: email} and a callback function async function(error, user {...}) as parameters. When the async function find a user with the email in the database, this async function returns a user object with all the user attributes and store this object into the user parameter. Within the scope of the async function I have now access to the user attributes using user.<attribute>.

Only in case the user with the email is found and the submitted password is correct the authentication is successful.

If the user is successfully authenticated, the category of the user is calculated based on the current date and the user’s birth date. Then a userData object is created in which various user attributes are stored. The data of the userData object are then attached to the session. More precisely, the object data is attached to the session with req.session.data and the value userData is assigned. Now the session is initialized and the session object is stored in the colsessions collection of the MongoDB.

Then the response is sent back to the browser.

In this response, the browser is instructed to call up a GET request to the GET endpoint /dashboard. The response is sent using res.status(200).redirect('/dashboard'). In the Response Header you see that the cookie with the name booking is set in the users browser using the set-cookie directive. The cookie only contain the session ID which has been signed and encrypted with the secret we provided in app.use(session( {... cookie: {...} })).

Then the browser send the GET request to the endpoint /dashboard. Lets have a look into the booking.js file again.

// booking.js 

...

// Redirect GET requests from not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.data) {
    res.redirect('/')

  } else {
    next()

  }
}

...

// GET dashboard route only for authenticated users. Anonym users redirected to home
app.get('/dashboard', redirectLogin, async (req, res) => {

  // Check admin authorization and render admin_dashboard
  if (req.session.data.role == 'admin') {

    const user_query = User.find( {} ).sort({lastname: 1, name: 1});
    var users = await user_query.exec();

    const training_query = Training.find( {} ).sort({date: 'desc'});
    var trainings = await training_query.exec();

    const location_query = Location.find( {} ).sort({location: 'desc'});
    var locations = await location_query.exec();

    const booking_query = Booking.find( {} ).sort({_booktrainingdate: 'desc'});
    var bookings = await booking_query.exec();

    const invoice_query = Invoice.find( {} ).sort({invoicedate: 'desc'});
    var invoices = await invoice_query.exec();

    res.status(200).render('admin_dashboard', {
      title: 'Admin Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      data_users: users,
      data_trainings: trainings,
      data_locations: locations,
      data_bookings: bookings,
      data_invoices: invoices,

      });

  // Check player authorization and render player_dashboard
  } else if (req.session.data.role == 'player') {

    var currentDate = new Date();
    console.log('current date: ' +currentDate);

    const availabletraining_query = Training.find( { _status: 'active', date: { $gte: currentDate } } ).sort({ date: 'desc' });
    var availabletrainings = await availabletraining_query.exec();

    const booking_query = Booking.find( { _bookuseremail: req.session.data.email, _bookparticipation: { $ne: 'invoice' } } ).sort({ _booktrainingdate: 'desc' });
    var bookings = await booking_query.exec();

    const myuser_query = User.findOne( { email: req.session.data.email } );
    var myuser = await myuser_query.exec();

    const invoice_query = Invoice.find( {invoiceemail: req.session.data.email} ).sort({invoicedate: 'desc'});
    var invoices = await invoice_query .exec();

    res.status(200).render('player_dashboard', {
      title: 'Player Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      email: req.session.data.email,
      data_availabletrainings: availabletrainings,
      data_bookings: bookings,
      data_myuser: myuser,
      data_myinvoices: invoices,
      });

  // Check coach authorization and render coach_dashboard
  } else if (req.session.data.role == 'coach') {
   
    var currentDate = new Date().setHours(00, 00, 00);
    console.log('currentDate: ' +currentDate);

    const training_query = Training.find( { _status: 'active', date: { $gte: currentDate } } ).sort({ date: 'asc' });
    var trainings = await training_query.exec();

    res.status(200).render('coach_dashboard', {
      title: 'Coach Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      data_trainings: trainings,
      });

  } else {
    // if user not authorized as admin, player or coach end request and send response
    var message = 'You are not authorized. Access prohibited';
    res.status(400).redirect('/400badRequest?message='+message);
  }

});

...

As you see above in the code we have first defined the middleware function redirectLogin. This middleware ensure that only users who are logged in see the dashboard page. In case the if-condition is true, the user is not logged in and the request is redirected to the home route, but in case the if-condition is false, the user is logged in and the next() function is called.

The GET HTTP request ask for the dashboard routingPath. The middleware function  redirectLogin is put in front of the routingHandler function. If the user is not logged in the  redirectLogin middleware redirect the request to the home route. In case the user is logged-in the routingHandler function is called using the request object and the response object as parameters.

app.get('/dashboard', redirectLogin, async (req, res) => {...})

If we look at the Request Header of this new GET request in the browser we can see that the cookie is dragged along unchanged with the GET request to the endpoint /dashboard. This happens from now on with every request until the session expires or until the user logout.

And now within the routingHandler function we do the user authorization check. The if condition check the users role using req.session.data.role. Depending on the role of the user different <role>dashboard HTML templates are rendered and for each role different HTML is sent back to the user’s browser. Various queries are executed beforehand because we need role specific data within each <role>dashboard HTML template. The return values ​​of the different queries find() and findOne() are only executed in case one of the if conditions become true. Then in each case the return values of the queries are stored in variables. In case all if conditions are false, meaning we cannot find a user with a role like admin, player or coach in the database for some reason the request is redirected to the Bad Request GET endpoint /400badRequest using the message as request parameter that this user is not authorized.

Within each if condition and for each role admin, player or coach, we create the response object by first setting the HTTP status to the value of 200 and then using the render method to render the respective HTML template.

res.status(200).render('<role>_dashboard', {...})

Within the render method, we now have the option of transferring a data object with different attributes to the HTML template. Later we can access these data in the respective HTML template and use it within the HTML template. How this works is not part of this article. But of course you can take a closer look at the templates admin dashboard, player dashboardand coach dashboardon my GitHub repository and you will immediately see how this works.

Create Authorizations

As I have already shown in the upper part, I work with middleware functions to control access to GET and POST endpoints in my app. Therefore these middleware functions are the Authorizations and you can find them in the code of my booking.js file.

// booking.js

...

// Authorizations
// Redirect GET requests from not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.data) {
    res.redirect('/')

  } else {
    next()

  }
}

// Redirect GET requests from authenticated users to dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.data) {
    res.redirect('/dashboard')

  } else {
    next()

  }
}

// Authorize POST requests only for not authenticated users
const verifyAnonym = (req, res, next) => {
  if (!req.session.data) {
    next()

  } else {
    var message = 'You are already logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for anonym and admin users
const verifyAnonymAndAdmin = (req, res, next) => {
  if (!req.session.data) {
    next()

  } else {

    if (req.session.data.role == 'admin') {
      next()

    } else {
      var message = 'You are no Admin. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);

    }

  }
}

// Authorize POST requests only for admin and player users
const verifyAdminAndPlayer = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'admin') {
      next()

    } else if (req.session.data.role == 'player') {
      next()

    } else {
      var message = 'You are no Admin, no Player. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);
  }

}

// Authorize POST requests only for admin users
const verifyAdmin = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'admin') {
      next()

    } else {
      var message = 'You are no Admin. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for player users
const verifyPlayer = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'player') {
      next()

    } else {
      var message = 'You are no Player. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for coach users
const verifyCoach = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'coach') {
      next()

    } else {
      var message = 'You are no Coach. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);

    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

...

As I have already explained above I use redirect functions as a middleware to control access to the GET endpoints home, register and dashboard. These middleware functions basically control access based on whether a user is logged-in or not. The redirect function redirectDashboard allow only not logged-in users access to the home endpoint and to the register endpoint, while users who are already logged-in have no access and would be redirected directly to the dashboard route if they try to access here. The redirectLogin middleware function allow only logged-in users access to the dashboard route while not logged-in users are redirected to the login or better to the home endpoint.

In addition to redirect functions I use verify functions as a middleware to control access to the POST endpoints. With the help of POST requests, data are sent via POST endpoints to the app. That is why it is particularly important to control who is allowed to send data and who is not. I use basically 5 types of POST endpoints.

Anonym POST endpoint. I only have one endpoint here. The loginusers endpoint can only be called by not logged-in users. Therefore the verifyAnonym  middleware is set before the routingHandler function to verify if the user is not logged-in.

// booking.js
...

// Anonym POST endpoint
// Login user available for anonym only
app.post('/loginusers', verifyAnonym, userController.loginUser)

...

Shared POST endpoints. The createusers endpoint can be called by not logged-in users and Admin users. The  verifyAnonymAndAdmin  middleware is set before the routingHandler function to verify if the user is not logged-in or if the user that is logged-in has the role admin. The  updateuseremail and setnewuserpassword endpoints can be called only by Admin and Player users. Therefore the  verifyAdminAndPlayer  middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is admin or player.

// booking.js
...

// Shared POST endpoints
// Create Users available for anonym and admin
app.post('/createusers', verifyAnonymAndAdmin, birthdateFormatValidation, userController.createUser)

// Update User-Email available for admin and player
app.post('/updateuseremail', verifyAdminAndPlayer, userController.updateUserEmail)

// Update User-Password available for admin and player
app.post('/setnewuserpassword', verifyAdminAndPlayer, userController.setNewUserPassword)

...

Admin POST endpoints. I have 19 endpoints here and each of these endpoint can only be called by Admin users. The  verifyAdmin middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is admin.

// booking.js
...

// Admin POST endpoints
// Admin User Management
app.post('/callupdateusers', verifyAdmin, userController.callUpdateUsers)

app.post('/updateuser', verifyAdmin, birthdateFormatValidation, userController.updateUser)

app.post('/terminateusers', verifyAdmin, userController.terminateUser)

app.post('/activateusers', verifyAdmin, userController.activateUser)

app.post('/removeusers', verifyAdmin, userController.removeUser)

// Admin Update Training
app.post('/callupdatetrainings', verifyAdmin, trainingController.callUpdateTrainings)

app.post('/updatetraining', verifyAdmin, trainingController.updateTraining)

// Admin Location Management
app.post('/createlocations', verifyAdmin, locationController.createLocation)

app.post('/callupdatelocations', verifyAdmin, locationController.callUpdateLocations)

app.post('/updatelocation', verifyAdmin, locationController.updateLocation)

app.post('/callcreatetrainings', verifyAdmin, trainingController.callCreateTrainings)

app.post('/createtraining', verifyAdmin, trainingController.createTraining)

// Admin Invoice Management
app.post('/createinvoice', verifyAdmin, invoiceController.createInvoiceUser)

app.post('/callcancelinvoice', verifyAdmin, invoiceController.callCancelInvoice)

app.post('/cancelinvoice', verifyAdmin, invoiceController.cancelInvoice)

app.post('/callpayinvoice', verifyAdmin, invoiceController.callPayInvoice)

app.post('/payinvoice', verifyAdmin, invoiceController.payInvoice)

app.post('/callrepayinvoice', verifyAdmin, invoiceController.callRePayInvoice)

app.post('/repayinvoice', verifyAdmin, invoiceController.rePayInvoice)

...

Player POST endpoints. I have 7 endpoints here and each of these endpoint can only be called by Player users. The  verifyPlayer middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is player.

// booking.js
...

// Player POST endpoints
// Player Booking Management
app.post('/callbooktrainings', verifyPlayer, bookingController.callBookTrainings)

app.post('/booktrainings', verifyPlayer, bookingController.bookTraining)

app.post('/bookingreactivate', verifyPlayer, bookingController.bookingReactivate)

app.post('/callcancelbookings', verifyPlayer, bookingController.callCancelBooking)

app.post('/cancelbookings', verifyPlayer, bookingController.cancelBooking)

// Player User Management
app.post('/callupdatemyuserdata', verifyPlayer, userController.callUpdateMyUserData)

app.post('/updatemyuserdata', verifyPlayer, birthdateFormatValidation, userController.updateMyUserData)

...

Coach POST endpoints. I have 2 endpoints here and each of these endpoint can only be called by Coach users. The  verifyCoach middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is coach.

// booking.js
...

// Coach POST endpoints
// Coach Confirmation Management
app.post('/callparticipants', verifyCoach, bookingController.callParticipants)

app.post('/callconfirmpatricipants', verifyCoach, bookingController.callConfirmPatricipants)

...

User logout

The user initiates a logout himself by clicking on the logout link in the navigation of the application. This sends a GET request to the /logout endpoint of the application. In this routing definition, the session is first deleted from the database using req.session.destroy() and then the cookie is removed from the browser and the user is redirected to the 200success site using res.status(200).clearCookie('booking').redirect().

// booking.js

...

// GET logout route only for authenticated users. Anonym users redirected to home
app.get('/logout', redirectLogin, (req, res) => {
  req.session.destroy(function(err) {
    if (err) {
      res.send('An err occured: ' +err.message);
    } else {
      var message = 'You have been successfully logged out';
      res.status(200).clearCookie('booking').redirect('/200success?message='+message)
    }
  });
})

...

How to setup such a production environment will be shown in the first chapter of this documentation.

The express app itself also contains some security features. The app contains a session based user authentication and HTTP headers which help to further secure the app. This is explained of the second chapter of this documentation.

Finally the express app should use the template engine PUG to render the HTML for us. This is describes of the third chapter of this documentation.

1. Setup the production environment

Installation of mongodb

The command brew tap without any arguments lists the GitHub repositories that are currently linked to your Homebrew installation.

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services

The formula mongodb has been removed from homebrew-core. But fortunately the MongoDB Team is maintaining a custom Homebrew tap on GitHub. Read the instructions in the README.md file.

Add the custom tap in the Mac OS terminal and install mongodb.

Patricks-MBP:~ patrick$ brew tap mongodb/brew

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services
mongodb/brew

Patricks-MBP:~ patrick$ brew install mongodb-community@4.2

After the installation the relevant paths are.

the configuration file (/usr/local/etc/mongod.conf)
the log directory path (/usr/local/var/log/mongodb)
the data directory path (/usr/local/var/mongodb)

Check the services with homebrew

brew services list

Name              Status  User    Plist
mongodb-community started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
  

Start and stop mongodb.

brew services start mongodb-community

brew services stop mongodb-community

Setup mongodb for the project

Setup an admin user

:# mongo

> use admin
switched to db admin

> db
admin

> db.createUser({ user: "adminUser", pwd: "adminpassword", roles: [{ role: "userAdminAnyDatabase", db: "admin" }, {"role" : "readWriteAnyDatabase", "db" : "admin"}] })

> db.auth("adminUser", "adminpassword")
1

> show users
{
    "_id" : "admin.adminUser",
    "userId" : UUID("5cbe2fc4-1e54-4c2d-89d1-317340429571"),
    "user" : "adminUser",
    "db" : "admin",
    "roles" : [
        {
            "role" : "userAdminAnyDatabase",
            "db" : "admin"
        },
        {
            "role" : "readWriteAnyDatabase",
            "db" : "admin"
        }
    ],
    "mechanisms" : [
        "SCRAM-SHA-1",
        "SCRAM-SHA-256"
    ]
}

> exit

Enable authentication with security: authorization: enabled

#> nano /usr/local/etc/mongod.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb
net:
  port: 27017
  bindIp: 127.0.0.1
security:
  authorization: enabled

Login and authenticate with admin

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b3e7f48a-a05c-4894-87db-996cb34eb1fb") }
MongoDB server version: 4.2.3

> show dbs
> db
test
> use admin
switched to db admin
> db
admin
> show dbs
> db.auth("adminUser", "adminpassword")
1
> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB

> 

If you login you dont see any databases when you call show dbs. The default database you are connected to is test.

Then you connect to admin database. For admin you setup the admin user with the roles userAdminAnyDatabase and readWriteAnyDatabase. With these permissions the admin user can manage users for any database and has read and write access to any database.

So wehen you logon to admin database with the admin user you are able to see all databases with show dbs.

Mongodb comes with 3 standard dbs pre installed:

• admin
• config
• local
  

Create a new database for our express-security app (authenticated as admin user – see above)

> use express-security
switched to db express-security
> db
express-security
> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
> 

The DB which you’ve created is not listed here. We need to insert at least one collection into it for displaying that database in the list.

> db
express-security

> db.createCollection("col_default")
{ "ok" : 1 }

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> exit

Create an owner user for express-security database using the admin user

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("79f79b63-9d08-489f-9e6c-bfc10d8cc09e") }
MongoDB server version: 4.2.3

> db
test

> show dbs

> use admin
switched to db admin

> db.auth("adminUser", "adminpassword")
1

> db
admin

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> use express-security
switched to db express-security

> db.createUser({ user: "owner_express-security", pwd: "passowrd", roles: [{ role: "dbOwner", db: "express-security" }] })
Successfully added user: {
	"user" : "owner_express-security",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "express-security"
		}
	]
}

> db
express-security

> show users
{
	"_id" : "express-security.owner_express-security",
	"userId" : UUID("7a0bafb2-d2ed-4d18-9aba-e2f15a503ec5"),
	"user" : "owner_express-security",
	"db" : "express-security",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "express-security"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

> exit 

Connection string to connect to express-security db using the owner_express-security user:

mongodb://owner_express-security:password@localhost/express-security

Installation of PM2

PM2 is a process manager for Node.js applications. It can daemonize applications to run them as a service in the background.

I install pm2 as a global npm package on my Mac.

Patricks-Macbook Pro:~ patrick$ npm install pm2 -g

Then navigate to your project directory.

Patricks-Macbook Pro:~ patrick$ cd Software/dev/node/articles/2020-05-15-express-security/express-security

Patricks-Macbook Pro:~ patrick$ ls -l 
total 112
drwxr-xr-x    5 patrick  staff    160 30 Mai 05:28 database
drwxr-xr-x  115 patrick  staff   3680 30 Mai 19:35 node_modules
-rw-r--r--    1 patrick  staff  34366 30 Mai 19:35 package-lock.json
-rw-r--r--    1 patrick  staff    339 30 Mai 19:35 package.json
-rw-r--r--@   1 patrick  staff  12343 30 Jun 05:00 secserver.js
drwxr-xr-x    3 patrick  staff     96 30 Mai 05:03 static

Patricks-Macbook Pro:express-security patrick$ 

Start your app using pm2

Patricks-Macbook Pro:express-security patrick$ pm2 start secserver.js

Patricks-Macbook Pro:~ patrick$ pm2 list
┌─────┬──────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name         │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0   │ secserver    │ default     │ 1.0.0   │ fork    │ 640      │ 16h    │ 0    │ online    │ 0%       │ 48.6mb   │ patrick  │ disabled │
└─────┴──────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

Patricks-Macbook Pro:~ patrick$ 

Other comands to control the process manager.

pm2 start secserver.js

pm2 start <id>

pm2 list

pm2 stop <id>

pm2 restart <id>

pm2 show <id>

Installation of nginx

nginx is an open source HTTP and an HTTP Reverse Proxy Server (also mail proxy and load balancer etc.). I install nginx on my Mac with Homebrew.

brew install nginx

You can list the brew services with the following command.

Patricks-MBP:digitaldocblog-V3 patrick$ brew services list

Name              Status  User    Plist
mongodb-community started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
nginx             started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.nginx.plist
Patricks-MBP:digitaldocblog-V3 patrick$

You can start and stop the brew services as follows.

brew install nginx

brew services start nginx

brew services stop nginx

Setup nginx with TLS/SSL

SSL/TLS works by using the combination of a public certificate and a private key.

The SSL key (private key) is kept secret on the server. It is used to encrypt content sent to clients.

The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

create private key

Patricks-MBP:express-security patrick$ cd /usr/local/etc/nginx

Patricks-MBP:nginx patrick$ ls -l
total 144
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  5 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  5 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types.default
-rw-r--r--  1 patrick  admin  3106 15 Mai 05:19 nginx.conf
-rw-r--r--  1 patrick  admin  2680  5 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params.default
drwxr-xr-x  3 patrick  admin    96 21 Jan 06:02 servers
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  5 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ mkdir ssl

Patricks-MBP:nginx patrick$ ls -l
total 152
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  5 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  5 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types.default
-rw-r--r--@ 1 patrick  admin   373 18 Mai 05:38 nginx.conf
-rw-r--r--  1 patrick  admin  2680  5 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--@ 1 patrick  admin  1390 17 Mai 05:19 nginx_old.conf
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params.default
drwxr-xr-x  5 patrick  admin   160 18 Mai 05:20 servers
drwxr-xr-x  4 patrick  admin   128 16 Mai 05:41 ssl
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  5 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ cd ssl

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl genrsa -out privateKey.pem 4096

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem

create certificate signing request (CSR)

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl req -new -key privateKey.pem -out csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  1740 16 Mai 05:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem

In case I would like to request an official certificate I must send this csr to the certificate authority. This authority would then create an authority signed certificate from the CSR and send it back to me.

This step is done by ourselves and this is the reason why we create a self signed certificate. This self signed certificate is not a official certificae and not trusted by any browser. It is not useful to use a self signed certificate in production because it produces error messages in the browsers. But for local development a self signed certificate is ok.

So create the self signed certificale. The csr file can then be removed.

create the self signed certificate

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in csr.pem -out selfsignedcertificate.pem -req -signkey privateKey.pem -days 365

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  1740 16 Mai 05:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 05:39 selfsignedcertificate.pem

Patricks-MBP:ssl patrick$ rm csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 05:39 selfsignedcertificate.pem
 

show certificate details

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in selfsignedcertificate.pem -text -noout

Configure nginx Servers with SSL

In our configuration we enforce ssl. Therefore we create a default Webserver listening on Port 80 with the server name servtest.rottlaender.lan.

Any request to servtest.rottlaender.lan:80 is redirected to my Reverse Proxy Server which is listening on servtest.rottlaender.lan:443.

The default Webserver is configured in /usr/local/etc/nginx/nginx.conf.

# /usr/local/etc/nginx/nginx.conf
# default Webserver

worker_processes  1;
error_log  /usr/local/etc/nginx/logs/error.log;

events {
    worker_connections  1024;
}

http {
    include       		mime.types;
    default_type  		application/octet-stream;
    sendfile        	on;
    keepalive_timeout  	65;

    access_log  /usr/local/etc/nginx/logs/access.log;

    # default Webserver redirect from port 80 to port 443 ssl
    
    server {
      	listen 80;
    	listen [::]:80;
    	server_name servtest.rottlaender.lan;
    	return 301 https://$host$request_uri;
    }
    
    include servers/*;
    
}

The Reverse Proxy Server is configured in /usr/local/etc/nginx/servers/reverse.

// /usr/local/etc/nginx/servers/reverse
// reverse Proxy Server

server {

    listen      443 ssl;
    server_name servtest.rottlaender.lan;

    ssl_certificate      ssl/selfsignedcertificate.pem;
    ssl_certificate_key  ssl/privateKey.pem;
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
       proxy_pass http://localhost:3300;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}

The server name servtest.rottlaender.lan is linked in /private/etc/hosts to the ip 192.168.178.20 which is the ip of my computer in my local network.

Patricks-MBP:digitaldocblog-V3 patrick$ cat /private/etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1			localhost
255.255.255.255		broadcasthost
::1             	localhost
192.168.178.20 		servtest.rottlaender.lan

2. Express Secure App (Security Features HTML version)

This is a very simple application but show the basic security features you should use when you run a node app in a production environment.

The app is a website with a simple layout and navigation.

The Home page contain static information and can be accessed by everyone.

On the register page, users can find a form to register. The user data entered here are saved in the database and the user is logged in at the same time. Known users can log in with their email and password after successful registration on the login page. The login and register page can only be accessed if the user is not logged in. If a user is logged in and tries to access the login or register, he will be redirected to the dashboard page.

The dashboard is a personalized area of the website. This area can only be accessed if the user is logged in. If a user is not logged in, he will be redirected to the login page.

Logout is not really a page but a link that contains a logout function. Users who are logged in can log out using this link. Users who are not already logged in will be redirected to the login page.

Download the code from GitHub

Pls. go to my GitHub site and clone the code. Here you find a some inline documentation in the code. The details are explained in this chapter.

Create your app home directory express-security

My app home directory is different to the one that is available after you cloned the code from GitHub.

Patricks-MBP:2020-05-15-express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security

Patricks-MBP:2020-05-15-express-security patrick$ mv node-part-5-express-security-with-db-pug express-security

Patricks-MBP:2020-05-15-express-security patrick$ cd express-security

Patricks-MBP:express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security

Manage environment variables

To manage environment variables for my app I use envy. First you need the files .env and .env.example in the root of your project directory. In .env.example you create a list of all potential environment variables without any values and in .env you use the defined variables and assign the values to them.

Patricks-MBP:express-security patrick$ ls -al
total 152
drwxr-xr-x   14 patrick  staff    448 23 Jun 05:29 .
drwxr-xr-x    4 patrick  staff    128 26 Mai 05:40 ..
-rw-------    1 patrick  staff    181 23 Jun 05:59 .env
-rw-r--r--    1 patrick  staff     53 23 Jun 05:59 .env.example
....

Patricks-MBP:express-security patrick$ cat .env.example
port=
mongodbpath=
sessionsecret=
sessioncookiename=

Patricks-MBP:express-security patrick$ cat .env
port=<YOUR_PORT>
mongodbpath=<YOUR_CONNECTION_STRING>
sessionsecret=<YOUR_SESSION_SECRET>
sessioncookiename=<YOUR_SESSION_COOKIE_NAME>

Patricks-MBP:express-security patrick$ 

Envy must be installed as dependency and required in the main application file secserver.js. Then you can set the environment variables as follows.

// secserver.js

....

// envy module to manage environment variables
const envy = require('envy');

// set the environment variables
const env = envy()
const port = env.port
const mongodbpath = env.mongodbpath
const sessionsecret = env.sessionsecret
const sessioncookiename = env.sessioncookiename
....

Start the MongoDB Server

To run the db server we install mongoose as dependency and require it in the db.js configuration file. The database connection will be initiated with mongoose.connect and the StartMongoServer function will be exported to be called in the main application file secserver.js.

const envy = require('envy')
const env = envy()

const mongodbpath = env.mongodbpath

const mongoose = require('mongoose');
mongoose.set('useNewUrlParser', true);
mongoose.set('useUnifiedTopology', true);

const StartMongoServer = async function() {
  try {

    await mongoose.connect(mongodbpath)
    .then(function() {
      console.log(`Mongoose connection open on ${mongodbpath}`);
    })
    .catch(function(error) {
      console.log(`Connection error message: ${error.message}`);
    })

  } catch(error) {
    res.json( { status: "db connection error", message: error.message } );
  }

};

module.exports = StartMongoServer;

Authentication and authorization

For user authentication we use the module express-session and to store session data in the session store in our database we use connect-mongodb-session. Therefore we install these modules as dependencies in our project and require the modules in our secserver.js main application file.

Then we create with new MongoDBStore a session storage in our MongoDB to store session data in collection col_sessions. errors are catched with store.on.

We use the session in our app with app.use( session({...}) ). With every request to our site a new session object is created with a unique session ID which include a session cookie object. The session object has keys options and the values for each key define how to deal with the session object. The session ID is created and signed using the secret option. We use name to provide a session cookie name and store to define where the session object should be stored (in case we store the session).

We can access the session object with req.session and the session ID with req.session.id. With every request we have a new session and this new session will be created but not stored anywhere so far. We say the session is uninitialized. The saveUninitialized false option ensure that a session will only be written to the store in case it has been modified. What does this mean?

We can modify the session when we store additional data into it. We always do this when the user is logging in via the login or the register route. When we post the data from the login- or from the registration-form to the server we call loginUser or the createUser module which is defined in database/controllers/userC.js. Both modules do basically the same thing: They create a userData Object and store the userData object into the session object and redirect the user to the dashboard when login or registration was successful.

....

var userData = { 
	userId: user._id, 
	name: user.name, 
	lastname: user.lastname, 
	email: user.email, 
	role: user.role 
	}

    req.session.userData = userData

    res.redirect('/dashboard')

....

If the user is successfully logged in the session is initialized (modified), the session object incl. the userData object are stored into the store and a cookie is stored into the requesting browser. The content of the cookie is only a hash of the session Id and with each request of a logged in user the session on the server is looked up.

The cookie in the browser will live max 1 week as we defined in the cookie object maxAge set to 1 week. Because of the cookie option sameSite true the cookie scope is limited to the same site.

Then the resave false option ensures that the session will not be updated with every request. This mean the session ID that has been created when the user has logged in will be kept until the user is logged out again.

// secserver.js

....

// server side session and cookie module
const session = require('express-session');
// mongodb session storage module
const connectMdbSession = require('connect-mongodb-session');

....

// Create MongoDB session storage
const MongoDBStore = connectMdbSession(session)
const store = new MongoDBStore({
  uri: mongodbpath,
  collection: 'col_sessions'
});

// catch errors in case store creation fails
store.on('error', function(error) {
  console.log(`error store session in session store: ${error.message}`);
});

// Create the express app
const app = express();

....

// use session to create session and session cookie
app.use(session({
  secret: sessionsecret,
  name: sessioncookiename,
  store: store,
  resave: false,
  saveUninitialized: false,
  // set cookie to 1 week maxAge
  cookie: {
    maxAge: 1000 * 60 * 60 * 24 * 7,
    sameSite: true
  },

}));

....

Secure HTTP headers

Response headers are HTTP header that come with the HTTP response from the server to the client. The http response header contain data that could possibly damage the integrity of the client. It is therefore important to secure the response header of your application.

To secure the http response headers I user the module helmet. This is a relatively easy-to-use module consisting of various middleware functionalities to secure various http response headers.

First we install helmetas a dependency of our project. Then we require helmet and use helmet right after we created the app.

// secserver.js

// hTTP module
const http = require('http');
// express module
const express = require('express');
// hTTP header security module
const helmet = require('helmet');

// Create the express app
const app = express();

....

// use secure HTTP headers using helmet
app.use(helmet())

Using simply app.use(helmet()) set the http header security to default. Then the following 7 out 11 helmet features can be used.

1. dnsPrefetchControl controls browser DNS prefetching
2. frameguard to prevent clickjacking
3. hidePoweredBy to remove the X-Powered-By header
4. hsts for HTTP Strict Transport Security
5. ieNoOpen sets X-Download-Options for IE8+
6. noSniff to keep clients from sniffing the MIME type
7. xssFilter adds some small XSS protections
   

When we then request our home page to retrieve the http headers using curl -k --head in the terminal we see the following output.

Patricks-MBP:express-security patrick$ curl -k --head https://servtest.rottlaender.lan
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Fri, 26 Jun 2020 16:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1734
Connection: keep-alive
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
ETag: W/"6c6-U2uWyDNyzlyBAbSI/Quxqo9RRQE"

Patricks-MBP:express-security patrick$ 

App routing

get routes: We have the following get routes and navigation.

• Home (/)
• Login (/login)
• Register (/register)
• Dashboard (/dashboard)
• Logout (/logout)
  

get routes involve an optional middleware and respond HTML back to the client.

app.get('/<route>', <optional: someMiddleware>, (req, res) => {
	
	res.send(`<some HTML>`)
})

I will not explain the HTML and css in detail. But as everyone can see, the HTML is the same for every route except for the <body>. Of course, this is not very nice and becomes a bit more efficient with the use of a template engine, which I will explain below using the PUG template engine. I will then rebuild the app using PUG.

Lets have a look at the middleware. If a request is made for a route and a middleware function is included, the middleware function is first executed before the next routing function function(req, res) is called. A condition is built into the middleware function which is checked. My middleware is built so that in case the condition is true the middleware code is executed directly and the next routing function is omitted. If the condition is false, the next routing function function(req, res) is called.

I have built 2 different middleware functions which each check

middleware 1 (login- and register route): a user is logged in

// secserver.js
....
// middleware 1 to redirect authenticated users to their dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.userData) {
    res.redirect('/dashboard')

  } else {
    next()
  }
}
....

If a user is logged in the request should be redirected to the dashboard route, in any other case (user is not logged in) the next routing function function(req, res) is called and respond the HTML to the browser. This middleware 1 is included in the /login- and /register route. This mean logged in users will be redirected to their dashboard, not logged in users will see the login- and register form.

middleware 2 (dashboard- and logout route): a user is not logged in.

// secserver.js
....

// middleware 2 to redirect not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.userData) {
    res.redirect('/login')
  } else {
    next()
  }
}
....

If a user is not logged in the request should be redirected to the login roure, in any other case (user is logged in) the next routing function function(req, res) is called and respond the HTML to the browser. This middleware 2 is included in the /dashboard- and /logout route. This mean not logged in users will be redirected to login route, logged in users will see the dashboard- or can log themselves out.

post routes: We have the following post routes.

• /login
• /register
  

The login and the register get routes contain a form in the HTML. With these forms the user provide the data to login and for user registration. When the user click the send button the action is to call the login- or register post route. This will happen for all not logged in users. The login and the register get routes have the middleware redirectDashboardto redirect the user to the dashbard if the user is already logged in.

// secserver.js
....

app.get('/login', redirectDashboard, (req, res) => {
....
res.send(`
....
<div class="form">
	<form id='register_form' method='post' action='/register'>
	......
	<label for='send'>
   		<input class='sendbutton' type='submit' name='send' value='Send'>
	</label>
	</form>
</div>

`)
)}
....

app.get('/register', redirectDashboard, (req, res) => {
....
res.send(`
....
<div class="form">
	<form id='login_form' method='post' action='/login'>
	......
	<label for='send'>
   		<input class='sendbutton' type='submit' name='send' value='Send'>
	</label>
	</form>
</div>
`)
)}
.....

The post routes contain functions to login- (loginUser) or register (createUser) the user.

// secserver.js
....
// Post routes to manage user login and user registration
app.post('/login', userController.loginUser);

app.post('/register', userController.createUser);
....

The loginUser function is defined in the user controller database/controllers/userC.js. This function lookup a user in the database based on the email address that has been provided by the request body. The data that are attached to the request body have been provided by the user vie the login form of the app. If no user could be found in the database login is not possible. If a user exist with the given email address then the provided password will be compared with the one stored in the database. If the password match fail login is not possible because the provided password is wrong. in any other case, the login takes place and a userData object is created and attached to the session object.

// database/controllers/userC.js

User.findOne({ email: req.body.email }, function(error, user) {
  if (!user) {
    res.status(400).send({ code: 400, status: 'Bad Request', message: 'No User found with this email' })

    } else {
      if (bcrypt.compareSync(req.body.password, user.password)) {
    
        var userData = { userId: user._id, name: user.name, lastname: user.lastname, email: user.email, role: user.role }

          req.session.userData = userData

          res.redirect('/dashboard')

        } else {
          res.status(400).send({ code: 400, status: 'Bad Request', message: 'Wrong User password' })
        }

      }
    })

  }

The createUser function is also defined in the user controller database/controllers/userC.js. This function create a new User object based on the data from the request body provided by the user via the form. The provided password will be hashed and stored together with all other data into the database. Finally a userData object is created and attached to the session and the user will be redirected to the dashboard after the registration was successful.

// database/controllers/userC.js

createUser: async function (req, res) {
    // assign input data from request body to input variables
    const name = req.body.name
    const lastname = req.body.lastname
    const email = req.body.email
    const password = req.body.password
    const role = req.body.role

    const newUser = new User({
      name: name,
      lastname: lastname,
      email: email,
      password: password,
      role: role
    })

    newUser.password = await bcrypt.hash(newUser.password, saltRounds)

    await newUser.save(function(err, user) {
          if (err) {
            // if a validation err occur end request and send response
            res.status(400).send({ code: 400, status: 'Bad Request', message: err.message })
          } else {
            // req.session.userId = user._id

            var userData = { userId: user._id, name: user.name, lastname: user.lastname, email: user.email, role: user.role }

            req.session.userData = userData

            res.redirect('/dashboard')
          }
        })
  },

And we have a default get route.

• /favicon.ico
  

Browsers will by default try to request /favicon.ico from the root of a hostname, in order to show an icon in the browser tab. As we dont use favicon so far we must avoid that these requests returning a 404 (Not Found). Here The /favicon.ico request will be catched and send a 204 No Content status.

// secserver.js
....
app.get('/favicon.ico', function(req, res) {
    console.log(req.url);
    res.status(204).json({status: 'no favicon'});
});
....

3. Express App (Pug Template Version)

From a functional point of view this app is pretty much the same app then the HTML version. The difference is that we use PUG templates instead of HTML in each res.send().

Setup a seperate Database

For the PUG version of my app I set up a new database to manage the users and the sessions.

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b3e7f48a-a05c-4894-87db-996cb34eb1fb") }
MongoDB server version: 4.2.3

> db
test

> use admin
switched to db admin

> db
admin

> db.auth("adminUser", "adminpassword")
1

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> use express-security-pug
switched to db express-security-pug

> db.createUser({ user: "owner_express-security-pug", pwd: "passowrd", roles: [{ role: "dbOwner", db: "express-security-pug" }] })

Successfully added user: {
    "user" : "owner_express-security-pug",
    "roles" : [
        {
            "role" : "dbOwner",
            "db" : "express-security-pug"
        }
    ]
}

> db
express-security-pug

> exit

Connection string to connect to express-security-pug db using the owner_express-security-pug user.

mongodb://owner_express-security-pug:password@localhost/express-security-pug

Download the code from GitHub

Pls. go to my GitHub site and clone the code. Here you find some inline documentation in the code.

Create your app home directory express-security-pug

My app home directory is different to the one that is available after you cloned the code from GitHub.

Patricks-MBP:2020-05-15-express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security

Patricks-MBP:2020-05-15-express-security patrick$ mv node-part-5-express-security-with-db-pug express-security-pug

Patricks-MBP:2020-05-15-express-security patrick$ cd express-security-pug

Patricks-MBP:express-security-pug patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security-pug

Install PUG and use it in your app

First we install PUG as a dependency.

Patricks-MBP:express-security-pug patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security-pug

Patricks-MBP:express-security-pug patrick$ npm install pug --save

PUG is already fully integrated into Express. Pls. read the documentation how to use template engines in Express.

After you installed PUG the view engine must be set in your main application file secserverpug.js.

// secserverpug.js
....
// use Pug Template Engine
app.set('view engine', 'pug')
app.set('views', './views')
....

These instructions tell your app that PUG template engine is used and that the templates can be found in /views directory.

PUG Directory setup

In /views I setup the templates for home, login, registration and an error template.

In /views/includes I setup the files containing HTML or JavaScript. These can be included in the templates.

Patricks-MBP:express-security-pug patrick$ ls -l
total 128
-rw-r--r--    1 patrick  staff    771  1 Jul 06:04 README.md
drwxr-xr-x    5 patrick  staff    160 29 Jun 05:26 database
drwxr-xr-x  150 patrick  staff   4800 29 Jun 06:11 node_modules
-rw-r--r--    1 patrick  staff  47547 29 Jun 06:11 package-lock.json
-rw-r--r--    1 patrick  staff    367 29 Jun 06:11 package.json
-rw-r--r--    1 patrick  staff   4393  2 Jul 05:34 secserverpug.js
drwxr-xr-x    3 patrick  staff     96 29 Jun 05:26 static
drwxr-xr-x    8 patrick  staff    256  2 Jul 05:45 views

Patricks-MBP:express-security-pug patrick$ ls -l views
total 40
-rw-r--r--  1 patrick  staff   549 30 Jun 05:35 dashboard.pug
-rw-r--r--  1 patrick  staff   522  2 Jul 05:50 err.pug
-rw-r--r--  1 patrick  staff   420 29 Jun 05:39 home.pug
drwxr-xr-x  6 patrick  staff   192 29 Jun 05:17 includes
-rw-r--r--  1 patrick  staff   735 30 Jun 05:02 login.pug
-rw-r--r--  1 patrick  staff  1067 30 Jun 05:08 register.pug

Patricks-MBP:express-security-pug patrick$ ls -l views/includes
total 32
-rw-r--r--  1 patrick  staff   76 29 Jun 05:39 foot.pug
-rw-r--r--  1 patrick  staff  167 29 Jun 05:24 head.pug
-rw-r--r--  1 patrick  staff  489  2 Jul 05:13 nav.pug
-rw-r--r--  1 patrick  staff  420 29 Jun 05:08 script.js

Patricks-MBP:express-security-pug patrick$ 

The responsive Website Design

Each site like home, login, register and dashboard has a specific site template in /views directory. The site content will be defined in the main section of each template. PUG enables files with HTML or JavaScript to be included. This makes the site templates clear and easy to maintain. The includes are located in /views/includes directory.

The website is build based on a grid design and each site template has the following structure.

doctype html

HTML

	Head
		include includes/head.pug	
	
	Body
		
		Grid-Container
			
				Header
					include includes/nav.pug
			
				Main
					... site template specific HTML ...
			
				Footer
					include includes/foot.pug
					
		<script>
			  include includes/script.js

The design of the website is defined in the css in static/css/style.css.

Here in the css we define the Site Structure as grid areas consisting of header, main and footer and link them to the grid-container.

....

.header { grid-area: header; background-color: #ffffff; border-radius: 5px;}
.main { grid-area: main; background-color: #ffffff; border-radius: 5px;}
.footer { grid-area: footer; background-color: #ffffff; border-radius: 5px;}

.grid-container {
  display: grid;
  grid-template-areas:
      "header"
      "main"
      "footer";
  grid-gap: 5px;
  background-color: #d1d1e0;
  padding: 50px;
}

....

The Navigation is defined in the Header area of the Grid-Container and the HTML comes into the template via include includes/nav.pug.

// includes/nav.pug

//(this) refers to the DOM element to which the onclick attribute belongs to
// the a DOM element will be given as parameter to the function

a(class="burgericon" onclick="myFunction(this)")
  div(class='burgerline' id='bar1')
  div(class='burgerline' id='bar2')
  div(class='burgerline' id='bar3')

a(class='link' href='/') Home
a(class='link' href='/login') Login
a(class='link' href='/register') Register
a(class='link' href='/dashboard') Dashboard
a(class='link' href='/logout') Logout

So the navigation design is then defined in the css. Each navigation object is an a link. We have a links with class link and burgericon. The burgericon is used to open the navigation bar onclick when the screen is smaller than 600px (like iphone displays etc., explained below), is cosisting of 3 burgerlines and these lines are created using 3 div objects with class burgerline. The burgelines will be transformed with speed 0.4s when you click on the burgericon (explained below). The burgericon is not visible and aligned on the right edge. All other navigation links are visible and aligned on the left edge.

/* static/css/style.css */
....

/* style the navigation links with float on the left (side by side) */
.header a.link {
  float: left;
  display: block;
  padding: 14px 16px;
  text-decoration: none;
  font-size: 1.4vw;
  color: #28283e;
}

/* hover effect for each navigation link */
.header a.link:hover {
  background-color: #28283e;
  color: #ffffff;
}

/* style the burgericon link on the right */
.header a.burgericon {
  float: right;
  display: none;
  padding: 14px 16px;
}

/* style each burgerline that create the burgericon */
.burgerline {
  width: 35px;
  height: 5px;
  background-color: #28283e;
  margin: 6px 0;
  transition: 0.4s;
}
....

When the display screen is lower than 600px the navigation links will not be shown and the burgericon (on the right side) will be faded in instead.

/* static/css/style.css */
....
/* for screens up to 600px remove the navigation links and show the burgericon instead */
@media screen and (max-width: 600px) {
  .header a.link { display: none; }
  .header a.burgericon { display: block; }
}
....

When you click on the burgericon the burgerlines will be transformed so that you will see a cross instead of the hamburger like icon. The 2nd burgerline with id='bar2' will not be shown at all while the other 2 burgelines will be rotated 45 degrees counterclockwise (burgerline with id='bar1') and clockwise (burgerline with id='bar1').

/* static/css/style.css */
....
/* style burgerlines after on onclick event */
/* the .change class will be added onclick with classList.toggle in the JavaScript */
/* rotate first bar */
.change #bar1 {
  /* rotate -45 degrees (counterclockwise) move 15px down in Y-direction */
  transform: rotate(-45deg) translateY(15px);
}
/* fade out the second bar */
.change #bar2 {
  opacity: 0;
}
/* rotate third bar */
.change #bar3 {
  /* rotate +45 degrees (clockwise) move 15px up in Y-direction */
  transform: rotate(45deg) translateY(-15px);
}
....

After clicking on the burgericon, the burgerlines are transformed as described. The links of the navigation menu are displayed one below the other (float none) and aligned left.

/* static/css/style.css */
....
/* for screens up to 600px and after onclick event the responsive class will be added to the header */
@media screen and (max-width: 600px) {
  /* show navigation links left with no float (links shown among themselves) */
  .header.responsive a.link {
    float: none;
    display: block;
    text-align: left;
  }
}
....

All onclick functionalities are controlled by the javascript which is embedded in the HTML of each site template (pls see above includes/nav.pug). In the HTML, the onclick event is initiated in the burgericon link and the function myFunction is called with onclick =" myFunction(this) ". With the parameter this the entire burgericon object is transferred to the javascript function.

With each click on the burger icon, the class change is added to each burgerline or, if available, removed. This is done by the toggle() function. If change is set, the hamburg icon is transformed into a cross according to the specification in the css (see above). If change is withdrawn with a new click, the hamburger icon is displayed again.

But it happens even more in the javascript when you click on the hamburger icon. The element that has the id responsivenav is searched for and the variablereponsiveNavElement is assigned to this element. Is the class of the reponsiveNavElement header the classresponsive is added after clicking on the hamburger icon. If the class responsive is set, as described above, the links of the navigation menu are displayed one below the other (float none) and aligned left. So it applies in the css .header.responsive a.link {....}

In all other cases only the class header is set. So it applies in the css .header a.link {....} and the navigation links are not shown.

// includes/script.js

// the (burgerlines) parameter represent the DOM element that has been given to the function
function myFunction(burgerlines) {
  burgerlines.classList.toggle('change');

  var reponsiveNavElement = document.getElementById('responsivenav');
    if (reponsiveNavElement.className === 'header') {
      reponsiveNavElement.classList.add('responsive')
    } else {
      reponsiveNavElement.className = 'header';
    }
  }

Finally at the end of the css we define the defaults for h1, for our text content, the forms, the input fields and the send buttons.

Summary and Outlook

In this part 4 of my little node.js series we have seen how to setup a production ready environment for our express app. I showed this using a Mac OS, but in principle this setup also applies to Linux, for example.

The basic setup is, to put it simply, the app runs as a service on the server in the background using a Process Manager, but has no interface to the client. This client interface regulates a reverse proxy which is upstream of the app and accepts all requests and forwards them to the app, as well as the responses from the app back to the client. The communication is SSL/TLS secured.

At the center of the setup is a separate local MongoDB that manages all application data. In our example, these are the users, but also the sessions. I prefer to set up my own MongoDB on my server but of course it is conceivable to use a cloud-based solution or to install another database locally.

The express app itself uses secure HTTP response headers so that HTTP attacks like clickjacking, MIME type sniffing or some smaller XSS attacks on the client are made as difficult as possible. Access to personal areas of the application is secured by session-based authentication and authorization. The session-relevant data is stored in the database and not in the browser cookie, which means additional security with regard to attacks on the client. The browser cookie only contains a hash of the session ID to query the relevant user data from the database.

I would like to end my node.js series with Part 4. I discussed and demonstrated the basic concepts and procedures in parts 1 to 4. Of course there will be other interesting articles on the topic node.js and web programming on Digitaldocblog. Just take a look.

The creation of the database and the cloning of the application root directory from my GitHub page will be briefly described in the following step-by-step instructions. This is followed by a description of the application functionality. Then I will explain more generally how data is modeled in a database-based Expressjs application with the help of mongoose and how the data is accessed so that the more basic software architecture is understood. The relevant code of our blog app is commented. In this respect, the explanations of the individual code passages can be found in the inline documentation.

Create a MongoDB Database

After the installation of MongoDB you have started the mongod service on your system. It is very important that you setup MongoDB with client authentication enabled. You have also created an admin user with a password on your admin default db. Pls. follow the description in my article Mongodb and Mongoose on Ubuntu Linux.

Then you create a database for your project and a user for your database using the mongo command on the console.

Patricks-MBP:digitaldocblog-V3 patrick$ mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("4e8c592e-104d-48ef-b495-62129e446d23") }
MongoDB server version: 4.2.3

> db
test

> show dbs

> use admin
switched to db admin

> db.auth("admin", "YOUR-ADMIN-PASSWD")
1

> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB

> use YOUR-DB-NAME
switched to db YOUR-DB-NAME

> db
YOUR-DB-NAME

> db.createCollection("col_default")
{ "ok" : 1 }

> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
YOUR-DB-NAME	    0.000GB

> db.createUser({ user: "YOUR-DB-USER", pwd: "YOUR-DB-USER-PASSWD", roles: [{ role: "dbOwner", db: "YOUR-DB-NAME" }] })
Successfully added user: {
	"user" : "YOUR-DB-USER",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "YOUR-DB-NAME"
		}
	]
}

> show users
{
	"_id" : "YOUR-DB-NAME.YOUR-DB-USER",
	"userId" : UUID("ef080b98-849f-41f4-b561-a658a88e4595"),
	"user" : "YOUR-DB-USER",
	"db" : "YOUR-DB-NAME",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "YOUR-DB-NAME"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
YOUR-DB-NAME	    0.000GB
> 

> exit
bye

If you enter the command mongo on the console, the MongoDB client connects to the MongoDB service. The MongoDB client is the so-called MongoDB shell with which the user can work on the console. After the connection is established, the MongoDB shell is displayed in the terminal and with the command db we see the output test.

The db command shows the database you are currently connected to. test is the default database and is actually no longer used. The show dbs command shows all available databases. Since we have activated client authentication nothing is shown here because we are not yet authenticated.

With use admin we switch to admin db and authenticate ourselves with db.auth. After we have entered the db.auth command in the MongoDB shell as described, the shell shows us a 1. This shows that the authentication was successful. As admin user we can now show all databases with show dbs.

With the command use YOUR-DB-NAME we create the database YOUR-DB-NAME and also switch from the database admin to the database YOUR-DB-NAME. With the command db.createCollection I always create a default collection in YOUR-DB-NAME to display the new database YOUR-DB-NAME with the show dbs command in the MongoDB shell. If you simply create the database and no collection the new database is not shown with the show dbs command.

Then I create a user with db.createUser for my new database YOUR-DB-NAME, assign the role dbOwner to this user and the password YOUR-DB-USER-PASSWD.

The database YOUR-DB-NAME is now created and with exit we leave the MongoDB shell.

Create the application root directory

Clone the Simple Blog App from my GitHub page and then switch to the directory node-part-3-express-blog-with-db-V2.

Patricks-MBP:~ patrick$ cd node-part-3-express-blog-with-db-V2

The application root directory then looks like this.

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l
total 72
-rw-r--r--   1 patrick  staff    768 9 Apr 05:19 README.md
-rw-r--r--   1 patrick  staff   2162 9 Apr 05:19 blog.js
drwxr-xr-x   6 patrick  staff    192 9 Apr 05:19 database
drwxr-xr-x   3 patrick  staff     96 9 Apr 05:19 modules
drwxr-xr-x  77 patrick  staff   2464 9 Apr 05:19 node_modules
-rw-r--r--   1 patrick  staff  22302 9 Apr 05:19 package-lock.json
-rw-r--r--   1 patrick  staff    188 9 Apr 05:19 package.json

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l database
total 16
drwxr-xr-x  4 patrick  staff  128  9 Apr 05:19 controllers
-rw-r--r--  1 patrick  staff  704  9 Apr 05:19 db_.js
drwxr-xr-x  4 patrick  staff  128  9 Apr 05:19 models

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l database/controllers
total 40
-rw-r--r--  1 patrick  staff  9464 9 Apr 05:19 blogController.js
-rw-r--r--  1 patrick  staff  5306 9 Apr 05:19 userController.js

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l database/models
total 16
-rw-r--r--  1 patrick  staff  1574 9 Apr 05:19 blogModel.js
-rw-r--r--  1 patrick  staff  1216 9 Apr 05:19 userModel.js

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l modules
total 8
-rw-r--r--  1 patrick  staff  175  9 Apr 05:19 logger.js

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ 

Read the README.md. After you run npm install You must adapt your database connection string and rename database/db_.js into database/db.js. Your database connection string ist as follows.

mongodb://YOUR-DB-USER:YOUR-DB-USER-PASSWD@localhost:27017/YOUR-DB-NAME

The application should run with npm blog.js. The application code is explained and commented inline. In addition, I will explain details in the following chapters.

How does the application work ?

The application looks like a very simple web application but already contains a lot of program logic. This program logic is mainly contained in the two user and blog controllers.

The application data are users and blogs. These data are stored in a MongoDB and can be created, changed or deleted by the application. When data is entered, i.e. when data sets are created and when data fields are changed, input validation takes place. This input validation is based on the mongoose built in input validation and especially for string values on the mongoose custom validation, whereby it is checked whether the input of a string corresponds to a defined regular expression (regex for ECMA Java Script).

The primary identifier for users is the email address. The users must be unique and can only be created if the email address is not already assigned to an existing user.

The users can be blog authors and therefore they have a reference to each blogid whose author they are. Whenever a blog post is created, the application checks whether a user with the specified email address already exists and whether the email address already belongs to an existing user. If the user does not exist or the specified email address belongs to an existing user, the blog post cannot be created. If a user exists and it is also his own email address, the new blog post can be created and the author’s name, first name and email address are stored on the blog in the database. In parallel the blogid of the new blog post is stored as reference on the user object.

We thus have a reference from users to their blogs where they are authors and a reference from blogs to the users or blog authors. When creating a blog, it is a condition that the author exists as a user in the database, but it may still be that we have blogs with authors in the database for which there is no longer a user. This is because it is possible to delete a user without deleting all blogs where the user is the author. Therefore it may still be that we have blogs with authors in the database for which there is no longer a user.

Consequently, when a blog is deleted, it is checked whether the author exists as a user. If the user no longer exists, the blog entry will still be deleted. If there is a user in the database, the blog post is deleted and the reference blogid on the user is deleted at the same time.

But we can also update the email address of a user. This is a very realistic use case because email addresses can change for users and consequently a user would like to change their changed email address accordingly in our application. Because the email address is the unique identifier of a user, we have to change the email address both on the user and on all blogs where the user is entered as the author.

Our application can also display blogs. All blogs are displayed or only the blogs for a specific year or month or only for a defined date. In order to display the blogs in this way, the blog data are selected accordingly from the database.

In addition, the application also displays a homepage and an about page, which have no further significance.

Data Models

To organize our application data in the MongoDB we define so called models using Mongoose. Models contain the essential data structure and data definitions that a database collection should have. Our database will have 2 collections one for the users and one for the blogs.

For each database collection exactly one model is defined in a seperate file. In our example, we need one model for each of the two collections col_users and col_blogs.

For col_users the model is defined in the file

• database/models/userModel.js
  

for the col_blogs the model is defined in the file

• database/models/blogModel.js.
  

I would like to show a standard model here.

// model.js
....

mongoose = require('mongoose')
var Schema = mongoose.Schema

var dataSchema = new Schema ({ key1:..., key2:..., ..., keyN:...}) 

var Data = mongoose.model('col_data', dataSchema)

module.exports = Data

With mongoose.Schema we create a database Schema. With new Schema(…) we define exactly what the Schema looks like and which keys or data fields are defined for each data object that will be stored in the collection. The new Schema is stored in a variable that typically describes the purpose of the new Schema (userSchema or blogSchema in our app). With mongoose.model() we create the model and map it to the database collection. The first argument in the mongoose.model() function is the singular name of the database collection. So if we create the model with col_data mongoose is looking for col_datas collection in the database. The second argument in the callback is the new Schema that we defined before. This model will be stored in the Variable Data and exported with module.exports.

It is extremely important that all input data are validated before the data is saved in the database. This ensures that the application only processes validated data.

In the model definition of or blog application, I mark all auto data fields with a preceding underscore. With auto data fields, the value is either created by the database (like the id) or a default value is always set (like created). This means that these data fields are never manipulated by data input.

Input data fields are subject to mongoose built-in validation. The topic of input validation is described in great detail in the mongoose documentation. Nevertheless, I would like to go into a few points of input validation here.

Basically the specification of validation properties takes place in the database Schema definition per key of the data object. We have built-in validation properties or short validators like required, minlength or maxlength but also the option to define a custom validation function using the property validate. Here in the example below the custom validation function of the key „name“ checks the input for the match with a regex. If the input value does not correspond to the regex, the function return false and the validation has failed. If the function return true, the data can be saved and the validation was successful.

// model.js

....

mongoose = require('mongoose')
var Schema = mongoose.Schema

var dataSchema = new Schema ({ 

 // define the auto data fields
  _id: {
    type: Schema.ObjectId,
    auto: true
  },

  _created: {
    type: Date,
    default: Date.now()
  },

  // define the input data fields
  name: {
    type: String,
    required: true,
    // input string validation function
    // input sting must match regex criteria
    validate: function(name) {
      return /^[a-zA-Z0-9üÜäÄöÖ.’-]+$/.test(name)
    }
  },  
....

})

....

Data Controllers

To organize our application data accesses and data manipulations using Mongoose we define so called data controllers in separate files.

In principle, each data model that we have defined in our data model files are representing a data perimeter. A data perimeter can be accessed via so-called controller modules which are defined in separate controller files. These controller modules are in turn defined as controller functions in the corresponding data controller.

These controller functions define the data operations like search queries, create, update and delete data within the controller functions. These data operations are defined in a try … block and if an error occurs when accessing the data, this error is forwarded to the catch … block. This catch block then takes over the error handling. I do this try catch thing for all operations except for save(). With save() errors are handled directly in the callback.

In addition to the req and res parameter, the controller functions also receive next as a parameter. This is required to call the default error handler of the server which is defined in our main application file blog.js.

If the request cannot be ended with a response due to an error accessing the data, then the request will be passed to the catch block. The catch block call the next(error) function with the error as parameter in order to pass control to the default error handler to respond to the request.

If the request cannot be ended with a response due to an err accessing the data, then the request will also be passed to the catch block. But the catch block then call an individual error handler to respond to the request.

In our application we have the user data perimeter and the blog data perimeter and for each of these different data perimeters there is a separate data controller.

User Controller

The user data controller exists to control data access and data manipulations for the user data perimeter.

• database/controllers/userController.js
  

The user data controller contain the following user controller modules and functions.

createUser

The createUser function create a new User object in the database. We try first to find a user with the given email address.

If an error occurs here in the query, the request error is passed to the default error handler via catch block. If no error occurs, either a user is found or not. If a user is found, the user cannot be created with createUser and the response is 400 bad request.

If no user is found, the user can be created. This means that the newUser object is saved and the input data is validated when it is saved. If an input data validation error occurs, we will respond with 400 bad request. otherwise there is a 200 ok response and the new user object has been successfully created in the database.

updateUserEmail

With the updateUserEmail function, the email address of a user object can be changed. In parallel also the author email will be adapted on the blog objects where the user is author of blogs.

First the user is searched for with his existing email address. If the function runs on an error, the request and the error are passed on to the default error handler with catch.

If no user is found, we respond with 400 bad request. If, on the other hand, a user is found, the email address can be updated by saving the user object with the new email address. When saving, the input validation takes place again. Only if the input validation was successful the storage of the user object is completed.

Now the email address must be updated on the blogs where the user is the author.

For this we are looking for all blogs where the author email corresponds to the existing email address of the user. If the search runs for an err, we will respond with an individual 502 bad gateway this time because the request has been partially processed up to this point. The email address on the user object has already been updated, but the data for updating the author email addresses on the blogs cannot be queried.

If there is no err in the blog query, the user may not be the author of any blog. Then we answer with 200 ok and confirm that only the email address on the user object has been changed.

If, on the other hand, the user is the author of blogs, we use the updateMany() function to update the email address of any blog found.

This updateMany() function receive a filter object with the previously existing email address of the user as first parameter and then the update object to replace this existing email address with the new email address of the user. This operation is executed on any blog found in the database and is also carried out in a try block. If an err occurs, the request is passed on for individual error handling to the catch block. Here in the catch block we answer with a 502 bad gateway because the update on the user object has been already performed successfully but the update of the author email address on the blogs found was not successful because the updateMany() operation failed.

If the blog update process was successful, we respond with 200 ok. In this case the email address has been changed on the user object and also on his blogs.

removeUser

With removeUser function a user object can be removed from the database. We try first to find the user that should be removed with the given email address. If an error occurs, the request error is passed to the default error handler via catch block.

If no error occurs, either a user is found or not.

If no user is found, the user cannot be removed and the response is 400 bad request.

If the user to be removed has been found we try to delete this user from the database using the deleteOne() function. If an error occurs here, the request error is also passed to the default error handler via catch block.

If the deletion was successful we respond with 200 ok.

Blog Controller

The blog data controller exists to control data access and manipulations for the blog data perimeter.

• database/controllers/blogController.js
  

The blog data controller contain the following blog controller modules and functions.

createBlog

The createBlog module create a new blog object in the database and update the new blog id reference on the author user object.

The new blog object can only be created if the author data email, first- and lastname match with a user object in the database. So the author must already exist as a user in the database.

We try first to find a user with the author email address. If an error occurs, the request error is passed to the default error handler via catch block. If no error occurs, either a user is found or not.

If no user is found, we respond with 400 bad request.

If a user is found, it is checked whether the provided first name or last name does not match the first name or last name of the user in the database found with the given email address. If one of the names does not match, the condition is false and we send a 400 bad request.

If both names are the same, there is a perfect match of email, first name and last name and the provided author data are validated. Then the blog object can be saved and the blog input data will be validated.

If an input data validation error occurs, we will respond with 400 bad request.

In any other case the new blog has been saved to the database and we try to update the user object by adding the new blog id on the user object as a blog reference. Therefore, we try to use the updateOne() function to find the user object based on the author’s email address to add the blog id of the newly created blog object as a blog reference.

This operation is carried out in a try block and if an err occurs, it is passed on to the catch block. In case of an err we answer with a 502 bad gateway. The blog object is created but the user object is still not updated because the updateOne() operation was not successful. If the updateOne() operation was successful we respond with 200 ok.

removeBlog

The removeBlog module remove a blog object from the database and update the user object by removing the relevant blog id reference on the author user object.

We try to find a blog object that should be removed based on the blog title. If an error occurs during that query the default error handling function will be invoked via the catch block and next(error). In case of no error the query return one blog object or null which means that no blog object has been found.

If no blog has been found we respond with 400 bad request.

If the query was successful and we found a blog object we try to find a user based on the author email address. If an error occurs the default error handling function will be invoked via the catch block. In case of no error the query return one user object or no user object.

If we have no user object found that mean that the author of the blog is no longer a user in our application. In this case we try to remove only the blog using the deleteOne() function. If this blog deletion function fail we have no success and an error. We catch this error in the catch block to invoke the default error handler function. If the deleteOne() function can be executed successfully, we respond with ok 200. The blog object with the title has been deleted.

If we have found a user object this means that this user is the author of the blog object we want to delete. In this case we first try to remove the blog object using the deleteOne() function and catch the error in the catch block to invoke the default error handler function when the execution of deleteOne() fail.

When the blog deletion was successful, we also try to delete the blog id as a reference on the user object using the updateOne() function. If updateOne throw an err we go ahead with catch and respond with 502 bad gateway as the blog could be removed but the user object update failed.

If the user update of the user object was successful we respond with 200 ok.

returnAllBlogs

The returnAllBlogs module is a function that searches all blog objects in the database and returns them in an array blogs when the requested url is /blogs. If the requested url is different than the next() function will be called to transfer to the next request handler in the row.

We try to search for all blog objects using the find() function. This find function return an error in case the query fail. In this case the catch block will invoke the default error handler function. In case the query was successful the function return an array blogs including the blog objects.

In case the array is empty or better the length is equal to 0 no blog object has been found in the database and we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into a blog array and return this blog array with 200 ok.

returnYearBlogs

The returnYearBlogs module is a function that searches blog objects of a certain year in the database and returns them in an array blogs when the requested url is /blogs/year. If the requested url is different than the next() function will be called to transfer to the next request handler in the row.

We try to search for all blog objects in a year using the find() function and return an error in case the query fail. In this case the catch block will invoke the default error handler function.

In case the query was successful the function return a query array blogs including the blog objects found for the relevant search criteria. In case no blog object has been found (length of blogs is equal to 0) in the database we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into another blog array and return this blog array with 200 ok.

returnMonthBlogs

The returnMonthBlogs module is a function that searches blog objects of a certain year and month in the database and returns them in an array blogs when the requested url is /blogs/year/month. If the requested url is different than the next() function will be called to transfer to the next request handler in the row.

We try to search for all blog objects in a year and month using the find() function and return an error in case the query fail. In this case the catch block will invoke the default error handler function.

In case the query was successful the function return a query array blogs including the blog objects found for the relevant search criteria. In case no blog object has been found (length blogs is equal to 0) in the database we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into another blog array and return this blog array with 200 ok.

returnDateBlogs

The returnDateBlogs module is a function that searches blog objects of a certain date in the database and returns them in an array blogs.

As this is the last routing handler in the app.get() routing function where the path is defined as /blogs/:year?/:month?/day? the requested url matches this definition or not.

In case the requested url is something like /blogs/year/month/day/else the url is not defined and then the default route error handler will be invoked and return 404 Not found.

In case the requested url path match /blogs/year/month/day we try to search for all blog objects for the date using the find() function and return an error in case the query fail. In case of an error the catch block will invoke the default error handler function.

In case the query was successful the function return a query array blogs including the blog objects found for the relevant search criteria.

In case no blog object has been found (length blogs is equal to 0) in the database we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into another blog array and return this blog array with 200 ok.

Data operations in detail

Our application perform the following data operations.

• create data
• search and return data
• update data
• delete data
  

These operations will be implemented in the controller modules. We use the following functions to perform these operations.

• save()
• find()
• findOne()
• updateOne()
• updateMany()
• deleteOne()
  

If we want to create a new Data object we call new Data( … ) and apply the key specifications according to the Schema defined in the model.

note: Please note that we are using new Data( … ) here because we export Data in the above model example definition. If you look in the model files of our blog app, you will notice that I export in the user model User and in the blog model Blog. Consequently, for example, when creating a new user object in the user data controller, the call new User( … ) is made.

The function save() receive a callback function with error as first parameter and data as second parameter. Within the callback we can work with error as the error object and with data as the data object that will be saved in the database. With save(), error handling takes place directly in the call back function using the if (error) … else … block.

// controller.js

newData = new Data({key1:..., key2:..., ..., keyN:...})

newData.save(function(error, data) {
	if (error) {

	// do something when an error occurs

	} else {

	// do what you need to do when the creation was successful

	} 
})

If we want to search and return data we call find() or findOne().

If we expect that a search return several data objects as a result we are using the find() function.

The find() function is covered with the try … catch … blog.

find() receive as first parameter a filter object as search criteria and as second parameter a callback function with error as first parameter and data as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with data which is an array containing all the data objects that have been found in the database. In case no data objects were found the data array is empty. In case the function fail we catch the error.

Suppose you want to search in Data perimeter for data objects that have a specificName. Then we pass the filter object name: specificName and then the callback function(error, data) … to the Data.find() function.

// controller.js

// create the filter object
var searchDataWithName = { name: specificName }

try {

	Data.find(searchDataWithName, function(error, data) {

	if (data.length == 0) {

	// do something when no data have been found 

	} else {

	// do something with the data array that have been found 

	}

	})

} catch (error) {

// do something when an error occurs

}

If a search should return only one data object as a result we are using the findOne() method.

The findOne() function is covered with the try … catch … blog.

findOne() receive as first parameter a filter object as search criteria and as second parameter a callback function with error as first parameter and data as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with data which is the object that has been found in the database. In case no data object has been found the value of data is null. In case the function fail we catch the error.

Suppose you want to search in Data perimeter for one data object that has a specificName. Then we pass the filter object name: specificName and then the callback function(error, data) … to the Data.findOne() function.

// controller.js


// create the filter object
var searchDataWithName = { name: specificName }

try {

Data.findOne(searchDataWithName, function(error, data) {

	if (!data) {

	// do something when no data have been found
	
	} else {

	// do something with the data in the data object that have been found 

	} 

	})

} catch (error) {

// do something when an error occurs

}

If we want to update an existing Data object we search the Data object, apply the changes and then call save(). We can also use updateOne() or updateMany().

If we do search and save() the data object first must be found in the database and will then be returned using the findOne() function. Then we can apply the changes to one or more data attribute values i.e. update the email address of a user and then call save() to save the updated data object back into the database. This procedure has the advantage that input validation is used with save() and is therefore recommended whenever we receive changes directly as input values and these input values must be validated.

// controller.js

// create the filter object
var searchDataWithName = { name: specificName }

try {

	Data.findOne(searchDataWithName, function(error, data) {

	if (!data) {

	// do something when no data have been found
	
	} else {

		// specify the update value
		var updateEmail = newEmail
	
		// update the value
		data.email = updateEmail

		data.save(function(err, updatedData) {

			if (err) {

			// do something when an err occurs
			
			} else {

			// do something when the update was successful

			}
		})
	} 
	})

} catch (error) {

// do something when an error occurs

}

If we use updateMany() or updateOne() the data objects or the one data object will be looked up and updated in the database in one step. The data object(s) will not be returned. This makes processing faster and saves network resources.

The functions updateMany() and updateOne() are both covered with the try … catch … blog.

updateMany() receive as first parameter a filter object as search criteria, as second parameter the update object and as third parameter a callback function with error as first parameter and result as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with result which is an object containing data about the processing result n: 1, nModified: 1, ok: 1 . This mean n objects found, n objects modified and the processing was ok (ok: 0 if the processing failed).

Suppose you want to find in Data perimeter for data objects that have a specificName and update all objects found with specificNewName. Then we pass the filter object name: specificName , the update object name: specificNewName and then the callback function(error, result) … to the Data.updateMany() function.

// controller.js

// create the filter object
var searchDataWithName = { name: specificName }

// create the update object
var updatedName = { name: specificNewName }

....

try {
      
   Data.updateMany(
      	
     searchDataWithName, 
      	
     updatedName, 
      	
     function(err, result) {
      
       // do something with the result
          
     })
     
} catch (err) {
     	
// do something with the err

}

....

updateOne() also receive as first parameter a filter object as search criteria, as second parameter the update object and as third parameter a callback function with error as first parameter and result as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with result which is an object containing data about the processing result n: 1, nModified: 1, ok: 1 . This mean n objects found, n objects modified and the processing was ok (ok: 0 if the processing failed).

Suppose you want to find in Data data perimeter for one data object that has a specificName and you want to update this single object with specificNewName. Then we pass the filter object name: specificName , the update object name: specificNewName and then the callback function(error, result) … to the Data.updateOne() function.

// controller.js

// create the filter object
var searchDataWithEmail = { name: specificEmail }

// create the update object
var updatedEmail = { name: specificNewEmail}

try {
   
   Data.updateOne(
      	
     searchDataWithEmail, 
      	
     updatedEmail, 
      	
     function(err, result) {
      
     // do something with the result
          
    })
     
} catch (err) {

// do something with the err

}

....

If we want to remove an existing Data object we search for the data object using findOne() to get the data object returned, and then call deleteOne().

We do this 2 step approach i.e. in removeBlogs because we use the returned blog object to determine first the id of the blog object and then use it to find and update the user object. You can do this in one single step only by using deleteOne(), but take into account that you will not receive a data object with which further queries or updates can be carried out.

deleteOne() receive as first parameter a filter object as search criteria and as second parameter a callback function with error as first parameter and result as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with result which is an object containing data about the processing result n: 1, deletedCount: 1, ok: 1 . This mean n objects found, n objects deleted and the processing was ok (ok: 0 if the processing failed). You can use the result object in the callback to check if the object to be removed has been found or not. But this an operation we don’t need to perform because we are using the 2 step approach with findOne() and then deleteOne().

....

if (result.n == 0) {
  // do something if no object to be removed has been found
} else {
  // do something when the delete was successful
}

....

Suppose you want to find in User data perimeter for a data object that has a specificName and you want to remove this object from the database. Then we pass the filter object name: specificName and then the callback function(error, result) … to the User.deleteOne() function.

// contoller.js

/ create the filter object
var searchDataWithName = { name: specificName }

try {

	User.findOne(searchDataWithName, function(error, user) {

	if (!user) {

	// do something when no data have been found
	
	} else {
	
		try {
		
		User.deleteOne(searchDataWithName, function(error, result) {
		
			// do something when the delete was successful
		
		})
		
		
		} catch (error) {
		
		// do something when an error occurs		
		}		
}
})


} catch (error) {

// do something when an error occurs

}

HTTP request Routing

In our application main file blog.js we define routes to process HTTP POST and HTTP GET requests.

HTTP POST requests are routed using the app.post(‚routingPath‘, routingHandler) method.

During the create-, update- and remove data operations app.post() routes are called and the request and response objects are transferred to the corresponding routingHandler to take over the request handling. The request body contain all relevant input data to process the request.

The routingHandler then calls the controller module and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends the response.

So for example if an HTTP POST request is made to the path /createusers the app.post(‚/createusers‘, userController.createUser) route is called and the createUser module take over the complete processing.

// blog.js

....

// load database controllers
const userController = require('./database/controllers/userController');
const blogController = require('./database/controllers/blogController');

....

// define routes (routing table)

....

app.post('/createusers', userController.createUser)

app.post('/updateuseremail', userController.updateUserEmail)

app.post('/removeusers', userController.removeUser)

app.post('/createblogs', blogController.createBlog)

app.post('/removeblogs', blogController.removeBlog)

....

You have to imagine it in reality with a real web application like this: On a website there is an HTML form and the end user of the application can enter user data such as last name, first name and email address to add for example a new user in the application or in the application database. The moment the user of the application clicks on the send button, the data entered in the form is transferred to the route /createusers with HTTP POST which means that the relevant app.post(‚/createusers‘, userController.createUser) function is called to handle this HTTP POST request.

Unfortunately we are not yet dealing with a real web application because there is no HTML form until now. This is exactly where Postman comes in to test an HTTP POST action. In principle, Postman replaces the form. With Postman I can (amongst other things) transfer HTTP POST requests to the application and thus I also transfer the request body. It is therefore necessary that you familiarize yourself with Postman to test the applications app.post() routes.

HTTP GET requests are routed using the app.get(‚routingPath‘, routingHandler) method.

So for example if an HTTP GET request is made to the path / the app.get(‚/‘, routingHandler) route is called and the routingHandler take over the complete processing and send directly the response. The same happens when an HTTP GET request is made on the route /about. This is easy and not spectacular.

// blog.js

....

// load database controllers
const userController = require('./database/controllers/userController');
const blogController = require('./database/controllers/blogController');

....

// define routes (routing table)

....

app.get('/', (req, res) => {
  res.send('Hello, this is the Blog Home Page.')
})

app.get('/about', (req, res) => {
  res.send('Hello, this is the Blog About Page.')
})

....

It is a bit more special when an HTTP GET request is made to the route app.get(‚/blogs/:year?/:month?/:day?‘, …. ).

The route definition here provides so-called request parameters. The /blogs route can be parameterized with the parameters year, month and day. The question mark behind each parameter indicates that the parameter is optional.

The request is forwarded to several routing handlers.

The first routing handler assigns the request url and the request parameters year, month and day to variables and calls next() to hand over the request to the next routing handler in the row.

The next routingHandler is blogController.returnAllBlogs. This routingHandler calls the controller module returnAllBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs as response if the request url is equal to /blogs otherwise next() is called to hand over the request to the next routing handler in the row.

The next routingHandler is blogController.returnYearBlogs. This routingHandler calls the controller module returnYearBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs in the specified year as response if the request url is equal to /blogs/year otherwise next() is called to hand over the request to the next routing handler in the row.

The next routingHandler is blogController.returnMonthBlogs. This routingHandler calls the controller module returnMonthBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs in the specified year and month as response if the request url is equal to /blogs/year/month otherwise next() is called to hand over the request to the next routing handler in the row.

The next and last routingHandler is blogController.returnDateBlogs. This routingHandler calls the controller module returnDateBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs from the specified date.

Any other route will end up in an error 404 Bad request.

// blog.js

....

// load database controllers
const userController = require('./database/controllers/userController');
const blogController = require('./database/controllers/blogController');

....

// define routes (routing table)

....

app.get('/blogs/:year?/:month?/:day?',

        (req, res, next) => {
          url = req.url
          year = req.params.year
          month = req.params.month
          day = req.params.day

          next()
        },

        blogController.returnAllBlogs,
        blogController.returnYearBlogs,
        blogController.returnMonthBlogs,
        blogController.returnDateBlogs

      )

....

Status Handling

In our application we work with the following HTTP Status codes.

• code 200: status: Ok. Request completed.
• code 500: status: Internal Server Error. Request failed due to server error.
• code 400: status: Bad Request. Request failed due to wrong input data.
• code 404: status: Not Found. Request failed due to wrong route.
• code 502: status: Bad Gateway. Request partly completed but not fully completed due to missing data.
  

Summary and Outlook

In this Part 3 of my node.js series I explained how to setup our express blog app using a MongoDB database. We learned how to create data models and controllers to control access to the data. But the blog app has still no HTML frontend or no HTML template rendering is used.

So in the next Part 4 of my node.js series I will introduce how to extend the blog app with an HTML interface. Therefore we implement the use of the template engine Pug. With Pug we can create an HTML interface for our blog application.

In Part 1 of my node.js series we have learned how to easily build a web server with node.js on-board resources. Lastly we installed the Express.js module and configured simple routes as application end points that respond to http client requests. In this part 2 I would like to take a closer look at the most important functionalities of Express.js with you.

In connection with Express, terms like framework or web api keep coming up. Everyone says Express is a node.js web application framework which is absolutely true. But what does this mean? I would like to explain this here with my own words.

Developers install the Express module in the application root directory and integrate the module with the require() function in the application main file. This is done with the following command line.

var express = require('express')
var app = express()

The express module is first referenced to the variable express with the require() function and an express() function is exported from the express module. The express() function generates an express application which is stored in the variable app. From this moment on, the developer has a whole range of Express Features available that can be used in the web application that is to be built. The range of express features is provided by the Express Framework.

The features provided relate to the development of web applications. That means all these features from the Express Framework are designed to program the response behavior of a web server to requests from a client exactly as it is desired. These features are essentially about Routing- and so-called Middleware-functionalities.

The Routing tools in the Express Framework enable the developer to program the response behavior of the web server to the request for a special application end point. An application end point is a route that is e.g. defined as „/“ or „/about“. The response from the server should be different here. The developer can program the end points in a very simplified way so that the request for „/“ is answered by the server with the Home Page, the request for „/about“ is answered with the About Page. The Request- and Response Objects are very central to the routing tools. The developer can access the properties of these objects using various methods and thus program his end points accordingly.

To continue the example above the server immediately responds to the client’s request and delivers either the Home Page or the About Page. If the server should process after the request and before the response some code, we are talking about Middleware. Express offers various methods to access express internal middleware and then switch it between the request and the response. You can also program your own middleware functions, which can then be integrated into the application with the require() function.

All in all, Express is a very powerful framework that is all about programming web servers or web applications. The Express Features serve as an application programming interface between the client and server and that is why these features are also called Express web api. We will take a closer look at Express Routing and Express Middleware in this Part 2 of the node.js series.

Chapter 2: Express Routing

We start here with the directory structure of the application root directory from Part 1 of my node.js series and explain the content of the server.js file.

Patricks-MBP:express-basic patrick$ ls -l
total 56
drwxr-xr-x  52 patrick  staff   1664 21 Mär 08:13 node_modules
-rw-r--r--   1 patrick  staff  14292 21 Mär 11:28 package-lock.json
-rw-r--r--   1 patrick  staff    162 21 Mär 10:16 package.json
-rw-r--r--   1 patrick  staff   2435 21 Mär 17:47 server.js
Patricks-MBP:express-basic patrick$ 

The server.js file is the application main file and contain the code of the application. The package.json file contain meta data of the application or project such as which external modules or dependencies have been installed to be used by the application. The package-lock.json file contain the complete directory tree of all installed dependencies and finally the directory node_modules contain all dependencies or external modules or packages that have been installed.

At the end of part 1 we already installed the external module Express.js with npm install express –save and can now load this module in our application. The server.js file looks like this.

// server.js

// load http module
const http = require('http')

// load third party Express module
const express = require('express')
const app = express()

// define the routes
app.get('/', (req, res) => {
  res.send('Hello, this is my home Page')
})

app.get('/about', (req, res) => {
  res.send('Hello, this is my about Page')
})

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My node.js web server is alive and running at port 3000')

We do the following with the code in the server.js file (from top to bottom).

1. load modules
2. define routes
3. create the server
4. define the port the server is listening
5. log a success message to the console when all the code runs without problems
   

The Routing method

As you can see in the code we already defined 2 simple routes or application end points (URIs). These routes are defined with routing methods and have the following notation.

app.routingMethod( 'routingPath', routingHandler(req, res) => { ... })

In our server.js file, we created an app as an instance of the express class by calling the express() function right after we loaded the express module. Routing methods can now be attached to this app. Each routing method corresponds to an http verb such as GET, POST, PUT or DELETE. In our example, we will deal with get() and post() here. The routing method receives 2 parameters:

1. routingPath: the path of the route end point starting from the root of the application „/“. Simply this is the path that the http request passed to the application. If the path passed meets an end point definition, the routingHandler code is executed.
2. routingHandler: the routingHandler is a callback function that receives parameters. Here in the notation above the callback receives 2 parameters. The first parameter is the request object, the second the response object. Because the parameters req and res were passed, the code has access to the methods and properties of the request and response objects. We will talk about these Objects later in this article.
   

While the routingPath is a pretty logical thing, we’ll take a closer look at the routingHandler. The routingHandler contains code that is executed after a request and sends the response.

The routing handler

The routing handler is a callback function with req and res as parameters and process code after the requests and before sending the response. The response is sent with the res.send() method back to the client.

app.routingMethod( 'routingPath', routingHandler(req, res) => { 

... 

})

Here the res.send() method send a string back to the client.

// server.js

.........

// define the routes
app.get('/', (req, res) => {

  res.send('Hello, this is my Home Page')

})

......

The routing handler can also receive a function as a parameter.

app.routingMethod( 'routingPath', routingHandler(req, res, next) => { 

... 

})

Here the routing handler receive req and res but also the function next() as parameter. The next() function forward the processing to the next function(req, res). This final function sen the response to the client with the res.send() method.

// server.js

.........

app.get('/', (req, res, next) => {
  console.log('this request will be responded by next()');
  next();
}, function (req, res) {
  res.send('This is the response to the request from next()');
})

.........

As we have seen above the routing handler can receive req and res and functions as parameter.

You can also collect routing handler functions into an array and pass the array as parameter into the routing method.

app.routingMethod( 'routingPath', [array] )

So lets define 3 functions and pass them as array into the routing method.

// server.js

.........

var r1 = function (req, res, next) {
  console.log('r1 ! The request will be responded by r3');
  next();
}

var r2 = function (req, res, next) {
  console.log('r2 ! The request will be responded by r3');
  next();
}

var r3 = function (req, res) {
  res.send('r3 ! Hello, this is the response !');
}

app.get('/example', [r1, r2, r3]);

.........

In the example above there are 3 routing handler functions that are collected in an array. The array is passed to the routing method as a parameter. r1 and r2 each execute a console.log instruction and then pass the routing on to the next routing handler function with next(). r3 then returns the response to the client using the res.send() method.

You can collect functions in an array and insert them into the routing method before a last routing handler.

app.routingMethod( 'routingPath', [array], (req, res, next) => { 

... 

})

The last of these functions in the array forward the request to the last routing handler for processing. This last routing handler also contains a next() function and processes code first before finally answering the request.

// server.js

.........

var r1 = function (req, res, next) {
  console.log('r1 ! The request will be responded by final');
  next();
}

var r2 = function (req, res, next) {
  console.log('r2 ! The request will be responded by final');
  next();
}

var r3 = function (req, res, next) {
  console.log('r3 ! The request will be responded by final');
  next();
}

app.get('/example', [r1, r2, r3], (req, res, next) => {
    console.log('log 4 ! The request will be responded by final');
    next();
  }, function (req, res) {
      console.log('final ! This is the response to the request')
})

.........

The response method res.send()

The res.send(body) method send the http response back to the client. The body parameter can be

• a string
• an object
• an array
• a buffer object.
  

res.send() with string response

When res.send() body is a string the method sets the Content-Type to “text/html”.

res.send(body); body = String.

// server.js

.........

// define the routes
app.get('/', (req, res) => {

  res.send('Hello, this is my Home Page')

})

......

Start the server with node server.js and run curl -i localhost:3000 in another terminal window and check the output.

Patricks-MBP:express-basic patrick$ curl -i localhost:3000
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 27
ETag: W/"1b-AX0brvVSaJSaZMvUvejQv+1wFQA"
Date: Sat, 21 Mar 2020 16:50:26 GMT
Connection: keep-alive

Hello, this is my Home Page

Patricks-MBP:express-basic patrick$ 

res.send() with object or array response

When the body is an Array or an Object, the Content-Type is application/json.

res.send(body); body = Object.

// server.js

.........

// define the routes
app.get('/', (req, res) => {

  res.send( { error: 'something wrong in my app' } )

})

......

Start the server with node server.js and run curl -i localhost:3000 in another terminal window and check the output.

Patricks-MBP:express-basic patrick$ curl -i localhost:3000
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 37
ETag: W/"25-8fYR5HXYWF9HTHAAgk3niZYRep4"
Date: Sat, 21 Mar 2020 13:34:03 GMT
Connection: keep-alive

{"error":"something wrong in my app"}

Patricks-MBP:express-basic patrick$ 

res.send(body); body = Array.

// server.js

.........

// define the routes
app.get('/', (req, res) => {

  var array = [1, 2, 3]
  res.send(array)
  
})

......

Start the server with node server.js and run curl -i localhost:3000 in another terminal window and check the output.

Patricks-MBP:express-basic patrick$ curl -i localhost:3000
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/json; charset=utf-8
Content-Length: 7
ETag: W/"7-nvUMyCrkdCefuOgolhQnArzLszo"
Date: Sat, 21 Mar 2020 13:35:57 GMT
Connection: keep-alive

[1,2,3]

Patricks-MBP:express-basic patrick$ 

res.send() with Buffer response

When the body is a Buffer object the Content-Type will be set to application/octet-stream.

res.send(body); body = Buffer.

// server.js

.........

// define the routes
app.get('/', (req, res) => {

 res.send(Buffer.from('<p>This is my buffer HTML</p>'))
  
})

......

Start the server with node server.js and run curl -i localhost:3000 in another terminal window and check the output.

Patricks-MBP:express-basic patrick$ curl -i localhost:3000
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: application/octet-stream
Content-Length: 29
ETag: W/"1d-TcgrXqCfH0MW//p+ASn/khisjHQ"
Date: Sat, 21 Mar 2020 13:39:23 GMT
Connection: keep-alive

<p>This is my buffer HTML</p>

Patricks-MBP:express-basic patrick$ 

Properties of the Request Object

The request object has properties that can be used on the request object. You can access various properties that have been provided with the request and process these data in your routing handler.

For example: You can access the request properties path and method with req.path and req.method. So when the client request i.e. the route „/users“ with HTTP GET then these properties will have the corresponing value.

// server.js

.........

app.get('/users', (req, res) => {

  path = req.path
  method = req.method

  console.log('Access / with GET. Path: ' +path +' and Method: ' +method);

  res.send('Hello, this is my Users Page')

})

.........

So start the server with node server.js in the server terminal and then access the route with curl -i localhost:3000/users in a different terminal. In the server terminal you see the following output.

Patricks-MBP:express-basic patrick$ node server.js
My node.js web server is alive and running at port 3000

Access /users with GET. Path: /users and Method: GET

For example: You have many users in your database and each user should have his own profile page in your web app. To provide such a profile page for each of your users you could create one route for each user. This mean you would have many routes and you must change your application by programming a new route whenever a new user join to your user population. This is not useful.

This can be avoided by using request parameters. Request parameters are defined in your route as follows.

// server.js

.........

app.get('/users/:name', (req, res) => {

  path = req.path
  method = req.method
  username = req.params.name

  console.log('Access /users/patrick with GET. Path: ' +path +' and Method: ' +method);
  
  console.log('Request Parameter name is: ' +username);

  res.send('Hello, this is my Home Page')

})

.........

Start the server with node server.js in the server terminal and then access the route with curl -i localhost:3000/users/patrick in a different terminal. In the server terminal you see the following output.

Patricks-MBP:express-basic patrick$ node server.js
My node.js web server is alive and running at port 3000

Access /users/patrick with GET. Path: /users/patrick and Method: GET

Request Parameter name is: patrick

Each of your users can access their individual profile page via a route /users/:name.

Suppose you want to show all users on the route /users in a list, you would expect that this would be possible without additional parameters when accessing the route /users. Unfortunately, this does not work in the current configuration.

Keep the server started in the server terminal and then access the route with curl -i localhost:3000/users in a different terminal. In the server terminal you see the following output.

Patricks-MBP:express-basic patrick$ curl -i localhost:3000/users
HTTP/1.1 404 Not Found
X-Powered-By: Express
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 144
Date: Sat, 21 Mar 2020 06:51:23 GMT
Connection: keep-alive

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /users</pre>
</body>
</html>
Patricks-MBP:express-basic patrick$ 

The 404 HTTP status code tell you that the route /users does not exist and nothing has been found. To solve this issue you can add a question mark to the name in your route definition. This tells Express that the request parameter is optional and that the route /users can also be requested.

// server.js

.........

app.get('/users/:name?', (req, res) => {

  path = req.path
  method = req.method
  username = req.params.name

  console.log('Access /users/patrick with GET. Path: ' +path +' and Method: ' +method);
  
  console.log('Request Parameter name is: ' +username);

  res.send('Hello, this is my Home Page')

})

.........

Start the server again in the server terminal and then access the route with curl -i localhost:3000/users in a different terminal again.

Patricks-MBP:express-basic patrick$ curl -i localhost:3000/users
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 27
ETag: W/"1b-AX0brvVSaJSaZMvUvejQv+1wFQA"
Date: Sat, 21 Mar 2020 08:23:03 GMT
Connection: keep-alive

Hello, this is my Home Page

Patricks-MBP:express-basic patrick$ 

Chapter 3: Middleware in Express

As I mentioned in the introduction, middleware simply means that code is executed between the request and the response in order to parameterize the response if necessary. Lets have a look at the following route definition from above to show how routing in express sometimes behave like a middleware.

....

app.get('/example', [r1, r2, r3], (req, res, next) => {
    console.log('log 4 ! The request will be responded by final');
    next();
  }, function (req, res) {
      console.log('final ! This is the response to the request')
      
....

After a GET request on the route /example, the routing handler functions r1, r2 and r3 are executed. Console.log commands are executed from r1 to r3 and the request is forwarded with next(). Until then, however, no response is returned. The request is forwarded by r3 to the last routing handler function, which also first executes a console.log instruction before the request is then answered by the last routing handler function. From r1 to the response, the routing handler functions behaved like middleware.

It is also possible to load modules as middleware and always execute the code of the middleware when processing a request with any route and before the response.

We assume the following directory structure.

Patricks-MBP:express-basic patrick$ ls -l
total 64
drwxr-xr-x   8 patrick  staff    256 21 Mär 16:06 modules
drwxr-xr-x  52 patrick  staff   1664 21 Mär 08:13 node_modules
-rw-r--r--   1 patrick  staff  14292 21 Mär 11:28 package-lock.json
-rw-r--r--   1 patrick  staff    162 21 Mär 10:16 package.json
-rw-r--r--   1 patrick  staff   1635 21 Mär 09:22 server.js

Patricks-MBP:express-basic patrick$ 

A very good but simple example of how middleware modules work is the use of a simple console logger. To demonstrate this, I create a logger.js file in the modules directory.

The looger.js file in the modules directory has the following code.

// modules/logger.js

var logger = function(req, res, next) {

	var method = req.method
	var path = req.path

	console.log(method +' ' +path);
	next()

}

module.exports = logger;

The logger.js file contain a simple function logger() that will be exported using module.exports. Whenever a request is made for any route, the req.method and the req.path will be logged in the terminal and then the processing will be forwarded with the next() function. So first the request is passed to the application, then the console.log instruction is executed and then the request will be forwarded to the respective routing method.

The routes are defined in our main application file server.js. The server.js file in the modules directory has the following content.

// server.js

// load node http module
const http = require('http')

// load third party Express module
const express = require('express')
const app = express()

// load own modules
const logger = require('./modules/logger')

// Integrate Middleware
app.use(logger);

// define the routes
app.get('/', (req, res) => {
  res.send('Hello, this is the Blog Home Page.')
})

app.get('/about', (req, res) => {
  res.send('Hello, this is the Blog About Page.')
})

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My node.js web server is alive and running at port 3000')

In the server.js main application file I load the module and store the logger() function into the variable logger. The middleware is integrated in the code with app.use.

After that I start the server with node server.js. In another terminal window I use curl to call routes „/“ and „/about“.

Patricks-MBP:express-blog patrick$ curl -i localhost:3000/
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 34
ETag: W/"22-jCGRti3cySGTHqyUOi9QHFbFw2U"
Date: Sat, 21 Mar 2020 12:53:32 GMT
Connection: keep-alive

Hello, this is the Blog Home Page.

Patricks-MBP:express-blog patrick$ curl -i localhost:3000/about
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 35
ETag: W/"23-hKbwboK7j6DObmy7eAWiCel3bMQ"
Date: Sat, 21 Mar 2020 12:53:35 GMT
Connection: keep-alive


Hello, this is the Blog About Page.

Patricks-MBP:express-blog patrick$

In the first terminal window in which the server is running we see that the logging has worked for both routes. The console.log command of the logger middleware module was executed for both requests.

Patricks-MBP:express-blog patrick$ node server.js
My node.js web server is alive and running at port 3000

GET /
GET /about

Chapter 4: Create a Small Blog app with express

The small Blog app should be a web app without any HTML rendering and no input validation. It should simply show how we can make use of the things we learned so far. This blog will have blogs and users stored in a data file. We will create routes end points to add-, and remove users and blogs and we will create modules which contain the program logic to manipulate data.

note: You should never ever implement an application in production without any input validation. From a security perspective, this is absolutely unacceptable. I will deal with the topic input validation with Express in an extra post here on Digitaldocblog.

You can find the code discussed here in this chapter on my GitHub page.

First we create a new application root directory express-blog. In this directory we create the main application file blog.js and a package.json file with the touch command.

Patricks-MBP:2020-03-08 patrick$ mkdir express-blog
Patricks-MBP:2020-03-08 patrick$ ls -l
total 0
drwxr-xr-x  9 patrick  staff  288 21 Mär 07:34 express-basic
drwxr-xr-x  2 patrick  staff   64 21 Mär 07:18 express-blog
drwxr-xr-x  6 patrick  staff  192 21 Mär 09:26 node-basic

Patricks-MBP:2020-03-08 patrick$ cd express-blog

Patricks-MBP:express-blog patrick$ ls -l

Patricks-MBP:express-blog patrick$ touch blog.js
Patricks-MBP:express-blog patrick$ touch package.json

Patricks-MBP:express-blog patrick$ ls -l
total 0
-rw-r--r--  1 patrick  staff  0 21 Mär 07:19 blog.js
-rw-r--r--  1 patrick  staff  0 21 Mär 07:19 package.json

Patricks-MBP:express-blog patrick$ 

We open the package.json file in our editor and edit the file as follows.

{
  "name": "simple_blog",
  "version": "0.0.1",
  "main": "blog.js",
  "author": "Patrick Rottlaender"
}

We install express with npm install express –save –save-exact to ensure that we will install express in the latest version but without updates to the version by future installs.

Patricks-MBP:express-blog patrick$ npm install express --save --save-exact
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN simple_blog@0.0.1 No description
npm WARN simple_blog@0.0.1 No repository field.
npm WARN simple_blog@0.0.1 No license field.

+ express@4.17.1
added 50 packages from 37 contributors and audited 126 packages in 2.167s
found 0 vulnerabilities

Patricks-MBP:express-blog patrick$ ls -l
total 40
-rw-r--r--   1 patrick  staff      0 21 Mär 07:19 blog.js
drwxr-xr-x  52 patrick  staff   1664 21 Mär 07:33 node_modules
-rw-r--r--   1 patrick  staff  14287 21 Mär 07:33 package-lock.json
-rw-r--r--   1 patrick  staff    155 21 Mär 07:33 package.json

Patricks-MBP:express-blog patrick$ 

Then we create a data.json file and a modules directory.

Patricks-MBP:express-blog patrick$ touch data.json

Patricks-MBP:express-blog patrick$ mkdir modules

Patricks-MBP:express-blog patrick$ ls -l
total 40
-rw-r--r--   1 patrick  staff      0 21 Mär 07:19 blog.js
-rw-r--r--   1 patrick  staff      0 21 Mär 07:44 data.json
drwxr-xr-x   2 patrick  staff     64 21 Mär 07:51 modules
drwxr-xr-x  52 patrick  staff   1664 21 Mär 07:33 node_modules
-rw-r--r--   1 patrick  staff  14287 21 Mär 07:33 package-lock.json
-rw-r--r--   1 patrick  staff    155 21 Mär 07:33 package.json

Patricks-MBP:express-blog patrick$ 

The structure of the application root directory has now been created and we can start developing the blog app.

First step: First I want to write the initial code in the blog.js main application file. The code loads the express module which we installed earlier and starts the server on port 3000 of the local host. In addition, a home route is already defined.

// blog.js

// load node http module
const http = require('http')

// load third party Express module
const express = require('express')
const app = express()

// define the routes
app.get('/', (req, res) => {
  res.send('Hello, this is the Blog Home Page.')
})

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My Blog server is running at port 3000')

To create users and blog data I should have 2 different modules. One that will create the users and the other one will create the blogs. Therefore I create 2 files in the modules directory.

Patricks-MBP:express-blog patrick$ touch modules/createUsers.js
Patricks-MBP:express-blog patrick$ touch modules/createBlogs.js

Patricks-MBP:express-blog patrick$ ls -l modules
total 0
-rw-r--r--  1 patrick  staff  0 21 Mär 08:02 createBlogs.js
-rw-r--r--  1 patrick  staff  0 21 Mär 08:02 createUsers.js

Patricks-MBP:express-blog patrick$ 

Second step: In the data.json file I will store the user and blog data. The data file is currently empty but will have the following structure.

{
	"users":[{name:..., lastname:..., email:..., id:... },{...},...],
	"blogs":[{title:..., author:..., date:..., id:... },{...},...]

}

There is a json data object with the keys users and blogs. These keys have an array value with objects as array elements.

i,j = 0, 1, 2 ...

data.users[i]
data.blogs[j]

I will now program the 2 modules usersAdd and blogsAdd in the createUsers.js and createBlogs.js files. These files contain functions that initially create user or blog records or add corresponding user or blog records.

The createUsers.js file contains the following code.

// modules/createUsers.js

// load node fs module
const fs = require('fs')

const usersAdd = function(req, res) {

    var dataExp = fs.readFileSync('./data.json', 'utf8')

    if(!dataExp) {
      console.log('no data available');
      var data = {}
      data.users = []

      var newUserName = req.body.name
      var newUserLastname = req.body.lastname
      var newUserEmail = req.body.email
      var newUserId = 100

      var newUser = {
        name: newUserName,
        lastname: newUserLastname,
        email: newUserEmail,
        id: newUserId
      }

    data.users.push(newUser)

    dataToFile = JSON.stringify(data)

      fs.writeFile('./data.json', dataToFile, function(err) {
        if (err) throw err;
        console.log('Intial User created');
        });

    } else {
      var data = JSON.parse(dataExp)

      if(!data.users) {
        console.log('no users are available')
        data.users = []

        var newUserName = req.body.name
        var newUserLastname = req.body.lastname
        var newUserEmail = req.body.email
        var newUserId = 100

        var newUser = {
          name: newUserName,
          lastname: newUserLastname,
          email: newUserEmail,
          id: newUserId
        }

      data.users.push(newUser)

      dataToFile = JSON.stringify(data)

        fs.writeFile('./data.json', dataToFile, function(err) {
          if (err) throw err;
          console.log('Intial User created');
          });

      } else {
        console.log('users are available')
        var newID
        var minID = data.users[0].id

        for (i = 0; i < data.users.length; i++) {
          if (data.users[i].id >= minID) {
              newID = data.users[i].id + 100
          }
        }

        var newUserName = req.body.name
        var newUserLastname = req.body.lastname
        var newUserEmail = req.body.email
        var newUserId = newID

        var newUser = {
          name: newUserName,
          lastname: newUserLastname,
          email: newUserEmail,
          id: newUserId
        }

      data.users.push(newUser)

      dataToFile = JSON.stringify(data)

      fs.writeFile('./data.json', dataToFile, function(err) {
        if (err) throw err;
        console.log('User added');
        });
      }
  }
}

module.exports = usersAdd;

The fs module is loaded first. This is a node internal module which is used to read files and also to write data into a file. Then we create the usersAdd function that will be exported with module.exports.

The usersAdd function read the data.json file with fs.readFileSync and store the received data into the dataExp variable. If dataExp is empty we have an empty data file and must first create the empty data Object and store this into the variable data. Then we create the users key on the data object with data.users and assign an empty array.

From the request body we receive the name, lastname and email and store these data into new user variables. For the initial user entry we define the id 100.

note: Later in this tutorial we will see the POST route definition /createusers in the application main file blog.js. This POST route will call usersAdd with the parameters req and res so that we have access to the req.body object in createUsers.js. We will see this later.

We create a newUser object with the keys like name, lastname, email, id and assign the new user variables we received from the request body to them as values. Then we push the newUser object into the array of the users key (created with data.users = [ ] at the beginning) which is linked to the data object that we created at the beginning with data = .

data.users.push(newUser)

After this we have the following JavaScript data object.

{

	"users": { [ {name:..., lastname:..., email:..., id:... } ] }

}

With JSON.stringify() we take the JavaScript data object, create a JSON string out of it and store this string into the dataToFile variable. With fs.writeFile we write the string into the so far empty data.json file.

note: If you use JavaScript to develop applications, JavaScript objects have to be converted into strings if the data is to be stored in a database or a data file. The same applies if you want to send data to an API or to a webserver. The JSON.stringify() function does this for us.

So far we know what the code does in case the data.json file is completely empty and the first if condition is true (if(!dataExp)). And now lets see what happens what the code does in case the first if condition is false which means that the data.json file already has data because that is the case if the variable dataExp is not empty.

As we have seen we store a JSON string in the data file. So when we read the file and store the data into the variable dataExp we have JSON string data in that variable. To process these data in our JavaScript code we must parse this JSON so that a JavaScript object is created from the JSON string. This is what we do with the JSON.parse() function.

Here we store the JavaScript object into the variable data.

var data = JSON.parse(dataExp)

Since our code is designed to store only one data object in the data.json file, the variable data is our data object. So when we have the data object we can ask if users are already existing.

Therefore we use the if condition to check if the key data.users exist.

In case the condition if(!data.users) is true, the key data.users is not existing which means that no users are available so far. At his point we create a data.users key and assign an empty array. In the following, a newUser object is created in the same way as described above and inserted into the array using the data.users.push() method. Now the data object has an additional key data.users and this key has a newUser Object as value. Now we can convert this entire data object into a string again and completely overwrite the data.json file.

In case the condition if(!data.users) is false, the key data.users is existing and users are available. If users exist the individual user records naturally have existing ids. So we first have to create an algorithm to get a new user ID for the new user.

We first create a variable newID and a variable minID. minID corresponds to the userID which has to be assigned at least since it corresponds to the userID of our first user data object. minID must be the id value of the first user data object record. Since the user objects are stored in an array, this must correspond to the user object at position zero.

var minID = data.users[0].id

newID should be plus 100 larger than the id of the last found user object.

That is why we iterate through all user objects, adding the value 100 to the id value of the last user object found. Finally, we take the id value of the last user object and add 100 and assign this value to newID.

In the following again a newUser object is created, the entire data object is converted into a string and stored into the data.json file.

The code for the blogsAdd function is implemented exactly according to the same scheme. The only difference is that the blog id only grows by 1.

The createBlogs.js file contains the following code.

// modules/createBlogs.js

// load node fs module
const fs = require('fs')

const blogsAdd = function(req, res) {

    var dataExp = fs.readFileSync('./data.json', 'utf8')

    if(!dataExp) {
      console.log('no data available')
      data = {}
      data.blogs = []

      var newBlogTitle = req.body.title
      var newBlogAuthor = req.body.author
      var newBlogDate = req.body.date
      var newBlogId = 1

      var newBlog = {
        title: newBlogTitle,
        author: newBlogAuthor,
        date: newBlogDate,
        id: newBlogId
      }

    data.blogs.push(newBlog)

    dataToFile = JSON.stringify(data)

      fs.writeFile('./data.json', dataToFile, function(err) {
        if (err) throw err;
        console.log('Intial Blog created');
        });

    } else {
      var data = JSON.parse(dataExp)

      if (!data.blogs) {
        console.log('no blog are available')
        data.blogs = []

        var newBlogTitle = req.body.title
        var newBlogAuthor = req.body.author
        var newBlogDate = req.body.date
        var newBlogId = 1

        var newBlog = {
          title: newBlogTitle,
          author: newBlogAuthor,
          date: newBlogDate,
          id: newBlogId
        }

      data.blogs.push(newBlog)

      dataToFile = JSON.stringify(data)

        fs.writeFile('./data.json', dataToFile, function(err) {
          if (err) throw err;
          console.log('Intial blog created');
          });

      } else {
        console.log('blogs are available');
        var newID
        var minID = data.blogs[0].id

        for (i = 0; i < data.blogs.length; i++) {
          if (data.blogs[i].id >= minID) {
              newID = data.blogs[i].id + 1
          }
        }

        var newBlogTitle = req.body.title
        var newBlogAuthor = req.body.author
        var newBlogDate = req.body.date
        var newBlogId = newID

        var newBlog = {
          title: newBlogTitle,
          author: newBlogAuthor,
          date: newBlogDate,
          id: newBlogId
        }

      data.blogs.push(newBlog)

      dataToFile = JSON.stringify(data)

      fs.writeFile('./data.json', dataToFile, function(err) {
        if (err) throw err;
        console.log('blog added');
        });
      }
  }
}

module.exports = blogsAdd;

Third Step: The blog.js file needs to be adjusted. First we integrate middleware that allows us to parse incoming http requests with urlencoded payloads.

app.use(express.urlencoded({ extended: false }))

Then the two modules must be loaded into the blog.js file with the require() function. Therefore we require the files createUsers.js and createBlogs.js.

const createUsers = require('./modules/createUsers')
const createBlogs = require('./modules/createBlogs')

note: I would like to explain the topic require() of modules again at this point. The files createBlogs.js and createUsers.js contain the usersAdd and blogsAdd functions which are each exported with module.exports. The module.exports instruction means that these functions can be called in other files of the application if the createBlogs.js and createUsers.js files are loaded with require(). usersAdd and blogsAdd have no return value and therefore the functions themselves are saved in the variables createBlogs and createUsers in blog.js. If e.g. createUsers(req, res) is called in a POST route, the usersAdd function is called and req an res are passed as parameters.

As we can see in the code of the two modules above, the values for the initial creation or the addition of user- or blog objects are passed to the functions usersAdd or blogsAdd via the request body. For this, POST routes /createusers and /createblogs must be created in the blog.js file. These POST routes call the functions usersAdd or blogsAdd via createUsers() or createBlogs().

The blog.js file contains the following code.

// blog.js

// load node http module
const http = require('http')

// load third party Express module
const express = require('express')
const app = express()

// parse application/x-www-form-urlencoded
app.use(express.urlencoded({ extended: false }))

// load own modules
const createUsers = require('./modules/createUsers')
const createBlogs = require('./modules/createBlogs')

// define the routes
app.get('/', (req, res) => {
  res.send('Hello, this is the Blog Home Page.')
})

app.post('/createusers', (req, res) => {
  createUsers(req, res)
  res.send('User has been added successfully.')
})

app.post('/createblogs', (req, res) => {
  createBlogs(req, res)
  res.send('Blog has been added successfully.')
})

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My node.js web server is alive and running at port 3000')

Now we are able to add blogs and users.

note: Here in this application we have no User interface to enter the user variables like name, lastname, email or the blog variables like title, author and date. With this application you cannot enter the data in a form and then transfer it to the application routes /createusers or /createblogs using HTTP POST. This is where Postman comes in. With Postman you can simulate a form by manually entering this data in the HTTP body of the HTTP POST request. Postman check whether the POST request was successful or not. I will not go into Postman at this point. So pls. check the documentation on the Postman website.

But there is still no way to remove data from the data.json. We now want to show this using the example of how to remove users.

The module which should take over exactly this functionality is the function usersRemove() which I will write in the file modules/usersRemove.js. To do this, we first create an empty file usersRemove.js in the modules directory.

Patricks-MBP:express-blog patrick$ pwd
/Users/patrick/Software/dev/node/myArticles/2020-03-08/express-blog
Patricks-MBP:express-blog patrick$ ls -l modules
total 24
-rw-r--r--  1 patrick  staff  2285 21 Mär 17:18 createBlogs.js
-rw-r--r--  1 patrick  staff  2319 21 Mär 17:17 createUsers.js
-rw-r--r--  1 patrick  staff   652 21 Mär 06:50 removeUsers.js

Patricks-MBP:express-blog patrick$ 

The removeUsers.js file contains the following code.

// modules/removeUsers.js

// load node fs module
const fs = require('fs')

const usersRemove = function(req, res) {
  var dataExp = fs.readFileSync('./data.json', 'utf8')
  var data = JSON.parse(dataExp)

  var delUserID = req.body.id - 0
  
  var elementFound = function(user) {
    return user.id === delUserID
  }

  var index = data.users.findIndex(elementFound)

  data.users.splice(index, 1)

  dataToFile = JSON.stringify(data)
 
  fs.writeFile('./data.json', dataToFile, function(err) {
    if (err) throw err;
    console.log('userID removed');
    });
 
}

module.exports = usersRemove;

We first load the node fs module again and then define the function usersRemove() which is then exported with module.exports at the end of the code. The function usersRemove() receive req and res as parameters, read the data.json file and parse the read data into a JavaScript object represented by the variable data.

To show you the data object I just run the script as follows.

// modules/removeUsers.js

// load node fs module
const fs = require('fs')

const usersRemove = function(req, res) {
  var dataExp = fs.readFileSync('./data.json', 'utf8')
  var data = JSON.parse(dataExp)
  
  console.log(data)

}

module.exports = usersRemove;

When you check your console you can see that we saved all the json-data from the file in the variable data. We see a JavaScript object that can be used in our code.

{
  blogs: [
    {
      title: 'First Title',
      author: 'First test author',
      date: '2020-03-28',
      id: 1
    },
    {
      title: 'Second Title',
      author: 'Second test author',
      date: '2020-03-28',
      id: 2
    }
  ],
  users: [
    {
      name: 'Oskar',
      lastname: 'Rottländer',
      email: 'o.rottlaender@test.com',
      id: 100
    },
    {
      name: 'Patrick',
      lastname: 'Rottländer',
      email: 'p.rottlaender@test.com',
      id: 200
    },
    {
      name: 'Carolin',
      lastname: 'Rottländer',
      email: 'c.rottlaender@test.com',
      id: 300
    }
  ]
}

In the code we take the userID from the req.body and store it into the variable delUserID. The value from the req.body is a string. As we operate with userIDs as numbers in our json data file we must convert the string into a number. This is done with „- 0“. Here we subtract the number value zero and tell JavaScript that the delUserID is a number instead of a string.

Then we must find the user object that need to be deleted. user objects are elements in the data.users array. Consequently we must find the user object with a userID from the req.body in the data.users array because this userID identify the user object that should then be deleted and therefore removed from the data.users array.

The callback function elementFound return true when the userID from the rep.body is equal to a userID of a user object in the data.users array.

The method data.users.findIndex() return the index of the first user object found in the data.users array where the userID from the rep.body is equal to the userID of the user object.

The data.users.splice(index, 1) remove one user object at the position of index. As the findIndex() method before returned the exact position of the user object that we want to delete the splice() method here ultimately removes the user object.

The complete data object consisting of blog objects and user objects is now reduced by one user object. the reduced data object is converted as a whole into a string and written again into the file data.json using the fs.writeFile() method.

The blogsRemove() function works identically to the usersRemove() function explained above. The file removeBlogs.js in the module directory therefore looks like this.

// modules/removeBlogs.js

// load node fs module
const fs = require('fs')

const blogsRemove = function(req, res) {
  var dataExp = fs.readFileSync('./data.json', 'utf8')
  var data = JSON.parse(dataExp)

  var delBlogID = req.body.id - 0

  var elementFound = function(blog) {
    return blog.id === delBlogID
  }

  var index = data.blogs.findIndex(elementFound)

  data.blogs.splice(index, 1)

  dataToFile = JSON.stringify(data)

  fs.writeFile('./data.json', dataToFile, function(err) {
    if (err) throw err;
    console.log('blogID removed');
    });
}

module.exports = blogsRemove;

In the code of the two removeUsers.js and removeBlogs.js modules above, the values for the to be removed user- or blog objects are passed to the functions usersRemove() or blogsRemove() via the request body. For this, POST routes /removeusers and /removeblogs must be created in the blog.js file. These POST routes call these functions usersRemove() or blogsRemove() which are defined in the modules removeUsers.js and removeBlogs.js. These modules must also be loaded into the blog.js file with the require() function.

The blog.js file now contains the following code.

// blog.js

// load node http module
const http = require('http')

// load third party Express module
const express = require('express')
const app = express()

// parse application/x-www-form-urlencoded
app.use(express.urlencoded({ extended: false }))

// load own modules
const createUsers = require('./modules/createUsers')
const removeUsers = require('./modules/removeUsers')
const createBlogs = require('./modules/createBlogs')
const removeBlogs = require('./modules/removeBlogs')

// define the routes
app.get('/', (req, res) => {
  res.send('Hello, this is the Blog Home Page.')
})

app.post('/createusers', (req, res) => {
  createUsers(req, res)
  res.send('User has been added successfully.')
})

app.post('/removeusers', (req, res) => {
  removeUsers(req, res)
  res.send('User has been successfully removed.')
})


app.post('/createblogs', (req, res) => {
  createBlogs(req, res)
  res.send('Blog has been added successfully.')
})

app.post('/removeblogs', (req, res) => {
  removeBlogs(req, res)
  res.send('Blog has been successfully removed.')
})

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My node.js web server is alive and running at port 3000')

Fourth step: Our application can currently create users and create blogs as well as delete blogs and users. Thats fine so far. Now another functionality should be added, namely the display of blogs. This is to be implemented in such a way that the GET request for one route sends different responses back to the server depending on the request parameters provided with the request.

What does this mean ?

For example: We have various blogs in our data file, all of them have been created for specific dates. We want to see all blogs, but also blogs from a certain year or month or blogs that were created on a specific date.

You want to access all blogs using this route.

app.get('/blog', (req, res) => {
  	....
  
  })

And you want to access all blogs in 2020 using this route.

app.get('/blog/2020', (req, res) => {
  	....
  
  })

And so on.

We can do this much more elegantly using request parameter. We create the following GET route in our blog.js file.

// blog.js

.....

app.get('/blog/:year?/:month?/:day?', (req, res) => {
  	....
  
  })
  
.....

The question mark indicates that the parameters year, month and day are optional and can also be omitted when requesting.

We have defined a route that should give different responds depending on the request parameters. That is why we first have to evaluate the request parameters in the route definition and then define conditions as to how the different inquiries should be answered.

// blog.js

.....

app.get('/blog/:year?/:month?/:day?', (req, res) => {

  url = req.url
  year = req.params.year
  month = req.params.month
  day = req.params.day

    if (url == '/blog') {
      	...    
    }

    else if (url == '/blog/'+year) {
 		...
    }

    else if (url == '/blog/'+year+'/'+month) {
    	 ...
    }

    else if (url == '/blog/'+year+'/'+month+'/'+day) {
     	...
    }

    else {
      res.status(404).send('Not Found')
    }

})

.....

I will come back to the main application file blog.js later.

Our goal is to display blog posts. Depending on the route, we want to show (1) all blogs or (2) all blogs of a year or (3) all blogs of a year and a month or (4) all blog posts from a specific day.

To collect all these data from our data.json file we basically need 4 new functions to perform those queries. Therefore I create a new queryModule in the modules directory. This queryModule contain all these 4 query functions and all of them will be made available within our application. The notation of the code in the queryModule file is slightly different than in the previous modules.

The reason is that we previously had one file as a module in the modules directory and only one function in that file. The export value was the function. These modules looked like this.

// modules/previousModule.js

var myFunction = function() {
	....
}
module.exports = myFunction

In our main application file we load the previousModule with the require function and store a function into the variable previousModule. We can call the function in the main application file directly with previousModule().

// mainApplicationFile.js

.....

const previousModule = require('./modules/previousModule')

.....

previousModule()

....

Here in the queryModule (which is our queryBlogs.js file in modules directory) we collect 4 functions but we do not export them. Instead we use module.exports to export an object with 4 keys. Each of these keys has exactly one value and that is the respective function.

// modules/queryModule.js

module.exports = {

	function1: function() {
				....	
			}
	
	function2: function() {
				....	
			}
			
	function3: function() {
				....	
			}
			
	function4: function() {
				....	
			}

}

In our main application file we load the queryModule with the require function and store an object into the variable queryModule. We can call the functions in the main application file via the object.key notation.

// mainApplicationFile.js

.....

const queryModule = require('./modules/queryModule')

.....

queryModule.function1()

queryModule.function2()

queryModule.function3()

queryModule.function4()

....

When we take a look at our new queryModule which is the queryBlogs.js file in the modules directory you see that we have an object with 4 different keys and those keys have a function() as values.

• queryAllBlogs
• queryBlogsYear
• queryBlogsYearMonth
• queryBlogsYearMonthDay
  

There is another important difference to the previous modules. Those functions in queryBlogs.js return each a value and this value can be used later in our main application file blog.js. This value is an array of blog posts that should be displayed when a certain route has been requested.

So the queryBlogs.js file looks as follows. I will explain the queryAllBlogs and the queryBlogsYear in detail. The other 2 functions queryBlogsYearMonth and queryBlogsYearMonthDay are schematically the same.

// modules/queryBlogs.js

// load node fs module
const fs = require('fs')

module.exports = {

  queryAllBlogs: function() {

    var allBlogs = []

    var dataExp = fs.readFileSync('./data.json', 'utf8')
    var data = JSON.parse(dataExp)

    for(i = 0; i < data.blogs.length; i++) {
      var dataSet = {
        title: data.blogs[i].title,
        author: data.blogs[i].author,
        date: data.blogs[i].date
      }
      allBlogs.push(dataSet)
    }
      return allBlogs
  },

  queryBlogsYear: function(allBlogs) {
    var yearBlogs = []
    for (i = 0; i < allBlogs.length; i++) {

      blogDate = allBlogs[i].date
      parsedBlogDate = new Date (Date.parse(blogDate))
      parsedBlogDateYear = parsedBlogDate.getFullYear()

      if (year == parsedBlogDateYear) {
        var dataset = {
          title: allBlogs[i].title,
          author: allBlogs[i].author,
          date: blogDate
        }
        yearBlogs.push(dataset)
      }
  }
  return yearBlogs
  },

  queryBlogsYearMonth: function(allBlogs) {
    var yearMonthBlogs = []
    for (i = 0; i < allBlogs.length; i++) {

      blogDate = allBlogs[i].date
      parsedBlogDate = new Date (Date.parse(blogDate))
      parsedBlogDateMonth = (parsedBlogDate.getMonth() + 1)
      parsedBlogDateYear = parsedBlogDate.getFullYear()

      if (year == parsedBlogDateYear && month == parsedBlogDateMonth) {
        var dataset = {
          title: allBlogs[i].title,
          author: allBlogs[i].author,
          date: blogDate
        }
        yearMonthBlogs.push(dataset)
      }
  }
  return yearMonthBlogs
  },

  queryBlogsYearMonthDay: function(allBlogs) {
    var yearMonthDayBlogs = []
    for (i = 0; i < allBlogs.length; i++) {

      blogDate = allBlogs[i].date
      parsedBlogDate = new Date (Date.parse(blogDate))
      parsedBlogDateDay = parsedBlogDate.getDate()
      parsedBlogDateMonth = (parsedBlogDate.getMonth() + 1)
      parsedBlogDateYear = parsedBlogDate.getFullYear()

      if (year == parsedBlogDateYear && month == parsedBlogDateMonth && day == parsedBlogDateDay) {
        var dataset = {
          title: allBlogs[i].title,
          author: allBlogs[i].author,
          date: blogDate
        }
        yearMonthDayBlogs.push(dataset)
      }
    }
    return yearMonthDayBlogs
  }

}

Our data is stored as a string in a data.json file. Every operation of our application in relation to the data requires that we have access to a JavaScript data object. So if we want to search or select data to display only a part of it in the application, e.g. only all blog posts from a certain year, it is still necessary to have read the entire data from the data file in beforehand. In a larger project with a larger amount of data, this approach is definitely not the right one.

note: With larger amounts of data, it makes sense to use a database such as MongoDB because we can then search for specific values directly in the database and it is not necessary to retrieve all data from the database. I will show you the use of a MongoDB in a node.js project in another article at a later time. For simplicity’s sake, let’s stick here in this article to a file-based approach.

Therefore the queryAllBlogs function serves as the basis for our other queries for displaying blogs. This function first defines an empty array allBlogs. Then, as already described above, the data is read completely from the data file with fs.readFileSync and converted into a JavaScript object with JSON.parse. Now we have access to blog posts with data.blogs. However, this is not entirely true. As we have already seen above, data.blogs is a key in the data object and this key has an array as value. Strictly speaking, this array contains elements that are the blog posts.

In any case, with the for loop we iterate through all elements of the array using data.blogs[i] so that each individual blog data record can be read from the data file and attached with the allBlogs.push() method as an dataSet object to the allBlogs array that was defined first. The function queryAllBlogs() then return allBlogs back to the caller. allBlogs contain all the dataSet objects. The allBlogs array looks like the following.

[
	{
	
	title: valueTitle,
	author: valueAuthor,
	data: valueDate
	
	},
	
	{
	
	...
	
	},
	
	...

]

The function queryBlogsYear take the allBlogs array as parameter (like any other function queryBlogs* defined in the module). This is because we want to select data from it in the queryBlogsYear function and therefore all blog posts must be available in the function scope.

First we create the empty array yearBlogs. This array will contain at the end of the function code all the data that should be returned later by the queryBlogsYear function.

Then we iterate through the allBlogs array using the for loop.

In the for loop, we first read at the value of the blog date with allblogs[i].date and store the value in the variable blogDate. In the data file the date is saved with ISO 8601 date format (YYYY-mm-dd). It is highly recommended to always save the date in this format since ISO 8601 is the date format preferred by JavaScript and so all other operations with the date are then made easier. In order to work well with the date string yyyy-mm-dd in JavaScript, this string date must first be converted into a number using. Then the number will be converted into a date object using the Date.parse() method.

note: Date.parse() receive the string date in ISO 8601 format yyyy-mm-dd and returns a number that represents the milliseconds between January 1st, 1970 00:00:00 UTC. This number is unique and is required to create the date object using the new Date() method. With this date object we have with JavaScript access to the year, month or day of the given date object.

Therefore, in our code Date.parse() is passed directly as parameter into the new Date() method so that the required date object can be generated directly from the returned number. The date object is stored in the variable parsedBlogDate.

Because parsedBlogDate represents an object, the parsedBlogDate.getFullYear() method gives us access to the year passed with the date. The year is stored in the parsedBlogDateYear variable.

note: Later we will use parsedBlogDate.getMonth() to retrieve the month from the date object. This method returns 0 for January and 11 for December. It is therefore necessary to add „+1“ to get the usual values 1 for January and 12 for December. And we will use parsedBlogDate.getDate() to retrieve the day of the month as 1 for the first day and max 31 for the last day of the respective month.

Then we compare in the if loop the year passed in the request (year = req.params.year, variable year is defined in blog.js) with the years of all blog posts. We only generate a dataset where the year from the request matches the year of the blog post. The array yearBlogs created in this way is then returned to the caller as a return value.

I will now come back to main application file blog.js and the specific route definition where we evaluate the request parameters and provide different responses. With this route we send, depending on the request parameters, the following responses.

• all blog posts – res.send(allBlogs)
• blog posts of given year – res.send(blogsYear)
• blog posts of a give year and month – res.send(blogsYearMonth)
• blog posts of a given date – res.send(blogsYearMonthDay)
  

// blog.js

....

const queryBlogs = require('./modules/queryBlogs')

....

app.get('/blog/:year?/:month?/:day?', (req, res) => {

  url = req.url
  year = req.params.year
  month = req.params.month
  day = req.params.day

  var allBlogs = queryBlogs.queryAllBlogs()

    if (url == '/blog') {
      if (allBlogs.length == 0) {
        res.send('no post found')
      } else {
        res.send(allBlogs)
      }
    }

    else if (url == '/blog/'+year) {

      const blogsYear = queryBlogs.queryBlogsYear(allBlogs)

      if(blogsYear.length == 0) {
        res.send('no post found for this year')
      } else {
        res.send(blogsYear)
      }

    }

    else if (url == '/blog/'+year+'/'+month) {

      const blogsYearMonth = queryBlogs.queryBlogsYearMonth(allBlogs)

      if(blogsYearMonth.length == 0) {
        res.send('no post found for this year and month')
      } else {
        res.send(blogsYearMonth)
      }

    }

    else if (url == '/blog/'+year+'/'+month+'/'+day) {

      const blogsYearMonthDay = queryBlogs.queryBlogsYearMonthDay(allBlogs)

      if(blogsYearMonthDay.length == 0) {
        res.send('no post found for this year and month and day')
      } else {
        res.send(blogsYearMonthDay)
      }

    }

    else {
      res.status(404).send('Not Found')
    }

})

....

We have defined a route that give different responds depending on the request parameters. We want to see all blogs when the request url == /blog, we see all blogs i.e. of a given year 2019 when the request url == /blog/2019 and so an.

First we load the queryBlogs module with the require() function and save the object in the variable queryBlogs (remember ? queryBlogs return an object of functions).

In the route definition we take the request parameters passed with the request and save them in the variables url, year, month and day. Then we access the queryAllBlogs() function defined in the queryBlogs module with queryBlogs.queryAllBlogs(). This function return all blog posts stored in an array. This array returned by the function will be stored in the variable allBlogs.

Then we create if and else if conditions to define the response behavior of our route definition in our application.

The first if condition is true in case the requested url is equal to /blog. The following if condition is true when the allBlogs array is empty. Then the response is no posts found res.send(’no post found‘). When this if condition is true blog posts have been found in the allBlogs array and this array is not empty. In this case all the data from the allBlogs array will be send back to the caller with res.send(allBlogs).

The first else if condition is true in case the requested url is equal to /blog/year (i.e. /blog/2019). When this else if condition is true we call the queryBlogsYear() function in our queryBlogs module and pass the allBlogs array as parameter. We call the function using queryBlogs.queryBlogsYear(allBlogs) and store the returned value in the array variable blogsYear.

If the blogsYear array is empty the following if condition is true. This mean no blog posts have been found for the requested year in allBlogs. Then the response is no posts found for this year res.send(’no post found for this year‘). When this if condition is false blog posts have been found for the requested year and the blogsYear array is not empty. Then all these data from the blogsYear array will be send back to the caller with res.send(blogsYear).

The second else if condition is true in case the requested url is equal to /blog/year/month (i.e. /blog/2019/04). When this else if condition is true we call the queryBlogsYearMonth() function in our queryBlogs module and pass the allBlogs array as parameter. We call the function using queryBlogs.queryBlogsYearMonth(allBlogs) and store the returned value in the array variable blogsYearMonth.

If the blogsYearMonth array is empty the following if condition is true. This mean no blog posts have been found for the requested year and month in allBlogs. Then the response is no posts found for this year and month res.send(’no post found for this year and month‘). When this if condition is false blog posts have been found for the requested year and month and the blogsYearMonth array is not empty. Then all these data from the blogsYearMonth array will be send back to the caller with res.send(blogsYearMonth).

The third else if condition is true in case the requested url is equal to /blog/year/month/day (i.e. /blog/2019/04/10). When this else if condition is true we call the queryBlogsYearMonthDay() function in our queryBlogs module and pass the allBlogs array as parameter. We call the function using queryBlogs.queryBlogsYearMonthDay(allBlogs)and store the returned value in the array variable blogsYearMonthDay.

If the blogsYearMonthDay array is empty the following if condition is true. This mean no blog posts have been found for the requested date in allBlogs. Then the response is no posts found for this year and month and day res.send(’no post found for this year and month and day‘). When this if condition is false blog posts have been found for the requested date and the blogsYearMonth array is not empty. Then all these data from the blogsYearMonthDay array will be send back to the caller with res.send(blogsYearMonthDay).

The first if condition is false in case the requested url does not match any of the defined request parameters (/blog/:year?/:month?/:day?). Then the else response is res.status(404).send(‚Not Found‘).

Summary and Outlook

In this Part 2 of my node.js series I explained some very important details of Express and how express can be used in web application development. Based on what we already learned about Express in Part 1, we looked at Express Routing in detail and how you can use middleware in Express relatively easily. With the knowledge we learned, we then built a simple blog web application and thus applied the knowledge.

This blog app has all the data in a file and no HTML frontend or no HTML template rendering is used. Express is ideally suited for the use of a database like MongoDB and also for the use of HTML templates to present the content. In the next Part 3 of my node.js series I will show you how our blog app uses a MongoDB database instead of the data file. Then we will continue to develop the blog app and implement HTML rendering.

Chapter 1: Under the hood of the simple web-server

Node.js is based on JavaScript. JavaScript has always been known as a front-end programming language that is executed in the browser. The code runs in the browser. This has changed since node.js because with node.js JavaScript code is executed on the server side. The code no longer runs in the browser but on the server. Node.js provides the build in http module which make it easy to implement a web-server with basic functionalities. Node.js is a runtime environment in which server-side code can be executed. In this respect, node.js is the basic technology for frameworks like Express.js.

Express.js is the most widespread node.js framework. In contrast to the pure node.js http module, Express.js offers extensive options for processing client http requests. Express.js also offers middleware functionalities that can be built in between the request and the response to execute the code build in routes depending on the middleware function. This includes authentication and authorization, logging and much more.

Chapter 2: Install node.js and npm

First of all, it must be ensured that node.js and also npm.js are installed on your system. npm is the node package manager and is absolutely necessary if we develop programs with node.

We should definitely check whether node and npm are already installed on the system. To do this, please enter the following commands on the command line.

Patricks-MBP:node patrick$ node --version
v13.8.0

Patricks-MBP:node patrick$ npm --version
6.13.7

Patricks-MBP:node patrick$ 

In this case, node is already installed in version 13.8.0 and npm in version 6.13.7 and nothing further needs to be done.

I work with Mac OS and there is the standard for installing command line tools homebrew. I wrote the article node.js and npm on Mac OS here on Digitaldocblog that shows step by step the installation of node and npm on a Mac OS. Other ways to install node.js can be found on [nodejs.org] (https://nodejs.org/en/download/).

Especially if we work with Ubuntu linux, node and npm can be installed using the Advanced Packaging Tool (APT). While on Mac OS the installation of node using homebrew includes npm, node and npm are installed separately on Ubuntu. Simply enter the following commands on the command line on your Ubuntu system (but check before if it is already installed).

#: sudo apt install nodejs

#: sudo apt install npm

Of course, it is also a basic requirement to install a suitable editor. You can of course also work with text editors like nano or vi, but that is certainly very cumbersome. From my point of view, the Atom editor is very suitable for node programming and my first choice.

After that, the following requirements should be met:

• node and npm installed
• text editor installed (i.e. Atom)
  

Chapter 3: The basics of node.js

I will briefly go over here to explain how node.js works. For this I will program a simple webserver with plain node.js and you can see what node already offers for creating a web app.

For this purpose, a separate directory for our first node web-app is created on the development system. This is the place we will store all files and the entire program code and called the application root directory. In my case this is the application root directory node-basic.

Patricks-MBP:2020-03-08 patrick$ mkdir node-basic

Patricks-MBP:2020-03-08 patrick$ cd node-basic

Patricks-MBP:node-basic patrick$ ls -al
total 0
drwxr-xr-x  2 patrick  staff  64 08 Mär 06:05 .
drwxr-xr-x  3 patrick  staff  96 08 Mär 06:05 ..

Patricks-MBP:node-basic patrick$ pwd
/Users/patrick/Software/dev/node/myArticles/2020-03-08/node-basic

Patricks-MBP:node-basic patrick$  

Change to the application root directory you just created, create a file server.js and open this file using your favorite editor. The file server.js is called the application main file and is the entry point of our node web-application.

Patricks-MBP:node-basic patrick$ touch server.js

Patricks-MBP:node-basic patrick$ ls -l
total 0

-rw-r--r--  1 patrick  staff  0 08 Mär 18:22 server.js

Patricks-MBP:node-basic patrick$ 

3.1 Simple node web-server using the integrated ‚http‘ module

Enter the following code in server.js. You can find the code on my GitHub Page.

// server.js

// 1. load http module
const http = require('http')

// 2. create the server
const server = http.createServer(function (req, res) {

    // 3. check conditions for requests
    if (req.url == '/') {

        // 4.1 set response header
        res.writeHead(200, { 'Content-Type': 'text/html' });

        // 4.2 set response content
        res.write('<html><body><p>This is my home Page.</p></body></html>');
        
        // 4.3 End response
        res.end();

    }
    else if (req.url == "/about") {

        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.write('<html><body><p>This is my about Page.</p></body></html>');
        res.end();

    }
    else if (req.url == "/impressum") {

        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.write('<html><body><p>This is my Impressum Page.</p></body></html>');
        res.end();

    }
   else {
      res.writeHead(404, { 'Content-Type': 'text/html' })
      res.write('<html><body><p>This path is not available. Invalid request</p></body></html>')
      res.end()
    }
});

// 5. server listen for any incoming requests
server.listen(3000);

console.log('My node.js web server is alive and running at port 3000')

A webserver is a relatively simple system that processes HTTP requests from a client. node.js contains the http module with which the developer can program http server but also http clients.

So lets see what happens in the code above.

1. The http module is loaded with the require() function and a reference is saved in the variable http. Various functions can be used on the http reference that are provided by the http module.
2. On the http reference a server is created with the http.createServer() function. http.createServer() creates a server object and the reference is saved in the variable server. The server object can listen to ports and receives a requestListener() function as parameter. This function is always executed when the server receives a request from the client and expects 2 parameters such as request and response. These parameters are usually noted with req and res.
3. req is always the first parameter and represent the request object. The request object refer to the IncomingMessage object. This object has properies and methods like the property url that can be used with req.url.
4. res is always the second parameter and represent the response object. The response object refer to the ServerResponse object. This object has properties and methods like the method writeHead() that can be used with res.writeHead().
5. In the if and else if block we compare the requested url req.url with the url we difined in our route (i.e. „/“ or „/about“). For example, if the URL corresponds to „/about“, the server should return an about page to the client. If the requested url req.url does not match any of the conditions specified with if and else if, an error message is sent back to the client.
6. res.writeHead(): the function writeHead() sends the http-status code 200 (Ok) as the standard response for successful HTTP requests and the response header back to the client. The response header is an object with the key content-type and the value text/html as here in the example. In case the requested url req.url does not match any of the conditions specified with if and else if writeHead()* sends the http-status code 404 (Not Found).
7. res.write(): The function write() sends a text stream back to the client. In our example write() send HTML that can be parsed by the browser to display a website content.
8. res.end(): the end() function tells the server that the request has ended.
9. At the end of the program, the listen() function is used on the server, which expects the port as the parameter. The port in this example is port 3000. listen() starts the server on port 3000 at the localhost.
10. The final statement log a message at the console that the server has been started.
    

3.2 Code encapsulation using self-developed modules

In our webserver example we load the node.js integrated http module with the require() function and we used the method createServer() with a requestListener() function that contains the entire server logic.

We write all the code in the server.js application main file which make the code confusing especially when we program more complex applications.

Node offers the possibility to encapsulate code in a separate file. When we have code in a separate file then we speak of a module. Like the integrated http module we load this module into the application main file server.js using the require() function.

With the help of directories you can structure your modules effectively. In this example I want to collect all my modules in a directory called modules. In this directory I create the logic.js file that contain all the server logic.

You can find the code on my GitHub Page.

Patricks-MBP:node-basic patrick$ ls -l
total 8
-rw-r--r--  1 patrick  staff  1282 08 Mär 05:58 server.js

Patricks-MBP:node-basic patrick$ mkdir modules

Patricks-MBP:node-basic patrick$ ls -l
total 8
drwxr-xr-x  2 patrick  staff    64 08 Mär 07:31 modules
-rw-r--r--  1 patrick  staff  1282 08 Mär 05:58 server.js

Patricks-MBP:node-basic patrick$ touch modules/logic.js

Patricks-MBP:node-basic patrick$ ls -l modules
total 0
-rw-r--r--  1 patrick  staff  0 08 Mär 07:31 logic.js

Patricks-MBP:node-basic patrick$ 

The server logic that is executed whenever the server receives a request should be in the modules/logic.js file. The modules/logic.js file contains the following code.

// modules/logic.js

const serverlogic = function(req, res) {
    // check conditions for requests
    if (req.url == '/') {

        // set response header
        res.writeHead(200, { 'Content-Type': 'text/html' });

        // set response content
        res.write('<html><body><p>This is my home Page.</p></body></html>');
        // End response
        res.end();

    }
    else if (req.url == "/about") {

        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.write('<html><body><p>This is my about Page.</p></body></html>');
        res.end();

    }
    else if (req.url == "/impressum") {

        res.writeHead(200, { 'Content-Type': 'text/html' });
        res.write('<html><body><p>This is my Impressum Page.</p></body></html>');
        res.end();

    }
    else {
      res.writeHead(404, { 'Content-Type': 'text/html' });
      res.write('<html><body><p>This path is not available. Invalid request</p></body></html>');
      res.end();
    }
  }

  module.exports = serverlogic;

The function with all instructions is assigned to the constant serverlogic. At the end of the file, the reference to the function is exported with module.exports. The function serverlogic() is now usable in other files of the application.

In order to integrate the new self-developed module into the server.js file, we use the require function again as follows.

// server.js

// load http integrated node module
const http = require('http')

// load self-developed module serverlogic from modules/logic.js
const serverlogic = require('./modules/logic');

// create the server
const server = http.createServer(serverlogic);

// 5. server listen for any incoming requests
server.listen(3000);

console.log('My node.js web server is alive and running at port 3000')

With the require() function we load the file modules/logic in the server.js application and assign the constant serverlogic. Now we can use the module in the createServer() method and the server runs exactly as before.

3.3 Using third-party modules

In our webserver example above we see how we can create a simple webserver using the node on-board resources. In our example we include the node.js integrated http module, which has been loaded with the require function and we used methods like createServer() that are provided by this integrated module. And we see how we can encapsulate the server logic to a separate file.

In general the scope of integrated node.js modules such as the http module is very minimal and is limited to the absolutely essential, which means that the node.js core is very compact and stable. The node.js core offers a very mature and stable runtime environment, but there is no comprehensive standard library to develop more complex web-applications easily. Therefore hundreds of thousands different third party modules from the node community are available.

Express.js is one of the most used third-party modules to create web-applications. The advantage of a module like Express is that this module contains many methods optimized for the http request handling. When using Express, for example the functions such as writeHead(), write() and end() can be replaced by one single function res.send(). And the node community offers many other third party modules that are tailored to the use of Express.

In order to use a third party module in a web application, it must be installed in the application root directory. The installation is done by the node package manager in the application root directory. Third party modules are also called dependencies. This means the application that lives in the application root directory and is defined by the various js-files contains dependencies of the installed third party modules.

I will therefore first delete the self-developed module so that I only have the application main file server.js in the application root directory.

Patricks-MBP:node-basic patrick$ ls -al
total 8
drwxr-xr-x  4 patrick  staff  128 08 Mär 18:10 .
drwxr-xr-x  3 patrick  staff   96 08 Mär 06:05 ..
drwxr-xr-x  3 patrick  staff   96 08 Mär 07:31 modules
-rw-r--r--  1 patrick  staff  387 08 Mär 07:51 server.js

Patricks-MBP:node-basic patrick$ rm -r modules

Patricks-MBP:node-basic patrick$ ls -al
total 8
drwxr-xr-x  3 patrick  staff   96 08 Mär 06:07 .
drwxr-xr-x  3 patrick  staff   96 08 Mär 06:05 ..
-rw-r--r--  1 patrick  staff  387 08 Mär 07:51 server.js

Patricks-MBP:node-basic patrick$

The third party module Express can now be easily installed in the application root directory as follows.

Patricks-MBP:node-basic patrick$ pwd
/Users/patrick/Software/dev/node/myArticles/2020-03-08/node-basic

Patricks-MBP:node-basic patrick$ npm install express
npm WARN saveError ENOENT: no such file or directory, open '/Users/patrick/Software/dev/node/myArticles/2020-03-08/node-basic/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/Users/patrick/Software/dev/node/myArticles/2020-03-08/node-basic/package.json'
npm WARN node-basic No description
npm WARN node-basic No repository field.
npm WARN node-basic No README data
npm WARN node-basic No license field.

+ express@4.17.1
added 50 packages from 37 contributors and audited 126 packages in 2.215s
found 0 vulnerabilities

Patricks-MBP:node-basic patrick$ ls -al
total 40
drwxr-xr-x   5 patrick  staff    160 08 Mär 06:14 .
drwxr-xr-x   3 patrick  staff     96 08 Mär 06:05 ..
drwxr-xr-x  52 patrick  staff   1664 08 Mär 06:14 node_modules
-rw-r--r--   1 patrick  staff  14240 08 Mär 06:14 package-lock.json
-rw-r--r--   1 patrick  staff    387 08 Mär 07:51 server.js

Patricks-MBP:node-basic patrick$ 

The third party module Express.js in version 4.17.1 has been installed successfully in the application root directory. Express.js is now available locally in our application perimeter and can therefore be used in the application.

In the application root directory we see the following changes.

• node_modules: the node_modules directory is created during the installation. npm stores all locally installed packages there. If you look into the directory you will see that there is currently more than just the Express module installed. This is due to the fact that Express itself requires modules which in turn were taken into account during this installation and were also installed.
• package-lock.json: The package-lock.json file is always generated automatically when npm changes the directory content in node_modules. The content of the package-lock.json shows the current complete directory tree of all modules and if you want to install an additional module or dependency it is ensured that the installation process creates exactly the directory tree at the end of the existing plus the directory of the newly installed module.
  

It is possible to install several modules in this way in order to use them in the application. If you continue installing modules, you will quickly lose track of which modules are installed in which version.

It is then no longer easy to depoloy the application, for example, from a Mac OS developer system to a Linux production system and keep exactly the same version of the modules used. The goal must be to install all the modules used in development in exactly the same way on another system or in another application root directory. Here node offers a possibility to do this with the help of the package.json file.

The package.json file is in your application root directory and list all modules or packages that have been installed in your application root directory. In node it is said that modules are packages and that an application depends on packages. In this respect, the package.json file lists all the packages on which your application depends on.

In order to demonstrate how we use a package.json file when installing the Express package, I will delete the node_modules directorory again and create a package.json file.

Patricks-MBP:node-basic patrick$ pwd
/Users/patrick/Software/dev/node/myArticles/2020-03-08/node-basic

Patricks-MBP:node-basic patrick$ ls -l
total 40
drwxr-xr-x  52 patrick  staff   1664 08 Mär 06:14 node_modules
-rw-r--r--   1 patrick  staff  14240 08 Mär 06:14 package-lock.json
-rw-r--r--   1 patrick  staff    524 08 Mär 09:34 server.js

Patricks-MBP:node-basic patrick$ rm -r node_modules

Patricks-MBP:node-basic patrick$ ls -l
total 40
-rw-r--r--  1 patrick  staff  14240 08 Mär 06:14 package-lock.json
-rw-r--r--  1 patrick  staff    524 08 Mär 09:34 server.js

Patricks-MBP:node-basic patrick$ touch package.json

Patricks-MBP:node-basic patrick$ ls -l
total 40
-rw-r--r--  1 patrick  staff  14240 08 Mär 06:14 package-lock.json
-rw-r--r--  1 patrick  staff      0 08 Mär 07:19 package.json
-rw-r--r--  1 patrick  staff    524 08 Mär 09:34 server.js

Patricks-MBP:node-basic patrick$

With the following command using the npm init –yes I create a default package.json file as follows. This guarantees a correct .json format with a standard structure and node or npm will not throw any error messages with this package.json file when you install packages.

Patricks-MBP:node-basic patrick$ ls -l
total 48
-rw-r--r--  1 patrick  staff  14240 08 Mär 06:14 package-lock.json
-rw-r--r--  1 patrick  staff    106 08 Mär 07:39 package.json
-rw-r--r--  1 patrick  staff    526 08 Mär 07:40 server.js

Patricks-MBP:node-basic patrick$ npm init --yes
Wrote to /Users/patrick/Software/dev/node/myArticles/2020-03-08/node-
basic/package.json:

{
  "name": "node-basic",
  "version": "1.0.0",
  "description": "",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


Patricks-MBP:node-basic patrick$

note: With the command npm init –yes default values are set in the package.json file. For example, the value of name is assigned to the directory name, the version is always 1.0.0 and main is usually index.js unless like in our case npm finds a server.js file that differs from index.js.

This package.json file can now be adjusted with our preferred editor according to our preferences. My personal preference is to keep the package.json file as simple as possible, so the default content for me is as follows.

{
  "name": "simple_webserver",
  "version": "0.0.1",
  "main": "server.js",
  "author": "Patrick Rottlaender"
}

Then I install Express.js again using the command npm install express --save.

Patricks-MBP:node-basic patrick$ ls -l
total 48
-rw-r--r--  1 patrick  staff  14240 08 Mär 06:14 package-lock.json
-rw-r--r--  1 patrick  staff    256 08 Mär 07:44 package.json
-rw-r--r--  1 patrick  staff    526 08 Mär 07:40 server.js

Patricks-MBP:node-basic patrick$ npm install express --save
npm WARN node-basic@1.0.0 No description
npm WARN node-basic@1.0.0 No repository field.

+ express@4.17.1
added 50 packages from 37 contributors and audited 126 packages in 2.895s
found 0 vulnerabilities

Patricks-MBP:node-basic patrick$ ls -l
total 48
drwxr-xr-x  52 patrick  staff   1664 08 Mär 07:46 node_modules
-rw-r--r--   1 patrick  staff  14286 08 Mär 07:46 package-lock.json
-rw-r--r--   1 patrick  staff    306 08 Mär 07:46 package.json
-rw-r--r--   1 patrick  staff    526 08 Mär 07:40 server.js

Patricks-MBP:node-basic patrick$ 

As you can see, the node_modules directory has also been created, in which the express module and all of its dependencies can now be found. Let’s take a look at the content of the package.json file, we see that a new object dependency has been added and we see that express in version 4.17.1 exists there.

{
  "name": "simple_webserver",
  "version": "0.0.1",
  "main": "server.js",
  "author": "Patrick Rottlaender",
  "dependencies": {
    "express": "^4.17.1"
  }
}

A very important point is the spelling of the version of the installed dependency in the package.json file. In our example express was installed in version „^ 4.17.1“. The preceding „^ …“ tells npm to install a version of at least version 4.17.1 or higher when calling npm install. This means that until the change to a new major version 5, npm would always update to the latest version. This can lead to problems in an application because when versions of dependencies are changed, the code may no longer be compatible and the code must be adapted.

Therefore I always recommend to remove the preceding „^ …“ to tell npm that the exact version has to be kept when calling npm install.

The following is the content of the package.json file.

{
  "name": "simple_webserver",
  "version": "0.0.1",
  "main": "server.js",
  "author": "Patrick Rottlaender",
  "dependencies": {
    "express": "4.17.1"
  }
}

note: It is also possible to tell npm at the time of the installation of a package that the latest version must be installed initially but the entry in the package.json file will be made without the preceding „^ …“. Then npm install does not update the version that is in the package.json file.

Using the example of the installation of express, the installation would be carried out with the following command: npm install express –save –save-exact.

If we assume we want to install exactly the same packages as dependencies in another directory or even on a different computer, we would proceed as follows. This is exactly the scenario when we have developed and tested an application and now want to bring this application into the production environment.

• we create a new target application root directory (new target directory)
• we copy the package.json file into the new target directory
• we copy the application files, in our case this is just the file server.js in the new target directory
• we change to the new target directory
• We run the command npm install without any other option
  

Patricks-MBP:node-basic patrick$ pwd
/Users/patrick/Software/dev/node/myArticles/2020-03-08/node-basic

Patricks-MBP:node-basic patrick$ ls -l
total 48
drwxr-xr-x  52 patrick  staff   1664 08 Mär 09:26 node_modules
-rw-r--r--   1 patrick  staff  14300 08 Mär 09:26 package-lock.json
-rw-r--r--   1 patrick  staff    171 08 Mär 09:26 package.json
-rw-r--r--   1 patrick  staff    526 08 Mär 07:40 server.js

Patricks-MBP:node-basic patrick$ cd ..

Patricks-MBP:2020-03-08 patrick$ ls -l
total 0
drwxr-xr-x  6 patrick  staff  192 08 Mär 09:26 node-basic

Patricks-MBP:2020-03-08 patrick$ mkdir express-basic

Patricks-MBP:2020-03-08 patrick$ ls -l
total 0
drwxr-xr-x  2 patrick  staff   64 08 Mär 09:39 express-basic
drwxr-xr-x  6 patrick  staff  192 08 Mär 09:26 node-basic

Patricks-MBP:2020-03-08 patrick$ cp node-basic/package.json express-basic
Patricks-MBP:2020-03-08 patrick$ cp node-basic/server.js express-basic

Patricks-MBP:2020-03-08 patrick$ cd express-basic

Patricks-MBP:express-basic patrick$ ls -l
total 16
-rw-r--r--  1 patrick  staff  163 08 Mär 10:03 package.json
-rw-r--r--  1 patrick  staff  526 08 Mär 09:40 server.js

Patricks-MBP:express-basic patrick$ cat package.json

{
  "name": "simple_webserver",
  "version": "0.0.1",
  "main": "server.js",
  "author": "Patrick Rottlaender",
  "dependencies": {
    "express": "4.17.1"
  }
}

Patricks-MBP:express-basic patrick$ npm install

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN simple_webserver@0.0.1 No description
npm WARN simple_webserver@0.0.1 No repository field.
npm WARN simple_webserver@0.0.1 No license field.

added 50 packages from 37 contributors and audited 126 packages in 1.864s
found 0 vulnerabilities

Patricks-MBP:express-basic patrick$ ls -l
total 48
drwxr-xr-x  52 patrick  staff   1664 08 Mär 10:05 node_modules
-rw-r--r--   1 patrick  staff  14292 08 Mär 10:05 package-lock.json
-rw-r--r--   1 patrick  staff    163 08 Mär 10:03 package.json
-rw-r--r--   1 patrick  staff    526 08 Mär 09:40 server.js

Patricks-MBP:express-basic patrick$  

Chapter 4: The simple webserver using the Express.js module

We are now working in the following directory. You can find the code on my GitHub Page.

Patricks-MBP:express-basic patrick$ pwd
/Users/patrick/Software/dev/node/myArticles/2020-03-08/express-basic

Patricks-MBP:express-basic patrick$ 

If we now want to use the Express module in our web application, we have to load the express module with the require() function. The express module returns a function and this function is stored in the constant express. When you call the express() function an app object will be returned. This app object will be now stored in the const app and can now be used in the server.js file. The code in server.js is now as follows.

// server.js

// load http module
const http = require('http')

// load the Express module
const express = require('express')

const app = express()

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My express web server is alive and running at port 3000')

As we can see in the code, we are now using the app as a parameter for the createServer() function. createServer() therefore creates and express web-server. Now lets run the code with node server.js and see what happens.

Patricks-MBP:express-basic patrick$ node server.js
My express web server is alive and running at port 3000

Perfect! The express server is running on localhost port 3000. But if we enter http://localhost:3000 in the browser we get an error message Cannot GET /. This is actually clear since we have not yet defined any routes in the code that tell the server how to respond to a request for the route „/“.

But there is something else interesting: the response of the server with the text Cannot GET / is already a website. That means Express already takes over for us the part of the error handling that we programmed without Express in the code above in a specific instruction.

This becomes even clearer when we issue an http request in the terminal with the help of the curl program.

Patricks-MBP:express-basic patrick$ curl -i http://localhost:3000
HTTP/1.1 404 Not Found
Content-Security-Policy: default-src 'none'
X-Content-Type-Options: nosniff
Content-Type: text/html; charset=utf-8
Content-Length: 139
Date: Sun, 15 Mar 2020 06:15:11 GMT
Connection: keep-alive

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /</pre>
</body>
</html>

Patricks-MBP:express-basic patrick$ 

note: curl is a program to transfer data from and to a server. curl supports some protocols including http. The -i option include the HTTP response headers in the output. The HTTP response headers include i.e. HTTP version, Content-Type, Date etc.

In the curl output above you can exactly see that the response from the server is a 404 Not Found http code and an html document to display an error HTML document in the browser. This is something that Express do for us.

So now we have to define routes to tell the server what to send in response when a particular route is requested. There are Express functions for certain http verbs like get(), post(), put() or delete() already available in Express. These functions are used on the app, which we referenced in our code with the constant app. So lets change our code as follows.

// server.js

// load http module
const http = require('http')

// load the Express module
const express = require('express')

const app = express()

// define the routes
app.get('/', (req, res) => {
  res.send('Hello, this is my home Page')
})

app.get('/about', (req, res) => {
  res.send('Hello, this is my about Page')
})

// create the server
const server = http.createServer(app);

// server listen for any incoming requests
server.listen(3000);

console.log('My express web server is alive and running at port 3000')

As you see I added a route definition for get(). The http verb functions like get() expect two parameters:

1. the route to be defined
2. the http handler which is transferred as a callback function with the parameters req and res. This http handler will give the server the instructions it needs to respond to the request for this particular route. Other routes like /about require a separate route definition.
   

We start server.js again and request the routes via the browser with http://localhost:3000 and http://localhost:3000/about and find that the response defined in res.send is sent as HTML from the server. Thats good.

With curl and requesting route /about we get the following terminal output.

Patricks-MBP:express-basic patrick$ curl -i http://localhost:3000/about

HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 28
ETag: W/"1c-qDZjsFWQH9PGrhmVTBEfY+9a0Fo"
Date: Sun, 15 Mar 2020 05:44:19 GMT
Connection: keep-alive

Hello, this is my about Page

Patricks-MBP:express-basic patrick$

The curl output now show 202 Ok http code and the html document content Hello, this is my about Page. So its all working fine so far.

Summary and Outlook

In this Part 1 I created a simple node.js web-server that respond to various http requests from a browser. I explained the basics of Node.js such as the installation of node and npm, the use of modules and configuration of the package.json file. Then I installed the Express.js framework as a dependency of my server application and configured it so that it can be used by the simple web-server.

In the following Part 2 of this node.js series I will explain the Express.js framework much more in detail. We will learn something about web api(s) and we will understand how we can use Express as middleware in a web application. And we will create a small blog application.

Note: I will not go into the installation of the Firebase CLI here. How this is installed on the computer is well described in Full-Stack Firebase and in the Google documentation.

Overview

Before I start a firebase project, I first think of a suitable name for the project and create one project for development (Firebase Project Dev) and another different one for production (Firebase Project Prod) in the Firebase Console. As a result, 2 projects with the following names are created in the Firebase Console (see the Browser side chapter below).

• myproject-Dev
• myproject-Prod
  

In addition, 2 directories must be created on the computer. One directory as Directory Dev and another as Directory Prod. Ideally, these directories have the same names as the projects in the Firebase Console (see the Command line side chapter below).

• myproject-Dev
• myproject-Prod
  

As shown in the figure above, the directories on the computer and the Firebase projects logically each represent a system. So in my example I have a Dev System and a Prod System. The two systems Prod and Dev are completely separated from each other.

The system development takes place on the computer. The files and directories for development are located on the computer directly in the myproject-Dev directory. The public subdirectory in the myproject-Dev directory contains all the files and directories relevant to the dev depolyment. The Dev deploy process copies these relevant files and directories for dev deployment there (see the Dev deployment chapter below).

Note: The system development is not part of this tutorial.

The files and directories for production are in the public subdirectory in the myproject-Prod directory. The process Prod deploy copies these relevant files and directories for the Prod deployment there (see the chapter Prod deployment below).

The directories and files for Firebase hosting are copied from the relevant public directory and are performed by the Firebase CLI function firebase deploy. In our example here we have hosting of the Dev System and hosting of the Prod System. The Dev System is used for testing under production conditions. The Prod System represents the actual live system on the web.

Browser side

The following activities are carried out in the browser.

How a Firebase project is created in the firebase console can be read in detail very well on Full-Stack Firebase. I will only go into the basic steps of creating a project here.

First we go to the Firebase Console in the browser. When initially creating a firebase project in the Firebase Console, the following steps are carried out. With this procedure we create the two projects myproject-Dev and myproject-Prod in the Firebase Console.

1. Click Add Project
2. Assign a project name
3. Switch on Google Analytics (default) and configure Analytics account
   

After the initial creation of the project in the Firebase Console, you go directly to the project overview in the project console of the newly created project. Here a Firebase app must be added to the project. In order to fully understand this tutorial, a web app is added to the project to create a website (iOS or Android mobile apps can also be added here later).

1. Click Add Web-App
2. Register the Alias name for the Web-App myproject-Dev-WebApp
3. Firebase hosting is to be set up for this web app
4. Then follow 3 steps for information that are simply noted (add Firebase SDK, install Firebase CLI, provide Firebase hosting)
5. Then go back to the project console project overview
   

In the project overview, click on the settings of the web app myproject-Dev-WebApp in the top left of the main window. The following configurations must be made here in the General area:

1. Google Cloud Platform (GCP) resource location
2. Support Email Address
   

When setting the default location for the cloud resources, it is important to know that this can only be done once here at this point. I choose here as the location of the cloud resources europe-west3. At this point, of course, any other location can be selected depending on which is meaningful or necessary for the developer.

This process must also be repeated for myproject-Prod.

Command line side

The following activities are carried out in the terminal or on the command line of the computer. The computer I use is a Mac. As mentioned above these steps require the installation of the Firebase CLI.

First you log on to the Firebase Console. To do this, the firebase login command must be entered in the command line. Then all other commands can be entered in the command line.

With the command firebase projects:list in the command line the projects created in the firebase console can now be listed.

|Project Display Name |Project ID |Resource Location ID |

|———————–|———————-|———————|

|myproject-Prod | myproject-prod-19842 | europe-west3 |
|myproject-Dev | myproject-dev-ace74 | europe-west3 |

Then I create two folders locally on my computer, one for development and one for production and initialize firebase in each of the folders.

Later in this article I’ll go into how depolyment from development to production takes place.

In order to initialize the Firebase project on the computer the following steps have to be followed.

Step 0: Create Folders on the computer

In the initial situation, I have already created 2 folders on my computer: One for development and a different one for production. These folders are:

• myproject-Dev
• myproject-Prod
  

Patricks-MBP:firebase-test patrick$ ls -l
total 0
drwxr-xr-x  2 patrick  staff  64 17 Dez 17:26 myproject-Dev
drwxr-xr-x  2 patrick  staff  64 17 Dez 17:25 myproject-Prod
Patricks-MBP:firebase-test patrick$

First I initialize firebase for myproject-Dev. The Steps 1 to the final step must be repeated in the same way for myproject-Prod.

To do this, I change to the myproject-Dev directory on my computer and enter the command firebase init on the command line of my computer.

Patricks-MBP:firebase-test patrick$ cd myproject-Dev
Patricks-MBP:myproject-Dev patrick$ firebase init

firebase init: The command firebase-init links the development project previously created in the Firebase Console to the myproject-Dev directory. After entering the command, the developer has to complete different stepts to setup the project. Once the process has been completed, various files are created in myproject-Dev, which I will explain in parts below.

The first step to be completed by the developer relates to the Firebase services to be used. I will use firestore, hosting and storage in this example.

Step 1: Initialize a Firebase project

Patricks-MBP:myproject-Dev patrick$ firebase init

     ######## #### ########  ######## ########     ###     ######  ########
     ##        ##  ##     ## ##       ##     ##  ##   ##  ##       ##
     ######    ##  ########  ######   ########  #########  ######  ######
     ##        ##  ##    ##  ##       ##     ## ##     ##       ## ##
     ##       #### ##     ## ######## ########  ##     ##  ######  ########

You're about to initialize a Firebase project in this directory:

  /Users/patrick/Sites/dev/tutorials/firebase-test/myproject-Dev

? Which Firebase CLI features do you want to set up for this folder? Press Space to select features, then Enter to confirm your choices.
 ◯ Database: Deploy Firebase Realtime Database Rules
 ◉ Firestore: Deploy rules and create indexes for Firestore
 ◯ Functions: Configure and deploy Cloud Functions
 ◉ Hosting: Configure and deploy Firebase Hosting sites
❯◉ Storage: Deploy Cloud Storage security rules
 ◯ Emulators: Set up local emulators for Firebase features

Firestore: Modern json database. This database can be accessed via the JavaScript SDK offered by Google in the JavaScript of the application. Access rights to the database are controlled via security rules that can be configured locally and loaded into the project using the deployment process.

Hosting: Self-explanatory. The project can be hosted on a Google webserver.

Storage: File server that can also be accessed via the JavaScript SDK in the application’s JavaScript. Access rights to the cloud storage are controlled via security rules that can be configured locally and loaded into the project using the deployment process.

After step 1 has been completed, the project is connected to the current folder on the computer (the directory myproject-Dev we are in).

After the first step has been completed with Enter, the second step is to connect the project directory we are in on the computer myproject-Dev to the Firebase Project myproject-Dev, which was previously created in the Firebase Console.

Step 2: Project Setup

=== Project Setup

First, let's associate this project directory with a Firebase project.
You can create multiple project aliases by running firebase use --add,
but for now we'll just set up a default project.

? Please select an option: Use an existing project
? Select a default Firebase project for this directory:  
❯ myproject-dev-ace74 (myproject-Dev)
  myproject-prod-19842 (myproject-Prod)

Steps 3 to 5 configure the services previously selected in Step 1.

Step 3: Firestore Setup

=== Firestore Setup

Firestore Security Rules allow you to define how and when to allow
requests. You can keep these rules in your project directory
and publish them with firebase deploy.

? What file should be used for Firestore Rules? firestore.rules

Firestore indexes allow you to perform complex queries while
maintaining performance that scales with the size of the result
set. You can keep index definitions in your project directory
and publish them with firebase deploy.

? What file should be used for Firestore indexes? firestore.indexes.json

Step 4 Hosting Setup

=== Hosting Setup

Your public directory is the folder (relative to your project directory) that
will contain Hosting assets to be uploaded with firebase deploy. If you
have a build process for your assets, use your build's output directory.

? What do you want to use as your public directory? public
? Configure as a single-page app (rewrite all urls to /index.html)? Yes
✔  Wrote public/index.html

Step 5: Storage Setup

=== Storage Setup

Firebase Storage Security Rules allow you to define how and when to allow
uploads and downloads. You can keep these rules in your project directory
and publish them with firebase deploy.

? What file should be used for Storage Rules? storage.rules

Final Step: Writing Files

i  Writing configuration info to firebase.json...
i  Writing project information to .firebaserc...
i  Writing gitignore file to .gitignore...

✔  Firebase initialization complete!

The directory myproject-Dev now has the following content.

Patricks-MBP:myproject-Dev patrick$ ls -l
total 32
-rw-r--r--  1 patrick  staff  379 18 Dez 07:30 firebase.json
-rw-r--r--  1 patrick  staff  563 18 Dez 07:27 firestore.indexes.json
-rw-r--r--  1 patrick  staff  132 18 Dez 07:27 firestore.rules
drwxr-xr-x  3 patrick  staff   96 18 Dez 07:29 public
-rw-r--r--  1 patrick  staff  138 18 Dez 07:30 storage.rules

Here you can see that the files for controlling access and indexes to the firestore and the files for controlling access to storage were created. Rules and indexes are transferred with the firebase deploy command. It is therefore important to enter any changes to the rules that may have been made during development in the Firebase Console here.

In addition, the public directory was created. The application is deployed from this directory with the firebase deploy command.

Note: I use a local apache web server for development on my Mac. This web server is configured so that websites in a local directory within my user directory are displayed as private websites (localhost). The directory myproject-Dev is created under localhost. In this respect, html files, css and Javascript files etc. can be created in myproject-Dev directory and displayed from there via the local web server in the browser by entering the url localhost/~user/myproject-Dev. This helps the developer to see his results and to do some debugging where necessary. A very good description of how to set up a local web server on a Mac can be found at Website Beaver.

As an example for a typical web application, the following directories and files must be created in the myproject-Dev directory.

• css Directory
  
  • mystyles.css File
• script Directotry
  
  • main.js File
• images Directory
  
  • .png or jpg. Files
• dev-versions Directory
• index.html File
• deploy-dev File
• deploy-prod File
  

The directories css, script and images and their files are created for web application development. The dev-versions directory is is used for versioning the web application development results.

The files deploy-dev and deploy-prod each contain a script for performing the respective deployment either in the Dev or Prod.

A typical directory structure in myproject-Dev looks like this.

Patricks-MBP:myproject-Dev patrick$ ls -l
total 48
drwxr-xr-x  3 patrick  staff   96 18 Dez 11:09 css
-rwxr--r--  1 patrick  staff  251 18 Dez 15:00 deploy-dev
-rwxr--r--  1 patrick  staff  240 18 Dez 14:54 deploy-prod
drwxr-xr-x  3 patrick  staff   96 18 Dez 15:01 dev-versions
-rw-r--r--  1 patrick  staff  379 18 Dez 07:30 firebase.json
-rw-r--r--  1 patrick  staff   10 18 Dez 12:31 firestore.indexes.json
-rw-r--r--  1 patrick  staff   31 18 Dez 11:05 firestore.rules
drwxr-xr-x  4 patrick  staff  128 18 Dez 12:32 images
-rw-r--r--  1 patrick  staff  386 18 Dez 11:31 index.html
drwxr-xr-x  8 patrick  staff  256 18 Dez 15:01 public
drwxr-xr-x  3 patrick  staff   96 18 Dez 11:10 script
-rw-r--r--  1 patrick  staff   31 18 Dez 11:05 storage.rules

The Firebase initialization for the development system is then completed.

As already mentioned above, this process must be repeated for myproject-Prod from step 1 in order to also initialize the production system accordingly. After initialization, only the prod-versions directory has to be created as subdirectory of the myproject-Prod directory. This directory is is used for versioning the web application.

A typical directory structure in myproject-Prod looks like this.

Patricks-MBP:myproject-Prod patrick$ ls -l
total 32
-rw-r--r--  1 patrick  staff  27 18 Dez 18:45 firebase.json
-rw-r--r--  1 patrick  staff  28 18 Dez 18:45 firestore.indexes.json
-rw-r--r--  1 patrick  staff  21 18 Dez 18:46 firestore.rules
drwxr-xr-x  2 patrick  staff  64 18 Dez 15:15 prod-versions
drwxr-xr-x  2 patrick  staff  64 18 Dez 18:35 public
-rw-r--r--  1 patrick  staff  19 18 Dez 18:46 storage.rules

Dev-Deployment

To deploy the application to the Dev-System in Firebase, all the necessary files must be copied to the public directory under myproject-Dev. The bash script deploy-dev does this.

Patricks-MBP:myproject-Dev patrick$ cat deploy-dev
#!/bin/bash

date=$(date +"%Y.%m.%d_%H.%M.%S")

mkdir dev-versions/dev-ver_$date

cp -Rip public dev-versions/dev-ver_$date

rm -R public/*

cp -ip index.html public
cp -Rip css public
cp -Rip script public
cp -Rip images public

deploy-dev stores the current date and time in the variable date. Then a subdirectory named like dev-ver_$date is created under the directory dev-versions and the entire public directory is copied there. The public directory is then emptied and the new application files are copied to the public directory.

Then the command firestore deploy must be called on the command line and the deployment from the current public directory to the dev system is started.

Note: The command firestore deploy could also be added to the deploy-dev script. I prefer to run this command separately afterwards.

After the Firebase Deployment is complete, you get a project url in the command line with which you can access via the web the Dev-System that has just been published. Now the tests can be performed using the Dev-System url.

Prod-Deployment

After the tests have all been successfully completed, the application can be published in the Prod system.

We are still in the myproject-Dev directory.

The goal is to copy all files from the public directory in myproject-Dev into the public directory of myproject-Prod. The bash script deploy-prod does this.

Patricks-MBP:myproject-Dev patrick$ cat deploy-prod
#!/bin/bash

date=$(date +"%Y.%m.%d_%H.%M.%S")

mkdir ../myproject-Prod/prod-versions/prod-ver_$date

cp -Rip ../myproject-Prod/public ../myproject-Prod/prod-versions/prod-ver_$date

rm -R ../myproject-Prod/public/*

cp -Rip public/* ../myproject-Prod/public

deploy-prod stores the current date and time in the variable date. Then on myproject-Prod a subdirectory named like prod-ver_$date is created under the directory prod-versions and the entire directory public is copied there. The public directory on myproject-Prod is then emptied.

The new application files are copied from the publicdirectory in myproject-Devto the public directory of myproject-Prod.

Then the command firestore deploy must be called on the command line and the deployment from the public directory to the prod system is started.

Note: The command firestore deploy could also be added to the deploy-prod script. I prefer to run this command separately afterwards.