Chapter 1: Initial Setup and Key Creation

This stage establishes your key pair and trust level.

Create Master Key Pair incl. encryption subkey

gpg --full-gen-key

On the following dialog you use ECC algorithms (ed25519 for signing, cv25519 for encryption) and chose option (9) ECC (sign, encryption) standard. You set an expiration date (e.g., 5 years) for forward compatibility. Finally you choose a strong, unique passphrase!

Create signing Subkey

Do not use the Master Key for signing in your daily work. Create an additional separate subkey for signing (S).

gpg --edit-key <KEY-ID>

gpg\> addkey

You type addkey and then option (10) ECC (only sign).

Then you check the generated Master Key including the subkeys

gpg --list-secret-keys

sec   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

sec: The secret master key (sec) is for Signing and Certification (SC).

KEY-ID: C957EA3BD909D1D3BA594D2CE14085D8197A0018.

ssb: The first subkey (ssb) is for encryption (E).

ssb: The second subkey (ssb) is for signing (S).

All Keys expire at 2030-12-24.

Set Trust Level

As you can see above my trust level is already ultimate. So here just for completion the command how to set the trust level on your keys.

Mark your own key as ultimately trusted so GnuPG relies on your certifications (key separation, subkey creation).

gpg --edit-key <KEY-ID>

gpg\> trust

Then type trust and select option 5 (ultimate).

Chapter 2: Security and Hardening

This stage prepares the necessary backups and secures the master key.

Backup the Master Key and generate Revocation Certificate

Do this immediately and store the fullback-up file and the revocation certificate file offline.

gpg --export-secret-keys --armor <KEY-ID> > fullbackup.asc	

gpg --output revocation.asc --gen-revoke <KEY-ID>

Implement Key Separation

Check the key setup again to identify clearly your 3 Key-IDs. Here we use the long format command.

gpg --list-secret-keys --keyid-format long

sec   ed25519/E14085D8197A0018 2025-12-25 \[SC] \[verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid              [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)

ssb   cv25519/AF0CB4F423538B80 2025-12-25 [E] [verfällt: 2030-12-24]

ssb   ed25519/56025D0BF8BF5B2F 2025-12-25 [S] [verfällt: 2030-12-24]

Master Key-ID (SC): E14085D8197A0018

Subkey 1-Key-ID (E): AF0CB4F423538B80

Subkey 2-Key-ID (S): 56025D0BF8BF5B2F

The goal is: Remove the Master Secret Key (sec) from your local machine using the delete command, turning it into a stub (sec#). This protects the core of your identity from malware. Keep both Subkeys for daily encryption (E) and signing (S).

Before you run the delete command, you must be 100% certain your backup file is valid. If you delete the master key now without a verified backup, you are back to the fatal „lost key“ situation. Run the following 2 checks.

Check 1:

gpg --list-packets /path/to/fullbackup.asc | grep "secret key packet"

:secret key packet:

The output must! show:

:secret key packet:

Check 2:

gpg --list-packets /path/to/fullbackup.asc

off=0 ctb=94 tag=5 hlen=2 plen=134
:secret key packet:
	version 4, algo 22, created 1766645842, expires 0
	pkey[0]: [80 bits] ed25519 (1.3.6.1.4.1.11591.15.1)
	pkey[1]: [263 bits]
	iter+salt S2K, algo: 7, SHA1 protection, hash: 2, salt: E5DD0BCA22F746A5
	protect count: 65011712 (255)
	protect IV:  d6 b2 c9 5d e2 3f 05 d3 35 fb cc 7b 87 41 cf f5
	skey[2]: [v4 protected]
	keyid: E14085D8197A0018
# off=136 ctb=b4 tag=13 hlen=2 plen=44
:user ID packet: "Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)"
# off=182 ctb=88 tag=2 hlen=2 plen=153
:signature packet: algo 22, keyid E14085D8197A0018
	version 4, created 1766645842, md5len 0, sigclass 0x13
	digest algo 10, begin of digest f0 54
	hashed subpkt 33 len 21 (issuer fpr v4 C957EA3BD909D1D3BA594D2CE14085D8197A0018)
	hashed subpkt 2 len 4 (sig created 2025-12-25)
	hashed subpkt 27 len 1 (key flags: 03)
	hashed subpkt 9 len 4 (key expires after 5y0d0h0m)
	hashed subpkt 11 len 4 (pref-sym-algos: 9 8 7 2)
	hashed subpkt 34 len 1 (pref-aead-algos: 2)
	hashed subpkt 21 len 5 (pref-hash-algos: 10 9 8 11 2)
	hashed subpkt 22 len 3 (pref-zip-algos: 2 3 1)
	hashed subpkt 30 len 1 (features: 07)
	hashed subpkt 23 len 1 (keyserver preferences: 80)
	subpkt 16 len 8 (issuer key ID E14085D8197A0018)
	data: [256 bits]
	data: [255 bits]
# off=337 ctb=9c tag=7 hlen=2 plen=139
:secret sub key packet:
	version 4, algo 18, created 1766645842, expires 0
	pkey[0]: [88 bits] cv25519 (1.3.6.1.4.1.3029.1.5.1)
	pkey[1]: [263 bits]
	pkey[2]: [32 bits]
	iter+salt S2K, algo: 7, SHA1 protection, hash: 2, salt: 2DE27015BEF91567
	protect count: 65011712 (255)
	protect IV:  f8 13 05 2b 8d a0 97 0d 22 99 3b b5 c3 62 43 be
	skey[3]: [v4 protected]
	keyid: AF0CB4F423538B80
# off=478 ctb=88 tag=2 hlen=2 plen=126
:signature packet: algo 22, keyid E14085D8197A0018
	version 4, created 1766645842, md5len 0, sigclass 0x18
	digest algo 10, begin of digest fc 2e
	hashed subpkt 33 len 21 (issuer fpr v4 C957EA3BD909D1D3BA594D2CE14085D8197A0018)
	hashed subpkt 2 len 4 (sig created 2025-12-25)
	hashed subpkt 27 len 1 (key flags: 0C)
	hashed subpkt 9 len 4 (key expires after 5y0d0h0m)
	subpkt 16 len 8 (issuer key ID E14085D8197A0018)
	data: [254 bits]
	data: [256 bits]
# off=606 ctb=9c tag=7 hlen=2 plen=134
:secret sub key packet:
	version 4, algo 22, created 1766645997, expires 0
	pkey[0]: [80 bits] ed25519 (1.3.6.1.4.1.11591.15.1)
	pkey[1]: [263 bits]
	iter+salt S2K, algo: 7, SHA1 protection, hash: 2, salt: 09FD8782B037DC77
	protect count: 65011712 (255)
	protect IV:  e3 fc 51 3f 7d 35 f8 78 a9 43 72 fe 73 f9 82 16
	skey[2]: [v4 protected]
	keyid: 56025D0BF8BF5B2F
# off=742 ctb=88 tag=2 hlen=2 plen=245
:signature packet: algo 22, keyid E14085D8197A0018
	version 4, created 1766645997, md5len 0, sigclass 0x18
	digest algo 10, begin of digest fe 58
	hashed subpkt 33 len 21 (issuer fpr v4 C957EA3BD909D1D3BA594D2CE14085D8197A0018)
	hashed subpkt 2 len 4 (sig created 2025-12-25)
	hashed subpkt 27 len 1 (key flags: 02)
	hashed subpkt 9 len 4 (key expires after 5y0d0h0m)
	subpkt 16 len 8 (issuer key ID E14085D8197A0018)
	subpkt 32 len 117 (signature: v4, class 0x19, algo 22, digest algo 10)
	data: [256 bits]
	data: [256 bits]

This is the expected output: You expect 3 packets most important is the :secret key packet:

Master Key package:

:secret key packet: secret master key packet with KeyID: E14085D8197A0018

:signature packet: This show the signature packet belonging to your secret key packet and issuer key ID E14085D8197A0018

Subkey 1 Key package:

:secret sub key packet: secret subkey packet with KeyID: AF0CB4F423538B80

:signature packet: This show the signature packet belonging to your secret sub key packet and issuer key ID E14085D8197A0018

Subkey 2 Key package:

:secret sub key packet: secret subkey packet with keyid: 56025D0BF8BF5B2F

:signature packet: This show the signature packet belonging to your secret sub key packet and issuer key ID E14085D8197A0018

Only in case :secret key packet: is available proceed!

GnuPG does not have a single command like —delete-only-master key. Instead, the standard „clean“ way to do this is a three-step process.

Step 1: Export ONLY your Secret Subkeys in a temporary subkeys-only file

gpg --export-secret-subkeys --armor E14085D8197A0018 > subkeys-only.asc

Step 2: Delete the current Secret Key (complete wipe) from your MacBook

gpg --delete-secret-keys E14085D8197A0018

Step 3: Import the Secret Subkeys

gpg --import subkeys-only.asc

Step 4: Remove the temporary subkeys-only file

rm subkeys-only.asc

Then check your local setup.

gpg --list-secret-keys

sec#  ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
      C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender <patrick@rottlaender.eu>
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

Here you see your Master Secret Key (sec) has been removed from your local machine and turning it into a stub (sec#).

Chapter 3: Reset and Restore (Scenario: Start From Scratch)

These steps show how to completely wipe your local key and restore it using the backup.

Delete Complete Key Set (Local Wipe)

Use both commands to ensure both the remaining subkeys and the Public Key are completely removed from your local keyring before restoring.

gpg --delete-secret-keys <KEY-ID> (Deletes subkeys/stubs)
gpg --delete-keys <KEY-ID> (Deletes public key)

Import Complete Key Set (Restore)

Use the verified backup file created in Step 2 to fully restore your Master Key and Subkeys back to the pre-separation state.

First you check the fullback file before you run the import.

#Check the backupfile
gpg --show-keys /path/to/fullbackup.asc

sec   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

#run the import
gpg --import /path/to/fullbackup.asc

Chapter 4: Useful commands

List public keys

gpg –list-keys

pub   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
sub   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
sub   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

List secret keys

gpg –list-secret-keys

sec   ed25519 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid        [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519 2025-12-25 [S] [verfällt: 2030-12-24]

List public keys showing all KeyIDs

gpg --list-keys --keyid-format long

pub   ed25519/E14085D8197A0018 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid              [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
sub   cv25519/AF0CB4F423538B80 2025-12-25 [E] [verfällt: 2030-12-24]
sub   ed25519/56025D0BF8BF5B2F 2025-12-25 [S] [verfällt: 2030-12-24]

List secret keys showing all KeyIDs

gpg --list-secret-keys --keyid-format long

sec   ed25519/E14085D8197A0018 2025-12-25 [SC] [verfällt: 2030-12-24]
	  C957EA3BD909D1D3BA594D2CE14085D8197A0018
uid              [ ultimativ ] Patrick Rottlaender [patrick@rottlaender.eu](mailto:patrick@rottlaender.eu)
ssb   cv25519/AF0CB4F423538B80 2025-12-25 [E] [verfällt: 2030-12-24]
ssb   ed25519/56025D0BF8BF5B2F 2025-12-25 [S] [verfällt: 2030-12-24]

Master Key (SC): E14085D8197A0018

Subkey 1 (E): AF0CB4F423538B80

Subkey 2 (S): 56025D0BF8BF5B2F

Encrypt

gpg -e -r patrick@rottlaender.eu file.txt

Decrypt

gpg -d file.txt.gpg > file.txt

Sign & Encrypt

gpg -se -r recipient@email.com file.txt

Decrypt & Verify

#shows the signature status automatically
gpg -d file.txt.gpg

Chapter 5: General note

ed25519 and cv25519 are algorithms based on Curve25519

ed25519 (Master Key & Signing Subkey)

Feature Description

Full Name: Edwards-curve Digital Signature Algorithm over Curve25519.

Cryptographic Role: Digital Signing and Verification. It is designed to create a mathematical signature that proves the origin (authenticity) and integrity of a message or file.

GnuPG Use: Used for your Master Key ([C]) to certify other keys, and your Signing Subkey ([S]) to sign emails or commits.

Security Property: It is highly resistant to various side-channel attacks and has a very reliable, straightforward implementation.

cv25519 (Encryption Subkey)

Feature Description

Full Name: Curve25519 variant used for Key Exchange.

Cryptographic Role: Key Exchange and Encryption. It is designed for the Diffie-Hellman (DH) key exchange process, which allows two parties to securely establish a shared secret (like a session key) over an insecure channel.

GnuPG Use: Used for your Encryption Subkey ([E]). It enables others to encrypt data to you, and you to decrypt it using the established session key.

Security Property: Highly efficient and designed specifically for confidential (private) communication.

Modern GnuPG key management strictly separates cryptographic roles signing and encryption for security reasons:

• You sign with ed25519: To prove you are the one you are (in my case I am Patrick!).
• You encrypt with cv25519: To ensure privacy.

This separation of duties means if one subkey were ever theoretically broken, it wouldn’t automatically compromise the function of the other key. For example, if your signing key (ed25519) was compromised, your ability to decrypt past files or conversations (protected by cv25519) would remain secure.

Under no circumstances should you start attacks on systems located on the internet, even if the systems belong to you, because from a legal point of view it is always a risk and you can get into trouble. You should build a game environment in your local network in which you install attacker and victim system. It is important that you only use private IP addresses and that the game environment does not route any data traffic to and from the Internet. This gaming environment is then your Hacking Lab in which you can try out different attack scenarios safely and securely.

The idea is to install a virtualization software on the operating system of the host computer which is in my case an M2 Apple Mac with Ventura 13.3. The virtualization software I use is UTM. UTM create the virtual guest machines in a virtual Lan (Vlan) isolated from the host operating system and the local network. The virtual guest machines can talk to each other but can not access the local network or the internet via the host. The host can talk to the virtual guest machines. This network mode is defined as host only mode.

Note: UTM provide various network operating modes. Details can be read in the UTM documentation.

The installation of UTM can be done in two ways:

1. Payed version in the Apple App Store
2. Free download from the UTM site

I recommend to install the payed version from the App Store. You will get all updates whenever there are updates available and you support the development of this fantastic tool. At the time of writing this article the costs for the payed version of UTM were 11.99 Euro.

Once UTM has been installed we need to create the attacker and the victim as virtual guest machines in UTM. The attacker is a Kali Linux Image the victim will be a host prepared with vulnarabilities from vulnerable hub.

On vulnerable hub you find vulnerable virtual machines. These machines are prepared from cyber security enthusiasts for other security enthusiasts. These machines are available with vulnerabilities and created specifically for the purpose of hacking them.

Create a vulnerable virtual machine in UTM

To create a victim in UTM you search for a machine which best suits your needs and download the virtual machine from vulnerable hub which is available there in the .ova format. OVA is basically a tar based archive that contains among other things, an .ovf file with the specification of the virtual machine and in most cases disk image files in the format .vdi and .vmdk. Other files contained in the .ova file we don’t consider.

On vulnerable hub we search for Robot and find 3 search results. We look for Mr-Robot: 1, click on it and find the download link to download the .ova file for the vulnerable virtual machine.

To install Mr-Robot: 1 as virtual machine in UTM we first must unpack the .ova file.

patrick % ls
total 2884016
mrRobot.ova

patrick % tar -xvf mrRobot.ova 
x mrRobot.ovf
x mrRobot.mf
x mrRobot-disk1.vmdk

patrick % ls
total 2884016
mrRobot-disk1.vmdk mrRobot.mf mrRobot.ova mrRobot.ovf

Then we convert the disk image file mrRobot-disk1.vmdk into the .qcow2 format which is the supported format by UTM. If you need some more detailed information there is a very good explanation on Xmodulo.

To convert the disk image from into .qcow2 we need the utility qemu-img. I install the complete qemu emulator from homebrew as the qemu utilities are not available as single option. If you don’t know homebrew or in case homebrew is not installed on your Mac just go to my blog site digitaldocblog and read the article how to intstall and use homebrew on your Mac.

patrick %  brew install qemu

patrick % ls
total 2884016
mrRobot-disk1.vmdk mrRobot.mf mrRobot.ova mrRobot.ovf

patrick % qemu-img convert -O \
qcow2 mrRobot-disk1.vmdk mrRobot.qcow2

patrick % ls
total 2884016
mrRobot-disk1.vmdk mrRobot.mf mrRobot.ova mrRobot.ovf
mrRobot.qcow2

Then you start UTM from the main menue and click create new virtual machine.

Then you click on Emulate. You click on Emulate because the virtual machine you are going to install is compiled for the x86 64 Intel CPU Architechture. This Intel CPU must be emulated by UTM to provide the basis that your host can run as guest on your Apple ARM Silicon Mac (M1 and M2). In the next chapter when we install the attacker Linux we choose a virtual machine that is compiled for the Apple ARM 64 Silicon architechture. Then we click Virtualize because this guest machine is compiled for your host architecture (M1 or M2).

After Emulate you click on Custom. Then you skip ISO boot, accept in the following all default values ​​and assign a name fitting your needs for your new virtual host. At the end, the configuration is completed with safe.

In the left panel you see the virtual machine you just created. Mark it and click on the settings menue at the top right side.

In the left panel of the settings go to the Drives section and delete the IDE drive. Stay in the Drives section and create a new drive. Here you click on Import and select the .qcow2 file on your hard disk (the one you created as you converted the .vmdk file). After this step go to the QEMU section where you unselect UEFI Boot. To set the correct network settings go to Network section and select Host only. Finally you click on safe and end the configuration.

Create the attacker virtual machine in UTM

To launch attacks against a vulnerable virtual host I recommend installing a standard Kali Linux machine in UTM. Most tools are already pre-installed here and you can start immediately.

We go on the Kali.org website an there into the section get Kali. I choose installer images and then we select the ARM Silicon (ARM64) for download the Kali especially compiled for our Apple ARM Silicon M2 architecture (if you have a M1 this is also working fine). Click on the recommended installer and download the iso image to your hard drive.

After you have downloaded the iso image the installation will be performed in 2 steps. First you create a new virtual machine in virtualization mode and then in step 2 you install the Kali Linux on it.

Click on the + sign in the top menue and then select Virtualize.

After that you can select the OS you want to install. Here you click on Linux and here you select the iso image you downloaded from Kali in the Boot-Iso-Image section.

Then you click continue and keep the default value for the RAM but set the CPU cores to 4.

In the following you can keep all default values i.e. for storage and always click on continue until you come to the Summary section. Here you can give you new virtual machine a name according to your needs. Finally you click on safe.

Then click on the settings to open the settings menue. Go to the Devices section and click on new and add a new serial device emulation. This is very important for the installation of Kali Linux on your virtual machine otherwise the installer will not start.

Run the virtual machine to start the installation. Here you see that 2 windows will be opened. Choose the window that says terminal one and start the installation. Don’t choose Graphical installation. Then follow the installation steps. You can see a very good video on YouTube that guide you through the standard installation.

Once you see the finish installation screen then use the navigation bar of this window terminal 1 and shut down the virtual machine. Close the second window that has been opened when you started with the installation.

Then you go to the main window, select the virtual machine you just installed, go to the main window and unmount the iso at the bottom of the page.

Open the settings and go to the device sections and remove the serial device you created before. To set the correct network settings go to Network section and select Host only. Finally you click on safe and end the configuration.

Check the network configuration

Run both virtual machines in UTM, Robot1VulnerableHost and the Attacker machine KaliLinux. When you run Robot1VulnerableHost you see the logon screen after the machine has beefed. Of course you don’t have login credentials because you want to hack into it instead of logging in. This is different on your KaliLinux. Here you created an account during the installation.

Log into KaliLinux and open a Terminal. Check the ip address from KaliLinux. The ip address is 192.168.128.5.

Then discover the network 192.168.128.0/24 to find other hosts in the same subnet. Type the following command.

kaliLinux % netdiscover -r 192.168.128.0/24

You see 2 hosts in your subnet 192.168.128.1 which is the bridge between your local host network and your vlan and you see 192.168.128.6 which is Robot1VulnerableHost on the same subnet.

You can ping Robot1VulnerableHost but you cannot ping the host computer which is my Mac on subnet 192.168.0.0/24 with the ip address 192.168.0.38. So the bridge does not route traffic from your vlan into the local network which is what we need. The guests can talk to each other but they can not talk to the outside world.

Then I ping the virtual guests from my Mac. Here you see the host can talk to them. So the network setup of the Hacking Lab is correct.

patrick % ping 192.168.128.5
PING 192.168.128.5 (192.168.128.5): 56 data bytes
64 bytes from 192.168.128.5: icmp_seq=0 ttl=64 time=0.979 ms
64 bytes from 192.168.128.5: icmp_seq=1 ttl=64 time=0.568 ms
64 bytes from 192.168.128.5: icmp_seq=2 ttl=64 time=0.811 ms
64 bytes from 192.168.128.5: icmp_seq=3 ttl=64 time=0.718 ms
^C

patrick % ping 192.168.128.6
PING 192.168.128.6 (192.168.128.6): 56 data bytes
64 bytes from 192.168.128.6: icmp_seq=0 ttl=64 time=2.663 ms
64 bytes from 192.168.128.6: icmp_seq=1 ttl=64 time=2.298 ms
64 bytes from 192.168.128.6: icmp_seq=2 ttl=64 time=2.330 ms
64 bytes from 192.168.128.6: icmp_seq=3 ttl=64 time=2.310 ms
64 bytes from 192.168.128.6: icmp_seq=4 ttl=64 time=2.318 ms
^C

patrick %

System Python is installed in /usr/bin directory which is owned by root. This mean all files and symlinks in /usr/bin are system owned which means managed by the system and not managed by the user. In /usr/bin you find python symlinks pointing to the Python binaries in /System/Library/Frameworks/Python.framework/Versions/2.7/bin directory.

patrick@PatrickMBNeu ~ % ls -l /usr/bin/python* 

lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

....

patrick@PatrickMBNeu ~ %

The symlinks python, python2 and python2.7 in /usr/bin each point to the same binary in /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7. So when you type one of these as shell command in your shell terminal the binary in /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 will be executed.

patrick@PatrickMBNeu ~ % python --version
Python 2.7.16

patrick@PatrickMBNeu ~ % python2 --version
Python 2.7.16

patrick@PatrickMBNeu ~ % python2.7 --version
Python 2.7.16

patrick@PatrickMBNeu

Type in the command without using the path like /usr/bin/python works because /usr/bin is configured in the shell PATH variable.

patrick@PatrickMBNeu ~ % echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

patrick@PatrickMBNeu ~ %  

The shell lookup for a command in the paths defined in the PATH variable and starts on the left side of the PATH. So python, python2 and python2.7 do not exist in /usr/local/bin but in /usr/bin.

When you came from Catalina and upgraded to BigSur you realized that System Python has not been upgraded from Python 2 to Python 3 for example. So obviously BigSur still relies on Python 2 which is ok for your Mac but maybe not for you as a Developer. You might need newer or different versions of Python on your system. You must install this them separately.

Many people use Homebrew to install software packages on their MacOS. Homebrew can also be used to install a Python version in addition to the System Python. You can check if Python is installed via Homebrew on your system using the command brew info <formula>.

patrick@PatrickMBNeu ~ % brew info python
python@3.9: stable 3.9.7 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@3.9/3.9.7_1 (3,191 files, 56MB) *
  Poured from bottle on 2021-10-15 at 07:09:27
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.9.rb
License: Python-2.0
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, mpdecimal ✔, openssl@1.1 ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.9/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.9/site-packages

tkinter is no longer included with this formula, but it is available separately:
  brew install python-tk@3.9

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 717,857 (30 days), 1,988,212 (90 days), 8,421,529 (365 days)
install-on-request: 322,364 (30 days), 784,390 (90 days), 2,811,664 (365 days)
build-error: 0 (30 days)

patrick@PatrickMBNeu ~ %

Here on my system Python has been installed with Homebrew in the directory /usr/local/bin which is owned by the user. That means all files and symlinks in this directory can be managed by the user. In /usr/local/bin you find the python symlinks python3 and python3.9 pointing to further symlinks in the Homebrew Cellar in /usr/local/Cellar.

patrick@PatrickMBNeu ~ % ls -l /usr/local/bin/python*

lrwxr-xr-x  1 patrick  admin  40 15 Okt 07:09 /usr/local/bin/python3 -> ../Cellar/python@3.9/3.9.7_1/bin/python3

lrwxr-xr-x  1 patrick  admin  42 15 Okt 07:09 /usr/local/bin/python3.9 -> ../Cellar/python@3.9/3.9.7_1/bin/python3.9

...

patrick@PatrickMBNeu ~ %

The python3 symlink in the Homebrew Cellar point again to another python3 symlink in /usr/local/Frameworks/Python.framework/Versions/3.9/bin and this symlink point finally to python3.9 binary /usr/local/Frameworks/Python.framework/Versions/3.9/bin. The python3.9 symlink from the Homebrew Cellar point directly to the python3.9 binary in /usr/local/Frameworks/Python.framework/Versions/3.9/bin.

patrick@PatrickMBNeu ~ % ls -l /usr/local/Frameworks/Python.framework/Versions/3.9/bin
total 136

lrwxr-xr-x  1 patrick  admin      9 30 Aug 21:19 python3 -> python3.9

-rwxr-xr-x  1 patrick  admin  50472 15 Okt 07:09 python3.9

...

patrick@PatrickMBNeu ~ % 

So If you enter python3 or python3.9 as a command in the terminal the shell lookup these symlinks in directory /usr/local/bin and find a match. In any case you always end up with the python3.9 binary in /usr/local/Frameworks/Python.framework/Versions/3.9/bin and execute the same binary.

patrick@PatrickMBNeu ~ % python3 --version
Python 3.9.7

patrick@PatrickMBNeu ~ % python3.9 --version
Python 3.9.7

patrick@PatrickMBNeu ~ %

It is not a problem to install other Python versions with Homebrew in addition to the System Python, but the Homebrew ecosystem has a special feature: dependencies. Python versions that are installed in the Homebrew perimeter can also represent a dependency on other applications. node, for example, needs Python 3.9. If we uninstall Python 3.9, we have a problem with node. Therefore, Python can never be seen completely isolated in the Homebrew ecosystem and this is exactly the problem we solve with pyenv.

patrick@PatrickMBNeu ~ % brew info node   
                                                 
node: stable 16.11.1 (bottled), HEAD
Platform built on V8 to build network applications
https://nodejs.org/
/usr/local/Cellar/node/16.11.1 (1,980 files, 45.4MB) *
  Poured from bottle on 2021-10-15 at 07:09:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/node.rb
License: MIT
==> Dependencies
Build: pkg-config ✔
Required: brotli ✔, c-ares ✔, icu4c ✔, libnghttp2 ✔, libuv ✔, openssl@1.1 ✔, python@3.9 ✔
==> Options
--HEAD
	Install HEAD version
==> Analytics
install: 501,912 (30 days), 1,345,537 (90 days), 4,731,822 (365 days)
install-on-request: 404,509 (30 days), 1,070,393 (90 days), 3,650,272 (365 days)
build-error: 0 (30 days)

patrick@PatrickMBNeu ~ %

pyenv create a complete isolated environment for your Python versions. You are able to install, use and uninstall Python versions and you can switch between them. pyenv can be easily installed with Homebrew. pyenv is therefore installed in the /usr/local/bin directory.

patrick@PatrickMBNeu ~ % which pyenv
/usr/local/bin/pyenv

patrick@PatrickMBNeu ~ % pyenv --version
pyenv 2.1.0

patrick@PatrickMBNeu ~ % 

We must adapt the PATH variable for the standard shell. Here in my example this is configured in the .zshrc file in my Home-Directory.

patrick@PatrickMBNeu ~ % cat .zshrc

# ~/.pyenv provides Python before others of the same name
export PYENV_ROOT=$(pyenv root)
export PATH="$PYENV_ROOT/shims:$PATH"

patrick@PatrickMBNeu ~ %

You check the versions managed by pyenv with the following command. As you see pyenv realize the System Python. With the command pyenv install 3.9.7 you install Python 3.9.7 in your pyenv environment. You can switch to your desired Python version using the command pyenv global <version> .

When you then run the command python you are directly connected to global python version you selected.

patrick@PatrickMBNeu ~ % pyenv versions 
*  system

patrick@PatrickMBNeu ~ % pyenv install 3.9.7

.....

patrick@PatrickMBNeu ~ % pyenv versions 
* system
  3.9.7 (set by /Users/patrick/.pyenv/version)

patrick@PatrickMBNeu ~ % pyenv global 3.9.7

patrick@PatrickMBNeu ~ % pyenv versions 
  system
* 3.9.7 (set by /Users/patrick/.pyenv/version)

patrick@PatrickMBNeu ~ % python
Python 3.9.7 (default, Oct 19 2021, 16:02:07) 
[Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

patrick@PatrickMBNeu ~ % pyenv global system

patrick@PatrickMBNeu ~ % pyenv versions     
* system (set by /Users/patrick/.pyenv/version)
  3.9.7

patrick@PatrickMBNeu ~ % python             

WARNING: Python 2.7 is not recommended. 
This version is included in macOS for compatibility with legacy software. 
Future versions of macOS will not include Python 2.7. 
Instead, it is recommended that you transition to using 'python3' from within Terminal.

Python 2.7.16 (default, Aug 30 2021, 14:43:11) 
[GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

patrick@PatrickMBNeu ~ %

Finally: To avoid a warning message from brew when you run brew doctor you must create a brew alias in your .zshrc file.

patrick@PatrickMBNeu ~ % cat .zshrc

alias brew='PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin brew'

# ~/.pyenv provides Python before others of the same name
export PYENV_ROOT=$(pyenv root)
export PATH="$PYENV_ROOT/shims:$PATH"

patrick@PatrickMBNeu ~ %

Patricks-Macbook Pro:~ patrick$ sudo cat /private/etc/sudoers
Password:
patrick is not in the sudoers file.  This incident will be reported.

Add a Standard User to sudoers

To be able to run any command as root I must add the Standard-User to my /private/etc/sudoers file or I must put the Standard-User patrick into the sudo Group.

To get some information I list all Users on the system using the dscl command (Directory Service Command Line Tool). Here I find both of my created Users the Admin-User AdminPR and the Standard-User patrick. With the id command I can check the group memberships. The User AdminPR is member of the admin Group and patrick ist not in the admin Group.

Patricks-Macbook Pro:~ patrick$ dscl . -list /Users
.....
_amavisd
_analyticsd
_appinstalld
_appleevents
_applepay
....
_wwwproxy
_xserverdocs
AdminPR
daemon
nobody
patrick
root

Patricks-Macbook Pro:~ patrick$ id -nG AdminPR
staff com.apple.sharepoint.group.1 everyone localaccounts _appserverusr admin _appserveradm _lpadmin com.apple.sharepoint.group.2 _appstore _lpoperator _developer _analyticsusers com.apple.access_ftp com.apple.access_screensharing com.apple.access_ssh com.apple.access_remote_ae

Patricks-Macbook Pro:~ patrick$ id -nG patrick
staff com.apple.sharepoint.group.2 everyone localaccounts access_bpf com.apple.sharepoint.group.1 _lpoperator

Patricks-Macbook Pro:~ patrick$ 

In the default User sudo specification any Users belonging to the admin Group are enabled to use sudo and can run any command as root. Therefore I must work with the admin user to make any changes to /private/etc/sudoers.

To check this I switch to my Admin-User using the su command in the Terminal and then I print the content of my /private/etc/sudoers file to the console using the cat command. In the User specification part of /private/etc/sudoers file it is defined that members of the admin Group can run all commands as root using sudo ( %admin … ).

Patricks-Macbook Pro:~ patrick$ su AdminPR
Password:

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.

bash-3.2$ sudo cat /etc/private/sudoers
Password:

.....

##
# User specification
##

# root and users in group wheel can run anything on any machine as any user
root        ALL = (ALL) ALL
%admin      ALL = (ALL) ALL

....

bash-3.2$ exit
Patricks-Macbook Pro:~ patrick$

To enable my Standard-User patrick to run root with sudo I basically have 2 options:

• I add the Standard-User patrick to the admin Group or
• I add the user patrick to the User specification part in /private/etc/sudoers file.
  

Both tasks can currently only be performed using the Admin-User as this is the only user that can run sudo commands.

I decide to add user patrick to sudoers. So I su into the AdminPR account again and add the Standard-User patrick to /private/etc/sudoers file in the User specification part as follows. I use my favorite editor nano for this.

Patricks-Macbook Pro:~ patrick$ su AdminPR
Password:

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.

bash-3.2$ sudo nano /etc/private/sudoers
Password:

##
# User specification
##

# root and users in group wheel can run anything on any machine as any user
root        ALL = (ALL) ALL
%admin      ALL = (ALL) ALL
patrick     ALL = (ALL) ALL

bash-3.2$ exit
Patricks-Macbook Pro:~ patrick$ 

Finally I successfully check that the configuration works.

Patricks-Macbook Pro:~ patrick$ sudo cat /private/etc/sudoers
Password:

#
# Sample /etc/sudoers file.
#
# This file MUST be edited with the 'visudo' command as root.
#
# See the sudoers man page for the details on how to write a sudoers file.

##
# Override built-in defaults
##
Defaults    env_reset
Defaults    env_keep += "BLOCKSIZE"
Defaults    env_keep += "COLORFGBG COLORTERM"
Defaults    env_keep += "__CF_USER_TEXT_ENCODING"
Defaults    env_keep += "CHARSET LANG LANGUAGE LC_ALL LC_COLLATE LC_CTYPE"
Defaults    env_keep += "LC_MESSAGES LC_MONETARY LC_NUMERIC LC_TIME"
Defaults    env_keep += "LINES COLUMNS"
Defaults    env_keep += "LSCOLORS"
Defaults    env_keep += "SSH_AUTH_SOCK"
Defaults    env_keep += "TZ"
Defaults    env_keep += "DISPLAY XAUTHORIZATION XAUTHORITY"
Defaults    env_keep += "EDITOR VISUAL"
Defaults    env_keep += "HOME MAIL"

Defaults    lecture_file = "/etc/sudo_lecture"

##
# User alias specification
##
# User_Alias    FULLTIMERS = millert, mikef, dowdy

##
# Runas alias specification
##
# Runas_Alias    OP = root, operator

##
# Host alias specification
##
# Host_Alias    CUNETS = 128.138.0.0/255.255.0.0
# Host_Alias    CSNETS = 128.138.243.0, 128.138.204.0/24, 128.138.242.0
# Host_Alias    SERVERS = master, mail, www, ns
# Host_Alias    CDROM = orion, perseus, hercules

##
# Cmnd alias specification
##
# Cmnd_Alias    PAGERS = /usr/bin/more, /usr/bin/pg, /usr/bin/less

##
# User specification
##

# root and users in group wheel can run anything on any machine as any user
root        ALL = (ALL) ALL
%admin        ALL = (ALL) ALL
patrick     ALL = (ALL) ALL

## Read drop-in files from /private/etc/sudoers.d
## (the '#' here does not indicate a comment)
#includedir /private/etc/sudoers.d

Patricks-Macbook Pro:~ patrick$ 

Homebrew Installation and Uninstallation

First check if Homebrew is (already) installed.

Macbook Pro:~ user$ brew --version
Homebrew 3.0.5
Homebrew/homebrew-core (git revision 4324a52c3e; last commit 2021-03-14)
Homebrew/homebrew-cask (git revision 16c4718ac7; last commit 2021-03-14)
Macbook Pro:~ user$

The above output show Homebrew 3.0.5.

In case Homebrew is not installed first check the requirements for Mac OS on the Homebrew Installation site. Here you find that Homebrew requires Command Line Tools (CLT) for Xcode. Therefore I first check if Command Line Tools are installed on my Mac.

Macbook Pro:~ user$ xcode-select -p
/Library/Developer/CommandLineTools
Macbook Pro:~ patrick$

The above command return that Command Line Tools for xcode (CLT) are installed on my Mac and show the path where the Command Line Tools are installed.

In case Command Line Tools for xcode are not installed run the following command before you start with the Homebrew installation.

:$ xcode-select --install

Then go to the Homebrew Website and check the actual installation instructions. At the time of writing this document you run the following script to install Homebrew.

:$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

After the installation I check the Homebrew version and run brew doctor.

Macbook Pro:~ user$ brew --version
Homebrew 3.0.5
Homebrew/homebrew-core (git revision 4324a52c3e; last commit 2021-03-14)
Homebrew/homebrew-cask (git revision 16c4718ac7; last commit 2021-03-14)

Macbook Pro:~ user$ brew doctor
Your system is ready to brew.
Macbook Pro:~ user$

The shell output show that Homebrew version 3.0.5 is installed on the system and that I am ready to brew which means that I am now ready to install software with Homebrew.

To uninstall Homebrew you run the following script.

:$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall.sh)"

Update Command Line Tools for xcode

Homebrew need Command Line Tools for xcode but sometimes Command Line Tools can be outdated. When you run brew doctor and see the following message that Command Line Tools are outdated you must ensure that the latest version of Command Line Tools is installed on your Mac.

Patricks-Macbook Pro:~ patrick$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: A newer Command Line Tools release is available.
Update them from Software Update in System Preferences or run:
  softwareupdate --all --install --force

If that doesn't show you any updates, run:
  sudo rm -rf /Library/Developer/CommandLineTools
  sudo xcode-select --install

Alternatively, manually download them from:
  https://developer.apple.com/download/more/.

Patricks-Macbook Pro:~ patrick$ ls -l /Library/Developer
total 0
drwxr-xr-x  5 root  admin  160 15 Sep 07:18 CommandLineTools
drwxr-xr-x  4 root  admin  128  9 Okt  2019 PrivateFrameworks

Patricks-Macbook Pro:~ patrick$ 

To update Command Line Tools for xcode they must first be uninstalled and then re- installed on the system. Pls. find additional information how to uninstall Command Line Tools for xcode and check the related Apple Developer Docs.

Patricks-Macbook Pro:~ patrick$ sudo rm -rf /Library/Developer/CommandLineTools

Patricks-Macbook Pro:~ patrick$ ls -l /Library/Developer
total 0
drwxr-xr-x  4 root  admin  128  9 Okt  2019 PrivateFrameworks

Patricks-Macbook Pro:~ patrick$ sudo xcode-select --install
xcode-select: note: install requested for command line developer tools

Patricks-Macbook Pro:~ patrick$ ls -l /Library/Developer
total 0
drwxr-xr-x  5 root  wheel  160 28 Feb 06:35 CommandLineTools
drwxr-xr-x  4 root  admin  128  9 Okt  2019 PrivateFrameworks

Patricks-Macbook Pro:~ patrick$ xcode-select -p
/Library/Developer/CommandLineTools

Patricks-Macbook Pro:~ patrick$

Homebrew Software Repositories and Tap(s)

With Homebrew package manager you have access to a wide range of software that can be installed on your system. After Homebrew has been installed on your system you find Homebrew under /usr/local/Homebrew on your system.

Homebrew differentiates between Core Software Packages and Cask Software or so-called Casks. All Software Packages are available in GitHub Repositories.

Homebrew Core Software Packages are free software developed from developers around the world to be installed on Mac OS platforms using Homebrew. Core software will be installed based on so called formulae. A formula contain the required information for the installation such as i.e. from where the installation files should be loaded, which dependencies exist and how the installation should be performed on the different Mac platforms. All formulae for Core Software Packages have been loaded from Homebrew’s GitHub Core-Repository Homebrew/homebrew-core into the core Tap on your machine. This Tap can be found on you machine in /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core

Homebrew Cask Software Packages are native Mac OS Apps like Google Chrome, Firefox or other Mac OS Apps that can be installed using the Homebrew Cask extension of the Homebrew Package Manager instead of installing the software via the Apple App Store or drag and drop a dmg file into the Application folder. How to use Homebrew Cask can be read on the Homebrew Cask GitHub site. Casks will be loaded from the Homebrew’s GitHub Cask-Repository Homebrew/homebrew-cask into the cask Tap on your machine. This Tap can be found on you machine in /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask.

It is also possible to install Software that is not available on Homebrew’s GitHub Core-Repository or Homebrew’s GitHub Cask-Repository. In this case you must first tap the GitHub Repository to let Homebrew know, from where it should load the formula or cask to install the software package.

For example the formula for MongoDB has been removed from the GitHub Core-Repository. But fortunately the MongoDB Team is maintaining a custom GitHub-Repository mongodb/homebrew-brew from where you can tap the formulae on your local machine.

:$ brew tap mongodb/homebrew-brew

On GitHub, a Homebrew repository must be named homebrew- in order to use the form of the brew tap with only one argument. When you use brew tap you can leave out the homebrew- prefix.

:$ brew tap mongodb/brew

This brew tap <username>/brew be used as a shortcut for the long version brew tap <username>/homebrew-brew. Homebrew will automatically add back the homebrew– prefix whenever it’s necessary.

The command brew tap without any arguments lists the GitHub repositories that are currently tapped on your local machine and available to install software from these sources.

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services
mongodb/brew

Where are Packages on your System

Homebrew create symlinks for each installed package in /usr/local/bin and usually point to the binary in /usr/local/Cellar/<package_name>/<version>/bin. This is the standard. Depending on the definitions in the formula of a certain package the binaries can be also linked somewhere else on your system.

Macbook Pro:~ user$ ls -l /usr/local/bin
total 0
lrwxr-xr-x  1 user  admin  28 11 Nov  2016 brew -> /usr/local/Homebrew/bin/brew
lrwxr-xr-x  1 user  admin  34  1 Nov  2017 ccmake -> ../Cellar/cmake/3.9.4_1/bin/ccmake
lrwxr-xr-x  1 user  admin  33  1 Nov  2017 cmake -> ../Cellar/cmake/3.9.4_1/bin/cmake
lrwxr-xr-x  1 user  admin  39  1 Nov  2017 cmakexbuild -> ../Cellar/cmake/3.9.4_1/bin/cmakexbuild
lrwxr-xr-x  1 user  admin  33  1 Nov  2017 cpack -> ../Cellar/cmake/3.9.4_1/bin/cpack
lrwxr-xr-x  1 user  admin  41 16 Dez  2016 cryptest.exe -> ../Cellar/cryptopp/5.6.5/bin/cryptest.exe
lrwxr-xr-x  1 user  admin  33  1 Nov  2017 ctest -> ../Cellar/cmake/3.9.4_1/bin/ctest
lrwxr-xr-x  1 user  admin  32  1 Nov  2017 evm -> ../Cellar/ethereum/1.7.2/bin/evm

.......

The Homebrew binary link named brew point to /usr/local/Homebrew/bin/brew. Any other binary link point into the Cellar in /usr/local/Cellar/<package_name>/<version>/bin. The Cellar is the place on your system where you usually find Homebrew installed software packages. As you see on my system I have installed a couple of software packages like boost, cmake and node.

Macbook Pro:~ user$ ls -l /usr/local/Cellar
total 0
drwxr-xr-x  4 user  admin  128  1 Nov  2017 boost
drwxr-xr-x  4 user  admin  128  1 Nov  2017 cmake
drwxr-xr-x  3 user  admin   96 16 Dez  2016 cryptopp
drwxr-xr-x  5 user  admin  160  1 Nov  2017 ethereum
drwxr-xr-x  4 user  admin  128  1 Nov  2017 gmp
drwxr-xr-x  5 user  admin  160  1 Nov  2017 go
drwxr-xr-x  4 user  staff  128  6 Okt 07:16 hugo
drwxr-xr-x  3 user  staff   96  3 Aug 06:48 icu4c
drwxr-xr-x  4 user  admin  128  1 Nov  2017 jsoncpp
drwxr-xr-x  4 user  staff  128  9 Okt 07:39 node
drwxr-xr-x  3 user  admin   96 12 Nov  2017 openssl
drwxr-xr-x  4 user  admin  128  1 Nov  2017 solidity

Cask software is installed in the following directory.

Patricks-MBP:~ patrick$ ls -l /usr/local/Caskroom
total 0
drwxr-xr-x  4 patrick  admin  128 11 Jun 09:00 opera

Homebrew Cask create a symlink in /usr/local/Caskroom/<package_name>/<version>/<symlink> and point it to the app in the Applications directory where the app will be copied.

Patricks-MBP:~ patrick$ ls -l /usr/local/Caskroom/opera/68.0.3618.165
total 0
lrwxr-xr-x  1 patrick  admin  23 11 Jun 09:00 Opera.app -> /Applications/Opera.app

Install, Uninstall and Update Software Packages

To install a package using the brew install command.

:$ brew install <formula>

To uninstall a package using the brew uninstall command.

:$ brew uninstall <formula>

To update and upgrade all packages with Homebrew use the following commands.

:$ brew update
:$ brew upgrade

You can also update and upgrade only a certain package.

:$ brew update <formula>
:$ brew upgrade <formula>

The update command load the new formulae and the upgrade command installs the newer software versions in the Cellar. Depending on the amount of packages you have installed this might take some time.

When you install Cask Software the app must be copied into your Applications Directory. Here you need to provide your user password.

You can list the Casks installed on your Mac.

patrick@PatrickMBNeu ~ % brew list --casks
opera
patrick@PatrickMBNeu ~ % 

You can search for packages

patrick@PatrickMBNeu ~ % brew search opera
==> Formulae
operator-sdk
==> Casks
opera ✔                                                    opera-mobile-emulator                                      operator
opera-gx                                                   opera-neon                                                 homebrew/cask-versions/opera-beta
opera-mail                                                 operadriver                                                homebrew/cask-versions/opera-developer
patrick@PatrickMBNeu ~ %

You install a cask package using the brew install command.

:$ brew install <cask-package>

You uninstall a cask package using the brew uninstall command.

patrick@PatrickMBNeu ~ % brew uninstall opera     
==> Uninstalling Cask opera
==> Purging files for version 68.0.3618.165 of Cask opera
patrick@PatrickMBNeu ~ % brew list --casks        
patrick@PatrickMBNeu ~ % 

But first of all I would like to describe my initial situation. I think some Mac users have python installed on their mac by themselves in one way or another and are surprised when checking the current version with python -V or when checking which python binary is currently being executed with the command which python. I installed python 2 and 3 with Homebrew and after executing these commands I was surprised and found that on my system different python versions were running than I expected.

I found that there must be a problem when you install different Python versions on a Mac and I was looking for an easy way to switch between these versions. Switching between the versions is necessary because when you are a developer you might have the need to run your code on a certain version. So it has to be easy to install different python versions on your Mac and to switch between them as needed. There are of course several ways of dealing with this matter. I decided to use the version manager Pyenv. But more on that later. First of all, back to my initial situation.

Initial Situation

It all started with a Homebrew error message after my update to Mac OS Big Sur (Mac OS 11). I had python 2 and 3 installed on my system with Homebrew. I had carried out these installations under Catalina. The brew doctor command gave me the error message saying I had installed kegs without a formula and I should uninstall python@2 because it is an outdated and no longer supported python version. What ?

My first check was on the /usr/local/bin directory because I wanted to know whether the python binaries had been linked correctly from Homebrew.

patrick@PatrickMBNeu ~ % ls -l /usr/local/bin/python*
lrwxr-xr-x  1 patrick  admin   32  6 Mär 10:23 python -> ../Cellar/python@2/2.7.17/bin/python
lrwxr-xr-x  1 patrick  admin   32  6 Mär 10:23 python2 -> ../Cellar/python@2/2.7.17/bin/python2
lrwxr-xr-x  1 patrick  admin   32  6 Mär 10:23 python2.7 -> ../Cellar/python@2/2.7.17/bin/python2.7

Here it seems to me that only python 2 has been installed by Homebrew and the binary links all refer to the python@2 directory in the Homebrew Cellar. My second check was on the Hombrew Cellar directory because I wanted to know whether there are Formulars of the two python versions available.

patrick@PatrickMBNeu ~ % ls -l /usr/local/Cellar/python*
drwxr-xr-x  3 patrick  staff  96 26 Feb 18:46 python@2
drwxr-xr-x  3 patrick  staff  96 26 Feb 18:46 python@3.8

This made it clear: Python 2 is correctly installed and linked in the Cellar (but out of date) but Python 3 was obviously not installed correctly (no binary links in /usr/local/bin). I understood that this must be the message from Homebrews error message. But then I was curious what the version check would tell me.

patrick@PatrickMBNeu ~ % which python
/usr/local/bin/python
patrick@PatrickMBNeu ~ % python --version
2.7.17
patrick@PatrickMBNeu ~ % which python3
/usr/bin/python3
patrick@PatrickMBNeu ~ % python3 --version
3.8.2

The command which python refers to the python 2 version installed with Homebrew in /usr/local/bin but what the hell is in /usr/bin ?

I googled around a bit and finally I found out that it must be the python version that comes with Mac OS. So the command which python3 refers to the python 3 version pre-installed on the Mac in /usr/bin. And I learned that it must be avoided in any case to remove these so-called System Python Versions from the system. Apple install in /usr/bin while Homebrew install in /usr/local/bin and /usr/local/Cellar. So stay away from everything that is installed in /usr/bin with regard to Python.

Then I wanted to understand what Apple installed in /usr/bin.

patrick@PatrickMBNeu ~ % ls -l /usr/bin/python*
lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel      75  1 Jan  2020 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
-rwxr-xr-x  1 root  wheel  137552  1 Jan  2020 /usr/bin/python3
patrick@PatrickMBNeu ~ % 

Apple apparently installs Python 2 and Python 3. But the command which python clearly tells me (see above) that the Python 2 version of Homebrew from /usr/local/bin is currently active in some way, while the command which python3 apparently refers to the System Python 3 version in /usr/bin.

How do I get the mess cleaned up ?

I decided to uninstall the python 2 version of Homebrew normally with the command brew uninstall python@2. And I also removed manually the /usr/local/Cellar/python@3.8 directory. Then I checked the python versions again.

patrick@PatrickMBNeu ~ % which python
/usr/bin/python
patrick@PatrickMBNeu ~ % python --version
2.7.16
patrick@PatrickMBNeu ~ % which python3
/usr/bin/python3
patrick@PatrickMBNeu ~ % python3 --version
3.8.2

I was satisfied with the result. Both System Pythons in /usr/bin are currently active on my system and this is exactly what I wanted to achieve with my clean up. I can now think about how I would like to manage other, different python versions on my system. And this is exactly where I come back to Pyenv.

Managing Python on Mac OS with Pyenv

First I install Pyenv with Homebrew.

patrick@PatrickMBNeu ~ % brew update
patrick@PatrickMBNeu ~ % brew install pyenv

Then I create a ~/.zshrc file in my home directory and add the command pyenv init at the end of the ~/.zshrc configuration file to enable pyenv shims in my PATH variable.

patrick@PatrickMBNeu ~ % echo -e 'eval "$(pyenv init -)" ' >> ~/.zshrc
patrick@PatrickMBNeu ~ % cat .zshrc

eval "$(pyenv init -)"

patrick@PatrickMBNeu ~ % 

It is very important to make sure that the command eval "$(pyenv init -)" is placed at the end of the shell configuration because it manipulates the PATH environment variable during the shell initialization.

Then I close the terminal and restart it again.

With the command pyenv versions you can check the existing python versions on your Mac. After my cleanup the only version on my Mac is the System Python and marked with system.

patrick@PatrickMBNeu ~ % pyenv versions     
* system (set by /Users/patrick/.pyenv/version)
patrick@PatrickMBNeu ~ % python --version
Python 2.7.16
patrick@PatrickMBNeu ~ % python3 --version
Python 3.8.2
patrick@PatrickMBNeu ~ %

When you check the version of the current System Python with the command python --``version and python3 --``version you see that I still run the System Python Versions 2.7.16 and 3.8.2. Thats what I expected.

With Pyenv you can also install Python using the pyenv install command. With the command pyenv install --list you can list all available Python version that can be installed. This would be a long list but you can pipe it into grep to restrict the displayed list a little. Here I restrict the list to all available Python 3.8.* and 3.9.* versions.

patrick@PatrickMBNeu ~ % pyenv install --list | grep " 3\.[89]"  
  3.8.0
  3.8-dev
  3.8.1
  3.8.2
  3.8.3
  3.8.4
  3.8.5
  3.8.6
  3.8.7
  3.9.0
  3.9-dev
  3.9.1
patrick@PatrickMBNeu ~ % 

Then I install Python 3.9.1 which is the latest Python available at the time of writing this article.

patrick@PatrickMBNeu ~ % pyenv install 3.9.1

Then I check the existing pythons again with pyenv versions and I can already see that the 3.9.1 version has been added. Now comes the real advantage of pyenv. With the command pyenv global I set the 3.9.1 version as the currently active one.

patrick@PatrickMBNeu ~ % pyenv versions     
* system (set by /Users/patrick/.pyenv/version)
  3.9.1
patrick@PatrickMBNeu ~ % pyenv global 3.9.1 
patrick@PatrickMBNeu ~ % pyenv versions   
  system
* 3.9.1 (set by /Users/patrick/.pyenv/version)
patrick@PatrickMBNeu ~ % python --version 
Python 3.9.1
patrick@PatrickMBNeu ~ % python3 --version 
Python 3.9.1
patrick@PatrickMBNeu ~ % python
Python 3.9.1 (default, Mar  8 2021, 18:50:54) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
patrick@PatrickMBNeu ~ %  

I check again if the default python is now set to 3.9.1 using the command python --``version. The output in the terminal shows the active version marked with a star. This is Python 3.9.1 and exactly what I expected. Then I login into the interactive python command interpreter and this also confirmed that it is currently the 3.9.1 version.

So I manage my Pythons on my Mac OS Big Sur from now on with Pyenv.

Make Pyenv working perfectly with Homebrew

When I finished installing Pyenv everything worked fine until I tried to use the brew doctor command to check my Homebrew installations.

patrick@PatrickMBNeu ~ % brew doctor
Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:

    /Users/patrick/.pyenv/shims/python-config
    /Users/patrick/.pyenv/shims/python3-config
    /Users/patrick/.pyenv/shims/python3.9-config

patrick@PatrickMBNeu ~ %

I googled a little and I found out this. Very interesting Article about the PATH settings I enforced in my ~/.zshrc file when I installed Pyenv (see above).

Then I checked my environment variables with the command env.

patrick@PatrickMBNeu ~ % env
TMPDIR=/var/folders/zy/3s2n4mh502z3320837q80qdm0000gp/T/
__CFBundleIdentifier=com.apple.Terminal
XPC_FLAGS=0x0
TERM=xterm-256color
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.ZIRHZWPkYD/Listeners
XPC_SERVICE_NAME=0
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=440
TERM_SESSION_ID=53DE5B52-55E2-4D61-A4E4-7F47D5C401AC
SHELL=/bin/zsh
HOME=/Users/patrick
LOGNAME=patrick
USER=patrick
PATH=/Users/patrick/.pyenv/shims:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/MacGPG2/bin
SHLVL=1
PWD=/Users/patrick
OLDPWD=/Users/patrick
PYENV_SHELL=zsh
LANG=de_DE.UTF-8
_=/usr/bin/env
patrick@PatrickMBNeu ~ % 

The problem seems to be the PATH which is set by the entry eval "$ (pyenv init -)" in my ~/.zshrc file.

patrick@PatrickMBNeu ~ % cat .zshrc
eval "$(pyenv init -)"

patrick@PatrickMBNeu ~ %

This entry leads to that /Users/patrick/.pyenv/shims is set at the beginning of the PATH variable when zsh is executed.

When you look at the complete PATH you see that the command brew doctor can be executed without specifying /usr/local/bin/brew because the required path specification /usr/local/bin is also available but further back. So brew reads first the path /Users/patrick/.pyenv/shims and only then the path /usr/local/bin and exactly there in /Users/patrick/.pyenv/shims are the *-config files used from Pyenv. So it seems to be that brew has problems with these config files.

Since these config files seems to confuse brew in some way, the best would be if brew could somehow ignore these files. This is possible if we define an alias environment variable. We can define brew as an alias and then instruct zsh that in the event that brew is called a specific PATH applies namely the PATH without the shims directory. For this we have to adapt the ~/.zshrc as follows.

patrick@PatrickMBNeu ~ % cat .zshrc
alias brew='PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin brew'
eval "$(pyenv init -)"
patrick@PatrickMBNeu ~ %

After restarting the terminal I was able to run the brew doctor command without this error message. The problem is solved.

patrick@PatrickMBNeu ~ % brew update 
Already up-to-date.
patrick@PatrickMBNeu ~ % brew doctor     
Your system is ready to brew.
patrick@PatrickMBNeu ~ % 

How to setup such a production environment will be shown in the first chapter of this documentation.

The express app itself also contains some security features. The app contains a session based user authentication and HTTP headers which help to further secure the app. This is explained of the second chapter of this documentation.

Finally the express app should use the template engine PUG to render the HTML for us. This is describes of the third chapter of this documentation.

1. Setup the production environment

Installation of mongodb

The command brew tap without any arguments lists the GitHub repositories that are currently linked to your Homebrew installation.

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services

The formula mongodb has been removed from homebrew-core. But fortunately the MongoDB Team is maintaining a custom Homebrew tap on GitHub. Read the instructions in the README.md file.

Add the custom tap in the Mac OS terminal and install mongodb.

Patricks-MBP:~ patrick$ brew tap mongodb/brew

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services
mongodb/brew

Patricks-MBP:~ patrick$ brew install mongodb-community@4.2

After the installation the relevant paths are.

the configuration file (/usr/local/etc/mongod.conf)
the log directory path (/usr/local/var/log/mongodb)
the data directory path (/usr/local/var/mongodb)

Check the services with homebrew

brew services list

Name              Status  User    Plist
mongodb-community started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
  

Start and stop mongodb.

brew services start mongodb-community

brew services stop mongodb-community

Setup mongodb for the project

Setup an admin user

:# mongo

> use admin
switched to db admin

> db
admin

> db.createUser({ user: "adminUser", pwd: "adminpassword", roles: [{ role: "userAdminAnyDatabase", db: "admin" }, {"role" : "readWriteAnyDatabase", "db" : "admin"}] })

> db.auth("adminUser", "adminpassword")
1

> show users
{
    "_id" : "admin.adminUser",
    "userId" : UUID("5cbe2fc4-1e54-4c2d-89d1-317340429571"),
    "user" : "adminUser",
    "db" : "admin",
    "roles" : [
        {
            "role" : "userAdminAnyDatabase",
            "db" : "admin"
        },
        {
            "role" : "readWriteAnyDatabase",
            "db" : "admin"
        }
    ],
    "mechanisms" : [
        "SCRAM-SHA-1",
        "SCRAM-SHA-256"
    ]
}

> exit

Enable authentication with security: authorization: enabled

#> nano /usr/local/etc/mongod.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb
net:
  port: 27017
  bindIp: 127.0.0.1
security:
  authorization: enabled

Login and authenticate with admin

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b3e7f48a-a05c-4894-87db-996cb34eb1fb") }
MongoDB server version: 4.2.3

> show dbs
> db
test
> use admin
switched to db admin
> db
admin
> show dbs
> db.auth("adminUser", "adminpassword")
1
> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB

> 

If you login you dont see any databases when you call show dbs. The default database you are connected to is test.

Then you connect to admin database. For admin you setup the admin user with the roles userAdminAnyDatabase and readWriteAnyDatabase. With these permissions the admin user can manage users for any database and has read and write access to any database.

So wehen you logon to admin database with the admin user you are able to see all databases with show dbs.

Mongodb comes with 3 standard dbs pre installed:

• admin
• config
• local
  

Create a new database for our express-security app (authenticated as admin user – see above)

> use express-security
switched to db express-security
> db
express-security
> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
> 

The DB which you’ve created is not listed here. We need to insert at least one collection into it for displaying that database in the list.

> db
express-security

> db.createCollection("col_default")
{ "ok" : 1 }

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> exit

Create an owner user for express-security database using the admin user

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("79f79b63-9d08-489f-9e6c-bfc10d8cc09e") }
MongoDB server version: 4.2.3

> db
test

> show dbs

> use admin
switched to db admin

> db.auth("adminUser", "adminpassword")
1

> db
admin

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> use express-security
switched to db express-security

> db.createUser({ user: "owner_express-security", pwd: "passowrd", roles: [{ role: "dbOwner", db: "express-security" }] })
Successfully added user: {
	"user" : "owner_express-security",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "express-security"
		}
	]
}

> db
express-security

> show users
{
	"_id" : "express-security.owner_express-security",
	"userId" : UUID("7a0bafb2-d2ed-4d18-9aba-e2f15a503ec5"),
	"user" : "owner_express-security",
	"db" : "express-security",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "express-security"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

> exit 

Connection string to connect to express-security db using the owner_express-security user:

mongodb://owner_express-security:password@localhost/express-security

Installation of PM2

PM2 is a process manager for Node.js applications. It can daemonize applications to run them as a service in the background.

I install pm2 as a global npm package on my Mac.

Patricks-Macbook Pro:~ patrick$ npm install pm2 -g

Then navigate to your project directory.

Patricks-Macbook Pro:~ patrick$ cd Software/dev/node/articles/2020-05-15-express-security/express-security

Patricks-Macbook Pro:~ patrick$ ls -l 
total 112
drwxr-xr-x    5 patrick  staff    160 30 Mai 05:28 database
drwxr-xr-x  115 patrick  staff   3680 30 Mai 19:35 node_modules
-rw-r--r--    1 patrick  staff  34366 30 Mai 19:35 package-lock.json
-rw-r--r--    1 patrick  staff    339 30 Mai 19:35 package.json
-rw-r--r--@   1 patrick  staff  12343 30 Jun 05:00 secserver.js
drwxr-xr-x    3 patrick  staff     96 30 Mai 05:03 static

Patricks-Macbook Pro:express-security patrick$ 

Start your app using pm2

Patricks-Macbook Pro:express-security patrick$ pm2 start secserver.js

Patricks-Macbook Pro:~ patrick$ pm2 list
┌─────┬──────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name         │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0   │ secserver    │ default     │ 1.0.0   │ fork    │ 640      │ 16h    │ 0    │ online    │ 0%       │ 48.6mb   │ patrick  │ disabled │
└─────┴──────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

Patricks-Macbook Pro:~ patrick$ 

Other comands to control the process manager.

pm2 start secserver.js

pm2 start <id>

pm2 list

pm2 stop <id>

pm2 restart <id>

pm2 show <id>

Installation of nginx

nginx is an open source HTTP and an HTTP Reverse Proxy Server (also mail proxy and load balancer etc.). I install nginx on my Mac with Homebrew.

brew install nginx

You can list the brew services with the following command.

Patricks-MBP:digitaldocblog-V3 patrick$ brew services list

Name              Status  User    Plist
mongodb-community started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
nginx             started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.nginx.plist
Patricks-MBP:digitaldocblog-V3 patrick$

You can start and stop the brew services as follows.

brew install nginx

brew services start nginx

brew services stop nginx

Setup nginx with TLS/SSL

SSL/TLS works by using the combination of a public certificate and a private key.

The SSL key (private key) is kept secret on the server. It is used to encrypt content sent to clients.

The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

create private key

Patricks-MBP:express-security patrick$ cd /usr/local/etc/nginx

Patricks-MBP:nginx patrick$ ls -l
total 144
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  5 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  5 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types.default
-rw-r--r--  1 patrick  admin  3106 15 Mai 05:19 nginx.conf
-rw-r--r--  1 patrick  admin  2680  5 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params.default
drwxr-xr-x  3 patrick  admin    96 21 Jan 06:02 servers
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  5 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ mkdir ssl

Patricks-MBP:nginx patrick$ ls -l
total 152
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  5 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  5 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types.default
-rw-r--r--@ 1 patrick  admin   373 18 Mai 05:38 nginx.conf
-rw-r--r--  1 patrick  admin  2680  5 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--@ 1 patrick  admin  1390 17 Mai 05:19 nginx_old.conf
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params.default
drwxr-xr-x  5 patrick  admin   160 18 Mai 05:20 servers
drwxr-xr-x  4 patrick  admin   128 16 Mai 05:41 ssl
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  5 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ cd ssl

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl genrsa -out privateKey.pem 4096

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem

create certificate signing request (CSR)

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl req -new -key privateKey.pem -out csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  1740 16 Mai 05:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem

In case I would like to request an official certificate I must send this csr to the certificate authority. This authority would then create an authority signed certificate from the CSR and send it back to me.

This step is done by ourselves and this is the reason why we create a self signed certificate. This self signed certificate is not a official certificae and not trusted by any browser. It is not useful to use a self signed certificate in production because it produces error messages in the browsers. But for local development a self signed certificate is ok.

So create the self signed certificale. The csr file can then be removed.

create the self signed certificate

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in csr.pem -out selfsignedcertificate.pem -req -signkey privateKey.pem -days 365

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  1740 16 Mai 05:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 05:39 selfsignedcertificate.pem

Patricks-MBP:ssl patrick$ rm csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 05:39 selfsignedcertificate.pem
 

show certificate details

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in selfsignedcertificate.pem -text -noout

Configure nginx Servers with SSL

In our configuration we enforce ssl. Therefore we create a default Webserver listening on Port 80 with the server name servtest.rottlaender.lan.

Any request to servtest.rottlaender.lan:80 is redirected to my Reverse Proxy Server which is listening on servtest.rottlaender.lan:443.

The default Webserver is configured in /usr/local/etc/nginx/nginx.conf.

# /usr/local/etc/nginx/nginx.conf
# default Webserver

worker_processes  1;
error_log  /usr/local/etc/nginx/logs/error.log;

events {
    worker_connections  1024;
}

http {
    include       		mime.types;
    default_type  		application/octet-stream;
    sendfile        	on;
    keepalive_timeout  	65;

    access_log  /usr/local/etc/nginx/logs/access.log;

    # default Webserver redirect from port 80 to port 443 ssl
    
    server {
      	listen 80;
    	listen [::]:80;
    	server_name servtest.rottlaender.lan;
    	return 301 https://$host$request_uri;
    }
    
    include servers/*;
    
}

The Reverse Proxy Server is configured in /usr/local/etc/nginx/servers/reverse.

// /usr/local/etc/nginx/servers/reverse
// reverse Proxy Server

server {

    listen      443 ssl;
    server_name servtest.rottlaender.lan;

    ssl_certificate      ssl/selfsignedcertificate.pem;
    ssl_certificate_key  ssl/privateKey.pem;
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
       proxy_pass http://localhost:3300;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}

The server name servtest.rottlaender.lan is linked in /private/etc/hosts to the ip 192.168.178.20 which is the ip of my computer in my local network.

Patricks-MBP:digitaldocblog-V3 patrick$ cat /private/etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1			localhost
255.255.255.255		broadcasthost
::1             	localhost
192.168.178.20 		servtest.rottlaender.lan

2. Express Secure App (Security Features HTML version)

This is a very simple application but show the basic security features you should use when you run a node app in a production environment.

The app is a website with a simple layout and navigation.

The Home page contain static information and can be accessed by everyone.

On the register page, users can find a form to register. The user data entered here are saved in the database and the user is logged in at the same time. Known users can log in with their email and password after successful registration on the login page. The login and register page can only be accessed if the user is not logged in. If a user is logged in and tries to access the login or register, he will be redirected to the dashboard page.

The dashboard is a personalized area of the website. This area can only be accessed if the user is logged in. If a user is not logged in, he will be redirected to the login page.

Logout is not really a page but a link that contains a logout function. Users who are logged in can log out using this link. Users who are not already logged in will be redirected to the login page.

Download the code from GitHub

Pls. go to my GitHub site and clone the code. Here you find a some inline documentation in the code. The details are explained in this chapter.

Create your app home directory express-security

My app home directory is different to the one that is available after you cloned the code from GitHub.

Patricks-MBP:2020-05-15-express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security

Patricks-MBP:2020-05-15-express-security patrick$ mv node-part-5-express-security-with-db-pug express-security

Patricks-MBP:2020-05-15-express-security patrick$ cd express-security

Patricks-MBP:express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security

Manage environment variables

To manage environment variables for my app I use envy. First you need the files .env and .env.example in the root of your project directory. In .env.example you create a list of all potential environment variables without any values and in .env you use the defined variables and assign the values to them.

Patricks-MBP:express-security patrick$ ls -al
total 152
drwxr-xr-x   14 patrick  staff    448 23 Jun 05:29 .
drwxr-xr-x    4 patrick  staff    128 26 Mai 05:40 ..
-rw-------    1 patrick  staff    181 23 Jun 05:59 .env
-rw-r--r--    1 patrick  staff     53 23 Jun 05:59 .env.example
....

Patricks-MBP:express-security patrick$ cat .env.example
port=
mongodbpath=
sessionsecret=
sessioncookiename=

Patricks-MBP:express-security patrick$ cat .env
port=<YOUR_PORT>
mongodbpath=<YOUR_CONNECTION_STRING>
sessionsecret=<YOUR_SESSION_SECRET>
sessioncookiename=<YOUR_SESSION_COOKIE_NAME>

Patricks-MBP:express-security patrick$ 

Envy must be installed as dependency and required in the main application file secserver.js. Then you can set the environment variables as follows.

// secserver.js

....

// envy module to manage environment variables
const envy = require('envy');

// set the environment variables
const env = envy()
const port = env.port
const mongodbpath = env.mongodbpath
const sessionsecret = env.sessionsecret
const sessioncookiename = env.sessioncookiename
....

Start the MongoDB Server

To run the db server we install mongoose as dependency and require it in the db.js configuration file. The database connection will be initiated with mongoose.connect and the StartMongoServer function will be exported to be called in the main application file secserver.js.

const envy = require('envy')
const env = envy()

const mongodbpath = env.mongodbpath

const mongoose = require('mongoose');
mongoose.set('useNewUrlParser', true);
mongoose.set('useUnifiedTopology', true);

const StartMongoServer = async function() {
  try {

    await mongoose.connect(mongodbpath)
    .then(function() {
      console.log(`Mongoose connection open on ${mongodbpath}`);
    })
    .catch(function(error) {
      console.log(`Connection error message: ${error.message}`);
    })

  } catch(error) {
    res.json( { status: "db connection error", message: error.message } );
  }

};

module.exports = StartMongoServer;

Authentication and authorization

For user authentication we use the module express-session and to store session data in the session store in our database we use connect-mongodb-session. Therefore we install these modules as dependencies in our project and require the modules in our secserver.js main application file.

Then we create with new MongoDBStore a session storage in our MongoDB to store session data in collection col_sessions. errors are catched with store.on.

We use the session in our app with app.use( session({...}) ). With every request to our site a new session object is created with a unique session ID which include a session cookie object. The session object has keys options and the values for each key define how to deal with the session object. The session ID is created and signed using the secret option. We use name to provide a session cookie name and store to define where the session object should be stored (in case we store the session).

We can access the session object with req.session and the session ID with req.session.id. With every request we have a new session and this new session will be created but not stored anywhere so far. We say the session is uninitialized. The saveUninitialized false option ensure that a session will only be written to the store in case it has been modified. What does this mean?

We can modify the session when we store additional data into it. We always do this when the user is logging in via the login or the register route. When we post the data from the login- or from the registration-form to the server we call loginUser or the createUser module which is defined in database/controllers/userC.js. Both modules do basically the same thing: They create a userData Object and store the userData object into the session object and redirect the user to the dashboard when login or registration was successful.

....

var userData = { 
	userId: user._id, 
	name: user.name, 
	lastname: user.lastname, 
	email: user.email, 
	role: user.role 
	}

    req.session.userData = userData

    res.redirect('/dashboard')

....

If the user is successfully logged in the session is initialized (modified), the session object incl. the userData object are stored into the store and a cookie is stored into the requesting browser. The content of the cookie is only a hash of the session Id and with each request of a logged in user the session on the server is looked up.

The cookie in the browser will live max 1 week as we defined in the cookie object maxAge set to 1 week. Because of the cookie option sameSite true the cookie scope is limited to the same site.

Then the resave false option ensures that the session will not be updated with every request. This mean the session ID that has been created when the user has logged in will be kept until the user is logged out again.

// secserver.js

....

// server side session and cookie module
const session = require('express-session');
// mongodb session storage module
const connectMdbSession = require('connect-mongodb-session');

....

// Create MongoDB session storage
const MongoDBStore = connectMdbSession(session)
const store = new MongoDBStore({
  uri: mongodbpath,
  collection: 'col_sessions'
});

// catch errors in case store creation fails
store.on('error', function(error) {
  console.log(`error store session in session store: ${error.message}`);
});

// Create the express app
const app = express();

....

// use session to create session and session cookie
app.use(session({
  secret: sessionsecret,
  name: sessioncookiename,
  store: store,
  resave: false,
  saveUninitialized: false,
  // set cookie to 1 week maxAge
  cookie: {
    maxAge: 1000 * 60 * 60 * 24 * 7,
    sameSite: true
  },

}));

....

Secure HTTP headers

Response headers are HTTP header that come with the HTTP response from the server to the client. The http response header contain data that could possibly damage the integrity of the client. It is therefore important to secure the response header of your application.

To secure the http response headers I user the module helmet. This is a relatively easy-to-use module consisting of various middleware functionalities to secure various http response headers.

First we install helmetas a dependency of our project. Then we require helmet and use helmet right after we created the app.

// secserver.js

// hTTP module
const http = require('http');
// express module
const express = require('express');
// hTTP header security module
const helmet = require('helmet');

// Create the express app
const app = express();

....

// use secure HTTP headers using helmet
app.use(helmet())

Using simply app.use(helmet()) set the http header security to default. Then the following 7 out 11 helmet features can be used.

1. dnsPrefetchControl controls browser DNS prefetching
2. frameguard to prevent clickjacking
3. hidePoweredBy to remove the X-Powered-By header
4. hsts for HTTP Strict Transport Security
5. ieNoOpen sets X-Download-Options for IE8+
6. noSniff to keep clients from sniffing the MIME type
7. xssFilter adds some small XSS protections
   

When we then request our home page to retrieve the http headers using curl -k --head in the terminal we see the following output.

Patricks-MBP:express-security patrick$ curl -k --head https://servtest.rottlaender.lan
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Fri, 26 Jun 2020 16:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1734
Connection: keep-alive
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
ETag: W/"6c6-U2uWyDNyzlyBAbSI/Quxqo9RRQE"

Patricks-MBP:express-security patrick$ 

App routing

get routes: We have the following get routes and navigation.

• Home (/)
• Login (/login)
• Register (/register)
• Dashboard (/dashboard)
• Logout (/logout)
  

get routes involve an optional middleware and respond HTML back to the client.

app.get('/<route>', <optional: someMiddleware>, (req, res) => {
	
	res.send(`<some HTML>`)
})

I will not explain the HTML and css in detail. But as everyone can see, the HTML is the same for every route except for the <body>. Of course, this is not very nice and becomes a bit more efficient with the use of a template engine, which I will explain below using the PUG template engine. I will then rebuild the app using PUG.

Lets have a look at the middleware. If a request is made for a route and a middleware function is included, the middleware function is first executed before the next routing function function(req, res) is called. A condition is built into the middleware function which is checked. My middleware is built so that in case the condition is true the middleware code is executed directly and the next routing function is omitted. If the condition is false, the next routing function function(req, res) is called.

I have built 2 different middleware functions which each check

middleware 1 (login- and register route): a user is logged in

// secserver.js
....
// middleware 1 to redirect authenticated users to their dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.userData) {
    res.redirect('/dashboard')

  } else {
    next()
  }
}
....

If a user is logged in the request should be redirected to the dashboard route, in any other case (user is not logged in) the next routing function function(req, res) is called and respond the HTML to the browser. This middleware 1 is included in the /login- and /register route. This mean logged in users will be redirected to their dashboard, not logged in users will see the login- and register form.

middleware 2 (dashboard- and logout route): a user is not logged in.

// secserver.js
....

// middleware 2 to redirect not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.userData) {
    res.redirect('/login')
  } else {
    next()
  }
}
....

If a user is not logged in the request should be redirected to the login roure, in any other case (user is logged in) the next routing function function(req, res) is called and respond the HTML to the browser. This middleware 2 is included in the /dashboard- and /logout route. This mean not logged in users will be redirected to login route, logged in users will see the dashboard- or can log themselves out.

post routes: We have the following post routes.

• /login
• /register
  

The login and the register get routes contain a form in the HTML. With these forms the user provide the data to login and for user registration. When the user click the send button the action is to call the login- or register post route. This will happen for all not logged in users. The login and the register get routes have the middleware redirectDashboardto redirect the user to the dashbard if the user is already logged in.

// secserver.js
....

app.get('/login', redirectDashboard, (req, res) => {
....
res.send(`
....
<div class="form">
	<form id='register_form' method='post' action='/register'>
	......
	<label for='send'>
   		<input class='sendbutton' type='submit' name='send' value='Send'>
	</label>
	</form>
</div>

`)
)}
....

app.get('/register', redirectDashboard, (req, res) => {
....
res.send(`
....
<div class="form">
	<form id='login_form' method='post' action='/login'>
	......
	<label for='send'>
   		<input class='sendbutton' type='submit' name='send' value='Send'>
	</label>
	</form>
</div>
`)
)}
.....

The post routes contain functions to login- (loginUser) or register (createUser) the user.

// secserver.js
....
// Post routes to manage user login and user registration
app.post('/login', userController.loginUser);

app.post('/register', userController.createUser);
....

The loginUser function is defined in the user controller database/controllers/userC.js. This function lookup a user in the database based on the email address that has been provided by the request body. The data that are attached to the request body have been provided by the user vie the login form of the app. If no user could be found in the database login is not possible. If a user exist with the given email address then the provided password will be compared with the one stored in the database. If the password match fail login is not possible because the provided password is wrong. in any other case, the login takes place and a userData object is created and attached to the session object.

// database/controllers/userC.js

User.findOne({ email: req.body.email }, function(error, user) {
  if (!user) {
    res.status(400).send({ code: 400, status: 'Bad Request', message: 'No User found with this email' })

    } else {
      if (bcrypt.compareSync(req.body.password, user.password)) {
    
        var userData = { userId: user._id, name: user.name, lastname: user.lastname, email: user.email, role: user.role }

          req.session.userData = userData

          res.redirect('/dashboard')

        } else {
          res.status(400).send({ code: 400, status: 'Bad Request', message: 'Wrong User password' })
        }

      }
    })

  }

The createUser function is also defined in the user controller database/controllers/userC.js. This function create a new User object based on the data from the request body provided by the user via the form. The provided password will be hashed and stored together with all other data into the database. Finally a userData object is created and attached to the session and the user will be redirected to the dashboard after the registration was successful.

// database/controllers/userC.js

createUser: async function (req, res) {
    // assign input data from request body to input variables
    const name = req.body.name
    const lastname = req.body.lastname
    const email = req.body.email
    const password = req.body.password
    const role = req.body.role

    const newUser = new User({
      name: name,
      lastname: lastname,
      email: email,
      password: password,
      role: role
    })

    newUser.password = await bcrypt.hash(newUser.password, saltRounds)

    await newUser.save(function(err, user) {
          if (err) {
            // if a validation err occur end request and send response
            res.status(400).send({ code: 400, status: 'Bad Request', message: err.message })
          } else {
            // req.session.userId = user._id

            var userData = { userId: user._id, name: user.name, lastname: user.lastname, email: user.email, role: user.role }

            req.session.userData = userData

            res.redirect('/dashboard')
          }
        })
  },

And we have a default get route.

• /favicon.ico
  

Browsers will by default try to request /favicon.ico from the root of a hostname, in order to show an icon in the browser tab. As we dont use favicon so far we must avoid that these requests returning a 404 (Not Found). Here The /favicon.ico request will be catched and send a 204 No Content status.

// secserver.js
....
app.get('/favicon.ico', function(req, res) {
    console.log(req.url);
    res.status(204).json({status: 'no favicon'});
});
....

3. Express App (Pug Template Version)

From a functional point of view this app is pretty much the same app then the HTML version. The difference is that we use PUG templates instead of HTML in each res.send().

Setup a seperate Database

For the PUG version of my app I set up a new database to manage the users and the sessions.

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b3e7f48a-a05c-4894-87db-996cb34eb1fb") }
MongoDB server version: 4.2.3

> db
test

> use admin
switched to db admin

> db
admin

> db.auth("adminUser", "adminpassword")
1

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> use express-security-pug
switched to db express-security-pug

> db.createUser({ user: "owner_express-security-pug", pwd: "passowrd", roles: [{ role: "dbOwner", db: "express-security-pug" }] })

Successfully added user: {
    "user" : "owner_express-security-pug",
    "roles" : [
        {
            "role" : "dbOwner",
            "db" : "express-security-pug"
        }
    ]
}

> db
express-security-pug

> exit

Connection string to connect to express-security-pug db using the owner_express-security-pug user.

mongodb://owner_express-security-pug:password@localhost/express-security-pug

Download the code from GitHub

Pls. go to my GitHub site and clone the code. Here you find some inline documentation in the code.

Create your app home directory express-security-pug

My app home directory is different to the one that is available after you cloned the code from GitHub.

Patricks-MBP:2020-05-15-express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security

Patricks-MBP:2020-05-15-express-security patrick$ mv node-part-5-express-security-with-db-pug express-security-pug

Patricks-MBP:2020-05-15-express-security patrick$ cd express-security-pug

Patricks-MBP:express-security-pug patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security-pug

Install PUG and use it in your app

First we install PUG as a dependency.

Patricks-MBP:express-security-pug patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security-pug

Patricks-MBP:express-security-pug patrick$ npm install pug --save

PUG is already fully integrated into Express. Pls. read the documentation how to use template engines in Express.

After you installed PUG the view engine must be set in your main application file secserverpug.js.

// secserverpug.js
....
// use Pug Template Engine
app.set('view engine', 'pug')
app.set('views', './views')
....

These instructions tell your app that PUG template engine is used and that the templates can be found in /views directory.

PUG Directory setup

In /views I setup the templates for home, login, registration and an error template.

In /views/includes I setup the files containing HTML or JavaScript. These can be included in the templates.

Patricks-MBP:express-security-pug patrick$ ls -l
total 128
-rw-r--r--    1 patrick  staff    771  1 Jul 06:04 README.md
drwxr-xr-x    5 patrick  staff    160 29 Jun 05:26 database
drwxr-xr-x  150 patrick  staff   4800 29 Jun 06:11 node_modules
-rw-r--r--    1 patrick  staff  47547 29 Jun 06:11 package-lock.json
-rw-r--r--    1 patrick  staff    367 29 Jun 06:11 package.json
-rw-r--r--    1 patrick  staff   4393  2 Jul 05:34 secserverpug.js
drwxr-xr-x    3 patrick  staff     96 29 Jun 05:26 static
drwxr-xr-x    8 patrick  staff    256  2 Jul 05:45 views

Patricks-MBP:express-security-pug patrick$ ls -l views
total 40
-rw-r--r--  1 patrick  staff   549 30 Jun 05:35 dashboard.pug
-rw-r--r--  1 patrick  staff   522  2 Jul 05:50 err.pug
-rw-r--r--  1 patrick  staff   420 29 Jun 05:39 home.pug
drwxr-xr-x  6 patrick  staff   192 29 Jun 05:17 includes
-rw-r--r--  1 patrick  staff   735 30 Jun 05:02 login.pug
-rw-r--r--  1 patrick  staff  1067 30 Jun 05:08 register.pug

Patricks-MBP:express-security-pug patrick$ ls -l views/includes
total 32
-rw-r--r--  1 patrick  staff   76 29 Jun 05:39 foot.pug
-rw-r--r--  1 patrick  staff  167 29 Jun 05:24 head.pug
-rw-r--r--  1 patrick  staff  489  2 Jul 05:13 nav.pug
-rw-r--r--  1 patrick  staff  420 29 Jun 05:08 script.js

Patricks-MBP:express-security-pug patrick$ 

The responsive Website Design

Each site like home, login, register and dashboard has a specific site template in /views directory. The site content will be defined in the main section of each template. PUG enables files with HTML or JavaScript to be included. This makes the site templates clear and easy to maintain. The includes are located in /views/includes directory.

The website is build based on a grid design and each site template has the following structure.

doctype html

HTML

	Head
		include includes/head.pug	
	
	Body
		
		Grid-Container
			
				Header
					include includes/nav.pug
			
				Main
					... site template specific HTML ...
			
				Footer
					include includes/foot.pug
					
		<script>
			  include includes/script.js

The design of the website is defined in the css in static/css/style.css.

Here in the css we define the Site Structure as grid areas consisting of header, main and footer and link them to the grid-container.

....

.header { grid-area: header; background-color: #ffffff; border-radius: 5px;}
.main { grid-area: main; background-color: #ffffff; border-radius: 5px;}
.footer { grid-area: footer; background-color: #ffffff; border-radius: 5px;}

.grid-container {
  display: grid;
  grid-template-areas:
      "header"
      "main"
      "footer";
  grid-gap: 5px;
  background-color: #d1d1e0;
  padding: 50px;
}

....

The Navigation is defined in the Header area of the Grid-Container and the HTML comes into the template via include includes/nav.pug.

// includes/nav.pug

//(this) refers to the DOM element to which the onclick attribute belongs to
// the a DOM element will be given as parameter to the function

a(class="burgericon" onclick="myFunction(this)")
  div(class='burgerline' id='bar1')
  div(class='burgerline' id='bar2')
  div(class='burgerline' id='bar3')

a(class='link' href='/') Home
a(class='link' href='/login') Login
a(class='link' href='/register') Register
a(class='link' href='/dashboard') Dashboard
a(class='link' href='/logout') Logout

So the navigation design is then defined in the css. Each navigation object is an a link. We have a links with class link and burgericon. The burgericon is used to open the navigation bar onclick when the screen is smaller than 600px (like iphone displays etc., explained below), is cosisting of 3 burgerlines and these lines are created using 3 div objects with class burgerline. The burgelines will be transformed with speed 0.4s when you click on the burgericon (explained below). The burgericon is not visible and aligned on the right edge. All other navigation links are visible and aligned on the left edge.

/* static/css/style.css */
....

/* style the navigation links with float on the left (side by side) */
.header a.link {
  float: left;
  display: block;
  padding: 14px 16px;
  text-decoration: none;
  font-size: 1.4vw;
  color: #28283e;
}

/* hover effect for each navigation link */
.header a.link:hover {
  background-color: #28283e;
  color: #ffffff;
}

/* style the burgericon link on the right */
.header a.burgericon {
  float: right;
  display: none;
  padding: 14px 16px;
}

/* style each burgerline that create the burgericon */
.burgerline {
  width: 35px;
  height: 5px;
  background-color: #28283e;
  margin: 6px 0;
  transition: 0.4s;
}
....

When the display screen is lower than 600px the navigation links will not be shown and the burgericon (on the right side) will be faded in instead.

/* static/css/style.css */
....
/* for screens up to 600px remove the navigation links and show the burgericon instead */
@media screen and (max-width: 600px) {
  .header a.link { display: none; }
  .header a.burgericon { display: block; }
}
....

When you click on the burgericon the burgerlines will be transformed so that you will see a cross instead of the hamburger like icon. The 2nd burgerline with id='bar2' will not be shown at all while the other 2 burgelines will be rotated 45 degrees counterclockwise (burgerline with id='bar1') and clockwise (burgerline with id='bar1').

/* static/css/style.css */
....
/* style burgerlines after on onclick event */
/* the .change class will be added onclick with classList.toggle in the JavaScript */
/* rotate first bar */
.change #bar1 {
  /* rotate -45 degrees (counterclockwise) move 15px down in Y-direction */
  transform: rotate(-45deg) translateY(15px);
}
/* fade out the second bar */
.change #bar2 {
  opacity: 0;
}
/* rotate third bar */
.change #bar3 {
  /* rotate +45 degrees (clockwise) move 15px up in Y-direction */
  transform: rotate(45deg) translateY(-15px);
}
....

After clicking on the burgericon, the burgerlines are transformed as described. The links of the navigation menu are displayed one below the other (float none) and aligned left.

/* static/css/style.css */
....
/* for screens up to 600px and after onclick event the responsive class will be added to the header */
@media screen and (max-width: 600px) {
  /* show navigation links left with no float (links shown among themselves) */
  .header.responsive a.link {
    float: none;
    display: block;
    text-align: left;
  }
}
....

All onclick functionalities are controlled by the javascript which is embedded in the HTML of each site template (pls see above includes/nav.pug). In the HTML, the onclick event is initiated in the burgericon link and the function myFunction is called with onclick =" myFunction(this) ". With the parameter this the entire burgericon object is transferred to the javascript function.

With each click on the burger icon, the class change is added to each burgerline or, if available, removed. This is done by the toggle() function. If change is set, the hamburg icon is transformed into a cross according to the specification in the css (see above). If change is withdrawn with a new click, the hamburger icon is displayed again.

But it happens even more in the javascript when you click on the hamburger icon. The element that has the id responsivenav is searched for and the variablereponsiveNavElement is assigned to this element. Is the class of the reponsiveNavElement header the classresponsive is added after clicking on the hamburger icon. If the class responsive is set, as described above, the links of the navigation menu are displayed one below the other (float none) and aligned left. So it applies in the css .header.responsive a.link {....}

In all other cases only the class header is set. So it applies in the css .header a.link {....} and the navigation links are not shown.

// includes/script.js

// the (burgerlines) parameter represent the DOM element that has been given to the function
function myFunction(burgerlines) {
  burgerlines.classList.toggle('change');

  var reponsiveNavElement = document.getElementById('responsivenav');
    if (reponsiveNavElement.className === 'header') {
      reponsiveNavElement.classList.add('responsive')
    } else {
      reponsiveNavElement.className = 'header';
    }
  }

Finally at the end of the css we define the defaults for h1, for our text content, the forms, the input fields and the send buttons.

Summary and Outlook

In this part 4 of my little node.js series we have seen how to setup a production ready environment for our express app. I showed this using a Mac OS, but in principle this setup also applies to Linux, for example.

The basic setup is, to put it simply, the app runs as a service on the server in the background using a Process Manager, but has no interface to the client. This client interface regulates a reverse proxy which is upstream of the app and accepts all requests and forwards them to the app, as well as the responses from the app back to the client. The communication is SSL/TLS secured.

At the center of the setup is a separate local MongoDB that manages all application data. In our example, these are the users, but also the sessions. I prefer to set up my own MongoDB on my server but of course it is conceivable to use a cloud-based solution or to install another database locally.

The express app itself uses secure HTTP response headers so that HTTP attacks like clickjacking, MIME type sniffing or some smaller XSS attacks on the client are made as difficult as possible. Access to personal areas of the application is secured by session-based authentication and authorization. The session-relevant data is stored in the database and not in the browser cookie, which means additional security with regard to attacks on the client. The browser cookie only contains a hash of the session ID to query the relevant user data from the database.

I would like to end my node.js series with Part 4. I discussed and demonstrated the basic concepts and procedures in parts 1 to 4. Of course there will be other interesting articles on the topic node.js and web programming on Digitaldocblog. Just take a look.

node contain various modules directly compiled into the node package like modules to access the network in asynchrons mode or modules to access the file system. Furthermore other modules can be embedded. These are precompiled files with .node extension or JavaScript modules. To manage these modules for node the node package manager (npm) exist. The npm package repository has more than 700.000 packages that can be installed with the npm package manager.

Installation of node and npm

node can be installed on a Mac with homebrew. node comes with npm.

Before node can be installed it must be checked if xcode command line tools are installed on the system.

Macbook Pro:~ user$ xcode-select -p
/Library/Developer/CommandLineTools
Macbook Pro:~ user$

The above command return that xcode command line tools are installed on your Mac and show the directory path where xcode command line tools are installed.

In case xcode command line tools are not installed run the following command in shell:

Macbook Pro:~ user$ xcode-select --install

Then it must be checked if Homebrew is installed.

Macbook Pro:~ user$ brew --version
Homebrew 1.7.7
Homebrew/homebrew-core (git revision 45d56; last commit 2018-10-08)
Homebrew/homebrew-cask (git revision 33e4d; last commit 2018-10-09)
Macbook Pro:~ user$

The above output show Homebrew 1.7.7 is installed on the Mac. In case Homebrew is not istalled so far run the follwoing command from shell:

Macbook Pro:~ user$ ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Check the Homebrew installation.

Macbook Pro:~ user$ brew doctor
Your system is ready to brew.
Macbook Pro:~ user$

To start with the node installation run the following command from shell:

Macbook Pro:~ user$ brew install node

After installation Homebrew created symlinks for node and npm in /usr/local/bin and symlink to the location where the relevant binaries have been installed.

Patricks-Macbook Pro:~ patrick$ ls -l /usr/local/bin
total 8

....

lrwxr-xr-x  1 patrick  admin   30  5 Jun 09:04 node -> ../Cellar/node/14.4.0/bin/node
lrwxr-xr-x  1 patrick  admin   38  5 Jun 14:22 npm -> ../lib/node_modules/npm/bin/npm-cli.js

....

We see that node has been installed in

/usr/local/Cellar/node/14.4.0/bin/node

and npm has been installed in

/usr/local/lib/node_modules/npm/bin

npm package management

npm install software packages always in node_modules directory. If you have not installed any software oackages with npm so far this directory does not exist and will be created during the installation process.

The installation of a software package with npm will be started with the following command.

npm install <package>

You can install software packages as Global Software Packages or as Local Software Packages.

Global Software Packages

If you want to use a software not only within a specific project but in global context on your machine then you provide the -g option in the install command.

npm install -g <package>

Then npm install this package in a global node_modules directory.

{prefix}/lib/node_modules

The prefix can be determined as follows

Macbook Pro:~ user$ npm config get prefix
/usr/local
Macbook Pro:~ user$

On my machine all global packages are installed under

/usr/local/lib/node_modules

To list all installed global packages on your machine you can ls -l the global node_modules directory.

Patricks-Macbook Pro:~ patrick$ ls -l /usr/local/lib/node_modules
total 0
drwxr-xr-x  24 patrick  admin  768  5 Jun 14:22 npm
drwxr-xr-x  21 patrick  admin  672  5 Jun 14:21 pm2

You can also list all global packages with the npm command.

Macbook Pro:~ user$ npm list -g --depth=0
/usr/local/lib
├── firebase-tools@5.0.1
└── npm@6.4.1
Macbook Pro:~ user$

To maintain your global packages run the following commands.

Check for outdated global packages

npm outdated -g --depth=0

Update a specific global package

npm update -g <package>

Update all global packages

npm update -g

Local Software Packages

Local Software Packages are dependencies of a node project. This mean these software packages are required by the project code to run correctly.

If you want to use a software package only in the context of a node project then you create a separate project subfolder on your system and change to this subfolder with cd. Any software package that will be installed under this subfolder is a dependency of your node project.

Patricks-Macbook Pro:test patrick$ pwd
/Users/patrick/Software/dev/node/test

Patricks-Macbook Pro:test patrick$ cd mytestproject

Patricks-Macbook Pro:mytestproject patrick$ pwd
/Users/patrick/Software/dev/node/test/mytestproject

Patricks-Macbook Pro:mytestproject patrick$ 

Now you are in the mytestproject application root directory. Run the command npm init from the application root directory.

Patricks-Macbook Pro:mytestproject patrick$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (mytestproject) 
version: (1.0.0) 
description: This is my Test Project
entry point: (index.js) 
test command: 
git repository: 
keywords: 
author: Patrick Rottlaender
license: (ISC) 
About to write to /Users/patrick/Software/dev/node/test/mytestproject/package.json:

{
  "name": "mytestproject",
  "version": "1.0.0",
  "description": "This is my Test Project",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Patrick Rottlaender",
  "license": "ISC"
}


Is this OK? (yes) yes
Patricks-Macbook Pro:mytestproject patrick$ ls -l
total 8
-rw-r--r--  1 patrick  staff  251 14 Jun 08:58 package.json

Patricks-Macbook Pro:mytestproject patrick$

A package.json file has been created in your application root directory. This package.json file contain meta data about your project. Now we create the index.js file as our main application file for the project.

You can name the main application file as you like. It is only important that under „main“ in the package.json file the file name match with the main application file name in your project root directory.

Patricks-Macbook Pro:mytestproject patrick$ touch index.js

Patricks-Macbook Pro:mytestproject patrick$ ls -l
total 8
-rw-r--r--  1 patrick  staff    0 14 Jun 09:08 index.js
-rw-r--r--  1 patrick  staff  251 14 Jun 08:58 package.json

Patricks-Macbook Pro:mytestproject patrick$

Dependencies are installed as local packages from a project root directory running the following command.

npm install <package_name> --save

npm will then create a node_modules directory under your project root directory and install the local packages there (incl. the dependencies of this new local package if there are any). npm will also create a package-lock.json file containing the current complete directory tree of all dependencies.

The package.json file is adapted and the new local software package is entered there as a dependency.

I have taken the following example 1:1 from the website of Tanja Rascia. If you want to read more details pls. go to Tanjas website.

Patricks-Macbook Pro:mytestproject patrick$ npm install left-pad --save

npm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN mytestproject@1.0.0 No repository field.

+ left-pad@1.3.0
added 1 package from 1 contributor and audited 1 package in 0.974s
found 0 vulnerabilities

Patricks-Macbook Pro:mytestproject patrick$ ls -l
total 16
-rw-r--r--  1 patrick  staff    0 14 Jun 09:08 index.js
drwxr-xr-x  3 patrick  staff   96 14 Jun 09:27 node_modules
-rw-r--r--  1 patrick  staff  366 14 Jun 09:27 package-lock.json
-rw-r--r--  1 patrick  staff  301 14 Jun 09:27 package.json

Patricks-Macbook Pro:mytestproject patrick$ cat package.json
{
  "name": "mytestproject",
  "version": "1.0.0",
  "description": "This is my Test Project",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "Patrick Rottlaender",
  "license": "ISC",
  "dependencies": {
    "left-pad": "^1.3.0"
  }
}
Patricks-Macbook Pro:mytestproject patrick$ 

After this we can write the code in index.js and use the functionality of left-pad. Therefore we must require the dependency at the beginning of the code.

// index.js

const leftPad = require('left-pad') // Require left pad
const output = leftPad('Hello, World!', 15) // Define output

// Send output to the console
console.log(output)

We run the programm using the following command and get the output on the console.

Patricks-Macbook Pro:mytestproject patrick$ node index.js
  Hello, World!
Patricks-Macbook Pro:mytestproject patrick$

For more information about npm and package management pls. check the getting started section on npm documentation site and the npm website .

Macbook Pro:~ user$ ls -l /Applications
total 0
drwxr-xr-x@  3 root     wheel   96 11 Jul 19:10 App Store.app
drwxr-xr-x@  3 root     wheel   96 21 Jun 18:48 Automator.app
drwxr-xr-x@  3 user     admin   96 14 Jun 09:32 Brackets.app
drwxr-xr-x@  3 root     wheel   96 21 Jun 18:48 Calculator.app
drwxr-xr-x@  3 root     wheel   96 21 Jun 18:48 Calendar.app
drwxr-xr-x@  3 root     wheel   96 11 Jul 19:10 Chess.app
drwxr-xr-x   3 user     admin   96  2 Okt 07:18 CleanMyMac 3.app
drwxr-xr-x   3 user     admin   96 19 Feb  2018 ClipGrab.app
drwxr-xr-x@  3 root     wheel   96 11 Jul 19:10 Contacts.app
drwxr-xr-x@  3 root     wheel   96 21 Jun 18:48 DVD Player.app

........

An alternative way to install software in your Mac is using a package manager like Homebrew. With Homebrew you have access to a large number of free software packages developed by developers for the installation via Homebrew.

Homebrew installation

Check if xcode command line tools are installed.

Macbook Pro:~ user$ xcode-select -p
/Library/Developer/CommandLineTools
Macbook Pro:~ patrick$

The above command return that xcode command line tools for xcode (CLT) are installed on your Mac and show the directory path where xcode command line tools are installed.

In case xcode command line tools are not installed run the following command in shell:

xcode-select --install

Check if Homebrew is (already) installed.

Macbook Pro:~ user$ brew --version
Homebrew 1.7.7
Homebrew/homebrew-core (git revision 45d56; last commit 2018-10-08)
Homebrew/homebrew-cask (git revision 33e4d; last commit 2018-10-09)
Macbook Pro:~ user$

The above output show Homebrew 1.7.7.

In case Homebrew is not istalled run the follwoing command from shell to install Homebrew:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

If you run the above shown ruby script you will be asked to run the following command.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

Check the Homebrew installation.

Macbook Pro:~ user$ brew --version
Homebrew 1.7.7
Homebrew/homebrew-core (git revision 02721; last commit 2018-10-21)
Homebrew/homebrew-cask (git revision 4bcf; last commit 2018-10-22)

Macbook Pro:~ user$ brew doctor
Your system is ready to brew.
Macbook Pro:~ user$

The shell output show that Homebrew version 1.7.7 is installed on the system and that you are ready to brew which means that you are ready to install software.

Homebrew Software Repositories and Tap(s)

With Homebrew package manager you have access to a wide range of software that can be installed on your system. After Homebrew has been installed on your system you find Homebrew under /usr/local/Homebrew on your system.

In principle, Homebrew differentiates between Core Software Packages, Non-Core Software Packages and so-called Cask(s). These Software Packages are available in GitHub Repositories or so called Tap(s).

Core Software Packages are free software developed from developers around the world to be installed on Mac OS platforms using Homebrew. Core software will be installed based on so called formulae. A formula contain the basic package definition and tell Homebrew i.e. how a package should be installed. This mean a formula of a certain software package contain the required information for the installation such as i.e. from where the installation files should be loaded, which dependencies exist and how the installation should be performed on the different Mac platforms. All formulae for Core Software Packages have been loaded from Homebrew’s GitHub Core-Repository-Tap Homebrew/homebrew-core into /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formulaon your machine.

It is also possible that Software can be installed with Homebrew but the formula is not available on Homebrew’s GitHub Core-Repository-Tap. Then we speak of Non-Core Software Packages because these Software Packages are not tap(ed) into the Homebrew Core-Repository-Tap. In this case you must first tap the Non-Core-Repository-Tap to let Homebrew know from where it should load the formula to install the software package.

Cask software or Casks are native Mac OS Apps like Google Chrome, Firefox or other native Mac OS Apps that can be also installed using the Homebrew Package Manager. All available Casks to install Mac OS native Apps have been loaded from Homebrew’s GitHub Profile Homebrew/homebrew-cask into /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask/Casks.

To get an overview which tap(s) are currently linked to your Homebrew Installation use th efollowing command.

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services
mongodb/brew

Where is Homebrew on your System

Homebrew create symlinks for each installed package in /usr/local/bin and symlink the relevant binary. Usually this symlink will point to /usr/local/Cellar/<package_name>/<version>/bin but it can also be anywhere else on your system.

So lets ls -l /usr/local/bin:

Macbook Pro:~ user$ ls -l /usr/local/bin
total 0
lrwxr-xr-x  1 user  admin  28 11 Nov  2016 brew -> /usr/local/Homebrew/bin/brew
lrwxr-xr-x  1 user  admin  34  1 Nov  2017 ccmake -> ../Cellar/cmake/3.9.4_1/bin/ccmake
lrwxr-xr-x  1 user  admin  33  1 Nov  2017 cmake -> ../Cellar/cmake/3.9.4_1/bin/cmake
lrwxr-xr-x  1 user  admin  39  1 Nov  2017 cmakexbuild -> ../Cellar/cmake/3.9.4_1/bin/cmakexbuild
lrwxr-xr-x  1 user  admin  33  1 Nov  2017 cpack -> ../Cellar/cmake/3.9.4_1/bin/cpack
lrwxr-xr-x  1 user  admin  41 16 Dez  2016 cryptest.exe -> ../Cellar/cryptopp/5.6.5/bin/cryptest.exe
lrwxr-xr-x  1 user  admin  33  1 Nov  2017 ctest -> ../Cellar/cmake/3.9.4_1/bin/ctest
lrwxr-xr-x  1 user  admin  32  1 Nov  2017 evm -> ../Cellar/ethereum/1.7.2/bin/evm

.......

Homebrew package manager binary is named brew and symlinked to /usr/local/Homebrew/bin/brew. The Homebrew package is installed in /usr/local/Homebrew This is the place on your system where Homebrew is installed.

Macbook Pro:~ user$ ls -l /usr/local/Homebrew
total 64
-rw-r--r--   1 user  admin    98 16 Dez  2016 CHANGELOG.md
-rw-r--r--   1 user  admin  3161  1 Nov  2017 CODE_OF_CONDUCT.md
-rw-r--r--   1 user  admin   827  9 Okt 07:38 CONTRIBUTING.md
-rw-r--r--   1 user  admin  1334  1 Nov  2017 LICENSE.txt
drwxr-xr-x   7 user  admin   224  9 Okt 07:38 Library
-rw-r--r--   1 user  admin  8471  9 Okt 07:38 README.md
-rw-r--r--   1 user  admin   899  9 Okt 07:38 azure-pipelines.yml
drwxr-xr-x   3 user  admin    96  9 Okt 07:38 bin
drwxr-xr-x   5 user  admin   160  1 Nov  2017 completions
drwxr-xr-x  49 user  admin  1568  9 Okt 07:38 docs
drwxr-xr-x   5 user  admin   160  9 Okt 07:38 manpages

Where are Core Software Packages on your System

Also the symlinks from other packages that you install with Homebrew have been created in /usr/local/bin directory and symlinked into /usr/local/Cellar/.... The Cellar is the place on your system where you usually find Homebrew installed software packages.

As you see on my system I have installed a couple of software packages like boost, cmake, node but also hugo which is my static website genrator.

Macbook Pro:~ user$ ls -l /usr/local/Cellar
total 0
drwxr-xr-x  4 user  admin  128  1 Nov  2017 boost
drwxr-xr-x  4 user  admin  128  1 Nov  2017 cmake
drwxr-xr-x  3 user  admin   96 16 Dez  2016 cryptopp
drwxr-xr-x  5 user  admin  160  1 Nov  2017 ethereum
drwxr-xr-x  4 user  admin  128  1 Nov  2017 gmp
drwxr-xr-x  5 user  admin  160  1 Nov  2017 go
drwxr-xr-x  4 user  staff  128  6 Okt 07:16 hugo
drwxr-xr-x  3 user  staff   96  3 Aug 06:48 icu4c
drwxr-xr-x  4 user  admin  128  1 Nov  2017 jsoncpp
drwxr-xr-x  4 user  staff  128  9 Okt 07:39 node
drwxr-xr-x  3 user  admin   96 12 Nov  2017 openssl
drwxr-xr-x  4 user  admin  128  1 Nov  2017 solidity

Install and Uninstall Core Software

To install formula with Homebrew to your system use the command.

brew install <formula>

To uninstall a formula from your system simply use the command.

brew uninstall <formula>

After installation the software package exist in the Cellar.

/usr/local/Cellar/<formula>/<version>

Update and Upgrade Core Software packages

First you update all formulae and Casks using the following command.

brew update

or if you want to update only a specific formula

brew update <formula>

Then you upgrade all your software packages using the following command.

brew upgrade

or if you want to upgrade only a specific formula

brew upgrade <formula>

The upgrade command installs newer software versions of all packages in the Cellar. Depending on the amount of packages you have installed this might take some time.

Where are Cask Software Packages on your System

Cask software is installed in the following directory.

Patricks-MBP:~ patrick$ ls -l /usr/local/Caskroom
total 0
drwxr-xr-x  4 patrick  admin  128 11 Jun 09:00 opera

From the output above you can see that I installed a GUI software package with the help of Homebrew. It is the Opera browser.

The installation takes place in the following path /usr/local/Caskroom/<package_name>/<version>/<symlink>. The symlink point to the /Applications directory where Homebrew copy the app.

Patricks-MBP:~ patrick$ ls -l /usr/local/Caskroom/opera
total 0
drwxr-xr-x@ 3 patrick  admin  96 11 Jun 09:00 68.0.3618.165
Patricks-MBP:~ patrick$ ls -l /usr/local/Caskroom/opera/68.0.3618.165
total 0
lrwxr-xr-x  1 patrick  admin  23 11 Jun 09:00 Opera.app -> /Applications/Opera.app

Install and uninstall Cask GUI Software Packages

When you install GUI Software the app must be copied into your Applications Directory. Here you need to provide your user password.

drwxr-xr-x  4 patrick  admin  128  9 Aug  2018 java
Patricks-MBP:~ patrick$ brew cask install opera
==> Downloading https://get.geo.opera.com/pub/opera/desktop/68.0.3618.165/mac/Opera_68.0.3618.165_Setup.dmg
######################################################################## 100.0%
==> Verifying SHA-256 checksum for Cask 'opera'.
==> Installing Cask opera
==> Moving App 'Opera.app' to '/Applications/Opera.app'.
Password:
🍺  opera was successfully installed!

To uninstall GUI Software Packeges you simply provide the uninstall command like this.

Patricks-MBP:~ patrick$ brew cask uninstall java
==> Uninstalling Cask java
==> Removing launchctl service com.oracle.java.Helper-Tool
==> Removing launchctl service com.oracle.java.Java-Updater
==> Uninstalling packages:
com.oracle.jdk-10.0.2
com.oracle.jre
==> Removing files:
/Library/Internet Plug-Ins/JavaAppletPlugin.plugin
/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents
/Library/PreferencePanes/JavaControlPanel.prefPane
/Library/Java/Home
/Library/Java/MacOS
==> Removing directories if empty:
/Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk
==> Purging files for version 10.0.2,13:19aef61b38124481863b1413dce1855f of Cask java

Update Cask GUI Software Packages

To update Casks you can check if some Casks are outdated using the following command brew cask outdated. Outdated Casks can be updated using the brew cask upgrade command.

More detailed information regarding the brew cask usage can be found on Homebrew-Cask.