In my booking system I give users different roles in my app and depending on their role, the users have different authorizations. An admin for example is able to access more sensitive data and functionalities than a normal player or a coach. So my app must know the role of a user to assign different authorizations to the particular user.

Clients, usually browsers send requests the app. The app responds to requests and is solely responsible for ensuring that the client only has access to the data that are intended for it. This request and response game is based on the HTTP protocol. HTTP is a stateless network protocol and requests cannot be related to each other. Each request is isolated and unrelated to previous requests and the server has no chance to recognize clients and does therefore not know their role.

This problem can be solved with sessions and cookies and means that session management must be implemented in the application. The application creates a session and stores session data such as the role of a requestor in this session. The session has a unique ID and the app saves only this ID in a cookie. The cookie is transferred to the browser and stored locally there. From now on, the browser always sends this cookie with the HTTP request and thus identifies itself to the application. The application can check the role of the requestor in the stored session data and control the appropriate access.

Basic setup of the server

First we need a working Server OS. I run Linux Ubuntu in production and have written an article about the basic setup of a production Linux server on my blog site Digitaldocblog. Since I am going to store the sessions in a MongoDB, MongoDB must be installed on the Linux server. I use MongoDB Community Edition but you can also install or upgrade to the MongoDB Enterprise Server version. In the lower part of the article you find the instructions how to install and setup your MongoDB Community Edition on your Linux System. In case you want to read the original documentation go on the MongoDB site and read how to install the MongoDB Community Edition for your OS.

In my express application I use a number of external modules or dependencies that have to be installed for the application in order for the application to run. In the repository of the bookingsystemon my GitHub account you find the package.json file which contains all the necessary dependencies. In principle, it is sufficient if you put this package.json file in your application main directory and install all dependencies with npm install.

Alternatively, of course, all modules can also be installed individually with

npm install <module> --save

Session Management

I discuss in the first part different code snippets in my application main file booking.js. The goal here is that you understand how session management is basically implemented.

// booking.js

// Load express module and create app
const express = require('express');
const app = express();
// Trust the first Proxy
app.set('trust proxy', 1);
// Load HTTP response header security module
const helmet = require('helmet');
// use secure HTTP headers using helmet with every request
app.use(
  helmet({
      frameguard: {
        action: "deny",
      },
      referrerPolicy: {
        policy: "no-referrer",
    },
    })
  );
// Load envy module to manage environment variables
const envy = require('envy');
const env = envy();

// Set environment variables
const port = env.port
const host = env.host
const mongodbpath = env.mongodbpath
const sessionsecret = env.sessionsecret
const sessioncookiename = env.sessioncookiename
const sessioncookiecollection = env.sessioncookiecollection

// Load server side session and cookie module
const session = require('express-session');
// Load mongodb session storage module
const connectMdbSession = require('connect-mongodb-session');
// Create MongoDB session store object
const MongoDBStore = connectMdbSession(session)
// Create new session store in mongodb
const store = new MongoDBStore({
  uri: mongodbpath,
  collection: sessioncookiecollection
});
// Catch errors in case session store creation fails
store.on('error', function(error) {
  console.log(`error store session in session store: ${error.message}`);
});
// Use session to create session and session cookie
app.use(session({
  secret: sessionsecret,
  name: sessioncookiename,
  store: store,
  resave: false,
  saveUninitialized: false,
  // set cookie to 1 week maxAge
  cookie: {
    maxAge: 1000 * 60 * 60 * 24 * 7,
    sameSite: true
  },
}));

... //further code not taken into account at this point

I create a server application using the Express-js  Web Application Framework. Therefore I load the Express-js module with the require() function and store the express() function in the constant app. Because my app is running behind a reverse proxy server I set the app to trust the first proxy server. Then I load the helmet module to use secure response headers in my app. I configure that all browsers should deny iFrames and that my app will set no referrer in the response header.

I use the envy module in my application to manage environment variables. Therefore I load the module with require() and store the envy() function in the constant env. With envy you can define your environment variables in your .env and .env.example files. These files must be stored in the application main directory as explained in the envy documentation.

Since my booking app is a real web application running on a web server in production I can not discuss the real environment variables because of security reasons. Therefore let us see how this work and make an example .env file.

// .env

port=myport
host=myhost
mongodbpath=myexamplemongodbpath
sessionsecret=myexamplesecret
sessioncookiename=booking
sessioncookiecollection=col_sessions

These variables have different values in my .env file. In the booking.jsfile above I define the constant env for the envy function with env = envy() . Then I have access to the environment variables defined in my .env file with env.<variable>. I define constants for each variable and assign the variable from the .env file with env.<variable>. These constants can now be used as values in the code.

I load the express-session module and the connect-mongodb-session module with the require() function. The session module stored in the constant session takes over the entire control of the session and cookie management.

The connect-mongodb-session stored in the constant connectMdbSession module is basically responsible for storing the session in the database. That is why we pass session as a parameter in the code and assign the constant MongoDBstore.

const MongoDBstore = connectMdbSession(session)

With new MongoDBStore I create a new store object. Here I pass the uri of the mongodb path and the collection where sessions should be stored.

// booking.js
...

const store = new MongoDBStore({
  uri: mongodbpath,
  collection: sessioncookiecollection
});

...

The store object initialized in this way contains all necessary parameters to successfully store a session object in my MongoDB database.

After we have defined the storage of the session object, we take care of the session object itself.

With app.use(session( {... cookie: {...} })) I create a session object with various options. The session object will be created with each request and also contains a cookie object. I pass the values for cookie: {...} and then other options likesecret: sessionsecret, the session object name with name: sessioncookiename as well as the location where the session object should be stored with store: store. Furthermore the session object has the option saveUninitialized: false and resave: false .

When the saveUninitialized option is set to false the session object is not stored into the store as long as the session is un-initialized. The option resave: false enforce that a session will not be saved back to the store even if the session is initialized. So we must understand what initialized and un-initialized mean. This must be explained.

A browser send a request to the app. More precisely, the browser sends the request to a defined endpoint in the app. An endpoint defines a path within the app that reacts to HTTP requests and executes code. Depending on the HTTP method GET or POST, the endpoint expects that the requestor requires a document back (GET) or that the requestor wants to send data to the app (POST).

In the example below the browser should send a GET request to GET home endpoint. The endpoint render the HTML template index and send the HTML back to the browser. Then the request is finished. So this process, which starts with the request and ends with the response is the runtime of the request.

In the code snippet below you see 2 GET endpoints in the booking.js file one for the home route and another one for the register route.

// booking.js

... //further code not taken into account at this point

// GET home route only for anonym users. Authenticated users redirected to dashboard
app.get('/', redirectDashboard, (req, res) => {
	
  console.log(req.url);
  console.log(req.session.id);
  console.log(req.session);

  res.render('index', {
      title: 'User Login Page',
    });

});

// GET register route only for anonym users. Authenticated users redirected to dashboard
app.get('/register', redirectDashboard, (req, res) => {
  
  console.log(req.url);
  console.log(req.session.id);
  console.log(req.session);

  res.render('register', {
      title: 'User Registration Page',
    });
});

... //further code not taken into account at this point

The code with app.use (session({ ... })) in my booking.js file ensures that a session object is always generated with each request. As long as a session object is not changed during the runtime of a request a separate session is created for each request and has its own session ID. The option saveUninitialized: false ensure that the session object will not be stored into the database. Each session object created in this way is un-initialized.

You can see the following output on the console for the home route and for the register route when we log the path, the session-ID and the session object on the console for each route.

/
BmbE8RVoTRcPP9nUnBm5JLE1w1mQiNyt
Session {
  cookie: {
    path: '/',
    _expires: 2021-04-24T04:27:04.265Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/register
awlPO-KpyVM51Gp6UAoeXGGmRWo-QFtP
Session {
  cookie: {
    path: '/',
    _expires: 2021-04-24T05:54:57.439Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}
  

The code of my app changes a session object during the runtime of a request by adding a data object when a user has successfully logged in. I will explain the code in detail in the next chapter but at the moment it is enough to know that. Therefore we play through the login of a user as follows.

The browser send a GET request to the home route as explained above, then the index template is rendered and the HTML page with the login form is sent back to the browser. During the runtime of this GET request a session object is created but the session object has not changed. We have already seen this above.

Then the user enters email and password in the login form and click submit. With this submit the browser send a POST request to the POST endpoint /loginusers and again a session object is generated for this POST request. During the runtime of the POST request, the code checks whether the transferred credentials are correct. If the credentials are correct, a data object with user data is generated and attached to the session object. Here the session object is changed during the runtime of the POST request. The existing session created with the POST request is now initialized at that moment. Because of the option saveUninitialized: false this session object is stored into the database store. When we look into the database store using the tool MongoDB Compass we see that the entire session object has been saved into the colsessions collection including the data object containing the required data of the user.

After the session initialization the code called by the POST endpoint redirect the request and send a new GET request to the /dashboard route. The code with app.use(session({ ... })) is called again but now there is an initialized session existing in the store. Because of the option resave: false the existing session object will not be updated and dragged along unchanged with every further request.

You see this in the output on the console when we log the path, the session-ID and the session object on the console for each route. The first output on the console is created when the GET request is sent to the home route. Then, the second output, after the user clicked submit the POST route /loginusers is called and a new session object is created. You see this from the different session IDs. During the runtime of this POST request the data object is added to the session object which initializes the session. Then, the third output, the GET route /dashboard is called and we see the same session object ID but the session object now contain the data object with the user data.

/
TEAZITdX7nLWBDc8uOk2HhXIiMZO7W-4
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:13.236Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/loginusers
gVlKut3bdEMiDHnK455FGjCi6YbPTBuZ
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:35.202Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    sameSite: true
  }
}

/dashboard
gVlKut3bdEMiDHnK455FGjCi6YbPTBuZ
Session {
  cookie: {
    path: '/',
    _expires: 2021-05-02T07:21:35.468Z,
    originalMaxAge: 604800000,
    httpOnly: true,
    secure: null,
    domain: null,
    sameSite: true
  },
  data: {
    userId: 5f716b7439777365c18639f1,
    status: 'active',
    name: 'Oskar David',
    lastname: 'Rottländer',
    email: 'oskar@test.com',
    role: 'player',
    age: 17,
    cat: 'youth'
  }
}

In summary, session management works as follows: A session object will be created with each request and the session object is only saved in the database when the user is logged in (saveUninitialized: false). As long as the user is logged in, the session object is not changed and the data of the session object in the database are not updated (resave: false).

But what happens to the cookie ? This will be explained in the next chapter.

User login

When the session has been initialized the cookie containing the session ID is stored in the browser of the requestor. With every request the browser provide the cookie to authenticate the requestor. To authenticate the requestor the code app.use(session({...})) is called and compare the session ID sent by the browser with the session IDs stored in the session store. If a session ID matches, the session object including the data object is attached to the request object to give the app access to the data object. Within the app we now have access to any attribute of the data object with req.session.data.<attribute>. Therefore we can now implement role based authorization by accessing the role of the requestor with req.session.data.role and use this information in conditions in the code to control access depending on the role of the requestor.

But lets start from the beginning with the login of the requestor or user as I call the requestor from now on. In order for a user to be able to login, he or she must first call the login page which can be displayed by calling up the home endpoint.

// booking.js

... // Code not discussed here

// Redirect GET requests from authenticated users to dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.data) {
    res.redirect('/dashboard')

  } else {
    next()

  }
}

... // Code not discussed here

// GET home route only for anonym users. Authenticated users redirected to dashboard

app.get('/', redirectDashboard, (req, res) => {

  res.render('index', {
      title: 'User Login Page',
    });

});

... // Code not discussed here

As you see above in the code I have first defined the middleware function redirectDashboard. This middleware ensure that only users who are not logged in see the login page. If we look at the code of the middleware function we can see that req.session.data is used in an if-condition to check whether a data object is attached to the current session object. In case the if-condition is true, the user is logged in and the request is redirected to the dashboard, but in case the if-condition is false, the user is not logged in and the next() function is called.

The GET endpoint has the routingPath to the home route. When a user visits the homepage of my booking application, the GET HTTP request ask for the home routingPath. The middleware function redirectDashboard is put in front of the routingHandler function. If the user is not logged in the routingHandler function render the HTML template index.pugand send the HTML back to the user or more precisely to the user’s browser.

So far so good. We now imagine a not logged in user who sees the index page in front of him or her now wants to login using his or her email and password.

As described above, the index page is nothing more than a login form for entering an email address and a password. When we look at the index.pug file we see that the form action attribute define that the form data email and password will be sent to the form handler /loginusers using the POST method when the Submit button is clicked.

...

form#loginForm.col.s12(
		method='post', 
		action='/loginusers'
		)

		input.validate(
			type='email', 
			name='email', 
			autocomplete='username' 
			required
			)
		...

		input.validate(
			type='password', 
			name='password', 
			autocomplete='current-password' 
			required
			)
		...

button.btn.waves-effect.waves-light(
		type='submit', 
		form='loginForm'
		)
...

Note: To understand the autocomplete attributes of the input tags I recommend reading the documentation of the Chromium Project. Most browsers have password management functionalities and automatically fill in the credentials after you provide a master password to unlock your local password store. By using these autocomplete attributes in login forms but also in user registration forms or change password forms you help browsers by using these autocomplete functions to better identify these forms.

When the user has entered his or her email and password in the HTML form and clicked the Submit button, the request body contain the Form Data attributes email and password. Then a POST HTTP request is sent via HTTPS to the POST endpoint /loginusers defined in my booking.js file (see above).

In the picture below you can see the output of the network analysis in the developer tool of the chrome browser. Here you can see that the Form Data are not encrypted on browser side but you also see that the POST request URL /loginusers is HTTPS. This mean that when the browser sent the POST request to the server these data are encrypted with SSL/TLS in transit from the browser to the server.

On the server side we have the web application behind a proxy server listening to HTTP requests addressed to the POST endpoint /loginusers. This POST endpoint is an anonym POST Route which means that the routingHandler controller function is restricted to not logged-in users only. This makes sense because a login function must not be used by already logged in users. So already logged in users can not send data to this POST endpoint. This check is controlled by the middleware function verifyAnonym which is put in front of the routingHandler.

So lets look at the relevant code snippets in booking.js.

// booking.js

...

// Load db controllers and db models
const userController = require('./database/controllers/userC');

...

// Verify POST requests only for anonym users
const verifyAnonym = (req, res, next) => {

  if (req.session.data) {
    var message = 'You are not authorized to perform this request because you are already logged-in !';
    res.status(400).redirect('/400badRequest?message='+message);

  } else {
    next()

  }
}

...

// Anonym POST Route
// Login user available for anonym only
app.post('/loginusers', verifyAnonym, userController.loginUser)

...

// GET bad request route render 400badRequest
app.get('/400badRequest', (req, res) => {
 
  res.status(400).render('400badRequest', {
    title: 'Bad Request',
    code: 400,
    status: 'Bad Request',
    message: req.query.message,
  })
})

...

At the beginning of the code I refer the constant userController to the user controller file userC.js using the require method. In userC.js all user functions are defined to control user related operations.

Note: When you look into the userC.js file you see that we export modules using module.exports = {...}. Using this directive we export in fact an object with various attributes and the values of these attributes are functions. So with module.exports = { loginUser: function(...) ...} we export the object including the attribute loginUser which contains a function as value. So when we refer the constant userController using the require() function in the booking.js file we store the complete exported object with all its attributes to the userController constant. Now we have access to any attribute of the exported object from userC.js file with userController.<attribute>. Because the attributes are in fact functions we call these functions with this statement.

In the verifyAnonym function req.session.data is used in the if-condition to check whether a data object is attached to the current session object. In case the if-condition is true, the user is already logged-in and is redirected to the Bad Request GET endpoint /400badRequest which is the standard route in my application to show the user that something went wrong. The user can see what went wrong from a message that has been attached to the request using the request parameter ?message=+message. In case the if-condition is false, the user is not logged-in and the next() function forwards the request to the routingHandler controller function that call the loginUser function using userController.loginUser. This function has access to the attributes email and password of the request body with req.body.email and req.body.password.

So lets look at the relevant code snippets in userC.js file.

// database/controllers/userC.js

// load the bcryptjs module
const bcrypt = require('bcryptjs');
// define hash saltrounds for password hashing
const saltRounds = 10;
// load the relevant Prototype Objects (exported from the models)
...

const User = require('../models/userM');

...

loginUser: function (req, res) {

    const inputemail = req.body.email
    const email = inputemail.toLowerCase()

    console.log(req.url);
    console.log(req.session.id);
    console.log(req.session);

    try {

      User.findOne({ email: email }, async function(error, user) {
        if (!user) {
          var message = 'User not found. Login not possible';
          res.status(400).redirect('/400badRequest?message='+message);

        } else {
          if (user._status !== 'active') {
            var message = 'Login not possible. Await User to be activated';
            res.status(400).redirect('/400badRequest?message='+message);

          } else {
              if (bcrypt.compareSync(req.body.password, user.password)) {

                var yearInMs = 3.15576e+10;
                var currentDate = new Date ()
                var currentDateMs = currentDate.getTime()
                var birthDateMs = user.birthdate.getTime()
                var age = Math.floor((currentDateMs - birthDateMs) / yearInMs)

                if (age < 18) {
                  var cat = 'youth'
                } else {
                  var cat = 'adult'
                };

                var userData = {
                  userId: user._id,
                  status: user._status,
                  name: user.name,
                  lastname: user.lastname,
                  email: user.email,
                  role: user.role,
                  age: age,
                  cat: cat,
                }

                req.session.data = userData

                res.status(200).redirect('/dashboard')

              } else {
                var message = 'Login not possible. Wrong User password';
                res.status(400).redirect('/400badRequest?message='+message);
              }
          }
        }
      })

    } catch (error) {
      // if user query fail call default error function
      next(error)

    }
  // End Module
  },

...

In order to authenticate a user, the loginUser function must find a user in the user database with the same email address as the one that was sent by the browser and attached to the request body by the app. If a user was found with the email, the function must check whether the transmitted password matches the password that is stored in the database for this user. If the email and password match the user is authenticated and the login is successful, if not, the login fails.

Passwords are never saved in plain text. Therefore I use the bcryptjs module to hash passwords. The bcryptjs module is loaded into the code with the require() function and assigned to the constant bcrypt. We set the constant saltRounds to the value of 10. This is the so called cost factor in the bcrypt hashing function and controls how much time bcrypt need to calculate a single bcrypt hash. Increasing the cost factor by 1 doubles the time and the more time bycrypt need to hash the more difficult it is to brute force stored passwords.

Then I load the user model from userM.js using the require() function and assign the constant User. Here at this point I have to explain the background. To do this, we also need a look at the userM.js file.

Note: I use MongoDB as the database and Mongoose to model the data. If you look in the file userM.js you see that a user object is created with the function new Schema() and saved in the variable userSchema. This userSchema object describes a user with all its attributes. At the end of the file, the mongoose.model() function is used to reference the userSchema to the collection colusers in my MongoDB. This reference is assigned to the variable User and exported using the function module.exports(). With User I have access to the user model meaning to all user objects and attributes in my database that are stored in the colusers collection. So that I can use this access in the code of my userC.js file I load userM.js with the require() function and assign the constant User. I can now use Mongoose functions for example to query user data from my colusers collection. This is exactly what we do with User.findOne() when we try to find a user with a certain email.

The actual user authentication now takes place in the userFindOne() function.

When we run User.findOne() we check the criteria that do not lead to a successful authentication.

1. No user found: We are looking for a user object that matches the email that has been submitted. If no user object is found with that email or the user found is not active, the request is redirected to the 400badRequest route. If we have found an active user, the submitted password string is hashed with bcrypt and compared with the saved password.
2. Wrong password: If the password comparison is not successful, the submitted password was wrong and the request is also redirected to the 400badRequest route.
   

Note: User.findOne() has a query object {email: email} and a callback function async function(error, user {...}) as parameters. When the async function find a user with the email in the database, this async function returns a user object with all the user attributes and store this object into the user parameter. Within the scope of the async function I have now access to the user attributes using user.<attribute>.

Only in case the user with the email is found and the submitted password is correct the authentication is successful.

If the user is successfully authenticated, the category of the user is calculated based on the current date and the user’s birth date. Then a userData object is created in which various user attributes are stored. The data of the userData object are then attached to the session. More precisely, the object data is attached to the session with req.session.data and the value userData is assigned. Now the session is initialized and the session object is stored in the colsessions collection of the MongoDB.

Then the response is sent back to the browser.

In this response, the browser is instructed to call up a GET request to the GET endpoint /dashboard. The response is sent using res.status(200).redirect('/dashboard'). In the Response Header you see that the cookie with the name booking is set in the users browser using the set-cookie directive. The cookie only contain the session ID which has been signed and encrypted with the secret we provided in app.use(session( {... cookie: {...} })).

Then the browser send the GET request to the endpoint /dashboard. Lets have a look into the booking.js file again.

// booking.js 

...

// Redirect GET requests from not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.data) {
    res.redirect('/')

  } else {
    next()

  }
}

...

// GET dashboard route only for authenticated users. Anonym users redirected to home
app.get('/dashboard', redirectLogin, async (req, res) => {

  // Check admin authorization and render admin_dashboard
  if (req.session.data.role == 'admin') {

    const user_query = User.find( {} ).sort({lastname: 1, name: 1});
    var users = await user_query.exec();

    const training_query = Training.find( {} ).sort({date: 'desc'});
    var trainings = await training_query.exec();

    const location_query = Location.find( {} ).sort({location: 'desc'});
    var locations = await location_query.exec();

    const booking_query = Booking.find( {} ).sort({_booktrainingdate: 'desc'});
    var bookings = await booking_query.exec();

    const invoice_query = Invoice.find( {} ).sort({invoicedate: 'desc'});
    var invoices = await invoice_query.exec();

    res.status(200).render('admin_dashboard', {
      title: 'Admin Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      data_users: users,
      data_trainings: trainings,
      data_locations: locations,
      data_bookings: bookings,
      data_invoices: invoices,

      });

  // Check player authorization and render player_dashboard
  } else if (req.session.data.role == 'player') {

    var currentDate = new Date();
    console.log('current date: ' +currentDate);

    const availabletraining_query = Training.find( { _status: 'active', date: { $gte: currentDate } } ).sort({ date: 'desc' });
    var availabletrainings = await availabletraining_query.exec();

    const booking_query = Booking.find( { _bookuseremail: req.session.data.email, _bookparticipation: { $ne: 'invoice' } } ).sort({ _booktrainingdate: 'desc' });
    var bookings = await booking_query.exec();

    const myuser_query = User.findOne( { email: req.session.data.email } );
    var myuser = await myuser_query.exec();

    const invoice_query = Invoice.find( {invoiceemail: req.session.data.email} ).sort({invoicedate: 'desc'});
    var invoices = await invoice_query .exec();

    res.status(200).render('player_dashboard', {
      title: 'Player Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      email: req.session.data.email,
      data_availabletrainings: availabletrainings,
      data_bookings: bookings,
      data_myuser: myuser,
      data_myinvoices: invoices,
      });

  // Check coach authorization and render coach_dashboard
  } else if (req.session.data.role == 'coach') {
   
    var currentDate = new Date().setHours(00, 00, 00);
    console.log('currentDate: ' +currentDate);

    const training_query = Training.find( { _status: 'active', date: { $gte: currentDate } } ).sort({ date: 'asc' });
    var trainings = await training_query.exec();

    res.status(200).render('coach_dashboard', {
      title: 'Coach Dashboard Page',
      name: req.session.data.name,
      lastname: req.session.data.lastname,
      role: req.session.data.role,
      data_trainings: trainings,
      });

  } else {
    // if user not authorized as admin, player or coach end request and send response
    var message = 'You are not authorized. Access prohibited';
    res.status(400).redirect('/400badRequest?message='+message);
  }

});

...

As you see above in the code we have first defined the middleware function redirectLogin. This middleware ensure that only users who are logged in see the dashboard page. In case the if-condition is true, the user is not logged in and the request is redirected to the home route, but in case the if-condition is false, the user is logged in and the next() function is called.

The GET HTTP request ask for the dashboard routingPath. The middleware function  redirectLogin is put in front of the routingHandler function. If the user is not logged in the  redirectLogin middleware redirect the request to the home route. In case the user is logged-in the routingHandler function is called using the request object and the response object as parameters.

app.get('/dashboard', redirectLogin, async (req, res) => {...})

If we look at the Request Header of this new GET request in the browser we can see that the cookie is dragged along unchanged with the GET request to the endpoint /dashboard. This happens from now on with every request until the session expires or until the user logout.

And now within the routingHandler function we do the user authorization check. The if condition check the users role using req.session.data.role. Depending on the role of the user different <role>dashboard HTML templates are rendered and for each role different HTML is sent back to the user’s browser. Various queries are executed beforehand because we need role specific data within each <role>dashboard HTML template. The return values ​​of the different queries find() and findOne() are only executed in case one of the if conditions become true. Then in each case the return values of the queries are stored in variables. In case all if conditions are false, meaning we cannot find a user with a role like admin, player or coach in the database for some reason the request is redirected to the Bad Request GET endpoint /400badRequest using the message as request parameter that this user is not authorized.

Within each if condition and for each role admin, player or coach, we create the response object by first setting the HTTP status to the value of 200 and then using the render method to render the respective HTML template.

res.status(200).render('<role>_dashboard', {...})

Within the render method, we now have the option of transferring a data object with different attributes to the HTML template. Later we can access these data in the respective HTML template and use it within the HTML template. How this works is not part of this article. But of course you can take a closer look at the templates admin dashboard, player dashboardand coach dashboardon my GitHub repository and you will immediately see how this works.

Create Authorizations

As I have already shown in the upper part, I work with middleware functions to control access to GET and POST endpoints in my app. Therefore these middleware functions are the Authorizations and you can find them in the code of my booking.js file.

// booking.js

...

// Authorizations
// Redirect GET requests from not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.data) {
    res.redirect('/')

  } else {
    next()

  }
}

// Redirect GET requests from authenticated users to dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.data) {
    res.redirect('/dashboard')

  } else {
    next()

  }
}

// Authorize POST requests only for not authenticated users
const verifyAnonym = (req, res, next) => {
  if (!req.session.data) {
    next()

  } else {
    var message = 'You are already logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for anonym and admin users
const verifyAnonymAndAdmin = (req, res, next) => {
  if (!req.session.data) {
    next()

  } else {

    if (req.session.data.role == 'admin') {
      next()

    } else {
      var message = 'You are no Admin. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);

    }

  }
}

// Authorize POST requests only for admin and player users
const verifyAdminAndPlayer = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'admin') {
      next()

    } else if (req.session.data.role == 'player') {
      next()

    } else {
      var message = 'You are no Admin, no Player. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);
  }

}

// Authorize POST requests only for admin users
const verifyAdmin = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'admin') {
      next()

    } else {
      var message = 'You are no Admin. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for player users
const verifyPlayer = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'player') {
      next()

    } else {
      var message = 'You are no Player. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);
    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

// Authorize POST requests only for coach users
const verifyCoach = (req, res, next) => {
  if (req.session.data) {
    if (req.session.data.role == 'coach') {
      next()

    } else {
      var message = 'You are no Coach. You are not authorized to perform this request !';
      res.status(400).redirect('/400badRequest?message='+message);

    }

  } else {
    var message = 'You are not logged-in. You are not authorized to perform this request !';
    res.status(400).redirect('/400badRequest?message='+message);

  }
}

...

As I have already explained above I use redirect functions as a middleware to control access to the GET endpoints home, register and dashboard. These middleware functions basically control access based on whether a user is logged-in or not. The redirect function redirectDashboard allow only not logged-in users access to the home endpoint and to the register endpoint, while users who are already logged-in have no access and would be redirected directly to the dashboard route if they try to access here. The redirectLogin middleware function allow only logged-in users access to the dashboard route while not logged-in users are redirected to the login or better to the home endpoint.

In addition to redirect functions I use verify functions as a middleware to control access to the POST endpoints. With the help of POST requests, data are sent via POST endpoints to the app. That is why it is particularly important to control who is allowed to send data and who is not. I use basically 5 types of POST endpoints.

Anonym POST endpoint. I only have one endpoint here. The loginusers endpoint can only be called by not logged-in users. Therefore the verifyAnonym  middleware is set before the routingHandler function to verify if the user is not logged-in.

// booking.js
...

// Anonym POST endpoint
// Login user available for anonym only
app.post('/loginusers', verifyAnonym, userController.loginUser)

...

Shared POST endpoints. The createusers endpoint can be called by not logged-in users and Admin users. The  verifyAnonymAndAdmin  middleware is set before the routingHandler function to verify if the user is not logged-in or if the user that is logged-in has the role admin. The  updateuseremail and setnewuserpassword endpoints can be called only by Admin and Player users. Therefore the  verifyAdminAndPlayer  middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is admin or player.

// booking.js
...

// Shared POST endpoints
// Create Users available for anonym and admin
app.post('/createusers', verifyAnonymAndAdmin, birthdateFormatValidation, userController.createUser)

// Update User-Email available for admin and player
app.post('/updateuseremail', verifyAdminAndPlayer, userController.updateUserEmail)

// Update User-Password available for admin and player
app.post('/setnewuserpassword', verifyAdminAndPlayer, userController.setNewUserPassword)

...

Admin POST endpoints. I have 19 endpoints here and each of these endpoint can only be called by Admin users. The  verifyAdmin middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is admin.

// booking.js
...

// Admin POST endpoints
// Admin User Management
app.post('/callupdateusers', verifyAdmin, userController.callUpdateUsers)

app.post('/updateuser', verifyAdmin, birthdateFormatValidation, userController.updateUser)

app.post('/terminateusers', verifyAdmin, userController.terminateUser)

app.post('/activateusers', verifyAdmin, userController.activateUser)

app.post('/removeusers', verifyAdmin, userController.removeUser)

// Admin Update Training
app.post('/callupdatetrainings', verifyAdmin, trainingController.callUpdateTrainings)

app.post('/updatetraining', verifyAdmin, trainingController.updateTraining)

// Admin Location Management
app.post('/createlocations', verifyAdmin, locationController.createLocation)

app.post('/callupdatelocations', verifyAdmin, locationController.callUpdateLocations)

app.post('/updatelocation', verifyAdmin, locationController.updateLocation)

app.post('/callcreatetrainings', verifyAdmin, trainingController.callCreateTrainings)

app.post('/createtraining', verifyAdmin, trainingController.createTraining)

// Admin Invoice Management
app.post('/createinvoice', verifyAdmin, invoiceController.createInvoiceUser)

app.post('/callcancelinvoice', verifyAdmin, invoiceController.callCancelInvoice)

app.post('/cancelinvoice', verifyAdmin, invoiceController.cancelInvoice)

app.post('/callpayinvoice', verifyAdmin, invoiceController.callPayInvoice)

app.post('/payinvoice', verifyAdmin, invoiceController.payInvoice)

app.post('/callrepayinvoice', verifyAdmin, invoiceController.callRePayInvoice)

app.post('/repayinvoice', verifyAdmin, invoiceController.rePayInvoice)

...

Player POST endpoints. I have 7 endpoints here and each of these endpoint can only be called by Player users. The  verifyPlayer middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is player.

// booking.js
...

// Player POST endpoints
// Player Booking Management
app.post('/callbooktrainings', verifyPlayer, bookingController.callBookTrainings)

app.post('/booktrainings', verifyPlayer, bookingController.bookTraining)

app.post('/bookingreactivate', verifyPlayer, bookingController.bookingReactivate)

app.post('/callcancelbookings', verifyPlayer, bookingController.callCancelBooking)

app.post('/cancelbookings', verifyPlayer, bookingController.cancelBooking)

// Player User Management
app.post('/callupdatemyuserdata', verifyPlayer, userController.callUpdateMyUserData)

app.post('/updatemyuserdata', verifyPlayer, birthdateFormatValidation, userController.updateMyUserData)

...

Coach POST endpoints. I have 2 endpoints here and each of these endpoint can only be called by Coach users. The  verifyCoach middleware is set before the routingHandler function to verify if the user is logged-in and if the users role is coach.

// booking.js
...

// Coach POST endpoints
// Coach Confirmation Management
app.post('/callparticipants', verifyCoach, bookingController.callParticipants)

app.post('/callconfirmpatricipants', verifyCoach, bookingController.callConfirmPatricipants)

...

User logout

The user initiates a logout himself by clicking on the logout link in the navigation of the application. This sends a GET request to the /logout endpoint of the application. In this routing definition, the session is first deleted from the database using req.session.destroy() and then the cookie is removed from the browser and the user is redirected to the 200success site using res.status(200).clearCookie('booking').redirect().

// booking.js

...

// GET logout route only for authenticated users. Anonym users redirected to home
app.get('/logout', redirectLogin, (req, res) => {
  req.session.destroy(function(err) {
    if (err) {
      res.send('An err occured: ' +err.message);
    } else {
      var message = 'You have been successfully logged out';
      res.status(200).clearCookie('booking').redirect('/200success?message='+message)
    }
  });
})

...

SSH public key authentication for the root user

First we create the ssh keys on the local machine.

PatrickMBNeu:~ patrick$ ssh-keygen -t rsa -b 4096 -C "your_email@domain.com"
Enter file in which to save the key (/home/patrick/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
PatrickMBNeu:~ patrick$ ls -l .ssh/id_*
-rw-------@ 1 patrick  staff  3434  4 Feb 14:58 .ssh/id_rsa
-rw-r--r--  1 patrick  staff   750  4 Feb 14:58 .ssh/id_rsa.pub
PatrickMBNeu:~ patrick$ 

We log in to the server with the root user via ssh and are in the root user’s home directory and create the .ssh directory and the authorized_keys file. The we log out from the server and install the public key from the local machine on the server.

PatrickMBNeu:~ patrick$ ssh root@85.134.111.90
root@85.134.111.90's password: 
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Fri Feb 19 07:10:43 2021 from 112.11.237.18
root@h2866085:~# ls -al
insgesamt 28
drwx------  3 root root 4096 Feb 19 07:10 .
drwxr-xr-x 23 root root 4096 Feb 19 06:41 ..
-rw-------  1 root root   12 Feb 19 07:10 .bash_history
-rw-r--r--  1 root root 3106 Aug 14  2019 .bashrc
drwx------  2 root root 4096 Feb 19 07:09 .cache
-rw-r--r--  1 root root  148 Aug 13  2020 .profile
-rw-r--r--  1 root root   20 Feb 19 06:37 .screenrc
root@h2866085:~# mkdir .ssh
root@h2866085:~# ls -al
insgesamt 32
drwx------  4 root root 4096 Feb 19 07:20 .
drwxr-xr-x 23 root root 4096 Feb 19 06:41 ..
-rw-------  1 root root   12 Feb 19 07:10 .bash_history
-rw-r--r--  1 root root 3106 Aug 14  2019 .bashrc
drwx------  2 root root 4096 Feb 19 07:09 .cache
-rw-r--r--  1 root root  148 Aug 13  2020 .profile
-rw-r--r--  1 root root   20 Feb 19 06:37 .screenrc
drwxr-xr-x  2 root root 4096 Feb 19 07:20 .ssh
root@h2866085:~# chmod 700 .ssh
root@h2866085:~# ls -al
insgesamt 32
drwx------  4 root root 4096 Feb 19 07:20 .
drwxr-xr-x 23 root root 4096 Feb 19 06:41 ..
-rw-------  1 root root   12 Feb 19 07:10 .bash_history
-rw-r--r--  1 root root 3106 Aug 14  2019 .bashrc
drwx------  2 root root 4096 Feb 19 07:09 .cache
-rw-r--r--  1 root root  148 Aug 13  2020 .profile
-rw-r--r--  1 root root   20 Feb 19 06:37 .screenrc
drwx------  2 root root 4096 Feb 19 07:20 .ssh
root@h2866085:~# cd .ssh
root@h2866085:~/.ssh# touch authorized_keys
root@h2866085:~/.ssh# ls -al
insgesamt 8
drwx------ 2 root root 4096 Feb 19 07:21 .
drwx------ 4 root root 4096 Feb 19 07:20 ..
-rw-r--r-- 1 root root    0 Feb 19 07:21 authorized_keys
root@h2866085:~/.ssh# chmod 600 authorized_keys
root@h2866085:~/.ssh# ls -al
insgesamt 8
drwx------ 2 root root 4096 Feb 19 07:21 .
drwx------ 4 root root 4096 Feb 19 07:20 ..
-rw------- 1 root root    0 Feb 19 07:21 authorized_keys
root@h2866085:~/.ssh# exit
Abgemeldet
Connection to 85.214.161.41 closed.

PatrickMBNeu:~ patrick$
PatrickMBNeu:~ patrick$ cat ~/.ssh/id_rsa.pub | ssh root@85.214.161.41 "cat >> ~/.ssh/authorized_keys"
PatrickMBNeu:~ patrick$ ssh root@85.134.111.90
root@85.214.161.41's password: 
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Fri Feb 19 07:18:57 2021 from 185.17.207.18

root@h2866085:~# cat .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCfV32OLcC90/0CE0nDsSRZwO5XtyRRBWkgKhkd+wlIad09Vi6URGO3XkUhac2bKWe6t9DP3GWSk23ruMj8M6UV9W1Fb7ZfFW3SXhz5+pRB1v0Uy5PDdxLH1foSz0hpbubCQ0AbEWWRNfMqKC6l2tFWrOfl5AXlbmZsHTH1Th9FSoBhqs8ZH33Oovs+lchzbpmObjUNzr0Y/ZWaNjNlAxvFtt8fMHxqEz3tw7ASub2eaVcGSiNioV3GKwlzbho62AF6b+KGbQkH92P5j4+KnDQpY92Ejd55c4kfq7DcG0pXLC2e77Ci/XnpROzllcOlSjmR5fsAIuWMw7dQyePCar2seVx7WBo0/Z/jnvF0exDJprtxPLlCbFRwj1nVMlKpUsqbE8mZs0L5k7Zh2GLkGJQekYR1X7zDthJHPMLeoepKw20onuCoTkquirwYhy4xCndjZ3VYk0033Rgu13ETrCB+eXc7UrbyyJJyTTs77BQZ/deTLZcXARYU96wQoQGzlevYjyWNhn6WEjkoBc2dcIHzV0Fp3enhLhptG6imHGsvAm+1uNkXbg46hYL4WZdJxkGXOoRo+oT/deRNvzMjDgL2SMUOgSzj7U+Krw0bUCY2LpkWp0lNAuT+YsF2O/k/TEVFBfKthrJd9f/PynTR+IFiRHK7jayhBQXIWSsqI9AlJw== p.rottlaender@icloud.com
root@h2866085:~# 

Ubuntu Version check

Another action we take is to check which OS version and kernel version we are dealing with. This provides important information when we install software later. Often we have to download special software releases for the Linux version used. Type any of the following commands to find OS Name and OS Version running on your Linux server.

root@h2866085:~$ cat /etc/os-release

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

root@h2866085:~$ lsb_release -a

No LSB modules are available.
Distributor ID:    Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:    18.04
Codename:    bionic

root@h2866085:~$

To print the Linux kernel version running on your server type the following command. I am running Linux kernel version 4.15.

root@h2866085:~$ uname -r
4.15.0
root@h2866085:~$

Create a user

Of course, we don’t want to log in to the system with the root user all the time. It is therefore advisable to create another user with whom we can then log in and work.

A user can be created in Ubuntu Linux with the command useradd or adduser. I use the adduser command because with this perl script, in addition to creating the user in /etc/passwd and creating a dedicated user group in /etc/group, the home directory in /home is also created for that user and default files are copied from /etc/skel if necessary.

As root run the command adduser mynewuser.

root@h2866085:~$ adduser mynewuser

Benutzer »mynewuser« wird hinzugefügt …
Neue Gruppe »mynewuser« (1001) wird hinzugefügt …
Neuer Benutzer »mynewuser« (1001) mit Gruppe »mynewuser« wird hinzugefügt …
Persönliche Ordner »/home/mynewuser« wird erstellt …
Dateien werden von »/etc/skel« kopiert …
Geben Sie ein neues UNIX-Passwort ein: 
Geben Sie das neue UNIX-Passwort erneut ein: 
passwd: password updated successfully
Changing the user information for mynewuser
Enter the new value, or press ENTER for the default
    Full Name []: Tech User
    Room Number []: 
    Work Phone []: 
    Home Phone []: 
    Other []: This user is only for tech 
Ist diese Information richtig? [J/N] J

root@h2866085:~$

This create the user mynewuser. In /etc/passwd you see that the user has been created with userID (1000) and a groupID (1000). In /etc/group you find the new group mynewuser. When you check the home directory you find the new home directory in /home/mynewuser.

root@h2866085:~$ grep mynewuser /etc/passwd
mynewuser:x:1000:1000::/home/mynewuser:/bin/bash
root@h2866085:~$ cat /etc/group
....
....
mynewuser:x:1000:
...
root@h2866085:~$ ls -l /home
drwxr-xr-x 2 mynewuser mynewuser 4096 Jan 23 05:31 mynewuser
root@h2866085:~$

The last entry in the /etc/passwd specifies the shell of the new user. Here the newly created user has the Bash shell. In general a Unix shell is a command processor running in a command line window. The user types commands and these commands will be executed and cause actions. For more details you can read the wiki article about Unix Shells.

Because there are some different shells available there might be the need to change the user’s shell which basically mean that you change the variety of available commands on your command line. To change the shell of a user type usermod --shell </path/toShell> <mynewuser>. To see which shells you can use pls. check the /etc/shells directory.

root@h2866085:~$ ls -l /etc/shells
# /etc/shells: valid login shells
/bin/sh
/bin/dash
/bin/bash
/bin/rbash
/usr/bin/screen
/bin/tcsh
/usr/bin/tcsh
root@h2866085:~$ usermod --shell /bin/sh mynewuser
root@h2866085:~$ grep mynewuser /etc/passwd
mynewuser:x:1001:1001:Tech User,,,,This user is only for tech:/home/mynewuser:/bin/sh

Sudo configuration

This newly created user can create files in his home directory and access those files. However, it cannot copy files to directories owned by root. To demonstrate this, I log in to the system with the newly generated user. I create a file in the home directory and then I try to copy this file into a directory owned by root. The error message is that I am not authorized. To do this copy you must be root.

A user can still execute commands as root. That’s why there is sudo. So when I try to execute the command with sudo I get the message that the user is not in the sudoers-file.

Patricks-MacBook-Pro:~$ ssh mynewuser@85.214.161.41
Welcome to Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
Last login: Fri Feb  5 08:57:52 2021 from 87.161.105.46
mynewuser@h2866085:~$  touch testfile
mynewuser@h2866085:~$ ls -l
insgesamt 4
-rw-rw-r-- 1 mynewuser mynewuser 24 Jan 23 06:20 testfile
mynewuser@h2866085:~$ cp testfile /etc/nginx/sites-available/testfile
cp: reguläre Datei '/etc/nginx/sites-available/testfile' kann nicht angelegt werden: Keine Berechtigung
mynewuser@h2866085:~$ sudo cp testfile /etc/nginx/sites-available/testfile
[sudo] Passwort für mynewuser: 
mynewuser ist nicht in der sudoers-Datei. Dieser Vorfall wird gemeldet.
mynewuser@h2866085:~$

Sudo allows the root user of the system to give other users on the system (or groups) the ability to run some (or all) commands as root. The sudo configuration is detailed in /etc/sudoers file.

In my sudo configuration file it is already preconfigured that all users in the sudo group can execute all commands as root. Please note the corresponding entry %sudo ALL=(ALL:ALL) ALL in the /etc/sudoers configuration file below.

I want that the the user mynewuser should be able to execute all commands as root. Therefore mynewuser must be added to the sudo group.

To make mynewuser a member of the sudo group I use the command usermod. Therefore I log in with root using the su root command. Then I run usermod with -a and -G option, which basically says to append a user to a Group.

Finally I check with cat /etc/group that the sudo group contain the mynewuser and logoff from the root account using exit.

mynewuser@h2866085:~$ su root
Passwort: 
root@h2866085:/home/mynewuser# 
root@h2866085:/home/mynewuser# cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults    env_reset
Defaults    mail_badpass
Defaults    secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo    ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d

root@h2866085:/home/mynewuser# usermod -a -G sudo mynewuser
root@h2866085:/home/patrick# cat /etc/group
root:x:0:
daemon:x:1:
bin:x:2:
...
sudo:x:27:mynewuser
....
root@h2866085:/home/patrick# exit
mynewuser@h2866085:~$

Advanced Package Tool on Ubuntu Linux

But before we start with software installations I need to give you some background info about Ubuntu or Debian package management.

Whenever you want to install software on your Ubuntu system, you can use the Advanced Package Tool or APT for short. Users interact with APT using the apt command. APT download the software package from a package source and then install it.

The software package can only be installed if the package source is known to the APT system. Therefore package sources are listed in the file /etc/apt/sources.list or in further files with the extension .list in the directory /etc/apt/sources.list.d.

When you call apt install <package-name> on your console APT first check the package sources listed in your /etc/apt/sources.list file. When the system find the package on one of these package sources APT will install the software from there.

When the software is not available on any package sources listed in /etc/apt/sources.list APT check the package sources listed in the .list files in your /etc/apt/sources.list.d directory.

When APT does not find the package on any package sources the software package cannot be installed until the package source is entered either in /etc/apt/sources.list or in a separate .list file in the /etc/apt/sources.list.d directory.

The entries in .list files regardless of whether they are in /etc/apt/sources.list or /etc/apt/sources.list.d/*.list are structured identically. The structure of the .list files is divided into 4 sections.

Example:
deb    ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic     main restricted universe
<type> <URI>                                       <archive>  <component>

The above listed package source is an ftp server that contain binary packages for the Ubuntu Bionic Distribution (bionic). The Binary Packages include packages that meet the Ubuntu licensing requirements and are supported by the Ubuntu team (main), software packages that the Ubuntu developers support because of their importance, but which are not under Ubuntu licensing (restricted) and a wide range of free software (no licensing restrictions) that is not officially supported by Ubuntu (universe).

Universe software is maintained by the Masters of the Universe (MOTU Developers). If a MOTU Developer want a software to be included in the Ubuntu package sources, this Developer must suggest the package for the Universe. MOTUs also maintain Multiverse software. Multiverse software is also managed by the MOTUs but the difference is that Multiverse software is not free Software. Multiverse software is subject to licensing restrictions. More details can be read in the Ubuntu Package Management Documentation on the Ubuntu Website.

The .list file can be crated manually using an editor like nano or with the following command. Here in this example to create a mongoDB source file.

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list

In order to check a package source for authenticity, the GPG key of the package source provided by the manufacturer must also be downloaded and added to the Ubuntu APT keyring using the apt-key add command. Here in this example to add the key to verify the mongoDB source.

wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -

With wget I download the public key from the mongodb.org server and pipe it into the apt-key add command which finally add the key to the keyring.

Alternatively you can use the add-apt-repository utility or script to automate package source entries in your /etc/apt/sources.list file. From Debian 8 on add-apt-repository is part of the Debian Package software-properties-common which must be installed in case you want to use this utility for your package source management.

apt-get update
apt-get install software-properties-common

Using add-apt-repository the package source will be appended to the /etc/apt/sources.listfile. No separate file will be created in /etc/apt/sources.list.d directory. Here is the example how to add the package source for the mongoDB.

sudo add-apt-repository 'deb [arch=amd64] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse'

The utility add-apt-repository can also be used to install software from Personal Package Archives (PPA). PPAs are a special service for MOTU Developers (Masters of the Universe) to provide Ubuntu packages that are built and published with Launchpad. When you use add-apt-repository to install a PPA a new .list file will be created in your /etc/apt/sources.list.d directory.

add-apt-repository ppa:user/ppa-name

When you use add-apt-repository the PPA’s key is automatically fetched and added to the keyring. You can read more about PPA packaging on the launchpad help page.

You should not use PPA software in production environments. The reason is that PPA software is often not maintained regularly by the MOTUs on launchpad. In general MOTUs publish their first versions on launchpad and then integrate them into the main Ubuntu Universe or Multiverse sources. So read the documentation and if possible try to install your production packages from main sources. You can see this in the following example when I install nodejs and npm from a main source.

Nodejs and NPM Installation

To run a node application you definitely need the node platform nodejs and the node package manager npm.

Go to the nodejs.org download page to find out the latest version of node to install. I recommend to install the latest LTS version (Long Term Support) which is at the time of writing this document node version 14.15.5 including npm in version 6.14.11.

When you scroll down on the nodejs.org download page you see the install options. I prefer to install nodejs via package manager. Here you select your distribution so I select Debian and Ubuntu based Linux distributions. The nodejs binaries are available on GitHub nodesource repository. I choose Debian and Ubuntu based distributions (deb) manual installation.

Since I did not installed nodejs via PPA before I can skip the first step described here in the documentation. So I start with the import of the package signing key to ensure that apt can verify the new node source.

$ wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -

The key ID at the time of writing this document is 1655A0AB68576280.

You can check the imported key using the apt-key command.

$ sudo apt-key fingerprint ABF5BD827BD9BF62

The output should be.

pub   rsa4096 2014-06-13 [SC]
      9FD3 B784 BC1C 6FC3 1A8A  0A1C 1655 A0AB 6857 6280
uid        [ unbekannt] NodeSource <gpg@nodesource.com>
sub   rsa4096 2014-06-13 [E]

Then I create the nodesource.list file in my /etc/apt/sources.list.d directory to make the nodejs package source known to apt. I create 2 entries in that file one for the nodejs debian binaries (deb) and one for the nodejs sources (deb-src).

# Replace $VERSION with Node.js Version you want to install: i.e. node_14.x
# $VERSION=node_14.x
# Replace $DISTRO with the output of the command lsb_release -s -c
# $DISTRO=bionic
$ echo "deb https://deb.nodesource.com/$VERSION $DISTRO main" | sudo tee /etc/apt/sources.list.d/nodesource.list
$ echo "deb-src https://deb.nodesource.com/$VERSION $DISTRO main" | sudo tee -a /etc/apt/sources.list.d/nodesource.list

Then I update the source package list and install.

$ sudo apt-get update
$ sudo apt-get install nodejs

Here is my manual installation.

patrick@h2866085:/etc/apt$ ls -l sources.list.d
insgesamt 0
patrick@h2866085:/etc/apt$ apt-key list
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid        [ unbekannt] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid        [ unbekannt] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid        [ unbekannt] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

patrick@h2866085:~$ sudo wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -
[sudo] Passwort für patrick: 
OK
patrick@h2866085:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2014-06-13 [SC]
      9FD3 B784 BC1C 6FC3 1A8A  0A1C 1655 A0AB 6857 6280
uid        [ unbekannt] NodeSource <gpg@nodesource.com>
sub   rsa4096 2014-06-13 [E]

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid        [ unbekannt] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid        [ unbekannt] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid        [ unbekannt] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

patrick@h2866085:~$ sudo apt-key fingerprint 1655A0AB68576280
pub   rsa4096 2014-06-13 [SC]
      9FD3 B784 BC1C 6FC3 1A8A  0A1C 1655 A0AB 6857 6280
uid        [ unbekannt] NodeSource <gpg@nodesource.com>
sub   rsa4096 2014-06-13 [E]

patrick@h2866085:~$ lsb_release -s -c
bionic
patrick@h2866085:~$ sudo echo "deb https://deb.nodesource.com/node_14.x bionic main" | sudo tee /etc/apt/sources.list.d/nodesource.list
deb https://deb.nodesource.com/node_14.x bionic main
patrick@h2866085:~$ sudo echo "deb-src https://deb.nodesource.com/node_14.x bionic main" | sudo tee -a /etc/apt/sources.list.d/nodesource.list
deb-src https://deb.nodesource.com/node_14.x bionic main
patrick@h2866085:~$ ls -l /etc/apt/sources.list.d
insgesamt 4
-rw-r--r-- 1 root root 110 Feb 20 08:14 nodesource.list
patrick@h2866085:~$ sudo cat /etc/apt/sources.list.d/nodesource.list
deb https://deb.nodesource.com/node_14.x bionic main
deb-src https://deb.nodesource.com/node_14.x bionic main
patrick@h2866085:~$ sudo apt-get update
OK:1 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic InRelease
Holen:2 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-updates InRelease [88,7 kB]
Holen:3 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-security InRelease [88,7 kB]
Holen:4 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic/main Translation-de [454 kB]
Holen:5 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic/restricted Translation-de [2.268 B]
Holen:6 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic/universe Translation-de [2.272 kB]                       
Holen:7 https://deb.nodesource.com/node_14.x bionic InRelease [4.584 B]                                                  
Holen:8 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-updates/universe Sources [446 kB]
Holen:9 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-updates/main amd64 Packages [1.885 kB]
Holen:10 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-updates/universe amd64 Packages [1.718 kB]
Holen:11 https://deb.nodesource.com/node_14.x bionic/main amd64 Packages [764 B]
Holen:12 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-updates/universe Translation-en [363 kB]
Holen:13 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-security/universe Sources [277 kB]
Holen:14 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-security/universe amd64 Packages [1.109 kB]
Holen:15 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-security/universe Translation-en [248 kB]
Es wurden 8.957 kB in 3 s geholt (3.459 kB/s).                    
Paketlisten werden gelesen... Fertig
patrick@h2866085:~$ sudo apt-get install nodejs
patrick@h2866085:~$ node -v
v14.15.5
patrick@h2866085:~$ npm -v
6.14.11
patrick@h2866085:~$ 

Basically we installed npm in a bundle together with nodejs using the apt package manager. But npm itself is also an additional package manager. We now have apt and npm as package managers on our system.

We need npm to have an additional source for software available on npmjs.com. We will for example install PM2 via npm (see next chapter) but but in particular we need npm to add local software packages or dependencies to our node application projects. So when we want to develop a node app based on the expressjs framework we will install express locally in our project directory. But this will be part of a separate tutorial.

When you ask npm to find outdated packages you can run the npm outdated command (using the -g option to show only global packages).

patrick@h2866085:~$ npm outdated -g --depth=0
Package  Current   Wanted  Latest  Location
npm      6.14.11  6.14.11   7.5.4  global

As you see from an npm point of view the latest version of npm is 7.5.4. But npm know that we installed npm together as a bundle with nodejs via apt and in the apt package sources we say that our wanted version is 14.15.5 (see above). This is a bit confusing but basically npm outdated tell us that that there is a higher npm version available (latest) but on our system we are up to date as the current version and the one we defined in our package sources (wanted) are equal.

We can update npm together only together with the nodejs update which must be initiated with apt-update. And here you see the packages are all up to date.

patrick@h2866085:~$ sudo apt update
OK:1 https://deb.nodesource.com/node_14.x bionic InRelease
OK:2 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic InRelease
OK:3 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-updates InRelease
OK:4 ftp://ftp.stratoserver.net/pub/linux/ubuntu bionic-security InRelease
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Alle Pakete sind aktuell.
patrick@h2866085:~$  

PM2 Installation with NPM

Go to npmjs.com and search for PM2 Process Manager. You find the PM2 package on npmjs.com. Follow the install instructions.

patrick@h2866085:~$ sudo npm install pm2 -g
patrick@h2866085:~$ pm2 -v
                        -------------

__/\\\\\\\\\\\\\____/\\\\____________/\\\\____/\\\\\\\\\_____
 _\/\\\/////////\\\_\/\\\\\\________/\\\\\\__/\\\///////\\\___
  _\/\\\_______\/\\\_\/\\\//\\\____/\\\//\\\_\///______\//\\\__
   _\/\\\\\\\\\\\\\/__\/\\\\///\\\/\\\/_\/\\\___________/\\\/___
    _\/\\\/////////____\/\\\__\///\\\/___\/\\\________/\\\//_____
     _\/\\\_____________\/\\\____\///_____\/\\\_____/\\\//________
      _\/\\\_____________\/\\\_____________\/\\\___/\\\/___________
       _\/\\\_____________\/\\\_____________\/\\\__/\\\\\\\\\\\\\\\_
        _\///______________\///______________\///__\///////////////__

                          Runtime Edition

        PM2 is a Production Process Manager for Node.js applications
                     with a built-in Load Balancer.

                Start and Daemonize any application:
                $ pm2 start app.js

                Load Balance 4 instances of api.js:
                $ pm2 start api.js -i 4

                Monitor in production:
                $ pm2 monitor

                Make pm2 auto-boot at server restart:
                $ pm2 startup

                To go further checkout:
                http://pm2.io/
                        -------------
[PM2] Spawning PM2 daemon with pm2_home=/home/patrick/.pm2
[PM2] PM2 Successfully daemonized
4.5.4
patrick@h2866085:~$ 

Nano Installation with apt

Nano is part of the Ubuntu 18.04 bionic main packages that are provided via the standard ftp package sources from my provider at stratoserver.net. These standard ftp package sources are in my /etc/apt/sources.list file and can be installed with apt.

patrick@h2866085:~$ sudo apt install nano
patrick@h2866085:~$

Nginx Installation with apt

Also Nginx is a package that is available under the standard Ubuntu main sources. Nginx is part of the Ubuntu 18.04 bionic main packages provided via the standard ftp package sources from my provider at stratoserver.net. Nginx can be installed from these sources using the apt command.

patrick@h2866085:~$ sudo apt install nginx
patrick@h2866085:~$

After the installation is complete the Nginx configuration files are in /etc/nginx directory.

patrick@h2866085:~$ ls -l /etc/nginx
insgesamt 64
drwxr-xr-x 2 root root 4096 Jan 10  2020 conf.d
-rw-r--r-- 1 root root 1077 Apr  6  2018 fastcgi.conf
-rw-r--r-- 1 root root 1007 Apr  6  2018 fastcgi_params
-rw-r--r-- 1 root root 2837 Apr  6  2018 koi-utf
-rw-r--r-- 1 root root 2223 Apr  6  2018 koi-win
-rw-r--r-- 1 root root 3957 Apr  6  2018 mime.types
drwxr-xr-x 2 root root 4096 Jan 10  2020 modules-available
drwxr-xr-x 2 root root 4096 Feb 20 12:46 modules-enabled
-rw-r--r-- 1 root root 1482 Apr  6  2018 nginx.conf
-rw-r--r-- 1 root root  180 Apr  6  2018 proxy_params
-rw-r--r-- 1 root root  636 Apr  6  2018 scgi_params
drwxr-xr-x 2 root root 4096 Feb 20 12:46 sites-available
drwxr-xr-x 2 root root 4096 Feb 20 12:46 sites-enabled
drwxr-xr-x 2 root root 4096 Feb 20 12:46 snippets
-rw-r--r-- 1 root root  664 Apr  6  2018 uwsgi_params
-rw-r--r-- 1 root root 3071 Apr  6  2018 win-utf
patrick@h2866085:~$ 

Then I edit the /etc/nginx.conf as follows.

patrick@h2866085:/etc/nginx$ sudo cat nginx.conf
##
# nginx.conf
##

user www-data; 
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
worker_connections 768;
}

http {

##
# Basic Settings
##

sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;

include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Timeout Settings
##
keepalive_timeout  30s; 
keepalive_requests 30;
send_timeout       30s;
##
# SSL Settings
##

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;

##
# Gzip Settings
##

gzip on;
gzip_vary on;
gzip_comp_level 2;
gzip_min_length  1000;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
gzip_disable "MSIE [4-6] \."; 
##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

patrick@h2866085:/etc/nginx$

In the directory /etc/nginx/sites-available you find the sever configuration files and in /etc/nginx/sites-enabled you find the sym links to the sever configuration files that are enabled on your nginx server.

patrick@h2866085:/etc/nginx$ ls -l sites-available
insgesamt 4
-rw-r--r-- 1 root root 2416 Apr  6  2018 default
patrick@h2866085:/etc/nginx$ ls -l sites-enabled
insgesamt 0
lrwxrwxrwx 1 root root 34 Feb 20 12:46 default -> /etc/nginx/sites-available/default
patrick@h2866085:/etc/nginx$ 

First we deactivate the default site by removing the default symlink in /etc/nginx/sites-enabled.

patrick@h2866085:/etc/nginx$ cd sites-enabled
patrick@h2866085:/etc/nginx/sites-enabled$ ls -l
insgesamt 0
lrwxrwxrwx 1 root root 34 Feb 20 12:46 default -> /etc/nginx/sites-available/default
patrick@h2866085:/etc/nginx/sites-enabled$ sudo unlink default
[sudo] Passwort für patrick: 
patrick@h2866085:/etc/nginx/sites-enabled$ ls -l
insgesamt 0
patrick@h2866085:/etc/nginx/sites-enabled$ 

I want to use my nginx server as reverse proxy server for node application servers running on the localhost. Therefore I create a new file prod-reverse-proxy in the directory sites-available .

patrick@h2866085:/etc/nginx/sites-enabled$ cd ..
patrick@h2866085:/etc/nginx$ 
patrick@h2866085:/etc/nginx$ cd sites-available
patrick@h2866085:/etc/nginx/sites-available$ ls -l
insgesamt 4
-rw-r--r-- 1 root root 2416 Apr  6  2018 default
patrick@h2866085:/etc/nginx/sites-available$ sudo touch prod-reverse-proxy
patrick@h2866085:/etc/nginx/sites-available$ ls -l
insgesamt 4
-rw-r--r-- 1 root root 2416 Apr  6  2018 default
-rw-r--r-- 1 root root    0 Feb 21 08:31 prod-reverse-proxy
patrick@h2866085:/etc/nginx/sites-available$ 

Then I put the following content in the file prod-reverse-proxy and link this file into /etc/nginx/sites-enabled.

patrick@h2866085:/etc/nginx/sites-available$ sudo nano prod-reverse-proxy
server {
        listen 80;
        listen [::]:80;
        server_name digitaldocblog.com www.digitaldocblog.com;

        access_log /var/log/nginx/prod-reverse-access.log;
        error_log /var/log/nginx/prod-reverse-error.log;

        location / {

	proxy_set_header HOST $host;

proxy_set_header X-Forwarded-Proto $scheme;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

	proxy_pass http://127.0.0.1:3000;


  }
}
patrick@h2866085:/etc/nginx/sites-available$ sudo ln -s /etc/nginx/sites-available/prod-reverse-proxy /etc/nginx/sites-enabled/prod-reverse-proxy
patrick@h2866085:/etc/nginx/sites-available$ cd ..
patrick@h2866085:/etc/nginx$ ls -l sites-enabled
insgesamt 0
lrwxrwxrwx 1 root root 50 Feb 21 08:37 prod-reverse-proxy -> /etc/nginx/sites-available/prod-reverse-proxy
patrick@h2866085:/etc/nginx$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
patrick@h2866085:/etc/nginx$ 

The production proxy server is running under the domain server names digitaldocblog.com and www.digitaldocblog.com and is listening on localhost port 80. This production proxy server pass all HTTP traffic from port 80 to a server running on localhost port 3000 (127.0.0.1:3000). Nginx configuration test was tested successful after using nginx -t.

The following commands can be used to check, start and stop the nginx server.

sudo systemctl status nginx

sudo systemctl start nginx 

sudo systemctl stop nginx 

sudo systemctl restart nginx

I restart the nginx server and then check the status. The basic configuration is now complete.

patrick@h2866085:/etc/nginx$ sudo systemctl restart nginx
patrick@h2866085:/etc/nginx$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2021-02-21 08:53:31 CET; 9s ago
     Docs: man:nginx(8)
  Process: 29759 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=0/SUCCESS)
  Process: 29761 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 29760 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 29762 (nginx)
    Tasks: 5 (limit: 60)
   CGroup: /system.slice/nginx.service
           ├─29762 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           ├─29763 nginx: worker process
           ├─29764 nginx: worker process
           ├─29765 nginx: worker process
           └─29766 nginx: worker process

Feb 21 08:53:31 h2866085.stratoserver.net systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 21 08:53:31 h2866085.stratoserver.net systemd[1]: Started A high performance web server and a reverse proxy server.
patrick@h2866085:/etc/nginx$ 

Letsencrypt SSL Certificate

To run your server with HTTPS you must install a certificate from an official Centification Authority (CA). Letsencrypt is such a CA where you can get free certificates. Letsencrypt recommends the use of certbot for easy creation and management of domain certificates.

On the certbot site you can select the webserver and your operating system. I choose nginx and Ubuntu 18.04 LTS bionic and get to a website with the install instructions. Since I am not interested in installing certbot with snap, I choose alternate installation instructions and get to the website with the install instructions of the operating system packages.

I must Install certbot and the certbot nginx plugin with apt.

$ sudo apt update
$ sudo apt-get install certbot
$ sudo apt-get install python-certbot-nginx

Then I run certbot --``nginx to request the letsencrypt certificate for my domain and domain servers.

patrick@h2866085:~$ sudo certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): p.rottlaender@icloud.com

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must
agree in order to register with the ACME server at
https://acme-v02.api.letsencrypt.org/directory
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(A)gree/(C)ancel: A

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: digitaldocblog.com
2: www.digitaldocblog.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for digitaldocblog.com
http-01 challenge for www.digitaldocblog.com
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/prod-reverse-proxy
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/prod-reverse-proxy

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/prod-reverse-proxy
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/prod-reverse-proxy

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://digitaldocblog.com and
https://www.digitaldocblog.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=digitaldocblog.com
https://www.ssllabs.com/ssltest/analyze.html?d=www.digitaldocblog.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/digitaldocblog.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/digitaldocblog.com/privkey.pem
   Your cert will expire on 2021-05-24. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

patrick@h2866085:~$ 

Then I check the new directory and the files in /etc/letsencrypt.

patrick@h2866085:/etc/letsencrypt$ sudo ls -l live
insgesamt 4
drwxr-xr-x 2 root root 4096 Feb 23 06:33 digitaldocblog.com
patrick@h2866085:/etc/letsencrypt$ sudo ls -l live/digitaldocblog.com
insgesamt 4
lrwxrwxrwx 1 root root  42 Feb 23 06:33 cert.pem -> ../../archive/digitaldocblog.com/cert1.pem
lrwxrwxrwx 1 root root  43 Feb 23 06:33 chain.pem -> ../../archive/digitaldocblog.com/chain1.pem
lrwxrwxrwx 1 root root  47 Feb 23 06:33 fullchain.pem -> ../../archive/digitaldocblog.com/fullchain1.pem
lrwxrwxrwx 1 root root  45 Feb 23 06:33 privkey.pem -> ../../archive/digitaldocblog.com/privkey1.pem
-rw-r--r-- 1 root root 682 Feb 23 06:33 README
patrick@h2866085:/etc/letsencrypt$ 

I also check the file prod-reverse-proxy in the directory /etc/nginx/sites-available and see that the file has been updated by certbot. There is a new server section defined (first) an this server is now listening to port 443 ssl and the links to the ssl certificates has been added. The original server section (second) has been changed so that all traffic for hosts digitaldocblog.com and www.digitaldocblog.com will be redirected to the https version of the site (return 301 https://$host$request_uri😉 and requests to port 80 will be answered with 404 site not found.

patrick@h2866085:~$ cd /etc/nginx/sites-available
patrick@h2866085:/etc/nginx/sites-available$ ls -l
insgesamt 12
-rw-r--r-- 1 root root 2416 Apr  6  2018 default
-rw-r--r-- 1 root root 1089 Feb 23 06:34 prod-reverse-proxy

patrick@h2866085:/etc/nginx/sites-available$ sudo cat prod-reverse-proxy
server {
server_name digitaldocblog.com www.digitaldocblog.com;

        access_log /var/log/nginx/prod-reverse-access.log;
        error_log /var/log/nginx/prod-reverse-error.log;

        location / {

	proxy_set_header HOST $host;

proxy_set_header X-Forwarded-Proto $scheme;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

	proxy_pass http://127.0.0.1:3000;


  }
    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/digitaldocblog.com/fullchain.pem; # managed by Certbot
      ssl_certificate_key /etc/letsencrypt/live/digitaldocblog.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {

    if ($host = www.digitaldocblog.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = digitaldocblog.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen 80;
        listen [::]:80;
        server_name digitaldocblog.com www.digitaldocblog.com;
        return 404; # managed by Certbot
}

patrick@h2866085:/etc/nginx/sites-available$ 

To renew all certificates I must run the following command.

$ certbot renew

To renew all certificates automatically I attach the following line to my system crontab.

40 6 * * * root /usr/bin/certbot renew > certrenew_log

The certbot renew command in this example run daily at 6:40 am (in the morning) as root and log the output in the logfile certrenew_log in the home directory of root. The command checks to see if the certificate on the server will expire within the next 30 days, and renews it if so.

Therefore you must edit the /etc/crontab file as follows.

patrick@h2866085:/etc$ sudo nano crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
15 * * * * root cd / && run-parts --report /etc/cron.hourly
28 0 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
9 5 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 0 20 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

40 6 * * * root /usr/bin/certbot renew > certrenew_log
#
patrick@h2866085:/etc$

Separate Letsencrypt SSL Certificate for another proxy server

I currently have 2 server files in my /etc/nginx/sites-available directory.

patrick@h2866085:/etc/nginx/sites-available$ ls -l
insgesamt 8
-rw-r--r-- 1 root root 2416 Apr  6  2018 default
-rw-r--r-- 1 root root 1089 Feb 23 06:34 prod-reverse-proxy
patrick@h2866085:/etc/nginx/sites-available$ 

The file default is disabled.

The file prod-reverse-proxy is enabled. The server run as reverse proxy server for the hostnames digitaldocblog.com and www.digitaldocblog.com already using SSL (see above).

For my hostname dev.digitaldocblog.com, which is a valid subdomain of domain digitaldocblog.com I create a separate proxy server file dev-reverse-proxy in the directory /etc/nginx/sites-available, link this file in ****/etc/nginx/sites-enabled, test the nginx configuration and restart nginx.

patrick@h2866085:/etc/nginx/sites-available$ sudo touch dev-reverse-proxy
patrick@h2866085:/etc/nginx/sites-available$ ls -al
insgesamt 16
drwxr-xr-x 2 root root 4096 Feb 23 13:42 .
drwxr-xr-x 8 root root 4096 Feb 23 06:34 ..
-rw-r--r-- 1 root root 2416 Apr  6  2018 default
-rw-r--r-- 1 root root    0 Feb 23 13:42 dev-reverse-proxy
-rw-r--r-- 1 root root 1089 Feb 23 06:34 prod-reverse-proxy
patrick@h2866085:/etc/nginx/sites-available$ sudo nano dev-reverse-proxy
server {
    listen 80;
    server_name dev.digitaldocblog.com;

        access_log /var/log/nginx/dev-reverse-access.log;
        error_log /var/log/nginx/dev-reverse-error.log;

        location / {

	proxy_set_header HOST $host;

proxy_set_header X-Forwarded-Proto $scheme;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            	proxy_pass http://127.0.0.1:3030;
  }

}
patrick@h2866085:/etc/nginx/sites-available$ cd ..
patrick@h2866085:/etc/nginx$ sudo ln -s /etc/nginx/sites-available/dev-reverse-proxy /etc/nginx/sites-enabled
patrick@h2866085:/etc/nginx$ ls -l sites-enabled
insgesamt 0
lrwxrwxrwx 1 root root 44 Feb 23 13:51 dev-reverse-proxy -> /etc/nginx/sites-available/dev-reverse-proxy
lrwxrwxrwx 1 root root 45 Feb 22 19:33 prod-reverse-proxy -> /etc/nginx/sites-available/prod-reverse-proxy
patrick@h2866085:/etc/nginx$ sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
patrick@h2866085:/etc/nginx$ sudo systemctl restart nginx
patrick@h2866085:/etc/nginx$

Then I run certbot --nginx to request a seperate letsencrypt SSL certificate only for the subdomain dev.digitaldocblog.com.

patrick@h2866085:/etc/nginx$ sudo certbot --nginx
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: digitaldocblog.com
2: dev.digitaldocblog.com
3: www.digitaldocblog.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 2
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for dev.digitaldocblog.com
Waiting for verification...
Cleaning up challenges
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/dev-reverse-proxy

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/dev-reverse-proxy

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations! You have successfully enabled https://dev.digitaldocblog.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=dev.digitaldocblog.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/dev.digitaldocblog.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/dev.digitaldocblog.com/privkey.pem
   Your cert will expire on 2021-05-24. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot again
   with the "certonly" option. To non-interactively renew *all* of
   your certificates, run "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

patrick@h2866085:/etc/nginx$

Then I check the new directory and the files in /etc/letsencrypt.

patrick@h2866085:/etc/letsencrypt$ pwd
/etc/letsencrypt
patrick@h2866085:/etc/letsencrypt$ sudo ls -l live
[sudo] Passwort für patrick: 
insgesamt 8
drwxr-xr-x 2 root root 4096 Feb 23 13:54 dev.digitaldocblog.com
drwxr-xr-x 2 root root 4096 Feb 23 06:33 digitaldocblog.com
patrick@h2866085:/etc/letsencrypt$ sudo ls -l live/dev.digitaldocblog.com
insgesamt 4
lrwxrwxrwx 1 root root  46 Feb 23 13:54 cert.pem -> ../../archive/dev.digitaldocblog.com/cert1.pem
lrwxrwxrwx 1 root root  47 Feb 23 13:54 chain.pem -> ../../archive/dev.digitaldocblog.com/chain1.pem
lrwxrwxrwx 1 root root  51 Feb 23 13:54 fullchain.pem -> ../../archive/dev.digitaldocblog.com/fullchain1.pem
lrwxrwxrwx 1 root root  49 Feb 23 13:54 privkey.pem -> ../../archive/dev.digitaldocblog.com/privkey1.pem
-rw-r--r-- 1 root root 682 Feb 23 13:54 README
patrick@h2866085:/etc/letsencrypt$ 

Directory Setup for node applications

I run my node applications on my server in /var/www/node directory. The node directory is owned by root.

All files of my Node Dev Applications will be copied to /var/www/node/dev using the user patrick. This directory is owned by patrick . According to the configuration in dev-reverse-proxy all http requests coming in for server name dev.digitaldocblog.com will be passed to localhost 127.0.0.1 server port 3030. All dev applications must listen on localhost 127.0.0.1 server port 3030.

All files of my Node Prod Application will be copied to /var/www/node/prod also using the user patrick. The prod directory is also owned by the user patrick. According to configuration in prod-reverse-proxy all http requests coming in for server names digitaldocblog.com and www.digitaldocblog.com will be passed to localhost 127.0.0.1 server port 3000. Prod application must listen on localhost 127.0.0.1 server port 3000.

patrick@h2866085:/etc$ cd /var
patrick@h2866085:/var$ ls
backups  cache  lib  local  lock  log  mail  opt  run  spool  tmp  www
patrick@h2866085:/var$ cd www
patrick@h2866085:/var/www$ ls -l
insgesamt 4
drwxr-xr-x 2 root root 4096 Feb 20 12:46 html
patrick@h2866085:/var/www$ sudo mkdir node
patrick@h2866085:/var/www$ ls -l
insgesamt 8
drwxr-xr-x 2 root root 4096 Feb 20 12:46 html
drwxr-xr-x 2 root root 4096 Feb 23 08:53 node
patrick@h2866085:/var/www$ cd node
patrick@h2866085:/var/www/node$ ls -l
insgesamt 0
patrick@h2866085:/var/www/node$ sudo mkdir prod
patrick@h2866085:/var/www/node$ sudo mkdir dev
patrick@h2866085:/var/www/node$ ls -l
insgesamt 8
drwxr-xr-x 2 root root 4096 Feb 23 08:56 dev
drwxr-xr-x 2 root root 4096 Feb 23 08:56 prod
patrick@h2866085:/var/www/node$ sudo chown patrick:patrick dev
patrick@h2866085:/var/www/node$ sudo chown patrick:patrick prod
patrick@h2866085:/var/www/node$ ls -al
insgesamt 16
drwxr-xr-x 4 root    root    4096 Feb 23 08:56 .
drwxr-xr-x 4 root    root    4096 Feb 23 08:53 ..
drwxr-xr-x 2 patrick patrick 4096 Feb 23 08:56 dev
drwxr-xr-x 2 patrick patrick 4096 Feb 23 08:56 prod
patrick@h2866085:/var/www/node$

MongoDB Community Server Installation

Most node applications also interact with a database. My preferred database is MongoDB. Go on the MongoDB website and read more about the free MongoDB Community Server. To read how to install the mongoDB in your environment go to the MongoDB Documentation Site.

I select the installation instructions of the MongoDB Community Server for my Ubuntu Linux 18.04 bionic LTS system.

Then I Import the public key used by the package management system to verify the package source.

$ wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -

Then I create the list file /etc/apt/sources.list.d/mongodb-org-4.4.list for my version of Ubuntu.

$ echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list

I reload the local package database with apt-get update and install a specific release including all relevant component packages with apt-get install such as

• server
• shell
• mongos
• tools
  
  
  $ sudo apt-get update
  $ sudo apt-get install -y mongodb-org=4.4.4 mongodb-org-server=4.4.4 mongodb-org-shell=4.4.4 mongodb-org-mongos=4.4.4 mongodb-org-tools=4.4.4
  

Mongodb can be started, restarted etc. with the following commands.

:$ sudo service mongod status

:$ sudo service mongod start

:$ sudo service mongod stop

:$ sudo service mongod restart

After the installation the mongoDB server must be started.

patrick@h2866085:~$ sudo service mongod status
● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: https://docs.mongodb.org/manual

patrick@h2866085:~$ sudo service mongod start
patrick@h2866085:~$ sudo service mongod status
● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-02-23 08:49:14 CET; 3s ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 27830 (mongod)
   CGroup: /system.slice/mongod.service
           └─27830 /usr/bin/mongod --config /etc/mongod.conf

Feb 23 08:49:14 h2866085.stratoserver.net systemd[1]: Started MongoDB Database Server.
patrick@h2866085:~$

Then I create the admin user myUserAdmin against the admin db.

patrick@h2866085:~$ mongo
MongoDB shell version v4.4.4
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("8df6776d-4228-43f5-9222-65893d6f146c") }
MongoDB server version: 4.4.4
---
The server generated these startup warnings when booting: 
        2021-02-23T08:49:15.047+01:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
        2021-02-23T08:49:15.718+01:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
        2021-02-23T08:49:15.718+01:00: You are running in OpenVZ which can cause issues on versions of RHEL older than RHEL6
---
---
        Enable MongoDB's free cloud-based monitoring service, which will then receive and display
        metrics about your deployment (disk utilization, CPU, operation statistics, etc).

        The monitoring data will be available on a MongoDB website with a unique URL accessible to you
        and anyone you share the URL with. MongoDB may use this information to make product
        improvements and to suggest MongoDB products and deployment options to you.

        To enable free monitoring, run the following command: db.enableFreeMonitoring()
        To permanently disable this reminder, run the following command: db.disableFreeMonitoring()
---
> use admin
switched to db admin
> db
admin
> db.createUser({ user: "myUserAdmin", pwd: "yourAdminPassword", roles: [{ role: "userAdminAnyDatabase", db: "admin" }, {"role" : "readWriteAnyDatabase", "db" : "admin"}] })
Successfully added user: {
"user" : "myUserAdmin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
},
{
"role" : "readWriteAnyDatabase",
"db" : "admin"
}
]
}
> db.auth("myUserAdmin", "yourAdminPassword")
1
> show users
{
"_id" : "admin.myUserAdmin",
"userId" : UUID("6b32b520-090b-411b-9569-4ecc70109707"),
"user" : "myUserAdmin",
"db" : "admin",
"roles" : [
{
"role" : "userAdminAnyDatabase",
"db" : "admin"
},
{
"role" : "readWriteAnyDatabase",
"db" : "admin"
}
],
"mechanisms" : [
"SCRAM-SHA-1",
"SCRAM-SHA-256"
]
}
> exit
bye
patrick@h2866085:~$

Then I open /etc/mongod.conf file in my editor an add the following lines to the file if it does not already exist security: authorization: enabled.

#security:
security:
  authorization: enabled

Then I restart the mongoDB service.

patrick@h2866085:/etc$ sudo service mongod restart

I create a new database dev-bookingsystem and the Db-User myUserDevBookingsystem as the Db-owner.

patrick@h2866085:/etc$ mongo
MongoDB shell version v4.4.4
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("48d8e3cb-8076-47cb-8166-8e48cfae3d9a") }
MongoDB server version: 4.4.4
> use admin
switched to db admin
> db.auth("myUserAdmin", "yourAdminPassword")
1
> use dev-bookingsystem
switched to db dev-bookingsystem
> db.createUser({ user: "myUserDevBookingsystem", pwd: "yourDbUserPassword", roles: [{ role: "dbOwner", db: "dev-bookingsystem" }] })
Successfully added user: {
"user" : "myUserDevBookingsystem",
"roles" : [
{
"role" : "dbOwner",
"db" : "dev-bookingsystem"
}
]
}
> db
dev-bookingsystem
> exit
bye

Then I create a default collection in dev-bookingsystem.

patrick@h2866085:/etc$ mongo
MongoDB shell version v4.4.4
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("d3494272-8651-4c7e-a781-231f48a60ec4") }
MongoDB server version: 4.4.4
> use admin
switched to db admin
> db.auth("myUserAdmin", "yourAdminPassword")
1
> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
> use dev-bookingsystem
switched to db dev-bookingsystem
> db.runCommand( { create: "col_default" } )
{ "ok" : 1 }
> show collections
col_default
> show dbs
admin              0.000GB
config             0.000GB
dev-bookingsystem  0.000GB
local              0.000GB
> db
dev-bookingsystem
> exit
bye
patrick@h2866085:/etc$

I do the same thing for my production database prod-digitaldocblog.

patrick@h2866085:~$ mongo
MongoDB shell version v4.4.4
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("2402f47f-d1dd-4399-931e-b327ea1cf4b4") }
MongoDB server version: 4.4.4
> use admin
switched to db admin
> db.auth("myUserAdmin", "yourAdminPassword")
1
> show dbs
admin              0.000GB
config             0.000GB
dev-bookingsystem  0.000GB
local              0.000GB
> use prod-digitaldocblog
switched to db prod-digitaldocblog
> db.createUser({ user: "myUserProdDigitaldocblog", pwd: "yourDbUserPassword", roles: [{ role: "dbOwner", db: "prod-digitaldocblog" }] })
Successfully added user: {
"user" : "myUserProdDigitaldocblog",
"roles" : [
{
"role" : "dbOwner",
"db" : "prod-digitaldocblog"
}
]
}
> db
prod-digitaldocblog
> show dbs
admin              0.000GB
config             0.000GB
dev-bookingsystem  0.000GB
local              0.000GB
> db
prod-digitaldocblog
> db.runCommand( { create: "col_default" } )
{ "ok" : 1 }
> show collections
col_default
> show dbs
admin                0.000GB
config               0.000GB
dev-bookingsystem    0.000GB
local                0.000GB
prod-digitaldocblog  0.000GB
> exit
bye
patrick@h2866085:~$

Now I can connect each app with their database with the following string.

mongodb://<youruser>:<yourpassword>@localhost/<yourdatabase>

Deploy the production node app

To deploy my node app into production I first copy the package.json file into the /var/www/node/prod directory and run npm install to install all dependencies. The directory node_modules now exists in my /var/www/node/prod directory.

patrick@h2866085:/var/www/node/prod$ ls -l
insgesamt 116
-rw-r--r--   1 patrick patrick   615 Jan 18 14:36 package.json
patrick@h2866085: npm install
patrick@h2866085:/var/www/node/prod$ ls -l
insgesamt 116
drwxrwxr-x 220 patrick patrick 12288 Feb 23 13:15 node_modules
-rw-r--r--   1 patrick patrick   615 Jan 18 14:36 package.json
-rw-rw-r--   1 patrick patrick 67705 Feb 23 13:15 package-lock.json
patrick@h2866085:

Then I copy all the application files on the server into the directory ****/var/www/node/prod .

patrick@h2866085:/var/www/node/prod$ ls -al
insgesamt 132
drwxr-xr-x  10 patrick patrick  4096 Feb 23 13:17 .
drwxr-xr-x   4 root    root     4096 Feb 23 08:56 ..
drwxr-xr-x   3 patrick patrick  4096 Feb 23 13:16 app
drwxr-xr-x   2 patrick patrick  4096 Feb 23 13:16 config
-rw-------   1 patrick patrick   167 Feb 19  2020 .env
-rw-r--r--   1 patrick patrick    33 Feb 19  2020 .env.example
drwxr-xr-x   2 patrick patrick  4096 Feb 23 13:16 middleware
drwxr-xr-x   2 patrick patrick  4096 Feb 23 13:16 modules
drwxrwxr-x 220 patrick patrick 12288 Feb 23 13:15 node_modules
-rw-r--r--   1 patrick patrick   615 Jan 18 14:36 package.json
-rw-rw-r--   1 patrick patrick 67705 Feb 23 13:15 package-lock.json
-rw-r--r--   1 patrick patrick  1281 Mai 17  2020 prod.digitaldocblog.js
drwxr-xr-x   2 patrick patrick  4096 Feb 23 13:17 routes
drwxr-xr-x   7 patrick patrick  4096 Feb 23 13:17 static
drwxr-xr-x   2 patrick patrick  4096 Feb 23 13:17 views
patrick@h2866085:/var/www/node/prod$ nano .env

I edit my environment variables in the .env file. Here I configure in particular the server port and the database connection string.

patrick@h2866085:/var/www/node/prod$ cat .env
port= 3000
host=127.0.0.1
mongodbpath=mongodb://<youruser>:<yourpassword>@localhost/<yourdatabase>
jwtkey=<yourjwtkey>
patrick@h2866085:/var/www/node/prod$

Finally I start the production application with PM2.

patrick@h2866085:/var/www/node/prod$ pm2 start prod.digitaldocblog.js

How to setup such a production environment will be shown in the first chapter of this documentation.

The express app itself also contains some security features. The app contains a session based user authentication and HTTP headers which help to further secure the app. This is explained of the second chapter of this documentation.

Finally the express app should use the template engine PUG to render the HTML for us. This is describes of the third chapter of this documentation.

1. Setup the production environment

Installation of mongodb

The command brew tap without any arguments lists the GitHub repositories that are currently linked to your Homebrew installation.

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services

The formula mongodb has been removed from homebrew-core. But fortunately the MongoDB Team is maintaining a custom Homebrew tap on GitHub. Read the instructions in the README.md file.

Add the custom tap in the Mac OS terminal and install mongodb.

Patricks-MBP:~ patrick$ brew tap mongodb/brew

Patricks-MBP:~ patrick$ brew tap
homebrew/cask
homebrew/core
homebrew/services
mongodb/brew

Patricks-MBP:~ patrick$ brew install mongodb-community@4.2

After the installation the relevant paths are.

the configuration file (/usr/local/etc/mongod.conf)
the log directory path (/usr/local/var/log/mongodb)
the data directory path (/usr/local/var/mongodb)

Check the services with homebrew

brew services list

Name              Status  User    Plist
mongodb-community started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
  

Start and stop mongodb.

brew services start mongodb-community

brew services stop mongodb-community

Setup mongodb for the project

Setup an admin user

:# mongo

> use admin
switched to db admin

> db
admin

> db.createUser({ user: "adminUser", pwd: "adminpassword", roles: [{ role: "userAdminAnyDatabase", db: "admin" }, {"role" : "readWriteAnyDatabase", "db" : "admin"}] })

> db.auth("adminUser", "adminpassword")
1

> show users
{
    "_id" : "admin.adminUser",
    "userId" : UUID("5cbe2fc4-1e54-4c2d-89d1-317340429571"),
    "user" : "adminUser",
    "db" : "admin",
    "roles" : [
        {
            "role" : "userAdminAnyDatabase",
            "db" : "admin"
        },
        {
            "role" : "readWriteAnyDatabase",
            "db" : "admin"
        }
    ],
    "mechanisms" : [
        "SCRAM-SHA-1",
        "SCRAM-SHA-256"
    ]
}

> exit

Enable authentication with security: authorization: enabled

#> nano /usr/local/etc/mongod.conf

systemLog:
  destination: file
  path: /usr/local/var/log/mongodb/mongo.log
  logAppend: true
storage:
  dbPath: /usr/local/var/mongodb
net:
  port: 27017
  bindIp: 127.0.0.1
security:
  authorization: enabled

Login and authenticate with admin

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b3e7f48a-a05c-4894-87db-996cb34eb1fb") }
MongoDB server version: 4.2.3

> show dbs
> db
test
> use admin
switched to db admin
> db
admin
> show dbs
> db.auth("adminUser", "adminpassword")
1
> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB

> 

If you login you dont see any databases when you call show dbs. The default database you are connected to is test.

Then you connect to admin database. For admin you setup the admin user with the roles userAdminAnyDatabase and readWriteAnyDatabase. With these permissions the admin user can manage users for any database and has read and write access to any database.

So wehen you logon to admin database with the admin user you are able to see all databases with show dbs.

Mongodb comes with 3 standard dbs pre installed:

• admin
• config
• local
  

Create a new database for our express-security app (authenticated as admin user – see above)

> use express-security
switched to db express-security
> db
express-security
> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
> 

The DB which you’ve created is not listed here. We need to insert at least one collection into it for displaying that database in the list.

> db
express-security

> db.createCollection("col_default")
{ "ok" : 1 }

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> exit

Create an owner user for express-security database using the admin user

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("79f79b63-9d08-489f-9e6c-bfc10d8cc09e") }
MongoDB server version: 4.2.3

> db
test

> show dbs

> use admin
switched to db admin

> db.auth("adminUser", "adminpassword")
1

> db
admin

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> use express-security
switched to db express-security

> db.createUser({ user: "owner_express-security", pwd: "passowrd", roles: [{ role: "dbOwner", db: "express-security" }] })
Successfully added user: {
	"user" : "owner_express-security",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "express-security"
		}
	]
}

> db
express-security

> show users
{
	"_id" : "express-security.owner_express-security",
	"userId" : UUID("7a0bafb2-d2ed-4d18-9aba-e2f15a503ec5"),
	"user" : "owner_express-security",
	"db" : "express-security",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "express-security"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

> exit 

Connection string to connect to express-security db using the owner_express-security user:

mongodb://owner_express-security:password@localhost/express-security

Installation of PM2

PM2 is a process manager for Node.js applications. It can daemonize applications to run them as a service in the background.

I install pm2 as a global npm package on my Mac.

Patricks-Macbook Pro:~ patrick$ npm install pm2 -g

Then navigate to your project directory.

Patricks-Macbook Pro:~ patrick$ cd Software/dev/node/articles/2020-05-15-express-security/express-security

Patricks-Macbook Pro:~ patrick$ ls -l 
total 112
drwxr-xr-x    5 patrick  staff    160 30 Mai 05:28 database
drwxr-xr-x  115 patrick  staff   3680 30 Mai 19:35 node_modules
-rw-r--r--    1 patrick  staff  34366 30 Mai 19:35 package-lock.json
-rw-r--r--    1 patrick  staff    339 30 Mai 19:35 package.json
-rw-r--r--@   1 patrick  staff  12343 30 Jun 05:00 secserver.js
drwxr-xr-x    3 patrick  staff     96 30 Mai 05:03 static

Patricks-Macbook Pro:express-security patrick$ 

Start your app using pm2

Patricks-Macbook Pro:express-security patrick$ pm2 start secserver.js

Patricks-Macbook Pro:~ patrick$ pm2 list
┌─────┬──────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name         │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0   │ secserver    │ default     │ 1.0.0   │ fork    │ 640      │ 16h    │ 0    │ online    │ 0%       │ 48.6mb   │ patrick  │ disabled │
└─────┴──────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

Patricks-Macbook Pro:~ patrick$ 

Other comands to control the process manager.

pm2 start secserver.js

pm2 start <id>

pm2 list

pm2 stop <id>

pm2 restart <id>

pm2 show <id>

Installation of nginx

nginx is an open source HTTP and an HTTP Reverse Proxy Server (also mail proxy and load balancer etc.). I install nginx on my Mac with Homebrew.

brew install nginx

You can list the brew services with the following command.

Patricks-MBP:digitaldocblog-V3 patrick$ brew services list

Name              Status  User    Plist
mongodb-community started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.mongodb-community.plist
nginx             started patrick /Users/patrick/Library/LaunchAgents/homebrew.mxcl.nginx.plist
Patricks-MBP:digitaldocblog-V3 patrick$

You can start and stop the brew services as follows.

brew install nginx

brew services start nginx

brew services stop nginx

Setup nginx with TLS/SSL

SSL/TLS works by using the combination of a public certificate and a private key.

The SSL key (private key) is kept secret on the server. It is used to encrypt content sent to clients.

The SSL certificate is publicly shared with anyone requesting the content. It can be used to decrypt the content signed by the associated SSL key.

create private key

Patricks-MBP:express-security patrick$ cd /usr/local/etc/nginx

Patricks-MBP:nginx patrick$ ls -l
total 144
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  5 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  5 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types.default
-rw-r--r--  1 patrick  admin  3106 15 Mai 05:19 nginx.conf
-rw-r--r--  1 patrick  admin  2680  5 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params.default
drwxr-xr-x  3 patrick  admin    96 21 Jan 06:02 servers
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  5 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ mkdir ssl

Patricks-MBP:nginx patrick$ ls -l
total 152
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf
-rw-r--r--  1 patrick  admin  1077  5 Apr 13:18 fastcgi.conf.default
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params
-rw-r--r--  1 patrick  admin  1007  5 Apr 13:18 fastcgi_params.default
-rw-r--r--  1 patrick  admin  2837  5 Apr 13:18 koi-utf
-rw-r--r--  1 patrick  admin  2223  5 Apr 13:18 koi-win
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types
-rw-r--r--  1 patrick  admin  5231  5 Apr 13:18 mime.types.default
-rw-r--r--@ 1 patrick  admin   373 18 Mai 05:38 nginx.conf
-rw-r--r--  1 patrick  admin  2680  5 Apr 13:18 nginx.conf.default
-rw-r--r--  1 patrick  admin  3091 21 Jan 05:40 nginx.conf.working
-rw-r--r--@ 1 patrick  admin  1390 17 Mai 05:19 nginx_old.conf
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params
-rw-r--r--  1 patrick  admin   636  5 Apr 13:18 scgi_params.default
drwxr-xr-x  5 patrick  admin   160 18 Mai 05:20 servers
drwxr-xr-x  4 patrick  admin   128 16 Mai 05:41 ssl
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params
-rw-r--r--  1 patrick  admin   664  5 Apr 13:18 uwsgi_params.default
-rw-r--r--  1 patrick  admin  3610  5 Apr 13:18 win-utf

Patricks-MBP:nginx patrick$ cd ssl

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl genrsa -out privateKey.pem 4096

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem

create certificate signing request (CSR)

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl req -new -key privateKey.pem -out csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 16
-rw-r--r--  1 patrick  admin  1740 16 Mai 05:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem

In case I would like to request an official certificate I must send this csr to the certificate authority. This authority would then create an authority signed certificate from the CSR and send it back to me.

This step is done by ourselves and this is the reason why we create a self signed certificate. This self signed certificate is not a official certificae and not trusted by any browser. It is not useful to use a self signed certificate in production because it produces error messages in the browsers. But for local development a self signed certificate is ok.

So create the self signed certificale. The csr file can then be removed.

create the self signed certificate

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in csr.pem -out selfsignedcertificate.pem -req -signkey privateKey.pem -days 365

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  1740 16 Mai 05:23 csr.pem
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 05:39 selfsignedcertificate.pem

Patricks-MBP:ssl patrick$ rm csr.pem

Patricks-MBP:ssl patrick$ ls -l
total 24
-rw-r--r--  1 patrick  admin  3247 16 Mai 05:22 privateKey.pem
-rw-r--r--  1 patrick  admin  1980 16 Mai 05:39 selfsignedcertificate.pem
 

show certificate details

Patricks-MBP:ssl patrick$ pwd
/usr/local/etc/nginx/ssl

Patricks-MBP:ssl patrick$ openssl x509 -in selfsignedcertificate.pem -text -noout

Configure nginx Servers with SSL

In our configuration we enforce ssl. Therefore we create a default Webserver listening on Port 80 with the server name servtest.rottlaender.lan.

Any request to servtest.rottlaender.lan:80 is redirected to my Reverse Proxy Server which is listening on servtest.rottlaender.lan:443.

The default Webserver is configured in /usr/local/etc/nginx/nginx.conf.

# /usr/local/etc/nginx/nginx.conf
# default Webserver

worker_processes  1;
error_log  /usr/local/etc/nginx/logs/error.log;

events {
    worker_connections  1024;
}

http {
    include       		mime.types;
    default_type  		application/octet-stream;
    sendfile        	on;
    keepalive_timeout  	65;

    access_log  /usr/local/etc/nginx/logs/access.log;

    # default Webserver redirect from port 80 to port 443 ssl
    
    server {
      	listen 80;
    	listen [::]:80;
    	server_name servtest.rottlaender.lan;
    	return 301 https://$host$request_uri;
    }
    
    include servers/*;
    
}

The Reverse Proxy Server is configured in /usr/local/etc/nginx/servers/reverse.

// /usr/local/etc/nginx/servers/reverse
// reverse Proxy Server

server {

    listen      443 ssl;
    server_name servtest.rottlaender.lan;

    ssl_certificate      ssl/selfsignedcertificate.pem;
    ssl_certificate_key  ssl/privateKey.pem;
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  5m;
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
       proxy_pass http://localhost:3300;
       proxy_set_header X-Forwarded-For $remote_addr;
    }
}

The server name servtest.rottlaender.lan is linked in /private/etc/hosts to the ip 192.168.178.20 which is the ip of my computer in my local network.

Patricks-MBP:digitaldocblog-V3 patrick$ cat /private/etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1			localhost
255.255.255.255		broadcasthost
::1             	localhost
192.168.178.20 		servtest.rottlaender.lan

2. Express Secure App (Security Features HTML version)

This is a very simple application but show the basic security features you should use when you run a node app in a production environment.

The app is a website with a simple layout and navigation.

The Home page contain static information and can be accessed by everyone.

On the register page, users can find a form to register. The user data entered here are saved in the database and the user is logged in at the same time. Known users can log in with their email and password after successful registration on the login page. The login and register page can only be accessed if the user is not logged in. If a user is logged in and tries to access the login or register, he will be redirected to the dashboard page.

The dashboard is a personalized area of the website. This area can only be accessed if the user is logged in. If a user is not logged in, he will be redirected to the login page.

Logout is not really a page but a link that contains a logout function. Users who are logged in can log out using this link. Users who are not already logged in will be redirected to the login page.

Download the code from GitHub

Pls. go to my GitHub site and clone the code. Here you find a some inline documentation in the code. The details are explained in this chapter.

Create your app home directory express-security

My app home directory is different to the one that is available after you cloned the code from GitHub.

Patricks-MBP:2020-05-15-express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security

Patricks-MBP:2020-05-15-express-security patrick$ mv node-part-5-express-security-with-db-pug express-security

Patricks-MBP:2020-05-15-express-security patrick$ cd express-security

Patricks-MBP:express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security

Manage environment variables

To manage environment variables for my app I use envy. First you need the files .env and .env.example in the root of your project directory. In .env.example you create a list of all potential environment variables without any values and in .env you use the defined variables and assign the values to them.

Patricks-MBP:express-security patrick$ ls -al
total 152
drwxr-xr-x   14 patrick  staff    448 23 Jun 05:29 .
drwxr-xr-x    4 patrick  staff    128 26 Mai 05:40 ..
-rw-------    1 patrick  staff    181 23 Jun 05:59 .env
-rw-r--r--    1 patrick  staff     53 23 Jun 05:59 .env.example
....

Patricks-MBP:express-security patrick$ cat .env.example
port=
mongodbpath=
sessionsecret=
sessioncookiename=

Patricks-MBP:express-security patrick$ cat .env
port=<YOUR_PORT>
mongodbpath=<YOUR_CONNECTION_STRING>
sessionsecret=<YOUR_SESSION_SECRET>
sessioncookiename=<YOUR_SESSION_COOKIE_NAME>

Patricks-MBP:express-security patrick$ 

Envy must be installed as dependency and required in the main application file secserver.js. Then you can set the environment variables as follows.

// secserver.js

....

// envy module to manage environment variables
const envy = require('envy');

// set the environment variables
const env = envy()
const port = env.port
const mongodbpath = env.mongodbpath
const sessionsecret = env.sessionsecret
const sessioncookiename = env.sessioncookiename
....

Start the MongoDB Server

To run the db server we install mongoose as dependency and require it in the db.js configuration file. The database connection will be initiated with mongoose.connect and the StartMongoServer function will be exported to be called in the main application file secserver.js.

const envy = require('envy')
const env = envy()

const mongodbpath = env.mongodbpath

const mongoose = require('mongoose');
mongoose.set('useNewUrlParser', true);
mongoose.set('useUnifiedTopology', true);

const StartMongoServer = async function() {
  try {

    await mongoose.connect(mongodbpath)
    .then(function() {
      console.log(`Mongoose connection open on ${mongodbpath}`);
    })
    .catch(function(error) {
      console.log(`Connection error message: ${error.message}`);
    })

  } catch(error) {
    res.json( { status: "db connection error", message: error.message } );
  }

};

module.exports = StartMongoServer;

Authentication and authorization

For user authentication we use the module express-session and to store session data in the session store in our database we use connect-mongodb-session. Therefore we install these modules as dependencies in our project and require the modules in our secserver.js main application file.

Then we create with new MongoDBStore a session storage in our MongoDB to store session data in collection col_sessions. errors are catched with store.on.

We use the session in our app with app.use( session({...}) ). With every request to our site a new session object is created with a unique session ID which include a session cookie object. The session object has keys options and the values for each key define how to deal with the session object. The session ID is created and signed using the secret option. We use name to provide a session cookie name and store to define where the session object should be stored (in case we store the session).

We can access the session object with req.session and the session ID with req.session.id. With every request we have a new session and this new session will be created but not stored anywhere so far. We say the session is uninitialized. The saveUninitialized false option ensure that a session will only be written to the store in case it has been modified. What does this mean?

We can modify the session when we store additional data into it. We always do this when the user is logging in via the login or the register route. When we post the data from the login- or from the registration-form to the server we call loginUser or the createUser module which is defined in database/controllers/userC.js. Both modules do basically the same thing: They create a userData Object and store the userData object into the session object and redirect the user to the dashboard when login or registration was successful.

....

var userData = { 
	userId: user._id, 
	name: user.name, 
	lastname: user.lastname, 
	email: user.email, 
	role: user.role 
	}

    req.session.userData = userData

    res.redirect('/dashboard')

....

If the user is successfully logged in the session is initialized (modified), the session object incl. the userData object are stored into the store and a cookie is stored into the requesting browser. The content of the cookie is only a hash of the session Id and with each request of a logged in user the session on the server is looked up.

The cookie in the browser will live max 1 week as we defined in the cookie object maxAge set to 1 week. Because of the cookie option sameSite true the cookie scope is limited to the same site.

Then the resave false option ensures that the session will not be updated with every request. This mean the session ID that has been created when the user has logged in will be kept until the user is logged out again.

// secserver.js

....

// server side session and cookie module
const session = require('express-session');
// mongodb session storage module
const connectMdbSession = require('connect-mongodb-session');

....

// Create MongoDB session storage
const MongoDBStore = connectMdbSession(session)
const store = new MongoDBStore({
  uri: mongodbpath,
  collection: 'col_sessions'
});

// catch errors in case store creation fails
store.on('error', function(error) {
  console.log(`error store session in session store: ${error.message}`);
});

// Create the express app
const app = express();

....

// use session to create session and session cookie
app.use(session({
  secret: sessionsecret,
  name: sessioncookiename,
  store: store,
  resave: false,
  saveUninitialized: false,
  // set cookie to 1 week maxAge
  cookie: {
    maxAge: 1000 * 60 * 60 * 24 * 7,
    sameSite: true
  },

}));

....

Secure HTTP headers

Response headers are HTTP header that come with the HTTP response from the server to the client. The http response header contain data that could possibly damage the integrity of the client. It is therefore important to secure the response header of your application.

To secure the http response headers I user the module helmet. This is a relatively easy-to-use module consisting of various middleware functionalities to secure various http response headers.

First we install helmetas a dependency of our project. Then we require helmet and use helmet right after we created the app.

// secserver.js

// hTTP module
const http = require('http');
// express module
const express = require('express');
// hTTP header security module
const helmet = require('helmet');

// Create the express app
const app = express();

....

// use secure HTTP headers using helmet
app.use(helmet())

Using simply app.use(helmet()) set the http header security to default. Then the following 7 out 11 helmet features can be used.

1. dnsPrefetchControl controls browser DNS prefetching
2. frameguard to prevent clickjacking
3. hidePoweredBy to remove the X-Powered-By header
4. hsts for HTTP Strict Transport Security
5. ieNoOpen sets X-Download-Options for IE8+
6. noSniff to keep clients from sniffing the MIME type
7. xssFilter adds some small XSS protections
   

When we then request our home page to retrieve the http headers using curl -k --head in the terminal we see the following output.

Patricks-MBP:express-security patrick$ curl -k --head https://servtest.rottlaender.lan
HTTP/1.1 200 OK
Server: nginx/1.19.0
Date: Fri, 26 Jun 2020 16:14:14 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 1734
Connection: keep-alive
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
ETag: W/"6c6-U2uWyDNyzlyBAbSI/Quxqo9RRQE"

Patricks-MBP:express-security patrick$ 

App routing

get routes: We have the following get routes and navigation.

• Home (/)
• Login (/login)
• Register (/register)
• Dashboard (/dashboard)
• Logout (/logout)
  

get routes involve an optional middleware and respond HTML back to the client.

app.get('/<route>', <optional: someMiddleware>, (req, res) => {
	
	res.send(`<some HTML>`)
})

I will not explain the HTML and css in detail. But as everyone can see, the HTML is the same for every route except for the <body>. Of course, this is not very nice and becomes a bit more efficient with the use of a template engine, which I will explain below using the PUG template engine. I will then rebuild the app using PUG.

Lets have a look at the middleware. If a request is made for a route and a middleware function is included, the middleware function is first executed before the next routing function function(req, res) is called. A condition is built into the middleware function which is checked. My middleware is built so that in case the condition is true the middleware code is executed directly and the next routing function is omitted. If the condition is false, the next routing function function(req, res) is called.

I have built 2 different middleware functions which each check

middleware 1 (login- and register route): a user is logged in

// secserver.js
....
// middleware 1 to redirect authenticated users to their dashboard
const redirectDashboard = (req, res, next) => {
  if (req.session.userData) {
    res.redirect('/dashboard')

  } else {
    next()
  }
}
....

If a user is logged in the request should be redirected to the dashboard route, in any other case (user is not logged in) the next routing function function(req, res) is called and respond the HTML to the browser. This middleware 1 is included in the /login- and /register route. This mean logged in users will be redirected to their dashboard, not logged in users will see the login- and register form.

middleware 2 (dashboard- and logout route): a user is not logged in.

// secserver.js
....

// middleware 2 to redirect not authenticated users to login
const redirectLogin = (req, res, next) => {
  if (!req.session.userData) {
    res.redirect('/login')
  } else {
    next()
  }
}
....

If a user is not logged in the request should be redirected to the login roure, in any other case (user is logged in) the next routing function function(req, res) is called and respond the HTML to the browser. This middleware 2 is included in the /dashboard- and /logout route. This mean not logged in users will be redirected to login route, logged in users will see the dashboard- or can log themselves out.

post routes: We have the following post routes.

• /login
• /register
  

The login and the register get routes contain a form in the HTML. With these forms the user provide the data to login and for user registration. When the user click the send button the action is to call the login- or register post route. This will happen for all not logged in users. The login and the register get routes have the middleware redirectDashboardto redirect the user to the dashbard if the user is already logged in.

// secserver.js
....

app.get('/login', redirectDashboard, (req, res) => {
....
res.send(`
....
<div class="form">
	<form id='register_form' method='post' action='/register'>
	......
	<label for='send'>
   		<input class='sendbutton' type='submit' name='send' value='Send'>
	</label>
	</form>
</div>

`)
)}
....

app.get('/register', redirectDashboard, (req, res) => {
....
res.send(`
....
<div class="form">
	<form id='login_form' method='post' action='/login'>
	......
	<label for='send'>
   		<input class='sendbutton' type='submit' name='send' value='Send'>
	</label>
	</form>
</div>
`)
)}
.....

The post routes contain functions to login- (loginUser) or register (createUser) the user.

// secserver.js
....
// Post routes to manage user login and user registration
app.post('/login', userController.loginUser);

app.post('/register', userController.createUser);
....

The loginUser function is defined in the user controller database/controllers/userC.js. This function lookup a user in the database based on the email address that has been provided by the request body. The data that are attached to the request body have been provided by the user vie the login form of the app. If no user could be found in the database login is not possible. If a user exist with the given email address then the provided password will be compared with the one stored in the database. If the password match fail login is not possible because the provided password is wrong. in any other case, the login takes place and a userData object is created and attached to the session object.

// database/controllers/userC.js

User.findOne({ email: req.body.email }, function(error, user) {
  if (!user) {
    res.status(400).send({ code: 400, status: 'Bad Request', message: 'No User found with this email' })

    } else {
      if (bcrypt.compareSync(req.body.password, user.password)) {
    
        var userData = { userId: user._id, name: user.name, lastname: user.lastname, email: user.email, role: user.role }

          req.session.userData = userData

          res.redirect('/dashboard')

        } else {
          res.status(400).send({ code: 400, status: 'Bad Request', message: 'Wrong User password' })
        }

      }
    })

  }

The createUser function is also defined in the user controller database/controllers/userC.js. This function create a new User object based on the data from the request body provided by the user via the form. The provided password will be hashed and stored together with all other data into the database. Finally a userData object is created and attached to the session and the user will be redirected to the dashboard after the registration was successful.

// database/controllers/userC.js

createUser: async function (req, res) {
    // assign input data from request body to input variables
    const name = req.body.name
    const lastname = req.body.lastname
    const email = req.body.email
    const password = req.body.password
    const role = req.body.role

    const newUser = new User({
      name: name,
      lastname: lastname,
      email: email,
      password: password,
      role: role
    })

    newUser.password = await bcrypt.hash(newUser.password, saltRounds)

    await newUser.save(function(err, user) {
          if (err) {
            // if a validation err occur end request and send response
            res.status(400).send({ code: 400, status: 'Bad Request', message: err.message })
          } else {
            // req.session.userId = user._id

            var userData = { userId: user._id, name: user.name, lastname: user.lastname, email: user.email, role: user.role }

            req.session.userData = userData

            res.redirect('/dashboard')
          }
        })
  },

And we have a default get route.

• /favicon.ico
  

Browsers will by default try to request /favicon.ico from the root of a hostname, in order to show an icon in the browser tab. As we dont use favicon so far we must avoid that these requests returning a 404 (Not Found). Here The /favicon.ico request will be catched and send a 204 No Content status.

// secserver.js
....
app.get('/favicon.ico', function(req, res) {
    console.log(req.url);
    res.status(204).json({status: 'no favicon'});
});
....

3. Express App (Pug Template Version)

From a functional point of view this app is pretty much the same app then the HTML version. The difference is that we use PUG templates instead of HTML in each res.send().

Setup a seperate Database

For the PUG version of my app I set up a new database to manage the users and the sessions.

#> mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("b3e7f48a-a05c-4894-87db-996cb34eb1fb") }
MongoDB server version: 4.2.3

> db
test

> use admin
switched to db admin

> db
admin

> db.auth("adminUser", "adminpassword")
1

> show dbs
admin               0.000GB
config              0.000GB
express-security    0.000GB
local               0.000GB

> use express-security-pug
switched to db express-security-pug

> db.createUser({ user: "owner_express-security-pug", pwd: "passowrd", roles: [{ role: "dbOwner", db: "express-security-pug" }] })

Successfully added user: {
    "user" : "owner_express-security-pug",
    "roles" : [
        {
            "role" : "dbOwner",
            "db" : "express-security-pug"
        }
    ]
}

> db
express-security-pug

> exit

Connection string to connect to express-security-pug db using the owner_express-security-pug user.

mongodb://owner_express-security-pug:password@localhost/express-security-pug

Download the code from GitHub

Pls. go to my GitHub site and clone the code. Here you find some inline documentation in the code.

Create your app home directory express-security-pug

My app home directory is different to the one that is available after you cloned the code from GitHub.

Patricks-MBP:2020-05-15-express-security patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security

Patricks-MBP:2020-05-15-express-security patrick$ mv node-part-5-express-security-with-db-pug express-security-pug

Patricks-MBP:2020-05-15-express-security patrick$ cd express-security-pug

Patricks-MBP:express-security-pug patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security-pug

Install PUG and use it in your app

First we install PUG as a dependency.

Patricks-MBP:express-security-pug patrick$ pwd
/Users/patrick/software/dev/node/articles/2020-05-15-express-security/express-security-pug

Patricks-MBP:express-security-pug patrick$ npm install pug --save

PUG is already fully integrated into Express. Pls. read the documentation how to use template engines in Express.

After you installed PUG the view engine must be set in your main application file secserverpug.js.

// secserverpug.js
....
// use Pug Template Engine
app.set('view engine', 'pug')
app.set('views', './views')
....

These instructions tell your app that PUG template engine is used and that the templates can be found in /views directory.

PUG Directory setup

In /views I setup the templates for home, login, registration and an error template.

In /views/includes I setup the files containing HTML or JavaScript. These can be included in the templates.

Patricks-MBP:express-security-pug patrick$ ls -l
total 128
-rw-r--r--    1 patrick  staff    771  1 Jul 06:04 README.md
drwxr-xr-x    5 patrick  staff    160 29 Jun 05:26 database
drwxr-xr-x  150 patrick  staff   4800 29 Jun 06:11 node_modules
-rw-r--r--    1 patrick  staff  47547 29 Jun 06:11 package-lock.json
-rw-r--r--    1 patrick  staff    367 29 Jun 06:11 package.json
-rw-r--r--    1 patrick  staff   4393  2 Jul 05:34 secserverpug.js
drwxr-xr-x    3 patrick  staff     96 29 Jun 05:26 static
drwxr-xr-x    8 patrick  staff    256  2 Jul 05:45 views

Patricks-MBP:express-security-pug patrick$ ls -l views
total 40
-rw-r--r--  1 patrick  staff   549 30 Jun 05:35 dashboard.pug
-rw-r--r--  1 patrick  staff   522  2 Jul 05:50 err.pug
-rw-r--r--  1 patrick  staff   420 29 Jun 05:39 home.pug
drwxr-xr-x  6 patrick  staff   192 29 Jun 05:17 includes
-rw-r--r--  1 patrick  staff   735 30 Jun 05:02 login.pug
-rw-r--r--  1 patrick  staff  1067 30 Jun 05:08 register.pug

Patricks-MBP:express-security-pug patrick$ ls -l views/includes
total 32
-rw-r--r--  1 patrick  staff   76 29 Jun 05:39 foot.pug
-rw-r--r--  1 patrick  staff  167 29 Jun 05:24 head.pug
-rw-r--r--  1 patrick  staff  489  2 Jul 05:13 nav.pug
-rw-r--r--  1 patrick  staff  420 29 Jun 05:08 script.js

Patricks-MBP:express-security-pug patrick$ 

The responsive Website Design

Each site like home, login, register and dashboard has a specific site template in /views directory. The site content will be defined in the main section of each template. PUG enables files with HTML or JavaScript to be included. This makes the site templates clear and easy to maintain. The includes are located in /views/includes directory.

The website is build based on a grid design and each site template has the following structure.

doctype html

HTML

	Head
		include includes/head.pug	
	
	Body
		
		Grid-Container
			
				Header
					include includes/nav.pug
			
				Main
					... site template specific HTML ...
			
				Footer
					include includes/foot.pug
					
		<script>
			  include includes/script.js

The design of the website is defined in the css in static/css/style.css.

Here in the css we define the Site Structure as grid areas consisting of header, main and footer and link them to the grid-container.

....

.header { grid-area: header; background-color: #ffffff; border-radius: 5px;}
.main { grid-area: main; background-color: #ffffff; border-radius: 5px;}
.footer { grid-area: footer; background-color: #ffffff; border-radius: 5px;}

.grid-container {
  display: grid;
  grid-template-areas:
      "header"
      "main"
      "footer";
  grid-gap: 5px;
  background-color: #d1d1e0;
  padding: 50px;
}

....

The Navigation is defined in the Header area of the Grid-Container and the HTML comes into the template via include includes/nav.pug.

// includes/nav.pug

//(this) refers to the DOM element to which the onclick attribute belongs to
// the a DOM element will be given as parameter to the function

a(class="burgericon" onclick="myFunction(this)")
  div(class='burgerline' id='bar1')
  div(class='burgerline' id='bar2')
  div(class='burgerline' id='bar3')

a(class='link' href='/') Home
a(class='link' href='/login') Login
a(class='link' href='/register') Register
a(class='link' href='/dashboard') Dashboard
a(class='link' href='/logout') Logout

So the navigation design is then defined in the css. Each navigation object is an a link. We have a links with class link and burgericon. The burgericon is used to open the navigation bar onclick when the screen is smaller than 600px (like iphone displays etc., explained below), is cosisting of 3 burgerlines and these lines are created using 3 div objects with class burgerline. The burgelines will be transformed with speed 0.4s when you click on the burgericon (explained below). The burgericon is not visible and aligned on the right edge. All other navigation links are visible and aligned on the left edge.

/* static/css/style.css */
....

/* style the navigation links with float on the left (side by side) */
.header a.link {
  float: left;
  display: block;
  padding: 14px 16px;
  text-decoration: none;
  font-size: 1.4vw;
  color: #28283e;
}

/* hover effect for each navigation link */
.header a.link:hover {
  background-color: #28283e;
  color: #ffffff;
}

/* style the burgericon link on the right */
.header a.burgericon {
  float: right;
  display: none;
  padding: 14px 16px;
}

/* style each burgerline that create the burgericon */
.burgerline {
  width: 35px;
  height: 5px;
  background-color: #28283e;
  margin: 6px 0;
  transition: 0.4s;
}
....

When the display screen is lower than 600px the navigation links will not be shown and the burgericon (on the right side) will be faded in instead.

/* static/css/style.css */
....
/* for screens up to 600px remove the navigation links and show the burgericon instead */
@media screen and (max-width: 600px) {
  .header a.link { display: none; }
  .header a.burgericon { display: block; }
}
....

When you click on the burgericon the burgerlines will be transformed so that you will see a cross instead of the hamburger like icon. The 2nd burgerline with id='bar2' will not be shown at all while the other 2 burgelines will be rotated 45 degrees counterclockwise (burgerline with id='bar1') and clockwise (burgerline with id='bar1').

/* static/css/style.css */
....
/* style burgerlines after on onclick event */
/* the .change class will be added onclick with classList.toggle in the JavaScript */
/* rotate first bar */
.change #bar1 {
  /* rotate -45 degrees (counterclockwise) move 15px down in Y-direction */
  transform: rotate(-45deg) translateY(15px);
}
/* fade out the second bar */
.change #bar2 {
  opacity: 0;
}
/* rotate third bar */
.change #bar3 {
  /* rotate +45 degrees (clockwise) move 15px up in Y-direction */
  transform: rotate(45deg) translateY(-15px);
}
....

After clicking on the burgericon, the burgerlines are transformed as described. The links of the navigation menu are displayed one below the other (float none) and aligned left.

/* static/css/style.css */
....
/* for screens up to 600px and after onclick event the responsive class will be added to the header */
@media screen and (max-width: 600px) {
  /* show navigation links left with no float (links shown among themselves) */
  .header.responsive a.link {
    float: none;
    display: block;
    text-align: left;
  }
}
....

All onclick functionalities are controlled by the javascript which is embedded in the HTML of each site template (pls see above includes/nav.pug). In the HTML, the onclick event is initiated in the burgericon link and the function myFunction is called with onclick =" myFunction(this) ". With the parameter this the entire burgericon object is transferred to the javascript function.

With each click on the burger icon, the class change is added to each burgerline or, if available, removed. This is done by the toggle() function. If change is set, the hamburg icon is transformed into a cross according to the specification in the css (see above). If change is withdrawn with a new click, the hamburger icon is displayed again.

But it happens even more in the javascript when you click on the hamburger icon. The element that has the id responsivenav is searched for and the variablereponsiveNavElement is assigned to this element. Is the class of the reponsiveNavElement header the classresponsive is added after clicking on the hamburger icon. If the class responsive is set, as described above, the links of the navigation menu are displayed one below the other (float none) and aligned left. So it applies in the css .header.responsive a.link {....}

In all other cases only the class header is set. So it applies in the css .header a.link {....} and the navigation links are not shown.

// includes/script.js

// the (burgerlines) parameter represent the DOM element that has been given to the function
function myFunction(burgerlines) {
  burgerlines.classList.toggle('change');

  var reponsiveNavElement = document.getElementById('responsivenav');
    if (reponsiveNavElement.className === 'header') {
      reponsiveNavElement.classList.add('responsive')
    } else {
      reponsiveNavElement.className = 'header';
    }
  }

Finally at the end of the css we define the defaults for h1, for our text content, the forms, the input fields and the send buttons.

Summary and Outlook

In this part 4 of my little node.js series we have seen how to setup a production ready environment for our express app. I showed this using a Mac OS, but in principle this setup also applies to Linux, for example.

The basic setup is, to put it simply, the app runs as a service on the server in the background using a Process Manager, but has no interface to the client. This client interface regulates a reverse proxy which is upstream of the app and accepts all requests and forwards them to the app, as well as the responses from the app back to the client. The communication is SSL/TLS secured.

At the center of the setup is a separate local MongoDB that manages all application data. In our example, these are the users, but also the sessions. I prefer to set up my own MongoDB on my server but of course it is conceivable to use a cloud-based solution or to install another database locally.

The express app itself uses secure HTTP response headers so that HTTP attacks like clickjacking, MIME type sniffing or some smaller XSS attacks on the client are made as difficult as possible. Access to personal areas of the application is secured by session-based authentication and authorization. The session-relevant data is stored in the database and not in the browser cookie, which means additional security with regard to attacks on the client. The browser cookie only contains a hash of the session ID to query the relevant user data from the database.

I would like to end my node.js series with Part 4. I discussed and demonstrated the basic concepts and procedures in parts 1 to 4. Of course there will be other interesting articles on the topic node.js and web programming on Digitaldocblog. Just take a look.

The creation of the database and the cloning of the application root directory from my GitHub page will be briefly described in the following step-by-step instructions. This is followed by a description of the application functionality. Then I will explain more generally how data is modeled in a database-based Expressjs application with the help of mongoose and how the data is accessed so that the more basic software architecture is understood. The relevant code of our blog app is commented. In this respect, the explanations of the individual code passages can be found in the inline documentation.

Create a MongoDB Database

After the installation of MongoDB you have started the mongod service on your system. It is very important that you setup MongoDB with client authentication enabled. You have also created an admin user with a password on your admin default db. Pls. follow the description in my article Mongodb and Mongoose on Ubuntu Linux.

Then you create a database for your project and a user for your database using the mongo command on the console.

Patricks-MBP:digitaldocblog-V3 patrick$ mongo

MongoDB shell version v4.2.3
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("4e8c592e-104d-48ef-b495-62129e446d23") }
MongoDB server version: 4.2.3

> db
test

> show dbs

> use admin
switched to db admin

> db.auth("admin", "YOUR-ADMIN-PASSWD")
1

> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB

> use YOUR-DB-NAME
switched to db YOUR-DB-NAME

> db
YOUR-DB-NAME

> db.createCollection("col_default")
{ "ok" : 1 }

> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
YOUR-DB-NAME	    0.000GB

> db.createUser({ user: "YOUR-DB-USER", pwd: "YOUR-DB-USER-PASSWD", roles: [{ role: "dbOwner", db: "YOUR-DB-NAME" }] })
Successfully added user: {
	"user" : "YOUR-DB-USER",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "YOUR-DB-NAME"
		}
	]
}

> show users
{
	"_id" : "YOUR-DB-NAME.YOUR-DB-USER",
	"userId" : UUID("ef080b98-849f-41f4-b561-a658a88e4595"),
	"user" : "YOUR-DB-USER",
	"db" : "YOUR-DB-NAME",
	"roles" : [
		{
			"role" : "dbOwner",
			"db" : "YOUR-DB-NAME"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

> show dbs
admin               0.000GB
config              0.000GB
local               0.000GB
YOUR-DB-NAME	    0.000GB
> 

> exit
bye

If you enter the command mongo on the console, the MongoDB client connects to the MongoDB service. The MongoDB client is the so-called MongoDB shell with which the user can work on the console. After the connection is established, the MongoDB shell is displayed in the terminal and with the command db we see the output test.

The db command shows the database you are currently connected to. test is the default database and is actually no longer used. The show dbs command shows all available databases. Since we have activated client authentication nothing is shown here because we are not yet authenticated.

With use admin we switch to admin db and authenticate ourselves with db.auth. After we have entered the db.auth command in the MongoDB shell as described, the shell shows us a 1. This shows that the authentication was successful. As admin user we can now show all databases with show dbs.

With the command use YOUR-DB-NAME we create the database YOUR-DB-NAME and also switch from the database admin to the database YOUR-DB-NAME. With the command db.createCollection I always create a default collection in YOUR-DB-NAME to display the new database YOUR-DB-NAME with the show dbs command in the MongoDB shell. If you simply create the database and no collection the new database is not shown with the show dbs command.

Then I create a user with db.createUser for my new database YOUR-DB-NAME, assign the role dbOwner to this user and the password YOUR-DB-USER-PASSWD.

The database YOUR-DB-NAME is now created and with exit we leave the MongoDB shell.

Create the application root directory

Clone the Simple Blog App from my GitHub page and then switch to the directory node-part-3-express-blog-with-db-V2.

Patricks-MBP:~ patrick$ cd node-part-3-express-blog-with-db-V2

The application root directory then looks like this.

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l
total 72
-rw-r--r--   1 patrick  staff    768 9 Apr 05:19 README.md
-rw-r--r--   1 patrick  staff   2162 9 Apr 05:19 blog.js
drwxr-xr-x   6 patrick  staff    192 9 Apr 05:19 database
drwxr-xr-x   3 patrick  staff     96 9 Apr 05:19 modules
drwxr-xr-x  77 patrick  staff   2464 9 Apr 05:19 node_modules
-rw-r--r--   1 patrick  staff  22302 9 Apr 05:19 package-lock.json
-rw-r--r--   1 patrick  staff    188 9 Apr 05:19 package.json

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l database
total 16
drwxr-xr-x  4 patrick  staff  128  9 Apr 05:19 controllers
-rw-r--r--  1 patrick  staff  704  9 Apr 05:19 db_.js
drwxr-xr-x  4 patrick  staff  128  9 Apr 05:19 models

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l database/controllers
total 40
-rw-r--r--  1 patrick  staff  9464 9 Apr 05:19 blogController.js
-rw-r--r--  1 patrick  staff  5306 9 Apr 05:19 userController.js

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l database/models
total 16
-rw-r--r--  1 patrick  staff  1574 9 Apr 05:19 blogModel.js
-rw-r--r--  1 patrick  staff  1216 9 Apr 05:19 userModel.js

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ ls -l modules
total 8
-rw-r--r--  1 patrick  staff  175  9 Apr 05:19 logger.js

Patricks-MBP:node-part-3-express-blog-with-db-V2 patrick$ 

Read the README.md. After you run npm install You must adapt your database connection string and rename database/db_.js into database/db.js. Your database connection string ist as follows.

mongodb://YOUR-DB-USER:YOUR-DB-USER-PASSWD@localhost:27017/YOUR-DB-NAME

The application should run with npm blog.js. The application code is explained and commented inline. In addition, I will explain details in the following chapters.

How does the application work ?

The application looks like a very simple web application but already contains a lot of program logic. This program logic is mainly contained in the two user and blog controllers.

The application data are users and blogs. These data are stored in a MongoDB and can be created, changed or deleted by the application. When data is entered, i.e. when data sets are created and when data fields are changed, input validation takes place. This input validation is based on the mongoose built in input validation and especially for string values on the mongoose custom validation, whereby it is checked whether the input of a string corresponds to a defined regular expression (regex for ECMA Java Script).

The primary identifier for users is the email address. The users must be unique and can only be created if the email address is not already assigned to an existing user.

The users can be blog authors and therefore they have a reference to each blogid whose author they are. Whenever a blog post is created, the application checks whether a user with the specified email address already exists and whether the email address already belongs to an existing user. If the user does not exist or the specified email address belongs to an existing user, the blog post cannot be created. If a user exists and it is also his own email address, the new blog post can be created and the author’s name, first name and email address are stored on the blog in the database. In parallel the blogid of the new blog post is stored as reference on the user object.

We thus have a reference from users to their blogs where they are authors and a reference from blogs to the users or blog authors. When creating a blog, it is a condition that the author exists as a user in the database, but it may still be that we have blogs with authors in the database for which there is no longer a user. This is because it is possible to delete a user without deleting all blogs where the user is the author. Therefore it may still be that we have blogs with authors in the database for which there is no longer a user.

Consequently, when a blog is deleted, it is checked whether the author exists as a user. If the user no longer exists, the blog entry will still be deleted. If there is a user in the database, the blog post is deleted and the reference blogid on the user is deleted at the same time.

But we can also update the email address of a user. This is a very realistic use case because email addresses can change for users and consequently a user would like to change their changed email address accordingly in our application. Because the email address is the unique identifier of a user, we have to change the email address both on the user and on all blogs where the user is entered as the author.

Our application can also display blogs. All blogs are displayed or only the blogs for a specific year or month or only for a defined date. In order to display the blogs in this way, the blog data are selected accordingly from the database.

In addition, the application also displays a homepage and an about page, which have no further significance.

Data Models

To organize our application data in the MongoDB we define so called models using Mongoose. Models contain the essential data structure and data definitions that a database collection should have. Our database will have 2 collections one for the users and one for the blogs.

For each database collection exactly one model is defined in a seperate file. In our example, we need one model for each of the two collections col_users and col_blogs.

For col_users the model is defined in the file

• database/models/userModel.js
  

for the col_blogs the model is defined in the file

• database/models/blogModel.js.
  

I would like to show a standard model here.

// model.js
....

mongoose = require('mongoose')
var Schema = mongoose.Schema

var dataSchema = new Schema ({ key1:..., key2:..., ..., keyN:...}) 

var Data = mongoose.model('col_data', dataSchema)

module.exports = Data

With mongoose.Schema we create a database Schema. With new Schema(…) we define exactly what the Schema looks like and which keys or data fields are defined for each data object that will be stored in the collection. The new Schema is stored in a variable that typically describes the purpose of the new Schema (userSchema or blogSchema in our app). With mongoose.model() we create the model and map it to the database collection. The first argument in the mongoose.model() function is the singular name of the database collection. So if we create the model with col_data mongoose is looking for col_datas collection in the database. The second argument in the callback is the new Schema that we defined before. This model will be stored in the Variable Data and exported with module.exports.

It is extremely important that all input data are validated before the data is saved in the database. This ensures that the application only processes validated data.

In the model definition of or blog application, I mark all auto data fields with a preceding underscore. With auto data fields, the value is either created by the database (like the id) or a default value is always set (like created). This means that these data fields are never manipulated by data input.

Input data fields are subject to mongoose built-in validation. The topic of input validation is described in great detail in the mongoose documentation. Nevertheless, I would like to go into a few points of input validation here.

Basically the specification of validation properties takes place in the database Schema definition per key of the data object. We have built-in validation properties or short validators like required, minlength or maxlength but also the option to define a custom validation function using the property validate. Here in the example below the custom validation function of the key „name“ checks the input for the match with a regex. If the input value does not correspond to the regex, the function return false and the validation has failed. If the function return true, the data can be saved and the validation was successful.

// model.js

....

mongoose = require('mongoose')
var Schema = mongoose.Schema

var dataSchema = new Schema ({ 

 // define the auto data fields
  _id: {
    type: Schema.ObjectId,
    auto: true
  },

  _created: {
    type: Date,
    default: Date.now()
  },

  // define the input data fields
  name: {
    type: String,
    required: true,
    // input string validation function
    // input sting must match regex criteria
    validate: function(name) {
      return /^[a-zA-Z0-9üÜäÄöÖ.’-]+$/.test(name)
    }
  },  
....

})

....

Data Controllers

To organize our application data accesses and data manipulations using Mongoose we define so called data controllers in separate files.

In principle, each data model that we have defined in our data model files are representing a data perimeter. A data perimeter can be accessed via so-called controller modules which are defined in separate controller files. These controller modules are in turn defined as controller functions in the corresponding data controller.

These controller functions define the data operations like search queries, create, update and delete data within the controller functions. These data operations are defined in a try … block and if an error occurs when accessing the data, this error is forwarded to the catch … block. This catch block then takes over the error handling. I do this try catch thing for all operations except for save(). With save() errors are handled directly in the callback.

In addition to the req and res parameter, the controller functions also receive next as a parameter. This is required to call the default error handler of the server which is defined in our main application file blog.js.

If the request cannot be ended with a response due to an error accessing the data, then the request will be passed to the catch block. The catch block call the next(error) function with the error as parameter in order to pass control to the default error handler to respond to the request.

If the request cannot be ended with a response due to an err accessing the data, then the request will also be passed to the catch block. But the catch block then call an individual error handler to respond to the request.

In our application we have the user data perimeter and the blog data perimeter and for each of these different data perimeters there is a separate data controller.

User Controller

The user data controller exists to control data access and data manipulations for the user data perimeter.

• database/controllers/userController.js
  

The user data controller contain the following user controller modules and functions.

createUser

The createUser function create a new User object in the database. We try first to find a user with the given email address.

If an error occurs here in the query, the request error is passed to the default error handler via catch block. If no error occurs, either a user is found or not. If a user is found, the user cannot be created with createUser and the response is 400 bad request.

If no user is found, the user can be created. This means that the newUser object is saved and the input data is validated when it is saved. If an input data validation error occurs, we will respond with 400 bad request. otherwise there is a 200 ok response and the new user object has been successfully created in the database.

updateUserEmail

With the updateUserEmail function, the email address of a user object can be changed. In parallel also the author email will be adapted on the blog objects where the user is author of blogs.

First the user is searched for with his existing email address. If the function runs on an error, the request and the error are passed on to the default error handler with catch.

If no user is found, we respond with 400 bad request. If, on the other hand, a user is found, the email address can be updated by saving the user object with the new email address. When saving, the input validation takes place again. Only if the input validation was successful the storage of the user object is completed.

Now the email address must be updated on the blogs where the user is the author.

For this we are looking for all blogs where the author email corresponds to the existing email address of the user. If the search runs for an err, we will respond with an individual 502 bad gateway this time because the request has been partially processed up to this point. The email address on the user object has already been updated, but the data for updating the author email addresses on the blogs cannot be queried.

If there is no err in the blog query, the user may not be the author of any blog. Then we answer with 200 ok and confirm that only the email address on the user object has been changed.

If, on the other hand, the user is the author of blogs, we use the updateMany() function to update the email address of any blog found.

This updateMany() function receive a filter object with the previously existing email address of the user as first parameter and then the update object to replace this existing email address with the new email address of the user. This operation is executed on any blog found in the database and is also carried out in a try block. If an err occurs, the request is passed on for individual error handling to the catch block. Here in the catch block we answer with a 502 bad gateway because the update on the user object has been already performed successfully but the update of the author email address on the blogs found was not successful because the updateMany() operation failed.

If the blog update process was successful, we respond with 200 ok. In this case the email address has been changed on the user object and also on his blogs.

removeUser

With removeUser function a user object can be removed from the database. We try first to find the user that should be removed with the given email address. If an error occurs, the request error is passed to the default error handler via catch block.

If no error occurs, either a user is found or not.

If no user is found, the user cannot be removed and the response is 400 bad request.

If the user to be removed has been found we try to delete this user from the database using the deleteOne() function. If an error occurs here, the request error is also passed to the default error handler via catch block.

If the deletion was successful we respond with 200 ok.

Blog Controller

The blog data controller exists to control data access and manipulations for the blog data perimeter.

• database/controllers/blogController.js
  

The blog data controller contain the following blog controller modules and functions.

createBlog

The createBlog module create a new blog object in the database and update the new blog id reference on the author user object.

The new blog object can only be created if the author data email, first- and lastname match with a user object in the database. So the author must already exist as a user in the database.

We try first to find a user with the author email address. If an error occurs, the request error is passed to the default error handler via catch block. If no error occurs, either a user is found or not.

If no user is found, we respond with 400 bad request.

If a user is found, it is checked whether the provided first name or last name does not match the first name or last name of the user in the database found with the given email address. If one of the names does not match, the condition is false and we send a 400 bad request.

If both names are the same, there is a perfect match of email, first name and last name and the provided author data are validated. Then the blog object can be saved and the blog input data will be validated.

If an input data validation error occurs, we will respond with 400 bad request.

In any other case the new blog has been saved to the database and we try to update the user object by adding the new blog id on the user object as a blog reference. Therefore, we try to use the updateOne() function to find the user object based on the author’s email address to add the blog id of the newly created blog object as a blog reference.

This operation is carried out in a try block and if an err occurs, it is passed on to the catch block. In case of an err we answer with a 502 bad gateway. The blog object is created but the user object is still not updated because the updateOne() operation was not successful. If the updateOne() operation was successful we respond with 200 ok.

removeBlog

The removeBlog module remove a blog object from the database and update the user object by removing the relevant blog id reference on the author user object.

We try to find a blog object that should be removed based on the blog title. If an error occurs during that query the default error handling function will be invoked via the catch block and next(error). In case of no error the query return one blog object or null which means that no blog object has been found.

If no blog has been found we respond with 400 bad request.

If the query was successful and we found a blog object we try to find a user based on the author email address. If an error occurs the default error handling function will be invoked via the catch block. In case of no error the query return one user object or no user object.

If we have no user object found that mean that the author of the blog is no longer a user in our application. In this case we try to remove only the blog using the deleteOne() function. If this blog deletion function fail we have no success and an error. We catch this error in the catch block to invoke the default error handler function. If the deleteOne() function can be executed successfully, we respond with ok 200. The blog object with the title has been deleted.

If we have found a user object this means that this user is the author of the blog object we want to delete. In this case we first try to remove the blog object using the deleteOne() function and catch the error in the catch block to invoke the default error handler function when the execution of deleteOne() fail.

When the blog deletion was successful, we also try to delete the blog id as a reference on the user object using the updateOne() function. If updateOne throw an err we go ahead with catch and respond with 502 bad gateway as the blog could be removed but the user object update failed.

If the user update of the user object was successful we respond with 200 ok.

returnAllBlogs

The returnAllBlogs module is a function that searches all blog objects in the database and returns them in an array blogs when the requested url is /blogs. If the requested url is different than the next() function will be called to transfer to the next request handler in the row.

We try to search for all blog objects using the find() function. This find function return an error in case the query fail. In this case the catch block will invoke the default error handler function. In case the query was successful the function return an array blogs including the blog objects.

In case the array is empty or better the length is equal to 0 no blog object has been found in the database and we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into a blog array and return this blog array with 200 ok.

returnYearBlogs

The returnYearBlogs module is a function that searches blog objects of a certain year in the database and returns them in an array blogs when the requested url is /blogs/year. If the requested url is different than the next() function will be called to transfer to the next request handler in the row.

We try to search for all blog objects in a year using the find() function and return an error in case the query fail. In this case the catch block will invoke the default error handler function.

In case the query was successful the function return a query array blogs including the blog objects found for the relevant search criteria. In case no blog object has been found (length of blogs is equal to 0) in the database we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into another blog array and return this blog array with 200 ok.

returnMonthBlogs

The returnMonthBlogs module is a function that searches blog objects of a certain year and month in the database and returns them in an array blogs when the requested url is /blogs/year/month. If the requested url is different than the next() function will be called to transfer to the next request handler in the row.

We try to search for all blog objects in a year and month using the find() function and return an error in case the query fail. In this case the catch block will invoke the default error handler function.

In case the query was successful the function return a query array blogs including the blog objects found for the relevant search criteria. In case no blog object has been found (length blogs is equal to 0) in the database we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into another blog array and return this blog array with 200 ok.

returnDateBlogs

The returnDateBlogs module is a function that searches blog objects of a certain date in the database and returns them in an array blogs.

As this is the last routing handler in the app.get() routing function where the path is defined as /blogs/:year?/:month?/day? the requested url matches this definition or not.

In case the requested url is something like /blogs/year/month/day/else the url is not defined and then the default route error handler will be invoked and return 404 Not found.

In case the requested url path match /blogs/year/month/day we try to search for all blog objects for the date using the find() function and return an error in case the query fail. In case of an error the catch block will invoke the default error handler function.

In case the query was successful the function return a query array blogs including the blog objects found for the relevant search criteria.

In case no blog object has been found (length blogs is equal to 0) in the database we return 200 ok no blogs found.

In case there are blog objects available we create for each blog object found a dataset object of blog attributes containing the title, the author (including name, firstname and email) and the date, push this dataset into another blog array and return this blog array with 200 ok.

Data operations in detail

Our application perform the following data operations.

• create data
• search and return data
• update data
• delete data
  

These operations will be implemented in the controller modules. We use the following functions to perform these operations.

• save()
• find()
• findOne()
• updateOne()
• updateMany()
• deleteOne()
  

If we want to create a new Data object we call new Data( … ) and apply the key specifications according to the Schema defined in the model.

note: Please note that we are using new Data( … ) here because we export Data in the above model example definition. If you look in the model files of our blog app, you will notice that I export in the user model User and in the blog model Blog. Consequently, for example, when creating a new user object in the user data controller, the call new User( … ) is made.

The function save() receive a callback function with error as first parameter and data as second parameter. Within the callback we can work with error as the error object and with data as the data object that will be saved in the database. With save(), error handling takes place directly in the call back function using the if (error) … else … block.

// controller.js

newData = new Data({key1:..., key2:..., ..., keyN:...})

newData.save(function(error, data) {
	if (error) {

	// do something when an error occurs

	} else {

	// do what you need to do when the creation was successful

	} 
})

If we want to search and return data we call find() or findOne().

If we expect that a search return several data objects as a result we are using the find() function.

The find() function is covered with the try … catch … blog.

find() receive as first parameter a filter object as search criteria and as second parameter a callback function with error as first parameter and data as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with data which is an array containing all the data objects that have been found in the database. In case no data objects were found the data array is empty. In case the function fail we catch the error.

Suppose you want to search in Data perimeter for data objects that have a specificName. Then we pass the filter object name: specificName and then the callback function(error, data) … to the Data.find() function.

// controller.js

// create the filter object
var searchDataWithName = { name: specificName }

try {

	Data.find(searchDataWithName, function(error, data) {

	if (data.length == 0) {

	// do something when no data have been found 

	} else {

	// do something with the data array that have been found 

	}

	})

} catch (error) {

// do something when an error occurs

}

If a search should return only one data object as a result we are using the findOne() method.

The findOne() function is covered with the try … catch … blog.

findOne() receive as first parameter a filter object as search criteria and as second parameter a callback function with error as first parameter and data as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with data which is the object that has been found in the database. In case no data object has been found the value of data is null. In case the function fail we catch the error.

Suppose you want to search in Data perimeter for one data object that has a specificName. Then we pass the filter object name: specificName and then the callback function(error, data) … to the Data.findOne() function.

// controller.js


// create the filter object
var searchDataWithName = { name: specificName }

try {

Data.findOne(searchDataWithName, function(error, data) {

	if (!data) {

	// do something when no data have been found
	
	} else {

	// do something with the data in the data object that have been found 

	} 

	})

} catch (error) {

// do something when an error occurs

}

If we want to update an existing Data object we search the Data object, apply the changes and then call save(). We can also use updateOne() or updateMany().

If we do search and save() the data object first must be found in the database and will then be returned using the findOne() function. Then we can apply the changes to one or more data attribute values i.e. update the email address of a user and then call save() to save the updated data object back into the database. This procedure has the advantage that input validation is used with save() and is therefore recommended whenever we receive changes directly as input values and these input values must be validated.

// controller.js

// create the filter object
var searchDataWithName = { name: specificName }

try {

	Data.findOne(searchDataWithName, function(error, data) {

	if (!data) {

	// do something when no data have been found
	
	} else {

		// specify the update value
		var updateEmail = newEmail
	
		// update the value
		data.email = updateEmail

		data.save(function(err, updatedData) {

			if (err) {

			// do something when an err occurs
			
			} else {

			// do something when the update was successful

			}
		})
	} 
	})

} catch (error) {

// do something when an error occurs

}

If we use updateMany() or updateOne() the data objects or the one data object will be looked up and updated in the database in one step. The data object(s) will not be returned. This makes processing faster and saves network resources.

The functions updateMany() and updateOne() are both covered with the try … catch … blog.

updateMany() receive as first parameter a filter object as search criteria, as second parameter the update object and as third parameter a callback function with error as first parameter and result as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with result which is an object containing data about the processing result n: 1, nModified: 1, ok: 1 . This mean n objects found, n objects modified and the processing was ok (ok: 0 if the processing failed).

Suppose you want to find in Data perimeter for data objects that have a specificName and update all objects found with specificNewName. Then we pass the filter object name: specificName , the update object name: specificNewName and then the callback function(error, result) … to the Data.updateMany() function.

// controller.js

// create the filter object
var searchDataWithName = { name: specificName }

// create the update object
var updatedName = { name: specificNewName }

....

try {
      
   Data.updateMany(
      	
     searchDataWithName, 
      	
     updatedName, 
      	
     function(err, result) {
      
       // do something with the result
          
     })
     
} catch (err) {
     	
// do something with the err

}

....

updateOne() also receive as first parameter a filter object as search criteria, as second parameter the update object and as third parameter a callback function with error as first parameter and result as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with result which is an object containing data about the processing result n: 1, nModified: 1, ok: 1 . This mean n objects found, n objects modified and the processing was ok (ok: 0 if the processing failed).

Suppose you want to find in Data data perimeter for one data object that has a specificName and you want to update this single object with specificNewName. Then we pass the filter object name: specificName , the update object name: specificNewName and then the callback function(error, result) … to the Data.updateOne() function.

// controller.js

// create the filter object
var searchDataWithEmail = { name: specificEmail }

// create the update object
var updatedEmail = { name: specificNewEmail}

try {
   
   Data.updateOne(
      	
     searchDataWithEmail, 
      	
     updatedEmail, 
      	
     function(err, result) {
      
     // do something with the result
          
    })
     
} catch (err) {

// do something with the err

}

....

If we want to remove an existing Data object we search for the data object using findOne() to get the data object returned, and then call deleteOne().

We do this 2 step approach i.e. in removeBlogs because we use the returned blog object to determine first the id of the blog object and then use it to find and update the user object. You can do this in one single step only by using deleteOne(), but take into account that you will not receive a data object with which further queries or updates can be carried out.

deleteOne() receive as first parameter a filter object as search criteria and as second parameter a callback function with error as first parameter and result as second parameter. Because we use try catch error handling is regulated in the catch block and we respond to the request in case of an error within this catch block.

Within the callback we can work with result which is an object containing data about the processing result n: 1, deletedCount: 1, ok: 1 . This mean n objects found, n objects deleted and the processing was ok (ok: 0 if the processing failed). You can use the result object in the callback to check if the object to be removed has been found or not. But this an operation we don’t need to perform because we are using the 2 step approach with findOne() and then deleteOne().

....

if (result.n == 0) {
  // do something if no object to be removed has been found
} else {
  // do something when the delete was successful
}

....

Suppose you want to find in User data perimeter for a data object that has a specificName and you want to remove this object from the database. Then we pass the filter object name: specificName and then the callback function(error, result) … to the User.deleteOne() function.

// contoller.js

/ create the filter object
var searchDataWithName = { name: specificName }

try {

	User.findOne(searchDataWithName, function(error, user) {

	if (!user) {

	// do something when no data have been found
	
	} else {
	
		try {
		
		User.deleteOne(searchDataWithName, function(error, result) {
		
			// do something when the delete was successful
		
		})
		
		
		} catch (error) {
		
		// do something when an error occurs		
		}		
}
})


} catch (error) {

// do something when an error occurs

}

HTTP request Routing

In our application main file blog.js we define routes to process HTTP POST and HTTP GET requests.

HTTP POST requests are routed using the app.post(‚routingPath‘, routingHandler) method.

During the create-, update- and remove data operations app.post() routes are called and the request and response objects are transferred to the corresponding routingHandler to take over the request handling. The request body contain all relevant input data to process the request.

The routingHandler then calls the controller module and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends the response.

So for example if an HTTP POST request is made to the path /createusers the app.post(‚/createusers‘, userController.createUser) route is called and the createUser module take over the complete processing.

// blog.js

....

// load database controllers
const userController = require('./database/controllers/userController');
const blogController = require('./database/controllers/blogController');

....

// define routes (routing table)

....

app.post('/createusers', userController.createUser)

app.post('/updateuseremail', userController.updateUserEmail)

app.post('/removeusers', userController.removeUser)

app.post('/createblogs', blogController.createBlog)

app.post('/removeblogs', blogController.removeBlog)

....

You have to imagine it in reality with a real web application like this: On a website there is an HTML form and the end user of the application can enter user data such as last name, first name and email address to add for example a new user in the application or in the application database. The moment the user of the application clicks on the send button, the data entered in the form is transferred to the route /createusers with HTTP POST which means that the relevant app.post(‚/createusers‘, userController.createUser) function is called to handle this HTTP POST request.

Unfortunately we are not yet dealing with a real web application because there is no HTML form until now. This is exactly where Postman comes in to test an HTTP POST action. In principle, Postman replaces the form. With Postman I can (amongst other things) transfer HTTP POST requests to the application and thus I also transfer the request body. It is therefore necessary that you familiarize yourself with Postman to test the applications app.post() routes.

HTTP GET requests are routed using the app.get(‚routingPath‘, routingHandler) method.

So for example if an HTTP GET request is made to the path / the app.get(‚/‘, routingHandler) route is called and the routingHandler take over the complete processing and send directly the response. The same happens when an HTTP GET request is made on the route /about. This is easy and not spectacular.

// blog.js

....

// load database controllers
const userController = require('./database/controllers/userController');
const blogController = require('./database/controllers/blogController');

....

// define routes (routing table)

....

app.get('/', (req, res) => {
  res.send('Hello, this is the Blog Home Page.')
})

app.get('/about', (req, res) => {
  res.send('Hello, this is the Blog About Page.')
})

....

It is a bit more special when an HTTP GET request is made to the route app.get(‚/blogs/:year?/:month?/:day?‘, …. ).

The route definition here provides so-called request parameters. The /blogs route can be parameterized with the parameters year, month and day. The question mark behind each parameter indicates that the parameter is optional.

The request is forwarded to several routing handlers.

The first routing handler assigns the request url and the request parameters year, month and day to variables and calls next() to hand over the request to the next routing handler in the row.

The next routingHandler is blogController.returnAllBlogs. This routingHandler calls the controller module returnAllBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs as response if the request url is equal to /blogs otherwise next() is called to hand over the request to the next routing handler in the row.

The next routingHandler is blogController.returnYearBlogs. This routingHandler calls the controller module returnYearBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs in the specified year as response if the request url is equal to /blogs/year otherwise next() is called to hand over the request to the next routing handler in the row.

The next routingHandler is blogController.returnMonthBlogs. This routingHandler calls the controller module returnMonthBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs in the specified year and month as response if the request url is equal to /blogs/year/month otherwise next() is called to hand over the request to the next routing handler in the row.

The next and last routingHandler is blogController.returnDateBlogs. This routingHandler calls the controller module returnDateBlogs and transfers the request and response object to this module. Then the module takes over the complete processing of the request and finally sends an array containing all blogs from the specified date.

Any other route will end up in an error 404 Bad request.

// blog.js

....

// load database controllers
const userController = require('./database/controllers/userController');
const blogController = require('./database/controllers/blogController');

....

// define routes (routing table)

....

app.get('/blogs/:year?/:month?/:day?',

        (req, res, next) => {
          url = req.url
          year = req.params.year
          month = req.params.month
          day = req.params.day

          next()
        },

        blogController.returnAllBlogs,
        blogController.returnYearBlogs,
        blogController.returnMonthBlogs,
        blogController.returnDateBlogs

      )

....

Status Handling

In our application we work with the following HTTP Status codes.

• code 200: status: Ok. Request completed.
• code 500: status: Internal Server Error. Request failed due to server error.
• code 400: status: Bad Request. Request failed due to wrong input data.
• code 404: status: Not Found. Request failed due to wrong route.
• code 502: status: Bad Gateway. Request partly completed but not fully completed due to missing data.
  

Summary and Outlook

In this Part 3 of my node.js series I explained how to setup our express blog app using a MongoDB database. We learned how to create data models and controllers to control access to the data. But the blog app has still no HTML frontend or no HTML template rendering is used.

So in the next Part 4 of my node.js series I will introduce how to extend the blog app with an HTML interface. Therefore we implement the use of the template engine Pug. With Pug we can create an HTML interface for our blog application.

Since the installation of mongodb is developed specifically for certain Ubuntu system versions, the system version of the Ubuntu system must first be found out. This is done by entering the following command in the terminal.

:# lsb_release -dc

Description:	Ubuntu 18.04.3 LTS
Codename:		bionic

So my system is an Ubuntu 18.04.3 LTS (Long Term Support) with the code name bionic. This information is important when, as shown in the following section, the list file entry is created.

The installation itself is carried out with the package manager apt.

Basically there is also a mongodb package managed by Ubuntu that could be installed with the standard apt or apt-get installation from the apt repository. mongodb is not the desired installation and should, if it was installed on the system beforehand, be removed with apt-get remove.

It is therefore recommended to install the mongodb-org package managed by MongoDB Inc.

Since this mongodb-org package comes from a source that has not yet been verified, it is strongly recommended to import the GPG verification key offered by MongoDB Inc. to enable the apt program to check the confidentiality of the package. The corresponding public key must therefore be imported from the mongodb server.

From a terminal, issue the following command to import the MongoDB public GPG Key used by the Ubuntu apt program.

wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -

note: wget is a program that can be used to download files from FTP or HTTP servers from a terminal. The program is often used to start a download of data with the code of a shell script. However, it is also possible to enter a wget command directly in the terminal and thus get data from servers.

The -q option prevents wget from displaying information on the console. The option -O <FILE> writes the content of the downloaded file to a file on the local file system. If you specify the - option after the -O option, wget writes the contents of the downloaded file directly to the standard output (stdout) of the terminal. The latter is useful, for example, for reading GPG key files if these keys are then to be added directly to the keychain.

And that’s exactly what we want to do with the apt-key command. With apt-key, various commands can be used to perform actions that are carried out with the keychain.

For example, the apt-key add command adds a new key to the keychain. This key can either be stored in a file and read from there or forwarded from a previous command via the standard output of the terminal (stdout) to the standard input (stdin) for processing apt-key add. The latter is instructed by - at the end of the command.

Since this mongodb-org package is located on external mongodb http or ftp servers, these external servers must be made known to the system as confidential sources by making a corresponding list file entry in the /etc/apt/source.list.d directory.

Create the list file /etc/apt/sources.list.d/mongodb-org-4.2.list for your version of Ubuntu.

echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list

note: With echo, strings and variables can be displayed line by line on the standard terminal output stdout. The string to be output is specified either with single quotation marks 'string' or with double quotation marks "string". Thus with the command echo 'this is an example' „this is an example“ can be shown directly in the terminal.

The pipe operator | belongs to the terminal redirects and forwards the output of a command instead of stdout directly to stdin of another command. The subsequent command can then process the result or the output of the first command. In this case here, the output of a string is forwarded to the tee command.

tee reads from the standard input stdin and doubles the read data. The data is then forwarded once to a file and to the standard edition stdout.

Now the GPG key has been added to the keychain and apt can use it to verify the mongodb-org package. In addition, the source directory of the installation packages is known to the system as a verified external source in the list file.

Installation with apt can now begin.

note: APT (Advanced Packaging Tool) is the command line tool for managing installation packages in Debian Linux. Ubuntu and other Linux distributions that come from Debian Linux also use APT. apt is the successor to apt-get and offers a different but also a wider range of functions. The commands of apt are essentially the same as those of apt-get.

Issue the following commands in the terminal.

:# sudo apt-get update

:# sudo apt-get install -y mongodb-org

The repository index is updated with upgrade. This should always be done before each installation. The -y option ensures that all interactive questions are answered with yes during the installation.

After the installation issue the following commands to check the status, start, stop and restart the mongod.

:# sudo service mongod status

:# sudo service mongod start

:# sudo service mongod stop

:# sudo service mongod restart

Mongodb comes with 3 standard dbs pre installed:

• admin
• config
• local
  

If you want to use mondodb for a project you setup your own database. databases in mongodb contain collections and collections contain the data. So for example you could have the following structure for your project:

your-db -> your-db-users-collection -> your-db-users-collection-user-data

MongoDB by default does not enable client authentication. So you could now setup your new database like your-db and connect to it without authentication. This would mean that everyone in the world is able to access your data.

Thefore you should enable client authentication and follow the given steps. Here we will implement a very simple security concept.

The following steps include the setup of an admin user in standard admin db and the creation of a project database with a seperate user. The admin user will be able to create databases and manage users for all databases in the mondodb instance. Each seperate project database become a seperate user and this user will be the owner and can do everything.

First we create an admin user in mongo shell. After we create that admin user we test if authentication works with db.auth and exit mongo shell.

:# mongo

> use admin
switched to db admin

> db
admin

> db.createUser({ user: "myUserAdmin", pwd: "adminpassword", roles: [{ role: "userAdminAnyDatabase", db: "admin" }, {"role" : "readWriteAnyDatabase", "db" : "admin"}] })

> db.auth("myUserAdmin", "adminpassword")
1

> show users
{
	"_id" : "admin.myUserAdmin",
	"userId" : UUID("5cbe2fc4-1e54-4c2d-89d1-317340429571"),
	"user" : "myUserAdmin",
	"db" : "admin",
	"roles" : [
		{
			"role" : "userAdminAnyDatabase",
			"db" : "admin"
		},
		{
			"role" : "readWriteAnyDatabase",
			"db" : "admin"
		}
	],
	"mechanisms" : [
		"SCRAM-SHA-1",
		"SCRAM-SHA-256"
	]
}

> exit

In this case the admin user got the role userAdminAnyDatabase and the role readWriteAnyDatabase. With userAdminAnyDatabase the admin user will be able to create, update, delete users on all databases except local and config.

readWriteAnyDatabase provides all the privileges to read plus the ability to modify data on all non-system collections.

Then open /etc/mongod.conf file in your editor an add the following lines to the file if they do not already exist. Therefore pls. note the security: option (security: authorization: enabled).

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


#processManagement:

#security:
security:
  authorization: enabled

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Pls. note the net:option (net: port: 27017 bindIp: 127.0.0.1). For security reasons it is necessary that the mongodb instance only connects on the loopback interface (127.0.0.1). Mongodb may only accept connections from this interface and never from outside, such as from an interface that is connected to the internet. Therefore bindIp must be configured with 127.0.0.1 in mongod.conf as is the case in this configuration above.

Then you should stop and start or simply restart your mongod.

:# sudo service mongod restart

Authentication is now enabled and we can logon with the admin user to a create a new project database with a user that will be the owner.

:# mongo

> use admin
switched to db admin

> db.auth("myUserAdmin", "adminpassword")
1

> use yourdatabase
switched to db yourdatabase

> db.createUser({ user: "youruser", pwd: "yourpassword", roles: [{ role: "dbOwner", db: "yourdatabase" }] })

> db.auth("youruser", "yourpassword")
1

> exit

The database owner can perform any administrative action on yourdatabase (database that has just been created by the admin). The owner role is something like the master role and combines privileges granted by the readWrite, dbAdmin and userAdmin roles.

Now you can connect to yourdatabase using the following connection string.

mongodb://youruser:yourpassword@localhost/yourdatabase

Install Mongoose local package in a project directory

Mongoose offers a schema-based solution to apply modeling and structure to application data and then to save the data in a Mongodb. mongoose is an open source node.js package and is ideal for the development of database-based node js web applications such as node express.

note: This article focuses on the installation and commissioning of Mongodb. Mongoose is required for the development of node js applications and is only briefly described here. Scheme and data modeling as well as CRUD application examples are explained in one of the next articles in the context of an express application.

Suppose we have created a project directory called express on our computer. Then we will now change to this directory and list all currently installed npm packages with the command npm list --depth 0.

:# cd /software/dev/express

:# npm list --depth 0

In the project directory, the following command is executed in the terminal to install mongoose for your project.

:# npm install mongoose

server@1.0.0 /home/patrick/software/dev/express
`-- mongoose@5.8.9  extraneous

**note: ** If you have problems with installing packages pls. create the package.json file with all the dependencies and then run npm install.

:# cat package.json

{
  "name": "server",
  "version": "1.0.0",
  "private": true,
  "description": "this is test node application",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js"
  },
  "author": "Patrick Rottländer",
  "license": "ISC",
  "dependencies": {
    "envy": "^2.0.0",
    "express": "^4.17.1",
    "mongoose": "^5.8.3"
  },
  "devDependencies": {}
}

:# npm install