Pusher Logo

    How to build a realtime application with Node.js and RethinkDB

    You will need Node 8.10+ and RethinkDB 2.3.6+ installed on your machine.

    In this tutorial, we’ll explore RethinkDB, a database system that was built with realtime functionality in mind. We’ll build a social app that pairs RethinkDB with Pusher Channels to bring an improved experience to users. Let’s go!

    In traditional web apps, database systems were used as “dumb” data stores—they only held data (usually textual or binary) for you and gave it back when you asked nicely. But over time, database systems have become more and more powerful. They can hold data in a wide range of formats and perform actions on (or with) the data.

    Prerequisites

    1. Node.js 8.10.0 or higher
    2. RethinkDB 2.3.6 or higher. Get it here.
    3. A Pusher account.

    Setting up

    We’ll start by using the express application generator:

        # if you don't already have it installed
    npm install express-generator -g
    
    # create a new express app with view engine set to Handlebars (hbs)
    express --view=hbs rethinkdb-pusher-nodejs-demo
    cd rethinkdb-pusher-nodejs-demo && npm install

    Then we’ll add our dependencies:

        npm install dotenv rethinkdb pusher

    We’ll use dotenv to load sensitive data (our Pusher app credentials) from a .env file. The rethinkdb and pusher libraries are the Node.js clients for communicating with RethinkDB and Pusher’s realtime APIs respectively.

    What is RethinkDB?

    RethinkDB is an open-source database built for realtime applications. It aims to improve application performance and simplify the developer experience by pushing data to the app, rather than having it poll the database for changes. RethinkDB is built to do this in an efficient and scalable way.

    RethinkDB is a NoSQL database, so it stores its data as JSON documents. However, it incorporates concepts from SQL (such as its query language, Rethink Query Language, or ReQL) to create an even better experience for its users. RethinkDB is used by thousands of developers, including NASA and Social Radar.

    We’ll build an application where people can create posts and all posts can be seen in realtime on the home page without having to reload the page. Our application will rely on RethinkDB to inform us of new posts in the database and use Pusher to push these changes to the frontend. Let’s get started.

    Building the feed

    The feed (the home page) will show a list of all posts, with the newest first. Let’s set up the route and view for this page. In the route, we retrieve all posts from the database and pass them to the view. Replace the code in routes/index.js with this:

        // routes/index.js
    const router = require('express').Router();
    const r = require('rethinkdb');
    
    let connection;
    r.connect({host: 'localhost', port: 28015, db: 'test'})
        .then(conn => {
          connection = conn;
        });
    
    /* Render the feed. */
    router.get('/', async (req, res, next) => {
      const posts = await r.table('posts').orderBy(r.desc('date')).run(connection)
          .then(cursor => cursor.toArray());
      res.render('index', { posts });
    });
    
    module.exports = router;

    Let’s walk through what’s going on here. We’re connecting to our database via the RethinkDB client (r). Then, in the home route, we retrieve all posts from the database. In ReQL, we can do this by calling table(). We want the results to have the newest posts first, so we add orderBy(r.desc('date')). This query will return all the documents in the table as a cursor. RethinkDB uses a cursor because it allows for easy pagination of large tables. We convert this cursor to an array of the documents by calling cursor.toArray(), then we render them on the frontend.

    Now we’ll create the view where we render the posts. We’ll use Bootstrap for some quick styling. Put the following code in your views/layout.hbs (the base layout):

        <!-- views/layout.hbs -->
    <!DOCTYPE html>
    <html>
      <head>
        <title>{{title}}</title>
          <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous">
      <body>
        {{{body}}}
      </body>
    </html>

    Then, in the file views/index.hbs:

        <!-- views/index.hbs -->
    <div class="container-fluid text-center" id="posts-list">
    
        {{#each posts }}
            <div class="card">
                <div class="card-body">
                    <h5 class="card-title">{{ this.title }}</h5>
                    <h6 class="card-subtitle mb-2 text-muted">{{ this.date }}</h6>
                    <p class="card-text">{{ this.content }}</p>
                </div>
            </div>
        {{/each}}
    
    </div>

    One more thing. Replace the contents of your app.js with the following:

        // app.js
    require('dotenv').config();
    const express = require('express');
    const path = require('path');
    const logger = require('morgan');
    
    const app = express();
    
    // view engine setup
    app.set('views', path.join(__dirname, 'views'));
    app.set('view engine', 'hbs');
    
    app.use(logger('dev'));
    app.use(express.urlencoded({ extended: false }));
    
    app.use('/', require('./routes/index'));
    
    // error handler
    app.use((err, req, res, next) => {
      res.locals.message = err.message;
      res.locals.error = err;
    
      // render the error page
      res.status(err.status || 500);
      res.render('error');
    });
    
    module.exports = app;

    Creating a post

    Next, we’ll make it possible for a user to create a post. Let’s add the necessary routes. Open up your route file (routes/index.js) and replace the last line (module.exports = router) with the following:

        // routes/index.js
    
    /* Show the view to create a new post. */
    router.get('/new', (req, res, next) => {
      res.render('new');
    });
    
    /* Save a new post to the database */
    router.post('/new', async (req, res, next) => {
        const post = {
            title: req.body.title,
            content: req.body.content,
            date: new Date(),
        };
      r.table('posts').insert(post).run(connection)
          .then(() => res.redirect('/'));
    });
    
    module.exports = router;

    Here we’re creating two routes. The first route will render the form for creating new posts when you visit /new in your browser. The second route is where the form submits to. Within this route, we pull data from the request and create a new post in the database. With RethinkDB, this is as easy as calling insert(post). For each post, we store a title, content and the date it was created.

    Next up, create the file views/new.hbs with the following contents:

        <!-- views/new.hbs -->
    
    <div class="container-fluid text-center mt-5">
    <h4>Create Post</h4>
    <form method="post" action="/new">
    
        <div class="form-group">
            <label>Title
                <input type="text" class="form-control" name="title">
            </label>
        </div>
    
        <div class="form-group">
            <label>Content
                <textarea class="form-control" name="content" rows="3"></textarea>
            </label>
        </div>
    
        <button type="submit" class="btn btn-primary">Publish post</button>
    </form>
    </div>

    At this point, you can take the app for a spin to be sure it works as expected. Here’s what you need to do:

    • Start your RethinkDB server by following the instructions for your operating system here.
    • RethinkDB comes with an included web interface where you can explore your database and modify or query data. After starting the server, you should be able to access this UI at http://localhost:8080.
    • You can run queries and commands from the Data Explorer tab. We’ll test it out by creating our posts table. Enter the command r.tableCreate('posts') in the text box and click the Run button. The result of the query will be shown in the lower pane, as shown in the screenshot below.

    rethinkdb-node-express-database-test

    • Start your Express app by running this terminal command in its directory:
        npm start
    • Visit your app on http://localhost:3000. You should see a blank page because our database is currently empty. Visit http://localhost:3000/new to create a new post. Create a couple of posts, and when you reload the home page, you should see your posts sorted from newest to oldest.

    Now that we’ve verified the app works, let’s make it realtime.

    Going realtime with RethinkDB changefeeds

    RethinkDB has a useful feature called changefeeds that makes it well-suited for realtime applications. Using changefeeds, we can ask RethinkDB to notify us when there are any changes to the database. We can even specify a query (such as “all items in Food category and with price greater than 500 dollars”), and RethinkDB will watch for changes to the result set.

    We’ll add changefeeds to our app so the server gets notified of any new posts. Note that since RethinkDB is a database system, it can’t talk directly to the frontend. It can only send the changes to our servers, then we can have the server send the changes to the frontend via a service like Pusher.

    To add changefeeds to our app, we need to make a small change to our routes/index.js. Replace this snippet (where we connect to the RethinkDB server):

        let connection;
    r.connect({host: 'localhost', port: 28015, db: 'test'})
        .then(conn => {
          connection = conn;
        });

    with this:

        let connection;
    r.connect({host: 'localhost', port: 28015, db: 'test'})
        .then(conn => {
          connection = conn;
          return r.table('posts').changes().run(connection);
        }).then(cursor => {
          cursor.each((err, row) => {
            if (err) throw err;
            const post = row.new_val;
            console.log(post);
            // publish row to the frontend
        });
    });

    We’re still connecting to the server as before, but this time, after the connection is established, we set up a changefeed by using the changes() method, telling RethinkDB to notify us of any changes to the data in the posts table. Once again, changes are returned as a cursor, which has keys new_val and old_val. For now, we simply log the new document to the console.

    Our backend is realtime now. Let’s make our frontend realtime too.

    Going realtime with Pusher Channels

    First, let’s get our credentials set up. Log in to your Pusher app dashboard and retrieve your app credentials from the App Keys tab. Then create a .env file in your project and add your credentials to it:

        # .env
    
    PUSHER_APP_ID=your-app-id
    PUSHER_APP_KEY=your-app-key
    PUSHER_APP_SECRET=your-app-secret
    PUSHER_APP_CLUSTER=your-app-cluster

    Now, we’ll modify our app to push changes to the frontend when notifications come in from RethinkDB. Modify your routes/index.js so it looks like this:

        // routes/index.js
    
    const router = require('express').Router();
    const r = require('rethinkdb');
    const Pusher = require('pusher');
    const pusher = new Pusher({
        appId: process.env.PUSHER_APP_ID,
        key: process.env.PUSHER_APP_KEY,
        secret: process.env.PUSHER_APP_SECRET,
        cluster: process.env.PUSHER_APP_CLUSTER
    });
    
    let connection;
    r.connect({host: 'localhost', port: 28015, db: 'test'})
        .then(conn => {
          connection = conn;
          return r.table('posts').changes().run(connection);
        }).then(cursor => {
          cursor.each((err, row) => {
            if (err) throw err;
            const post = row.new_val;
            pusher.trigger('post-events', 'new-post', { post }, (err) => console.log(err));
        });
    });
    
    /* Render the feed. */
    router.get('/', async (req, res, next) => {
      const posts = await r.table('posts').orderBy(r.desc('date')).run(connection)
          .then(cursor => cursor.toArray());
      res.render('index', { posts, appKey: process.env.PUSHER_APP_KEY });
    });
    
    /* Show the view to create a new post. */
    router.get('/new', (req, res, next) => {
      res.render('new');
    });
    
    /* Save a new post to the database */
    router.post('/new', async (req, res, next) => {
        const post = {
            title: req.body.title,
            content: req.body.content,
            date: new Date(),
        };
      r.table('posts').insert(post).run(connection)
          .then(() => res.redirect('/'));
    });
    
    module.exports = router;

    The final step is to update our frontend to add the newly created post to the list when it receives the event via Pusher. Add the following code to the end of your views/index.hbs:

        <!-- views/index.hbs -->
    
    <script src="https://js.pusher.com/4.3/pusher.min.js"></script>
    <script>
        // Enable pusher logging
        Pusher.logToConsole = true;
        const pusher = new Pusher("{{ appKey }}", {
            cluster: 'eu'
        });
        const channel = pusher.subscribe('post-events');
        channel.bind('new-post', (data) => {
            const postsList = document.getElementById('posts-list');
            postsList.insertBefore(createPostCard(data.post), postsList.firstChild);
        });
    
        function createPostCard(post) {
            let cardElement = document.querySelector('.card');
            let newCard = cardElement.cloneNode(true);
            newCard.querySelector('.card-title').innerText = post.title;
            // we wrap the date here and call toString() 
            // because RethinkDB returns it in a different format for changefeeds
            newCard.querySelector('.card-subtitle').innerText = new Date(post.date).toString();
            newCard.querySelector('.card-text').innerText = post.content;
            return newCard;
        }
    </script>

    All done! Now our app is ready.

    Make sure your RethinkDB server is running. Start the app by running npm start. If you had it running earlier, be sure to exit and restart it. Tinker with the app as before, creating new posts at http://localhost:3000/new. This time, leave the homepage (http://localhost:3000) open in another tab. You should see your posts showing up in realtime on the feed, as in the GIF below:

    rethinkdb-node-express-demo

    Conclusion

    In this tutorial, we’ve explored RethinkDB, a database system built for realtime applications. Without RethinkDB’s native changefeeds functionality, we’d have to spend time writing code that listens for any database inserts and dispatches events, introducing a lot of complexity. When paired with Pusher, RethinkDB allows us to deliver pleasant realtime experiences to our users. It’s a great choice worth checking out if you’re building a realtime application. You can visit the RethinkDB docs for more, and check out the source code of the completed application on GitHub.

    JavaScript Node.js

    11 March 2019

    by Shalvah Adebayo