The world needs everything uber-fast now. There are plenty of data streams being generated by different systems everyday. Realtime monitoring and analysis has become very important for taking decisions in many industries today. For example: realtime monitoring of website traffic, server performance, weather updates, IOT sensors etc. It is important to analyse and interpret this burst of data, for which interactive Charts and Graphs are an excellent solution.

In this blog post, we will be building a Node.js Server to expose APIs to provide historic data for a metric (in this case, weather in London City) and also provide an API to ingest new data points. We will also be building a front end app with a Line Chart to display the temperature changes in London weather in realtime. The application we build will look something like this:

Signup for Pusher

The first step to start this tutorial is to Signup at Pusher or login with your existing credentials if you already have an account. After logging in, you will need to create a new app and select ‘Vanilla JS’ for the front end along with ‘Node.js’ for the backend. You will then be brought to a landing page containing the ‘getting started’ code for both front end and backend which we will use later on in the tutorial.

Node.js Server APIs for Monitoring & Analytics System

The most essential APIs for any analytics systems for any metric or entity are:

1. Ingestion API - An API to ingest the new data points for any particular entity. In our server for this blog post, we will make an API to ingest new temperature data at a particular time for London city. This API can be called by any global weather system or any IOT sensor.
2. Historical Data API - This API will return all the data within a range from this date in time. For our server, we will create a simple API which will return some static historical data with limited data points for London’s temperature values for any day.

Node.js Express Server Skeleton

We will create a basic Express Server along with instantiating the Pusher library server instance. We will create a new folder for our project and create a new file server.js. Add the following code to this file:

var express = require('express');
var path = require('path');
var bodyParser = require('body-parser');

var Pusher = require('pusher');

var pusher = new Pusher({
  appId: '<your-app-id>',
  key: '<your-app-key>',
  secret: '<your-app-secret>',
  cluster: '<your-app-cluster>',
  encrypted: true
});

var app = express();

app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(express.static(path.join(__dirname, 'public')));

// Error Handler for 404 Pages
app.use(function(req, res, next) {
    var error404 = new Error('Route Not Found');
    error404.status = 404;
    next(error404);
});

module.exports = app;

app.listen(9000, function(){
  console.log('Example app listening on port 9000!')
});

API to get historical temperature data

Now, we will add some static data regarding London’s temperature at certain times during a day and store it in any JavaScript variable. We will also expose a route to return this data whenever someone invokes it using a GET HTTP call.

var londonTempData = {
    city: 'London',
    unit: 'celsius',
    dataPoints: [
      {
        time: 1130,
        temperature: 12 
      },
      {
        time: 1200,
        temperature: 13 
      },
      {
        time: 1230,
        temperature: 15 
      },
      {
        time: 1300,
        temperature: 14 
      },
      {
        time: 1330,
        temperature: 15 
      },
      {
        time: 1406,
        temperature: 12 
      },
    ]
  }

app.get('/getTemperature', function(req,res){
  res.send(londonTempData);
});

API to ingest temperature data point

Now we will add the code for exposing an API to ingest the temperature at a particular time. We will expose a GET HTTP API with temperature and time as query parameters. We will validate that they are not empty and store them by pushing in the dataPoints array of our static Javascript variable londonTempData. Please add the following code to the server.js file:

app.get('/addTemperature', function(req,res){
  var temp = parseInt(req.query.temperature);
  var time = parseInt(req.query.time);
  if(temp && time && !isNaN(temp) && !isNaN(time)){
    var newDataPoint = {
      temperature: temp,
      time: time
    };
    londonTempData.dataPoints.push(newDataPoint);
    pusher.trigger('london-temp-chart', 'new-temperature', {
      dataPoint: newDataPoint
    });
    res.send({success:true});
  }else{
    res.send({success:false, errorMessage: 'Invalid Query Paramaters, required - temperature & time.'});
  }
});

In the above code, apart from storing in the data source, we will also trigger an event ‘new-temperature’ on a new channel ‘london-temp-chart’. For every unique data source or a chart, you can create a new channel.

The event triggered by our server will be processed by the front end to update the chart/graph in realtime. The event can contain all the important data which the chart needs to display the data point correctly. In our case, we will be sending the temperature at the new time to our front end.

Building the Front End App using Vanilla JS & Chart.js

Now, we will build the front end application to display a Line Chart representing the changes in temperature for London City at different times throughout the day. The key approach for displaying realtime graphs is

1. We have to make an initial Ajax call to fetch historical data, and render the graph with the existing data.
2. We will subscribe to any events for new data points being stored on a particular channel.

Basic HTML Template

We will create a new folder called public in our project root and then create a new file index.html in this folder. This file will contain the basic HTML code to render a simple header and a sub-header with the app name along with few icons. We will also import the Pusher Javascript library from its CDN URL.

<!DOCTYPE>
<html>
    <head>
        <title>Realtime Analytics</title>
        <link rel="stylesheet" href="https://unpkg.com/purecss@0.6.2/build/pure-min.css" integrity="sha384-UQiGfs9ICog+LwheBSRCt1o5cbyKIHbwjWscjemyBMT9YCUMZffs6UqUTd0hObXD" crossorigin="anonymous">
        <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Raleway:200">
        <link rel="stylesheet" href="./style.css">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
    </head>
    <body>
        <header>
            <div class="logo">
                <img src="./assets/pusher-logo.png" />
            </div>
            <div id="logout" class="logout">
               <a href="/logout">Logout</a>
            </div>
        </header>
        <section class="subheader">
            <img class="weather-icon" src="./assets/weather.png" />
            <h2>Realtime Weather Updates for London !</h2>
            <img class="weather-icon" src="./assets/london-icon.png" height="70px" />
        </section>
        <section>
           <div id="loader" class="loader">
           </div>
        </section>
        <script src="https://js.pusher.com/4.0/pusher.min.js"></script>
        <script type="text/javascript" src="./app.js"></script>
    </body>
</html>

Adding Charts Library

In Javascript and HTML apps, we have to use either of the two technologies to build graphical components to represent mathematical graphs, SVG or Canvas. There are numerous open source libraries which can help you render different chart types, such as Bar Charts, Pie Charts, Line Charts and Scatter Charts. Here are links to a few of the popular libraries:

• Highcharts - http://www.highcharts.com/
• Chart.js - http://www.chartjs.org/
• Google Charts - https://developers.google.com/chart/
• Flot - http://www.flotcharts.org/
• D3 JS - http://d3js.org/

For our project, we will choose Chart.js as it has fairly simple API and renders robust charts using Canvas HTML tag. You can choose any charting library but keep in mind that the library should have a means to update the chart without completely re-rendering it. Chart.js provides a method on any instantiated chart to update it.

Add the following code to your index.html file at appropriate places

...
<section>
   <div id="loader" class="loader">
    Loading...
   </div>
   <canvas id="weatherChart">
   </canvas>
</section>
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.5.0/Chart.bundle.min.js"></script>
<script src="https://js.pusher.com/4.0/pusher.min.js"></script>
...

Adding JS File & Instantiating Pusher client side library

Now we will create a new file app.js in our public folder and also add the following code to instantiate the Pusher client side library.

// Using IIFE for Implementing Module Pattern to keep the Local Space for the JS Variables
(function() {
    // Enable pusher logging - don't include this in production
    Pusher.logToConsole = true;

    var serverUrl = "/",
        members = [],
        pusher = new Pusher('<your-api-key>', {
          encrypted: true
        }),
        channel,weatherChartRef;

    function showEle(elementId){
      document.getElementById(elementId).style.display = 'flex';
    }

    function hideEle(elementId){
      document.getElementById(elementId).style.display = 'none';
    }

    function ajax(url, method, payload, successCallback){
      var xhr = new XMLHttpRequest();
      xhr.open(method, url, true);
      xhr.setRequestHeader("Content-Type", "application/json;charset=UTF-8");
      xhr.onreadystatechange = function () {
        if (xhr.readyState != 4 || xhr.status != 200) return;
        successCallback(xhr.responseText);
      };
      xhr.send(JSON.stringify(payload));
    }

})();

In the above code, we have also added few utility methods to make an Ajax call and also show or hide elements from the DOM API.

Adding Code to fetch Historical Data

Now, we will add the code to fetch the historical temperature data to display the graph with the initial values. We will also instantiate a new Chart object with a specific config to render a Line Chart. You can read more about how to construct these configs at the Chart.js documentation.

Please add the following code to the app.js file:

   function renderWeatherChart(weatherData) {
      var ctx = document.getElementById("weatherChart").getContext("2d");
      var options = { };
      weatherChartRef = new Chart(ctx, {
        type: "line",
        data: weatherData,
        options: options
      });
   }

   var chartConfig = {
      labels: [],
      datasets: [
         {
            label: "London Weather",
            fill: false,
            lineTension: 0.1,
            backgroundColor: "rgba(75,192,192,0.4)",
            borderColor: "rgba(75,192,192,1)",
            borderCapStyle: 'butt',
            borderDash: [],
            borderDashOffset: 0.0,
            borderJoinStyle: 'miter',
            pointBorderColor: "rgba(75,192,192,1)",
            pointBackgroundColor: "#fff",
            pointBorderWidth: 1,
            pointHoverRadius: 5,
            pointHoverBackgroundColor: "rgba(75,192,192,1)",
            pointHoverBorderColor: "rgba(220,220,220,1)",
            pointHoverBorderWidth: 2,
            pointRadius: 1,
            pointHitRadius: 10,
            data: [],
            spanGaps: false,
         }
      ]
   };

   ajax("/getTemperature", "GET",{}, onFetchTempSuccess);

   function onFetchTempSuccess(response){
      hideEle("loader");
      var respData = JSON.parse(response);
      chartConfig.labels = respData.dataPoints.map(dataPoint => dataPoint.time);
      chartConfig.datasets[0].data = respData.dataPoints.map(dataPoint => dataPoint.temperature);
      renderWeatherChart(chartConfig)
  }

In the above code, we have added a function named renderWeatherChart which will be used to render the chart using latest data which is embedded in the chartConfig variable under the key datasets. If we want to draw multiple line charts on same canvas, we can add more elements to this array.

The data key in each of the elements of the array will display the different points on the graph. We will make an ajax request to the /getTemperature api to fetch all the data points and put them into this key. We will call the rendering method to display the graph then. Now we can run the command node server.js and then go to the browser with the following URL to see the initial chart rendered using the data.

http://localhost:9000/

In order to style our app properly, please add the following CSS into a new style.css file inside the public folder. Add the following code to that file:

body{
    margin:0;
    padding:0;
    overflow: hidden;
    font-family: Raleway;
}

header{
    background: #2b303b;
    height: 50px;
    width:100%;
    display: flex;
    color:#fff;
}

.logo img{
  height: 45px;
}

.subheader{
    display: flex;
    align-items: center;
    margin: 0px;
}

.logout{
    flex:1;
    justify-content: flex-end;
    padding:15px;
    display: none;
}

.logout a{
    color:#fff;
    text-decoration: none;
}

#weatherChart{
    height: 80% !important;
    width: 95% !important;
    margin: 0 auto;
}

Code to Update Graph on new event received

Now we want to subscribe to the unique channel on which our server will be sending update events for this graph. For our project the channel is named london-temp-chart and the event is named new-temperature. Please add the following code to process the event and then update the chart in realtime:

channel = pusher.subscribe('london-temp-chart');
channel.bind('new-temperature', function(data) {
    var newTempData = data.dataPoint;
    if(weatherChartRef.data.labels.length > 15){
      weatherChartRef.data.labels.shift();  
      weatherChartRef.data.datasets[0].data.shift();
    }
    weatherChartRef.data.labels.push(newTempData.time);
    weatherChartRef.data.datasets[0].data.push(newTempData.temperature);
    weatherChartRef.update();
});

In order to see this code in action, you have to refresh the browser and you will see the initial chart. Now we have to ingest a new data point, for which you would need to call the following API either by using some mock API calling tool or using the following URL with different values in the browser.

http://localhost:9000/addTemperature?temperature=17&time=1500

In order to test your chart update code, you can use the following temporary code in your app.js Javascript file which will make dummy Ajax requests to the above URL after a specific time interval.

/* TEMP CODE FOR TESTING */
  var dummyTime = 1500;
  setInterval(function(){
    dummyTime = dummyTime + 10;
    ajax("/addTemperature?temperature="+ getRandomInt(10,20) +"&time="+dummyTime,"GET",{},() => {});
  }, 1000);

  function getRandomInt(min, max) {
      return Math.floor(Math.random() * (max - min + 1)) + min;
  }
/* TEMP CODE ENDS */

The Github repo for reference to complete code is https://github.com/mappmechanic/realtime-analytics

Conclusion

Finally our realtime analytics app is ready. We will see the weather temperature chart for London city updating in realtime.

We can use the code from this blog post for any charts library and also to render any type of chart like Bar Chart, Scatter Chart or Pie Chart to update in realtime.

This code can also be used in multiple Enterprise Apps like Monitoring dashboards, analytics reports, sensor regulatory apps, financial apps etc. The Pusher library helps us to send realtime events to all connected client side apps which can consume the data to update the charts in realtime.