Build a live analytics dashboard using Go and MongoDB
You will need Go 1.7+ and MongoDB 3+ installed on your machine.
One of the most important step to take while taking a website or app into production is analytics and usage statistics. This is important as it allows you to see how users are actually using your app, improve usability and inform future development decisions.
In this tutorial, I will describe how to monitor all requests an application is going to receive, we will use the data gotten from monitoring to track a few metrics such as:
- Most visited links
- Response time for each link
- Total number of requests
- Average response time
Prerequisites
- Golang >=1.7. You can install the Golang toolchain by following this guide.
- MongoDB >=3. You can install this by following this guide.
- A Pusher account.
Starting out
We will start out by setting up our project directory. You will need to create a directory called analytics-dashboard. The location of this directory will depend on the version of the Go toolchain you have:
- If you are running
<=1.11, you should create the directory in$GOPATH/src/github.com/pusher-tutorials/analytics-dashboard - If you are running
1.12or greater, you can create the directory anywhere.
In the newly created directory, create a .env in the root directory with the following command:
$ touch .env
In the .env file, you will need to add your credentials. Copy and paste the following contents into the file:
// analytics-dashboard/.env
PUSHER_APP_ID=PUSHER_APP_ID
PUSHER_APP_KEY=PUSHER_APP_KEY
PUSHER_APP_SECRET=PUSHER_APP_SECRET
PUSHER_APP_CLUSTER=PUSHER_APP_CLUSTER
PUSHER_APP_SECURE="1"
Please make sure to replace the placeholders with your own credentials.
MongoDB
MongoDB is going to be used as a persistent datastore and we are going to make use of it’s calculation abilities to build out the functionality I described above.
Since we are building the application in Golang, we will need to fetch a client library that will assist us in connecting and querying the MongoDB database. To that, you should run the following command:
$ go get -u -v gopkg.in/mgo.v2/...
Once the above command succeeds, you will need to create a new file called analytics.go. In this file, paste the following code:
// analytics-dashboard/analytics.go
package main
import (
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
const (
collectionName = "request_analytics"
)
type requestAnalytics struct {
URL string `json:"url"`
Method string `json:"method"`
RequestTime int64 `json:"request_time"`
Day string `json:"day"`
Hour int `json:"hour"`
}
type mongo struct {
sess *mgo.Session
}
func (m mongo) Close() error {
m.sess.Close()
return nil
}
func (m mongo) Write(r requestAnalytics) error {
return m.sess.DB("pusher_tutorial").C(collectionName).Insert(r)
}
func (m mongo) Count() (int, error) {
return m.sess.DB("pusher_tutorial").C(collectionName).Count()
}
type statsPerRoute struct {
ID struct {
Method string `bson:"method" json:"method"`
URL string `bson:"url" json:"url"`
} `bson:"_id" json:"id"`
NumberOfRequests int `bson:"numberOfRequests" json:"number_of_requests"`
}
func (m mongo) AverageResponseTime() (float64, error) {
type res struct {
AverageResponseTime float64 `bson:"averageResponseTime" json:"average_response_time"`
}
var ret = []res{}
var baseMatch = bson.M{
"$group": bson.M{
"_id": nil,
"averageResponseTime": bson.M{"$avg": "$requesttime"},
},
}
err := m.sess.DB("pusher_tutorial").C(collectionName).
Pipe([]bson.M{baseMatch}).All(&ret)
if len(ret) > 0 {
return ret[0].AverageResponseTime, err
}
return 0, nil
}
func (m mongo) StatsPerRoute() ([]statsPerRoute, error) {
var ret []statsPerRoute
var baseMatch = bson.M{
"$group": bson.M{
"_id": bson.M{"url": "$url", "method": "$method"},
"responseTime": bson.M{"$avg": "$requesttime"},
"numberOfRequests": bson.M{"$sum": 1},
},
}
err := m.sess.DB("pusher_tutorial").C(collectionName).
Pipe([]bson.M{baseMatch}).All(&ret)
return ret, err
}
type requestsPerDay struct {
ID string `bson:"_id" json:"id"`
NumberOfRequests int `bson:"numberOfRequests" json:"number_of_requests"`
}
func (m mongo) RequestsPerHour() ([]requestsPerDay, error) {
var ret []requestsPerDay
var baseMatch = bson.M{
"$group": bson.M{
"_id": "$hour",
"numberOfRequests": bson.M{"$sum": 1},
},
}
var sort = bson.M{
"$sort": bson.M{
"numberOfRequests": 1,
},
}
err := m.sess.DB("pusher_tutorial").C(collectionName).
Pipe([]bson.M{baseMatch, sort}).All(&ret)
return ret, err
}
func (m mongo) RequestsPerDay() ([]requestsPerDay, error) {
var ret []requestsPerDay
var baseMatch = bson.M{
"$group": bson.M{
"_id": "$day",
"numberOfRequests": bson.M{"$sum": 1},
},
}
var sort = bson.M{
"$sort": bson.M{
"numberOfRequests": 1,
},
}
err := m.sess.DB("pusher_tutorial").C(collectionName).
Pipe([]bson.M{baseMatch, sort}).All(&ret)
return ret, err
}
func newMongo(addr string) (mongo, error) {
sess, err := mgo.Dial(addr)
if err != nil {
return mongo{}, err
}
return mongo{
sess: sess,
}, nil
}
type Data struct {
AverageResponseTime float64 `json:"average_response_time"`
StatsPerRoute []statsPerRoute `json:"stats_per_route"`
RequestsPerDay []requestsPerDay `json:"requests_per_day"`
RequestsPerHour []requestsPerDay `json:"requests_per_hour"`
TotalRequests int `json:"total_requests"`
}
func (m mongo) getAggregatedAnalytics() (Data, error) {
var data Data
totalRequests, err := m.Count()
if err != nil {
return data, err
}
stats, err := m.StatsPerRoute()
if err != nil {
return data, err
}
reqsPerDay, err := m.RequestsPerDay()
if err != nil {
return data, err
}
reqsPerHour, err := m.RequestsPerHour()
if err != nil {
return data, err
}
avgResponseTime, err := m.AverageResponseTime()
if err != nil {
return data, err
}
return Data{
AverageResponseTime: avgResponseTime,
StatsPerRoute: stats,
RequestsPerDay: reqsPerDay,
RequestsPerHour: reqsPerHour,
TotalRequests: totalRequests,
}, nil
}
In the above, we have implemented a few queries on the MongoDB database:
StatsPerRoute: Analytics for each route visitedRequestsPerDay: Analytics per dayRequestsPerHour: Analytics per hour
The next step is to add some HTTP endpoints a user can visit. Without those, the code above for querying MongoDB for analytics is redundant. You will also need to create a logging middleware that writes analytics to MongoDB. And to make it realtime, Pusher Channels will also be used.
To get started with that, you will need to create a file named main.go. You can do that via the command below:
$ touch main.go
You will also need to fetch some libraries that will be used while building. You will need to run the command below to fetch them:
$ go get github.com/go-chi/chi
$ go get github.com/joho/godotenv
$ go get github.com/pusher/pusher-http-go
In the newly created main.go file, paste the following code:
// analytics-dashboard/main.go
package main
import (
"encoding/json"
"flag"
"fmt"
"html/template"
"log"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"github.com/go-chi/chi"
"github.com/joho/godotenv"
"github.com/pusher/pusher-http-go"
)
const defaultSleepTime = time.Second * 2
func main() {
httpPort := flag.Int("http.port", 4000, "HTTP Port to run server on")
mongoDSN := flag.String("mongo.dsn", "localhost:27017", "DSN for mongoDB server")
flag.Parse()
if err := godotenv.Load(); err != nil {
log.Fatal("Error loading .env file")
}
appID := os.Getenv("PUSHER_APP_ID")
appKey := os.Getenv("PUSHER_APP_KEY")
appSecret := os.Getenv("PUSHER_APP_SECRET")
appCluster := os.Getenv("PUSHER_APP_CLUSTER")
appIsSecure := os.Getenv("PUSHER_APP_SECURE")
var isSecure bool
if appIsSecure == "1" {
isSecure = true
}
client := &pusher.Client{
AppId: appID,
Key: appKey,
Secret: appSecret,
Cluster: appCluster,
Secure: isSecure,
HttpClient: &http.Client{
Timeout: time.Second * 10,
},
}
mux := chi.NewRouter()
log.Println("Connecting to MongoDB")
m, err := newMongo(*mongoDSN)
if err != nil {
log.Fatal(err)
}
log.Println("Successfully connected to MongoDB")
mux.Use(analyticsMiddleware(m, client))
var once sync.Once
var t *template.Template
workDir, _ := os.Getwd()
filesDir := filepath.Join(workDir, "static")
fileServer(mux, "/static", http.Dir(filesDir))
mux.Get("/", func(w http.ResponseWriter, r *http.Request) {
once.Do(func() {
tem, err := template.ParseFiles("static/index.html")
if err != nil {
log.Fatal(err)
}
t = tem.Lookup("index.html")
})
t.Execute(w, nil)
})
mux.Get("/api/analytics", analyticsAPI(m))
mux.Get("/wait/{seconds}", waitHandler)
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *httpPort), mux))
}
func fileServer(r chi.Router, path string, root http.FileSystem) {
if strings.ContainsAny(path, "{}*") {
panic("FileServer does not permit URL parameters.")
}
fs := http.StripPrefix(path, http.FileServer(root))
if path != "/" && path[len(path)-1] != '/' {
r.Get(path, http.RedirectHandler(path+"/", 301).ServeHTTP)
path += "/"
}
path += "*"
r.Get(path, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fs.ServeHTTP(w, r)
}))
}
func analyticsAPI(m mongo) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
data, err := m.getAggregatedAnalytics()
if err != nil {
log.Println(err)
json.NewEncoder(w).Encode(&struct {
Message string `json:"message"`
TimeStamp int64 `json:"timestamp"`
}{
Message: "An error occurred while fetching analytics data",
TimeStamp: time.Now().Unix(),
})
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(data)
}
}
func analyticsMiddleware(m mongo, client *pusher.Client) func(next http.Handler) http.Handler {
return func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
startTime := time.Now()
defer func() {
if strings.HasPrefix(r.URL.String(), "/wait") {
data := requestAnalytics{
URL: r.URL.String(),
Method: r.Method,
RequestTime: time.Now().Unix() - startTime.Unix(),
Day: startTime.Weekday().String(),
Hour: startTime.Hour(),
}
if err := m.Write(data); err != nil {
log.Println(err)
}
aggregatedData, err := m.getAggregatedAnalytics()
if err == nil {
client.Trigger("analytics-dashboard", "data", aggregatedData)
}
}
}()
next.ServeHTTP(w, r)
})
}
}
func waitHandler(w http.ResponseWriter, r *http.Request) {
var sleepTime = defaultSleepTime
secondsToSleep := chi.URLParam(r, "seconds")
n, err := strconv.Atoi(secondsToSleep)
if err == nil && n >= 2 {
sleepTime = time.Duration(n) * time.Second
} else {
n = 2
}
log.Printf("Sleeping for %d seconds", n)
time.Sleep(sleepTime)
w.Write([]byte(`Done`))
}
While the above might seem like a lot, basically what has been done is:
- Line 31 - 33: Parse environment variables from the
.envcreated earlier.
Another reminder to update the
.envfile to contain your actual credentials
- Line 36 - 56: A server side connection to Pusher Channels is established
- Line 68 - 95: Build an HTTP server.
- Line 139 - 171: A lot is happening here.
analyticsMiddlewareis used to capture all requests, and for requests that have the pathwait/{seconds}, a log is written to MongoDB. It is also sent to Pusher Channels.
Before running the server, you need a frontend to visualize the analytics. The frontend is going to be as simple and usable as can be. You will need to create a new directory called static in your root directory - analytics-dashboard . That can be done with the following command:
$ mkdir analytics-dashboard/static
In the static directory, create two files - index.html and app.js. You can run the command below to do just that:
$ touch static/{index.html,app.js}
Open the index.html file and paste the following code:
// analytics-dashboard/static/index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>Realtime analytics dashboard</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
</head>
<body>
<div class="container" id="app"></div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/handlebars.js/4.1.2/handlebars.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/axios/0.18.0/axios.min.js"></script>
<script src="https://js.pusher.com/4.3/pusher.min.js"></script>
<script src="/static/app.js"></script>
</body>
</html>
While that is an empty page, you will make use of JavaScript to fill it up with useful data. So you will also need to open up the app.js file. In the app.js file, paste the following code:
// analytics-dashboard/static/app.js
const appDiv = document.getElementById('app');
const tmpl = `
<div class="row">
<div class="col-md-5">
<div class="card">
<div class="card-body">
<h5 class="card-title">Total requests</h5>
<div class="card-text">
<h3>\{{total_requests}}</h3>
</div>
</div>
</div>
</div>
<div class="col-md-5">
<div class="card">
<div class="card-body">
<h5 class="card-title">Average response time</h5>
<div class="card-text">
<h3>\{{ average_response_time }} seconds</h3>
</div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-5">
<div class="card">
<div class="card-body">
<h5 class="card-title">Busiest days of the week</h5>
<div class="card-text" style="width: 18rem">
<ul class="list-group list-group-flush">
{{#each requests_per_day}}
<li class="list-group-item">
\{{ this.id }} (\{{ this.number_of_requests }} requests)
</li>
{{/each }}
</ul>
</div>
</div>
</div>
</div>
<div class="col-md-5">
<div class="card">
<div class="card-body">
<h5 class="card-title">Busiest hours of day</h5>
<div class="card-text" style="width: 18rem;">
<ul class="list-group list-group-flush">
{{#each requests_per_hour}}
<li class="list-group-item">
\{{ this.id }} (\{{ this.number_of_requests }} requests)
</li>
{{/each}}
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-5">
<div class="card">
<div class="card-body">
<h5 class="card-title">Most visited routes</h5>
<div class="card-text" style="width: 18rem;">
<ul class="list-group list-group-flush">
{{#each stats_per_route}}
<li class="list-group-item">
\{{ this.id.method }} \{{ this.id.url }} (\{{ this.number_of_requests }} requests)
</li>
{{/each}}
</ul>
</div>
</div>
</div>
</div>
</div>
`;
const template = Handlebars.compile(tmpl);
writeData = data => {
appDiv.innerHTML = template(data);
};
axios
.get('http://localhost:4000/api/analytics', {})
.then(res => {
console.log(res.data);
writeData(res.data);
})
.catch(err => {
console.error(err);
});
const APP_KEY = 'PUSHER_APP_KEY';
const APP_CLUSTER = 'PUSHER_CLUSTER';
const pusher = new Pusher(APP_KEY, {
cluster: APP_CLUSTER,
});
const channel = pusher.subscribe('analytics-dashboard');
channel.bind('data', data => {
writeData(data);
});
Please replace
PUSHER_APP_KEYandPUSHER_CLUSTERwith your own credentials.
In the above code, we defined a constant called tmpl, it holds an HTML template which we will run through the Handlebars template engine to fill it up with actual data.
With this done, you can go ahead to run the Golang server one. You will need to go to the root directory - analytics-dashboard and run the following command:
$ go build
$ ./analytics-dashboard
Make sure you have a MongoDB instance running. If your MongoDB is running on a port other than the default 27017, make sure to add
-mongo.dsn "YOUR_DSN"to the above command
Also make sure your credentials are in
.env
At this stage, you will need to open two browser tabs. Visit http://localhost:4000 in one and http://localhost:4000/wait/2 in the other. Refresh the tab where you have http://localhost:4000/wait/2 and go back to the other tab to see a breakdown of usage activity.
Note you can change the value of 2 in the url to any other digit.
Conclusion
In this tutorial, we’ve built a middleware that tracks every request, and a Golang application that calculates analytics of the tracked requests. We also built a dashboard that displays the relevant data. With Pusher Channels, we’ve been able to update the dashboard in realtime. The full source code can be found on GitHub.
1 May 2019
by Lanre Adelowo