Monitor APIs in realtime using Go
A basic understanding of Go and JavaScript is needed to follow this tutorial.
REST is a popular architectural style for providing standards between computer systems on the web, making it easier for systems to communicate with each other. It is mostly used by APIs to provide data to other systems requiring them.
Sometimes, the providers of APIs would like to monitor its use. Monitoring APIs helps provide useful information, such as which endpoints are called most frequently, or what regions are the largest audience using request IP Addresses. This information can then be used to optimize the API.
In this article, we will implement realtime monitoring of a small API built with GoLang, using Pusher. Here’s a preview of what it should look like at the end:
Requirements
To follow along with this article, you will need the following:
- An IDE of your choice e.g. Visual Studio Code.
- Go installed on your computer.
- Basic knowledge of GoLang.
- Basic knowledge of JavaScript (ES6 syntax) and jQuery.
- Basic knowledge of using a CLI tool or terminal.
Once you have all the above requirements, let’s proceed.
Setting up our codebase
To keep things simple, we’ll be using an already written GoLang CRUD API, which is available on GitHub. We will fork the repository and set it up following the README.md guidelines on installation.
Next, we will set up Pusher in the API project. Pusher is a service that provides a simple implementation of realtime functionality for our web and mobile applications. We’ll be using the Channels API to provide realtime updates to our API monitor dashboard.
Let’s head over to Pusher.com, you can create a free account if you don’t already have one. On the dashboard, create a new Channels app and copy out the app credentials (App ID, Key, Secret, and Cluster). We will use these credentials in our API.
Now that we have our Channels app, we will install the Pusher Go library by running:
$ go get github.com/pusher/pusher-http-go
Monitoring our API
We have so far set up a functional CRUD API, and we will now implement monitoring calls to it. In this article, we will monitor:
- The endpoints called with details like name, request type (GET, POST, etc) and URL.
- For each call to an endpoint, we will also take note of:
- The requesting IP address remove,
- The response status code for the particular call.
Now that we have defined what to monitor, we will begin by creating models to keep track of the data we acquire.
Creating models for monitoring
Based on our specifications above, we will create two new model files EndPoints.go and EndPointCalls.go. As was used in the base API, we will use the GORM (the GoLang ORM) for managing data storage.
💡 Our new model files will exist in the models directory and belong to the models package.
In EndPoints.go, we will define the EndPoints object and a method to save endpoints:
package models
import (
"github.com/jinzhu/gorm"
)
// EndPoints - endpoint model
type EndPoints struct {
gorm.Model
Name, URL string
Type string `gorm:"DEFAULT:'GET'"`
Calls []EndPointCalls `gorm:"ForeignKey:EndPointID"`
}
// SaveOrCreate - save endpoint called
func (ep EndPoints) SaveOrCreate() EndPoints {
db.FirstOrCreate(&ep, ep)
return ep
}
In the code block above, our model did not re-initialize the GORM instance db, yet it was used. This is because the instance defined in the Movies.go file was global to all members of the package, and so it can be referenced and used by all members of package models.
💡 Our EndPoints model has an attribute
Callswhich is an array ofEndPointCallsobjects. This attribute signifies the one to many relationship betweenEndPointsandEndPointCalls. For more information on model associations and relationships see the GORM documentation.
Next, we’ll fill in the model definitions and methods for our EndPointCalls model in the EndPointCalls.go file:
package models
import (
"github.com/jinzhu/gorm"
"github.com/kataras/iris"
)
// EndPointCalls - Object for storing endpoints call details
type EndPointCalls struct {
gorm.Model
EndPointID uint `gorm:"index;not null"`
RequestIP string
ResponseCode int
}
// SaveCall - Save the call details of an endpoint
func (ep EndPoints) SaveCall(context iris.Context) EndPointCalls {
epCall := EndPointCalls{
EndPointID: ep.ID,
RequestIP: context.RemoteAddr(),
ResponseCode: context.GetStatusCode(),
}
db.Create(&epCall)
return epCall
}
As shown above, our EndPointCalls model defines a SaveCall method, which stores the requesting IP address and the response code of an existing EndPoint object.
Finally, we will update the model migration in the index.go file to include our new models:
// index.go
// ...
func main() {
// ...
// Initialize ORM and auto migrate models
db, _ := gorm.Open("sqlite3", "./db/gorm.db")
db.AutoMigrate(&models.Movies{}, &models.EndPoints{}, &models.EndPointCalls{})
// ...
}
Saving endpoint data for monitoring
Using our newly created models, we will edit the MoviesController.go file to save relevant data when an endpoint is called.
To do this, we will add a private helper method to MoviesController.go, which will save endpoint data with our models. See how below:
// MoviesController.go
// ...
func (m MoviesController) saveEndpointCall(name string) {
endpoint := models.EndPoints{
Name: name,
URL: m.Cntx.Path(),
Type: m.Cntx.Request().Method,
}
endpoint = endpoint.SaveOrCreate()
endpointCall := endpoint.SaveCall(m.Cntx)
}
The saveEndpointCall method takes the name of the endpoint as a parameter. Using the controller’s iris.Context instance, it reads and saves the endpoint path and request method.
Now that this helper method is available, we will call it in each of the endpoint methods in the MoviesController.go file:
// MoviesController.go
// ...
// Get - get a list of all available movies
func (m MoviesController) Get() {
movie := models.Movies{}
movies := movie.Get()
go m.saveEndpointCall("Movies List")
m.Cntx.JSON(iris.Map{"status": "success", "data": movies})
}
// GetByID - Get movie by ID
func (m MoviesController) GetByID(ID int64) {
movie := models.Movies{}
movie = movie.GetByID(ID)
if !movie.Validate() {
msg := fmt.Sprintf("Movie with ID: %v not found", ID)
m.Cntx.StatusCode(iris.StatusNotFound)
m.Cntx.JSON(iris.Map{"status": "error", "message": msg})
} else {
m.Cntx.JSON(iris.Map{"status": "success", "data": movie})
}
name := fmt.Sprintf("Single Movie with ID: %v Retrieval", ID)
go m.saveEndpointCall(name)
}
// ...
As shown in the snippet above, the saveEndpointCall helper method will be called in each CRUD method.
💡 The
saveEndpointCallmethod is called as a Goroutine. Calling it this way calls it concurrently with the execution of the endpoint’s method, and allows our monitoring code to not delay or inhibit the response of the API.
Creating the endpoint monitor dashboard
Now that we have implemented monitoring our API’s calls, we will display the data we have accrued on a dashboard.
Registering our template engine
The GoLang framework, Iris, has the ability to implement a range of template engines, which we will take advantage of.
In this section, we will implement the Handlebars template engine, and in our index.go file, we will register it to the app instance:
// index.go
package main
import (
"goggles/controllers"
"goggles/models"
"github.com/jinzhu/gorm"
"github.com/kataras/iris"
)
func main() {
app := iris.New()
tmpl := iris.Handlebars("./templates", ".html")
app.RegisterView(tmpl)
// ...
app.Run(iris.Addr("127.0.0.1:1234"))
}
💡 We have defined our template engine (Handlebars), to render
.htmlfiles contained in thetemplatesdirectory.
Creating the dashboard’s route and controller
Now that we have registered our template engine to the application, we will add a route in index.go to render our API monitor dashboard:
// index.go
// ...
func main() {
app := iris.New()
// ...
app.Get("/admin/endpoints", func(ctx iris.Context) {
dashBoard := controllers.DashBoardController{Cntx: ctx}
dashBoard.ShowEndpoints()
})
app.Run(iris.Addr("127.0.0.1:1234"))
}
Above, we have added definitions for the path /admin/endpoints, where we intend to render details of our API endpoints and its calls. We have also specified that the route should be handled by the ShowEndpoints method of DashBoardController.
To create DashBoardController, we will create a DashBoardController.go file in the controllers directory. And in our DashBoardController.go file, we will define the DashBoardController object and its ShowEndpoints method:
// DashBoardController.go
package controllers
import (
"goggles/models"
"github.com/kataras/iris"
"github.com/kataras/iris/mvc"
)
// DashBoardController - Controller object for Endpoints dashboard
type DashBoardController struct {
mvc.BaseController
Cntx iris.Context
}
// ShowEndpoints - show list of endpoints
func (d DashBoardController) ShowEndpoints() {
endpoints := (models.EndPoints{}).GetWithCallSummary()
d.Cntx.ViewData("endpoints", endpoints)
d.Cntx.View("endpoints.html")
}
In ShowEndpoints(), we retrieve our endpoints and a summary of their calls for display. Then we pass this data to our view using d.Cntx.ViewData("endpoints", endpoints), and finally we render our view file templates/endpoints.html using d.Cntx.View("endpoints.html").
Retrieving endpoints and call summaries
To retrieve our list of endpoints and a summary of their calls, we will create a method in the EndPoints.go file called GetWithCallSummary.
Our GetWithCallSummary method should return the endpoints and their call summaries ready for display. For this, we will define a collection object EndPointWithCallSummary with the attributes we need for our display in the EndPoints.go file:
// EndPoints.go
package models
import (
"github.com/jinzhu/gorm"
)
// EndPoints - endpoint model
type EndPoints struct {
gorm.Model
Name, URL string
Type string `gorm:"DEFAULT:'GET'"`
Calls []EndPointCalls `gorm:"ForeignKey:EndPointID"`
}
// EndPointWithCallSummary - Endpoint with last call summary
type EndPointWithCallSummary struct {
ID uint
Name, URL string
Type string
LastStatus int
NumRequests int
LastRequester string
}
And then define GetWithCallSummary method to use it as follows:
// EndPoints.go
// ...
// GetWithCallSummary - get all endpoints with call summary details
func (ep EndPoints) GetWithCallSummary() []EndPointWithCallSummary {
var eps []EndPoints
var epsWithDets []EndPointWithCallSummary
db.Preload("Calls").Find(&eps)
for _, elem := range eps {
calls := elem.Calls
lastCall := calls[len(calls)-1:][0]
newElem := EndPointWithCallSummary{
elem.ID,
elem.Name,
elem.URL,
elem.Type,
lastCall.ResponseCode,
len(calls),
lastCall.RequestIP,
}
epsWithDets = append(epsWithDets, newElem)
}
return epsWithDets
}
// ...
Above, the GetWithCallSummary method leverages the Calls attribute of EndPoints, which defines its relationship with EndPointCalls. When retrieving our list of endpoints from the database, we eager load its EndPointCalls data using db.Preload("Calls").Find(&eps).
For more information on eager loading in GORM, see the documentation.
GetWithCallSummary initializes an array of EndPointWithCallSummary, and loops through the EndPoints objects returned from our database to create EndPointWithCallSummary objects.
These EndPointWithCallSummary objects are appended to the initialized array and returned.
💡 The
EndPointWithCallSummaryis not a model. It is a collection object and does not need to have a table in our database. This is why it does not have its own file and is not passed toindex.gofor migration.
Implementing the dashboard and displaying data
Now that we have the dashboard’s route, controller and data for display, we will implement the dashboard view to achieve a simple list display of endpoints and their summary data.
Let’s update templates/endpoints.html to have the following code:
<!-- templates/endpoints.html -->
<!DOCTYPE html>
<html>
<head>
<title>Endpoints Monitor Dashboard</title>
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/css/bootstrap.min.css" />
</head>
<body>
<div>
<nav class="navbar navbar-default navbar-static-top">
<div class="container">
<div class="navbar-header">
<a class="navbar-brand" href="http://127.0.0.1:1234/">
Goggles - A Real-Time API Monitor
</a>
</div>
</div>
</nav>
<div class="container">
<div class="row">
<div class="col-xs-12 col-lg-12">
<div class="endpoints list-group">
{{#each endpoints}}
<a id="endpoint-{{ID}}" href="#" class="list-group-item
list-group-item-{{status_class LastStatus}}">
<strong>{{name}}</strong>
<span class="stats">
{{type}}: <strong>{{url}}</strong> |
Last Status: <span class="last_status">
{{LastStatus}}</span> |
Times Called: <span class="times_called">
{{NumRequests}}</span> |
Last Request IP: <span class="request_ip">
{{LastRequester}}</span>
</span>
</a>
{{/each}}
</div>
</div>
</div>
</div>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script>
</body>
</html>
Above, we render our endpoints list using Bootstrap and our Handlebars template engine. We have also created and used a template function status_class, to colour code our list based on their last call status LastStatus.
We define the status_class template function in index.go after initialising our template engine:
// index.go
// ...
func main() {
app := iris.New()
tmpl := iris.Handlebars("./templates", ".html")
tmpl.AddFunc("status_class", func(status int) string {
if status >= 200 && status < 300 {
return "success"
} else if status >= 300 && status < 400 {
return "warning"
} else if status >= 400 {
return "danger"
}
return "success"
})
app.RegisterView(tmpl)
}
Then in our view file we call the function as:
class="list-group-item list-group-item-{{status_class LastStatus}}"
💡 In the above LastStatus is the function’s parameter.
Adding realtime updates to our dashboard
So far in this article, we have monitored the calls to an API and displayed the data via a dashboard. We will now use Pusher Channels to provide realtime data updates to our dashboard.
Sending realtime data from the backend
Earlier, we installed the Pusher Go library, which we will use to trigger an event when an endpoint is called. In the MoviesController.go file, where the API requests are handled, we will initialize the Pusher client:
// MoviesController.go
package controllers
import (
// ...
"github.com/pusher/pusher-http-go"
)
// MoviesController - controller object to serve movie data
type MoviesController struct {
mvc.BaseController
Cntx iris.Context
}
var client = pusher.Client{
AppId: "app_id",
Key: "app_key",
Secret: "app_secret",
Cluster: "app_cluster",
}
// ...
Here, we have initialized the Pusher client using the credentials from our earlier created app.
⚠️ Replace
app_id, app_key, app_secret and app_clusterwith your app credentials.
Next, we will use our Pusher client to trigger an event, which would include the endpoint’s data to be displayed in our view. We will do this in the saveEndpointCall method, which logs an endpoint and its call:
// MoviesController.go
// ...
func (m MoviesController) saveEndpointCall(name string) {
endpoint := models.EndPoints{
Name: name,
URL: m.Cntx.Path(),
Type: m.Cntx.Request().Method,
}
endpoint = endpoint.SaveOrCreate()
endpointCall := endpoint.SaveCall(m.Cntx)
endpointWithCallSummary := models.EndPointWithCallSummary{
ID: endpoint.ID,
Name: endpoint.Name,
URL: endpoint.URL,
Type: endpoint.Type,
LastStatus: endpointCall.ResponseCode,
NumRequests: 1,
LastRequester: endpointCall.RequestIP,
}
client.Trigger("goggles_channel", "new_endpoint_request", endpointWithCallSummary)
}
Above, we create an EndPointWithCallSummary object from EndPoints (the endpoint) and EndPointCalls. This EndPointWithCallSummary object has all the data required for display on the dashboard, so will be passed to Pusher for transmission.
Displaying data in realtime on the dashboard
To display the realtime updates of our endpoints, we will use the Pusher JavaScript client and jQuery libraries.
In our view file, templates/endpoints.html, we will first import and initialize a Pusher instance using our app’s credentials:
<!-- endpoints.html -->
<script src="//cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="//cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/4.0.0-beta.3/js/bootstrap.min.js"></script>
<script src="//js.pusher.com/4.1/pusher.min.js"></script>
<script>
const pusher = new Pusher('app_id', {cluster: "app_cluster"});
</script>
⚠️ Replace
app_id and app_clusterwith values from your app’s credentials.
Next, we will define the following:
- The template for adding new endpoints to our view.
- The functions to append a new endpoint and get the status class of the endpoint.
Finally, we will subscribe to the goggles_channel and listen to the new_endpoint_request event, where our endpoint updates will be transmitted:
<!-- endpoints.html -->
<script>
// ...
const channel = pusher.subscribe("goggles_channel");
channel.bind('new_endpoint_request', function(data) {
let end_point_id = data.ID;
if ( $('#endpoint-' + end_point_id).length > 0 ) {
let status_class = getItemStatusClass(data['LastStatus']),
endpoint = $('#endpoint-' + end_point_id);
let calls = 1 * endpoint.find('span.times_called').text()
endpoint.find('span.last_status').text(data['LastStatus']);
endpoint.find('span.times_called').text( (calls + 1) )
endpoint.removeClass('list-group-item-success');
endpoint.removeClass('list-group-item-danger');
endpoint.removeClass('list-group-item-warning');
endpoint.addClass('list-group-item-' + status_class);
} else {
addNewEndPoint(data);
}
});
// ...
In the new_endpoint_request event handler, the endpoint data is categorized into either an update scenario (where the endpoint already exists on the dashboard) or a create scenario (where a new list item is created and appended).
Finally, you can build your application and when you run it you should see something similar to what we have in the preview:
Conclusion
In this article, we were able to monitor the realtime requests to a REST API and demonstrate how Pusher works with GoLang applications.
10 February 2018
by Neo Ighodaro