Pusher Channels as an alternative messaging queue
You will need Go 1.9+ and Node 7+ installed on your machine.
Introduction
In this tutorial, we will be building a message queue backed up by Pusher Channels. The application we will build will be a typical login service which upon a successful authentication, an email is sent to the authenticated user informing him of the authentication process and where it originated from. This is quite common with web applications - Twitter, GitHub and Slack do this all the time. We will build the login service in Golang while the email service will be written in NodeJS. The Golang application will publish the data to Pusher channels while the Node.js service will be subscribe to the particular channel and send the email to the user.
Messaging queues are an interesting technique used to improve scalability and a bit of abstraction between the producer and the receiver/consumer as they don’t have to be connected in whatever form. A message queue is nothing much more than a list of messages being sent between two or more applications. A message is basically data produced by an application usually called the producer. That data is then sent into the queue to be picked up by another totally different application - known as the consumer.
Prerequisites
- Golang (
>= 1.9) - Node.js (
>= 7) - A Pusher Channels application. Create one here.
Building the login service
Let’s set up a simple login Golang service. Due to simplicity reasons this application will only handle authentication and will use a memory-mapped list of users.
To get started, we will need to set up our project root directory. We need to create the directory pusher-channels-queue somewhere in $GOPATH. Ideally, this should resolve to $GOPATH/src/github.com/pusher-tutorials/pusher-channels-queue.
After doing the above, we will need to create a go directory since that is where our Golang application will live.
$ mkdir go
The only external library we will need here are the Channel’s Golang SDK and a library to help us load our Pusher Channels keys. You can fetch that by running the command below:
$ go get github.com/pusher/pusher-http-go
$ go get github.com/joho/godotenv
To get started, you will need to create an .env file with the following contents:
// github.com/pusher-tutorials/pusher-channels-queue/go/.env
PUSHER_APP_ID="YOUR_APP_ID"
PUSHER_APP_KEY="YOUR_APP_KEY"
PUSHER_APP_SECRET="YOUR_APP_SECRET"
PUSHER_APP_CLUSTER="YOUR_APP_CLUSTER"
PUSHER_APP_SECURE="1"
Once this has been done, we will need to create a main.go file.
// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
package main
func main() {
port := flag.Int("http.port", 1400, "Port to run HTTP service on")
flag.Parse()
err := godotenv.Load()
if err != nil {
log.Fatal("Error loading .env file")
}
appID := os.Getenv("PUSHER_APP_ID")
appKey := os.Getenv("PUSHER_APP_KEY")
appSecret := os.Getenv("PUSHER_APP_SECRET")
appCluster := os.Getenv("PUSHER_APP_CLUSTER")
appIsSecure := os.Getenv("PUSHER_APP_SECURE")
var isSecure bool
if appIsSecure == "1" {
isSecure = true
}
client := &pusher.Client{
AppId: appID,
Key: appKey,
Secret: appSecret,
Cluster: appCluster,
Secure: isSecure,
}
mux := http.NewServeMux()
mux.Handle("/login", http.HandlerFunc(login(client)))
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), mux))
}
In the above, we created an HTTP server that responds to the login route. We will go on to implement the login function subsequently.
Since we will be using a memory mapped list of users to prevent complications that might drive us away from the main focus of the tutorial. We will need to go ahead to create those. Paste the following code in the main.go file.
// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
type User struct {
Email string
Password string
}
var (
validUsers = map[string]User{
"admin": User{
Email: "youremail@gmail.com",
Password: "admin",
},
"lanre": User{
Email: "youremail@gmail.com",
Password: "lanre",
},
}
)
You should replace
youremail@gmail.comwith your real email address so as to get the email when we get to the end of the tutorial.
Now back to the login function, you can go ahead to paste the following code in main.go
// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
func encode(w io.Writer, v interface{}) {
json.NewEncoder(w).Encode(v)
}
func login(client *pusher.Client) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
var request struct {
UserName string `json:"userName"`
Password string `json:"password"`
}
type response struct {
Message string `json:"message"`
Success bool `json:"success"`
}
// Make sure to only respond to the "/login" route
// due to limitations in the standard HTTP router
if r.URL.Path != "/login" {
w.WriteHeader(http.StatusNotFound)
return
}
// Only HTTP posts are accepted
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
w.WriteHeader(http.StatusBadRequest)
encode(w, response{"Invalid request body", false})
return
}
// Check if the user exists in our memory mapped list.
user, ok := validUsers[request.UserName]
if !ok {
w.WriteHeader(http.StatusBadRequest)
encode(w, response{"User not found", false})
return
}
// Do the passwords match ?
if user.Password != request.Password {
w.WriteHeader(http.StatusBadRequest)
encode(w, response{"Password does not match", false})
return
}
w.WriteHeader(http.StatusOK)
encode(w, response{"Login successful", true})
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
fmt.Fprintf(w, "userip: %q is not IP:port", r.RemoteAddr)
return
}
var ip = host
if host == "::1" {
ip = "127.0.0.1"
}
client.Trigger("auth", "login", &struct {
IP string `json:"ip"`
User string `json:"user"`
Email string `json:"email"`
}{
User: request.UserName,
IP: ip,
Email: user.Email,
})
}
}
While it is pretty easy to grok through the code above due to the inline comments, I will still like to go through the last few lines. Especially from Line 59.
- We get the IP of the user from
r.RemoteAddr.
Please note that if you end up running something that does this kind of IP fetching in production, this might not be the right approach if your Go application is behind a proxy.
- We also check to make sure we have a valid IP address by making use of the
net.SplitHostPortutility function. - Then we finally publish the data to the
authchannel.
At this point, the entire main.go should look like the following:
// github.com/pusher-tutorials/pusher-channels-queue/go/main.go
package main
import (
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net"
"net/http"
"os"
"github.com/joho/godotenv"
pusher "github.com/pusher/pusher-http-go"
)
type User struct {
Email string
Password string
}
var (
validUsers = map[string]User{
"admin": User{
Email: "youremail@gmail.com",
Password: "admin",
},
"lanre": User{
Email: "youremail@gmail.com",
Password: "lanre",
},
}
)
func main() {
port := flag.Int("http.port", 1400, "Port to run HTTP service on")
flag.Parse()
err := godotenv.Load()
if err != nil {
log.Fatal("Error loading .env file")
}
appID := os.Getenv("PUSHER_APP_ID")
appKey := os.Getenv("PUSHER_APP_KEY")
appSecret := os.Getenv("PUSHER_APP_SECRET")
appCluster := os.Getenv("PUSHER_APP_CLUSTER")
appIsSecure := os.Getenv("PUSHER_APP_SECURE")
var isSecure bool
if appIsSecure == "1" {
isSecure = true
}
client := &pusher.Client{
AppId: appID,
Key: appKey,
Secret: appSecret,
Cluster: appCluster,
Secure: isSecure,
}
mux := http.NewServeMux()
mux.Handle("/login", http.HandlerFunc(login(client)))
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", *port), mux))
}
func encode(w io.Writer, v interface{}) {
json.NewEncoder(w).Encode(v)
}
func login(client *pusher.Client) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
var request struct {
UserName string `json:"userName"`
Password string `json:"password"`
}
type response struct {
Message string `json:"message"`
Success bool `json:"success"`
}
if r.URL.Path != "/login" {
w.WriteHeader(http.StatusNotFound)
return
}
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
if err := json.NewDecoder(r.Body).Decode(&request); err != nil {
w.WriteHeader(http.StatusBadRequest)
encode(w, response{"Invalid request body", false})
return
}
user, ok := validUsers[request.UserName]
if !ok {
w.WriteHeader(http.StatusBadRequest)
encode(w, response{"User not found", false})
return
}
if user.Password != request.Password {
w.WriteHeader(http.StatusBadRequest)
encode(w, response{"Password does not match", false})
return
}
w.WriteHeader(http.StatusOK)
encode(w, response{"Login successful", true})
host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
fmt.Fprintf(w, "userip: %q is not IP:port", r.RemoteAddr)
return
}
var ip = host
if host == "::1" {
ip = "127.0.0.1"
}
client.Trigger("auth", "login", &struct {
IP string `json:"ip"`
User string `json:"user"`
Email string `json:"email"`
}{
User: request.UserName,
IP: ip,
Email: user.Email,
})
}
}
Run the Go program:
$ cd $GOPATH/src/github.com/pusher-tutorials/pusher-channels-queue/go
$ go run main.go
You can try to send requests to the service with cURL by:
$ curl -X POST localhost:1400/login -d '{"username" : "admin", "password" :"admin"}'
This will produce a response such as:
{"message":"Login successful","success":true}
Building the Node.js email service
We have made progress by publishing the events to Pusher Channels. You can verify that the events are published by looking at the Debug Console of the dashboard.
To build our Node.js email service, we will need to go back to the root directory, pusher-channels-queue. After which we will create the node directory as it will house our Node.js application.
$ mkdir node
We will need a couple libraries for the application;
pusher-js- the NodeJS SDK for Pusher Channels.nodemailer- We need this to send emails.dotenv- We need this to load environment variables from a file.handlebars- We need to dynamically replace contents of the email before sending it. Things like username and IP address come to mind here.fs- We need to be able to read the content of the email template from the filesystem. You can have a look at the email template here.
To install the above, you will need to create a package.json file that contains the following:
// github.com/pusher-tutorials/pusher-channels-queue/node/package.json
{
"dependencies": {
"dotenv": "^6.2.0",
"fs": "^0.0.1-security",
"handlebars": "^4.0.12",
"nodemailer": "^4.7.0",
"pusher-js": "^4.3.1"
}
}
You will need to run npm install to get install those dependencies.
Since we need to subscribe to Pusher Channels, we need to first include the required values in .env.
// github.com/pusher-tutorials/pusher-channels-queue/node/.env
PUSHER_APP_CLUSTER="YOUR_APP_CLUSTER"
PUSHER_APP_SECURE="1"
PUSHER_APP_KEY="YOUR_APP_KEY"
MAILER_EMAIL="you@gmail.com"
MAILER_PASSWORD="Password"
Then create an index.js file
// github.com/pusher-tutorials/pusher-channels-queue/node/index.js
require('dotenv').config();
const Pusher = require('pusher-js');
const nodemailer = require('nodemailer');
const handlebars = require('handlebars');
const fs = require('fs');
const pusherSocket = new Pusher(process.env.PUSHER_APP_KEY, {
forceTLS: process.env.PUSHER_APP_SECURE === '1' ? true : false,
cluster: process.env.PUSHER_APP_CLUSTER,
});
const transporter = nodemailer.createTransport({
service: 'gmail',
auth: {
user: process.env.MAILER_EMAIL,
pass: process.env.MAILER_PASSWORD,
},
});
const channel = pusherSocket.subscribe('auth');
channel.bind('login', data => {
fs.readFile('./index.html', { encoding: 'utf-8' }, function(err, html) {
if (err) {
throw err;
}
const template = handlebars.compile(html);
const replacements = {
username: data.user,
ip: data.ip,
};
let mailOptions = {
from: '"Pusher Tutorial demo" <foo@example.com>',
to: data.email,
subject: 'New login into Pusher tutorials demo app',
html: template(replacements),
};
transporter.sendMail(mailOptions, function(error, response) {
if (error) {
console.log(error);
callback(error);
}
});
});
console.log(data);
});
In the above code, we read the contents of index.html and process it like a handlebars template with handlebars.compile(html). This is because we are dynamically replacing {{ username }} and {{ ip }}.
So far, we have not created the index.html . You will need to create the aforementioned file and paste the following contents:
// github.com/pusher-tutorials/pusher-channels-queue/node/index.html
<!doctype html>
<html>
<head>
<meta name="viewport" content="width=device-width" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Simple Transactional Email</title>
<style>
/* -------------------------------------
GLOBAL RESETS
------------------------------------- */
/*All the styling goes here*/
img {
border: none;
-ms-interpolation-mode: bicubic;
max-width: 100%;
}
body {
background-color: #f6f6f6;
font-family: sans-serif;
-webkit-font-smoothing: antialiased;
font-size: 14px;
line-height: 1.4;
margin: 0;
padding: 0;
-ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%;
}
table {
border-collapse: separate;
mso-table-lspace: 0pt;
mso-table-rspace: 0pt;
width: 100%; }
table td {
font-family: sans-serif;
font-size: 14px;
vertical-align: top;
}
/* -------------------------------------
BODY & CONTAINER
------------------------------------- */
.body {
background-color: #f6f6f6;
width: 100%;
}
/* Set a max-width, and make it display as block so it will automatically stretch to that width, but will also shrink down on a phone or something */
.container {
display: block;
Margin: 0 auto !important;
/* makes it centered */
max-width: 580px;
padding: 10px;
width: 580px;
}
/* This should also be a block element, so that it will fill 100% of the .container */
.content {
box-sizing: border-box;
display: block;
Margin: 0 auto;
max-width: 580px;
padding: 10px;
}
/* -------------------------------------
HEADER, FOOTER, MAIN
------------------------------------- */
.main {
background: #ffffff;
border-radius: 3px;
width: 100%;
}
.wrapper {
box-sizing: border-box;
padding: 20px;
}
.content-block {
padding-bottom: 10px;
padding-top: 10px;
}
.footer {
clear: both;
Margin-top: 10px;
text-align: center;
width: 100%;
}
.footer td,
.footer p,
.footer span,
.footer a {
color: #999999;
font-size: 12px;
text-align: center;
}
/* -------------------------------------
TYPOGRAPHY
------------------------------------- */
h1,
h2,
h3,
h4 {
color: #000000;
font-family: sans-serif;
font-weight: 400;
line-height: 1.4;
margin: 0;
margin-bottom: 30px;
}
h1 {
font-size: 35px;
font-weight: 300;
text-align: center;
text-transform: capitalize;
}
p,
ul,
ol {
font-family: sans-serif;
font-size: 14px;
font-weight: normal;
margin: 0;
margin-bottom: 15px;
}
p li,
ul li,
ol li {
list-style-position: inside;
margin-left: 5px;
}
a {
color: #3498db;
text-decoration: underline;
}
/* -------------------------------------
BUTTONS
------------------------------------- */
.btn {
box-sizing: border-box;
width: 100%; }
.btn > tbody > tr > td {
padding-bottom: 15px; }
.btn table {
width: auto;
}
.btn table td {
background-color: #ffffff;
border-radius: 5px;
text-align: center;
}
.btn a {
background-color: #ffffff;
border: solid 1px #3498db;
border-radius: 5px;
box-sizing: border-box;
color: #3498db;
cursor: pointer;
display: inline-block;
font-size: 14px;
font-weight: bold;
margin: 0;
padding: 12px 25px;
text-decoration: none;
text-transform: capitalize;
}
.btn-primary table td {
background-color: #3498db;
}
.btn-primary a {
background-color: #3498db;
border-color: #3498db;
color: #ffffff;
}
/* -------------------------------------
OTHER STYLES THAT MIGHT BE USEFUL
------------------------------------- */
.last {
margin-bottom: 0;
}
.first {
margin-top: 0;
}
.align-center {
text-align: center;
}
.align-right {
text-align: right;
}
.align-left {
text-align: left;
}
.clear {
clear: both;
}
.mt0 {
margin-top: 0;
}
.mb0 {
margin-bottom: 0;
}
.preheader {
color: transparent;
display: none;
height: 0;
max-height: 0;
max-width: 0;
opacity: 0;
overflow: hidden;
mso-hide: all;
visibility: hidden;
width: 0;
}
.powered-by a {
text-decoration: none;
}
hr {
border: 0;
border-bottom: 1px solid #f6f6f6;
Margin: 20px 0;
}
/* -------------------------------------
RESPONSIVE AND MOBILE FRIENDLY STYLES
------------------------------------- */
@media only screen and (max-width: 620px) {
table[class=body] h1 {
font-size: 28px !important;
margin-bottom: 10px !important;
}
table[class=body] p,
table[class=body] ul,
table[class=body] ol,
table[class=body] td,
table[class=body] span,
table[class=body] a {
font-size: 16px !important;
}
table[class=body] .wrapper,
table[class=body] .article {
padding: 10px !important;
}
table[class=body] .content {
padding: 0 !important;
}
table[class=body] .container {
padding: 0 !important;
width: 100% !important;
}
table[class=body] .main {
border-left-width: 0 !important;
border-radius: 0 !important;
border-right-width: 0 !important;
}
table[class=body] .btn table {
width: 100% !important;
}
table[class=body] .btn a {
width: 100% !important;
}
table[class=body] .img-responsive {
height: auto !important;
max-width: 100% !important;
width: auto !important;
}
}
/* -------------------------------------
PRESERVE THESE STYLES IN THE HEAD
------------------------------------- */
@media all {
.ExternalClass {
width: 100%;
}
.ExternalClass,
.ExternalClass p,
.ExternalClass span,
.ExternalClass font,
.ExternalClass td,
.ExternalClass div {
line-height: 100%;
}
.apple-link a {
color: inherit !important;
font-family: inherit !important;
font-size: inherit !important;
font-weight: inherit !important;
line-height: inherit !important;
text-decoration: none !important;
}
.btn-primary table td:hover {
background-color: #34495e !important;
}
.btn-primary a:hover {
background-color: #34495e !important;
border-color: #34495e !important;
}
}
</style>
</head>
<body class="">
<table role="presentation" border="0" cellpadding="0" cellspacing="0" class="body">
<tr>
<td> </td>
<td class="container">
<div class="content">
<!-- START CENTERED WHITE CONTAINER -->
<table role="presentation" class="main">
<!-- START MAIN CONTENT AREA -->
<tr>
<td class="wrapper">
<table role="presentation" border="0" cellpadding="0" cellspacing="0">
<tr>
<td>
<p>Hi {{ username }},</p>
<p>You’ve successfully signed into the demo app.</p>
<p>You signed in from the IP address, {{ ip }}</p>
<table role="presentation" border="0" cellpadding="0" cellspacing="0" class="btn btn-primary">
<tbody>
<tr>
<td align="left">
<table role="presentation" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td> <a href="https://pusher.com"
target="_blank">Visit
Pusher</a> </td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- END MAIN CONTENT AREA -->
</table>
<!-- END CENTERED WHITE CONTAINER -->
</div>
</td>
<td> </td>
</tr>
</table>
</body>
</html>
We listen for the login event and pick out the important data from there. In this case, the user’s name and IP address from which they logged in. After which we send the email to the user.
You will need to start the Node.js service by running node index.js. After doing that, you can send login requests to the Golang service again.
You should check your email:
Please note that you might need to allow “Insecure apps”. Please visit https://support.google.com/accounts/answer/6010255?hl=en
Conclusion
In this tutorial, we have leveraged Pusher Channels as a messaging queue between two different applications. While we used this to send email notifications, we can use this for much more interesting patterns depending on your application’s needs.
The entire source code of this tutorial can be found on GitHub.
4 February 2019
by Lanre Adelowo