How to Add Simple Real Time Streaming to Your Express App with Server Sent Events

A lot of web applications these days require fast and lightweigh real time communication.

Real time analytics, real time measurement, games, chat applications, syncing with other devices, syncing with other people when coloborating on a document, etc.

Last time I showed you a solution to all of those problems with WebSockets. It works, most of the time, but often it is too complex.

What’s more it uses a completely different protocol from HTTP. Some users don’t support it and some host providers don’t allow it.

Wouldn’t it be nice if you could have

  • realtime communication but only over HTTP?
  • very simple to use that can be easily added to any web stack?
  • reconnection automatically without your intervention?

Everybody is talking only about the syxy WebSockets and as a result, we often forget its little sister Server Sent Events (SSE).

SSE can help you with all of the above and give you a little bit more, too.

What is SSE?

Server Sent Events is another technology for real time communication with the server.

Its main purpose is to stream events from the server to the clients because unlike WebSockets it works only in this direction.

Clients cannot use SSE to send back events to the server.

One of the big advantages of SSE is that it is done only over HTTP. You don’t require a fancy new protocol, and you avoid all of the problems that come with it.

Another nice feature is automatic reconnection. Whenever, the connection drops your browser is resopnsible for automatically reestablishing it. You don’t have to do anything yourself.

You will see that SSE is that simple that you can actually implement it yourself even on clients which are not browsers like mobile devices for example.

Web is often only one way realtime

Now that you know that SSE is capable of sending lightweight real time events only from the server to the client, maybe you are a little disappointed and you want to go back to WebSockets.

However, for many applications this is all they need. For example, receiving status updates on Twitter can be done with SSE, also getting the current stock price and its updates.

Even the usual example of WebSockets tutorials with chat application needs real time communication only from the server to the client and SSE is more than enough.

No matter how fast you write your messages AJAX is more than enough to sent them and SSE to provide you with all messages from your friends.

You only need two way real time communication when you have a lof of messages in both ways. More often than not, you only need events sent from the server and AJAX is enough for the rest.

How SSE works?

Server sent events are very simple. All you have to do is to construct a plain text response with the Content-Type header set to text/event-stream and then keep the connection open. Then you can send more plain text responses over the same connection.

This is it.

Your plane text response is expected to be a single line like this:

data: What a great message!\n\n

The double “\n\n” (new line character) is how you signal the end of the message to the browser.

Only after you do that it is delivered. The text after data: is the actual message that the browser will receive.

However, sometimes you might want to send a message on multiple lines. In this case it should look like this

data: An even better message\n
data: Don't you think?\n\n

This whole piece is delivered as a single multiline message. Again \n\n signals the end of the message and between lines you are using only a single \n. However, don’t forget that every line should start with data:.

On the client side in the browser you can listen for those messages in the following way:

if (!!window.EventSource) {
  var source = new EventSource('/path/to/sse/')

  source.addEventListener('message', function(e) {
    console.log(e.data)
  }, false)

  source.addEventListener('open', function(e) {
    console.log("Connection was opened")
  }, false)

  source.addEventListener('error', function(e) {
    if (e.readyState == EventSource.CLOSED) {
      console.log("Connection was closed")
    }
  }, false)
}

As you can see it is extremely simple. First, you check whetehr SSE is supported on the borwser.

Then you connect to your web app and it creates a source, an EventSource, where you can listen for events.

In e.data you will find your messages from the server and it works for both single and multiline messsages.

As you can see by default there are three type of events. First, an event when a connection is open, then an event when it is closed or reconnecting and finally an event when you receive a new message.

A neat little feature is that you can attach unique id to each event:

id: 5678\n
data: Hello, World!\n\n

If the connection drops, when the browser restores it automatically. You will receive on the server a header Last-Event-ID containing the last id it received, which can help you with replying all missed events.

By default the browser reconnects after 3 seconds, but you can control this

retry: 10000\n
data: Don't talk to me too much\n\n

This value is in milliseconds so the browser will try to reconnect after 10 seconds in case of a lost connection.

Last but not least, by default you only have three events: open, error and message. The first two are automatically generated by the browser.

You can easily create more events like this

event: spilledWater\n
data: {"user": "Peter", "spilledFrom": "bottle"}\n\n

Then you can listen for this event

source.addEventListener('spilledWater', function(e) {
    var data = JSON.parse(e.data);
    console.log('This person spilled the water:' + data.user);
}, false)

This will not fire the message event, only the spilledWater event.

Example: Voting system

Let’s see all of this in action. We are going to build a real time voting app.

It will have two pages. One page for people to answer. Another where the overall results of the answers will be updated in real time.

For example during a conference the participants can vote and they can see the result of their vote on a big screen in real time.

We are going to use Express with Jade as templating language.

$ npm install express jade

Our project structure will be very simple

views/
  vote.jade
  result.jade
sse.js
app.js

Unlike WebSockets and other technologies, we are not going to install an NPM module to help us.

SSE is so simple that instead, we are going to create a middleware to handle our needs.

SSE middleware

module.exports = function (req, res, next) {
  res.sseSetup = function() {
    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive'
    })
  }

  res.sseSend = function(data) {
    res.write("data: " + JSON.stringify(data) + "\n\n");
  }

  next()
}

This is all you need. On method to setup SSE and one more to send events, after you’ve called the previous method.

Your App

The entry point of your application is app.js

var express = require('express')
  , app = express()
  , sse = require('./sse')

var connections = []
  , votes = {yes: 0, no: 0}

app.engine('jade', require('jade').__express) //__
app.set('view engine', 'jade')

app.use(sse)

app.get('/', function(req, res) {
  res.render('vote')
})

app.get('/result', function(req, res) {
  res.render('result')
})

app.get('/vote', function(req, res) {
  if (req.query.yes === "true") votes.yes++
  else votes.no++

  for(var i = 0; i < connections.length; i++) {
    connections[i].sseSend(votes)
  }

  res.sendStatus(200)
})

app.get('/stream', function(req, res) {
  res.sseSetup()
  res.sseSend(votes)
  connections.push(res)
})

app.listen(3000, function() {
  console.log('Listening on port 3000...')
})

This is a standard Express application. It renders a voting page and a result page. It also provides a url to reiceive votes and a url to listen to a stream of events.

You can see that the simple SSE middleware is enough to create this advance real time app. You only have to load it with app.use(sse) or even directly with app.use(require('./sse')).

However, don’t forget to load it before you define the url which will use it.

Client Side Voting

Let’s see how the part where the users vote looks like

doctype html
html
  head
    title Voting for the world peace
  body
    h1 Will "Hello, World!" bring world peace?
    <div id="yes">Yes</div>
    <div id="no">No</div>
    <div id="message"></div>
  script(src="//code.jquery.com/jquery-2.1.4.min.js")
  script.
    function vote(yes) {
      $.get("/vote?yes=" + yes, function() {
        $("#yes").hide()
        $("#no").hide()
        $("#message").text("Thank you for your vote!")
      })
    }
    
    $("#yes").on("click", function() {
      vote(true)
    })

    $("#no").on("click", function() {
      vote(false)
    })

There is yes and no on the page. Cliking either of it send an AJAX request to your server with the chosen value.

As you saw in the server code above, it accumulates those responses and send them back to all connections.

Client Side Results

Next is where all the magic happen. The results of the voting are displayed with this.

doctype html
html
  head
    title Voting results for world peace
  body
    div(id="votes") Yes: 0, No: 0
    div(id="state") Disconnected
  script(src="//code.jquery.com/jquery-2.1.4.min.js")
  script.
    if (!!window.EventSource) {
      var source = new EventSource('/stream')

      source.addEventListener('message', function(e) {
        votes = JSON.parse(e.data)
        $("#votes").text("Yes: " + votes.yes + ", No: " + votes.no)
      }, false)

      source.addEventListener('open', function(e) {
        $("#state").text("Connected")
      }, false)

      source.addEventListener('error', function(e) {
        if (e.target.readyState == EventSource.CLOSED) {
          $("#state").text("Disconnected")
        }
        else if (e.target.readyState == EventSource.CONNECTING) {
          $("#state").text("Connecting...")
        }
      }, false)
    } else {
      console.log("Your browser doesn't support SSE")
    }

This page is not only updating the number of votes, but it is also keeping up to date with the current connection state.

Let’s play

You built it. Let’s play and vote to see how it will work.

First, try to open several tabs. Two with results and two more for voting. It is even better if you open them in different windows and put them side by side.

Then vote and watch how the pages with the results update immediately.

Then kill your node server, but keep the results pages open and you will see the state chaning to Connecting, because the browser is try to restore the lost connection.

Run the server again and the state will automatically return to Connected, but the results will be nulified because we stored them in the memory.

Isn’t it fun? :-)

As you can see building real time apps with SSE is very easy.

It’s not all roses

Just like with WebSockets, it is not all roses for SSE. There are still clients which don’t support it.

SSE is also producing long open connections, which altough lightweight, might still saturate your server abilities to accept new connections.

Next

Real time is shiny but you better use it with care only when you need it. It is very easy to fall into a trap of always using the latest technology.

Everybody is talking about WebSockets, and use it without much thinking about it whenever they need real time communication. You should check very well wether you need duplex comunication like in WebSockets or SSE is all that you want.

Finally, the SSE middleware that you built above is very simple. Try to enhance it by using unique ids and custom events.


Other articles that you may like

Did you like this article?

Please share it on
Enter your email and get our NPM Cheat Sheet for NodeJS Developers and the links to our 5 most popular articles which have helped thousands of developers build faster, more reliable and easier to maintain Node applications.
100% privacy. No spam.

Best Practices for Node & Express Web Apps

Express makes it very simple to develop your web app, but it lacks structure. It is easy to begin but when your app grows in complexity and your team in size developing and maintaining your app is no more simple.

This is not how it should be. Node & Express promise to make your life easier.

We have created a video course for you to teach you what we have learned from years of development with Node & Express.

It is all about best practices which make development and maintenance of Express apps simple again.

Join our video course on best practices in ExpressJS

Freelancing is hard. You feel alone. You have to manage everything yourself and some clients make it even harder than it should be.

Never ending projects. Clients constantly calling you for no reason.

Sounds familiar? Join hundreds of freelancers with our free

7-part guide to dealing with difficult clients

We are Stefan Fidanov & Vasil Lyutskanov. We share actionable advice about development with Node, Express, React and other web & mobile technologies.

It is everything that we have learned from years of experience working with customers from all over the world on projects of all sizes.


© 2016 Terlici Ltd · Terms · Privacy