Server-Sent Events with Node.js (Express)

09 November 2011

First, an overview...

What are Server-Sent Events?

From the specification:

"...an API for opening an HTTP connection for receiving push notifications from a server in the form of DOM events. The API is designed such that it can be extended to work with other push notification schemes such as Push SMS." -http://dev.w3.org/html5/eventsource/

So, another way of sending data to a client...but different in a few ways. Here are the unique characteristics of SSEs over other things you may be thinking of...

SSEs are/do:

  1. A (standard) way for you to open a connection to many clients and push data unilaterally from the server to the client.
  2. Automatically re-establish a connection in the event that an error is raised or the connection is lost. So, your browser will reconnect for you...you don't have to handle that in your client-side code.
  3. Allows you to "pick-up where you left off" in the event that you have dropped a connection and missed a message in real-time. SSE-compatible clients (ie: browsers that support the feature) will include a "Last-Event-ID" header when re-establishing a connection. On the server-side, if you sniff for this header and respond accordingly, you can re-play/send the messages that specific client missed if necessary.
  4. Operate over standard HTTP connections...less issues to worry about with regards to open ports or other protocols a piece of software in your stack needs to support (eg: your web or application server should be compatible no matter what).

SSEs are/do NOT:

  1. A replacement for polling. Polling is when you ask the server over & over whether it has new data. SSEs are a single long-running, open connection.
  2. A replacement for long-polling. Long-polling is when a client (browser) open a connection to a server/stream and waits until it receives some data...at which point the connection is closed and a new connection is re-established. SSEs are a single long-running, open connection.
  3. Enable bi-directional, full-duplex communication with the server. Servers are allowed to push data out to clients, but communication back would have to be done via other methods/tools.

Big-picture-wise, what are we dealing with?

Client-side walk-through

You use the EventSource constructor to set up an SSE connection. For Brevity's sake, in the example below we'll go through the simplest case: connecting to a locally-running node.js application ("simplest" is arguable, I suppose).

Note: I may post a follow-up article with some examples of how to configure this behind a web server and set up CORS support. So, let me know if that would be of interest to you...as it will increase the likelihood of me actually putting another article together on the topic.

A bare-bones client-side use of EventSource would entail something like the following:

<script type="text/javascript">
  var source = new EventSource('/stream');

  source.addEventListener('message', function(e) {
    console.log(e.data);
  }, false);
</script>

...which will print whatever you receive from the server out to your console. Not awesome, but...easy to follow.

Now, a few things to note about this are:

  1. How small it is. No extra libraries necessary.
  2. No reconnection logic. If your cycle your application servers, web server, or the user loses their internet connection...it's fine. The browser will attempt to reconnect every 3-seconds or so (which can be configured by the server).
  3. In this example we are just listening for onmessage events, there are other standard events (onopen, onerror). Additionally, you can send custom events as well. For example, you could send an event named my-custom-event, and any client who has registered for it will receive those messages in a separate "stream" of messages. So, you don't have to have one really chatty message pipe that all clients have to listen to and parse for only the messages that apply to their situation...you can just make extra "streams" to subscribe to.

Server-side walk-through

The server-side is surprisingly simple. Here are the highlights:

  1. When the client defines the EventSource entity on the client side, the browser sends a request to the server with an Accept header of text/event-stream.
  2. The server immediatly responds with a 200 status code and a keepalive header...leaving the connection....alive...(ie: 'open').
  3. Each message from the server has a defined format of 2-3 fields: event (optional), id (what uniquely identifies this set of data...more later), data (the actual content of the event you have set up the listener for)...followed by an empty newline.
  4. If you define the event field, the client must have explicitly subscribed to updates from the defined event in order to receive messages on that "stream". For example, if you were to send the following message over the wire:

    event: newcontent
    id: 10222
    data: this is new stuff on the 'newcontent' event
    

...the only "clients" who would recieve this specific event would be those who had something along the lines of the following:

  <script type="text/javascript">
    var source = new EventSource('/stream');

    source.addEventListener('newcontent', function(e) { // note: 'newcontent'
      console.log(e.data);
    }, false);
  </script>

In the event that the connection was dropped (internet lost on client-side, server died on the server side, etc), the client will attempt to reconnect to the originally defined URL (/stream above). On the server-side, when the client (browser) attempts to reconnect, it will set a header of Last-Event-Id. If applicable, the server can sniff out the Last-Event-id header and re-send any missed transactions to the client.

Sounds cool. Let's see it in action...

I'll assume you have Node.js set up on your machine. I used Express, as it makes it pretty easy to clearly see what's going on without getting hung up on how the request & URL parameters are being handled. I also used Redis to show how you can use the pub/sub feature to send data down the pipe to your clients...instead of just a timer (or whatever)...as it seems relatively realistic.

You can see the whole codebase as a single unit by cloning the repo at https://github.com/tomkersten/sses-node-example.

Here are the important parts:

app.js

The brains of the application...all the supported/required actions are defined in here. There are three "URLs" defined:

  1. / - a bland HTML page used to render the data delivered across the wire via SSEs
  2. /update-stream - the URL the EventSource connects to in order to receive updates from the app
  3. /fire-event/:event_name - a URL you can hit to "kick off" a message to be sent to all clients.

views/layout.jade

A simple layout...

views/index.jade

Ridiculously-simple index page rendered when you hit '/'. Data the client-side receives is appended to the 'data' ul.

What's going on?

When you hit /, the index page is rendered inside the layout...and in doing so, sets up an EventSource entity on the client-side...subscribing to all messages sent over that channel.

When the server receives this request (to /update-stream), it immediately responds with the necessary headers (200 status code, keepalive, et cetera), sets up the subscriber to listen to the appropriate pub/sub channel (updates).

Whenever you visit /fire-event/:event_name (where :event_name is whatever you want), the other Redis client (publisherClient) sends a message out over the updates channel...which the subscriber client receives & shoves down the pipe to the clients.

It's that easy.

What'd I miss?

In addition to the outline above, I've tried to comment the code where I thought it would be important and/or potentially unclear to a reader. Please let me know if you see any errors or content you think should be described in a bit more detail.

I picked up a lot of stuff while figuring this out, some of which is not outlined here. Hopefully, this post has left you with a bit of a head start on where I started and a little bit of inspiration to try it out for yourself.