Live Page-View Counter, Comet server and JSON-push

Overview

A "page-view counter" or "hit counter" is a mechanism that displays the number of page-views on an HTML page. It uses a server side of script that counts the page-views, dynamically generates an HTML page on the server side, and returns it back to the browser. Although it accurately displays the number of page-views at the point when the HTTP request was made to fetch the HTML page, it will not reflect additional changes in the page-views after that point because there is no further communication between the server and the client (unless you hit F5 to refresh the page).

This "Live Page-View Counter" is an experimental implementation of "live" version of "page-view counter", which displays the "current" page-views, dynamically updating the counter on the HTML page, utilizing Comet server and JSON, which I call "JSON-push".

[Notes] Regarding Comet server, I strongly recommend you to read "Comet: Low Latency Data for the Browser" by Alex Russell, before you proceed to the next section.

See it and believe it

Take a look at the sample implementation below, pasted to this blog entry using IFRAME. Pay attention to the counter, which will automatically increments when a new visitor comes to this blog. You just need a regular browser (it works on IE, Firefox and Safari) and don't need to hit F5.

You also see footstep icons below the page counter, which change their colors occasionally. Those changes are not initiated by Javascript on this page or the server side of script, but other users who are reading this blog while you are looking at this blog. This is a service called "Footstep live!", which I will explain later.

How It Works

Unlike a traditional "page-view counter", the "live" version returns a static HTML page instead of dynamically generated HTML page. This HTML page, however, has following line that acts as a "page-view" signal to the server.

<script type="text/javascript"
  src="http://lab.uievolution.com:8080/~onload">
</script>

"http://lab.uievolution.com:8080" specifies the Comet server and its port, and "/~onload" is an URI that indicates that the page is being loaded. When the Comet server receives this HTTP request, it increments the internal page counter, performs a few additional tasks (to be described later [*1]) and returns following script immediately (it acts like a regular HTTP server in this particular request).

onLoadCallback({"Count": N});

where N is the value of the page counter (the Comet server running behind this sample returns additional information, but they are not essential and you can ignore them for now). The "onLoadCallback" function on the HTML page displays this value in an appropriate format (the onLoadCallback function on this sample actually stores the json object in a global variable temporarily, then populates its values to the HTML page when it gets "onload" event on BODY tag).

This is how this page displays the initial page-view number when the page is loaded, which is the same value as the traditional page-view counter would display. The only difference is that the data-binding (the binding of "view" and "data") is done on the client-side, instead of the server-side.

After the page is completely loaded, the browser calls the following function.

function connectComet() {
    aObj = new JSONscriptRequest('http://lab1.uievolution.com:8080/~listen');
    aObj.buildScriptTag();
    aObj.addScriptTag();
}

This function uses JSONScriptRequest, which is in jsr_class.js written by Jason Levitt. This function dynamically creates a new script tag that is equivalent to.

<script type="text/javascript"
  src="http://lab1.uievolution.com:8080/~listen">
</script>

This is also an HTTP request to the same Comet server (ignore that fact that "lab" became "lab1" for now - I will explain it later [*2]), but the URI "/~listen" indicates that the client is now ready for JSON-push. When the Comet server receives this request, it adds the socket to the "JSON-push client list" -- the list of sockets waiting for JSON-push, instead of returning an HTTP response immediately.

JSON-push is essentially a mechanism for the Comet server to push any Javascript to those clients in the "JSON-push client list", either to a particular set of clients or to the all the clients.

For example, the Comet server may send the following script to all the JSON-push clients when it increments the counter (when the Comet server receives "/~onload" request. See [*1] above).

setCount({"Count": N});
reconnectComet();

setCount() updates the counter on the HTML page (performs the same task as onLoadCallback()), and reconnectComet() will re-establish the connection to the Comet server. Here is the implementation of reconnectComet().

function() {
    aObj.removeScriptTag();
    connectComet();
}

JSON-dispatch

Although the mechanism described above was sufficient to implement the "live" version of page-view counter, I soon encountered the backward compatibility issue. When I was testing a new feature using the same Comet server (I have no staging server to test), I have realized that I can't broadcast an arbitrary Javascript (such as "updateFootstep();") to the all the JSON-clients, because it causes script errors on existing HTML pages and will break the connections to the Comet server.

In order to work-around this problem, I added an indirection mechanism, which I call JSON-dispatch.

Instead of directly pushing function calls that may or may not exist on the client side, the Comet server wraps each function call in a function call to jsonCallback() with the function identifier as a "callback" property of the JSON object. For example, the example described above,

setCount({"Count": N});
reconnectComet();

becomes

jsonCallback({"Count": N, "callback": "counter"});
reconnectComet();

where the "counter" is the function identifier. The implementation of jsonCallback() would become like this:

function jsonCallback(json)
{
    // Dispatch JSON callback functions
    id = json.callback;
    if (id=='counter') {
        setCount(json);
    }
}

Because of this mechanism, those HTML pages can gracefully ignore the unknown JSON-push like this:

jsonCallback({"Id": 1, "Color": "red", "callback": "footstep"});

DNS hack

I have mentioned above that it uses "lab.uievolution.com:8080" for "/~onload", but "lab1.uievolution.com:8080" for "~listen" (see [*2]), although both URL are mapped to the Comet server by DNS. Originally, I was using the same URL, and the page works fine without this hack. I, however, noticed that the page counter completely stops working when I open two instances of this page at the same time. Thanks to Kenn (Kentaro Ejima, who developed the Lingr chat service, which also uses Comet server), I have learned that this problem was caused by the maximum number of simultaneously opened HTTP requests to the same server, which is "two" by default. Because the Comet server does not immediately respond to the "/~listen" request from the client and hang onto it, we hit this limitation when the user opens the second instance of the page, and any further HTTP requests (including the request to port 80, where I am running a regular Apache server) will be deferred until the Comet server responds to those requests (which could be several seconds later).

While I was tracking this problem, Kenn taught me how to work around, which is this DNS hack. Simply mapping another server name (in this case 'lab1.uievolution.com') to the same IP address, the browser is able to open more than two HTTP requests simultaneously. In other word, this is an intentional Cross Domain Comet.

The "/~listen" requests will still encounter this limitation when the user opens more than two instances of this page, but both "/~onload" requests and regular HTTP requests always works (they always immediately return). This is still not perfect, but sufficient practically.

The Service Architecture

The Comet server I am using in this experiment is the "Micro Comet Server", which I have implemented in C++ from scratch while I was learning Linux programming (I am quite new to Linux). Based on KISS principle (Keep It Simple, Stupid!), I have made it quite simple and efficient. The code size is quite small (less than 20KB at this moment - and I would like to keep it as small as possible), and it is able to handle millions of transactions (HTTP requests) per day (I don't know the exact limit yet, but I am guessing it is two to three million transactions on a single core Pentium machine). It theoretically supports up to 1024 concurrent clients, although I have tested only 200 so far. It is supposed to linearly scale (to the number of transactions - not to the number of clients), although I don't have enough data to prove it yet.

Because I have only one physical server to play with, I am running an Apache server on port 80 to process regular HTTP requests and the Micro Comet Server on port 80 to process "/~onload" and "/~listen" requests from the client. The Micro Comet Server pushes its own scripts to the clients (such as the page-view update event), but also provides a web service for the Apache server to push arbitrary javascripts to the Comet clients. It allows me to implement variety of Comet-based service applications in regular scripting languages (such as PHP, Perl and Ruby) without mixing any application specific code to the Micro Comet Server.

Footstep live!

The "Footstep Live!" is another application that demonstrates the unique user experience enabled by a Comet server. It allows users to change the colors of footstep icons by simply clicking them, and share those changes with all the other people who are looking at the same page simultaneously.

Comments

[Rob]

Thanks for this well written description of how Comet works.

[Ioana]

Hello,
I am very interested in your ideea. But I was wondering if you could share the sources?
I've found very few implemented exemples over the web. There are a lot of articles about Comet, but not enough examples.

[Hanson]

If I could not find the working source code for the example, I don't want to read the article.

[Øystein Pedersen]

Would it be at all possible to receive the server code for this example?

[phero]

The footstep example doesn't work, i change the color in IE, but nothing happens in Firefox, also the counter is not updated. Shame, it sounded promising.

[phero]

The footstep example doesn't work, i change the color in IE, but nothing happens in Firefox, also the counter is not updated. Shame, it sounded promising.

[LE LONG Vu]

Thank for your post. It’s very good but can you share complete project code of this example.
The reason is that I can easy to apply it for solve my problem.
I am working on the project which is related to JSON and JavaScript, your post is very useful for me. However if you can implement it to a detail example (e.g.: just a small project or small html file).
I appreciate it very much.