Monday, July 16, 2012

Realtime Web Chat with Socket.io and Gevent

In a previous post, I described how to build a realtime analytics graph with socket.io. That was nearly a year ago, however, and both socket.io and the Python library gevent-socketio have both undergone significant changes. In this post, then, I'll use the latest versions of each to build a web chat server.

Requirements

Before you get started, you should make sure you have the prerequisite libraries installed. For this post, I used version 0.3.5-rc2 of gevent-socketio and version 0.9.6 of the socket.io client library. To install gevent-socketio, you can use pip:

$ pip install gevent-socketio==0.3.5-rc2

The socket.io file socket.io.min.js can be retrieved from the socketio-client Github repository.

Building the HTML

Our HTML structure is fairly simple: there is a login pane at the top to set your and a two-column chat pane where the left column displays current users and the right column displays the chat:

<h1>Socket.io Chatterbox</h1>
<div id="status" class="border">Disconnected</div>
<form id="login">
  <label for="login-input">Login:</label>
  <input id="login-input">
</form>
<div>
  <div class="border left">
    <h3>Nicks</h3>
    <ul id="nicks">
    </ul>
  </div>
  <div class="border right">
    <form id="chat">
      <label for="chat-input">Chat:</label>
      <input id="chat-input">
    </form>
    <div id="chat-data">
    </div>
  </div>

At the end of this, we have our javascripts:

<script type="text/javascript" src="/js/jquery.min.js"></script>
<script type="text/javascript" src="/js/socket.io.min.js"></script>
<script type="text/javascript" src="/js/test.js"></script>

Add a bit of minimal styling in the <head> and we're done with the HTML:

<style>
  .border { border: 1px solid black; }
  input { width: 35em; }
  .left { width: 25%; float: left; margin: 20px; }
  .right { width: 60%; float: left; margin: 20px;}
  #chat-data { height: 400px; overflow: auto; }
</style>

Building the Python server

The biggest change to gevent-socketio since my last post is the addition of namespaces. A Namespace is an object that manages a connection to a particular socket.io client. The basic idea is that the client-side socket can emit an event with data attached, triggering a handler method in the Namespace. For instance, if the client sent the login event using socket.emit('login', username), the server would call namespace.on_login(username).

Building a gevent-socketio Namespace

In our example here, we'll use the ChatNamespace class to keep a registry of ChatNamespace objects so they can communicate. To begin our namespace, we set up the registry:

from socketio.namespace import BaseNamespace

class ChatNamespace(BaseNamespace):
    _registry = {}

The next thing we can do is define initialize and disconnect methods on the namespace. These will be called when we get a new connection and when that connection dies, respectively. In our initialize method, we add ourselves to the registry and send a message to the client noting that we're connected. In disconnect, we just make sure to de-register this client from the registry:

    def initialize(self):
        self._registry[id(self)] = self
        self.emit('connect')
        self.nick = None

    def disconnect(self, *args, **kwargs):
        if self._nick:
            self._users.pop(self._nick, None)
        super(ChatNamespace, self).disconnect(*args, **kwargs)

Now we can define our event handlers. The event's we're interested in are the following:

  • login - This event fires when the user chooses a nickname
  • chat - This event fires when a chat message is received from a client

To handle login, we do a few things:

  • If we're already logged in under another nickname, broadcast an "exit" event noting that we're leaving the chat.
  • Update our nickname
  • Notify other clients that we have a new chat participant by broadcasting the "enter" event.
  • Send a "users' event to the client with a list of current chat participants
    def on_login(self, nick):
        if self.nick:
            self._broadcast('exit', self.nick)
        self.nick = nick
        self._broadcast('enter', nick)
        self.emit('users',
                  [ ns.nick
                    for ns in self._registry.values()
                    if ns.nick is not None ])

Handling "chat" events is fairly straightforward: if the user is logged in, broadcast the event (along with the nickname) to other clients. Otherwise, send a message back to the client asking them to log in:

    def on_chat(self, message):
        if self.nick:
            self._broadcast('chat', dict(u=self.nick, m=message))
        else:
            self.emit('chat', dict(u='SYSTEM', m='You must first login'))

Finally, our _broadcast method just emit()s an event to each client in turn:

    def _broadcast(self, event, message):
        for s in self._registry.values():
            s.emit(event, message)

Hooking up the Namespace to a WSGI application

Once we have our namespace, we need to actually connect it to a WSGI application. Our main WSGI app is fairly simple: if the path requested starts with '/socket.io', we'll route it to the gevent-socketio handler socketio_manage. Otherwise, we'll serve a static file. Note in particular that we're passing the ChatNamespace defined earlier to socketio_manage so it knows how to route chat messages:

def chat(environ, start_response):
    if environ['PATH_INFO'].startswith('/socket.io'):
        return socketio_manage(environ, { '/chat': ChatNamespace })
    else:
        return serve_file(environ, start_response)

Our serve_file function is fairly straightforward. (In a production system, I'd recommend serving static files from a dedicated server instead.)

def serve_file(environ, start_response):
    path = os.path.normpath(
        os.path.join(public, environ['PATH_INFO'].lstrip('/')))
    assert path.startswith(public), path
    if os.path.exists(path):
        start_response('200 OK', [('Content-Type', 'text/html')])
        with open(path) as fp:
            while True:
                chunk = fp.read(4096)
                if not chunk: break
                yield chunk
    else:
        start_response('404 NOT FOUND', [])
        yield 'File not found'

Finally, since this is a small stand-alone app, we'll go ahead and put a small WSGI server at the end of our script:

sio_server = SocketIOServer(
    ('', 8080), chat, 
    policy_server=False)
sio_server.serve_forever()

The Javascript

Our Javascript is mostly glue code, consisting of

  • Code to cause form submissions to send socket.io events instead
  • Code to respond to socket.io events

We start off by creating our socket.io connection and grabbing some elements from the DOM:

(function() {

    // Create and connect sockert
    var socket = io.connect('/chat');

    // Grab some references
    var $status = $('#status');
    var $login_form = $('#login');
    var $chat_form = $('#chat');
    var $nicks = $('#nicks');
    var $chat = $('#chat-data');

Next, we'll make our two forms (login and chat) emit socket.io events:

    // Bind the login form
    $login_form.bind('submit', function() {
        var $input = $(this).find('input');
        socket.emit('login', $input.val());
        $input.val('');
        return false;
    });

    // Bind the chat form
    $chat_form.bind('submit', function() {
        var $input = $(this).find('input');
        scroll_chat();
        socket.emit('chat', $input.val());
        $input.val('');
        return false;
    });

Next, we'll keep a list of the users so we can incrementally update it when we receive enter and exit events:

    // List of currently logged-in users
    var users = [];

Now we'll respond to some socket.io events indicating our connection status so the user gets some notification of disconnects and errors:

    // Bind events to the socket
    socket.on('connect', function() {
        $status.html('<b>Connected: ' + socket.socket.transport.name + '</b>');
    });
    socket.on('error', function() {
        $status.html('<b>Error</b>');
    });
    socket.on('disconnect', function() {
        $status.html('<b>Closed</b>');
    });

Next, we handle our enter, exit, and users events by updating our users array and rerendering the list of users in the room:

    socket.on('enter', function(msg) {
        users.push(msg);
        render_nicks();
        append_chat($('<em>' + msg + '</em> has entered the room<br/>'));
    });
    socket.on('exit', function(msg) {
        users = $.grep(users, function(value, index) {
            return value != msg });
        render_nicks();
        append_chat($('<em>' + msg + '</em> has left the room<br/>'));
    });
    socket.on('users', function(msg) {
        users = msg;
        render_nicks();
    });

Finally, handling our chat event is trivial:

    socket.on('chat', function(msg) {
        append_chat($('<em>' + msg.u + '</em> &mdash; ' + msg.m + '<br>'));
    });

To keep all the preceding code succinct, we need a few helper functions. The first one renders our $nicks chat participant list:

    // Some helper functions
    function render_nicks() {
        var result = $.map(users, function(value, index) {
            return '<li>' + value + '</li>';
        });
        $nicks.html(result.join('\n'));
    }

Next, we have a function that ensures the last line of the chat is visible:

    function scroll_chat() {
        var new_scrolltop = $chat.prop('scrollHeight') - $chat.height();
        $chat.prop({ scrollTop: new_scrolltop});
    }

Finally, our function to append chat messages makes sure that, if we're already scrolled to the bottom, we are still at the bottom when new messages come in:

    function append_chat(msg) {
        var old_scrolltop = $chat.prop('scrollTop');
        var new_scrolltop = $chat.prop('scrollHeight') - $chat.height();
        var scroll = old_scrolltop == new_scrolltop;
        $chat.append(msg);
        if(scroll) {
            scroll_chat();
        }
    }
})();

Conclusion

So that was a lot of code, but hopefully it gave you a feel for how you can bend socket.io and gevent-socketio to you will in your own applications. All the code is also available on my PyGotham SourceForge repository (the chat server was initially developed for a talk given at PyGotham). So what do you think? Is socket.io something you'd use? Do you like the new Namespace abstraction or hate it? Let me know in the comments below!

Posted by Rick Copeland at 12:37 PM
Labels: , ,

33 comments:

  1. Markus5:24 PM

    Just out of curiosity, is it possible to port the sine example to this? I implemented it with juggernaut as it seemed more straight forward...

    ReplyDelete
    Replies
    1. Rick Copeland7:53 AM

      Thanks for the comment! I can look at updating that example as well.

      Delete
    2. Philip Neustrom1:35 AM

      You should check out the 'live CPU graph' in gevent-socketio's examples/ -- it's very similar

      Delete
    3. Rick Copeland9:31 AM

      Thanks Philip, I will check that out.

      Delete
  2. Anonymous7:04 AM

    Splendid tutorial! This is exactly what I was looking for!

    ReplyDelete
    Replies
    1. Rick Copeland7:53 AM

      Thanks for the comment! Glad to hear the tutorial was helpful!

      Delete
  3. Maksat1:51 AM

    Guys, I've installed gevent-socketio, run example simple_chat successfully, but when i run your chat, I get this type of error:

    socket = environ['socketio']
    KeyError: 'socketio'

    Could you please, find me out, what is going wrong...

    ReplyDelete
    Replies
    1. Rick Copeland9:19 AM

      That code only works on obsolete versions of gevent-socketio; the code in this post doesn't use environ['socketio']

      Delete
  4. Anonymous5:05 PM

    Great post!

    In your last post, a year ago, you included ZeroMQ in the mix. With the changes to Socket.IO and Gevent, how would you integrate ZeroMQ between two Socket.io servers to have multi-server chat?

    ReplyDelete
    Replies
    1. Rick Copeland8:28 PM

      Thanks for the comment! If you wanted multi-server chat, you could certainly use something like ZeroMQ to broadcast to all the other servers. I left it out this time because I felt like it complicated the example without providing much value (in this particular example).

      Thanks again!

      Delete
  5. Unknown11:08 PM

    Hey thanks for great tutorial, it was really very educative. But I'm having the little problem with getting it to work. Im getting the IOError: [Errno 21] Is a directory error. Here's the gist https://gist.github.com/3421900, if you could help me I would be very thankfull.

    ReplyDelete
    Replies
    1. Rick Copeland10:52 AM

      Glad you liked it! I've commented directly on your gist with what I believe to be the problem. Let me know if you have any further questions!

      Delete
  6. Anonymous1:20 PM

    Hi, Do you have any website or demo or similar that I could see? I'm just learning Python and I'm not sure if your process is what I'm looking for.

    I have a python script for a chat, that connects to a local server via sockets. I run the script through the terminal. What I want is to replace the terminal with a browser page for eventual remote access for multiple users.
    Thanks

    ReplyDelete
    Replies
    1. Rick Copeland12:19 PM

      Thanks for the comment! All the code is available at http://sf.net/u/rick446/pygotham for you to play with/enhance for your own purposes, and it does implement a web-based chat server.

      Delete
    2. Anonymous10:04 PM

      Thanks for the code. I had downloaded it, but hadn't tried it. Was exactly what I needed. Props!

      Delete
    3. Rick Copeland2:06 PM

      Great! Glad to hear it helped you out!

      Delete
    4. Anonymous4:07 PM

      I'm having trouble with having test.html being the default webpage for browsers. When I don't include the /test.html, the error trace shows that public is a folder. So I guess I need to set test.html as default while keeping public in the path (where there's a folder with the webpage media). I appreciate any help with that.

      Delete
    5. Rick Copeland4:20 PM

      If you want the static test.html to be served as the default, you'll need to set up a handler for that. You might start off by naming it 'index.html' and see if that works for you.

      Delete
  7. Anonymous4:24 PM

    Sorry for being too brief and simplistic. It is named index.html.

    ReplyDelete
  8. Anonymous4:25 PM

    By the way, when trying to access without /index.html, the browser shows "Internal Server Error"

    ReplyDelete
    Replies
    1. Rick Copeland4:29 PM

      OK, in that case you'll need to put a special case in the serve_file to serve up index.html when you request the directory (as a minimal example). Also keep in mind that this code is intended as a tech demo for real-time server-side notifications, not so much as a production-ready server.

      Delete
  9. Unknown1:19 PM

    awesome tut. Thanks for taking the time to do that.
    Would this be called ajax based tech? or what, i'm still confides as to what ajax is. I wan to make app that takes use input computes and prints out the result below the input string, king of like ipython notebook, is that still under the ajax umbrella?

    ReplyDelete
    Replies
    1. Rick Copeland1:28 PM

      Hi juin - thanks for the comment!

      AJAX usually refers to Javascript making asynchronous requests to the server in the background, so it's a technique that *can* be used to implement the server "push" tech described here, but WebSockets is another (and Socket.io can use both). In general, AJAX refers to communication initiated by the browser, not by the server.

      For your app, if everything's happening in the browser, it's definitely not considered ajax; ajax only relates to communication between browser and server.

      Thanks again for the comment!

      Delete
  10. Anonymous3:18 AM

    Hi Rick
    This works great. I was wondering if you could show how to run this example inside a django app.

    ReplyDelete
    Replies
    1. Rick Copeland11:19 AM

      Thanks for the comment. I'm not sure of the details of how to run this inside django, since I'm not a django programmer, but since django can run as a WSGI app, presumably you could take the "chat" function defined and use it to wrap the django app at the highest level.

      Delete
  11. Meeks10:49 AM

    Nice tutorial, Rick. You did a great job tying all the pieces together in a very small package. (And yes, I realize that I am about a year late, but it is still a helpful post.)

    ReplyDelete
    Replies
    1. Rick Copeland10:52 AM

      Thanks for the comment! Glad you liked it!

      Delete
  12. aya8:46 AM

    Hi Rick,

    I cloned sources,
    installed gevent-socketio ($ pip install gevent-socketio==0.3.5-rc2),
    ran: $python socketio_test.py
    Open test.html in browser, and get status: discontected.
    What might I miss?

    Thanks!

    ReplyDelete
    Replies
    1. Rick Copeland3:39 PM

      Hm, the only thing that comes to mind is ensuring that you're opening test.html as http://localhost:8080/test.html and not file:///test.html or something like that? Otherwise, I'm not sure.

      Delete
  13. Default Winner6:30 AM

    Hello Rick,
    I need to create a chat app for android and iPhone as well. Just want to know that whether socket.io will be sufficient for the purpose and if not , what architecture you would suggest .

    ReplyDelete
    Replies
    1. Rick Copeland7:59 AM

      Hi Default,

      I don't really know much about doing mobile app development, so I'm afraid I can't help you much. My only idea would be to look into doing an HTML5 app and using something like Phonegap to make it native. In that case, I'd expect that socket.io would work fine for you.

      Sorry I'm not much more help there. Thanks for the comment!

      Delete
  14. Unknown8:10 AM

    could it be that BaseNamespace has changed to Namespace?

    ReplyDelete
  15. Unknown8:14 AM

    hi!
    i can import only Namespace there is no BaseNamespace
    like above , are they the same?

    ReplyDelete

Subscribe to: Post Comments (Atom)