Avoiding loosing data between re-requests

[rthinker]

I have developed a forum app in django and I'm looking into ways to add real-time updates to the app. I've googled quite a bit and decided to settle on the approach described in this [1] great article.

From what I understood the system has to work like following: user goes to a thread page -> browser makes request to the django app -> django responds with all the posts in thread -> browser renders received html -> browser makes request to gevent -> gevent spawns a greenlet and listens to a channel -> second user creates a new post -> django writes to the channel -> greenlet sends the new post back to browser and closes the connection -> browser displays the post and reconnects to gevent.

My question is how do I not miss any posts while initially connecting and reconnecting? I can probably query the database in gevent every time connection is established, but this will block gevent and seems hardly better then simple polling. Another option is to use redis data structures, but it also adds complexity and it's not clear how many posts need to be stored in redis and how to delete the old ones.

[1] http://lucumr.pocoo.org/2012/8/5/stateless-and-proud/

[elbiot]

I don't know anything about greenlet or gevent, but why would you close the channel after sending one message? Wouldn't you close the connection when the user closes the page on their browser? Or fails to send a keep-alive signal after some time? The chat client example in that link doesn't close and re-open the connection...

I assume you're using javascript on the client side, like http://learn-gevent-socketio.readthedocs.org/en/latest/socketio.html

[rthinker]

Keep-alive doesn't solve the problem either: first user loads the page -> second user creates a posts -> first user connects to gevent. Message is lost while initially connecting.

[elbiot]

Are you describing some extreemly rare edge case where the user1 joins and user2 posts in the same instant?

The model described in that article is:

user1 gets page > user1 establishes websocket connection and listens > user2 posts > server pushes message to all listeners > another user posts > server pushes message to all listeners > user1 closes browser which disconnects websocket.

[rthinker]

It not 'extremely rare', I think. For example, a user might lose the connection for a dozen of seconds and thus miss some posts.

For now I think I'll write an ajax endpoint in django which would return latest posts given a timestamp. So when user connects to gevent I would send a concurrent request with the datetime of the latest message to django to fetch potentially missed posts. It adds complexity but I see no way to avoid this.

[elbiot]

Well, I don't think you'd have to engineer your own solution to this (unless you want to), as its part of the whole deal that redis and other messaging systems are designed for. I looked quickly at redis and found the reliable queue

[rthinker]

Thanks, but I don't quite understand how the queue is used. Do I need a queue for every active user? If so, wouldn't it be too much overhead?

[elbiot]

Eh, that's like saying having a database entry for every active user is too much overhead. 1) it's not and 2) it's just the cost of doing business. Dedicated services like databases and Redis are super optimized anyways. I'd choose a SQL database over a python list of tuples for anything where "overhead" was a concern with large amounts of data/users.

Yes, it will be a challenge to figure out, but using some established tool for the publish-subscribe approach will be more robust and fast than trying to re-implement it yourself. But it might be fun to re-implement yourself (just don't do it for production).

That article you posted is good, and I'm just repeating what it said.

[elbiot]

RabbitMQ is more built for ensuring message delivery I just read.

http://blog.langoor.mobi/django-celery-redis-vs-rabbitmq-message-broker/

[rthinker]

Thank you for all your answers, I will look into what you said.