Activity Stream using Redis

29 April 2013 by Abhijit Rao 1 comment

One of the common UI patterns in modern web applications and services is a stream of recent events – commonly called the ‘activity stream’. An activity stream keeps the user updated with the latest happenings and hence keeps the user engaged. Activity streams can show updates about a variety of activities. Typically information regarding new comments, new friends, new posts etc. is shown.

Activity streams can be realtime push based or regular pull based. The push ones require some more infrastructure like, for example, Meteor to push latest updates to the client (the browser or a native app). We will discuss a backend implementation of a typical activity stream which could support both push or pull updates.An activity stream is generally transient information which we do not want to store permanently. This can be modeled using a FIFO (First In , First Out) queue of fixed size. Newer items are pushed into one end of the queue and the same number of old items are pushed out from the other end. So at any time we maintain ‘n’ newest items – ‘n’ being the size of the queue. This ‘n’ can be tuned according to the loads and the available infrastructure. A bigger ‘n’ will need more resources for storage. In case we want to store all the data, we would need another storage backend like Cassandra.

We can implement this queue using a table in a RDBMS and do the housekeeping to implement the functionality of a queue. But a better way is to use a native queue. Redis is a NoSQL database/datastore which provides lists which can be used as a base for a queue implementation. Redis stores information in memory for performance and scalability. It writes periodic snapshots to disk for persistence and recovery. A fixed size queue suits Redis very well as we can predict memory requirements and resulting performance.

ACTIVITY-STREAM1

In our hypothetical example of a web application, each user will have an activity stream of his own. This is because users will get updates to comments on posts/comments they have made, which will be different for different users.

We will need a unique identifier representing a user’s stream. Let us use the unique user-id for this – actistrm: will give us a unique key for each user’s stream ( is a variable here). The key does not need to be stored and can be constructed at the time of stream retrieval.

LPUSH actistrm:
This will push the activity-id into the queue.
Note: Here the activity-id should uniquely identify the activity. In case we have activities of multiple types then we have to have a composite value which we push into the queue. For example “|“. This we will deconstruct when we retrieve the list. We can store any amount of information in a composite value with only storage constraints. The deconstructing is cheap.

LTRIM actistrm: 0 1000
This trims the list (queue) to a length of 1000 pushing out older items.

LREM actistrm: 0
This removes the activity-id from the users stream.

LRANGE actistrm: 0 99
This will fetch the latest 100 elements from the queue

Once the list of the latest activities is available then the relevant items can be simply fetched form relevant stores and rendered. As Redis is very efficient a server with moderate hardware can easily serves 1000’s of requests per second. Redis has clients in most of the common languages. The client will take care of establishing the connection to the redis-server and other details while providing a said-language friendly API for consumption.

A detailed note on using Redis with Ruby on Rails can be found here.

Abhijit Rao

Follow me on Twitter

1 thought on “Activity Stream using Redis”

  1. cool article. one question i have is, what happens if one of those activities, for example, a comment, or a follow, is removed from the system. how would you know how to map the “undoing” of that activity from that list?

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe To Our Blog

Get access to proven marketing ideas, latest trends and best practices.

Next up home

Contact

Lets build cool stuff

Share your contact information & we will get in touch!

I want (Tell us more about your dream project)