Cooking up a big data storm

At PeerIndex, we use a distributed fault-tolerant real-time computational framework called Storm for our real-time big data analytics. Originally created by Nathan Marz in his start-up Backtype, it was quickly snapped up by Twitter to power its analytics platform – and rightly so. Apart from Twitter, it’s used by a number of companies like Flipboard and RocketFuel, so we’re in good company.

After the Storm London big data meetup

After the Storm (London Meetup)

Storm is a scalable framework for processing streams, in the same way that Hadoop is a scalable framework for batch processing big data.  It is a great distributed framework for processing streams which allows people to scale up their applications quickly. However, when our engineering team tried to find other Storm users in London to mutually share knowledge with, there was a distinct lack of meetups in the area –  despite the large number of London-based businesses working with data analytics!  The next step was obvious: they set up the Storm London Usergroup, and held their inaugural meetup this Monday. PeerIndex supports this activity because we believe that strong ties with the wider developer community are a cornerstone of our business, and we wanted to contribute to the London ecosystem.

Organised by members of our engineering team (say hello to Mischa, Enno, Ferenc) and kindly supported with a great venue by the Open Data Institute (thanks Tom!), 40 Storm chasers congregated on Monday evening for an introduction to the system by Michael Vogiatzis from Social Artisan and Stephen Elliot from QuBit.

Michael started off by giving us an overview of what Storm can do and how people are using it. He also explained its architecture and the inner workings of Storm with a live demo. Check out his presentation and code example.

Stephen had in store interesting insights from his experience of using Storm in the Qubit web analytics platform, e.g. what data is processed with it, how it fits into the rest of their technology stack, their motivation for using the Trident abstraction, how they employ monitoring and alerting and much more. His slides showed some great stats on the impact Storm’s real-time processing has had on Qubit’s product.

Polishing off an evening of big data learning with pizza and beer seemed like the appropriate ending to a great night – and Mischa, Enno and Ferenc hope to see you at the next Storm London meetup! In the meantime, you can view all materials in the meetup group.

[button link=""]Join the meetup[/button]