IBM touts 'stream computing' for real-time data analysis

IBM is bringing "stream computing" to the IT industry with new software that the company says analyzes thousands of simultaneous data streams to provide insight that helps businesses solve their most challenging problems.
##CONTINUE##
Stemming from seven years of development and technology from more than 100 IBM patent filings, stream computing is moving out of the prototype stage and into the commercial market with an offering called System S, IBM said Wednesday. IBM stream computing takes a fundamentally different approach to business analytics by analyzing data in continuously updated streams of information from multiple sources, rather than static files pre-loaded into a data warehouse.

"System S can analyze hundreds or thousands of simultaneous data streams – stock prices, retail sales, weather reports, etc. – and deliver nearly instantaneous analysis to business leaders who need to make split-second decisions," IBM says. "The software can help all organizations that need to react to changing conditions in real time, such as government and law enforcement agencies, financial institutions, retailers, transportation companies, healthcare organizations, and more."

Several institutions including the Swedish Institute of Space Physics, the Marine Institute of Ireland and TD Securities are already using stream computing.

"TD Securities is using System S to ingest more than 5 million bits of trading data per microsecond to make faster financial trading decisions," IBM says. "To match the capacity of the system, a trader would have to be able to read the entire works of Shakespeare 10 times in less than one second and then identify and execute a stock trade faster than a hummingbird flaps its wings."

The stream computing project, first unveiled two years ago, was spearheaded by System S chief scientist Nagui Halim, who says the software is designed for clusters of commodity Linux servers. List price is about $400,000, but prices can range anywhere from $100,000 to $1 million, depending on the size of the system, he says.

Instead of files and directories, System S processes data in the namesake streams, which Halim compared to the windows in Microsoft Windows because they provide the means with which users interact with the system. "Here the stream becomes a live entity you're connected to, and that's what you process," he says. You can say "give me a stream, or ten streams … name them, turn them on and off, monitor them. The streams become the underlying key element."

The stream project began in 2003, and included contributions from more than 80 people, Halim says, noting that it posed challenges in many areas, including mathematics, communications, scheduling, failover and networking.

"One thing that was interesting in retrospect is it took several iterations to get the language right," he says. "We tried a number of different approaches, but I'm extremely confident that after the third or fourth try we hit on something very powerful."

There are other systems that process data in real time as it arrives from external sources, but IBM built System S with a high degree of sophistication in terms of performance, scalability and analysis, according to Halim.

IBM has also opened a Stream Computing Center in Dublin, Ireland, to focus on research, customer support and advanced testing. A trial version of the System S code is available, with developer tools and adapters to help customers understand the software's capabilities and how it might help their businesses.

IBM has not announced plans to make System S available over the Web in a hosted model, but Halim said to stay tuned.

-----------------------------
BY Jon Brodkin, Framingham
Source：Resellernews

© Fairfax Media Business Group
Fairfax New Zealand Limited, 2009.