Traditional Big Data technologies too storage-centric, experts say.

New technologies are likely to yield benefit in harnessing Big Data in capital markets.

Such systems leverage massively parallel processing architectures, in-memory processing and appliance technologies for predictive analytics on structured information, Hadoop appliances for unstructured information and purpose-built appliances for simulations.

“Traditional Big Data solutions are storage-centric processing models with relatively long computational or response cycles,” Richard Tibbetts, chief technology officer at software provider StreamBase Systems, told Markets Media.

“These big data storage and analysis technologies have a place in capital markets, but we are seeing slow adoption because trading firms need so much more,” Tibbetts said.

Many trading firms are already sophisticated in their use of market data, which will be difficult for Hadoop et al to match. At the same time, tick databases don’t extend naturally to unstructured data or massively parallel map reduce jobs.

“Until some storage and historical analytics technology bridges the gap the leading firms will use a combination of tick stores and Hadoop for historical data storage and analysis,” said Tibbetts.

In order to make the shift to real time analytics, organizations will need a combination of real-time data management technology and the right analytics. Getting the right analytics requires the ability to derive the best parameters for those real time decisions from a combination of historical data, statistical analysis, and domain expertise.

“Look for firms to adopt a variety of approaches to getting analytics off the desktop and into real-time data flows,” said Tibbetts. “The best solutions will empower quantitative analysts and domain experts to participate in operationalizing their ideas, reducing time to market and assuring the best analytics are driving real time decisions.”

The goal of the analysis of large amounts of data in capital markets is to drive real-time decisions about working orders in the market, whereas in other industries it’s more likely to be monthly or weekly decisions about website provisioning, design and management, certainly not real-time decisions that are measured in milliseconds.
Front desk trading activities have long been measured by milliseconds, but middle and back offices still struggle to maintain a real-time view of positions, markets and operations.

StreamBase, a provider of complex-event processing (CEP) technology, has launched StreamBase LiveView, which enables firms to address the challenges of managing high-speed order flow and reacting in real-time.