The ability to analyze massive amounts of market data and other data that’s generated both internally and externally is guiding the development of algorithmic trading.

“Our focus in 2013 is on real-time ‘big data’,” said James McInnes, chief executive of financial technology provider Cyborg Trading. “We are taking our technology and applying it to bigger financial data feeds.”

The explosion of data in capital markets, as well as the development of high-performance databases that can take in and process this data, leads “to the idea of massive scalability, whereby you can create trading opportunities by analyzing all the data”.

Cyborg Trading is developing “scanner algorithms” that can be used to set customizable conditions and sense when a trading opportunity exists, said McInnes.

Cyborg Trading’s Algorithm Development Kit (ADK), part of its Cloud Trader platform, provides a customizable simulation environment in which the functionality, correctness and performance of algorithms can be tested before they are launched into production.

“You can create algorithms that get sent to our cloud, which runs server-side, and then takes action,” McInnes said.

Different degrees of human interaction can get built into the algorithm’s design. If, for example, the trader wishes the algorithm to be fully automated, it can be designed to spot certain conditions and then initiate a trade when the conditions are met. If, on the other hand, the trader prefers to have some human intervention, that can be designed as well.

“There’s a spectrum of automation, from black box to grey box to white box, which can be engineered into the algos that can be created using our platform,” McInnes said.

Research firm Celent estimates that big data spending in capital markets will be $1.2 billion in 2013 and will grow to $2.4 billion by 2015, as more firms move from testing to implementation of big data strategies.

“The capital markets industry has vast amounts of data that needs to be processed in real time, which is the perfect context for investing in big data,” said Celent analyst Bill Fearnley in a report.

Increasingly, firms are combining information from multiple internal business units as well as external data sources. “Big Data tools and technology can help firms capture, analyze and visualize this data and glean new investment ideas,” said Fearnley.

In addition to being “maximally scalable” to operate with large sets of data, algorithms need to be capable of being designed to change when market conditions change, something that most algorithms lack.

“A lot of the algorithmic strategies being developed today are hard-coded,” said McInnes at Cyborg Trading. “Our differentiator is in allowing you to create a customizable strategy dynamically, and then deploy it through the cloud.”

Celent recommends that capital market firms begin experimenting with big data and new open source technologies such as Hadoop and NoSQL, by establishing “big data sandboxes” that can be carved out of the existing IT infrastructure.

As a “local sandbox test environment”, Cyborg Trading’s ADK provides a dedicated local development machine in order to test functionality throughout the development process. The simulations use authentic market dynamics, including realistic fills and order queue.

“The benefit of this approach is that you can write an algorithm once, run it against a live simulation and then deploy it globally,” McInnes said. “This is especially important as firms move towards trading multiple asset classes and geographies.”