The U.S. Securities and Exchange Commission has denied an application by the Nasdaq stock market to provide algorithms called ‘benchmark orders’ to its members, citing a lack of adequate pre-trade risk controls.

Under the Nasdaq proposal, the benchmark orders would seek to achieve the performance of a specified benchmark over a specific period of time, where all terms are defined by the entering party.

A third-party system dedicated to processing benchmark orders would process each benchmark order and generate one or more ‘child orders’ for execution.

Under the Nasdaq application, risk controls required under Rule 15c3-5—known as the Market Access Rule—would not be applicable to child orders. Nasdaq had said that it would impose “substantial risk controls” to govern its proposed benchmark orders, in particular with respect to the child orders to which the Market Access Rule would not apply.

But the SEC, in denying the application, said that Nasdaq had failed to adequately address the issue or detail its proposed commitments with respect to benchmark orders.

The SEC’s Market Access Rule has been widely cited over the Knight Capital trading glitch of August 1, 2012, when erroneous orders from the U.S. market maker inundated the New York Stock Exchange for 40 minutes, nearly bankrupting the firm.

The rule requires broker-dealers to implement risk controls that would block erroneous orders from reaching exchanges and other market centers.

This has put the spotlight on the testing of trading software, whether for risk management or investment purposes.

One of the pitfalls with testing is the ‘data mining fallacy’, or the risk of extrapolating based on a limited set of inputs.

“When you run a simulation and get a desired result, the question is whether it’s based on a true pattern in the data or you’re just lucky,” said Tucker Balch, professor of computer science at Georgia Institute of Technology.

“It’s like trying to flip a coin and get heads ten times in a row,” said Balch. “It’s highly improbable for an individual, but if you have 10,000 people flipping coins there’s a reasonable chance that someone will do it. Similarly, if you’re testing 10,000 investment strategies, there’s a good chance that one of them will work.”

The Securities Industry and Financial Markets Association (Sifma), an industry trade group, vigorously opposed the Nasdaq proposal, saying that Nasdaq had provided no details on how child orders generated by the benchmark orders would comport with Nasdaq’s existing rules, including those intended to enforce the Market Access Rule.

“This lack of detail raises concerns about the potential for market disruptions that Nasdaq’s proposed algorithmic functionality could cause,” said Theodore Lazo, managing director and associate general counsel at Sifma, in a comment letter.

Sifma also questioned whether it was appropriate for Nasdaq, as a national securities exchange, to offer products such as benchmark orders which would compete with the algorithms that broker-dealers offer.

The commercial offerings of a national securities exchange are distinct from the functions that an exchange carries out in its role as a self-regulatory organization, Sifma said.