While the markets continue to struggle to gain traction thus far in 2012, uncertainty surrounding impending regulation is likely to add to the difficulties.

One of the most controversial and potentially game-changing of these regulations is the Volcker Rule, which prohibits banks from partaking in proprietary trading.

“Eliminating proprietary trading changes everything, especially for banks,” said Sean Owens, director of fixed income for consultants Woodbine Associates. “Then it becomes a trading volume game for firms.”

By eliminating a significant part of a bank’s revenue stream, it will become necessary to key into other activities. There will be a renewed focus on increasing customer bases in order to boost order flow. A lot of the firms who are marginal players will likely need to shut down the parts of their businesses that are underperforming. However, this opens the door for smaller firms to step in.

“On the flip side, this creates a huge opportunity for non-bank dealers as well as well-capitalized hedge funds,” added Owens. “Buy-side firms will take on more execution risk in markets. People will change the way they trade on both sides.”

Jefferies has been among the firms hinting that it will step in and take the place once held by the bulge bracket banks. But some of the largest buy-side institutions, including hedge funds, have also been mentioned as those who can take advantage of the big banks bowing out of the game.

“I would not look to hedge funds as a group to rely on for liquidity,” said Edward Provost, executive vice-president and chief business development officer at the Chicago Board Options Exchange. “We need to have organizations whose primary role is to be a liquidity provider. Hedge funds on the other hand are opportunistic; they are there when they need to be and they’re not when they don’t need to be there. Options in particular need to rely on dedicated market-makers, not on hedge funds, for liquidity.”

According to rough industry estimates, when looking at the pool of options market makers, about a third of those businesses will be directly impacted by the Volcker Rule.

Aside from proprietary trading, regulators are also taking a look at high-frequency trading.

The Securities and Exchange Commission will investigate the communications between exchanges and high-frequency trading firms. While most exchanges accept and welcome order flow from HFT firms, a few in particular thrive on it, offering maker-taker pricing, which rewards liquidity providers. Regulators have also sent letters to a number of high-speed firms, such as Getco and Tradebot, requesting information about their trading activities and communications with exchanges, according to reports.

While the impending regulation is set to have a significant impact on the markets, the gradual roll-out of the new rules will be a welcome sight by market participants.

“Rather than roll it out abruptly, regulators have chosen to phase in the regulations with a gradual approach,” said Owens at Woodbine Associates. “Over the next couple of years as Volcker is phased in, we expect a long implementation period to allow the markets to adjust.”