Flatlining. That, in a word, is what mutual fund assets have been doing over the past four years, according to the Investment Company Institute, slipping even from their $12 trillion level at the end of 2007. No wonder investors are frustrated with the fund industry -- and increasingly moving their cash from active managers to blind market-tracking vehicles like ETFs. But to hear some fund firms tell it, investors just need to have a little more faith in them. (Say what?) Yes, say firms: Portfolio managers could provide more value if only they were freed from the stifling trading limitations spelled out in their own fund prospectuses.

To push the point, fund companies have been rolling out a slew of anything-goes funds in recent months -- so many, in fact, that fund rater Morningstar has recently created some new categories to track their performances (such as the new "nontraditional bond" rubric introduced this past fall). Often advertised by the fund companies themselves as "absolute return" or "unconstrained," these open-marriage offerings allow managers to invest in pretty much anything -- from currency bets to complex derivatives such as credit-default swaps. And while it's far too early to see whether the approach is working, one thing is perfectly clear: Investors are betting big on them -- pouring some $37 billion into nontraditional and alternative-strategies funds in 2011, while yanking more than $50 billion from traditional U.S. stock funds and another $4.6 billion from U.S. government bond funds.

Still, many experts doubt that taking off the leash will let these managers run any faster or straighter. Some may simply be trying to imitate the glamour of the hedge fund industry, say analysts -- with the result often being higher costs and more risks for investors. Geoffrey Bobroff, a fund-industry consultant in East Greenwich, R.I., for one, worries that "investors don't really understand what they're buying" when they opt for such funds.

Some portfolio managers who have embraced the anything-goes model say that the markets themselves have forced their hand. Stocks have been disappointing for a decade; Treasurys offer such measly yields -- currently about 2 percent on the 10-year bond -- that investors stand to earn almost nothing and could face sharp losses if interest rates rise. "There's no income in fixed income," says William Eigen, manager of the $14 billion JPMorgan Strategic Income Opportunities, which uses a mix of individual bonds and derivatives such as credit-default and interest-rate swaps to try to post positive returns in weak markets.

Steve Podnos, a financial adviser in Merritt Island, Fla., has put some client money into the new breed of funds, but only in those run by firms with big enough research teams to find bargains in hard-to-analyze things like convertible securities and derivatives. Podnos is also careful to make sure the manager has at least some track record with alternative investment strategies. "I don't want some untested cowboy trying to goose returns," he says.