IT'S MIDMORNING AT Robert Toll's office, more than two years into the housing slump, and still the CEO of Toll Brothers doesn't seem fazed. He's on speakerphone with one of his executives, firing off talking points. The company Toll founded with his brother 41 years ago has posted a third-straight quarter of losses, and the luxury home builder's shares are down 60 percent since the boom. But Toll is defiantly upbeat. "Our business is good," the punchy 67-year-old blusters from behind his desk. His office, at Toll's nondescript headquarters outside Philadelphia, buzzes with urgency. Another line rings. A reporter waits. Another meeting is minutes away. To kill time, his marketing manager points out a photo of Toll trading jabs with Muhammad Ali.

Toll has taken a different sort of hit recently. Today's ongoing housing bleakness isn't what he predicted; back in 2006, he said we were "bouncing along the bottom" and that the next season would pick up. He even ventured into new territory: Known for building pricey suburban McMansions, Toll muscled into New York's cutthroat condo market, with upscale high-rises in Manhattan.

Toll's misplaced confidence has angered many big investors. This year major pension funds and a big labor union opposed Toll's board reelection. "Robert Toll's dramatic proclamations turned out to be clearly wrong," says Richard Metcalf, of the Laborers' International Union of North America, which has several funds that own Toll shares. "There's been bad performance and an underestimation and miscalculation of problems facing the industry."

For more SmartMoney Magazine features, turn to the September issue.

But Toll, a physically unassuming man with thin hair and bushy eyebrows, bridles at such critiques. He says that over the long term, the growing number of wealthy middle-agers will want his homes, and with $1 billion in cash, Toll Brothers will be ready. Many on Wall Street agree. Jay McCanless, an analyst at FTN Midwest Securities, calls the firm "one of the best-managed builders out there." SmartMoney's Paulette Miniter caught up with Toll to talk about the industry's troubles, his regrets and how Barack Obama could fix everything.

It doesn't seem as if you were prepared for the real estate bust.

We recognized there was a speculative overexuberance taking place, but we didn't think it was to such an extent that it dwarfed demographics. Wealth was increasing. You had a roaring stock market. Unemployment was at an all-time low. Credit was free, and all the money in the world was sloshing all over the place.

Weren't there also signs of froth? Double-digit price increases, no-money-down loans...

That's a lot of topics. Froth? Yes, but we thought the underlying fundamentals were strong enough to support the market, even if the froth blew away. The free mortgages were an accident a result of greed and the capitalist system. It's the best system we've figured out, but without checks and balances, it can run amok.

Was there a point when you thought about slowing down?

You don't stop building. But you do raise your threshold. And we did, toward the end of the cycle.

Yet you have some six years' worth of land in a down market and are taking losses.

Because we didn't foresee the serious decrease in the business any more than anybody else did. Management of any firm has to look at both sides. You want to have an exit strategy, but you also don't want to miss opportunity, because shareholders aren't paying you to just sit. They want expansion, extra profit. We could've balanced it a lot better if we'd had that crystal ball. But we didn't.

You did have option agreements, which let you opt out of land purchases. Wasn't that supposed to protect you?

It did protect us to some extent. And now we don't have to go out and acquire more land to keep things rolling. So now all we'll be buying are very special good deals.

You were leery of condos a few years ago, but now you have several high-rises. Is this a growing part of the business?

We're very happy we got into the urban tower business the New York tower business being where we are, with one or two exceptions. If we hadn't gone into it, we wouldn't be as well off as we are today. If the opportunity is there, you ramp up. If it's not, you back down.

Have you had to lower your condo prices?

We have introduced some incentives to keep pace up, but on a minor scale. I do wish we'd gotten in earlier.

Did you think Toll was a bit immune to problems in the mortgage market, being a luxury builder for wealthier folks?

Yes, to some extent we thought we were more insulated than the starter market, which is on a tighter budget. We knew that if our buyers couldn't sell to their buyers, then we would have trouble. But we didn't think it'd be as bad as it turned out.

Has the downturn lasted longer than you thought it would?

I did think certain markets, such as Washington, D.C.-Northern Virginia, would turn that haven't.

So how do we get out of it?

We're looking forward to the presidential election campaign potentially picking up consumer confidence. We've made such horrible blunders, lost so much treasure and soldiers' lives. Campaigns have a way of redressing that, especially with an orator as good as Barack Obama. Instead of focusing on our misery, he can help us focus on the potential of America again. If you have a change in confidence, this market will take off.

You're also lobbying for tax rebates for home buyers.

Yeah, I feel bad speaking about it, because I've received e-mails saying, "Isn't it a little disingenuous for Toll to be begging Congress to help it out when it made billions in the past few years?" I have to say, I'm not pushing for Toll to be saved we'll make it through anyway. I'm pushing for what, in my opinion, is a more sensible way to stimulate the economy than some of the bells and whistles being proposed.

How so?

Gerald Ford and a Democratic Congress did it successfully, with a $2,000 tax credit for people who bought new homes. Today that equates to $15,000. It's my belief that if Congress did that, buyers would say, "Congress is calling a bottom," and you'd see the market turn around. If we don't stop the slide in housing prices, we'll be in the same spot a year from now.

You sold a lot of shares when Toll's stock was peaking. Good timing?

It turns out to have been tremendous timing. It looks as though I knew what the market didn't, but I reacted simply to the fact that we had come to new highs and it was a good time to diversify. It still leaves me as the largest shareholder, and the vast majority of my wealth is still in Toll Brothers. So I don't think I did anything wrong by the public.