ByKEN BENSINGER
|
It used to be pretty tough to get a high rating from this famous burgundy book. But we crunched the numbers and discovered the latest dining trend at Zagat: grade inflation. Should you trust the reviews? |
THE BACK TABLE AT New York's exclusive Jean Georges restaurant looks out over an airy, sunlight-splashed room buzzing with lunchtime activity. It is the best seat in the house, a big bay window onto some of the most exquisite food and discriminating diners in the restaurant world. And it's no accident that it's often occupied by Tim and Nina Zagat.
It's also no accident that the Zagats are the center of attention. The restaurant manager has already come tableside to greet them. Out pops Jean-Georges Vongerichten himself, in chef's attire, to join a line of admirers saluting the couple. Eugene "Tim" Zagat, a gruff-voiced bear of a man in a dark suit, along with his wife, Nina, are the brains behind the famous burgundy Zagat guidebooks, and even a four-star restaurant like this affords them a special level of respect. But fine dining isn't the only thing on the agenda today. Digging into his second plate of sea urchin, Tim Zagat excitedly discusses his plans to go well beyond restaurants, building a sprawling "surveying content" empire, with reviews of hotels, museums, movies and, well, anything that can be rated on his famous zero-to-30 scale. "Every way you can slice and dice data," he says, "we want to do that."
On an impressive scale, they already have, with a library that has grown to some 50 titles spanning three continents. Thanks to Zagat, we know that the Grant's Farm petting zoo in St. Louis is a higher-rated attraction than the Washington Monument; that the Cheesecake Factory in Las Vegas serves up tasty 21-rated food, while its counterpart in San Jose, Calif., manages a barely adequate 16; and that the staff at the Cobbler's Cove hotel in Barbados is "a bit aloof." It doesn't stop there. Zagat has found its way onto the Web and your PDA, and is also conquering the business world, where companies order custom guides to give away to their customers. In 2004, Bank of America ordered five million copies of the Zagat Movie Guide equivalent to the entire paperback run of "The DaVinci Code."
The transformation from a two-page pet project launched from an uptown New York apartment to an international publishing juggernaut is a testament to the power of a good idea. But so much growth can put a strain on any business, and a close look at Zagat Survey LLC suggests that finding all those enthusiastic diners and shoppers to play critic as well as keeping up the standards of its legendary rating system has become a challenge. Over the course of several months, SmartMoney tracked down former Zagat editors, crunched the guides' numbers and discovered a few surprises about how the Zagat sausage is made. Among the most important: a curious case of grade inflation.
The guide which millions of consumers have come to rely on bills itself as an industry "report card" with grades handed out by the voting public. But what's become of all those tough, if not picky, reviewers? When the Zagats started selling their 1983 New York restaurant guide, it was no mean feat for a chef to score a food rating of 20 or higher, the benchmark for "very good to excellent" in Zagat terms. Only one in four New York restaurants did so at the time. Today fully 70% reach those heights. It's as if the bottom tier dropped out: Just over a decade ago 189 out of 1,300 New York restaurants rated 15 or below; today only 23 do, despite the fact that the guide now rates more than 1,500 restaurants. Step outside restaurants and the numbers look even more buoyant including a rather impressive handicap in the golf guides, where two clubs have managed a perfect 30 for their courses.
|
The Zagats, both in their 60s, say the guides continue to tell it like it is, and without a doubt, many of their trademark biting comments still appear in the books. The numbers are up, they say, only because the quality of the businesses they rate has improved and because they often eliminate lesser establishments for space considerations. "We are in the business of providing customers with information they want and need," the couple explains. "Given a choice of listing a good restaurant and one that's not as good, for reasons of space, we choose the former."
This approach, though, isn't necessarily what readers expect to find. Only a handful of the guides are identified as rating the "top" of any category, while the majority would appear to be comprehensive for their market. And while the New York guide may need to leave some restaurants on the cutting-room floor, it's hard to see such space crunches in cities where even restaurant chains like Applebee's are reviewed. Instead, critics wonder about the couple's perceived coziness with the industry they cover and whether the company's rapid expansion is affecting its surveying system. Bring these criticisms up with the Zagats, however, and you'll get a passionate defense of their system. "The integrity of our survey process is our highest priority," the couple says.
IT'S NO SECRET THE
Zagats are wildly successful, but few publishing mavens have ever developed such iconic status. A recent promotional video reminds us how the Zagat name is embedded in the modern lexicon it's been dropped everywhere from "Seinfeld" to "The Sopranos." Their simple notion to add a measure of democracy to the otherwise staid, snooty world of restaurant reviewing put their guides in seemingly every airport, bookstore and newsstand in America. The little book has also played a fundamental role in the much-ballyhooed "food revolution" that has swept the country, serving as a kind of field guide to a nation of diners. Little wonder that everyone from city mayors to foreign dignitaries court the couple's attention. The Zagats, says star chef Mario Batali, are "celebrities in their own right."
Even so, very little is known about the company they run. Headquartered in midtown Manhattan with about 120 full-time staffers, Zagat remains in private hands and won't discuss its finances. Attempts to bring in outside management, around the time Zagat received $31 million in venture capital money in 2000, were short-lived, as were plans to launch an initial public stock offering. Former staffers describe a secretive and compartmentalized operation, which relies on a sprawling network of freelancers, who say they are paid about $20 a review. Indeed, about the only thing publicly known is the basic formula for its ratings: Reviewers log on to Zagat.com and rate hotels, tourist attractions or anything else Zagat covers on a scale of zero to 3 in several categories and can add a short comment. The votes are summed, averaged and multiplied by 10, giving a maximum score of 30, which is accompanied by a short written review culled from the comments.
This fairly straightforward system is clearly not intended as a scientific poll, especially since anyone is allowed to vote. Still, it's hard to ignore the fact that a product fundamentally based on numbers is seeing a consistent upward trend. SmartMoney totaled the ratings in numerous Zagat guides to come up with an average for each. The average food rating for the New York restaurant guide has risen 26% since 1983, while the number of restaurants rated 25 or above in San Francisco has jumped nearly 60% in the past five years. Ratings of establishments other than restaurants can be even rosier perfect 30s have been awarded to a handful of musicians, golf courses, hotels and a museum, a feat no restaurant has matched. And while many people might consider New York City a consumer's paradise, it's still surprising to see that 44 shops almost one in 20 merited a near-flawless quality score of 29 in the 2006 New York City shopping guide. In all, more than a third of all retailers in that guide landed in the "extraordinary to perfect" cohort.
Theories abound about why all this might be happening. "The ratings are increasingly suspect," says Steven Shaw, a food writer and cofounder of nonprofit food society eGullet, who has written about what he sees as flaws in the Zagat system. "And yet there's never been an official explanation of how they gather their data." Certainly, the couple haven't endeared themselves to some restaurant consultants and owners for their close relationship with the industry their books critique. One commonly cited example is New York's Restaurant Week, a promotional event run by the city's tourism board that promotes restaurants offering special discounts. Restaurants love the publicity, but there's an important requirement: They have to be included in the Zagat guide to participate. Tim Zagat not only cofounded the event, but also continues to help oversee it as a member of the board's executive committee. "What better way could there be to make sure the restaurants participating in Restaurant Week are top-notch" than to use the survey, the Zagats ask. The couple make a point of never voting in their guides and say restaurants earn high ratings only when they "make their customers happy."
Lately, the Zagats have also raised a few eyebrows by selling plaques to the businesses they rate. While they still give out window stickers for free, they charge $149 to $199 for restaurants that want customized plaques. (One twist: The better the rating, the fancier and the costlier the plaque.) Restaurants are told that refusing to participate in the "recognition program" won't affect their scores, yet some can't help but wonder. "There's a little bit of a feeling like, I'd better buy this," says Betsy Alger, owner of the Frog and the Peach Restaurant in New Brunswick, N.J. The Zagats say that "the people who handle our plaques have no connection with any aspect of our editorial process."
Some of Zagat's critics attribute the grade inflation to the company's growing pains and a less than transparent system for surveying. Casual readers might assume they are reading reviews and making dining choices based on the opinion of hundreds, if not thousands, of customers. This is, after all, one of the world's most popular guides, and Zagat's books refer to an army of "250,000 active surveyors." But read a little closer and you will discover that only 1,300 people took the time to fill out surveys in the most recent Seattle dining guide. And if you think that means the commendable 24 food rating at local favorite Ezell's Famous Chicken is the average of 1,300 votes, think again; surveyors can weigh in on as many or as few restaurants as they please, reducing the pool of opinion makers quite considerably.
Indeed, one of Zagat's most closely guarded secrets is just how many votes are required to get in the book in the first place. That threshold is "determined on a survey-by-survey basis by our research experts," according to the Zagats. Ratings for restaurants that have low vote totals are defined in the guides as statistically "less reliable" and identified by a triangle. Data sheets for New York from 2001 and 2002 reviewed by SmartMoney show that as few as 36 votes earned those much-followed ratings, while other restaurants in the same guide may have more than 4,000 votes. "Less reliable" reviews amount to about 16% of the total in the latest New York guide and 22% in the Los Angeles/Southern California guide.
Such vote shortages may aggravate what has long been a complaint about Zagat's polling. The voters essentially select themselves, rather than, say, being randomly called by a pollster, as is common practice in modern surveying. Critics also say there is a tendency for the same reviewers to vote year after year. According to Gary King, a statistics professor at Harvard, that sort of methodology goes against some of the basic rules of polling. "The selection process is everything," he says, adding that it can lead to statistical oddities or strange-looking comparisons, such as an apparent preference for Japanese food among Zagat surveyors (16 of the top 68 restaurants in this year's New York guide) or the fact that Brooklyn pizzeria Grimaldi's rates the same as the legendary Four Seasons.
IF YOU OWN THE GROCERY
, a small Brooklyn restaurant that seemingly came out of nowhere to tie for the highest rating in the 2004 New York guide, you're not going to complain about any of the finer points of surveying. And for many establishments, just getting in the guide and being able to display the ubiquitous Zagat sticker on the window helps bring people through the front door. Even so, the process remains a mystery to restaurants. Dan Constantine, manager of New York's Amaranth, is still trying to figure out why his restaurant was dropped from the guide in 2005. "I don't know why it happened," he says. "But it hurt." Zagat put Amaranth back in the 2007 guide.
Jilted restaurants have been known to take their complaints to court. Upset when its 1996 review said "the only thing authentic is the heartburn," Cajun, a New York restaurant, sued the Zagats for defamation. The couple denied any wrongdoing and the case was dropped, but according to a letter from Zagat's lawyer, reviewed by SmartMoney not before Zagat sent an advance copy of the following year's review to the restaurant. Cajun co-owner Arlene Lichterman says the phrase "gone South" was omitted at her request; the Zagats say they occasionally show restaurants their reviews as a courtesy "that's the responsible thing to do" but deny changing any language at the restaurant's request. Either way, Cajun was dropped from future editions, a fate that another restaurant, Lucky Cheng's, faced after its defamation suit was dismissed by the courts in 2004. "We reserve the right not to include people who sue us," says a Zagat spokesperson.
Litigation aside, exactly how any establishment gets into the guide goes to the heart of Zagat's pitch as the quintessential American guidebook: of the people, by the people, for the people. "Each place's ratings and review are really a free market study of its own customers," explains the New York restaurant guide. But talk to former Zagat editors and freelancers, and you'll discover a less-than-populist process in cases when the survey tally is low. "We weren't getting the responses we wanted," says Kevin Arnovitz, who was co-editor of the 2004/2005 Los Angeles nightlife guide. His solution? Go on a daylong marathon to his favorite bars 23 in all then write up summaries. Similarly, Virginia Rainey, co-editor of the 2002 Salt Lake City/Denver restaurant guide, which was published in anticipation of the Olympics, found it "hard to get responses." A local food writer, Rainey says she was asked to use her own experience to round out a guide that was lacking in popular voice. "They figured with the Olympics it was an easy market," she says. "I felt like we were starting from scratch."
The Zagats say the response rate has generally been strong in most markets and add that they inform editors that without survey results write-in reviews should be "direct, objective and fact-based." Such reviews appear in the guides with no ratings, just dashes. In the end, this may be among Zagat's least understood practices. Devoid of their trademark consumer quips, these reviews tend to be positive often overwhelmingly so yet they are sprinkled liberally throughout the pages of the guides. "The guide isn't necessarily what your average reader thinks it is," says Zak Pelaccio, co-owner of two highly rated New York restaurants. "How can it really be democratic when, at the end of the day, Zagat makes all the decisions?"
At Zagat's offices overlooking Central Park, an editorial staff of nearly three dozen works busily on dozens of Zagat guides, sorting survey data and adjusting reviews to fit the Zagat tone. Tim Zagat is in his office, jammed with trophies, awards, mementos, commendations and an oversize painting of himself. Dignitaries from around the world seek an audience with this ratings overlord, and in publishing circles, his operation is revered for its ability to produce so many guides so efficiently. According to the Zagats, the New York restaurant guide alone sells about 650,000 copies a year. "They do what everyone in the industry wants to do," says Jan Nathan of the PMA, the Independent Book Publishers Association. "And they do it very well."
But surrounded by such adoration and roaring success, not to mention a largely friendly food press, they can be sensitive to criticism. When pressed in an interview for answers to questions such as why the numbers are so high in so many areas, why New York's restaurant average has increased by nearly five points since the 1983 guide was published or what effect limited survey response has on the ratings, Tim Zagat grows agitated and eventually storms out, leaving his wife to answer the questions. Calmer in demeanor, Nina Zagat says, "Food has improved so incredibly." And it's not just restaurants, she adds, but hospitality, shopping, nightlife everything has soared in quality.
Some credit the Zagats with helping to create higher standards as well as a brand that for a quarter of a century has made the chore of finding a place to eat both convenient and simple. Indeed, there will probably always be plenty of Americans thumbing through the burgundy bible to find out if there's a sushi place nearby. Brian Buckley, a New York restaurant consultant, is one of them. He uses the guides as a reference manual to find restaurants and their phone numbers. But the ratings? He pays less attention to them. "It's still a real handy book," Buckley says. "But I think people should take it with a grain of salt."



- LinkedIn
- Fark
- del.icio.us
- Reddit
X