Women’s Funding Network Sex Trafficking Study Is Junk Science

Schapiro Group data wasn’t questioned by mainstream media.

By Nick Pinto
published: March 23, 2011
Courtesy of David Finkelhor

David Finkelhor, the director of the Crimes Against Children Research Center at the University of New Hampshire, says reporters need to resist the urge to cite bogus studies 

Steve Doig, the Knight Chair in Journalism at Arizona State University, says the Schapiro study is based on a logical fallacy

 During the same September hearing of a subcommittee of the House Judiciary Committee, members of Congress listened to vivid and chilling accounts regarding underage prostitution. They heard testimony from half a dozen nonprofit executives and law enforcement officials. But the most alarming words of the day came from Deborah Richardson, the chief program officer of the Women’s Funding Network (WFN), who told legislators that juvenile prostitution is exploding at an astronomical rate.

“An independent tracking study released today by the Women’s Funding Network shows that over the past six months, the number of underage girls trafficked online has risen exponentially in three diverse states,” Richardson claimed. “Michigan: a 39.2 percent increase; New York: a 20.7 percent increase; and Minnesota: a staggering 64.7 percent increase.”

In the wake of this bombshell revelation, her disturbing figures found their way into some of the biggest newspapers in the country. USA Today, the Houston Chronicle, The Miami Herald, the Minneapolis Star Tribune, and the Detroit Free Press all repeated the dire statistics as gospel. The successful assault on Craigslist was followed by a cross-country tour by Richardson and the WFN.

None of the media that published Richardson’s astonishing numbers bothered to examine the study at the heart of her claim. If they had, they would have found what we did after asking independent experts to examine the research: It’s junk science.

After all, the numbers are guesses. The data are based merely on looking at photos on the Internet. There is no science.

Eric Grodsky, a sociologist at the University of Minnesota who teaches about proper research construction, says that the study is fundamentally flawed. “The method’s not clean,” he says. “You couldn’t get this kind of thing into a peer-reviewed journal. There are just too many unanswered questions about [the WFN's] methodology.”

Ric Curtis, the chairman of the anthropology department at John Jay College of Criminal Justice in New York City, led a Justice Department–funded study on juvenile prostitution in New York City in 2008. He’s highly skeptical of the WFN study’s claims. “I wouldn’t trust those numbers,” he says. “This new study seems pretty bogus.”

In fact, the group behind the study admits as much. It’s now clear it used fake data to deceive the media and lie to Congress. And it was all done to score free publicity and a wealth of public funding.

“We pitch it the way we think you’re going to read it and pick up on it,” says Kaffie McCullough, the director of Atlanta-based antiprostitution group A Future Not a Past. “If we give it to you with all the words and the stuff that is actually accurate — I mean, I’ve tried to do that with our PR firm, and they say, ‘They won’t read that much.’”

A Future Not a Past is a product of the Atlanta Women’s Foundation, the Juvenile Justice Fund, and Harold and Kayrita Anderson’s foundation. To measure the amount of juvenile prostitution in the state, the consortium hired the Schapiro Group, an Atlanta business-consulting operation.

The Schapiro Group members weren’t academic researchers, and had no prior experience studying prostitution. In fact, the group was best known for research paid for by the American Chamber of Commerce Executives. The study found — surprise — that membership in the Chamber of Commerce improves businesses’ image.

The consultants came up with a novel, if not very scientific, method for tabulating juvenile prostitutes: They counted pictures of young-looking women on online classified sites.

“That’s one of the first problems right there,” Grodsky says. “These advertisers are in the business of making sales, and there’s a market for young-looking women. Why would you trust that the photographs are accurate?”

In other words, the ads, like the covers of women’s magazines, are relentlessly promoting fantasy. Anyone who has tried online dating understands the inherent trouble in trusting photographs.

Even if the person placing the advertisement is the one in the picture, there’s no telling how old the photo is, says David Finkelhor, the director of the Crimes Against Children Research Center at the University of New Hampshire. “How do you know when the pictures were taken?” he asks. “It’s not illegal for an 18-year-old who’s selling sex to put up a picture of herself from when she was 16.”

And if, for the sake of argument, the photos were an accurate portrayal, how do you train those viewing the photographs to guess the correct age? In fact, you don’t.

Before conducting its full study, the Schapiro Group tested the accuracy of its method in a sample of 100 observers. At one point, observers are described as a “random sample.” Elsewhere, they are described as “balanced by race and gender.” These 100 adults were shown pictures of teenagers and young adults whose ages were known, and were asked to guess whether they were younger than 18.

“The study showed that any given ‘young’-looking girl who is selling sex has a 38 percent likelihood of being under age 18,” reads a crucial passage in the study’s explanation of methodology. “Put another way, for every 100 ‘young’-looking girls selling sex, 38 are under 18 years of age. We would compute this by assigning a value of .38 to each of the 100 ‘young’ girls we encounter, then summing the values together to achieve a reliable count.”

This is dense gibberish posing as statistical analysis.

When the team went on to conduct its full statewide study, it simply treated this 38 percent success rate as a constant. Six new observers were then turned loose to count “young-looking” sex ads on online classifieds sites like Craigslist and Backpage. That total count was then multiplied by .38 to come up with a guesstimate of how many children were being trafficked.

“This is a logical fallacy,” says Steve Doig, the Knight Chair in Journalism at Arizona State University, who reviewed the study at our request. “Consider this analogy: Imagine that 100 people were shown pictures of various automobiles and asked to identify the make, and that 38 percent of the time people misidentified Fords as Chevrolets. Using the Schapiro logic, this would mean that 38 percent of Fords on the street actually are Chevys.”

But the Georgia sponsors were happy with the results — after all, the scary-sounding study agreed with what they were saying all along. So the Women’s Funding Network paid Schapiro to expand the study to Michigan, Minnesota, New York, and Texas. (Georgia’s Kayrita Anderson sits on the board of the Women’s Funding Network.) The WFN says it would like to run the study in all 50 states.

The count of online classifieds featuring “young women” is repeated every three months to track how the numbers change over time. That’s the source of Richardson’s claim of a 64 percent increase in child prostitution in Minnesota in a matter of months.

Finkelhor says that’s not how a scientific study is supposed to work. “They don’t tell you what the confidence intervals are, so these changes could just be noise,” he says. “When the Minnesota count goes from 102 to 112, that’s probably just random fluctuations.” There’s a more fundamental issue, of course: “The trend analysis is simply a function of the number of images on these site. … It’s not necessarily an indication that there’s an increase in the number of juveniles involved.”

Despite these flaws, the Women’s Funding Network, which has held rallies across the nation, has been flogging the results relentlessly through national press releases and local member organizations. In press releases, the group goes so far as to compare its conjured-up data to actual hard numbers for other social ills.

“Monthly domestic sex trafficking in Minnesota is more pervasive than the state’s annually reported incidents of teen girls who died by suicide, homicide, and car accidents (29 instances combined); infants who died from SIDS (6 instances); or women of all ages murdered in one year (37 instances),” the study reads. Of course, those other figures are rigorously compiled medical and law enforcement records of documented incidents, so it’s not exactly an apples-to-apples comparison.

The police who tally many of those actual statistics — as well as records of face-to-face encounters with juvenile prostitutes — don’t seem to be very impressed by the statistics put forward by the Women’s Funding Network. “The methodology that they used doesn’t really show the numbers that back it up,” says Sgt. John Bandemer, who heads the Gerald D. Vick Human Trafficking Task Force in St. Paul, Minn. “We take it with a grain of salt.”

The experts we consulted all agreed the Schapiro Group’s published methodology raises more questions than it answers. So we went to the Schapiro Group to ask those questions.

Beth Schapiro founded the Schapiro Group in 1984, starting out mostly with political consulting. The bulk of her group’s work, she says, consists of public opinion research. In 2007, the group installed its own phone-banking center. The group’s website advertises services including customer satisfaction surveys and “voter persuasion calls.”

Counting hard-to-find exploitation victims wasn’t exactly in the company’s repertoire when it was asked by A Future Not a Past to devise a study on juvenile prostitution in 2007, but Schapiro jumped at the opportunity.

The Georgia studies included efforts to count juvenile prostitutes on the street, at hotels, and in escort services, but they also marked the debut of the problematic online classifieds study that would later be reproduced in other states.

In a phone call this month, Schapiro insisted that her study was the first effort to scientifically determine the number of juvenile prostitutes — a claim that would likely surprise the authors of dozens of previous studies, several of which are footnoted in her report.

When we asked Schapiro and Rusty Parker, the leader of the classifieds study, to fill in some of the missing pieces in their methodology, they had a hard time coming up with straight answers. In fact, Parker couldn’t remember key information about how he constructed the study. When asked where he got the sample pictures used to calibrate the all-important 38 percent error rate, he wasn’t sure. “It was a while back,” he says. “I forget exactly where we got them from.”

Parker was equally fuzzy on how the researchers knew the ages of the people pictured in the control group. “Um … I’m afraid I do not remember,” he says.

You might say that this is important information. The Schapiro Group has been telling the world that it cracked the alchemical code that transforms dumb guesses into hard statistics, and that the magic number is .38. But the leader of the study can’t remember the procedure he followed to get that number.

Neither Schapiro nor Parker had any answers when asked whether there was any empirical reason to believe their two critical assumptions: that online photos always represent what the prostitutes actually look like, and that the six handpicked observers conducting the state studies have exactly the same error rate as the initial test batch of 100 random citizens.

Instead, Schapiro beat a hasty retreat, saying the study results shouldn’t be read as actual incidents of prostitution. “We’re the first to tell you, this is not a precise count of the number of girls being prostituted,” she says. “We make no bones about that.”

Of course, a precise count of the number of girls being prostituted is exactly what the statistics are being presented as in the media, in press releases, and in Schapiro’s own study. When this is pointed out, she reverses herself: “Well, yes, these are specific numbers. … And yes, they are hard numbers, and they are numbers that we stand completely behind.”

This is the kind of cognitive whiplash you have to endure if you try to follow Schapiro’s reasoning. She insists that the numbers have the weight of fact, and can properly be cited as actual incidents of juvenile prostitution. But when pressed to justify her study’s broad and unsupported assumptions, she says it is a work in progress and the numbers are only approximations.

Schapiro’s grasp on empirical rigor is such that when asked point-blank to choose between her two contradictory interpretations — estimates or facts — she opts for both. “I would square the circle by saying that you can look at them both ways,” she says.

Any reporters who had read the methodology of the Schapiro report would have been left with doubts, and any reporters who followed up would probably have been treated to the same baffling circuit of nonanswers. The fact that the study’s findings continue to be rebroadcast by national news outlets suggests that not one of these reporters has bothered to read the study about which they are writing.

“You see this kind of thing a lot, unfortunately,” says Rick Edmonds, a media business analyst for the Poynter Institute who writes frequently about statistics. “The kind of skepticism that reporters apply to a statement by a politician just doesn’t get applied to studies.”

Finkelhor says he understands the pressure on reporters to cite figures when they’re writing about juvenile prostitution, but it’s something they need to resist, because despite what groups like the Women’s Funding Network would have you believe, there simply are no good statistics. “You have to say, ‘We don’t know. Estimates have been made, but none of them have a real scientific basis to them,’” he says. “All you can say is, ‘This is the number the police know about, and we think there are more than that, but we don’t know how many more.’”

In her own online photos, the woman who commissioned the Schapiro Group study looks to be in her 50s, with blue eyes, graying hair, and a taste for dangly earrings. Kaffie McCullough first approached the group about conducting a study of juvenile prostitution in Georgia in 2007 when, as director of A Future Not a Past, she realized that having scientific-sounding numbers makes all the difference.

In early 2007, McCullough approached the Georgia Legislature to ask for money for a regional assessment center to track juvenile prostitution. “We had no research, no nothing. The legislators didn’t even know about it,” she recalls. “We got a little bit. We got about 20 percent of what we asked for.”

Later that year, the Schapiro Group made its first counts, and when McCullough returned to the Legislature the following session, she had the study’s statistics in hand. “It gave us traction — night and day,” she says. “That year, we got all the rest of that money, plus we got a study commission.”

McCullough touts the fundraising benefits of the study whenever she can. Since the Schapiro study was picked up for replication nationwide by the Women’s Funding Network, she has acted as a sort of technical consultant for state groups as they debate whether to invest in the project. Whenever she’s asked, she tells the local groups that the money they spend will come back to them with hefty dividends. “I would say, ‘The research costs money, but we’ve been able to broker — I don’t know what it is now, I think it’s over $1.3, $1.6 million in funding that we never would have gotten,’” she says.

McCullough initially maintained that she stands by the Schapiro Group study, in part because she says she was told that “it is the same scientific methodology that science has been using for a long time to measure endangered species.” But when pressed on whether she really believes that counting Internet photos is reliable, she grants that the sex-work industry isn’t exactly the gold standard of truth in advertising. “That’s absolutely correct,” she says. “That’s part of how that business operates: It’s a bait-and-switch.”

And given the tricky nature of the photographs, she admits that counting pictures isn’t a precise way to measure juvenile prostitutes. “I can’t guarantee that any picture that four of those six people said looked young — that may not be the girl that you’d get if you called up,” she concedes.

Asked whether she has any reason to believe that the six observers in the study have the identical 38 percent error rate as the 100 random citizens who were the initial test subjects, she allows that it might be worth revisiting that question.

The basic truth is that the study exists in service of the advocacy, and if news outlets present the Schapiro Group’s numbers as gospel, it certainly doesn’t hurt the advocates’ cause.

Admitting that there is no authoritative scientific count of juvenile prostitution, as Finkelhor recommends, isn’t an option for McCullough. She recalls an early presentation she made in Nebraska, when a politician gave her a piece of advice that stuck.

“He said, ‘If you all as a movement don’t start having numbers, you are going to lose the money,’” McCullough recalls. “‘How can you justify millions of dollars when there are only hundreds of victims that you’re actually serving?’”

Editor’s conclusion: On March 16, the drumbeat continued in the U.S. Senate with a briefing on domestic sex trafficking of minors that featured actress Mira Sorvino and the startling statistic that 100,000 children are trafficked for sex annually in America. Trafficking — in labor and sex — became a defining issue in the administration of President George W. Bush. But as an investigation by the Washington Post in 2007 revealed, victims in the sex trade were difficult to come by. Today, advocates have shifted media attention to allegations of trafficking in children. But facts to suggest a plague of underage perversion simply do not exist, despite claims to the contrary.

In a deficit-obsessed Congress, there is a long line of those seeking tax dollars to raise awareness of trafficking: government agencies, nonprofits, religious groups, the well-intentioned, as well as abolitionists opposed to pornography and adult services.

It is no surprise that some seek to use children as a wedge.

Responsible parties prosecute predators and rescue victims. Not everyone with a microphone is responsible. The challenge of keeping children out of the hands of exploiters is real, but solutions are not clear in an atmosphere of hyped hysteria.

http://www.sfweekly.com/2011-03-23/news/sex-trafficking-fake-study-womans-funding-network-deborah-richardson-nick-pinto/