Facebook
Twitter
Pocket
Reddit
Email
xxx.lanl.gov. The address was cryptic, with a tantalizing whiff of government secrets, or worse.
The server itself was exactly the opposite. Government, yes—it was hosted by Los Alamos National Laboratory—but openly accessible in a way that, in those early Internet days of the 1990s, was totally new, and is still game-changing today.
The site, known as arXiv (pronounced “archive,” and long since decamped to the more wholesome address “arXiv.org” and to the stewardship of the Cornell University Library), is a vast repository of scientific preprints, articles that haven’t yet gone through the peer-review process or aren’t intended for publication in refereed journals. (Papers can also appear, often in revised form, after they have been published elsewhere.) As of July 2016, there were more than a million papers on arXiv, leaning heavily toward the hardest of the hard sciences: math, computer science, quantitative biology, quantitative finance, statistics, and above all, physics.
ArXiv is the kind of library that, 30 years ago, scientists could only dream of: totally searchable, accessible from anywhere, free to publish to and read from, and containing basically everything in the field that’s worth reading. At this golden moment in technological history, when you can look up the history of atomic theory on Wikipedia while waiting in line at Starbucks, this might seem trivial. But in fact it was revolutionary.
Practically, arXiv has leveraged new technologies to create a boon for its community. What is less visible, though, is that it has had to answer a difficult philosophical question, one which resonates through the rest of the scientific community: What, exactly, is worth reading? What counts as science?
Before arXiv, preprint papers were available only within small scientific circles, distributed by hand and by mail, and the journals in which they were ultimately published months later (if they were published at all) were holed up in university libraries. But arXiv has democratized the playing field, giving scientists instant access to ideas from all kinds of colleagues, all over the world, from prestigious chairs at elite universities to post-docs drudging away at off-brand institutions and scientists in developing countries with meager research support.
Paul Ginsparg set up arXiv in 1991, when he was a 35-year-old physicist at Los Alamos. He expected only about 100 papers to go out to a few hundred email subscribers in the first year. But by the summer of 1992, more than 1,200 papers had been submitted. It was a good problem to have, but still a problem. While Ginsparg had no intention of giving incoming papers the top-to-tail scrutiny of peer review, he did want to be sure that readers could find the ones they were most interested in. So he started binning the incoming papers into new categories and sub-categories and bringing on more and more moderators, who took on the work as volunteers, as a service to their scientific community.
Will unclassifiable papers get lost in the muck of the truly incoherent?
The arXiv credo is that papers should be “of interest, relevance, and value” to the scientific disciplines that arXiv serves. But as the site and its public profile grew, it began attracting papers from outside the usual research circles, and many of those papers didn’t pass the test. They weren’t necessarily bad science, says Ginsparg. Bad science can be examined, tested, and refuted. They were “non-science”—sweeping theories grandly claiming to overturn Einstein, Newton, and Hawking; to reveal hidden connections between physics and ESP and UFOs; and to do it all almost entirely without math and experiment.
The arXiv’s default stance is acceptance—papers are innocent “until proven guilty,” Ginsparg says—but the non-science papers were a waste of scholarly readers’ time. And if they were allowed to share the same virtual shelf space with legitimate science, they could create confusion among arXiv’s growing audience of journalists and policymakers. So, paper by paper, moderators had to make the call: What is and isn’t science?
Most arXiv users were satisfied with the moderators’ decisions. But some people felt papers were getting tossed aside that should have made it into arXiv, and some scientists—especially those on the academic fringes—accused arXiv moderators of censoring against-the-grain ideas.
The problem that arXiv’s moderators were tackling wasn’t a new one. In 1959, philosopher of science Thomas Kuhn called it “the essential tension”: the conflict between traditional scholarly limits, which place scientific questions and practices in- or out-of-bounds, and the kind of freewheeling inquiry that embraces maverick ideas and methods. To move forward, science needs both, the thinking goes. If innovative ideas often spring up in the spaces between well-established disciplines, will unclassifiable but credible papers get lost in the muck of the truly incoherent?
ArXiv moderators, though, don’t have much time for Kuhnian rumination. Many users probe the site daily and arXiv wants to stay as fresh and up-to-date as possible. So, from the beginning, arXiv imposed a “very unforgiving 24-hour turnaround,” says Ginsparg. Any paper that comes in before 4 p.m. Eastern on a weekday is cued to go online at 8 p.m. that night. Moderators have less than a day to flag a paper for rejection or further review—and sometimes as little as four hours. Mindful of the daily dash, Ginsparg had an idea for how to give his volunteer moderators a helping hand: a computer program that would do some of the thinking for them.
Credible ideas do sometimes land on the wrong side of arXiv’s human and computer filters.
Ginsparg didn’t intend to design a program that could tell science from non-science. His initial goal was modest: to build an algorithm that could classify papers by subject category, saving moderators the hassle of looking through misclassified submission. It took just a few hours of coding to create a program that could ingest a paper, pick out meaningful words, count up how many times each word appeared, and classify it based on how moderators had classified similar papers in the past. Rather than keying in the lexicon of key words himself, he let the algorithm figure out on its own which words were the best predictors. Because moderators reviewed every classification, the algorithm got instant feedback, and it got smarter and better, one paper at a time, hundreds of papers every day.
This computerized classification had a surprising side effect: It had an uncanny ability to distinguish “good” papers from “bad” ones. Without meaning to, arXiv had reduced Kuhn’s essential tension to practice. How?
By reviewing thousands of papers, the classifier algorithm had gotten a fix on a telltale sign of real science: language. As the program learned scientific language, it made judgments similar to those of the human gatekeepers. Outlier papers, the ones that got rejected, didn’t line up with the usual language norms of any scientific discipline. The deviation might have been obvious, as it is in chimeric papers that fuse scientifically disparate fields. Or it might have been subtle: the wrong distribution of seemingly content-less words like “and,” “or,” “it,” or “that.”
The attention to language was a direct mirror of what academic editors, peer reviewers, and scientists already do in judging work. ArXiv reflects “properties about the [scientific] community,” and demarcates that community, says Luis Reyes-Galindo, a sociologist of science at Cardiff University in Wales.
Language also serves as a biomarker of pseudoscience. John Baez, a mathematical physicist at the University of California, Riverside, has a “Crackpot Index” that scores papers on 37 figures of crackpot merit (or demerit): five points for each word in all caps, 10 points “for claiming that your work is on the cutting edge of a ‘paradigm shift,’ ” a 50-point bounty for “claiming you have a revolutionary theory but giving no concrete testable predictions” (the last of these admittedly having less to do with language).
There’s a reason language is a good tell. Authors who go through the formal scientific training process—four years of college, many more years as a graduate student and post-doc—do more than just absorb a set of facts and a method of thinking and experimenting. They also learn to communicate in a specific style. “Being a scientist is 90 percent language—knowing how to speak, talking the talk,” says Reyes-Galindo.
Papers the algorithm considers “outliers” aren’t automatically rejected; they are just flagged for extra human scrutiny. But because science aspires to be an intellectual standard-bearer where ideas are evaluated on substance, not style, some argue that there is still something unsettling about identifying outsiders and casting them out based on the words they choose. What, for example, to make of scientists who cross disciplines and make valuable contributions?
“Long before the Internet, we can find examples of curious work on the boundaries,” says Massachusetts Institute of Technology physicist and historian of science David Kaiser. “Some of it rightly has gone nowhere, but some of it really does cash out.” Plenty of today’s take-it-for-granted science principles—the heliocentric solar system, the notion that invisible fields convey physical forces, or that physical laws can be described by mathematical equations—were seen as radical in their day. And some investigations that were dismissed in the 1970s as hazy, drug-induced blather are now indispensable to research on quantum theory, Kaiser adds.
In his 2011 book How the Hippies Saved Physics, Kaiser wrote, “Many of the ideas that now occupy the core of quantum information science once found their home amid an anything-goes counterculture frenzy, a mishmash of spoon-bending psychics, Eastern mysticism, LSD trips, CIA spooks chasing mind-reading dreams, and comparable ‘Age of Aquarius’ enthusiasms.” The physicists who laid the groundwork for quantum computing, quantum encryption, and quantum teleportation took an approach that was unfashionable and academically marginalized, but time has proved it to be essential. “It is doubtful whether some of the more colorful ideas or approaches from that earlier period would have made it onto arXiv,” says Kaiser.
Today, too, credible—or at least not out-and-out nutty—ideas do sometimes land on the wrong side of arXiv’s human and computer filters. “I know three cases in my own field where I can judge, of good papers by professional physicists, including tenured professors at research universities with good publication records, being excluded or challenged,” says Lee Smolin, a theoretical physicist at the Perimeter Institute for Theoretical Physics in Waterloo, Ontario.
Paper by paper, moderators had to make the call: What is and isn’t science?
Many researchers who have had to fight to see their papers on arXiv aren’t eager to talk about the experience, fearing a black mark on them and their work. One exception is Nicolas Gisin, an accomplished physicist at the University of Geneva, who, in December 2015, issued a public grievance on behalf of two of his students. The students, he wrote, had submitted a paper on black holes to arXiv. The paper’s conclusion, that it is impossible to fall into a black hole, was unorthodox, and its authors weren’t part of the usual circle of researchers working on black holes. But, Gisin felt, their calculations were clear and their logic lucid. “I appreciated my students’ eagerness to transgress barriers established between different subfields of physics,” he wrote in an essay for a blog hosted by the Institute for Quantum Optics and Quantum Information—Vienna.
After a wait of several months—an eternity in the fast-turnaround world of arXiv—the paper was rejected. It had been flagged by the system and then turned down by the moderator. Then a second paper by the pair was rejected, too, this time so quickly that Gisin concluded the students were “now on a black list,” he wrote on the blog.
Ginsparg denies that arXiv uses any such black list. He wasn’t involved in the case of Gisin’s students, but looked over their work and felt that the papers contained elementary errors. Referring to the case in an article about arXiv for The EMBO Journal, Ginsparg wrote that arXiv “moderators work above and beyond the call to spare ill-advised graduate students unnecessary embarrassment.” The first student paper was ultimately published in Physics Letters A, a peer-reviewed journal.
Ginsparg explains that arXiv’s automated screening may flag users whose previous submissions have been rejected, and will reject papers coming from users who re-submit rejected papers under different aliases. ArXiv also offers users an appeals process: An author can request that a rejected paper be re-reviewed by the original moderator, and if it gets turned down again, the author can initiate a third round of review by an appellate moderator. If a paper is rejected by arXiv and later accepted by a peer-reviewed journal, it also gets back-door acceptance to arXiv. Gisin’s students didn’t try to appeal, though, and the paper wasn’t published to arXiv even after it was accepted by Physics Letters A.1
In spite of occasional brushfires of controversy, arXiv’s standards are remarkably liberal. Ginsparg says that, in the last year, less than 1 percent of submissions were turned away due to content issues, and many of the site’s critics, like Gisin, remain devoted users.
Yet arXiv’s process can still be opaque. “ArXiv has no habeas corpus,” says Kaiser. When moderators reject a paper, they need not give an explanation beyond arXiv’s boilerplate removal criteria. And successfully leveling an appeal after a rejection can be an exercise in frustration, says Smolin. “My sense from the few cases I know about is that their appeals process is weak and doesn’t follow the best professional practices, given the importance of arXiv for careers in physics.”
There is one middle option between acceptance and rejection: Dubious papers can be shunted to a category called “general physics,” or “gen-ph.” At best, gen-ph is a home for papers that don’t fit neatly into any other category; at worst, it is a crackpot dumping ground. Either way, it allows moderators to dodge the drama that comes with out-and-out rejections. For many authors, though, being relegated to gen-ph is just as galling as being rejected outright: As one such researcher told Reyes-Galindo, having his paper reclassified to gen-ph felt like a deliberate attempt to diminish the paper’s impact. Still, gen-ph makes up just a sliver of the arXiv: between July 2014 and June 2015, just 302 papers wound up in gen-ph, out of more than 100,000 papers published on the arXiv.
Frustration with arXiv has boiled up into an alternative pre-print site called viXra, founded by Philip Gibbs, an independent physicist in England. Anyone may post anything on viXra, though house rules do prohibit “vulgar, libellous, plagiaristic or dangerously misleading” content. To those whose work finds a home there, viXra is a kind of intellectual sanctuary. But to many in the scientific community, viXra is more like the Island of Misfit Toys, a pitiable exile for oddball and broken playthings. (Regarding the value of viXra, Ginsparg quips: “It’s a great thing! Any time anybody asks us why we need filtering, we point them to [viXra].”) To Gibbs, who has also published a 17-part retrospective called “Crackpots Who Were Right” on the viXra blog, viXra stands as a historical document preserving the lineage of ahead-of-their-time ideas for a future in which their importance may be recognized.
It’s tempting to think the Internet has created a genuinely new problem for thinkers trying to draw the line between science and pseudoscience. But, says Kaiser, arXiv is managing a jumbo-sized version of the same old tension. “Scientists and scholars have complained for literally a millennium about needing tools to curate and help deal with the most precious commodity of all, which is limited attention.” The arXiv may have remodeled the gates of science—made them swing wider and more smoothly—but still, someone always winds up stuck outside.
Kate Becker writes about physics, astronomy, and other wonders of the universe. She lives in Brookline, Massachusetts.
Photo collage credit: PhotoAlto/Matthieu Spohn/Getty Images; szefei,Rommel Canlas/Shutterstock
Get the Nautilus newsletter
Cutting-edge science, unraveled by the very brightest living thinkers.
