Power, Pollution and the Internet

Ethan Pines for The New York Times

Data centers are filled with servers, which are like bulked-up desktop computers, minus screens and keyboards, that contain chips to process data.

SANTA CLARA, Calif. - Jeff Rothschild's machines at Facebook had a problem he knew he had to solve immediately. They were about to melt.

The company had been packing a 40-by-60-foot rental space here with racks of computer servers that were needed to store and process information from members' accounts. The electricity pouring into the computers was overheating Ethernet sockets and other crucial components.

Thinking fast, Mr. Rothschild, the company's engineering chief, took some employees on an expedition to buy every fan they could find - “We cleaned out all of the Walgreens in the area,” he said - to blast cool air at the equipment and prevent the Web site from going down.

That was in early 2006, when Facebook had a quaint 10 million or so users and the one main server site. Today, the information generated by nearly one billion people requires outsize versions of these facilities, called data centers, with rows and rows of servers spread over hundreds of thousands of square feet, and all with industrial cooling systems.

They are a mere fraction of the tens of thousands of data centers that now exist to support the overall explosion of digital information. Stupendous amounts of data are set in motion each day as, with an innocuous click or tap, people download movies on iTunes, check credit card balances through Visa's Web site, send Yahoo e-mail with files attached, buy products on Amazon, post on Twitter or read newspapers online.

A yearlong examination by The New York Times has revealed that this foundation of the information industry is sharply at odds with its image of sleek efficiency and environmental friendliness.

Most data centers, by design, consume vast amounts of energy in an incongruously wasteful manner, interviews and documents show. Online companies typically run their facilities at maximum capacity around the clock, whatever the demand. As a result, data centers can waste 90 percent or more of the electricity they pull off the grid, The Times found.

To guard against a power failure, they further rely on banks of generators that emit diesel exhaust. The pollution from data centers has increasingly been cited by the authorities for violating clean air regulations, documents show. In Silicon Valley, many data centers appear on the state government's Toxic Air Contaminant Inventory, a roster of the area's top stationary diesel polluters.

Worldwide, the digital warehouses use about 30 billion watts of electricity, roughly equivalent to the output of 30 nuclear power plants, according to estimates industry experts compiled for The Times. Data centers in the United States account for one-quarter to one-third of that load, the estimates show.

“It's staggering for most people, even people in the industry, to understand the numbers, the sheer size of these systems,” said Peter Gross, who helped design hundreds of data centers. “A single data center can take more power than a medium-size town.”

Energy efficiency varies widely from company to company. But at the request of The Times, the consulting firm McKinsey & Company analyzed energy use by data centers and found that, on average, they were using only 6 percent to 12 percent of the electricity powering their servers to perform computations. The rest was essentially used to keep servers idling and ready in case of a surge in activity that could slow or crash their operations.

A server is a sort of bulked-up desktop computer, minus a screen and keyboard, that contains chips to process data. The study sampled about 20,000 servers in about 70 large data centers spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies.

“This is an industry dirty secret, and no one wants to be the first to say mea culpa,” said a senior industry executive who asked not to be identified to protect his company's reputation. “If we were a manufacturing industry, we'd be out of business straightaway.”

These physical realities of data are far from the mythology of the Internet: where lives are lived in the “virtual” world and all manner of memory is stored in “the cloud.”

Power, Pollution and the Internet

Ethan Pines for The New York Times

Data centers are filled with servers, which are like bulked-up desktop computers, minus screens and keyboards, that contain chips to process data.

SANTA CLARA, Calif. - Jeff Rothschild's machines at Facebook had a problem he knew he had to solve immediately. They were about to melt.

The company had been packing a 40-by-60-foot rental space here with racks of computer servers that were needed to store and process information from members' accounts. The electricity pouring into the computers was overheating Ethernet sockets and other crucial components.

Thinking fast, Mr. Rothschild, the company's engineering chief, took some employees on an expedition to buy every fan they could find - “We cleaned out all of the Walgreens in the area,” he said - to blast cool air at the equipment and prevent the Web site from going down.

That was in early 2006, when Facebook had a quaint 10 million or so users and the one main server site. Today, the information generated by nearly one billion people requires outsize versions of these facilities, called data centers, with rows and rows of servers spread over hundreds of thousands of square feet, and all with industrial cooling systems.

They are a mere fraction of the tens of thousands of data centers that now exist to support the overall explosion of digital information. Stupendous amounts of data are set in motion each day as, with an innocuous click or tap, people download movies on iTunes, check credit card balances through Visa's Web site, send Yahoo e-mail with files attached, buy products on Amazon, post on Twitter or read newspapers online.

A yearlong examination by The New York Times has revealed that this foundation of the information industry is sharply at odds with its image of sleek efficiency and environmental friendliness.

Most data centers, by design, consume vast amounts of energy in an incongruously wasteful manner, interviews and documents show. Online companies typically run their facilities at maximum capacity around the clock, whatever the demand. As a result, data centers can waste 90 percent or more of the electricity they pull off the grid, The Times found.

To guard against a power failure, they further rely on banks of generators that emit diesel exhaust. The pollution from data centers has increasingly been cited by the authorities for violating clean air regulations, documents show. In Silicon Valley, many data centers appear on the state government's Toxic Air Contaminant Inventory, a roster of the area's top stationary diesel polluters.

Worldwide, the digital warehouses use about 30 billion watts of electricity, roughly equivalent to the output of 30 nuclear power plants, according to estimates industry experts compiled for The Times. Data centers in the United States account for one-quarter to one-third of that load, the estimates show.

“It's staggering for most people, even people in the industry, to understand the numbers, the sheer size of these systems,” said Peter Gross, who helped design hundreds of data centers. “A single data center can take more power than a medium-size town.”

Energy efficiency varies widely from company to company. But at the request of The Times, the consulting firm McKinsey & Company analyzed energy use by data centers and found that, on average, they were using only 6 percent to 12 percent of the electricity powering their servers to perform computations. The rest was essentially used to keep servers idling and ready in case of a surge in activity that could slow or crash their operations.

A server is a sort of bulked-up desktop computer, minus a screen and keyboard, that contains chips to process data. The study sampled about 20,000 servers in about 70 large data centers spanning the commercial gamut: drug companies, military contractors, banks, media companies and government agencies.

“This is an industry dirty secret, and no one wants to be the first to say mea culpa,” said a senior industry executive who asked not to be identified to protect his company's reputation. “If we were a manufacturing industry, we'd be out of business straightaway.”

These physical realities of data are far from the mythology of the Internet: where lives are lived in the “virtual” world and all manner of memory is stored in “the cloud.”

Free Speech in the YouTube Age

San Francisco

COMPANIES are usually accountable to no one but their shareholders.

Internet companies are a different breed. Because they traffic in speech - rather than, say, corn syrup or warplanes - they make decisions every day about what kind of expression is allowed where. And occasionally they come under pressure to explain how they decide, on whose laws and values they rely, and how they distinguish between toxic speech that must be taken down and that which can remain.

The storm over an incendiary anti-Islamic video posted on YouTube has stirred fresh debate on these issues. , which owns YouTube, restricted access to the video in Egypt and Libya, after the killing of a United States ambassador and three other Americans. Then, it pulled the plug on the video in five other countries, where the content violated local laws.

Some countries blocked YouTube altogether, though that didn't stop the bloodshed: in Pakistan, where elections are to be scheduled soon, riots on Friday left a death toll of 19.

The company pointed to its internal edicts to explain why it rebuffed calls to take down the video altogether. It did not meet its definition of hate speech, YouTube said, and so it allowed the video to stay up on the Web. It didn't say very much more.

That explanation revealed not only the challenges that confront companies like Google but also how opaque they can be in explaining their verdicts on what can be said on their platforms. Google, Facebook and Twitter receive hundreds of thousands of complaints about content every week.

“We are just awakening to the need for some scrutiny or oversight or public attention to the decisions of the most powerful private speech controllers,” said Tim Wu, a Columbia University law professor who briefly advised the Obama administration on consumer protection regulations online.

Google was right, Mr. Wu believes, to selectively restrict access to the crude anti-Islam video in light of the extraordinary violence that broke out. But he said the public deserved to know more about how private firms made those decisions in the first place, every day, all over the world. After all, he added, they are setting case law, just as courts do in sovereign countries.

Mr. Wu offered some unsolicited advice: Why not set up an oversight board of regional experts or serious YouTube users from around the world to make the especially tough decisions?

Google has not responded to his proposal, which he outlined in a blog post for The New Republic.

Certainly, the scale and nature of YouTube makes this a daunting task. Any analysis requires combing through over a billion videos and overlaying that against the laws and mores of different countries. It's unclear whether expert panels would allow for unpopular minority opinion anyway. The company said in a statement on Friday that, like newspapers, it, too, made “nuanced” judgments about content: “It's why user-generated content sites typically have clear community guidelines and remove videos or posts that break them.”

Privately, companies have been wrestling with these issues for some time.

The Global Network Initiative, a conclave of executives, academics and advocates, has issued voluntary guidelines on how to respond to government requests to filter content.

And the Anti-Defamation League has convened executives, government officials and advocates to discuss how to define hate speech and what to do about it.

Hate speech is a pliable notion, and there will be arguments about whether it covers speech that is likely to lead to violence (think Rwanda) or demeans a group (think Holocaust denial), just as there will be calls for absolute free expression.

Behind closed doors, Internet companies routinely make tough decisions on content.

Apple and Google earlier this year yanked a mobile application produced by Hezbollah. In 2010, YouTube removed links to speeches by an American-born cleric, , in which he advocated terrorist violence; at the time, the company said it proscribed posts that could incite “violent acts.”

ON rare occasions, Google has taken steps to educate users about offensive content. For instance, the top results that come up when you search for the word “Jew” include a link to a virulently anti-Jewish site, followed by a promoted link from Google, boxed in pink. It links to a page that lays out Google's rationale: the company says it does not censor search results, despite complaints.

Susan Benesch, who studies hate speech that incites violence, said it would be wise to have many more explanations like this, not least to promote debate. “They certainly don't have to,” said Ms. Benesch, director of the Dangerous Speech Project at the World Policy Institute. “But we can encourage them to because of the enormous power they have.”

The companies point out that they obey the laws of every country in which they do business. And their employees and algorithms vet content that may violate their user guidelines, which are public.

YouTube prohibits hate speech, which it defines as that which “attacks or demeans a group” based on its race, religion and so on; Facebook's hate speech ban likewise covers “content that attacks people” on the basis of identity. Google and Facebook prohibit hate speech; Twitter does not explicitly ban it. And anyway, legal scholars say, it is exceedingly difficult to devise a universal definition of hate speech.

Shibley Telhami, a political scientist at the University of Maryland, said he hoped the violence over the video would encourage a nuanced conversation about how to safeguard free expression with other values, like public safety. “It's really about at what point does speech becomes action; that's a boundary that becomes difficult to draw, and it's a slippery slope,” Mr. Telhami said.

He cautioned that some countries, like Russia, which threatened to block YouTube altogether, would be thrilled to have any excuse to squelch speech. “Does Russia really care about this film?” Mr. Telhami asked.

International law does not protect speech that is designed to cause violence. Several people have been convicted in international courts for incitement to in Rwanda.

One of the challenges of the digital age, as the YouTube case shows, is that speech articulated in one part of the world can spark mayhem in another. Can the companies that run those speech platforms predict what words and images might set off carnage elsewhere? Whoever builds that algorithm may end up saving lives.

Commander of All Apps, Except the One for Guilt

IN San Francisco recently, I was running late for a meeting, desperate for a ride. But no buses or cabs were in sight.

Then I remembered a service called Lyft. I pulled out my smartphone and quickly downloaded the application, which lets regular people act as chauffeurs for a fee. As instructed, I entered my location - then crossed my fingers and waited.

Five anxious minutes later, a clean black Audi pulled up, with a chatty young man at the wheel. I jumped in and we made friendly small talk. I gave him advice for his future trip to New York, and he invited me to a party that night in the city. We reached my destination, and I slid out of my seat, transferred $10 to the service through the Lyft app, and walked into my meeting with a smile.

It was the start of a series of encounters with useful apps providing services in the nick of time. At first, these successes emboldened me. The next day, for example, I found myself in yet another bind. I realized that a booking error had left me without a hotel room in San Francisco for one night. Normally, such a travel glitch would have caused me to panic. But I calmly searched until I found an application called HotelTonight, which shows last-minute rooms available in a particular city. I found and reserved a place to stay, all while drinking iced coffee and sitting outside in a park.

What couldn't I do, what problems couldn't be solved, with the tap of an app? A few hours later, I realized that the device I'd brought to supply my laptop with an Internet connection was on the fritz. I needed it for a press event starting bright and early the next day, but I also had to get to an important dinner. But, again, I didn't have a meltdown. I simply pulled out my phone and found an application called Exec. It quickly matched me with a human assistant who was willing to track down and deliver a replacement, so long as I paid him $25 an hour, which I was quite willing to do.

But a few hours later, the first signs of trouble began. While sitting at the dinner, receiving texts and questions from the patient young man who was buying my mobile hot spot at Best Buy, I started to feel guilty. Was I taking advantage of him? Was I paying him enough? Was my time worth more than his?

No doubt these services are helpful; in all three instances, they were lifesavers. This emerging on-demand economy, made up of a wave of mobile applications and services, is certainly convenient. It is intended to deliver almost anything you need or want with the flick of a finger, if you have a smartphone and the cash to spare. A game changer? Definitely.

But are there murkier issues, about the haves and have-nots in a tech world, lurking just below the surface? I certainly felt uneasy about how simple it was to command an army of mobile helpers via my phone.

My discomfort, it turns out, is not uncommon, according to Justin Kan, the founder of Exec, an online service that provides helpers for a variety of tasks. “I don't think it comes that naturally to people at first,” Mr. Kan said. “There is friction. But when they see other people using it, they will find ways to use it and that it is O.K.”

Rachel Botsman, one of the authors of “What's Mine Is Yours: The Rise of Collaborative Consumption,” notes that “it can be quite uncomfortable asking a stranger to do something for you, even if you are paying them.” Yet, she says, many people have overcome this aversion.

She cited Airbnb, a service that lets people rent their homes to travelers, for rates comparable to a hostel or hotel, with the company taking a slice of the fee. At first, letting a stranger crash in your spare bedroom may seem hard to imagine - yet, many have found that making extra cash from their homes is appealing enough to join in. And enough travelers have enjoyed the authenticity of staying in someone else's home to have made the service quite popular. Since 2008, when Airbnb was introduced, it has helped people book 10 million nights in more than 26,000 cities around the globe.

Such services are useful enough that at least some of them will thrive, Ms. Botsman said.

“The economics are bigger than the individual,” she said. “This is changing the way things get done. There are massive socioeconomic implications for the unemployed and underemployed.”

She gave the example of a skilled chef who hadn't been able to secure a culinary job. Through one of these apps, the chef could take quick requests for a caterer or a personal cook, thus getting a foot in the door.

In addition, small businesses, she said, could scale back on staff or even office space by using on-demand services like Exec and TaskRabbit, which also help find people to perform errands on the spot.

“There has always been a demand for people who will do odd jobs,” said Kartik Hosanagar, a professor of online commerce at the Wharton School of the University of Pennsylvania. “There probably always was a good amount of supply as well, although that supply has increased dramatically since the 2008 recession. However, there wasn't an efficient way to match supply and demand previously.”

The bigger question is whether these companies can grow beyond their existing markets. Airbnb is in 26,000 cities worldwide, but Lyft and Exec are relatively young companies and currently operate only in San Francisco.

“There's a fundamental issue of intimacy,” said Susan Etlinger, an analyst at the Altimeter Group, who advises companies on how best to use the Web in their business. “Will people feel safe and comfortable on a larger scale? They'll have to do more of what Airbnb has done, spread the word about how well regulated and managed the service is. But even then, there's always tension between the person who is the employer and the person who is the employee that could turn users off.”

FOR consumers, it may be a matter of getting accustomed to the new services, and deciding how useful they are and whether they are worth the cost. I found myself weighing my discomfort against the convenience of being able to vastly increase the number of things I could do in a short time, and all for a relatively small amount of money.

On the final night on that San Francisco trip, for example, I faced yet another crunch.  I needed to be in three places at once - buying a new charger, picking out a birthday present for a friend and meeting friends from college for a drink. I stared at my phone, debating whether to use a mobile app to enlist an assistant for help with the shopping errands.

This time, at least, I decided that I didn't need the help. I'd deal with the errands at another time. So I put down my phone and headed straight to the bar.

The Internet? We Built That

Illustration by Tom Gauld

Who created the Internet and why should we care? These questions, so often raised during the Bush-Gore election in 2000, have found their way back into the political debate this season - starting with one of the most cited texts of the preconvention campaign, Obama's so-called “you didn't build that” speech. “The Internet didn't get invented on its own,” Obama argued, in the lines that followed his supposed gaffe. “Government research created the Internet so that all the companies could make money off the Internet.” In other words: business uses the Internet, but government made it happen.

About a week after Obama's speech, The Wall Street Journal's Gordon Crovitz took on those lines from Obama's speech, claiming it was an “urban legend” that the government built the Internet. Credit for the early networking innovations, Crovitz argued, belonged to private-sector companies like Xerox and Apple. It was no accident, he observed, that the Net languished in relative obscurity for two decades until private corporations and venture capitalists turned their focus to it.

So what had once seemed to be a relatively stable narrative grounding has in recent months erupted with all sorts of political tremors. For most of the past two decades, the story of the Internet's origins followed a fairly standardized plot: the Internet was originally developed by computer scientists whose research was heavily financed by the federal government, most notably through Darpa, the research arm of the Defense Department. Some narratives emphasized the decentralized network architecture designed by Paul Baran to survive a nuclear strike; others gave credit to the British programmer Tim Berners-Lee, whose World Wide Web gave the Internet a more accessible hypertextual layer. And of course there were all those Al Gore jokes.

The renewed political stakes in the details of this origin story are obvious. If you believe Big Government built the most important communications platform of our time, then that success is a powerful riposte to all the standard claims about bureaucratic inefficiencies and incompetence. Government might be able to out-innovate the private sector, given the right focus and commitment (and freedom from being beholden to stockholders). But if you believe that the Internet's success is largely attributable to the private sector, all the usual libertarian homilies remain untarnished.

So was the Internet created by Big Government or Big Capital? The answer is: Neither. This is what's most notable about the debate over the Net's origins: it misses the most interesting part of the story. We live in a world that assumes that the most important and original products in society - bridges, cars, iPads, hospitals, 787s, houses - are created either by states or by corporations. And yet, against all odds, the Internet came from somewhere else entirely.

Like many of the bedrock technologies that have come to define the digital age, the Internet was created by - and continues to be shaped by - decentralized groups of scientists and programmers and hobbyists (and more than a few entrepreneurs) freely sharing the fruits of their intellectual labor with the entire world. Yes, government financing supported much of the early research, and private corporations enhanced and commercialized the platforms. But the institutions responsible for the technology itself were neither governments nor private start-ups. They were much closer to the loose, collaborative organizations of academic research. They were networks of peers.

Peer networks break from the conventions of states and corporations in several crucial respects. They lack the traditional economic incentives of the private sector: almost all of the key technology standards are not owned by any one individual or organization, and a vast majority of contributors to open-source projects do not receive direct compensation for their work. (The Harvard legal scholar Yochai Benkler has called this phenomenon “commons-based peer production.”) And yet because peer networks are decentralized, they don't suffer from the sclerosis of government bureaucracies. Peer networks are great innovators, not because they're driven by the promise of commercial reward but rather because their open architecture allows others to build more easily on top of existing ideas, just as Berners-Lee built the Web on top of the Internet, and a host of subsequent contributors improved on Berners-Lee's vision of the Web.