The Internet Is Broken, and Shellshock Is Just the Start of Our Woes

Shellshock is one of the oldest known bugs in history. But its story is not that usual. Early this year, security researchers discovered another bug, called Heartbleed, that has languished in open source software for years. Both bugs are indicative a problem that could continue to plague the internet unless we change the way we write and audit software. The truth is that the net is littered with code that dates back decades, and some of it never gets looked at.
cracked2
WIRED

Brian Fox drove from Boston to Santa Barbara, with two tapes stashed in his trunk.

These weren't music tapes or video tapes. They were computer tapes---two massive reels loaded with software code and data, the sort you can see spinning on furniture-sized computers in classic movies like Dr. Strangelove and Three Days of the Condor.

The year was 1987, and as Fox drove cross-country to his new home, the tapes held a software program called Bash, a tool that Fox had built for the UNIX operating system and tagged with a license that let anyone use the code and even redistribute it to others. Fox---a high school dropout who spent his time hanging out with MIT computer geeks such as Richard Stallman---was a foot soldier in an ambitious effort to create software that was free, hackable, and unencumbered by onerous copy restrictions. It was called the Free Software Movement, and the idea was to gradually rebuild all of the components of the UNIX operating system into a free product called GNU and share them with the world at large. It was the dawn of open source software.

Brian Fox.

Wikipedia CC 3.0

Fox and Stallman didn't know it at the time, but they were building the tools that would become some of the most important pieces of our global communications infrastructure for decades to come. After Fox drove those tapes to California and went back to work on Bash, other engineers started using the software and even helped build it. And as UNIX gave rise to GNU and Linux---the OS that drives so much of the modern internet---Bash found its way onto tens of thousands of machines. But somewhere along the way, in about 1992, one engineer typed a bug into the code. Last week, more then twenty years later, security researchers finally noticed this flaw in Fox's ancient program. They called it Shellshock, and they warned it could allow hackers to wreak havoc on the modern internet.

Shellshock is one of the oldest known and unpatched bugs in the history of computing. But its story isn't that unusual. Earlier this year, researchers discovered another massive internet bug, called Heartbleed, that had also languished in open source software for years. Both bugs are indicative of a problem that could continue to plague the internet unless we revamp the way we write and audit software. Because the net is built on software that gets endlessly used and reused, it's littered with code that dates back decades, and some of it never gets audited for security bugs.

When Bash was built, no one thought to audit it for internet attacks because that didn't really make sense. "Worrying about this being one of the most [used] pieces of software on the planet and then having malicious people attack it was just not a possibility," Fox says. "By the time it became a possibility, it had been in use for 15 years." Today, it's used by Google and Facebook and every other big name on the internet, and because the code is open source, any of them can audit it at any time. In fact, anyone on earth can audit it at anytime. But no one thought to. And that needs to change.

How the Web Was Built

In digital terms, Fox's Bash program was about the same size as, say, a photograph snapped with your iPhone. But back in 1987, he couldn't email it across the country. The internet was only just getting off the ground. There was no world wide web, and the most efficient way to move that much data across the country was to put it in the trunk of a car.

More on Shellshock:Hackers Are Already Using the Shellshock Bug to Launch Botnet AttacksThe Internet Braces for the Crazy Shellshock WormBash is a shell utility, a black-boxy way of interfacing with an operating system that predates the graphical user interface. If you've used Microsoft's Windows command prompt, you get the idea. That may seem like an archaic thing, but as the internet took off, fueled by web browsers and the Apache web server, the Bash shell became a simple yet powerful way for engineers to glue web software to the operating system. Want your web server to get information from the computer's files? Make it pop up a bash shell and run a series of commands. That's how the web was built---script by script.

Today, Bash is still an important part of the toolkit that helps power the web. It's on the Mac, and virtually any company that runs the Linux operating system, the descendant of UNIX, uses it as a quick and easy way to connect computer programs---web server software, for example---with the underlying operating system.

But the lead developer of the program doesn't work for any of these big names. He doesn't even work for a tech company. His name is Chet Ramey, and he's a coder at Case Western Reserve University in Cleveland. He works on Bash in his spare time.

'Quite a Long Time'

In the late 1980s, Ramey took over from Brian Fox as the lead developer of Bash, and this September 12, he received an email from a security researcher named Stephane Chazelas that identified the Shellshock bug. It was a serious security vulnerability that the world learned about last week. Within hours, hackers had released code that could take over vulnerable machines and turn them into a malicious botnet.

Chet Ramey in an undated family photo.

Chet Ramey

Ramey doesn't have access to the project's source code revision logs dating back to the early '90s, but he thinks that he probably wrote the buggy code himself, sometime around 1992. That would make it the oldest, significant-yet-unpatched bug what we've heard of here at WIRED. We checked with someone who would know---Purdue University Professor Eugene Spafford---and he couldn't top it. "I can't recall any others that were [unpatched] quite as long as this," he says. "There are undoubtedly a number that have been out there longer, but the combination of age and potential impact would not be as large."

But it's a situation that feels eerily familiar to people familiar with Heartbleed, which was discovered in an widely used open-source project called OpenSSL.1 Like the OpenSSL software, Bash has never had a full-blown security audit, and it's developed by a skeleton crew with virtually no financial support. That, unfortunately, is the story of the internet.

The Lie of 'Many Eyes'

For Robert Graham, the CEO of consultancy Errata Security, Shellshock gives lie to a major tenet of open-source software: that open-source code permits "many eyes" to view and then fix bugs more quickly than proprietary software, where the code is kept out of view from most of the world. It's an idea known as Linus's Law. "If many eyes had been looking at bash over the past 25 years, these bugs would've been found a long time ago," Graham wrote on his blog last week.

Linus Torvalds---the guy that Linus's Law is named after and the guy who created the Linux operating system---says that the idea still stands. But the fallacy is the idea that all open-source projects have many eyes. "[T]here's a lot of code that doesn't actually get very many eyes at all," he says. "And a lot of open-source projects don't actually have all that many developers involved, even when they are fairly core."

This kind of issue comes up with any software code---whether it's open source or not. After all, it's even harder to tell how many bugs like this may lurk in closed-source software such as Oracle's database. About a decade ago, Microsoft faced serious security problem because parts of its software weren't properly audited. But after the Blaster worm tore though systems running Microsoft's Windows operating system in 2003, the company made security audits a priority. Over the course of the next decade, it improved the standards of its code. Microsoft spent millions on security audits and it hired white-hat hackers, called pen testers, to test out its software. Now, the open source community is starting to do the same thing.

This May, not long after the public first learned about the Heartbleed vulnerability, the Linux Foundation amassed a $6 million war chest to shore up the security on a few widely used open source projects, including OpenSSL, OpenSSH, and the Network Time Protocol. But Bash wasn't on the list. "This was not predicted," says Jim Zemlin, the Foundation's executive director. "But certainly, my guys are reaching out to those folks to see how we can help as we speak."

That's all well and good. But the trick is to shore up the internet before the bugs are found. Hopefully, the Linux Foundation---and the Googles and the Facebooks---can do so.

Even with Shellshock, Brian Fox is still proud of the project he once drove across the country. "It's been 27 years of that software being out there before a bug was found," he says. "That's a pretty impressive ratio of usage to bugs found."

1Correction: 13:10 EDT 09/29/14 An earlier version of this story incorrectly identified OpenSSH as the source of the Heartbleed bug. The name of the project is OpenSSL.