How Red Hat killed its core product—and became a billion-dollar business
By Jon Brodkin | Published 10 days ago on Ars Technica
Read the entire article on their site, better links, related articles, comments, and serendipitous reading opportunities.
I am saving the article here as I am an unabashed RedHat supporter. Disclaimer: I have taught RedHat Academy in High School and taught as a Red Hat Certified Instructor for three years at Working Connections.
A decade ago, Linux developer Red Hat faced a decision that would make or break the company: whether to stop producing the very product that gave Red Hat its name. The company was built on Red Hat Linux, but when Paul Cormier—now the head of Red Hat’s technologies and products group—joined the company as vice president of engineering in 2001, he knew Red Hat’s devotion to open source alone couldn’t create a business model capable of standing up to the Microsofts and Oracles of the world. He pushed for drastic action.
To move from small player to big-time enterprise software competitor, Cormier argued that Red Hat had to ditch the freely downloadable Red Hat Linux. Instead, it should replace Red Hat Linux with a more robust enterprise software package that maintained the principles of free (as in freedom) software without actually being free (as in price) to customers.
People within Red Hat told Cormier he was crazy. "A lot of engineers at the time didn’t care about a business model," he told Ars. "They wanted to work on Red Hat Linux. We had some level of turmoil inside the company with going to this new model. Some engineers left, but more stayed."
Cormier’s vision required a "bet the farm decision," and he won out over the doubters when he convinced then-CEO Matthew Szulik to stop producing Red Hat Linux. The last stable release of the operating system appeared in 2003 at the same time RHEL—Red Hat Enterprise Linux—hit the market. Since then, the company has been on a steady climb in revenue and profitability, aided by the growing popularity of Linux-based servers and Red Hat’s expansion into new markets.
The company now faces new challenges as it looks to expand its empire through a big bet on the virtualization market—a market so utterly dominated by VMware that Red Hat may stand little chance. Yet Red Hat’s prominence in the enterprise Linux server market ensures growing revenue and profits for some time to come.
This week, at the end of its fiscal year on February 29, Red Hat marks a major milestone: it becomes the first billion-dollar-a-year company making its revenue entirely (or almost entirely) from building and maintaining open source software.
In hindsight, Cormier’s risky idea looks prescient; to some at the time, it seemed ridiculous. How did a successful company like Red Hat manage to take such a risk and turn it in to such reward?
Software, scooters, and Nerf gun fights
Red Hat was founded in 1993 by Bob Young and Marc Ewing—with its software having been named after the hat Ewing wore around Carnegie Mellon University—but Ewing had left and Young was gradually stepping back when Cormier joined. He arrived at the same time a new release of Red Hat Linux debuted, and he vividly remembers his early conversations, including one with the engineering manager.
A Red Hat timeline
1993: Bob Young founds the ACC Corporation, Marc Ewing founds Red Hat.
1994: Ewing releases his own Linux distribution—Red Hat Linux.
1995: Young buys Ewing’s business, the two merge into Red Hat Software. Young becomes CEO.
1999: Red Hat becomes a publicly traded company. Bob Young steps down as CEO later in the year, Ewing leaves the company entirely.
2001: Paul Cormier joins Red Hat, sparking a re-examination of the company’s business model.
2003: The last stable release of Red Hat Linux debuts. Red Hat Enterprise Linux hits the market.
2005: Young leaves Red Hat’s Board of Directors, officially leaving the company.
2006: Red Hat buys JBoss.
2008: Red Hat buys Qumranet, maker of the KVM virtualization platform, laying the groundwork for Red Hat Enterprise Virtualization.
2009: Red Hat enters the virtualization market with RHEV. Existing competitors VMware, Citrix, and Microsoft already have an established market share within the field.
2011: During December earnings announcement, Red Hat reports $836 million in revenue through nine months of the fiscal year.
2012: Red Hat expects to eclipse the $1 billion mark in revenue for the fiscal year ending in February.
"I said, ‘So obviously we just shipped something, I assume you can recreate the bits, the binary-compatible bits.’ He opened up his top drawer of his desk, a bunch of CDs were inside and said, ‘I think the source is somewhere in here.’ I said, ‘Wow, we have a lot of work to do.’ It was like that, people riding around the halls in scooters and having Nerf gun fights. It was all cool. It was all good stuff."
Most of Red Hat’s revenue at the time came from telephone support, and Cormier knew the model wouldn’t scale.
"We really didn’t have a business model," he said. "We had Red Hat Linux in bookstores for $29.95 a box, the same Red Hat Linux people could download for free. We had, maybe, at the time, ten commercial customers that were primarily paying for us to talk on phone. We knew that wouldn’t scale. They were saying, ‘Why should I pay you for this?’"
Cormier recalls many late nights with top executives, hammering out a strategy to replace Red Hat Linux with Red Hat Enterprise Linux, which was ultimately sold through a subscription including updates, patches, and bug fixes. The late night talks were challenging, Cormier says, because Red Hat wanted to both satisfy open source licensing requirements and pave the way for a multi-billion dollar business.
After the transition, Red Hat didn’t completely stop giving software away. While RHEL would bring in the money, the spirit of the original Red Hat Linux lived on in Fedora, a Red Hat-sponsored community project, which was first released in 2003 and initially based on Red Hat Linux code. Fedora and RHEL have a mutually beneficial relationship, with Fedora code serving as something of a testing ground for the enterprise features delivered to Red Hat’s paying customers.
RHEL source code is freely available under the GPL (GNU General Public License) for those who want to compile it themselves, but the actual finished product costs money. Yes, there is CentOS, a free-to-download clone of RHEL compiled from the source code by CentOS developers. But Red Hat charges a premium for RHEL because it’s (theoretically) guaranteed to work—Red Hat and third-party software vendors make sure that applications running on RHEL are not broken when the operating system is updated.
"Red Hat Enterprise Linux, the bits, the ones and the zeros, are not free," Red Hat CEO Jim Whitehurst told Ars. "The source code is free and it’s freely available. But we compile those bits and we make it enterprise-class."
Most importantly for major customers, Red Hat creates a long-term support edition every two years. It commits to support this for a full decade, to the point of taking critical fixes from the Linux community and back-porting them to the older versions.
"If you’re the New York Stock Exchange, the last thing you want is rapid change in your underlying operating system," Whitehurst said.
If all Red Hat did was give away software and sell technical support, "we’d be a tiny little company that is marginally profitable," he said.
Red Hat Linux was much like today’s Fedora, releasing new versions quickly to get the bleeding-edge technology out to users. But new versions and patches could break old applications, and there was no ecosystem of software and hardware vendors supporting applications running on Red Hat. With RHEL, Red Hat gives the enterprise what it wants: a stable lifecycle and roadmap, and a more careful system for inserting patches without breaking application compatibility. That model has certainly proven its worth.
Patent trolls follow the money
With the increased cash flow, Red Hat became a target of patent lawsuits and a staunch defender of open source users. The company remains unusually outspoken in its opposition to even the notion of software patents, yet it also forged a pragmatic strategy of acquiring defensive patents. Red Hat fights lawsuits where fundamental issues are at stake and settles others that aren’t worth the legal bills.
"If you believe in the concept of modular innovation where a lot of different people add to works that came before them, patents clearly slow that down," Whitehurst said. "I don’t think we’re different from any other software company, it’s just we have to deal with a lot of patent trolls."
Red Hat CEO Jim Whitehurst
Red Hat is always busy in court, particularly in the Eastern District of Texas (well-known as a hotbed of patent lawsuits). Red Hat teamed up with longtime rival Novell to win one case against a subsidiary of "patent licensing" vendor Acacia in 2010. The pair even filed an amicus brief on behalf of Microsoft in one patent dispute.
In other cases, Red Hat simply paid up. It settled lawsuits from Acacia in April 2011 and paid $4.2 million to settle a suit brought by FireStar Software in 2008, for instance.
Among the benefits of a Red Hat subscription is indemnification against lawsuits related to using Red Hat software. Patent trolls sometimes target Red Hat customers directly, and Red Hat generally takes those cases over.
"Our customers call us and say ‘we’re being sued because we use your product,’ and we take over the defense," Whitehurst said. "We reach whatever settlement we reach with the trolls. Some of those we fight because many of these aren’t applicable patents for whatever reason, and others we ultimately settle. We generally settle not just for our customers, but for all downstream users of that code."
Despite the patent troubles, Red Hat has become one of the top names in the enterprise server market by focusing on what it does best: building great software.
"We never ever go into a customer and say, ‘There’s an application that’s not supported on RHEL," Cormier said. "If it’s supported on Windows, it’s supported on RHEL."
Daniel Vande More, chief architect for a company called Net-Results and an Ars reader who has used Red Hat for more than a decade, agrees the shift to a more stable release strategy helped make Red Hat as appealing as it is today.
"Packaging for distributions that shift often is like trying to catch a falling knife," Vande More said. "Everything from new glibc versions to simple things like MySQL version changes can affect applications dramatically and often introduce so much variability that testing all possible configurations is at odds with making good use of developer time. Once Red Hat was able to smooth out these bumps by supporting their RHEL releases for years, I believe it became a much more attractive operating system for software vendors."
Reaching $1 billion
With RHEL, Red Hat expanded and soon outstripped companies like Novell, which was sold to Attachmate in 2010 after a long, slow decline. Moving far beyond operating systems, Red Hat now sells almost any type of software a data center might need to deploy and run applications, with various middleware, storage, security, virtualization, and management products. Having reported $836 million in revenue (and net income of $111 million) for the nine months ending Nov. 30, Red Hat is set to soar past the billion-dollar mark when its fiscal year concludes at the end of February (results will be reported in mid-March).
Just four years ago, Red Hat reported $523 million in revenue at the end of its fiscal year. Assuming current trends continue, Red Hat will more than double that in Fiscal 2012. Most of its money comes from subscriptions, the rest from training and services.
"It’s just another validation in the line of many that open source software is a fundamental part of IT infrastructure," said Jim Zemlin, head of the Linux Foundation. "They have shown that this is a solid business model, that this is a better way to distribute software and it can be done profitably."
Red Hat’s growth may not stop here. It made its most significant acquisition in 2006 when it bought middleware vendor JBoss. It has continued to expand with the November 2010 purchase of platform-as-a-service provider Makara and the October 2011 buy of storage vendor Gluster. But moving up in the virtualization market will be difficult.
In virtualization, VMware maintains a "huge lead" over Red Hat
Red Hat has occasionally boasted about providing all the goodness of Microsoft’s data center offerings—without the restrictions imposed by closed-source software. But Red Hat’s top executives say their biggest rival isn’t in Redmond.
"We almost never see Microsoft," Whitehurst says. He notes Red Hat is mostly converting Unix shops to Linux, a market in which it competes against Oracle and IBM instead. Whitehurst says Red Hat’s future growth depends on providing the back-end infrastructure for public and private cloud networks, as well as software-as-a-service vendors. Here, the biggest competition is virtualization vendor VMware.
"They’re way ahead of us in virtualization, we’re way ahead of them in operating systems and middleware."
This is also where many expect Red Hat to fall short. VMware dominates the server virtualization market and its most serious threat comes from Microsoft’s Hyper-V. Citrix with the Xen open source hypervisor is third. Red Hat, along with Oracle, currently fight it out for fourth place.
Red Hat’s virtualization platform is based on KVM, which has less than one percent of today’s market share, according to Gartner analyst Chris Wolf. Red Hat’s technology is solid, but Wolf said the company will "have a challenge just staying ahead of Oracle, because Oracle has taken virtualization very seriously."
Red Hat has a brand-new release of Red Hat Enterprise Virtualization out. It adds many new features and finally drops the Windows Server-based management system for one relying entirely on open source software, namely Java, JBoss, and RHEL. But Wolf said Red Hat is far behind VMware when it comes to integrating the virtualization stack with third-party backup, security and disaster recovery products.
Whitehurst, of course, gives Red Hat a better shot.
"They’re way ahead of us in virtualization, we’re way ahead of them in operating systems and middleware," he said. And when it comes to making applications portable across data centers and cloud networks, "all three of those components are important. They have a huge lead in one and we have a huge lead in the other two."
A question of definition
Some take issue with calling Red Hat the first billion-dollar open source company. As Linux kernel maintainer Greg Kroah-Hartman notes, Google, Facebook, and Amazon were built on open source software, and IBM made billions off of Linux.
"There are lots of companies that rely entirely on open source software, and give back in various ways to the communities involved," he said. "Loads of them are above the billion-dollar mark."
Still, Red Hat is unique when you consider that it earns its money almost entirely by selling and building open source software. Red Hat is also the top corporate contributor to the Linux kernel, something Kroah-Hartman praises. "They are doing wonderful work, allowing their community members to contribute in great numbers," he said. "I’m really happy with all they have been doing."
Kroah-Hartman points out that Red Hat "does have and sell proprietary software," calling into question the nature of Red Hat as an open source-only company.
However, even when Red Hat acquires proprietary technology, it eventually converts the code to open source. A Red Hat spokesperson tells us the only Red Hat technology today that is "not fully open source" is the OpenShift platform-as-a-service software.
"OpenShift is built on open source technologies and we are in the process of setting up the licensing and governance model for that now," a company spokesperson says. The company says RHEV, based on the 2008 acquisition of Qumranet, was fully open sourced as of November 2011.
Red Hat drew some attention to its practices for distributing source code last year when it implemented a change designed to prevent Oracle from cannibalizing its RHEL business.
Oracle’s Linux distribution is based on RHEL—much as CentOS is—with Oracle trying to divert customers away from Red Hat by selling support. In retaliation, Red Hat started shipping Linux kernel source code in a big tarball with the patches already applied, making it more difficult to build Linux distributions from the RHEL source.
Kroah-Hartman claims it was a non-issue, saying it "affected the kernel developers not one bit at all." The move hasn’t prevented CentOS from making regular releases. It all fits in with Red Hat’s longtime strategy of making money without violating the terms of the GPL.
"To put it bluntly, Oracle has come in very clearly saying ‘This is a copy of Red Hat, and we are offering it at lower price than Red Hat,’" Whitehurst said. "We need to protect our business model because if we’re not successful I seriously doubt Oracle would be giving the code away the way we do to CentOS and everyone else."
Old versions of Red Hat Linux
The next Red Hat?
There’s plenty of room for another Red Hat, Whitehurst said. Enterprise software is a $180-$200 billion market, and the company competes for about $30-$40 billion of that.
"There are huge categories where we aren’t even there," he said. "Databases, for example."
Zemlin sees a lot of innovation from newer open source companies, and he singles out Hadoop vendor Cloudera and SugarCRM. Zemlin believes "there will be some more Red Hats out there," but notes the success of open source lies just as much in companies creating innovative products built on top of open source as it does with companies like Red Hat that build and sell open source software directly.
Salesforce.com (a Red Hat customer), Google, Amazon, and Facebook couldn’t have reached the heights they did without open source, Zemlin argued.
"I think there will be more companies like Red Hat, but that question limits how you characterize success using open source," he said. "You really have to look at the hundreds and hundreds of companies that achieve massive business value by taking open source and using it in different ways."