How Cisco Got Bullwhipped

In the late 1990s, Cisco sat at the heart of the internet boom, selling networking equipment like routers and switches. To demonstrate the internet's revolutionary potential, Cisco made itself into the nimble, fast-moving corporation of the future. A highly praised e-commerce website, rapid daily financial reports, and sophisticated data sharing with outsourcing manufacturers, all made possible with the web. So as the telecom bubble began to implode, Cisco management felt confident enough to double down. They got bullwhipped. In this video, we look at how Cisco misread the signals. This video is brought to you by the Asianometry Patreon. Cisco Systems' origin story is famous. Leonard and Sandra first met as master students at Stanford. They were kind of opposites. Sandy aggressive, while Len rather non-confrontational, but enjoyed each other's sense of humor and married. After graduating, Sandy became Director of Computer Facilities at the Stanford Business School and Len Director at the Computer Science Department. After some time, the two grew frustrated by a computer communication issue. The early 1980s saw a Cambrian explosion of new and different mini-computers and PCs. Great for competition and variety, perhaps not so great for intercommunications. The Stanford campus in 1982 had 5,000 computers operating within self-contained networks. These devices can easily communicate with others within their own network, but passing data between networks was not so easy. In January 1980, a Stanford employee named Bill Yeager, working in networking, was asked by his boss to produce something. They wanted to connect together the various devices in the networks of the computer CS and EE departments as well as the medical center. It was a mess of incompatible machines like DEC10s, DEC VAXs, TI Explorers, Symbolic Systems, Xerox PARC Lisp machines, and Alto file servers. In an interview for Network World, Igor recalls wiring up these computers and putting together a special, quote, network operating system, end quote, with some routing code running on a DEC PDP-1105. Within six months, a workable demo of the router was installed in Pine Hall, midway between the CS department and the medical center. Igor later added IP and other protocols to turn it into the first multi-protocol router. Later, he shifted the hardware from the memory-constrained PDP to a beefier Motorola 68000 circuit board. This was provided to him by a doctoral student named Andy Bechtolsheim, who would later use this very type board to found Sun Microsystems. The product did its job, and by 1983 it was widely adopted across campus. Len and Sandy realized that this was a good product to build a company around. In December 1984, they founded Cisco Systems and later acquired the router technology and its code for commercialization. Cisco began as a small, scrappy business, making a few million dollars a year and employing relatively few people. Len and Sandy worked 100 hour weeks, mortgaging their house, maxing out their credit cards, and even taking a second job to fund this business. At first, Cisco sold its routers to universities, research centers, and government facilities. They worked closely with their customers, even letting them tinker with the software's source code. Cisco's routers added more protocols and data transmission standards, becoming a crucial bridge between various networks. Despite the fast growth, however, problems emerged. Cisco was turning high profits but somehow remained chronically short in cash and Sandy's credit cards could no longer make up the difference. At the same time, competition was emerging. So Sandy and Len decided to raise venture capital. After 75 plus rejections, Don Valentine, the legendary general partner of Sequoia Capital, agreed in 1987 to invest $2.5 million for a 30% stake, as well as the final right to make management decisions. Valentine became Cisco's chairman, professionalizing its procedures, bringing in new management like CEO John Morgridge, and changing the target customer from universities to corporations. The business immediately became successful. Valentine said that "...customers were tearing the hinges off the door to get the products." The company didn't even need the $2.5 million in VC money it got from him. But the founders and new managers clashed. The friction eventually led to Sandy and Len leaving and selling their 60-plus percent stake soon after the company's IPO in 1990. They then reportedly gave most of the $170 million they earned to charity. The company matured at the right time. The 1980s saw the opening up of NSFnet, the immediate predecessor to the modern internet. Built in the hub-and-spoke structure of feeding networks connected to a central backbone, NSFnet encourages the creation of many new networks. As noted by the Computer History Museum, between 1986 and 1987 the number of Internet hosts across the world surged from 2,000 to nearly 30,000. At the same time, companies were buying PCs for workers and networking them together. They desperately needed a way to inter-network all of these local area networks so their employees can collaborate. Cisco wasn't the only company with a router-type product. Others included Proteon, Synoptics, Banyan, Bridge Communications, and 3Com. But Cisco's routers did the best job of moving between all these competing network protocols like DECnet or Xerox's XNS or Simple Internet Protocol or AppleTalk or Novell IPX. Their routers also had the Inter-Network Operating System, or iOS software. Based on Jaeger's original router code, iOS was smarter than a simple bridge, which does not scale, in that it routed traffic in such a way that sub-networks can remain distinct but still connected to the wider world. The company placed iOS at the heart of its proprietary lock-in strategy, licensing it to other companies building networking products or services. Another major sales multiplier was the Cisco Partner Program. This recruited third-party distributors and resellers to sell and support Cisco's routers. The program amplified the reach of Cisco's sales network and let it grow far faster than it could with only a direct sales force. Sales grew rapidly throughout the early 1990s, encouraged along by smart, timely acquisitions like the switch company Crescendo in 1993. In 1991, they made $183 million in revenue. Three years later in 1994, the company cleared $1.2 billion in revenue. Just in case you're wondering, like I was, a switch connects devices within segments of a single network. A router forwards data packets along networks. The two product categories contributed about two-thirds of Cisco's revenues. Anyway, the next year, John Chambers, a former IBM and Wang Labs executive, took over as CEO. He then goes on to aggressively acquire and integrate literally dozens of startups to build an end-to-end networking empire. Between 1993 and 2000, Cisco bought a staggering 60 startups, paying mostly in its own stock as the price soared into the stratosphere. Considering Cisco's central role in the internet boom, the company strove to present itself as a leader in e-business. E-business back then meant using the internet to conduct front and back-end business processes. John Chambers believed that the internet would change how people live, work, and play. Not wrong. And he wanted his company to be at the forefront of that. Cisco was an early adopter of online purchasing. They sold products to their thousands of sales partners through their official website titled Cisco Connection Online. The site began offering simple online transactions and customer support all the way back in 1994. Then under Chief Information Officer Peter Solvik, the site was greatly expanded. Thanks to the power of the internet, customers can receive real-time quotes, configure their orders, and be alerted if their order has any errors. New orders are routed to the right salesperson over the internet, saving the company a reported half a million dollars a year in FedEx delivery charges. Running on Sun Hardware and Solaris, the new website showed up in 1996 and generated $75 million of sales within its first five months. By 1998, over 65% of Cisco's sales came in through Customers loved it, and it won awards from CIO Magazine in 1996, 1997, and 1998. Mortgage and Chambers both believed in sticking to your core competencies. And to them, Cisco's core competencies lay in corporate strategy, engineering, design, and customer service. They did not believe that Cisco can manufacture better than other companies So starting in 1992, but continuing steadily throughout the 1990s, Cisco outsourced the production of its products to third-party assemblers like Jabil and Selectron. Jabil Circuit is one of the OG assemblers. Founded by Bill Morian and James Golden in 1966, the company started off in Detroit fixing circuit board assemblies for control data systems. In 1979, they struck a turnkey service partnership with General Motors that changed their whole future. They then moved to Florida, where they picked up a service contract with IBM to assemble the IBM PC. They then expanded their services to include product design and engineering and went public in 1993. Since then, they have grown to become one of the country's largest assemblers. The other major manufacturing partner is Selectron. Founded in 1977 by Roy Kusumoto and two Indian Americans as a small assembly shop, the firm found its initial edge adopting total quality management and cutting-edge manufacturing like surface mounting technology. Assembly shops and contract manufacturers have been around in the Silicon Valley for years. But until the 1990s, they were largely used as overflow capacity. This was not a pleasant role because those orders can get cut for any reason, leaving the assembler in for a bad time. Then in the late 1980s, former IBMer Dr. Koichi Nishimura joined as COO. Nishimura, a second-generation Japanese-American who was interned at Manzanar during World War II, transformed what was still a local assembly shop into a globalized giant. Cisco became a major customer of both Jabil and Selectron. By the late 1990s, some 70-80% of Cisco's production was outsourced. It saved them an estimated 30% on their costs and stood alongside their control of the iOS software, and the sales partner program as their key competitive advantages. It also let them brag that they had the best supply chain in the industry. Cisco built an information network, called an extranet, that let them right away communicate orders and specifications to their various third-party manufacturers. Then, once the products are built, they can be shipped directly to the customers. If done right, a Cisco employee never touches the product. Carl Redfield, Cisco's SVP of manufacturing, said, We've cleaned up the supply chain. You don't add value by having multiple people touch a product. Authors described it as like a Japanese keiretsu, a family of companies that are ostensibly independent of Cisco, but treated as if under their roof. The extranet, like that rug in the Big Lebowski, really tied it all together. Cisco Vice President Paul Mountford told CFO Magazine, "...our suppliers can come straight through and look at our order book, and then manufacture to the appropriate level." It also gave management what seemed to be an unprecedented real-time view into demand. Executives can generate financial reports on the fly, showing the company's total worldwide sales for the day, and cut by region, product, or customer. It let them perform daily tactical operating movements. And then you got the virtual close, a big part of their e-business aura. CFO Larry Carter would humblebrag in the Harvard Business Review that they can close their quarterly accounting books in hours, a process that took an ordinary company weeks. Anyway, John Chambers called what they had created a virtual supply chain, endowed with unlimited capacity, outputting extraordinary reliability, and enabling incredible agility. It seemed the perfect asset as the market began to enter a wild boom. The Telecoms Act of 1996 remade the telecom market, opening it up to competition. Also, in 1996 and 1997, companies like Amazon and Yahoo IPO'd and their stock soared on the first day. Their quote unquote successes seemingly demonstrating the viability of e-commerce, email, and other e-business. Over 300 dot coms stampeded to go public as well. Driven by the belief that the internet was going to change everything, and myths like that internet traffic was doubling every 100 days, these telecom companies worked themselves up into a bubble of their own. Entrants like Level 3, Williams, and Quest raised staggering amounts of money from the stock and bond markets to build large-scale internet infrastructure. In 1995, the year before the new law, the telecom industry put in $47 billion of capex. By the peak year 2000, the SF Federal Reserve estimated that the telecom capex investment reached about $120 billion, or about $233 billion today. The telecom bubble deserves more detail, so let me save that for a future video. To bring things back to Cisco, these new telecom firms most often bought their networking gear from Cisco or its two major competitors Lucent and Nortel. Lucent and Nortel were larger as measured by revenue, but it was Cisco that became the business world's darling. The staggering sales and profit growth. The fresh startup smell. The e-business savvy. If you weren't trying to be like Cisco, then you were falling behind. In 1998, Cisco's market cap reached $100 billion. They achieved that milestone in just eight years after their IPO. Seven years faster than it took Mr. Softee. Revenues grew at 50% each year from $8.5 billion in 1998 to $18.9 billion in 2000. To keep up with the rapidly changing market, the company bought even more startups. 17 purchases in 1999 alone, including shelling out an ankle-breaking $7 billion to snag a fiber optic startup called Sarent, which turned only $15 million in revenue. Optics valuations in the bubble era were as hilarious as a donkey eating your figs. Just kills me. The frenzy pays off, though, with a daring transition. Up until then, most telecoms carried internet data over voice circuits. Cisco feels that this is not the future. Rather, their vision is a packet-based network that treats data, voice, and video signals all the same. In 1998, Cisco declares their intention to build such packet-switched data networks, including to send those signals over optic fiber. This puts Lucent and Nortel on their back foot, and they struggle to make a similar transition. Lucent tried to bolt it onto their legacy circuit equipment, while Nortel spent $9 billion to buy Bay Networks. Neither strategy really works. This leaves Cisco dominant in the lucrative internet data networking field, the clear future of the industry. As the fiber bubble drove on over the next two years, Cisco was the clear beneficiary. SoElectron benefited from the bubble frenzy too. In 1999, they became the world's largest contract manufacturer, with over $8 billion in sales. Thanks in part to the strategy of acquiring their customers' factories, revenues grew over 50% a year from 1992 to 1999. The business press labeled SoElectron the quote-unquote Intel of contract manufacturing, and at the time, that was considered high praise. The dot-com bubble was of course a bubble, but primarily one in the stock markets. Most sensible people recognized that things there were crazy. Telecoms on the other hand seemed different because they were laying down fiber lines, hard assets. And for decades, telecom meant AT&T, plain old telephone service that is boring and thus safe to plunge your nest egg into. But that was in the before times. People were slow to catch on to the fact that the industry really had changed. This wasn't your grandpa's monopoly anymore. Moreover, few noticed that the fiber everyone was building out was duplicative. Level 3 and Quest and their peers focused on building out the easy stuff. Long distance fiber loops cutting through rural areas. Far fewer dollars were being spent on the last mile connections actually going into people's homes. Because unfortunately, that was the hardest part. For every dollar spent laying down long distance cable, $20 must be spent on last mile. There were also the shady vendor financing practices. And if you've read an Nvidia bear report recently, then you might have heard of these. Building infrastructure takes a lot of upfront capital expenditure that can only be paid back over a long period of time. These telecom startups lacked that kind of cash flow, so they turned to their own vendors. Vendors can help startups get off the ground by directly investing in said startup, accepting stock in exchange for product, or loaning money that they can then turn around to buy product. Cisco was actually one of the better actors in this. By late 2000, Cisco had only $2.4 billion committed, of which $625 million had been drawn. To compare, Nortel showed about $3 billion in vendor financing commitments and Lucent a staggering $8.1 billion in total. One super fun example was WinStar Communications. Between 1995 and 2000, they raised billions to build the quote, world's first global end-to-end broadband network, end quote. This included a very unusual $500 million from Lucent in October 1998, funding a staggering 40 plus percent of the price of Lucent equipment. Traditional norms number 10 to 15 percent at the most. Some tried to get the word out. Andrew Odlyzko of AT&T Labs Published a paper in late 1998 debunking the internet 100-day traffic doubling factoid, which had by then become fundraising gospel. A 1999 issue, the McKinsey Quarterly noted that if all the fiber planned by the U.S. telcos were to be fully utilized, then U.S. backbone capacity would rise by 200 times over the next three to five years. They noted significant oversupply then, but added that there was no technological reason why that backbone wouldn't be fully utilized. In March 2000, Cisco became the world's largest company with a market cap of $531 billion. The stock had tripled over the past year. An analyst predicted that the company would be the first to ever make the trillion-dollar market cap number driven by the internet tidal wave. In May 2000, Fortune magazine called John Chambers a money-making machine and mooted him as the world's greatest CEO alongside Jack Welch, Lou Gerstner, and Sony's Nobuyuki Ide. By then, the dot-com bubble had already begun to burst. Yet even as the dot-coms made their merry little way to the slaughterhouse, the telecom bubble continued on. And in some aspects, it even intensified. For instance, in July 2000, Nortel reported higher profits and, thanks to all the optical gear customers were storming to buy, even raised their sales expectations for the rest of the year from 35% growth to 40% growth. This final burst of energy created shortages. In early 2000, Cisco started noticing that lead times on some of their networking equipment had grown to four to six months. Such shortages were estimated to have cost the company 10% of revenue last year. Trusting their numbers and forecasting 60% revenue growth for the forthcoming fiscal year starting in July 2000, Cisco went aggressive. They moved forward on long-term commitments with its manufacturing partners and component suppliers. Selectron now faced a dilemma. Its own internal forecasts looked at the worsening general economy and preached caution. They warned Cisco that they were buying more communications chips, optical lasers, and assembly boards than what seemed necessary. Cisco replied that their business would keep growing. Dot-coms were less than 5% of their business. Some dot-coms still had money and were still going to buy. So Cisco was still going to need those extra components. The Wall Street Journal reported Chambers saying, "...to not plan to meet that growth is the quickest way to lose customers." You can see these starting to appear on Cisco's financial statements. Their year 2010-K, covering the year ended July 2000, showed inventories rising 87% from the previous year due to, "...increased purchases to secure the supply of certain components." The following 10Q statement, covering the quarter ending October 2000, showed Cisco's inventories rising another 58.8%, again in part because of further component purchases. So Cisco wasn't backing off. They were doubling down. And to be fair, the signs were mixed, especially in that final quarter of 2000. In that quarter I mentioned earlier, ending in October, Cisco reported record revenue growth of 65%. Their executives insisted that they had not seen any sign of a slowdown. But the stock was down 32% from its March peak. And Wall Street analysts were now openly saying that the telecom boom was slowing. The telecom giants were pulling back on their equipment purchases. And what was especially worrisome was that the telecom startups were losing the ability to raise cash to finish their networks. The first eight months of the year, telecoms raised $5.6 billion a month selling stocks and bonds. In September, that fell to just $1.2 billion. October, $900 million. November, $700 million. Some of those startups were going bankrupt. A notable one was One was ICG Communications in Colorado, which owed Cisco $32 million for equipment. Ominously, Cisco's other partners, not just Selectron, were revising forecasts. A notable one was Xilinx, a maker of FPGA silicon chips that go into routers. In September 2000, they cut their internal growth forecast from 16% to 20% to just 10%. Other partners warned Cisco that shipments were slowing. But just to give you a sense how hard predicting the future is, signals were very mixed. Morgan Stanley's Christopher Sticks reported that his own channel check still found wait times of three months for Cisco gear. And tens of millions of people were still joining the internet every month. So even if telecom sales are indeed slowing, sales in corporate networking and database internet switches appear to have so much runway ahead. The Wall Street Journal later reported that Cisco's management didn't really look at the overall economy or the cash positions of their telecom customers. They of course noticed the downturn, but framed it as a consolidation that would produce stronger telecom customers in the end. So in the moment, two or three random negative signals did not seem a big deal. Twice before, Cisco's management hit the gas rather than pulling back. Like during the Asian financial crisis. And twice before, the gamble paid off. The company had grown for literally dozens of quarters. A slowdown did not seem feasible. On December 4th, John Chambers told analysts, "...I have never been more optimistic about the future of our industry as a whole or of Cisco." End quote. About a week later, in mid-December 2000, he ran his daily financial report and noticed that the day's sales had fallen short of projections. After the trend persisted for a week, he got worried enough to start calling customers. So what was happening? Earlier in the year, Cisco customers, facing months-long delays in getting product, started putting down double or even triple orders to get the quantities they needed. Cisco salespeople sometimes added their own buffer to the numbers, hoping to get attention. Sometimes customers went to multiple resellers and suppliers, putting orders in with them too. The result was an exaggerated demand signal that Cisco took to heart. They faithfully transmitted that to the rest of the supply chain and put in big component orders to meet them. When things started to go south later in the year, all those orders got cancelled. Buffers that the company thought it had, like a massive $3.8 billion sales backlog in late September, equivalent to 8-10 weeks of sales, evaporated like raindrops in Death Valley. The company and wider market initially held to some hope that things would come back. News articles mentioning component shortages for certain switches impeding sales were still coming out in early January 2001. But in mid-January 2001, Chambers admitted that the quarter was more challenging than anticipated due to a slowdown in customer spending. Cisco later reported strong revenue growth numbers but missed its earning numbers. The company tried to keep a cheery tone. They emphasized a positive 3-5 year outlook, because many businesses still have to modernize to leverage more internet. But in the end, they had to face the music. Two months later, in April 2001, Cisco took responsibility for its bad forecast. They admitted that sales will fall 30% from the prior quarter, cut 8,500 jobs, and took an estimated $2.5 billion financial charge on unused inventory held by Jabal and Selectron, including stuff they committed to even if there wasn't a contract. The actual number turned out to be about $2.2 billion. They sold some of the components, others they warehoused in the hope of the market coming back. When it didn't, the inventory had to be destroyed because it had been custom made for Cisco. Cisco's rapid buildup and then abrupt cut reverberated through its supply chain partners. In March 2001, Cisco announced that its revenues had surged 85% with profits meeting Wall Street's targets. Yet at the same time, they were cutting 8,200 jobs. And I did a video about JDS Uniphase, ancestor to today's Lumentum, which supplied many of the optical fiber components like lasers to Cisco and others. They cut a staggering 16,000 jobs in 2001 alone. And a stock price that had once been over $130 in 2000 spiraled to $7 a year later. In an oral history for the Computer History Museum, John Morgridge, the former CEO who then stepped up to chairman, recalled seeing the internal trends data deteriorating in the autumn of 2000. But Chambers was by nature an optimistic guy. Faced with these numbers, he decided to hit the pedal just as he did before. And in the end, the downturn turned out to be far worse than almost everyone anticipated. Dips happen all the time. By 2001, the telecom bubble was special. So Cisco got bullwhipped. The bullwhip effect is, formally speaking, a supply chain phenomenon where shifts in retail amplify in size the deeper you go down the chain. I first mentioned it in a prior video about TSMC and Silicon Valley. The 2000 Cisco inventory write-off is a particularly pure form of the effect. A short period of shortages triggered excessive inventory buildup within the supply chain. Layers in that chain amplified those demand signals. Cisco's management then took those signals to heart, overriding the instincts of their manufacturing partners, who are a bit more conservative. Fortunately for Cisco, the company had the money to weather the storm. They took on their partner's financial obligations and in doing so were able to keep the supply chain intact. And that helped them survive and scale up for when the downturn finally ended. Unfortunately, not everyone in that industry had the money to do what Cisco did. Alright everyone, that's it for tonight. Thanks for watching. Subscribe to the channel, sign up for the Patreon, and I'll see you guys next time.