Fulfilling the world’s surging demand for AI chatbots and image generators depends on mundane components as much as glitzy GPUs. Server cabinets, heavy-duty metal enclosures to store GPU systems, have at times been a crucial bottleneck.
As it scrambled to build up its facilities, CoreWeave once ordered 1,400 of the wrong cabinets. It was a costly mistake because supply chain backups have delayed new shipments by months. “You’re moving so fast, and somewhere along the line a process fails—and you don’t realize until you have 17 tractor trailers with cabinets outside the door and you have to turn them all away,” Venturo says.
But in an example of the shrewdness that’s been key to rapid expansion, Venturo’s team in that crisis quickly put aside frustration and decided to buy used cabinets off what he called the gray market. The move prevented a significant delay. “This was just one of the instances of challenges we faced and overcame to make sure we delivered for our partners,” he says.
To keep things moving, CoreWeave has turned to the “gray market” for more than just cabinets. It’s bought networking switches and routers from eBay to sidestep waits as long as two years for new gear, a former employee says. The security and reliability of used parts can be questionable, but amid the urgency of the AI boom, some conventional practices had to be brushed aside, the person says.
In Plano, CoreWeave last year outfitted four 1-megawatt data center halls in under three days each, a feat that normally takes weeks. “We can go in with the gloves off and build really quickly,” Venturo says.
CoreWeave was forced into another creative solution when an internet provider was slow to install a broadband connection at a new site—a problem familiar to many home internet users. Three senior executives met by phone one morning to decide how to avoid delaying the project. All happened to offer the same fix: Buy satellite internet through the convenient but not cheap Starlink service of Elon Musk’s SpaceX until the fiber provider showed up. It eliminated weeks of potential delay. “We have to be incredibly flexible,” Venturo says.
Helter Skelter
Lessons from early projects inform CoreWeave’s standard procedures today. CoreWeave has opted to pay a premium for custom manufacturing of tens of thousands of fiber-optic cables because its special design means installation takes just an hour, instead of 10. After US customs authorities held up one vital shipment of important equipment, CoreWeave immediately began pushing orders through multiple alternate ports. To avoid shortages, the company now orders far more parts than it needs, betting that the leftovers can be shipped to the next project that comes along.
Haste has sometimes had unwanted consequences. CoreWeave’s data center in Las Vegas still smells like burning plastic, one source says, because it blew out some electrical components when it fired up too many GPUs at once when the site was set up several years ago.
At the core of CoreWeave’s operations are its data center technicians. The most skilled of them operate like a special operations unit, flying from site to site to get new data centers fired up instead of working at one campus full-time. Venturo declines to say how many miles his crack squad of technicians have logged, but he says they installed about 6,000 miles of fiber-optic cabling last year. “I probably have the most interaction with that team, more than any other team in the company, just because they are so incredibly important,” he says.