Colossus Data Center Compute Hall Elon Musk’s big-picture vision for various tech sectors has now focused on artificial intelligence (AI) with xAI, a company created for the purpose of AI development. At the center of this effort is Colossus, one of the world’s most powerful supercomputers that could radically redefine the capabilities of AI. The creation of Colossus marks a key achievement, not only for Musk’s xAI, but also for the AI community at large, which wants to play a leading role in the technology's adoption. Origins and Vision Behind xAI xAI was officially established in mid-2023 by Musk, CEO of Tesla and SpaceX, with the aim to “discover what the real world is like." According to its mission statement, "xAI is a company working on building artificial intelligence to accelerate human scientific discovery. We are guided by our mission to advance our collective understanding of the universe." According to Musk, he founded the company because he began to worry about the dangers of unregulated AI. xAI has the stated goal of using AI for scientific discovery but in a manner that is not exploitative. The xAI supercomputer is designed to drive cutting-edge AI research, from machine learning to neural networks with a plan to use Colossus to train large language models (like OpenAI’s GPT-series) and extend the framework into areas including autonomous machines, robotics and scientific simulations. Colossus Colossus was launched in September 2024 in Memphis, Tennessee. The data center is located at a former Electrolux manufacturing site (photo here) in a South Memphis industrial park. The Tennessee Valley Authority has approved an arrangement to furnish greater than 100 megawatts of power to the site. The Colossus system started with 100,000 Nvidia H100 GPUs, which made it one of the world’s most significant AI training platforms. The deployment of these GPUs, in 19 days, underscored xAI’s focus on scaling its AI infrastructure quickly. Consider that configuring such an extensive infrastructure usually takes months, even years to deploy, the deployment itself drew significant attention from the media and the data center/AI industry. This initial setup of 100,000 GPUs allowed it to hit a high-level of processing, making xAI capable of tackling highly complex AI models at cutting-edge speeds. This speed and efficiency are essential given the ever-increasing complexity and size of contemporary AI models, which need to be fed massive datasets and use enormous computational power. Very much a “if you build it, they will come” model, the LLM designs focused on making use of the processing power available. Expansion Plans and Upgrades In November 2024, xAI announced it would double the capacity of Colossus through a multibillion-dollar deal. The firm plans to raise $6 billion in the coming years, with the bulk of it coming from Middle Eastern sovereign wealth funds. It will cover the cost of adding 100,000 more GPUs to the existing set, bringing it to 200,000. The planned upgrade would add Nvidia’s new Blackwell H200 GPUs that are even more powerful than the H100 GPUs originally shipped. NVIDIA Hits a Snag The H200 GPUs provide considerable improvements in performance and efficiency, and will allow xAI to train AI models faster and more accurately. These GPUs are optimized for performing deep learning and neural network training, thus making them a perfect fit for xAI’s larger AI projects. According to Nvidia, the Blackwell GPUs can be up to 20 times faster than their previous generation GPUs, depending on the workload. However, the delivery of the Blackwell GPU to customers has hit a snag. The next generation chip delivery to customers had already been delayed by a quarter, due to Nvidia discovering and fixing some design flaws. A new delay has cropped up as it has been reported that the 72 GPU configuration was overheating in Nvidia’s custom-designed server racks. Yahoo Finance reported that the announcement of the problem caused an almost 3% drop in the value of Nvidia stock, even though a potential delay in this 2025 delivery of the GB200 had not been confirmed, nor was Nvidia willing to comment on whether the final design for the server racks been finalized. This larger Colossus infrastructure will make it much easier for xAI to build and test its AI models (particularly the Grok LLMs). They are meant to challenge, and possibly even exceed, currently dominant AI systems such as OpenAI’s GPT-4 and Google’s Bard. Designed for AI Colossus is different from other supercomputers not simply for its underlying computing power but also for its tailor-made AI infrastructure. The system is built to manage the special needs of AI training — handling massive amounts of data and running highly advanced algorithms that must be parallelized. As widely reported, both Dell Technologies and Supermicro partnered with xAI to build the supercomputer. The combination of Nvidia’s H100 and H200 GPUs will put Colossus at a distinct advantage when it comes to speed and efficiency. These GPUs also feature dedicated tensor cores that help accelerate deep learning algorithms. Additionally, the memory bandwidth of these GPUs is powerful enough to efficiently handle the big data sets needed to train the latest AI models. The primary building block of Colossus is the Supermicro 4U Universal GPU Liquid Cooled system. Each 4U server is equipped with eight NVIDIA H100 Tensor Core GPUs, providing substantial computational power for AI training tasks. The servers are organized into racks, with each rack containing eight 4U servers, totaling 64 GPUs per rack. Between each 4U server is a manifold for the liquid cooling, taking up 1U of rack space, and the base of each rack contains a 4U CDU pumping system providing redundant cooling, and management unit. Ethernet Networking The servers are interconnected using NVIDIA's Spectrum-X Ethernet networking platform, enabling high-bandwidth, low-latency communication essential for AI training. Each server is equipped with multiple 400GbE connections, running on 800 GBE capable cabling, rather than the Infiniband option also supported by Nvidia for large scale deployments. In the current architecture, each GPU in a cluster gets a dedicated 400 GB network interface card, with an additional 400 GBE NIC dedicated to the server, for a potential total bandwith of 3.6 TB per server. There are 512 GPUs per array (8 racks of 64 GPUs) and almost 200 total arrays. In October, NVIDIA head Jensen Huang announced that the entire initial 100,000 GPU supercomputer, was set up in only 19 days, comparing that to what he described as the normal four-year build process an average data center would require. Roadmap