Amazon is set to roll out its latest artificial intelligence chips as the Big Tech group seeks to recoup its multibillion-dollar investments in semiconductors and reduce its dependence on market leader Nvidia.
Executives at Amazon’s cloud computing division are spending big on custom chips in hopes of improving the efficiency of its dozens of data centers, thereby reducing its own costs as well as those of Amazon Web Services‘ customers.
The effort is led by Annapurna Labs, an Israeli chip startup that Amazon acquired in early 2015 for $350 million and which operates in Austin.
Annapurna’s latest work is expected to be showcased next month when Amazon announces the widespread availability of Trainium 2, part of a line of AI devices. chips intended to train the largest models.
Trainium 2 is already being tested by Anthropic – OpenAI’s competitor which has secured $4 billion in backing from Amazon – as well as Databricks, Deutsche Telekom and Japan’s Ricoh and Stockmark.
AWS and Annapurna’s goal is to take on Nvidia, one of the most valuable companies in the world thanks to its dominance of the AI processor market.
“We want to be absolutely the best place to run Nvidia,” said Dave Brown, vice president of compute and networking services at AWS. “But at the same time, we think it’s healthy to have an alternative.” Amazon said Inferentia, another of its specialized AI chip lines, was already 40% cheaper to use to generate responses from AI models.
“The price (of cloud computing) tends to be much higher when it comes to machine learning and AI,” Brown said. “When you save 40 percent of $1,000, it won’t really affect your choice. But when you save 40 percent on tens of millions of dollars, it actually does.
Amazon now plans about $75 billion in capital spending in 2024, with the majority going toward technology infrastructure. During the company’s latest earnings conference call, Chief Executive Officer Andy Jassy said he expected the company to spend even more in 2025.
That’s an increase from 2023, when it spent $48.4 billion for the entire year. The biggest cloud providers, including Microsoft and Google, are all engaged in an AI spending spree that shows no signs of slowing down.
Amazon, Microsoft and Meta are all big customers of Nvidia, but are also designing their own data center chips to lay the groundwork for what they hope will be a wave of AI growth.
“Each of the big cloud providers is moving feverishly toward a more verticalized and, if possible, homogenized and integrated (chip technology) stack,” said Daniel Newman of the Futurum Group.
“Everyone from OpenAI to Apple is looking to build their own chips,” Newman noted, as they sought “lower production costs, higher margins, greater availability and more control” .
“It’s not (just) about the chip, it’s about the complete system,” said Rami Sinno, Annapurna’s director of engineering and a veteran of SoftBank’s Arm and Intel.
For Amazon’s AI infrastructure, that means building everything from scratch, from the silicon wafer to the server racks they fit into, all supported by Amazon’s proprietary architecture and software. ‘Amazon. “It’s really difficult to do what we do at scale. Not many companies can,” Sinno said.
After starting to build a security chip for AWS called Nitro, Annapurna has since developed several generations of Graviton, its Arm-based central processing units that offer a low-power alternative to traditional servers supplied by Intel or AMD.
“The big advantage of AWS is that its chips can use less power and its data centers can maybe be a little more efficient,” which reduces costs, said G Dan Hutcheson, an analyst at TechInsights. If Nvidia’s graphics processing units were powerful general-purpose tools — in automotive terms, like a station wagon or station wagon — Amazon could optimize its chips for specific tasks and services, like a compact or sedan, he said. -he declared.
However, AWS and Annapurna have barely dented Nvidia’s dominance in AI infrastructure.
Nvidia reported $26.3 billion in revenue for AI data center chip sales during its fiscal second quarter 2024. That figure is the same as Amazon reported for its entire AWS division during of its own fiscal second quarter – of which only a relatively small fraction can be attributed. to customers running AI workloads on Annapurna infrastructure, according to Hutcheson.
As for the raw performance of AWS chips versus Nvidia’s, Amazon avoids making direct comparisons and does not subject its chips to independent performance testing.
“Benchmarks are good for that first question: ‘Hey, should I even consider this chip?’ “, said Patrick Moorhead, chip consultant at Moor Insights & Strategy, but the real test came when they were placed “in multiple racks together in a fleet. “.
Moorhead said he was confident that Amazon’s claims of a four-fold increase in performance between Trainium 1 and Trainium 2 were accurate, after scrutinizing the company for years. But performance numbers may matter less than simply providing customers with more choice.
“People like all the innovation that Nvidia is bringing, but no one is comfortable with Nvidia having 90% market share,” he added. “This can’t last long.”
This article has been edited to clarify the origin and location of Annapurna Labs.