DeepSeek's new chatbot boasts an impressive introduction: "Hi, I was created so you can ask anything and get an answer that might even surprise you." This AI, a product of the Chinese startup DeepSeek, has quickly become a major market player, even contributing to a significant drop in NVIDIA's stock price. Its success stems from a unique architecture and training methodology, incorporating several innovative technologies.
Multi-token Prediction (MTP): Unlike traditional word-by-word prediction, MTP forecasts multiple words simultaneously, analyzing various sentence parts for enhanced accuracy and efficiency.
Mixture of Experts (MoE): This architecture utilizes multiple neural networks to process input data, accelerating AI training and improving performance. DeepSeek V3 employs 256 neural networks, activating eight for each token processing task.
Multi-head Latent Attention (MLA): MLA focuses on crucial sentence elements, repeatedly extracting key details from text fragments to minimize information loss and capture subtle nuances.
DeepSeek initially claimed a remarkably low training cost of $6 million for its powerful DeepSeek V3 model, using only 2048 GPUs. However, SemiAnalysis revealed a far more substantial infrastructure: approximately 50,000 Nvidia Hopper GPUs (including 10,000 H800, 10,000 H100, and additional H20 GPUs) distributed across multiple data centers. This translates to a server investment of roughly $1.6 billion and operational expenses estimated at $944 million.
DeepSeek, a subsidiary of the Chinese hedge fund High-Flyer, owns its data centers, unlike many startups that rely on cloud services. This provides greater control over optimization and faster innovation implementation. The company's self-funded nature enhances flexibility and decision-making speed. Furthermore, DeepSeek attracts top talent, with some researchers earning over $1.3 million annually, primarily recruiting from leading Chinese universities.
The initial $6 million figure, DeepSeek clarifies, only reflects pre-training GPU costs, excluding research, refinement, data processing, and overall infrastructure. The company's total investment in AI development exceeds $500 million. Despite this substantial investment, DeepSeek's streamlined structure allows for efficient innovation implementation.
DeepSeek's success highlights the competitive potential of a well-funded independent AI company. While the "revolutionary budget" claim is arguably exaggerated, the company's achievements are undeniable, resulting from significant investment, technological breakthroughs, and a strong team. The contrast is stark when comparing training costs: DeepSeek's R1 cost $5 million, while ChatGPT4 cost a reported $100 million—demonstrating DeepSeek's relative cost-effectiveness, even with its substantial overall investment.
