In 2023, a small but powerful Chinese company called DeepSeek emerged as a disruptive player in the field of generative artificial intelligence (AI). Its R1 model, which is free and open-source, demonstrated performance that rivaled and even surpassed that of the latest version of ChatGPT. Despite facing restrictions on acquiring cutting-edge chips due to embargoes, the DeepSeek team successfully advanced their innovative project. Now, the creators unveil their secrets in an article published in the journal ‘Nature’, highlighting the key to their success: a focus on reinforcement learning.
The Generative AI Revolution
Generative artificial intelligence enables the creation of texts, images, videos, and audio based on user instructions. This type of AI primarily relies on deep learning, a methodology that has revolutionized the sector over the past decade by using algorithms to analyze vast databases in search of patterns. The essence of this technique lies in the AI learning from the data.
Historically, machine learning had mainly developed using supervised learning, a method that involves processing millions of texts and manually adjusting the responses. However, the DeepSeek team decided to approach the problem differently by focusing on reinforcement learning, an approach similar to how a child learns to play a video game, through trial and error.
Reinforcement Learning: The Key to Success
DeepSeek's focus on reinforcement learning is based on incentivizing the reasoning capabilities of language models without relying on patterns predefined by humans. "We demonstrate that the reasoning capabilities of large language models (LLMs) can be stimulated through pure reinforcement learning techniques, avoiding the need to introduce human-labeled reasoning guides," explain the authors of the article.
Daphne Ippolito, a professor at Carnegie Mellon University and a specialist in natural language models, notes that DeepSeek's approach allows these LLMs to learn to reason without having been exposed to previous examples of human reasoning.
Innovation in Methods
The DeepSeek researchers concentrated their efforts on tasks where they could establish clear objectives and numerical rewards. The model's mission was to achieve the highest possible score, even without receiving explicit instructions on how to do so. This methodology proved effective, as the model outperformed others that were trained through traditional supervised learning.
"We achieved superior performance on verifiable tasks such as mathematics and programming proficiency, surpassing models trained conventionally with human demonstrations," highlights Wenfeng Liang, a member of the AI team at DeepSeek.
However, due to the lack of corrections in the model's responses, unexpected results sometimes occurred, such as mixing languages in the same text. To address this, the researchers decided to incorporate elements of supervised learning, seeking a balance between generating correct and comprehensible responses.
An Approach as Efficient as It Is Innovative
The team's strategy not only succeeded in terms of performance but also allowed them to optimize resources. "For LLMs to exhibit reasoning capabilities in the pre-training phase, a considerable amount of computational resources is required," notes the DeepSeek team. However, the innovation of using well-designed examples and minimalist prompts contributed to enhancing these capabilities.
Another formula for success was the use of distilled generative AI models, which allowed them to leverage existing technologies and avoid the arduous task of developing from scratch. This meant access to powerful AI with reduced energy consumption.
A Promising Future for AI
The DeepSeek team believes that their work on reinforcement learning could "unlock more advanced levels of capabilities in LLMs, paving the way for more autonomous and adaptable models in the future." Ippolito emphasizes that the study raises important questions about the nature of reasoning in AI: "The question of what makes a model reason well is both philosophical and technical. What kind of answers does a user seek when posing complicated questions to an AI system? Should we be concerned if the mode of reasoning is unintelligible, as long as it arrives at a correct answer?"
Conclusion
DeepSeek has demonstrated that with innovative approaches and a deep understanding of learning processes, it is possible to create technologies that not only compete with those of tech giants but also challenge the established norms in artificial intelligence. Its R1 model is not only a technological breakthrough but also a turning point for how AI models are developed and utilized worldwide.
For more information on the latest innovations and developments in technology and science, I invite you to keep exploring my blog. Don't miss out!
