In this article, Tian Yuandong provides profound insights into several key issues in current AI research, particularly his skepticism towards the Scaling Law and his advocacy for generative AI.
Tian first discusses the limitations of the Scaling Law. Proposed by OpenAI in 2020, the Scaling Law suggests that the ultimate performance of large models is primarily determined by computational power, model parameter size, and the amount of training data, rather than the specific structure of the models. However, Tian points out that as model performance approaches human levels, acquiring new data becomes increasingly difficult, and further improvements become harder to achieve. Additionally, he highlights that many real-world long-tail needs involve scenarios with very little data, which cannot be addressed by relying on the Scaling Law alone. This could eventually lead to a situation where everyone is isolated on their own "data islands," unable to share and utilize each other's data.
Tian then emphasizes the advantages of generative AI. He believes that generative AI can generate large amounts of content from minimal prompts, reducing the need for manual input and repetitive labor. Generative AI can work similarly to teaching a child, where minimal guidance allows it to extrapolate and create more, significantly boosting productivity. This is because generative AI can work around the clock, has low replication costs, and replicating engineers is very difficult.
Furthermore, Tian presents his views on breakthroughs in data efficiency. He argues that achieving truly data-efficient artificial general intelligence (AGI) requires 2-3 major breakthroughs. While the Scaling Law may be effective in some aspects, it is not the complete solution, as it represents a very pessimistic future.
Regarding the interpretability of AI, Tian believes that AI models based on neural networks are interpretable, and eventually, humans will understand how these models are trained. Despite many currently inexplicable aspects, he argues that this should not be a reason to abandon exploration.
Lastly, Tian discusses the diversity of technology in Silicon Valley, noting that everyone has their own methods, and technological progress does not necessarily rely on current mainstream approaches. Non-mainstream explorations could potentially drive the next technological revolution. He also suggests abandoning the notion that "the brain is the controller of humans," asserting that every part of the body has a vote in behavioral expressions, and future integrated AI will have a vote as well.
Through Tian Yuandong's perspective, this article offers readers unique insights into the development of AI, the limitations of data-driven models, and the prospects of generative AI, making it a valuable read for deeper understanding and contemplation.