Author | Li Shuiqing, Editor | Mo Ying
On the morning of July 18, the "Foundation Model Architecture Innovation and Ecosystem Cooperation Forum," hosted by SenseTime as the most anticipated forum at WAIC 2026, attracted the attention of countless AI researchers, industry experts, and investment institutions. This was because it directly addressed the most core anxiety in the current large model industry: As the Scaling Law approaches its physical limits, is multimodality truly the "antidote" to break the deadlock, or merely an extension of old wine in a new bottle? The "Chief Scientists" roundtable dialogue, moderated by Dahua Lin, Chief Scientist at SenseTime, and jointly participated in by Professor Xipeng Qiu from Fudan University, Deli Zhao, Chief Scientist at DAMO Academy, Xiangyu Zhang, Chief Scientist at StepFun, and Professor Ziwei Liu from Nanyang Technological University, Singapore, covered a top-tier judgment matrix. It can be described as the first collective review and in-depth debate of China's AI industry after multimodal AI entered the "deep water zone."
Over the past month, the latest international models such as GPT-5.6 have been successively unveiled. In his opening remarks, Dahua Lin pointed out that today's most advanced large models have almost turned multimodal understanding capabilities into a "standard feature." But another question emerged: Is multimodality merely an extension of the capability boundaries of large language models, or the true starting point of next-generation intelligence? Further speaking, can today's large models, built upon Next Token Prediction, truly understand the physical world?
Amid high-density intellectual collisions, the five scientists engaged in a "life-and-death" debate sufficient to reconstruct industry cognition, focusing on the endogenous essence of multimodality, core bottlenecks, and the ultimate landscape five years from now.
01."Stepping Out of the Cave"Revisiting Whether Multimodality is a "Plugin" or the "Soul"?
Dahua Lin threw out this core question right at the beginning, which also represented the most central intellectual collision of the entire forum. Currently, there are two completely different voices in the industry.
One view holds that large language models have become the core of future AI, and multimodality is merely a natural extension of the capability boundaries of language models; the other view argues that when AI truly moves from the digital world to physical space and begins to build world models, multimodality is the most fundamental and native component of intelligence, rather than a plugin for language models. Should future AI be language-centric or world-centric? Dahua Lin put this question directly on the table.
In response, Xipeng Qiu's viewpoint is clear: intelligence still needs to rely on language. He believes that the future focus of multimodal development is not to replace language, but how to truly align multimodal information with language. However, in his view, today's large models are still far from truly understanding the world. "What models truly lack is not just multimodality, but Context." Here, Context refers to the ability to understand complex situations in the real world. Future models will face not just more images and more videos, but how to transform complex real-world situations into information that models can understand. The future development of multimodality will likely require a complete set of Harness built around the real environment to support the model's understanding of the real world. This contrasts with Xipeng Qiu's viewpoint.
Deli Zhao believes that multimodal-based models are the paradigm of next-generation intelligence. He first summarized four paths to intelligence from a broader perspective: large language models, robots moving into open spaces, simulations based on digital life, and interactions based on neural signals. These four paths share a common characteristic: they are all related to the source signals that generate human intelligence, and they are all multimodal, with no single modality. "Large language models are the compression of knowledge, and video generation models are also the compression and recording of human behavior. Large language models can learn intelligence, and video generation models can equally learn intelligence." Deli Zhao emphasized, citing examples of cheetahs hunting and ants finding their way. These superb skills are not learned through language, but through interaction, imitation, and feedback with the environment. Deli Zhao further pointed out that the physical world requires the modeling of four-dimensional causal relationships: physical space, physical attributes, causal relationships, and actions and movements, whereas large language models are only one-dimensional. Therefore, "whether from the principle of intelligence generation or the logic of algorithms, multimodal-based models are the paradigm of next-generation intelligence."
Unlike the perspectives of Xipeng Qiu and Deli Zhao, Xiangyu Zhang shifted the focus. He believes the core is not the route debate between language and vision, but the "learning paradigm": how to enable agents to learn autonomously, also known as postnatal learning or online learning. "If this is not solved, it doesn't matter whether you go with language or vision; it's actually the same." Why do language models seem more successful today? Xiangyu Zhang believes that our current training data is static, collected, cleaned, and organized through massive human effort. Language naturally has high information density, so it naturally has the advantage under this "imitation learning" paradigm. Visual signals, however, are sparse and high-dimensional, and do not work well under this paradigm. But here comes the problem, Xiangyu Zhang changed the subject, as large models move towards the agent era, everything is changing. "Agents interact with the environment, but they cannot efficiently and automatically learn from feedback, lacking the ability of self-evolution." He gave a vivid comparison: "A Codex subscription makes one person equal to ten, but no matter how long you use it, the improvement is limited; an intern you hire might not be good at first, but improves quickly through actual work. This is the ability of self-improvement, which our models lack."
Seeing the discussion forming tension between language-centrism and world-centrism, Ziwei Liu provided a profound judgment from a philosophical perspective. He cited Plato's Allegory of the Cave—language is a low-dimensional projection of the real world. "We observe the projections in the cave and summarize the laws of the world, but ultimately we need to step out of the cave and move towards the real world." Specifically at the data level, he compared language to fossil fuels: "Humans have accumulated it for a long time, but it is ultimately limited. We will definitely move to the next era and utilize other forms of energy—multimodal data." At the task level, "many high-value tasks, such as manufacturing, are inseparable from multimodality, and multimodality can truly emerge next-generation intelligence." It can be seen that from Dahua Lin's question of "language center vs. world center," to Xipeng Qiu's "contextual intelligence," Deli Zhao's "first principles," Xiangyu Zhang's "continuous evolution," and Ziwei Liu's "moving from virtual to real," although the scholars' entry angles are different, the trends they point to are highly consistent: multimodality is moving from an "auxiliary tool" to "core infrastructure." And the true driving force behind this transformation is the irreversible migration of AI application scenarios from digital space to physical space.
02.Data, Architecture, ParadigmWhat is Missing for the Next Leap in Multimodality?
After looking forward to the long-term AI prospects from first principles, Dahua Lin brought the question back to reality: Are current mainstream AI methods, such as data, model architecture, and training paradigms, sufficient to support us moving to the next stage? Where is the real bottleneck? This question has already shown a clear division of opinions in the industry.
One judgment is optimistic: current model architectures already possess sufficient potential, and the real limitation lies in the scarcity of high-quality multimodal data. In particular, data with temporal relationships, spatial relationships, behavioral processes, and outcome feedback is harder to acquire, annotate, and use than Internet text. As long as the data and computing power issues are solved, the existing route still has a lot of room for improvement.
Another voice is more cautious: the problem is not just insufficient data volume, but that the training method based on next token prediction may naturally be poor at learning physical laws, causal relationships, and continuously changing environments. True breakthroughs may require new paradigms such as native multimodal architectures, world models, reinforcement learning, active exploration, or continuous learning.
So what exactly is the primary factor limiting further breakthroughs in multimodal AI? Is expanding data and computing power still effective, or have we reached a stage where we must change model architectures and training methods?
First, Ziwei Liu gave a calm judgment: "None of these three aspects (data, model architecture, training paradigm) are sufficient to push to the next stage; all have room for improvement." He first deconstructed "why language works and multimodality doesn't" from the data level: language models got lucky, with 20 years of Internet-accumulated corpus and GPU computing power developed for playing games—all elements happened to be in place. But what about multimodality? "There are many videos and images on the Internet, but most are relatively static and lack strong long-range correlations." Long-range tasks like doing housework or daily work are hard to record. "Language models accumulated data by serendipity, while multimodality needs a better paradigm to record, save, and annotate data. This is not just a technical issue, but an organizational and social issue."
Immediately after, he pointed his spear at the architecture: "The overall architecture is still centered on language models, with multimodal capabilities entering in a bridging manner." He proposed the challenge of "native multimodality": how to achieve native on the input side? How to achieve native on the output side? Long context, sparse coding, at which layer to do representation—these questions all remain unanswered. Finally, falling back on the learning paradigm, he admitted this is a "bigger problem." "Everyone doing vision has this feeling; people were talking about this problem 10 years ago, and it still hasn't been solved."
Ziwei Liu's "triple strike" broke the problem down very thoroughly, but Xiangyu Zhang perhaps felt it could be more thorough. He took over the proposition of "continuous evolution" to dig deeper. He believes that autonomous learning is a consensus, but there are divergences in specific paths. Today's models have relatively long contexts, but "it's still far from long enough," and there is severe context degradation. Engineering-wise, relying on upper-level Harness for context management and Self Evolve is a route some are trying, but the problems are obvious. Multimodality makes this problem even more severe. "Multimodal information is too long; compressing multimodal signals is not as natural as compressing text.
What if it's stuffed with images of 1 million tokens? The compression rate is very limited." Then he threw out a sharp question: "Can the agent itself truly evolve only through external memory? Just like a student who is very good at taking notes, but if his hippocampus can no longer memorize new things, can he make long-term progress relying solely on external memory? No, he can't." Xiangyu Zhang continued to deconstruct, stating that the next generation of autonomous learning needs to solve three major problems: how to learn (the online learning problem), what to learn (exploration and curiosity modeling), and how to learn efficiently (the efficiency problem).
And what is most worth investing in now and most likely to yield results in the short term is online learning. Online learning faces two core challenges, he continued to elaborate. The first is hierarchical memory modeling. Today's Transformers are excellent at modeling short-term memory, but completely lack instantaneous memory (sensory memory). Long-term memory is divided into episodic memory and semantic memory, corresponding to different mechanisms. "The context of a Transformer is more like a kind of working memory in principle; it is lossless, but the effective length is relatively short. You are using a method that tries to lengthen working memory to model fundamentally different memories, which is definitely problematic."
The second is the learning algorithm itself. Many people are betting on reinforcement learning, but Xiangyu Zhang proposed a different view: "RL was very successful in the past reasoning era with high data efficiency, but this efficiency largely comes from On Policy, generating data by itself and then distilling it back. It's not just RL that satisfies this characteristic; distillation itself and Self Correction also satisfy it." He believes On Policy might be the way out to solve the problem of sparse supervision signals, while world models are another key to solving the efficiency problem.
Dahua Lin interjected with a follow-up question, steering the discussion towards the Memory mechanism. "Current models, whether Dense or MoE, essentially rely on Context to bear memory. Context is pushed from 1 million to 2 million, but the effective part is still just a small segment at the beginning. Doesn't this indicate that we need significant innovation because we are using a short-term working memory mechanism to bear functions that should belong to long-term structured memory?" Xiangyu Zhang immediately nodded in agreement: "Exactly, this is also a very important frontier research point: how to model the Memory mechanism." Human memory mechanisms are hierarchical, including instantaneous sensory memory, short-term working memory lasting 2-4 seconds, and long-term episodic and semantic memory. The Context of a Transformer is more like working memory in principle and characteristics; it doesn't lose information, but the effective length is inherently limited. Once visual signals are input, they stay in the Context forever, and the mechanism does not allow them to be compressed away. Humans, however, will actively select, compress, and clear. This is a fundamental difference.
At this point, Deli Zhao joined the discussion on "where the bottleneck is." He first mentioned an industry phenomenon: "Model architectures are gradually converging, no one cares about the algorithms themselves anymore, and everyone is going to do applications next." As a scientist with an algorithm background, he believes this is a huge misconception. "As AI moves from digital space to the physical world, the first thing it faces is the data problem." Deli Zhao pointed out that the data for language models and video generation models all come from Internet accumulation, while data in the physical world is extremely scarce. "Taking robots as an example, last year it was awesome if anyone could have 100,000 hours of data; this year people are just talking about several hundred thousand hours, but it is far from enough to support training at the GPT-3.5 level."
However, Deli Zhao changed the subject and gave a judgment with deep industry insight: "China's industrial scenarios, service industry scenarios, and home appliance scenarios have diversity and demands far greater than Europe and the US. All data comes from actual scenarios; from this perspective, China has a natural huge advantage in the embodied direction." Regarding training paradigms and algorithm architectures, Deli Zhao emphasized a fact that is easily overlooked: AI has a tradition of technology sharing, with papers and technical reports being public, which "gives people a misconception that technology is not important." But in reality, "introducing reinforcement learning into foundation models, specific algorithms like PPO and GRPO, architectural innovations like Sparse Attention and Sliding Window, the underlying optimization mechanisms of DeepSeek—every technological advance drives the development of the field."
Returning to Memory innovation, Deli Zhao believes it involves two levels: first, the deep coupling of software and hardware with chip architectures; second, how personal data and work data can be supplied to models for long-term use. "DeepSeek's Ingram algorithm injects knowledge into the model, couples it into an integrated whole, and conducts end-to-end training, which is a specific example of architectural innovation." Thus, the discussion on "what the bottleneck is" presents a four-layer progression: Dahua Lin first positioned the problem, Ziwei Liu deconstructed the limitations in data, architecture, and paradigm, Xiangyu Zhang delved into the technical details of memory mechanisms and online learning, and Deli Zhao found a breakthrough from the reality of industrial implementation and data acquisition. The perspectives of the scientists interweave and complement each other.
03.Overestimating Agents, Underestimating Social Change:Five-Year Predictions
Dahua Lin left the last question to the time scale. Looking back at the AI industry from late 2022 to early 2023, many things happening now were completely unpredictable. So, looking at the present from five years in the future, what will be overestimated? What will be underestimated? Dahua Lin first shared his own feelings. In early 2023, everyone was most anxious about whether AI would replace human jobs, but looking at it today, AI is more about changing the way we work rather than eliminating jobs, which might be overestimated; what is underestimated is the speed of paradigm shifts. In 2023, few people could have predicted that reinforcement learning would become a core method again in 2025-2026, nor did anyone predict that "test-time scaling" would become an independent dimension of competition.
Xipeng Qiu frankly admitted: "Prediction has always been very difficult. Most of the future AI should not be what we imagine today. If we deliberately pursue certain things, it will not necessarily develop as expected. On the contrary, what seems less attended to now might instead achieve full development."
He gave a judgment: The next-generation paradigm shift might be "AI for AI." "Since Coding Models and Agentic Models have already made breakthroughs, why not take them to the extreme first, and then let them design what the future multimodal models will look like and whether there are better paradigms." Xipeng Qiu emphasized the power of gradual evolution. "ChatGPT looked like a huge paradigm innovation in 2023, but it was also developed step by step. The Transformer came later; even earlier was the Seq2Seq paradigm, which allowed me to unify all natural language processing tasks. With this kind of cognition of great potential, we then optimize the model until a certain mutation node, and suddenly it succeeds. The future will definitely be quite different from what we predict now."
Contrasting with Xipeng Qiu's "serendipity" viewpoint, Deli Zhao's prediction is relatively specific. He believes what might be overestimated is "the autonomous capability of Agents when landing in specific enterprise-level and commercial-level scenarios." What is underestimated is "the change AI will bring to the way society operates." Five years is a very large cycle for AI. He gave a very specific example: payment. "We are gradually moving into interactions between humans and agents, and between agents and agents. This involves a simple question: how do agents make payments? The entire payment system is built around humans, and mobile payment is the most effective tool for increasing social efficiency. But what about agent payments? Behind this is the reconstruction of the entire way society operates." One focuses on "unpredictable evolution," and the other focuses on "social certainty." These two prediction directions seem vastly different but are actually complementary: Xipeng Qiu reminds us to maintain an open mindset towards technological paths, while Deli Zhao hints at how AI will penetrate the social fabric.
04.Conclusion: MultimodalityIs Becoming the New Main Battlefield of the AI Industry
On the path to physical world intelligence, language models are crucial, but far from sufficient. From Xipeng Qiu's "contextual intelligence," Deli Zhao's "four paths to intelligence," Xiangyu Zhang's "autonomous learning paradigm," to Ziwei Liu's "stepping out of the cave," every top scholar points to the same direction from different dimensions: multimodality is not an appendage to large language models, but the native foundation of next-generation intelligence.
Current multimodal AI is at a critical turning point. Innovations in algorithmic architecture are far from stopping, from the redesign of Memory mechanisms to the deep integration of reinforcement learning, from the construction of world models to breakthroughs in online learning paradigms. These are all thresholds that must be crossed to reach physical world intelligence. In terms of industrial implementation, China's diversity advantages in embodied intelligence, industrial scenarios, and service industry scenarios are expected to become key variables in the next stage of global AI competition.
As Deli Zhao said, five years from now we might be surprised to find that the change AI brings to the way society operates is far more profound than imagined today. Multimodality is becoming the new main battlefield of the AI industry. And the debate over "soul" and "plugin" may not have a standard answer in the short term, but it determines the ultimate direction of the second half of physical AI.