Yunqing Zhao

Multimodal LLMs, agents & media generation

Teach it which context to seek. Agents that go and get what they need.

CodeDance used RL to teach a multimodal model when a tool is worth calling; ThinkGen used it to teach one model to write another’s instructions. At Tencent Hy I work on multimodal LLMs and agents with a data-centric approach: teaching them what to look at, which tool to call, and when the evidence they have gathered is enough and accurate.

Some of this work shows up in HyCreator, a long-video agent that turns a short brief into a film. Before it assembles picture and sound, it creates references for each character, location and prop, asks a subagent for an independent review, and checks every clip for continuity.

Multimodal LLMs & tool use

Let the model build its context. Pixels, time and code, turned into evidence it can read.

Today’s multimodal models don’t just read their context; they have to build it. Pixels must become tokens a language model can read and write: a vision encoder pretrained to speak (GenLIP), and segmentation masks written as compact text (Text4Seg++). A long video mixes timestamps, saliency and captions, so each kind of token gets its own expert (TimeExpert). One model can even compose another’s context: an MLLM reasons first, then writes the instruction a diffusion transformer draws from (ThinkGen). And when one look isn’t enough, the model writes and runs code that draws boxes, lines and plots back into its own context, with a reward that keeps it from calling tools it doesn’t need (CodeDance). This body of work also contributed training data and evaluation to Pistis, a family of 27B and 9B multimodal models.

Evidence on demand CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning Executable code as a general visual solver: it composes tools and renders evidence it can check. A balanced tool-calling reward curbs overuse, and unseen tool compositions emerge during RL. CVPR 2026 · tech lead, corresponding author Vision that speaks Let ViT Speak: Generative Language-Image Pre-training Pretrain the ViT to emit language tokens directly, so its features arrive in the language model’s own terms. ECCV 2026 · co-author Masks as text Text4Seg++: Advancing Image Segmentation via Generative Language Modeling Extends Text4Seg (ICLR 2025): box-wise semantic descriptors turn masks into “semantic bricks”, so segmentation becomes next-brick prediction, with no mask decoder. TPAMI 2026 · tech lead, corresponding author Routing time TimeExpert: An Expert-Guided Video LLM for Video Temporal Grounding A mixture-of-experts Video-LLM that routes each task’s tokens to a specialist, reaching state of the art on dense captioning, moment retrieval and highlight detection. ICCV 2025 · co-author Think, then draw ThinkGen: Generalized Thinking for Visual Generation Chain-of-thought for image generation: the MLLM plans, the diffusion transformer draws, and alternating RL (SepGRPO) trains both. CVPR 2026 · co-author Data & evaluation Pistis Technical Report 27B and 9B multimodal models trained with IDRL — alternating on-policy distillation and RL in one loop — for stronger reasoning and long-horizon agentic tasks. Technical Report · training data & evaluation

AI safety for multimodal models

Guard what the model reads. An open input is an attack surface, and a key.

If a context can teach a model, it can also mislead it. Once large models read images and prompts at inference time, the context became an interface, and whoever writes into it can steer the model. With only black-box access, an adversarial image makes open VLMs such as MiniGPT-4 and LLaVA give the attacker’s chosen response (AttackVLM). The same channel can serve the owner: a trigger prompt only they know makes a watermarked Stable Diffusion draw a predefined watermark, such as a scannable QR code, while other prompts behave as before (WatermarkDM, US patent).

Few-shot learning & knowledge transfer

Learn from a little context. Fewer examples, so the prior matters more.

In few-shot image generation, a new domain arrives as about ten images, and the only place to put them is the weights. Adaptation becomes a negotiation between context and prior: what should a pretrained model keep, and what should the few examples overwrite? We found that it mostly fails by collapsing onto the examples (A Closer Look), that the prior worth keeping depends on the target (AdAM), and that some of it has to be actively forgotten (RICK). FS-BAN asks the same question of classifiers facing domains they have never seen. In-context learning answers it without gradients; this was the gradient version.