Articles

The article deeply analyzes the current chaos of various 'techniques' in the PPO algorithm within the RL4LLM field, pointing out the community's widespread phenomenon of 'technique stacking,' which makes it difficult for engineers to choose. To solve this problem, the authors conducted detailed experimental verification of the four core techniques—Normalization, Clipping, Loss Aggregation, and Overlong Filtering—based on a unified ROLL framework and the principle of univariate ablation. The article reveals the true effects of these techniques under different models (Base/Instruct), data difficulty, and reward settings, such as recommending the 'group-mean + batch-std' normalization method, and pointing out that Clip-Higher is only effective for Instruct models, while Token-level Loss yields greater benefits for Base models. Ultimately, the article proposes a streamlined and high-performance 'Lite PPO' solution based on the experimental results, and provides a practical 'cheat sheet' for different scenarios, calling on the community to improve transparency and standardization, and offering valuable, evidence-based guidance for RL4LLM implementation.

This article provides an in-depth analysis of the Pass@k training method presented in the latest ACL 2025 paper. Pass@k aims to address the risk aversion and local optimization issues associated with the traditional Pass@1 reward mechanism in large language model reinforcement learning. By enabling the model to generate k answers simultaneously and rewarding it if at least one is correct, Pass@k effectively promotes exploration, facilitating a natural curriculum learning process from exploration to exploitation without incurring additional labeling costs. The article details the principles behind Pass@k and its core code implementation (requiring only approximately 20 lines of modification). Experimental results using the Qwen-7B model on GSM8K, MATH, and Maze tasks demonstrate Pass@k's ability to significantly enhance model performance, even surpassing GPT-4o. Finally, the article offers easy-to-implement upgrade steps for existing projects and highlights potential pitfalls, emphasizing the method's efficiency and practical benefits.

As a practical guide, the article meticulously breaks down 22 core experiences of Large Language Model Supervised Fine-Tuning (SFT). It first clarifies the positioning of SFT in the LLM lifecycle. It then provides a detailed comparative analysis with technologies such as Pre-training, RLHF, RAG, Incremental Pre-training, and In-context learning. The article delves into the prerequisites of SFT, including the selection of the Base Model (Base vs Chat), the construction of high-quality training data (format, quantity, quality control), and emphasizes that high-quality data is the core of SFT success, along with proper software and hardware environment configuration. In addition, it provides practical advice on hyperparameter tuning, effect evaluation, potential adverse consequences (such as the decline of general capabilities and overfitting), and avoidance strategies during the SFT training process. Finally, the article combines practical applications to discuss advanced topics such as inference time estimation and SFT Packing, and provides key code examples, providing LLM developers with extremely valuable practical guidance.

This article delves into SageAttention, a low-bit quantization (such as 8-bit, 4-bit) optimization algorithm library specifically designed for the Transformer attention mechanism. SageAttention aims to match or surpass the accuracy of libraries like FlashAttention and xFormers, while using fewer resources and reducing latency. Subsequently, it details several versions of SageAttention: v1 introduced smoothing and adaptive quantization of K, achieving a 2x acceleration compared to FlashAttention2; v2 further optimized through thread-level quantization and Q smoothing, with the speed increased to 3 times that of FlashAttention2; v2++ further enhances inference efficiency by refining the accumulation method from v2; v3 implemented the first mxFP4 attention for inference acceleration, with a speed 5 times that of FlashAttention, and explored the feasibility of 8-bit attention in training. The second half of the article provides specific steps for integrating SageAttention into the Hugging Face Transformers library, including environment configuration, installation methods, and code examples for replacing the standard attention module in the Qwen3 model with SageAttention, offering developers actionable implementation guidance.

This article explores the core mechanisms of GPU memory footprint during the training of large language models (LLMs). It begins by explaining fundamental concepts, such as model parameter sizes (e.g., 7B, 32B), different storage precisions (fp16, fp32), and the relationship between bytes and bits. Then, it calculates the memory occupied by model parameters, gradients, and Adam optimizer states, excluding activation values, and points out that their sum is approximately 16 times the model parameter size (calculated with fp16/fp32 mixed precision). For example, a 7B model requires about 112GB of GPU memory. To address this memory bottleneck, the article focuses on DeepSpeed's ZeRO optimization techniques (ZeRO-1, ZeRO-2, ZeRO-3), clarifying how these stages significantly reduce memory consumption per GPU by partitioning optimizer states, gradients, and model parameters. Finally, the article previews the next part, which will delve into the GPU memory footprint estimation of intermediate activation values, equipping readers with a clear framework for understanding the resource requirements and optimization strategies for large model training.

The article provides an in-depth introduction to Zhipu AI's newly released GLM-4.5 series open-source large model. The model's core breakthrough lies in its MoE (Mixture of Experts) sparse activation architecture with 355 billion total parameters (32 billion activated parameters) and a hybrid inference engine, achieving a balance between high computational efficiency and performance. GLM-4.5 aims to establish a 'golden triangle' of Agent, Reasoning Ability, and Coding Ability and has demonstrated excellent performance in authoritative benchmarks like TAU-Bench, BrowseComp, AIME 24, and SWE-bench Verified, surpassing some top proprietary models, particularly in agent and coding capabilities. The article also details the model's three-stage training process: 23T of high-quality data pre-training, 128K context mid-training, and expert distillation post-training. The open-source nature of GLM-4.5, especially its lightweight version GLM-4.5-Air, simplifies AI agent development, ushering in a new era of AI that 'thinks and acts'.

This article details the algorithm design and engineering practices of AgenticLLM in conjunction with Reinforcement Learning (AgenticRL). It begins by introducing a case study of AgenticLLM implementing tool usage via SFT, without Reinforcement Learning (e.g., Qwen-Agent solving math problems), and identifies issues related to path dependence and data distribution. Subsequently, the article focuses on how AgenticRL addresses these problems through RL training, demonstrating performance improvements with examples such as ToRL. From an engineering perspective, the article analyzes the computation process of RLHF and discusses the impact of multi-turn agent interactions on reasoning efficiency. It proposes and details asynchronous service solutions, such as vLLM's AsyncLLM, to address the complex interaction requirements of AgenticLLM. Finally, the article explores a fully asynchronous solution that decouples generation and training, and discusses the relationship between off-policy methods and SFT/DPO, offering insights into the future development of AgenticRL.

The article reports in detail on the user controversies triggered by the release of OpenAI GPT-5 and the emergency response from Sam Altman and his team. User reviews of GPT-5's performance are mixed, especially particularly missing certain features of GPT-4o. To address this feedback, OpenAI announced it would double the usage limit for GPT-5, allow Plus users to continue using 4o, fix model switching glitches, and optimize the UI. In the subsequent AMA Q&A session, OpenAI executives further responded to user questions about model version selection, unlimited mode, reasoning capabilities, voice features, bias handling, and context length, emphasizing their commitment to providing a more personalized model experience. They also acknowledged that computational power limitations have prevented them from achieving the 1 million Token context goal.

This article provides a detailed analysis of the performance bottlenecks that AI models (especially Large Language Models) may encounter when running on GPUs: Compute Bottleneck and Memory Bandwidth Bottleneck. The article first defines GPU compute capability (FLOP/s) and bandwidth (Byte/s), and introduces the peak compute intensity of a GPU. Subsequently, it introduces model memory footprint and model computation, as well as the model's compute intensity. The core part clearly distinguishes between compute bottlenecks (insufficient GPU computing power, model computing demands exceed GPU processing capability) and memory bandwidth bottlenecks (GPU memory read/write speed becomes the limitation, GPU computing power is available but waiting for data) through analysis and comparisons. The article uses A100 GPU as an example for quantitative explanation and emphasizes the importance of understanding these bottlenecks for optimizing model runtime efficiency. This understanding is crucial for further exploration of optimization techniques such as MQA and Flash Attention.