BestBlogs.dev Highlights Issue #7

This article, written by Siyuan Chai from Zhipu AI, details the application of RAG technology in enterprise service scenarios. RAG technology solves the hallucination problem of large models, reduces implementation costs, and improves the traceability of answers through three steps: Indexing, Retrieval, and Generation. Zhipu AI provides a complete RAG technology solution, including file parsing, fine-tuning the Embedding model for specific tasks, retrieval strategies, and tools for knowledge construction and question-answering processes. The article showcases the practical application of RAG technology through a specific case of intelligent customer service practice in the public affairs customer service question-answering scenario, addressing the high maintenance costs and frequent knowledge updates of traditional customer service systems. Finally, the article looks forward to the future development of RAG technology and introduces Zhipu AI's continuous exploration and practice in related fields.

The article reports on the release of Claude 3.5 Sonnet, which has outperformed GPT-4o in terms of performance and cost-effectiveness. Key highlights include its 10x faster coding speed, the introduction of the Artifacts feature for real-time code generation and execution, and its potential to replace a significant portion of users' work. The article also covers various user tests and comparisons, showcasing Claude 3.5 Sonnet's capabilities in creating games, visualizing neural networks, and more.

The article provides a detailed exposition of the challenges of professionalism, collaboration, accountability, and security faced by AI Agents in corporate production scenarios. Huawei Cloud, through practical scenarios, adopts a combination of multi-faceted technologies to address these challenges.

Specific practices include a progression from basic to advanced levels across three stages and seven steps, as well as key technical practices tailored to the challenges, such as the construction of corporate vocabularies, integration of external knowledge bases, implementation of anti-degradation mechanisms, model orchestration strategies, and security risk mitigation measures. Additionally, the article illustrates the application effects of AI Agents in scenarios such as customer service assistants, meeting minutes generation assistants, and production command assistants through three corporate case studies. Finally, the article forecasts the emergence of interactive, transactional, and device-oriented AI Agents in future corporate scenarios and emphasizes the importance of building a management and collaborative communication network compatible with multiple Agent runtimes.

Academician Sun Ninghui discusses the development of intelligent computing in China, highlighting four major challenges and potential paths forward. The article emphasizes the importance of AI in driving down costs and expanding user bases, with a focus on empowering the real economy. It also covers the history of computing technology, the evolution of intelligent computing, and the impact of large AI models like ChatGPT.

The article discusses the evaluation of Retrieval-Augmented Generation (RAG) tools, covering the assessment process and results, component-based and end-to-end evaluation methods, and tools for evaluating RAG quality such as TruLens and RAGAS, as well as tools for automating RAG evaluation like LangSmith and Langfuse.

The open source model enables every company to have the potential to become an AI company. With the widespread application of large models across various industries, the open source large model ecosystem is rapidly developing. Researching open source large models is not only a crucial exploration towards achieving Artificial General Intelligence (AGI) but also a key driver for the widespread application of AI. Open source large models offer broader user coverage and greater innovation freedom, demonstrating strong innovation dynamics in user experience, technology, and product iteration. As the number of products based on open source large models increases, these models are expected to become a significant force in the widespread adoption of AI, covering various scenarios in both toC and toB products. Therefore, Jia Zi Guang Nian has released the '2024 Open Source Large Model Ecosystem Research Report,' which studies the development of AI and open source large models, sorts out the ecosystem of open source large models, discusses commercial practices in the field, and forecasts future industry trends.

The era of large language models (LLMs) is filled with exciting opportunities. Over the past year, LLMs have become 'good enough' for real-world applications, and are expected to drive around $200 billion in artificial intelligence investment by 2025. LLMs have also widely enabled everyone, not just machine learning engineers and scientists, to incorporate AI into their products. This article shares best practices on the core components of LLM technology, including prompting techniques to enhance quality and reliability, strategies for evaluating outputs, improving retrieval-augmented generation, and adjusting and optimizing workflows. It also discusses how to design workflows with human involvement.

The article begins by noting that the design of generative AI assistants differs from traditional product design, requiring special attention to user experience. With the advent of technologies like ChatGPT, AI assistants are redefining the way products are used. The article analyzes the key design elements of AI assistants from a UX design perspective, including functions, intelligent agents, input boxes, answer message bodies, dialog bubble functions, generation process interactions, and voice calls.

In terms of functions, the article discusses functional guidance, intelligent agent centers, shortcut commands, recommended features, and provides detailed explanations on the input methods of the input box, functional elements, as well as the clear/new dialog button. In terms of answer message bodies, it explores different types of message bodies such as text messages, multimedia messages, card-style messages, and interactive forms. The discussion on dialog bubble functions includes display methods and feedback operations. The generation process interaction section emphasizes immediate feedback, interruptibility, and the provision of optimization suggestions. Finally, the article also examines the call flow and key elements of voice call functionality.

Additionally, the article proposes eight design principles, which include the visualization of natural language processing capabilities, context awareness and continuity, multi-modal interaction, immediate feedback and confirmation, personalization and customization, transparency and explainability, error handling and correction mechanisms, and emotional understanding and feedback.

The article introduces a course on generative AI available on freeCodeCamp.org, which is designed for learners of all levels and avoids complex technical details. Developed by Abdul from 1littlecoder, the course covers a brief introduction to generative AI, a comparison between its past and present, the reasons for its current feasibility, and an in-depth discussion of topics such as the concept of decentralized AI and the introduction and analysis of LLM (Large Language Model) APIs at the application level. The course also includes content on Q&A systems, chatbots, RAG (Retrieval-Augmented Generation) solutions, and the application of large language models in natural language processing tasks and the development of intelligent AI agents. Finally, the course prospects the potential of large language model operating systems and provides a comprehensive explanation of the past, present, and future of generative AI.

Mobile-Agent-v2 is an automated mobile device operation tool based on a pure visual approach that operates without relying on system-level UI files. The recently released Mobile-Agent-v2 has demonstrated significant improvements in various aspects, including retaining the pure visual scheme, implementing a multi-agent collaborative architecture, enhancing task decomposition and cross-application operation capabilities, and adding multi-language support.

Its application scenarios range from assisting the elderly and visually impaired in hailing rides to managing chat messages. The paper and code for Mobile-Agent-v2 have been made public, and it has already been integrated into ModelScope-Agent by the Magic combination team.

Demonstration videos showcase Mobile-Agent-v2's capabilities in automated ride-hailing tasks, handling messages in chat applications, and operating social media platforms.

From a technical implementation perspective, Mobile-Agent-v2 addresses the challenge of tracking long operation histories through the collaborative work of planning, memory, and reflection agents. It has shown comprehensive improvements in tests conducted on both English and non-English applications. Ablation studies have validated the importance of the planning agent, decision-making agent, and memory units for the system's performance.

This webpage is the daily report from the ShowMeAI Research Center, summarizing the latest developments in the fields of deep learning and artificial intelligence, including the open-sourcing of DeepSeek's local private deployment service and its large model, the completion of the LLM course at Shanghai Jiao Tong University, the basic video tutorials for ComfyUI, the sharing of experiences from the founder of Devv AI search engine, a comprehensive guide to GenAI design patterns, and a historical review of deep learning, among other content.

The application of large models in the field of artificial intelligence is rapidly expanding, from the initial prompt engineering to the pursuit of general artificial intelligence (AGI). This article explores the progress of large models in practical applications and how they pave the way for AGI. It covers prompt engineering, RAG, AI Agent, knowledge base, knowledge graph, and other applications, providing a comprehensive overview of the development and prospects of large models in AI.

A comprehensive 75-page report on prompt techniques for Large Language Models (LLM) has been released by over 30 researchers from institutions including the University of Maryland, OpenAI, Stanford, and Microsoft. The report details various prompt techniques and their impact on LLM performance, highlighting the sensitivity of LLMs to specific details in prompts and the importance of careful engineering in enhancing model accuracy.

AutoGen Studio, developed by Microsoft Research, is a low-code interface built on the open-source AutoGen framework, designed to facilitate the rapid creation, testing, customization, and sharing of multi-agent AI solutions with minimal coding. It leverages the power of multiple AI agents working collaboratively to tackle complex tasks, making it accessible to a wide range of users including researchers and developers.

This article discusses the development of video-to-audio technology that uses video pixels and text prompts to create synchronized soundtracks for silent videos. The research aims to enhance creative control and provide a range of sound options for various video content.

BigCodeBench is a new benchmark for evaluating large language models (LLMs) on their ability to solve practical and challenging programming tasks. It addresses shortcomings of existing benchmarks like HumanEval, which are considered too simple and not representative of real-world programming. BigCodeBench features 1,140 tasks that involve complex instructions, diverse library calls, and rigorous testing. The benchmark includes two variants: BigCodeBench-Complete, where LLMs complete function implementations based on detailed instructions, and BigCodeBench-Instruct, which tests instruction-tuned LLMs' ability to translate natural language instructions into code.

Meta has recently unveiled several new AI models and datasets, including the Chameleon multi-modal model, Multi-Token Prediction, JASCO for text-to-music generation, AudioSeal for AI voice detection, and PRISM dataset for enhancing language model diversity. These releases aim to advance AI research and applications across various domains.

The MiCo team from Hong Kong Chinese University and other institutions has proposed a large-scale full-modal pre-training paradigm, Multimodal Context (MiCo), which supports 10 modalities and 25 cross-modal understanding tasks. The paradigm introduces more modalities, data, and model parameters into the pre-training process, achieving impressive performance in multimodal learning. The model has set 37 SOTA records in 18 multimodal benchmarks, showcasing its capability in coherent multimodal understanding.

Huawei Pangu 5.0 has been unveiled at the Huawei Developer Conference on June 21. The new version features upgrades in three main areas: full series, multi-modal, and strong thinking capabilities. Key highlights include:

1. Introduction of models with different parameter specifications to suit various business scenarios.
2. Enhanced multi-modal capabilities for precise understanding and generation of high-resolution images and videos.
3. Integration of advanced thinking chain and strategy search technologies to improve mathematical and complex task planning abilities.
4. Application of Pangu 5.0 in various fields such as autonomous driving, industrial design, and traditional Chinese medicine.
5. Introduction of new architectures and data synthesis methods to enhance model efficiency and performance.

Character.AI, founded by Noam Shazeer, achieves 20% of Google search traffic with 2 million inference requests per second. The article discusses the optimization techniques used to achieve this, including memory-efficient architecture design, attention state caching, and int8 precision training.

Alibaba Cloud has launched its first AI programmer based on the Tongyi large model, capable of completing end-to-end application development in minutes, significantly improving development efficiency and expected to achieve a 100-fold increase in productivity. The AI programmer has an innovative multi-agent architecture, with different agents responsible for tasks such as requirement understanding, task decomposition, code writing, testing, problem fixing, and deployment. It has the skills of architects, developers, and testers. At the Shanghai AI Summit, Alibaba Cloud demonstrated the AI programmer's ability to independently complete an Olympic schedule application in just 10 minutes, a task that would take a human programmer at least half a day. The AI programmer is still in the early research stage but can already complete simple tasks such as searching for tools online, debugging, and testing iterations. Alibaba Cloud aims to replace 70% of repetitive work with AI, allowing programmers to focus on more complex and valuable tasks. The internal AI code generation rate has reached 26%, and the goal is to reach 70% in the future.

The Luanchen Open-Sora team has achieved a breakthrough in the quality and generation time of 720p HD text-to-video, seamlessly producing high-quality short videos in any style. Surprisingly, they have chosen to bring another shock to the open-source community by continuing to open-source all their work. Visit their GitHub: https://github.com/hpcaitech/Open-Sora

This article from Alibaba Technology details a comprehensive approach to creating a song and its music video from scratch using AIGC large models and Multi-Agent systems. It begins by outlining the traditional music video production process and then demonstrates how to innovate by integrating large model capabilities. The creation process is divided into three stages: pure manual, human-AI interaction, and interface automation. The article delves into the breakdown of agents and the use of prompts, using director agents, art agents, and sound director agents as examples. It explains how these agents can be used to create storyboards, keyframes, and theme songs, ultimately culminating in a finished video. The article also mentions other tools and platforms, such as Mjdjourney, pika, audiocraft, and chattts, showcasing the wide-ranging applications and future potential of AI in music and video production. It concludes by looking ahead to the future of Multi-Agent systems, envisioning a time when multi-modal large model interfaces are fully open, enabling AI to efficiently complete complex creative tasks and revolutionize the music and video production landscape.

This article provides an in-depth exploration of the AI search engine Perplexity, featuring a 3-hour interview with the CEO and a full text of 50,000 words. It discusses the product's unique features, such as AI-assisted question formulation and subsequent retrieval, and its potential impact on the search engine market, particularly in comparison to Google. The article also delves into the technical aspects of machine learning, retrieval-augmented generation, thought-chain reasoning, web indexing, and user experience design.

Innovative Music Creation: Suno utilizes AI music generation tools to create complete songs with simple text prompts, revolutionizing traditional music creation processes. Key points include: 1. Promoting social and personalized music creation through collaboration. 2. Innovations in audio tokenization for managing continuous signals. 3. Importance of aesthetics in evaluating AI audio models through extensive listening and A-B testing. 4. Suno's journey from text processing to audio AI and its focus on music over speech technology.

Wang Xiaochuan, the founder of Baichuan Intelligence, believes that healthcare is the 'hard but right thing' on the path to AGI. He emphasizes that while many AI applications focus on entertainment (killing time) or efficiency (saving time), healthcare has the potential to 'add time' by improving quality of life and longevity. This viewpoint reflects his focus on developing AI applications that address real-world problems with significant impact, rather than simply showcasing technology for its own sake. He also cautions against 'laying eggs along the way,' as creating too many applications, even if successful, can drain resources and distract from the pursuit of AGI.

At the 2024 Developer Conference, Huawei unveiled its deepened strategy and infrastructure layout for the age of intelligence, introducing the new HarmonyOS NEXT developer beta. This release features a novel system architecture and AI integration, aiming to redefine the cross-device user experience. Huawei's Pangu 5.0 large-scale model has seen enhancements in multi-modality and robust reasoning, with applications spanning industrial design, media production, and autonomous driving, among other sectors. In tandem, Huawei Cloud announced the debut of the Pangu 5.0 large-scale model at the event, marking its first joint unveiling with HarmonyOS NEXT. This move underscores Huawei's commitment to deep integration within the AI domain and its aspirations for the intelligent future.

Huawei Cloud has crafted an AI-native cloud environment through comprehensive, system-level AI innovations, encompassing data centers, cloud platform architecture, and infrastructure services. This initiative equips AI developers with an AI-native foundational infrastructure. Furthermore, Huawei Cloud has elevated its AI development production line, ModelArts, to establish the ModelArts Studio platform, which delivers hosting services for a myriad of third-party large-scale models, accommodating a wide array of scenarios.

This article explores the capabilities of Apple's self-developed large model and its collaboration with OpenAI. It reveals that Apple's large model is highly competitive, matching the performance of mainstream 7B models and even reaching GPT-4 Turbo levels. The collaboration with OpenAI is not about integrating OpenAI's models into Apple's systems but rather using OpenAI's services to enhance user experiences. The article also discusses the implications of this technology for future hardware and AI integration.

Jensen Huang, CEO of NVIDIA, delivered a commencement speech at Caltech's 2024 graduation ceremony. He shared insights from his career, encouraged graduates to engage in the AI revolution, and discussed the transformative impact of accelerated computing and deep learning. Key points include: 1. The importance of AI and accelerated computing. 2. The evolution of NVIDIA and its contributions to technology. 3. Encouragement for graduates to seize opportunities in AI. 4. Reflections on the future of computing and AI's role in it. 5. Personal anecdotes and lessons learned from his journey.

Sam Altman discusses the positive impacts of AI on productivity and the challenges such as cybersecurity. He highlights the progress in language coverage with GPT-4o and the commitment to improve language fairness. Altman also emphasizes the importance of balancing safety and efficiency in AI governance, predicting that China will develop a unique large language model. He reflects on how AI might make humans more humble, prompting a reevaluation of our place in the universe.