Articles

The article presents FilBench, a new, systematic evaluation suite designed to assess the performance of Large Language Models (LLMs) in Philippine languages, specifically Tagalog, Filipino, and Cebuano. Developed in response to the lack of clear, systematic understanding of LLM capabilities in these languages despite high local usage, FilBench covers four main categories: Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation, across 12 distinct tasks. The suite is built on Hugging Face's Lighteval framework for ease of use and integrates non-translated content to ensure faithfulness to natural language use. Through evaluating over 20 state-of-the-art LLMs, the authors found that while region-specific LLMs show promise and parameter efficiency, they still lag behind closed-source models like GPT-4o. The study also revealed that Filipino translation remains a significant challenge for LLMs, often resulting in verbose or incorrect outputs. Finally, FilBench identified that open-weight LLMs offer a cost-effective alternative to commercial models for Filipino language tasks without sacrificing significant performance, making them suitable for regions with limited internet infrastructure and lower incomes. The authors hope FilBench will catalyze further research and development in Filipino NLP.

The article addresses the inefficiency of manual research discovery, which involves tedious switching between platforms like arXiv, GitHub, and Hugging Face. It outlines three layers of abstraction: manual research, scripted tools, and MCP integration. While scripts automate some tasks, they are prone to errors and require constant maintenance. The Model Context Protocol (MCP) elevates this by enabling AI systems to interact with these tools using natural language, orchestrating complex queries, filling information gaps, and reasoning about results. This approach, likened to Software 3.0, promises to significantly automate and streamline the research discovery process, making it more efficient, albeit still requiring human guidance for optimal quality. It provides quick setup instructions for using the Research Tracker MCP via Hugging Face settings and resources for building custom MCP tools.

This article introduces TextQuests, a novel benchmark designed to assess Large Language Models (LLMs) in dynamic, interactive environments, contrasting with their strong performance on static knowledge benchmarks. Built upon 25 classic Infocom interactive fiction games, TextQuests challenges LLMs to demonstrate sustained, self-directed reasoning over long and continuously growing contexts, and to learn through exploration via trial-and-error. The benchmark evaluates LLMs based on 'Game Progress' (tracking objectives) and 'Harm' (ethical behavior). Key findings reveal that current LLMs struggle with long-context reasoning, often hallucinating or repeating actions as context windows exceed 100K tokens, particularly in spatial reasoning tasks. The article also touches on the importance of 'Dynamic Thinking,' highlighting the trade-off between computational efficiency and performance. TextQuests aims to provide an open-source tool for researchers to better understand and improve LLM agents' capabilities in complex, exploratory settings.