AI PAPERS/DAILY
Subscribe free Open sourceView on GitHub Star This project is open sourceRead the code or use it in your own project. If you like it, don’t forget to give it a ★ star on GitHub.

All papersReasoning & AgentsTechnische Universität Berlin

τ-bench-LILT: Evaluating Conversational Agents on Realistic Multilingual Tool-Use Tasks

/ Paper arXiv 2611.07203

Key numbers

61by up to 61 points in pass^4

From the paperCopied from the abstract, not written by AI.

Abstract

From the paperThe authors' own words.

Frontier evaluations of tool-using conversational agents are conducted almost exclusively in English, leaving a critical gap for organizations deploying agents in other languages and regions. We present $τ$-bench-LILT, a multilingual extension of τ^3-bench covering German, Korean, Turkish, and Farsi in the airline, retail, telecom, and banking domains. We natively localize every layer of the benchmark: customer databases are populated with culturally appropriate records, user-simulator prompts are refined iteratively through contrastive annotation by native-language experts, and tasks are updated according to the localized database and to reflect realistic tasks in the target region. Evaluation across eight agent models shows the largest drops compared to English in the Korean and Farsi retail domains, by up to 61 points in pass^4. Code and data released at https://github.com/lilt/tau-bench-lilt.

From the full paper

From the paperCopied from the paper itself by our code.

Reading the paper on arXiv…

Try it

Code, models and demos that use this paper.

Datasets:
lilt-ai/tau-bench-lilt ↗1 dataset on Hugging Face

Ask this paper

/ Ask a questionAnswers come from the full paper and name the section

    / Your notes

    Select text in the abstract to highlight it or add a note. They stay in this browser.

      / Read next, same idea

      Read the PDF