Project summary

Instead of focusing on unambiguous harmful queries used in multilingual safety benchmarks and checking for compliance quantitatively, this project studies the divergences in communicating values across languages and cultures. Initial findings showed how LLMs answer to ambiguous or hazardous queries differ across Indic languages (Hindi, Punjabi and Tamil, as opposed to English). Models seemed to favour evidence-based outputs in English, and community-building orientation in Indic languages.

The pilot of our research and judges’ feedback can be found here: https://apartresearch.com/project/traduttore-traditore-llm-languagedependent-safety-answers-in-community-contexts-d8n7 

What are this project's goals? How will you achieve them?

The goal of this project is to scale our pilot research and verify if patterns hold in a more robust safety evaluation. Then, we will be able to determine whether the themes and categories that appeared remain observed or if new ones emerge from the data. To do so, we will:

• Increase the number of prompts

• Test more models

• Expand the languages (based on time and capacity allowed with funding)

• And most importantly, repeat evaluations extensively

We also aim to publicly store all the material used and created to facilitate replication, as well as inspiration for future research. Therefore, all the prompts, the corpus of outputs, our coding framework, the documentation used and final benchmark materials will be public.

How will this funding be used?

The team is made up of early-career researchers. Funding is essential to scale our study, while making it more robust and its evaluation reproducible. All funding would be split equally in the team. The minimal funding would serve to finance AI tools and premium subscriptions needed to fulfil such research. Additional funding would allow us to commit to more hours in the research and dive deeper into smaller variations, such as incorporating more languages and areas in the world, while maintaining a focus in the Global South (Latin America, Africa, Asia). Below is a breakdown of our hopes to scale this research further and obtain deeper insights into thresholds:

$600 [Technical Costs]

API costs for current frontier models from OpenAI, Anthropic, and Google, we estimate multiplying the data collected by at least 5 times. The costs would serve to increase prompts and repeat generations. We aim at sharing our evaluation dataset publicly at this stage.

$1,800 [Mandarin & Cantonese]

The addition of widely spoken Asian languages would deepen the focus on Asia, while providing findings that would reflect a great majority of the continent’s population. The statistical robustness could be augmented through increasing models (including DeepSeek for instance) and repetition further.

$3,000 [Romance Languages]

Deeper focus on whether the location grounding or the linguistic nature is the origin of the variation. For this purpose, the integration of Spanish and French with location-specific grounding in Latin America and Europe on the one hand, while North America, Europe and Africa on the other.

$3,600 [ACL 2027]

Ultimate objective, finishing our project with all the variations in languages, creating an open-source benchmark, and submitting the paper and benchmark to the Association of Computational Linguistics (ACL) conference of 2027.

Who is on your team? What's your track record on similar projects?

Karan Verma: Work covers English, Hindi, Punjabi and Tamil. B.Tech. in Computer Science and Engineering from DAV Institute of Engineering and Technology (Jalandhar, India). Independent researcher working on multilingual AI evaluation, AI safety, and LLM evaluation methodologies. GitHub: https://github.com/karanverma; Hugging Face: https://huggingface.co/karanverma19.

Jeanne Marie Jacqueline Vincendeau: Work covers French, Spanish, English, Italian, Mandarin. Double Degree Master’s of international relations between LUISS Guido Carli (Rome) and China Foreign Affairs University (Beijing). My research focuses on the intersection between political violence and AI governance. Previous works: https://orcid.org/0009-0001-6835-5532.

What are the most likely causes and outcomes if this project fails?

Based on our experience in the hackathon, and judges' feedback, the idea has a lot of potential and the paper’s contributions to the literature are original. The main hurdles we have faced are time and tools, funding could help us fix both of these issues, specifically by giving us the flexibility of spending hours collecting large amounts of data and analysing it sharply.

Ultimately, if we obtain a null or insignificant result when scaling the research, it would still have contributed to the literature on multilingual and multicultural AI safety evaluations. Our methods on making evaluations’ prompts more grounded on daily life contexts through the tripartite crafting could also be a valuable framework for future research. Additionally, this research would show the robustness of multilingual AI safety evaluations and leave behind dense datasets that could be reused for future research.

How much money have you raised in the last 12 months, and from where?

No money was raised to work on this project so far, it is our first time filing a funding request.