Innovating with generative AI to better understand community needs

El Salvador is the smallest country in Central America, with a population of around 6.3 million people. Of those, more than 71,000 — some 1.1% of all families — are internally displaced. Many more have previously experienced violence and displacement; some have returned to the country after attempts to migrate north; and an increasing number of people in mixed movements are now transiting the territory. This all adds up to a complex human mobility landscape.

UNHCR, the UN Refugee Agency, needs to have an up-to-date and nuanced understanding of the experience of people around the country in order to fulfill its protection mandate, address evolving needs, and ensure communities’ priorities shape the decisions affecting their lives. One way the organization seeks to do this is by, quite simply, talking to people.

In 2023 alone, UNHCR conducted some 122 focus group discussions (FGDs) in El Salvador, with more than 1,300 people. This generated more than 400 hours of audio, which became more than 8,000 pages of transcription. But how do you find time to read 8,000 pages, let alone analyze it?

Finding a solution — after a sharp pivot

To address this challenge, a team of Information Management and Community-Based Protection colleagues in El Salvador explored a potential solution with the support of the Data Innovation Fund. Initially, they planned to use typical data analysis approaches, to organize the unstructured information into data visualizations that quantified terms and themes, to drive evidence-informed programming.

But rapid technological advancements soon after they started their project made clear that this was no longer the most effective approach. “Last year, there was an amazing evolution in artificial intelligence technologies,” says Sebastian Salazar Tapia, Associate Information Management Officer. “It’s remarkable how you can chat with these tools: You can ask questions and the tool gives you answers like a human. But the problem was that these tools answer your questions using all the information on the internet.”

A multistage innovation process

What the El Salvador team needed was a genAI instance that only had access to their core dataset — the FGD transcriptions — and that guaranteed data protection and accuracy. So, they designed a process to gather, clean, and ‘mine’ the data in four main steps:

1. Gathering high-quality recordings: The first crucial step was to ensure the audio recordings of FGDs were of a consistently high quality. The team acquired six professional recorders and developed a set of recording guidelines to support the Protection colleagues who run these meetings. For the pilot, the Information Management team managed the recordings themselves.
    
2. Converting speech to text: Once the team had completed their recordings, they went through a process of reviewing and editing the hundreds of hours of audio, before transforming the speech into text using open-source, AI-supported text transcription tools including Sonix and Whisper. “We got very good results,” Sebastian says, with roughly 80% confidence of accuracy.
    
3. Cleaning and anonymizing the transcriptions: In the FGDs, people often mention the names of community members, locations, institutions. Using a database of common names, in conjunction with named entity recognition (NER) technologies, the team searched the transcriptions to identify personal information and anonymize it, thereby reducing protection risks.
    
4. Mining the data for specific insights: The team’s final dataset was more than 8,000 pages of clean, anonymized transcription. It was ready for analysis.
    

By this time, ChatGPT had been released, and the Azure OpenAI initiative by the Innovation Service and UNHCR’s Division of Information Systems and Telecommunications (DIST) to pilot secure genAI chatbot instances in a variety of use cases was still in its very early stages. The team in El Salvador were often innovating ahead of official recommendations — experimenting with new tools and approaches. “We really wanted to personalize our tool, make changes, and go farther with the possibilities of artificial intelligence,” recalls Sebastian.

So, they experimented with LangChain and open-source large language models (LLMs) to create their own ChatGPT instance — one trained exclusively on their dataset and operated from a personal computer, without recourse to the internet. Sebastian notes:

“It was a lot of work learning how to use this. It was a really new, new, new technology.”

Chatting to the data

The result of this process was a secure, accurate genAI tool run on a device used by the Information Management team. Called ‘SIVAR+’, the tool was fed the many thousands of pages of transcript and, when prompted, could deliver a concise summary of what communities said on a given topic.

In response to a question like ‘What types of support is the community interested in receiving from UNHCR regarding access to employment?’ the tool provided broad information on expressed needs. Asked a more specific question, it provided more specific detail. “The great thing is that you can do this with natural language, as if you were talking with a person, and the answer will be given in the same way,” says Sebastian. Since SIVAR+ could only utilize data from the transcripts, the risk of ‘hallucinations’ was mitigated.

Given that the FGDs are informal, and the conversation jumps around thematically, it’s usually very difficult and time-consuming to gather information on a specific topic. SIVAR+ made this simple. It also enabled users to search specific FGDs, request direct quotations, and identify where in the transcript the information is drawn from. The FGDs are “a great conversation with the community that we have every year,” says Sebastian. SIVAR+ offered “a better way to analyze that information.”

Impact and outlook

The El Salvador operation uses information gleaned from FGDs to develop planning and results reports. Usually, such reports take two or three months to compile, using less detailed summaries of what communities have said. With SIVAR+, this work was completed in less than half the time — a process that was both easier and more accurate, because the tool made it possible to directly access what people said, in various levels of granularity. UNHCR’s Representative in El Salvador, Laura Almirall, says:

“SIVAR+ evolved into an invaluable tool for seamlessly accessing information shared by communities, ensuring that their needs and priorities consistently inform every decision we make.”

The tool not only met a very specific need, but has also informed the ongoing Azure OpenAI collaboration by the Innovation Service and DIST. We’re continuing to explore different ways of responsibly and creatively leveraging generative AI and other AI technologies to further the mission of UNHCR, equipped with learnings from projects like this one.