Multilingual Voice AI Agent

The challenge

Front-desk teams handle the same routine calls all day, often after hours and across languages. Callers wait on hold, and simple requests consume staff time.

The objective

Answer calls around the clock with a natural-sounding agent that handles bookings and guest questions on its own, recognizes repeat callers, leaves a structured summary for staff and escalates anything it should not handle.

Our approach

A streaming voice pipeline runs over real telephony. Speech is transcribed as the caller talks, grounded against a knowledge base, answered by an LLM and spoken back sentence by sentence so the caller hears a reply quickly. Callers are identified by phone number, older turns are compressed into an LLM-written memory summary, and when a call ends a structured booking record is extracted and a confirmation email is sent.

How it works

1. 01

   Caller dials in via Telnyx and audio streams to the agent in real time.

2. 02

   Whisper transcribes each turn and the caller is recognized by number.

3. 03

   Relevant facts are retrieved from the knowledge base (RAG).

4. 04

   An LLM generates the reply, streamed back as speech.

5. 05

   On hang-up, a booking record is extracted and a confirmation is emailed.

The build

Telephony & voice

Telnyx Call Control, media streaming, streaming TTS

Speech

faster-whisper, Groq Whisper large-v3-turbo

LLM

Groq, Fireworks, Ollama (provider-swappable)

RAG

Qdrant, BGE-small embeddings

Backend & dashboard

Python, FastAPI, WebSockets, SQLite, Nuxt 3, Docker Compose

The result