Giving a voice agent hands with MCP

A voice agent that can only talk is a nice demo. A voice agent that can do things during the call is a product. On VYVA, a voice companion for senior care, my job was the second part: the tools an ElevenLabs agent calls while it is on the phone with someone.

The backend is FastAPI on PostgreSQL. Each capability the agent needs is exposed as an MCP tool, so the agent can look up a medication, change it or start a brain-coach activity without the person ever touching a screen.

One tool, one job

The medication tools are a good example. Rather than one “manage medications” tool with a mode switch, there are separate tools to add a medication, update one and end one. That sounds like more work, but it pays off in a live conversation:

• The agent picks the right tool from its name and description alone, so narrow tools with plain-English descriptions get chosen correctly far more often.
• Each tool validates a small, typed input, so a misheard dose fails loudly instead of quietly writing bad data.
• When something goes wrong, the call log tells you exactly which action the agent tried.

The description is the agent’s only documentation. I write it the way I would brief a new colleague: what the tool does, when to use it, and what it needs.

The call does not end when the agent hangs up

Reminder calls taught me that a voice feature is really a workflow. Placing the call is the easy bit. You also need to know whether it connected, whether the person answered, and what to do if they didn’t. I added a call-status check and fixed the medication reminder flow so the system acts on what actually happened, not on what was scheduled.

Time zones turned out to matter more than expected. A reminder at “9 in the morning” means nothing until you know whose morning, so I added time zone support to user records and updated the call-logging tool to match.

Conversations have follow-ups

The brain coach runs cognitive exercises, trivia and memory games. The first version answered one request at a time. Real conversations don’t work like that: people finish an activity and want another one straight away. I worked on a second version of the question-retrieval flow that supports follow-up activities in the same call, so the agent keeps the session going instead of starting over.

Boring infrastructure, real reliability

The brain-coach tables started out managed by raw SQL scripts. I replaced them with Alembic migrations. It is not glamorous, but when an agent depends on a schema in production, every change needs to be versioned, repeatable and reversible.

What I would tell anyone starting out

1. Design tools around the actions a person would ask for out loud, not around your database tables.
2. Treat every tool description as a prompt, because it is one.
3. Follow each action through to its real-world outcome: the call connected, the dose changed, the activity finished.
4. Put the unglamorous parts, such as migrations, time zones and logging, in place early. Voice makes every failure audible.

The full case study, including the parts I built, is on the VYVA project page.