How to build an AI phone agent you can call

Short answer: build an agent that can reach your calendar, email and CRM, then put a phone number in front of it so you can call it like a person. You ring the number, ask for what you need in plain speech, and it reads back the answer or does the task. It works from the car, on a walk, anywhere you are away from a desk.

Every AI agent I had built lived on my computer. Which is useful right up until I leave my computer, and most of my week happens away from a desk. In the car, walking, between meetings.

So I gave one a phone number. I call it Jarvis, and I ring it the way I would ring a person.

What it actually does

This is the demo, start to finish, in one call:

• "What is on my calendar today?" It reads back the day. Book club at six, Q3 financials at half eight, a client at nine.
• "Any new emails?" Six unread, with what matters in them.
• "How many leads in my CRM are ready to book?" Eight, all in phase two, discovery.
• "I will send you a link to some London cafes. Research their opening hours and locations and build me an itinerary." It takes the link, does the research, files it in Notion.
• "Text me my schedule for the rest of the day." The message arrives while we are still talking.

One phone call, no screen. That is the part that changes how the day feels.

The four pieces you need

Whatever tools you choose, a phone callable agent is always these four parts. Get the shape right and the specific products become a preference rather than a decision.

1. A number to call

A dedicated number that routes to software rather than a handset. Any of the voice platforms will sell you one for a few dollars a month. Save it as a contact so you can say "call Jarvis" rather than dialling.

2. A voice layer

Something that answers, turns your speech into text, and speaks the reply back. This is where the experience is won or lost. Test it on speakerphone in a moving car before you commit, because that is the real use case, and most demos are recorded in a silent room.

3. The brain

Claude, holding the instructions about who you are, how you talk, and what it is allowed to do. This is the same thing as any other agent you have set up. The voice layer is just a different door into it.

4. The connectors

Calendar, email, CRM, Notion. Each one you connect adds a question you can ask. My rule: connect one, use it for a week, then add the next. An agent that can reach eight systems badly is worse than one that reaches two properly.

What I would tell you before you build it

Start with three questions, not thirty. Mine were the calendar, the inbox and the CRM. Everything else came later, and only because I kept reaching for it.

Make it talk like a person. The first version read out my calendar like a database. Useless while driving. Now it gives me the shape of the day and I ask for detail if I want it.

Decide what it may do without asking. Reading is safe. Sending is not. Mine reads freely, drafts freely, and sends nothing without me saying so, except texts to my own number.

Latency matters more than intelligence here. A brilliant answer four seconds later feels broken on a phone call. A good answer in one second feels like magic.

How to build an AI phone agent with Claude

You can have something you can ring answering your calendar questions in a weekend, and the build belongs in Claude Code because it has to keep answering when nobody is at a laptop.

Use Claude Code. A phone agent runs unattended, holds API keys for the voice platform and your connectors, and has to respond in under a second. That is Code's job, not Cowork's. You will still use Cowork for the writing and research side of your week.

Before you start you need a Claude account with Code, an account with a voice platform that will sell you a number, and at least one system worth asking about, which for most people is the calendar.

1. Buy a number from a voice platform and save it in your phone as a contact, so you can say "call Jarvis" rather than dialling.
2. Connect the voice layer to Claude and test it on speakerphone in a moving car, because that is the real use case and every demo is recorded in a silent room.
3. Connect ONE system first. The calendar. Use it for a week before you add anything else.
4. Write down what it may do without asking you, and what it must never do.
5. Give it the prompt below, then ring it and ask it three things.
6. Add the inbox, then the CRM, a week apart, only if you keep reaching for them.

Build me a phone callable assistant.

How it should sound: like a colleague giving me the shape of things, not a database reading out rows. I am usually driving or walking when I call.

Who I am: [WHAT YOU DO, IN ONE LINE]
What I will ask it most: [YOUR THREE MOST COMMON QUESTIONS]

Connect to [CALENDAR / EMAIL / CRM / NOTES] and nothing else for now.

What it may do on its own: read anything, draft anything, text ME.
What it must never do without asking: send an email, message anyone else, change or delete a record, spend money.

Rules:
- Answer in under a second even if the answer is shorter. Speed matters more than completeness on a phone call.
- Give me the shape first and the detail only if I ask.
- If you are not sure, say so out loud rather than guessing.

Ask me anything you still need. Then set up the calendar only, and tell me what to say when I ring it the first time.

Expect latency to be the thing that makes or breaks it. A brilliant answer four seconds later feels broken on a call, and a good answer in one second feels like magic. Start with three questions, not thirty.

Frequently asked questions

Do I need to be technical to build this?

You need to be willing to wire four things together and test them. If you have set up a Claude agent with connectors before, this is a step up in fiddliness, not a different sport.

What does it cost to run?

The number and the voice minutes are the only real running costs, and they are small at personal volume. The thinking runs on the Claude subscription you already pay for.

Is it safe to give an agent my email and CRM?

It has the access you grant and no more. Start read only. Add permission to act once you have watched it work for a week, and never give it send rights to clients until you have read a hundred of its drafts.

Can it call me, rather than me calling it?

Yes, and that is the next thing worth building: a morning call with the day ahead, rather than you remembering to ask.

If you want this built for your business rather than as a weekend project, that is what we do in a workshop.

Related reading

If you want the same agent to run on a schedule rather than only when you call it, read how to automate repeat work with Claude. To give it somewhere to file what it finds, see building your own CRM in Notion.

If you want help working out which parts of your business are worth handing over, have a look at how I work.