Most people building AI agents stop at the chat interface. They set up a bot, embed it on a website, and call it done. I did the same thing for a while, and honestly, it felt like leaving money on the table.
A phone number changes everything. People trust a number they can call. It feels real in a way that a chat widget never does.
I put this together after spending way too long looking for a clear walkthrough that didn’t assume I already knew five other tools.
The setup I landed on uses ElevenLabs for the voice agent and Twilio for the phone number. The whole thing took me about 10 minutes once I stopped second-guessing myself, and the free credits from Twilio covered the cost of getting started.
What you end up with is an AI assistant that picks up calls, answers questions about your business, and pulls from whatever you feed its knowledge base.
If someone calls at 2 am asking about your services, the agent handles it. No voicemail, no missed leads, no follow-up required from you.
This guide walks through every step.
What You Need Before You Start
The setup is straightforward, but having everything ready before you begin saves you from bouncing between tabs mid-process. Here is what you need:
- An ElevenLabs account (free tier works to start)
- A Twilio account ($15 in free credits on signup, no card required initially)
- Your website URL and any business info you want the agent to know
- About 10 minutes
The free credits from Twilio are enough to buy a phone number and run test calls.
A standard US number costs around $1, so you have plenty of runway to get everything working before spending anything real.
How to Set Up Your Voice Agent in ElevenLabs
This is where your assistant gets its personality, its voice, and its knowledge. ElevenLabs does the heavy lifting here.
You are building the brain before you give it a way to receive calls.
Step 1: Create a new agent
Log into ElevenLabs and go to the Agents section. Select New Agent, then choose the Personal Assistant template.
This gives you a solid base to work from without starting from scratch.
Step 2: Configure the basics
Pick a voice from the library. Take a minute here because this is what callers will hear. Set your time zone and write a first message.
That first message is what the agent says when someone calls, so keep it natural. Something like “Hey, thanks for calling. How can I help you today?” works better than anything that sounds scripted.
Step 3: Build the knowledge base
Add your website URL and any supporting documents you want the agent to draw from. This is what the agent references when someone asks about your services, pricing, hours, or anything else.
The knowledge base is static, meaning it does not update automatically, so make sure the information you add is accurate and current.
| Knowledge Base Input | Best Used For |
|---|---|
| Website URL | General business info, services, FAQs |
| PDF or doc upload | Pricing sheets, menus, detailed guides |
| Plain text | Custom scripts, specific instructions |
| Webhook (via Make.com or n8n) | Live data like calendar events or inventory |
For most use cases, the website URL alone gets you 80% of the way there.
If you need the agent to pull live calendar data or real-time availability, the webhook option is worth setting up, and we cover that later.
How to Get a Phone Number Through Twilio
Twilio is where your agent gets its actual phone number.
The platform can look intimidating at first glance, but you only need three things from it: a phone number, an account SID, and an auth token.
Step 1: Create your Twilio account
Go to Twilio and sign up. You get $15 in free credits automatically. No card is required to get started, and that credit is enough to buy a number and run plenty of test calls.
Step 2: Buy a phone number
From the Twilio dashboard, go to Phone Numbers and search for an available number in your country or area code. A standard US number costs roughly $1 per month. Select one and confirm the purchase.
Step 3: Grab your credentials
Stay on the dashboard. You need three pieces of information to connect Twilio to ElevenLabs:
- Account SID — found on the main dashboard homepage
- Auth Token — right next to the Account SID, click to reveal it
- Phone number — the number you just purchased, in E.164 format (example: +12025551234)
Copy all three somewhere handy. You will paste them into ElevenLabs in the next step.
| Twilio Credential | Where to Find It |
|---|---|
| Account SID | Twilio dashboard homepage |
| Auth Token | Twilio dashboard homepage (click to reveal) |
| Phone Number | Phone Numbers section, in E.164 format |
Do not share your auth token publicly. It gives full access to your Twilio account, so treat it the same way you would a password.
How to Connect Your Twilio Number to Your ElevenLabs Agent
This is the step where everything comes together. You are linking the phone number you bought to the agent you built, so that when someone calls, ElevenLabs picks it up and your agent responds.
Step 1: Go to Phone Numbers in ElevenLabs
Inside your ElevenLabs account, find the Phone Numbers section in the left navigation. This is where you manage any numbers connected to your agents.
Step 2: Import from Twilio
Select Import from Twilio. You will see three fields waiting for the credentials you copied from your Twilio dashboard.
- Paste your Account SID into the first field
- Paste your Auth Token into the second field
- Paste your phone number in E.164 format into the third field
Step 3: Assign your agent
Once ElevenLabs verifies the credentials, you will be prompted to assign an agent to the number. Select the Personal Assistant you set up earlier. Save the configuration.
Step 4: Make a test call
Call the number from your phone. Your agent should pick up within a couple of seconds and deliver the first message you wrote during setup.
Ask it something about your business and see how it responds based on the knowledge base you built.
| Step | Action | Where |
|---|---|---|
| 1 | Navigate to Phone Numbers | ElevenLabs left nav |
| 2 | Select Import from Twilio | Phone Numbers section |
| 3 | Paste SID, token, and number | Import form fields |
| 4 | Assign agent to the number | Dropdown after verification |
| 5 | Call the number to test | Your phone |
If the agent does not answer or gives an error, double-check that the phone number is in E.164 format and that the auth token was copied without any extra spaces.
Those are the two most common issues I ran into during setup.
How to Add Live Data to Your AI Agent with a Webhook
The knowledge base you built in ElevenLabs is static. It knows what you told it, and that is it.
For most business info like services, FAQs, and general details, that is perfectly fine. Where it falls short is anything time-sensitive, like your calendar availability, current stock, or live appointment slots.
This is where a webhook changes the picture. Connecting your agent to a tool like Make.com or n8n lets it pull real-time data on demand, so when a caller asks if you are free Thursday at 2pm, the agent actually checks instead of guessing.
Step 1: Choose your automation tool
Make.com is the easier starting point if you have not worked with webhooks before. n8n is a good option if you prefer open-source or want to self-host.
Both support calendar integrations and can feed data back to ElevenLabs.
Step 2: Build the webhook scenario
In Make.com, create a new scenario and add a Webhook module as the trigger. Copy the webhook URL it generates.
In n8n, create a new workflow and add a Webhook node to get the same result.
Step 3: Connect your calendar or data source
Add a Google Calendar module (or whichever calendar you use) after the webhook trigger.
Configure it to return available slots or upcoming events based on the incoming request from your agent.
Step 4: Add the webhook to ElevenLabs
Back in your ElevenLabs agent settings, find the Tools or Webhook section and paste the URL from Make.com or n8n.
Define what the agent should send when it hits the webhook, typically the caller’s question or a structured query, and what it expects to receive back.
| Use Case | Data Source | Automation Tool |
|---|---|---|
| Calendar availability | Google Calendar | Make.com or n8n |
| Appointment booking | Calendly or Cal.com | Make.com |
| Live inventory | Google Sheets or Airtable | Make.com or n8n |
| CRM lookup | HubSpot or Notion | n8n |
The webhook setup takes longer than the initial 10-minute build, but it is worth it if your business runs on scheduling or availability.
Without it, your agent is answering from a snapshot. With it, the agent is actually useful for anything time-sensitive.
At RoboRhythms.com we have covered a range of AI agent setups, and the ones that get the most traction are always the ones built around real workflows rather than demos. This is one of those setups.
