Building Multi Agent AI Systems: From Orchestration to Production
Building one AI agent is relatively easy.
You give an LLM some instructions, connect a few tools, and let it perform a task.
For example:
1User
2 ↓
3Agent
4 ↓
5LLM
6 ↓
7Tools
8 ↓
9Answer
But imagine the task becomes much larger.
You want an AI system that can:
- Understand a complex request
- Break the request into smaller tasks
- Search the web
- Read internal documents
- Write code
- Analyze data
- Ask another agent for help
- Remember previous work
- Check its own results
- Ask a human before taking risky actions
- Continue working even when one step fails
One agent can try to do everything.
But as the system becomes more complex, it may make sense to divide the work among several specialized agents.
This gives us a multi agent system.
Modern agent frameworks support patterns where a central manager calls specialist agents, or where one agent hands control to another specialist. OpenAI describes both manager based orchestration and agent handoffs as common multi agent patterns.
But adding more agents does not automatically make a system better.
A production multi agent system needs much more than several LLMs talking to each other.
You need:
1Agents
2
3Orchestration
4
5Harness
6
7State
8
9Communication
10
11Tools
12
13Memory
14
15Planning
16
17Routing
18
19Context Management
20
21Guardrails
22
23Evaluation
24
25Human Approval
26
27Observability
28
29Failure Handling
In this article, we will understand how all these pieces work together.
First, What Is an AI Agent?
Let us start with the simplest definition.
An AI agent is usually an LLM connected to instructions and capabilities.
For example:
1 Agent
2 ↓
3 ┌────────┼────────┐
4 ↓ ↓ ↓
5Instructions LLM Tools
OpenAI describes an agent as an LLM configured with instructions, tools, and optional runtime behavior such as handoffs and guardrails.
Suppose we create a database agent.
Its instruction might be:
1You are a database expert.
2
3Help users analyze data.
4
5Use the database tools when necessary.
6
7Never modify production data without approval.
Its tools might include:
1search_schema
2
3run_sql
4
5get_table_metadata
6
7explain_query
The LLM decides when to use those tools.
That is already an agent.
What Is a Multi Agent System?
Now imagine we have several agents.
1Database Agent
2
3Research Agent
4
5Coding Agent
6
7Analytics Agent
8
9Review Agent
Each agent has a specific job.
The system might look like:
1 User
2 ↓
3 Main Agent
4 ↓
5 ┌──────────────┼──────────────┐
6 ↓ ↓ ↓
7 Research Agent Coding Agent Data Agent
8 ↓ ↓ ↓
9 Tools Tools Tools
The main agent does not need to be an expert at everything.
It can ask specialist agents for help.
Think about a company.
A CEO does not personally write every SQL query, design every page, and fix every server.
There are specialists.
1CEO
2 ↓
3Engineering
4Marketing
5Finance
6Legal
7Operations
A multi agent system uses a similar idea.
Do You Always Need Multiple Agents?
No.
This is extremely important.
If one agent can reliably complete the task, use one agent.
More agents mean more:
1LLM calls
2
3Latency
4
5Cost
6
7State
8
9Failure possibilities
10
11Communication
12
13Debugging
14
15Evaluation
Anthropic recommends starting with simpler systems and adding more autonomous agent behavior only when the task actually requires it. Their agent guidance separates predictable workflows from more autonomous agents and encourages using the simplest design that solves the problem.
For example, imagine you want:
1User asks question
2 ↓
3Search database
4 ↓
5Generate answer
You probably do not need five agents.
But imagine you want:
1Research a company
2
3Analyze its financial data
4
5Read recent news
6
7Compare competitors
8
9Create charts
10
11Write a report
12
13Review the report
14
15Verify every important claim
Now specialization may become useful.
The Big Picture
A production multi agent system might look something like this:
1 User
2 ↓
3 API Layer
4 ↓
5 Agent Harness
6 ↓
7 Orchestrator
8 ↓
9 Router
10 ↓
11 ┌────────────────┼────────────────┐
12 ↓ ↓ ↓
13 Research Data Coding
14 Agent Agent Agent
15 ↓ ↓ ↓
16 Tools Tools Tools
17 ↓ ↓ ↓
18 └────────────────┼────────────────┘
19 ↓
20 Shared State
21 ↓
22 Memory
23 ↓
24 Evaluator
25 ↓
26 Guardrails
27 ↓
28 Human Approval
29 ↓
30 Response
Do not worry if this looks complicated.
We will go through each piece.
1. Orchestration
The first major problem is orchestration.
Orchestration answers:
Who should do what, and in what order?
Imagine a user asks:
1Research NVIDIA's latest earnings,
2compare them with AMD,
3analyze the numbers,
4and write a short investment report.
We might have three agents:
1Research Agent
2
3Financial Analysis Agent
4
5Report Agent
Someone needs to coordinate them.
That is orchestration.
The Orchestrator Pattern
One common design uses a main agent.
1 Orchestrator
2 ↓
3 Understand the task
4 ↓
5 Create subtasks
6 ↓
7 ┌───────────────┼───────────────┐
8 ↓ ↓ ↓
9 Research Agent Finance Agent Report Agent
The orchestrator acts like a manager.
Anthropic described a similar orchestrator and worker pattern in its multi agent research system, where a lead agent coordinates several worker agents.
OpenAI also supports a manager style pattern where one agent keeps control and calls specialist agents as tools.
Code Controlled Orchestration
The LLM does not always need to control everything.
You can define the workflow in code.
For example:
1Research
2 ↓
3Analyze
4 ↓
5Review
6 ↓
7Write
This is predictable.
Your application decides the order.
This works well when you already know the correct process.
LLM Controlled Orchestration
Sometimes the correct process depends on the problem.
For example:
1User:
2Find out why our recommendation system performed worse yesterday.
The orchestrator might decide:
1First inspect metrics.
2
3Then check recent deployments.
4
5Then inspect training data.
6
7Then ask the experimentation agent.
8
9Then compare the results.
For another question, it may create a completely different plan.
OpenAI's orchestration guidance describes both approaches: letting an LLM decide what should happen next, or controlling the workflow directly through application code.
Which Should You Use?
A simple rule is:
1Predictable task
2 ↓
3Code controlled workflow
1Unpredictable task
2 ↓
3Agent controlled workflow
You can also combine them.
For example:
1Code controls the major stages
2
3LLM decides what happens inside each stage
This often gives you a useful balance between flexibility and control.
2. The Agent Harness
The agent harness is one of the most important concepts in production agent systems.
The LLM is only the brain.
The harness is everything around the brain that helps it work.
Think about a race car driver.
The driver may be extremely skilled.
But the driver still needs:
1Car
2
3Steering
4
5Brakes
6
7Dashboard
8
9Radio
10
11Safety system
12
13Navigation
14
15Pit crew
The model is the driver.
The harness provides the rest.
A simplified harness might contain:
1 Agent Harness
2 ↓
3 ┌────────────────┼────────────────┐
4 ↓ ↓ ↓
5 Tools State Memory
6 ↓ ↓ ↓
7 Planning Context Guardrails
8 ↓ ↓ ↓
9 Retries Logging Evaluation
Anthropic describes agent harnesses as systems around the model that provide capabilities such as tools, context management, planning, and execution support for long running tasks.
The quality of the harness can make a huge difference.
A very strong model with a poor harness may perform worse than a slightly weaker model with excellent tools, context, and controls.
3. State Management
Now imagine an agent is doing a ten step task.
It has already completed five steps.
Where do we store that information?
That is state management.
Suppose the task is:
1Build a market analysis report.
The state might look conceptually like:
1Task:
2Market analysis
3
4Status:
5In progress
6
7Completed:
8Company research
9Competitor research
10
11Currently running:
12Financial analysis
13
14Remaining:
15Charts
16Report writing
17Review
The system needs this information outside the model.
Why Not Keep Everything Inside the Prompt?
Because the prompt is not your application database.
If the process crashes, you may lose information.
If another agent needs the information, sharing it becomes difficult.
If the task runs for hours, the context may become too large.
Instead, keep important state in a proper storage system.
For example:
1PostgreSQL
2
3Redis
4
5Firestore
6
7Spanner
8
9Object Storage
Then agents can read and update that state.
OpenAI's Agents SDK also separates runtime context and resumable run state, and supports saving state when workflows pause for approvals or other interruptions.
A Shared State Example
Imagine three agents.
1Research Agent
2
3Analysis Agent
4
5Writer Agent
Instead of sending huge messages between them, they update shared state.
1 Shared State
2 ↓
3 ┌───────────────┼───────────────┐
4 ↓ ↓ ↓
5 Research Analysis Writer
6 Agent Agent Agent
The research agent writes:
1research_status = complete
2
3research_document = report_123
The analysis agent reads it and writes:
1analysis_status = complete
2
3analysis_document = analysis_456
The writer then reads both.
This is much cleaner than putting everything into one giant conversation.
4. Communication Between Agents
Now we need agents to communicate.
There are several ways to do this.
Method 1: Direct Handoff
Agent A transfers control to Agent B.
1User
2 ↓
3Agent A
4 ↓
5Agent B
6 ↓
7User
OpenAI calls this a handoff. A handoff allows one agent to delegate the conversation to another specialized agent.
Imagine customer support.
1General Support Agent
2 ↓
3"This is a refund issue"
4 ↓
5Refund Agent
Now the Refund Agent takes responsibility.
Method 2: Agent as a Tool
The main agent stays in control.
1 Main Agent
2 ↓
3 calls Research Agent
4 ↓
5 gets result
6 ↓
7 calls Finance Agent
8 ↓
9 gets result
10 ↓
11 creates answer
OpenAI supports this manager pattern by allowing agents to be exposed as tools to another agent.
This pattern is useful when you want one agent to own the final answer.
Method 3: Shared State
Agents communicate indirectly through storage.
1Agent A
2 ↓
3Shared State
4 ↓
5Agent B
This works well for longer workflows.
Method 4: Events and Queues
For larger systems, agents may communicate through events.
For example:
1Research Agent
2 ↓
3"research.completed"
4 ↓
5Message Queue
6 ↓
7Analysis Agent
You could use systems such as:
1Pub/Sub
2
3Kafka
4
5RabbitMQ
This can be useful when agents run independently or when tasks take a long time.
What Should Agents Send Each Other?
Do not automatically send the entire conversation.
Send the information the next agent actually needs.
For example, instead of:
150,000 tokens of research history
send:
1Research Summary
2
3Important Findings
4
5Sources
6
7Open Questions
8
9Confidence
This saves tokens and keeps the next agent focused.
Anthropic has described subagents as useful for context isolation because they can work in separate context windows and return only relevant information to the orchestrator.
5. Tool Access
Agents become much more useful when they can take actions.
A model without tools can mainly generate text.
A model with tools can do things.
For example:
1Search web
2
3Query database
4
5Read files
6
7Write files
8
9Run Python
10
11Call APIs
12
13Send email
14
15Create ticket
16
17Deploy service
OpenAI describes tools as a way for agents to fetch information, run code, call APIs, interact with computers, and perform other actions.
Different Agents Should Have Different Tools
Imagine this system:
1Research Agent
2
3Database Agent
4
5Deployment Agent
Do not automatically give every tool to every agent.
The Research Agent may need:
1web_search
2
3read_web_page
The Database Agent may need:
1read_schema
2
3run_sql
The Deployment Agent may need:
1deploy_staging
2
3check_logs
4
5restart_service
This makes tool selection easier.
It also improves safety.
The research agent probably should not have permission to delete a production database.
Tool Design Matters
Suppose you give an agent a tool named:
1execute
What does it execute?
The model has little information.
A better tool might be:
1search_customer_orders
with a description explaining:
1Search customer orders using a customer ID.
2
3Use this when the user asks about previous or current orders.
Clear tool names, descriptions, inputs, and outputs help models choose tools more reliably. Anthropic's guidance on agent tools emphasizes that tool interfaces should be designed carefully for the model using them.
6. Planning
Complex tasks often need a plan.
Imagine:
1Build a competitor analysis for five AI companies.
The agent might create:
11. Identify the companies.
2
32. Collect product information.
4
53. Collect pricing.
6
74. Find recent announcements.
8
95. Compare features.
10
116. Analyze strengths and weaknesses.
12
137. Write the report.
14
158. Review the claims.
Now the system has a roadmap.
Why Planning Helps
Without planning, an agent can jump around.
It might:
1Search one company
2
3Start writing
4
5Realize information is missing
6
7Search again
8
9Rewrite everything
10
11Forget another company
A plan gives the task structure.
Plans Should Be Editable
A plan should not always be fixed.
Suppose the agent discovers:
1Company B has been acquired.
The plan may need to change.
So think of planning as:
1Create plan
2 ↓
3Execute step
4 ↓
5Observe result
6 ↓
7Update plan
8 ↓
9Execute next step
Not:
1Create plan once
2 ↓
3Blindly follow forever
7. Routing
Routing answers another important question:
Which agent should receive this task?
Imagine we have:
1Coding Agent
2
3Finance Agent
4
5Support Agent
6
7Research Agent
The user says:
1My invoice amount is incorrect.
The router decides:
1Finance Agent
Another user says:
1Fix this Python function.
The router chooses:
1Coding Agent
A Router Can Be an LLM
You can give an LLM:
1Available agents:
2
3Coding Agent
4Finance Agent
5Research Agent
6Support Agent
Then ask it to select the best one.
A Router Can Also Be Code
Some routing is simple enough for rules.
For example:
1refund request
2 ↓
3Refund Agent
1account password
2 ↓
3Account Agent
If routing can be reliable with simple logic, you may not need another LLM call.
Routing Can Be Hierarchical
Large systems may have many agents.
Imagine 100 specialist agents.
One router should probably not choose directly among all 100.
You might use:
1 Main Router
2 ↓
3 ┌───────────────┼───────────────┐
4 ↓ ↓ ↓
5 Engineering Business Support
6 Router Router Router
7 ↓ ↓ ↓
8 Specialists Specialists Specialists
This creates a hierarchy.
8. Memory
State and memory sound similar, but they solve different problems.
State usually describes the current task.
Memory stores useful information from the past.
For example:
1State:
2
3Current task is analyzing Q2 revenue.
Memory:
1The user usually wants revenue broken down by region.
Memory can be divided into several useful types.
1Semantic Memory
2
3Facts and knowledge
1Episodic Memory
2
3Previous events and experiences
1Procedural Memory
2
3How tasks should be performed
Agent systems can combine tools, retrieval, and memory so the model has useful information beyond its immediate prompt.
Example
Suppose a user says:
1Analyze this experiment.
Semantic memory might provide:
1The team's primary metric is conversion rate.
Episodic memory might provide:
1Last month's experiment had a logging problem on Android.
Procedural memory might provide:
1Always check sample ratio mismatch before evaluating experiment results.
Together, these memories can help the agent make better decisions.
9. Context Management
This is one of the hardest problems in agent systems.
LLMs have limited context.
Imagine a long running agent generates:
1User messages
2
3Agent thoughts
4
5Tool calls
6
7Tool outputs
8
9Search results
10
11Documents
12
13Subagent results
14
15Logs
16
17Plans
18
19Memories
After enough work, the context becomes enormous.
You cannot simply keep adding everything forever.
Context Is Like a Desk
Imagine you are working at a desk.
You need the documents related to your current task.
If someone puts 10,000 random documents on your desk, having more information does not necessarily help.
It may make your work harder.
An agent has a similar problem.
Good context management means deciding:
1What should stay?
2
3What should be removed?
4
5What should be summarized?
6
7What should be stored externally?
8
9What should be retrieved later?
Anthropic's context engineering guidance recommends keeping the model's working context focused on relevant information and using external persistence or summaries for information that does not need to remain in the immediate context.
Context Compaction
Suppose the current context contains:
140,000 tokens
Much of it describes completed work.
You can summarize it.
140,000 tokens
2
3 ↓
4
5Summary
6
7 ↓
8
94,000 tokens
The summary may contain:
1Completed tasks
2
3Important decisions
4
5Important results
6
7Current plan
8
9Remaining tasks
Anthropic has described compaction and external memory as ways for long running agents to continue working without keeping every previous token in active context.
Subagents Can Help With Context
Imagine your main agent is researching five companies.
Instead of putting all research into one context:
1Main Agent
2
3Company A
4Company B
5Company C
6Company D
7Company E
create separate subagents.
1 Main Agent
2 ↓
3 ┌────────────┼────────────┐
4 ↓ ↓ ↓
5 Agent A Agent B Agent C
6 Company A Company B Company C
Each agent gets its own context.
The main agent receives summaries.
This can reduce context pressure and allow independent tasks to run at the same time.
10. Guardrails
Giving agents tools creates risk.
Imagine an agent can:
1Send emails
2
3Delete files
4
5Update databases
6
7Deploy code
8
9Issue refunds
You do not want the model to perform every action without checks.
This is where guardrails become useful.
OpenAI describes guardrails as checks and validations that can run on agent inputs and outputs.
Input Guardrails
Check what enters the system.
For example:
1User Request
2 ↓
3Input Guardrail
4 ↓
5Agent
The guardrail might detect:
1Unsupported request
2
3Malicious input
4
5Sensitive information
6
7Invalid parameters
Output Guardrails
Check what the agent produces.
1Agent Output
2 ↓
3Output Guardrail
4 ↓
5User
You might check:
1Does the answer contain private information?
2
3Does it follow the required structure?
4
5Did the agent invent data?
6
7Is a required field missing?
Tool Guardrails
Tool actions deserve special attention.
For example:
1Agent wants to:
2
3DELETE production database
The system should not simply execute that action.
You can place checks before the tool call.
1Agent
2 ↓
3Tool Request
4 ↓
5Permission Check
6 ↓
7Approval
8 ↓
9Execute
11. Human in the Loop
Some decisions should involve a person.
Imagine an agent wants to:
1Send $50,000
2
3Delete a production database
4
5Publish a public statement
6
7Deploy a major production change
8
9Cancel a customer's subscription
Even if the agent is confident, you may want human approval.
This is called human in the loop.
Modern agent systems can pause a run before a sensitive tool executes, ask a human to approve or reject the action, save the run state, and continue later. OpenAI's Agents SDK provides this approval and resume pattern.
Example
1Agent:
2
3I want to issue a $2,000 refund.
4
5 ↓
6
7System pauses.
8
9 ↓
10
11Human:
12
13Approve
14or
15Reject
16
17 ↓
18
19Agent continues.
This gives the agent autonomy for normal work while keeping humans involved in important decisions.
Not Every Action Needs Approval
If every action requires approval:
1Search web
2
3Approve?
4
5Read document
6
7Approve?
8
9Run simple query
10
11Approve?
12
13Generate summary
14
15Approve?
The system becomes frustrating.
Instead, use approval based on risk.
For example:
1Read database
2 ↓
3No approval
1Modify production database
2 ↓
3Approval required
1Draft email
2 ↓
3No approval
1Send email
2 ↓
3Approval required
This is much more practical.
12. Evaluation
This is where many agent projects become difficult.
A normal software function is easy to test.
12 + 2
Expected:
14
An agent can take many different paths and still produce a good result.
For example:
1Task:
2Research a company.
Agent A might use five searches.
Agent B might use eight searches.
Agent C might use three searches and two database queries.
All three may produce correct reports.
So evaluating agents requires more than checking one final string.
Anthropic's agent evaluation guidance emphasizes evaluating both outcomes and agent behavior, because agents may take many turns, call tools, modify state, and adapt based on intermediate results.
What Should You Evaluate?
You can evaluate several layers.
Final Answer Quality
1Was the answer correct?
2
3Was it complete?
4
5Did it follow the instructions?
Tool Selection
1Did the agent choose the right tool?
Tool Arguments
1Did the agent send correct parameters?
Routing
1Did the request go to the correct specialist?
Planning
1Did the agent create a useful plan?
Memory
1Did it retrieve the right memories?
Safety
1Did it avoid dangerous actions?
Efficiency
1How many LLM calls?
2
3How many tool calls?
4
5How many tokens?
6
7How much time?
8
9How much cost?
Evaluate the Journey, Not Only the Destination
Imagine an agent gives the correct answer.
But internally it:
1Called 30 unnecessary tools
2
3Used 500,000 tokens
4
5Failed six times
6
7Accidentally modified data
8
9Recovered by luck
The final answer alone does not tell you whether this is a good agent.
You also need to inspect the path it took.
13. Evaluator Agents
You can sometimes use another model to evaluate the result.
For example:
1Research Agent
2 ↓
3Creates report
4 ↓
5Evaluator Agent
6 ↓
7Checks report
8 ↓
9Feedback
10 ↓
11Research Agent improves report
This creates a generator and evaluator pattern.
Anthropic has also experimented with multi agent designs where one agent generates work and another evaluates it.
But the evaluator is also an LLM.
It can make mistakes.
So do not assume:
1LLM evaluated it
2=
3It must be correct
Use deterministic tests whenever possible.
For example:
1Schema validation
2
3SQL tests
4
5Unit tests
6
7Required fields
8
9Numeric checks
10
11Permission checks
Then use LLM evaluation for things that are difficult to measure with simple rules.
14. Observability
When a multi agent system fails, you need to understand why.
Imagine the user gets a bad answer.
What happened?
Maybe:
1Router selected wrong agent
2
3Agent selected wrong tool
4
5Tool returned bad data
6
7Context was missing
8
9Memory retrieval failed
10
11Agent misunderstood tool output
12
13Evaluator approved a bad result
Without observability, debugging becomes guessing.
You should record useful information such as:
1Request ID
2
3Agent used
4
5Model used
6
7Prompt version
8
9Tool calls
10
11Tool results
12
13Handoffs
14
15State changes
16
17Memory retrieval
18
19Latency
20
21Token usage
22
23Cost
24
25Errors
26
27Final result
Think about the entire workflow as a trace.
1User Request
2
3 ↓ 200 ms
4
5Router
6
7 ↓
8
9Research Agent
10
11 ↓ 1.4 sec
12
13Search Tool
14
15 ↓
16
17Research Agent
18
19 ↓
20
21Analysis Agent
22
23 ↓ 800 ms
24
25Python Tool
26
27 ↓
28
29Evaluator
30
31 ↓
32
33Final Answer
Now you can see where time and failures occur.
15. Failure Handling
Agents will fail.
Tools will fail.
APIs will time out.
Models may return invalid output.
Workers may crash.
Your architecture should expect this.
Retries
Suppose an API temporarily fails.
1Tool call
2 ↓
3Error
4 ↓
5Retry
But do not retry everything forever.
Use limits.
1Attempt 1
2
3Attempt 2
4
5Attempt 3
6
7Stop
Be Careful With Actions
Imagine:
1Agent calls:
2
3send_payment($1000)
The request succeeds.
But the response is lost.
The agent thinks it failed and retries.
Now:
1Payment 1 = $1000
2
3Payment 2 = $1000
That is a serious problem.
For important actions, design tools so repeated calls do not accidentally repeat the action.
For example, use a unique operation ID.
1payment_request_id = abc123
If abc123 was already processed, the system does not process it again.
Save Progress
Long running agents should not restart from zero whenever something crashes.
Imagine:
1Task has 20 steps
2
318 completed
4
5Server crashes
Bad design:
1Start again at Step 1
Better design:
1Load saved state
2
3Continue from Step 19
OpenAI documents durable execution integrations for agent runs that may include long waits, retries, restarts, and human approval steps.
16. Parallel Agents
Multiple agents become especially useful when tasks are independent.
Imagine you need research about four companies.
You could do:
1Company A
2 ↓
3Company B
4 ↓
5Company C
6 ↓
7Company D
That is sequential.
Or:
1 Orchestrator
2 ↓
3 ┌──────────┼──────────┐
4 ↓ ↓ ↓
5 Company A Company B Company C
6 ↓
7 Company D
Several research tasks can happen at the same time.
Anthropic has described parallel subagents as one of the benefits of multi agent architectures, particularly when separate parts of a task can be investigated independently.
But Parallelism Has a Cost
If ten agents run simultaneously, you may also create:
110 model requests
2
3More tokens
4
5More API calls
6
7More database traffic
8
9More memory usage
10
11More cost
Use parallel agents where parallel work actually makes sense.
17. Context Boundaries Between Agents
One useful design principle is:
Each agent should receive only the context it needs.
Imagine a software engineering system.
You have:
1Frontend Agent
2
3Backend Agent
4
5Database Agent
6
7Security Agent
The frontend agent may need:
1UI requirements
2
3Design system
4
5Frontend code
6
7API specification
It probably does not need:
1Every database migration
2
3Every security log
4
5Every backend test result
Smaller focused contexts can make specialized agents easier to control.
18. Give Agents Clear Responsibilities
Bad agent design:
1Agent 1:
2Help with stuff.
3
4Agent 2:
5Also help with stuff.
6
7Agent 3:
8Do whatever is needed.
Now the agents overlap.
Routing becomes difficult.
Good design:
1Research Agent
2
3Responsibility:
4Collect and verify information.
5
6Tools:
7Web search
8Document search
1Data Agent
2
3Responsibility:
4Analyze structured data.
5
6Tools:
7SQL
8Python
9BigQuery
1Writer Agent
2
3Responsibility:
4Turn verified findings into a clear report.
5
6Tools:
7Document editor
Clear boundaries make the system easier to understand and evaluate.
19. A Full Example
Let us build a multi agent system for business analysis.
The user asks:
1Analyze why our revenue decreased this month
2and prepare a report for leadership.
We have these agents:
1Orchestrator
2
3Analytics Agent
4
5Experiment Agent
6
7Research Agent
8
9Writer Agent
10
11Reviewer Agent
Step 1: Request Arrives
1User
2 ↓
3Analyze why revenue decreased.
Step 2: Orchestrator Understands the Goal
The orchestrator creates a plan.
11. Check revenue metrics.
2
32. Find which segments declined.
4
53. Check recent product experiments.
6
74. Check known incidents.
8
95. Identify likely causes.
10
116. Write report.
12
137. Review claims.
Step 3: Router Sends Work
1 Orchestrator
2 ↓
3 ┌───────────────┼───────────────┐
4 ↓ ↓ ↓
5 Analytics Agent Experiment Agent Research Agent
Step 4: Agents Use Tools
Analytics Agent:
1BigQuery
2
3Python
4
5Metrics API
Experiment Agent:
1Experiment database
2
3Feature flag system
Research Agent:
1Incident logs
2
3Internal documents
Step 5: Agents Update Shared State
1Analytics:
2
3Revenue decreased 12%.
4
5Largest decline came from mobile users in Germany.
1Experiments:
2
3New checkout experiment launched seven days ago.
1Research:
2
3Payment failures increased after a payment provider change.
Step 6: Orchestrator Combines Results
Now the orchestrator sees:
1Revenue decline
2
3Mobile Germany decline
4
5Checkout experiment
6
7Payment failures
It may ask the Analytics Agent:
1Compare payment failure rate before and after
2the checkout experiment.
This is important.
Agent systems are not always one straight line.
They can create new tasks after discovering new information.
Step 7: Writer Creates the Report
The Writer Agent receives only the useful findings.
1Verified metrics
2
3Important events
4
5Likely causes
6
7Supporting evidence
Then creates the report.
Step 8: Reviewer Checks It
1Report
2 ↓
3Reviewer Agent
4 ↓
5Check claims
6 ↓
7Check numbers
8 ↓
9Check unsupported conclusions
Step 9: Human Reviews Important Conclusions
Before the report goes to leadership:
1AI Report
2 ↓
3Human Review
4 ↓
5Approved
6 ↓
7Leadership
Now we have a full multi agent workflow.
The Architecture
The final architecture could look like:
1 User
2 ↓
3 API
4 ↓
5 Harness
6 ↓
7 Orchestrator
8 ↓
9 Router
10 ↓
11 ┌─────────────────┼─────────────────┐
12 ↓ ↓ ↓
13 Analytics Research Experiment
14 Agent Agent Agent
15 ↓ ↓ ↓
16 Tools Tools Tools
17 ↓ ↓ ↓
18 └─────────────────┼─────────────────┘
19 ↓
20 Shared State
21 ↓
22 Memory
23 ↓
24 Writer Agent
25 ↓
26 Reviewer Agent
27 ↓
28 Guardrails
29 ↓
30 Human Approval
31 ↓
32 Final Output
Around everything, we also need:
1Logging
2
3Tracing
4
5Evaluation
6
7Retries
8
9Permissions
10
11Cost Monitoring
A Simple Mental Model
If this article feels like a lot of information, remember this analogy.
Imagine a company.
The agents are employees.
1Researcher
2
3Engineer
4
5Analyst
6
7Writer
The orchestrator is the manager.
1Who should work on what?
The router is the receptionist.
1Who should receive this request?
The tools are the software employees use.
1Database
2
3Browser
4
5Python
6
7Email
The state is the project board.
1What are we currently doing?
The memory is the company's knowledge.
1What have we learned before?
The context is the information currently sitting on an employee's desk.
1What does this person need right now?
The guardrails are company rules.
1What are employees allowed to do?
The human approval system is management authorization.
1Does a person need to approve this action?
The evaluation system is quality control.
1Did we do the work correctly?
The observability system is the activity log.
1What happened and why?
The harness is the operating environment that connects all these pieces.
What Makes a Good Multi Agent System?
A good multi agent system is not the system with the most agents.
It is the system where responsibilities are clear.
You want something like:
1Clear Agents
2
3 ↓
4
5Clear Responsibilities
6
7 ↓
8
9Clear Tool Access
10
11 ↓
12
13Good Routing
14
15 ↓
16
17Controlled Context
18
19 ↓
20
21Reliable State
22
23 ↓
24
25Useful Memory
26
27 ↓
28
29Safe Actions
30
31 ↓
32
33Strong Evaluation
34
35 ↓
36
37Observable Behavior
Common Mistake 1: Creating Too Many Agents
It is easy to create:
1Planning Agent
2
3Thinking Agent
4
5Research Agent
6
7Search Agent
8
9Web Agent
10
11Review Agent
12
13Quality Agent
14
15Supervisor Agent
16
17Manager Agent
18
19Manager of Managers Agent
The architecture looks impressive.
But it may perform worse.
Every extra agent creates another point where information can be lost or misunderstood.
Start small.
For example:
1Orchestrator
2
3Specialist A
4
5Specialist B
Add another agent only when you can explain exactly why it is needed.
Common Mistake 2: Letting Every Agent See Everything
More context does not always mean better performance.
Do not send:
1Entire conversation
2
3Entire database schema
4
5All available tools
6
7Every previous agent message
8
9All memories
to every agent.
Retrieve what is relevant.
Common Mistake 3: Giving Agents Too Much Power
Do not give every agent:
1Database delete access
2
3Production deployment access
4
5Email sending access
6
7Payment access
Use the minimum permissions each agent needs.
Common Mistake 4: No Evaluation
A demo working three times does not mean the system is ready.
Build an evaluation set.
For example:
1100 routing examples
2
3100 tool selection examples
4
550 failure scenarios
6
750 memory retrieval examples
8
9100 end to end tasks
Run them whenever you change:
1Model
2
3Prompt
4
5Tools
6
7Router
8
9Memory
10
11Agent architecture
Common Mistake 5: No Trace of What Happened
If the final response is wrong and all you saved is:
1Final answer
debugging will be painful.
You want to know:
1Which agent ran?
2
3Why was it selected?
4
5Which tools were called?
6
7What did the tools return?
8
9Which memories were used?
10
11What state changed?
12
13Where did the failure begin?
When Should You Use Multi Agent Systems?
Multi agent systems make the most sense when the problem naturally contains separate areas of expertise or independent work.
For example:
1Deep research
2
3Software development
4
5Business analysis
6
7Data science
8
9Customer support
10
11Security investigation
12
13Complex operations
14
15Long running workflows
They can also help when separate tasks can run in parallel or when separate contexts prevent one agent from becoming overloaded.
But if your task is:
1Question
2 ↓
3Search
4 ↓
5Answer
keep it simple.
Final Takeaway
Building a multi agent system is not mainly about connecting several LLMs.
The difficult part is building the system around the LLMs.
You need to answer questions such as:
1Who decides what happens next?
2
3Which agent owns each task?
4
5How do agents communicate?
6
7Where is task state stored?
8
9Which tools can each agent use?
10
11What should the system remember?
12
13What context should each agent receive?
14
15How should the system create and update plans?
16
17How should tasks be routed?
18
19Which actions need protection?
20
21When should a human approve something?
22
23How do we measure whether the agents are good?
24
25How do we understand failures?
That is why the harness matters so much.
The LLM provides intelligence.
The harness provides structure.
A simple way to remember the whole system is:
1 User
2 ↓
3 Harness
4 ↓
5 Orchestrator
6 ↓
7 Router
8 ↓
9 ┌────────────┼────────────┐
10 ↓ ↓ ↓
11 Agent A Agent B Agent C
12 ↓ ↓ ↓
13 Tools Tools Tools
14 ↓ ↓ ↓
15 └────────────┼────────────┘
16 ↓
17 State
18 ↓
19 Memory
20 ↓
21 Review
22 ↓
23 Guardrails
24 ↓
25 Human Approval
26 ↓
27 Output
And surrounding the whole system:
1Context Management
2
3Evaluation
4
5Tracing
6
7Permissions
8
9Retries
10
11Cost Control
The goal is not to create as many agents as possible.
The goal is to create the smallest group of agents that can work together reliably to solve a problem that one agent cannot solve well enough on its own.
That is the foundation of a strong multi agent AI system.