Operate the Agent → Control the Agent → Understand Its Economics

An AI agent is not production-ready because it can complete a demo.

It becomes production-ready when three things are true:

1. The team can operate it.
2. The business can control it.
3. The company understands its economics.

Those three requirements belong together.

Operate without control, and the agent becomes a security and governance risk. Control without economics, and the system may be safe but financially unsustainable. Measure cost without operating discipline, and the numbers explain the failure only after customers have already experienced it.

The practical sequence is:

Operate the agent → Control the agent → Understand its economics.

This is the operating model I would use before letting any AI agent touch customers, money, production data or public content.

What does this framework mean?

The framework separates three different questions that teams often mix together.

AWS now describes AgentOps as the discipline for deploying, managing and continuously improving agents in production. Its 2026 guidance organizes the work around governance and security, build and operations, evaluation, and observability. The important point is that agent operation is broader than model access or application uptime. AWS AgentOps guidance treats every agent, tool and memory configuration as a versioned production asset.

Step 1: How do you operate an AI agent?

Operating an agent means being able to reconstruct what happened, detect quality changes and improve the system without guessing.

Traditional software monitoring starts with infrastructure signals:

• Availability
• Latency
• Error rate
• CPU and memory
• Queue depth

Agents need those signals too. They also need behavioral visibility.

An agent can return a technically successful response while choosing the wrong tool, using stale context, repeating an unnecessary loop or failing the actual business task.

That is why operation begins with traces.

Trace the full run, not only the final answer

A useful trace should connect:

• The original request
• The selected model and configuration
• Retrieved context
• Tool selection and arguments
• Tool results
• Handoffs between agents
• Guardrail and approval decisions
• Retries, loops and fallbacks
• The final action
• The business outcome

The OpenAI Agents SDK tracing system records model generations, tool calls, handoffs, guardrails and custom events. OpenTelemetry is also developing shared conventions for GenAI observability so agent telemetry can move across tools without locking the company into one dashboard. Its 2026 guidance models an agent run as a parent span with child spans for model calls and tool execution. OpenTelemetry’s GenAI observability guidance is useful because it treats AI behavior as part of the wider production trace.

Evaluate before and after deployment

Pre-release tests answer one question: did this version pass the known cases?

Production evaluation answers another: does the system still work under real traffic?

A practical evaluation program should test four levels:

1. Tool level: Did the tool call use valid arguments and return the expected result?
2. Turn level: Was the response accurate, grounded and policy-compliant?
3. Session level: Did the agent complete the workflow?
4. Outcome level: Did the business result improve?

Production traffic should be sampled and scored continuously. When quality drops, the result should enter a human review queue or trigger a rollback path.

This is the larger lesson behind why AI projects fail after the demo works: capability is visible early, but reliable operation is built later.

Version every dependency that can change behavior

An agent release is more than application code.

Behavior can change when any of these change:

• The model
• The system prompt
• A tool description
• The available tool list
• The retrieval source
• The memory policy
• The approval threshold
• The routing logic

Each production trace should show which versions produced the run.

Without versioning, the team may know that quality declined but still have no reliable way to identify the change that caused it.

Prepare for agent incidents

An agent incident may not look like an outage.

It may be:

• A sharp increase in wrong-tool selection
• A cost spike caused by repeated reasoning loops
• Duplicate customer contact
• A model update that reduces task completion
• A prompt-injection pattern reaching production
• Sensitive data appearing inside trace content

The incident plan should include a kill switch, human fallback, trace preservation, external-action containment, rollback and a permanent evaluation case created from the failure.

Observability is useful only when it shortens recovery and improves the next release.

Step 2: How do you control an AI agent?

Once an agent can take action, the main question changes.

It is no longer only: what can the model do?

It becomes: what is this specific agent allowed to do, under which identity, for which user, inside which boundary?

NIST’s 2026 work on software and AI-agent identity focuses on identification, authorization, auditing and non-repudiation because agents increasingly access several datasets, applications and tools. The NIST concept paper makes the risk clear: agent access must be governed as seriously as any other non-human identity.

Give every agent its own identity

An agent should not borrow:

• A founder’s administrator account
• A shared API key
• A generic service credential used by several workflows
• A human user’s session

A production agent needs a dedicated identity with:

• Minimum required permissions
• Separate read and write scopes
• Workspace and tenant isolation
• Credential rotation
• Revocation
• Action-level audit logs

The identity should tell the system which agent acted, on whose behalf, against which resource and under which policy.

Use least privilege at the action level

Broad access is convenient during development and dangerous in production.

A publishing agent may need permission to create a draft. It may not need permission to publish, delete media or change billing settings.

A CRM agent may need permission to update a lead stage. It may not need permission to export the entire customer database.

Permissions should match the exact workflow, not the maximum technical capability of the integration.

Place human approval where impact increases

Human approval should be risk-based.

Approval is not the opposite of automation.

It is how automation stays accountable where consequences rise.

Enforce policy inside the runtime

A policy document does not control an agent unless the runtime enforces it.

Controls should include:

• Allowed and blocked tools
• Input and output validation
• Maximum turns and tool calls
• Time and cost limits
• Data-region restrictions
• Approval gates
• Audit requirements
• Automatic stop conditions

This is where BackCTL and H22T fit into the wider AI-native business stack: one layer defines controlled data and actions, while the other provides the environment, telemetry and recovery path underneath them.

Step 3: How do you understand an AI agent’s economics?

An agent may be reliable and controlled while still being a bad business decision.

Economics must be measured at the level of the completed outcome.

Token price alone is not enough.

A single agent task may include:

• Several model calls
• Large retrieved context
• Tool execution
• Memory reads and writes
• Retries
• Fallback models
• Infrastructure
• Human review
• Support and incident handling

The AWS guidance on agentic AI economics argues that organizations should move beyond simple labor-versus-token comparisons and evaluate total economic impact, risk, decision quality and long-term value.

Track cost per completed task

The basic unit should be the business task, not the model request.

Useful measures include:

• Cost per run
• Cost per successful run
• Cost per completed task
• Cost per approved outcome
• Failure and retry cost
• Human review cost
• Cost by customer, workspace and workflow

The OpenAI Agents SDK automatically records request count, input tokens, output tokens, total tokens and per-request usage. Its usage tracking documentation shows the level of detail needed to connect individual model calls to a larger run.

Measure value beside cost

Cost without value creates the wrong optimization.

A cheaper agent that completes fewer tasks may cost more per useful outcome. A more expensive model may be justified if it reduces human correction or increases successful completion.

Track both sides:

The correct question is not “How many tokens did the agent use?”

It is “What did the business receive for the full cost of this workflow?”

Set economic guardrails before scale

Agent loops need budgets.

Before launch, define:

• Maximum cost per run
• Maximum turns
• Maximum tool calls
• Maximum execution time
• Fallback model rules
• Human escalation thresholds
• Daily and monthly workspace budgets

A run that never throws an exception can still become a financial incident.

Route work by difficulty

Not every task needs the most capable model.

A practical routing system may use:

• Deterministic logic for simple rules
• A smaller model for classification or extraction
• A stronger model for complex reasoning
• A human for high-risk uncertainty

This keeps the model layer replaceable and prevents the whole product from inheriting the cost profile of its hardest task. That is one of the reasons the model was never the full moat.

How do the three stages work together?

Consider a hypothetical social-publishing agent.

1. It receives an approved draft.
2. It selects the correct account and platform.
3. It validates format and media.
4. It requests approval for publication.
5. It publishes through the platform API.
6. It records status, cost and result.

Operation asks:

• Can we reconstruct the entire run?
• Did the tool call complete?
• Did the post actually publish?
• Which version produced the result?

Control asks:

• Which account can the agent access?
• Can it publish without approval?
• Can it delete content?
• What happens if the wrong workspace is selected?

Economics asks:

• What did generation, validation, retries and publishing cost?
• How much human review remained?
• Was the workflow cheaper or more reliable than the manual process?

The same workflow must answer all three sets of questions.

A production scorecard for AI agents

Before expanding an agent, score it against these twelve questions.

1. Can we trace every important action?
2. Can we identify the model, prompt, tool and policy version?
3. Do we evaluate task completion in production?
4. Can we detect loops and silent behavioral failures?
5. Does the agent have a dedicated identity?
6. Are permissions limited to the exact workflow?
7. Which actions require human approval?
8. Can we pause or revoke access immediately?
9. Do we know the cost per completed task?
10. Do we include retries, failures and human review in the cost?
11. Do we measure business value beside cost?
12. Who owns the final outcome?

If the team cannot answer these questions, the agent is still a prototype with production access.

What should a small team implement first?

Start with the minimum operating layer.

1. Add end-to-end tracing.
2. Create a small regression dataset from real tasks.
3. Give the agent a dedicated identity.
4. Reduce permissions to the minimum required actions.
5. Add approval for high-impact work.
6. Set turn, tool, time and cost limits.
7. Measure cost per completed task.
8. Create a human fallback and kill switch.

You do not need a large AgentOps platform on day one.

You do need visibility, boundaries and economics before scale.

Frequently asked questions

What is the difference between AgentOps and AI governance?

AgentOps covers the daily deployment, observability, evaluation and improvement of agents. Governance defines the policies, ownership, permissions and accountability under which those operations occur. Production systems need both.

What is the best cost metric for an AI agent?

Cost per completed and accepted business outcome is usually more useful than token cost. It captures model use, tools, retries, infrastructure and human review.

Should every agent action require human approval?

No. Approval should increase with risk. Low-impact drafting can be automated. Actions involving customers, money, publication, deletion, permissions or regulated decisions need stronger control.

Can one model be used for every task?

It can, but it may be economically inefficient. Model routing lets the system use deterministic logic, smaller models, stronger models or humans according to task difficulty and risk.

When is an AI agent ready for production?

An agent is production-ready when the business can observe its behavior, limit its authority, recover from failures and prove that the workflow creates acceptable value at an acceptable total cost.

The final takeaway

The first question about an AI agent is usually: can it do the task?

The production questions are harder:

• Can we operate it?
• Can we control it?
• Can we afford it?

That is the sequence.

Operate the agent. Control the agent. Understand its economics.

When all three are visible, the agent becomes a managed business system instead of an unpredictable model with API access.

Reviewing an agent before production? Send the workflow, available tools and current cost model to hi@turjo.me. I will help identify the missing operating, control and economics layers.