Agent observability, evaluation and governance
Take any AI agent to production, and keep it there.
Framework · delivered through engagements
AELSHA™ ASTRA is how we deliver every engagement, not software you buy. Your agent works in the demo; running it on real work every day raises new questions: what did it do, was it right, what did it cost, and who allowed it?
When we build or run an agent with you, you receive:
- Checks the agent must pass before it goes live
- A record of every step it takes, and what each run costs
- Rules for what it may do alone, and what a person approves
Tell us what you want to automate. We reply by email.
AELSHA™ ASTRA, short for Agentic Systems: Trust, Reliability & Assurance, is a tool-agnostic framework for taking AI agents from proof of concept to production and keeping them there. It defines a six-phase lifecycle with quality gates and an observability control tower built on OpenTelemetry GenAI conventions. It is the framework behind every Aelsha Innovations engagement: how we deliver, not a product you buy.
Technical detail: standards, methods and what we check ↓Terms used on this page
- AI agent
- software that reads a request, decides what to do and acts in your systems.
- Pilot / production
- a pilot shows it can do the task once; production is when it runs on real work every day.
- Evaluation set
- a fixed collection of real examples the agent is tested against before each release.
- Tracing / observability
- a record of every step an agent takes, so you can see what it did and what it cost.
- OpenTelemetry
- an open, vendor-neutral framework for recording what software does, so records from different tools can be read together.
- Red-teaming
- attacking a system, with written permission, to find weaknesses before someone else does.
- Autonomy tier
- how much an agent may do before a person must approve.
- Governance
- the rules for what may run, who approved it, and the evidence kept.
The six AELSHA™ ASTRA lifecycle phases, clockwise from the top. Each ends at a quality gate, and phase 6 feeds back into phase 1. Observability, evaluation and governance run through every phase.
01 Frame
- Objective:
- Define value and risk.
- Key activities:
- Use case, business value, autonomy tier, success criteria, guardrail policy.
- Quality gate:
- A signed value and risk brief, with KPIs defined.
02 Build
- Objective:
- Assemble the agent.
- Key activities:
- Model choice, prompts and context, tools, memory, orchestration, instrumentation.
- Quality gate:
- The agent runs end to end, and its traces are emitting.
03 Evaluate
- Objective:
- Prove quality offline.
- Key activities:
- Golden datasets, offline evaluations, LLM-as-judge scoring, red-teaming.
- Quality gate:
- Evaluation thresholds met, and the regression suite is green.
04 Harden
- Objective:
- Make it safe and affordable.
- Key activities:
- Guardrails, security tests, cost and latency budgets, load tests.
- Quality gate:
- Security and FinOps sign-off, with budgets set.
05 Deploy
- Objective:
- Release safely.
- Key activities:
- Canary or human-in-the-loop rollout, SLOs, dashboards and alerts live.
- Quality gate:
- SLOs met in the canary, and rollback tested.
06 Observe & Improve
- Objective:
- Run and learn.
- Key activities:
- Tracing, online evaluation, drift detection, feeding the improvement loop.
- Quality gate:
- An improvement backlog that feeds the next iteration, back at phase 1.
Why agents need more than application monitoring
Agents fail in ways classic application performance monitoring (APM) was never built to see. They are non-deterministic, act in the real world, choose tools at runtime, keep memory and chain sub-agents. Our view: what most often holds agents back between proof of concept and production is an observability and lifecycle gap, not model capability.
| Dimension | Traditional application (APM) | Agentic AI |
|---|---|---|
| Unit of work | A request or transaction | A multi-step task and its path across agents |
| What a step is | Service and database calls | Reasoning steps, tool calls, memory operations, handoffs |
| Determinism | Repeatable | Non-deterministic: the same input can take a different path |
| How it fails | Errors, latency, 5xx responses | Wrong tool, fabricated output, low confidence, goal drift |
| Quality | Implicit: it ran | Must be evaluated for accuracy, faithfulness and safety |
| Cost | Stable per request | Varies from task to task |
| Security | A known attack surface | Prompt injection, tool misuse, memory poisoning |
Six design principles
Six principles shape every part of AELSHA™ ASTRA. They keep it independent of any one vendor, make quality something measured rather than asserted, and treat governance, cost and the needs of each role as design inputs from the first day, not additions at the end.
Tool-agnostic, standards-first
Built on OpenTelemetry GenAI conventions, so traces from any agent framework land in any backend.
Eval-driven
Quality is measured continuously, not asserted. Evaluations gate every release.
Closed-loop
Production signals become datasets, evaluations and improvements.
Governance- and security-by-design
Controls, autonomy tiers and human oversight are built in, not bolted on.
FinOps-aware
Cost per outcome is a first-class signal, next to latency and quality.
Persona-based
Executives, product, operations, data science, FinOps and risk each get the view they need.
The observability control tower
The control tower is how AELSHA™ ASTRA observes agents: five signal pillars flowing through three planes, framed by governance and standardised on OpenTelemetry. It is a reference architecture, not a product. Each part can be filled by open-source software, a commercial service or your cloud provider’s own tools.
The backbone is the OpenTelemetry GenAI semantic conventions, a shared vocabulary for agent telemetry with operation names such as invoke_agent and execute_tool. Because the schema is shared, traces from different agent frameworks reach the same backend without custom parsing. The conventions are in Development status (checked 26 October 2026).
Source: OpenTelemetry GenAI conventions
What a well-instrumented trace contains
A trace is the record of one task; each timed step in it is a span. AELSHA™ ASTRA names six span types:
- LLM call
- Tool call
- Memory operation
- Handoff
- Reasoning step
- Guardrail event
Five signal pillars rise through three planes, from capture to action, inside a governance frame and on the OpenTelemetry backbone. Improvement actions feed back to the agent.
| Plane | What it does |
|---|---|
| 1 · Instrument & Capture | Instruments the agent framework and emits standard spans. |
| 2 · Pipeline & Store | Collects, samples and stores telemetry. Every trace with an error or a cost breach is kept; the rest are sampled. |
| 3 · Analyze & Act | Turns data into dashboards, evaluations, alerts, cost analysis and improvement actions that feed back to the agent. |
The five signal pillars
- Telemetry:
- Traces, metrics and logs of every agent step.
- Evaluation:
- Online and offline scoring of quality.
- Quality & Safety:
- Guardrail events, faithfulness, toxicity, policy violations.
- Cost & FinOps:
- Tokens, model spend, cost per task and per outcome.
- Business / Outcome:
- Task success, resolution rate, satisfaction, hours saved.
A catalogue of exactly what to capture
The signal catalogue says what to capture and how: 47 signals in 10 domains, each with a definition, unit, source span attribute, capture method and alert threshold, and an 11-table data model aligned to OpenTelemetry so trace IDs join cleanly. The full catalogue is delivered in engagements; this page shows its shape.
- 47 signals
- 10 domains
- 11 tables in the data model
The 47 signals grouped by domain, one block per signal.
47 signals in 10 domains
- Telemetry5
- LLM7
- Tool use5
- Quality5
- Confidence3
- Safety5
- Cost5
- Reliability5
- Business4
- Human in the loop3
The evaluation flywheel
Evaluation is the engine of AELSHA™ ASTRA. Golden datasets (curated inputs with known good answers) and a regression suite run in CI/CD, so no change ships without a measured quality check, and the same metrics run online in production. A falling online score is a drift signal that arrives before a user complaint does.
Wherever a check can be code, the framework prefers code to a judge model: tool correctness is a deterministic comparison of the tools and arguments an agent called against the ideal ones.
The improvement loop, six steps clockwise from the top around a growing base of datasets and knowledge.
- 01 Monitor. Trace and score behaviour in production.
- 02 Detect & diagnose. Find failures, drift and low-confidence answers.
- 03 Curate. Turn high-signal cases and human corrections into golden and edge-case datasets.
- 04 Evaluate. Measure each candidate change against those datasets.
- 05 Improve. Pick the right lever: prompts and context, model routing, tool fixes, guardrail tuning or fine-tuning.
- 06 Release. Ship only after the regression gate passes.
Governance and autonomy tiers
Governance runs through AELSHA™ ASTRA rather than being added at the end, because you cannot govern what you cannot see. Each of the ten risks in the OWASP Top 10 for Agentic Applications is mapped to a control and to the signal that detects it, and autonomy is earned through demonstrated reliability.
Source: OWASP Agentic Top 10 (Dec 2025)
Four autonomy tiers
Oversight is matched to risk. The gold mark shows where a person approves actions.
T0Assist
- The agent
- Suggests; a person performs every action.
- Oversight
- A person decides and acts.
T1Supervised
- The agent
- Acts, with a person approving each consequential step.
- Oversight
- A person approves each consequential step.
T2Conditional
- The agent
- Acts alone within defined limits, and escalates outside them.
- Oversight
- Limits are enforced outside the model; a person handles escalations.
T3High
- The agent
- Acts with broad independence.
- Oversight
- Review after the fact, with a tamper-evident audit trail.
The maturity ladder, L1 to L5
A five-level maturity model shows where an organisation stands and plans the climb. It is scored across six dimensions: instrumentation, evaluation, governance, FinOps, operating model and continuous improvement. Each level has visible hallmarks, and an assessment places you on the ladder and sets the next level to aim for.
The five maturity levels as a staircase, from reactive to self-improving.
L1 Reactive
Ad hoc
Logs only, no agent traces, debugging by guesswork.
L2 Instrumented
Visible
OpenTelemetry traces with cost and latency; basic dashboards live.
L3 Managed
Controlled
Online and offline evaluations; SLOs, alerts and governance gates.
L4 Optimized
Proactive
Drift detection, FinOps guardrails, automatic evaluation in CI/CD.
L5 Self-improving
Autonomous
A closed-loop data flywheel that curates, tunes and canaries.
How AELSHA™ ASTRA is delivered
AELSHA™ ASTRA is delivered through fixed-scope engagements, not sold as software. Seven offerings map onto the same Validate, Scale and Operate steps as all our consulting work. Each has a fixed scope, and we agree the scope and the schedule with you at kick-off. You can start with one, stop after it, and keep what was delivered.
- 01
Validate
Find out where you stand.
Agent observability and readiness assessment
Maturity scoring, scored use cases, a review of data and systems, a 90-day roadmap and a pilot, defer or stop recommendation.
- 02
Scale
Build the foundation and take agents to production.
AELSHA™ ASTRA foundation build
Stand up the control tower: OpenTelemetry, stores, dashboards as code, governance.
Agent productionisation
Take one agent through the six phases to production.
Evaluation and quality engineering
Evaluation harness, golden datasets, judge calibration, CI/CD gates.
Governance, trust and security for agents
OWASP-aligned controls, autonomy tiers, audit trail, responsible-AI review.
Agent FinOps and cost optimisation
Cost-per-outcome instrumentation, model routing, caching, budgets.
- 03
Operate
Run and improve agents from real results.
Managed AgentOps
We observe, evaluate and improve your agents in production with your team, on a scope and response times agreed in the contract.
Questions about AELSHA™ ASTRA
AELSHA™ ASTRA is a framework, not software you install or subscribe to. Aelsha Innovations delivers it through fixed-scope engagements, on the observability tools you already run or the ones you choose, and it starts wherever your agents are today. The answers below say what it is, what it works with and where to start.
Is AELSHA™ ASTRA a software product?
No. It is a framework: a lifecycle with quality gates, a reference architecture for observability, a signal catalogue, a maturity model and the practices that connect them. Aelsha Innovations delivers it through engagements, on the observability tools you already run or the ones you choose.
Which observability tools does it work with?
It is tool-agnostic. Because it standardises on the OpenTelemetry GenAI semantic conventions, traces from different agent frameworks land in the same backend, whether that is open-source software, a commercial service or your cloud provider’s own observability. Changing backends need not mean re-instrumenting your agents.
Does it replace our existing monitoring?
No. It works with what you already run and adds what application monitoring cannot see: reasoning steps, tool calls, evaluation scores and cost per outcome, on the same OpenTelemetry foundation.
Do we need to reach a maturity level before we start?
No. The maturity model exists to show where you are. An engagement can begin with an assessment that scores your current level against the five levels and sets a target.
Is AELSHA™ ASTRA related to other projects called ASTRA?
No. Other, unrelated organisations and projects use the name ASTRA. AELSHA™ ASTRA is the framework described on this page, from Aelsha Innovations.
Further reading
Find out where your agents stand
Tell us what your agents do today and what is keeping them from production. We reply by email.