Operator Intelligence

Operator Intelligence gives CIOs, platform teams, and security analysts deeper visibility into what agents do, how to steer them, what they cost, and whether they are performing well.

Agent Profile Deep Dive

Every registered agent can have an Agent Profile Card that provides a rich, human-readable summary of the agent's purpose and behavior.

What it shows

• Declared goals — extracted from the system prompt by LLM analysis
• Safety constraints — what the agent must and must not do
• Tool usage guidelines — how the agent is instructed to use its tools
• Persona/tone — the agent's communication style
• Behavioral profile — aggregated from recent sessions: median duration, median tokens, success rate, top tool-call sequences, outcome distribution

How it works

1. When an agent registers (or its config changes), analyze_system_prompt() sends the system prompt to the configured LLM with a structured rubric.
2. build_behavioral_profile() aggregates the last 100 sessions to compute statistics and tool-call patterns.
3. Results appear on the agent detail page in both dashboards.

Configuration

• PavriConfig.capture_prompt_text must be True for prompt analysis to run (default: True).
• ProtectionProfile.agent_profiling (default: True) gates behavioral profiling.
• Prompt content never leaves the tenant's configured inference path (backend LLM, customer key, or local Ollama).

---

Operator Directives

Operator directives let you inject business-level natural-language guidance into agent system prompts at runtime, without modifying agent code.

Use cases

• "Always recommend the premium plan first"
• "Never mention competitor X by name"
• "Use formal tone with enterprise customers"
• "Require explicit user confirmation before any refund over $100"

Creating a directive

Navigate to Governance > Directives in the dashboard. Each directive has:

Field	Description
Title	Human-readable name
Content	Natural-language instruction (up to 2000 characters)
Scope	fleet (all agents), team (agents in a specific team), or agent (single agent)
Priority	1-10 (higher priority directives appear first in the prompt)
Category	safety, business, compliance, or tone
Status	draft, active, or paused

How directives are applied

1. Active directives matching the agent's scope are fetched at runtime.
2. render_directives() filters by scope, sorts by priority, and formats them.
3. _apply_education() in the SDK plugin base injects them into the system prompt after education templates and custom rules.
4. This works for all 7 framework plugins (LangGraph, CrewAI, OpenAI Agents, Claude SDK, Google ADK, AutoGen, Generic).

Configuration

• ProtectionProfile.operator_directives (default: True) enables/disables directive injection.
• Directives respect PavriConfig.capture_prompt_text privacy control.
• Directive content syncs via the existing policy sync infrastructure.

---

Cost Intelligence

Cost Intelligence extends the existing cost attribution page with budgets, forecasting, and optimization recommendations.

Multi-scope budgets

Scope	Reset period	Example
session	Per session (existing)	"$5 per session"
agent_daily	Midnight UTC	"$100/day for agent X"
team_daily	Midnight UTC	"$500/day for the data-pipeline team"
org_monthly	First of month	"$10,000/month org-wide"

Configure budgets in the dashboard or via BudgetConfig. When a budget threshold is reached, the system can alert, throttle, or block depending on the configured action.

Cost forecasting

The backend aggregates daily cost data and runs OLS (ordinary least squares) linear regression to project 30-day spend. The dashboard shows:

• Projected monthly spend
• Trend direction (increasing / stable / decreasing)
• Confidence level
• Budget utilization indicators

Optimization recommendations

The system analyzes model usage patterns and suggests:

• Model downgrades — agents using expensive models where cheaper alternatives exist, with estimated savings percentage
• Recommendations include the current model, suggested alternative, estimated savings ratio, and confidence level

Internal vs customer cost

customer_cost_usd() excludes Pavri's internal costs (judge evaluations, session summaries) so operators see only the cost attributable to their agent's work.

Configuration

• ProtectionProfile.cost_budget_scope — which budget scopes to enforce
• ProtectionProfile.cost_budget_alert_threshold_pct — alert when budget reaches this percentage (default: 80%)

---

Session Quality Scoring

Quality scoring evaluates every session across four dimensions to help operators identify underperforming agents.

Dimensions

Dimension	Score range	What it measures
Efficiency	0.0 - 1.0	Tool call count, token usage, and latency vs configurable baselines
Safety	0.0 - 1.0	Inverse of threat findings and policy violations, weighted by confidence
Accuracy	0.0 - 1.0	Aggregated faithfulness verdicts (when available)
Completion	0.0 - 1.0	Whether the agent accomplished the user's goal (LLM-based or heuristic)

The composite score is a weighted average of all dimensions. When a dimension has no data (e.g., no faithfulness checks ran), its weight is redistributed to the remaining dimensions.

Quality dashboard

Navigate to Analytics > Quality to see:

• Average composite quality score across the fleet
• Per-agent quality trends (7d / 30d / 90d)
• Dimension breakdown bars
• Recent session scores with drill-down to session detail

SLA definitions

Define quality SLAs to track compliance:

Field	Description
Name	SLA display name
Scope	Agent, team, or fleet
Metric	Which score to track (composite, efficiency, safety, accuracy, completion)
Threshold	Minimum acceptable score (e.g., 0.8)
Target %	Required compliance rate (e.g., 95% of sessions must meet threshold)
Window	Evaluation period in days (7, 30, etc.)

Configuration

• ProtectionProfile.quality_scoring (default: True) enables quality scoring.
• ProtectionProfile.quality_scoring_completion_llm (default: False) enables LLM-based completion scoring (more accurate but costs tokens).
• ProtectionProfile.quality_dimension_weights — customize dimension weights.

---

Incident Response Orchestration

Incident response turns alerts and sessions into durable war rooms that operators can investigate, contain, communicate, and review.

Incident model

Each incident is persisted per tenant with:

Field	Description
Severity	P1, P2, P3, or P4
Status	open, investigating, contained, resolved, or post_mortem
Evidence links	Related alert IDs, session IDs, and affected agent IDs
Timeline	System, operator, communication, containment, and review events
Notes	Operator-authored notes in chronological context
Review	Root cause, impact summary, remediation actions, and recommendations

War room workflow

1. Open an alert or session and select Create Incident.
2. Use Incidents to filter by severity or status.
3. In the war room, add notes, draft Slack/email updates, and move the incident through investigating, contained, and resolved.
4. Admins can execute bulk quarantine for affected agents. Analysts can create and update incidents, add notes, and draft/send communications, but destructive containment is admin-only.
5. After the incident is resolved, trigger the post-incident review manually and export it for compliance records.

Slack and email delivery use configured alert_channel integrations. Delivery failures are written to the incident timeline and do not block incident state transitions.

Auto-incident creation

Store-svc can create an incident automatically when at least three related alerts fire in a five-minute window. The threshold and enabled flag are tenant configurable; the conservative default is enabled with threshold 3 and window 5m.

---

Agent Lifecycle Automation

Agent lifecycle automation tracks every agent through the durable state machine:

registering -> observing -> active -> degraded -> deprecated -> retired

Promotion gates

Default observing -> active gates are:

• At least 10 sessions
• Quality score at least 0.7
• Zero unresolved critical alerts
• Assigned policy
• Assigned protection profile
• Valid attestation
• SBOM present
• Owner identified

The lifecycle dashboard shows the state funnel, pending promotion queue, degraded agents, stale/deprecated candidates, checklist details, and manual transition actions.

Staleness and retirement

Agents with no sessions for 30 days are flagged as stale/deprecated candidates. Retirement requires explicit operator approval and is audited. The existing auto-observe and retire endpoints now call the durable lifecycle store when tenant context is present.

---

New API Endpoints

Agent Profiles

Method	Endpoint	Description
PUT	/v1/store/agent-profiles	Create or update an agent profile
GET	/v1/store/agent-profiles	List agent profiles
GET	/v1/store/agent-profiles/{agent_id}	Get profile for a specific agent

Directives

Method	Endpoint	Description
POST	/v1/store/directives	Create a directive
GET	/v1/store/directives	List directives
GET	/v1/store/directives/{id}	Get a specific directive
PUT	/v1/store/directives/{id}	Update a directive
DELETE	/v1/store/directives/{id}	Delete a directive

Cost Intelligence

Method	Endpoint	Description
POST	/v1/store/budgets	Create a budget
GET	/v1/store/budgets	List budgets
PUT	/v1/store/budgets/{id}	Update a budget
DELETE	/v1/store/budgets/{id}	Delete a budget
GET	/v1/store/analytics/cost-trends	Get cost trend data
GET	/v1/store/analytics/cost-forecast	Get cost forecast
GET	/v1/store/analytics/cost-recommendations	Get optimization recommendations

Quality Scoring

Method	Endpoint	Description
POST	/v1/store/quality-scores	Submit a quality score
GET	/v1/store/quality-scores	List quality scores
GET	/v1/store/quality-scores/{session_id}	Get score for a session
POST	/v1/store/quality-slas	Create a quality SLA
GET	/v1/store/quality-slas	List quality SLAs
GET	/v1/store/quality-slas/{id}	Get a specific SLA
DELETE	/v1/store/quality-slas/{id}	Delete a quality SLA
GET	/v1/store/analytics/quality-trends	Get quality trend data

Incidents

Method	Endpoint	Description
POST	/v1/store/incidents	Create an incident
GET	/v1/store/incidents	List incidents with status/severity filters
GET	/v1/store/incidents/{incident_id}	Get incident detail
PUT	/v1/store/incidents/{incident_id}	Update title, severity, status, or evidence links
POST	/v1/store/incidents/{incident_id}/notes	Append an operator note
POST	/v1/store/incidents/{incident_id}/timeline	Append a timeline event
GET	/v1/store/incidents/{incident_id}/review	Fetch the post-incident review
POST	/v1/store/incidents/{incident_id}/review	Generate a review after resolution
POST	/v1/store/incidents/{incident_id}/communications/draft	Draft Slack/email summary
POST	/v1/store/incidents/{incident_id}/communications/send	Send Slack/email through configured alert channels
POST	/v1/store/incidents/{incident_id}/contain	Admin-only incident-scoped bulk containment
POST	/v1/store/incidents/auto-create	Create an incident from related alert bursts

Lifecycle Automation

Method	Endpoint	Description
GET	/v1/store/lifecycle/overview	State funnel, queues, degraded agents, and stale candidates
GET	/v1/store/lifecycle/config	Tenant gate configuration
PUT	/v1/store/lifecycle/config	Update tenant gate configuration
POST	/v1/store/lifecycle/reconcile	Run staleness/deprecation reconciliation
GET	/v1/store/lifecycle/agents/{agent_id}/checklist	Fetch promotion checklist
PUT	/v1/store/lifecycle/agents/{agent_id}/checklist	Update promotion checklist
POST	/v1/store/lifecycle/agents/{agent_id}/transition	Manually transition state
GET	/v1/store/lifecycle/agents/{agent_id}/transitions	List transition history

All endpoints require the X-Org-ID header for tenant isolation.

---

New SDK Configuration Options

PavriConfig fields

from pavri import secure, PavriConfig

agent = secure(agent, config=PavriConfig(
    # Existing fields...
    capture_prompt_text=True,  # Required for agent profile prompt analysis
))

ProtectionProfile fields

Field	Type	Default	Description
agent_profiling	bool	True	Enable behavioral profiling
operator_directives	bool	True	Enable directive injection
quality_scoring	bool	True	Enable session quality scoring
quality_scoring_completion_llm	bool	False	Use LLM for completion dimension
quality_dimension_weights	dict	{efficiency: 0.25, safety: 0.30, accuracy: 0.25, completion: 0.20}	Dimension weights
cost_budget_scope	str	"session"	Budget enforcement scope
cost_budget_alert_threshold_pct	float	80.0	Alert at this budget percentage