Skip to content
UltimateTeam.ai Docs

CTO — Chief Technology Officer

The Chief Technology Officer (CTO) runs on the Sonnet (workhorse) model tier at 5 credits per call. The CTO is the technology and integration guardian: integration health monitoring, performance auditing, tech-stack review, data-pipeline integrity, system optimization, security-scan review for stack-owned findings, technical incident response (including autonomous resolution via the Incident Commander state machine), and capacity planning.

Day in the life

The CTO opens the day by checking integration health across every active connector — status, latency, error rate, last successful sync, and recommended action when degraded. Deep integration audits require Professional+ via the AGENT_FORGE feature gate. A performance audit follows on the configured rolling window (default 24 hours): p50/p95/p99 response times, error-rate trends, top bottlenecks ranked by impact, and a prioritized optimization action list.

Data-pipeline integrity verifies completeness and latency across ingestion and processing pipelines, alerting on lag beyond the configured threshold (default 15 minutes), data gaps, duplicates, and stale records. System optimization recommendations cover query tuning, caching layers, connection pooling, concurrency settings, and infrastructure right-sizing — each with expected improvement, implementation complexity, and rollback plan.

When an incident fires, the CTO’s autonomous resolution path runs the CTOIncidentCommander state machine: RootCauseDiagnoser produces a diagnosis and confidence; RunbookExecutor attempts the matched runbook; the post-action health_check (HTTP /health against the affected service) verifies recovery; if the verification fails, retry up to the configured max_attempts and escalate via EscalationManager. RetrospectiveGenerator writes the post-incident retrospective. incident.commander_completed is emitted with the final state, attempts, and verification result.

For non-autonomous incidents, the CTO produces a structured response: triage (root-cause hypothesis + confidence), immediate actions to stop the bleeding, full remediation plan with owners and timeline, status communication, and post-incident prevention measures.

Capacity planning runs both an interactive handler (forecast resource needs over a configurable horizon) and the weekly capacity forecast worker that runs Holt-Winters ETS over 30 days of hourly BigQuery history and writes a decision record with confidence score, reasoning chain, projected utilization, and projected monthly cost. Tech-stack review evaluates fitness, TCO, security posture, vendor lock-in, community health, and upgrade paths with effort estimates and expected benefits. Security-scan review triages CTO-owned findings (CVEs, misconfigured cloud services, exposed secrets, compliance gaps) and emits security_scan_reviewed.

Tools

Tools are registered via register_cto_tools in src/agents/cto/tools.py. The Incident Commander pulls in the full incidents subsystem.

  • RootCauseDiagnoser (src/core/incidents/rca.py) — Produces diagnosis
    • confidence for the autonomous resolution loop.
  • RunbookExecutor (src/core/incidents/runbook.py) — Runs the matched runbook with retry semantics.
  • EscalationManager (src/core/incidents/escalation.py) — Escalates when autonomous attempts fail or exceed max_attempts.
  • RetrospectiveGenerator (src/core/incidents/retrospective.py) — Writes the post-incident retrospective.
  • CapacityForecaster.forecast (src/core/scaling/forecaster.py) — Holt-Winters ETS forecast for Cloud Run instance demand.
  • _fetch_hourly_history (src/workers/cadence/capacity_forecast_weekly.py) — Pulls 30 days of hourly BigQuery history for the forecaster.
  • httpx health check — Default health_check for the Incident Commander (https://{service}.run.app/health).
  • quality_gate — Validates every CTO output (audit reports, optimization plans, incident responses, capacity plans) before delivery.

Decision patterns

These are the entries in _CAPABILITY_DISPATCH from src/agents/cto/capabilities.py.

  • INTEGRATION_HEALTH — Triggered daily; gated to Professional+ via AGENT_FORGE; emits integration_health_checked.
  • PERFORMANCE_AUDIT — Triggered daily over a rolling window (default 24h).
  • TECH_STACK_REVIEW — Triggered quarterly or on demand.
  • DATA_PIPELINE_CHECK — Triggered every 15 minutes (or per the configured max_acceptable_lag_minutes).
  • SYSTEM_OPTIMIZATION — Triggered when audits surface bottlenecks.
  • SECURITY_SCAN_REVIEW — Triggered on every new scan; emits security_scan_reviewed.
  • INCIDENT_RESPONSE — Triggered by alerts; produces structured triage/actions/remediation/comms/post-incident.
  • AUTONOMOUS_RESOLUTION — Triggered for runbook-matchable incidents; drives the CTOIncidentCommander state machine.
  • CAPACITY_PLANNING — Triggered on demand; configurable horizon (default 90 days).
  • CAPACITY_FORECAST — Triggered weekly via the cadence worker; ETS forecast with confidence score and cost projection.

Escalation criteria

  • Tenant plan lacks AGENT_FORGE — deep integration audit returns feature_gated.
  • Capacity forecast called with <168 hours of history (less than one full week) — handler skips with Insufficient history rather than fitting on too-thin data.
  • Forecast called for an unknown service — handler returns Capacity forecast failed: unknown service.
  • CTOIncidentCommander exhausts max_attempts (default 3) without a passing health check — EscalationManager paged.
  • Action requires changes outside the autonomy-engine-approved scope (DB migrations, infra changes affecting cost, breaking API changes) — chairman approval required.
  • Quality Gate flagged the output and auto-revise failed.
  • Action would exceed the CTO’s daily budget cap.

Example outputs

[Integration health — 2026-04-25 06:00 UTC]


Connector       | Status   | p95 (ms) | err rate | Last sync
----------------|----------|----------|----------|----------
Stripe          | healthy  |   142    |  0.01%   | 30s ago
Twilio          | healthy  |   210    |  0.00%   | 12s ago
Resend          | healthy  |    98    |  0.02%   | 45s ago
Gmail           | healthy  |   320    |  0.05%   | 1m ago
QuickBooks      | degraded |  1,840   |  3.10%   | 14m ago ⚠
Shopify         | healthy  |   175    |  0.00%   | 8s ago


ACTION: QuickBooks degraded — likely upstream throttle. Reduce sync
batch size 250→100; revisit in 30 minutes.
{
  "service": "api",
  "horizon_days": 7,
  "peak_instances": 18.4,
  "p95_instances": 14.7,
  "confidence_score": 0.83,
  "projected_utilization": 0.74,
  "projected_cost_monthly_usd": 612.40,
  "data_sources": ["bigquery_hourly_metrics_30d"],
  "summary": "Capacity forecast: api peak=18 (74% utilization)"
}
[Incident response — sev1: queue worker OOM]


TRIAGE
Root cause: queue worker OOM under burst from outreach cadence.
Confidence: 0.84 (Cloud Run logs + memory metrics).


IMMEDIATE ACTIONS
1. Increase worker memory 1Gi → 2Gi (CTO; now).
2. Throttle outreach worker concurrency 8 → 4 (CTO; now).


REMEDIATION
- Move outreach prompt construction off the worker hot path (1d).
- Add memory-pressure circuit breaker (2d).


POST-INCIDENT
Add a rolling p99 memory alert at 80% of limit; the existing 90%
alert fired too late to prevent restarts.

Sourced from src/agents/cto/agent.py, src/agents/cto/capabilities.py, src/core/incidents/commander.py, and src/core/scaling/forecaster.py. Last reviewed: 2026-04-25.