Every incident leaves footprints. Ichnois follows them.
Ichnois turns a production error into an evidence-backed GitHub issue: the commit behind the failing line, a bounded AI explanation, and your approval. The AI explains. It never decides.
Autonomous Investigation Graph
Fingerprint: 06c9cb5852aa | Active claim acquired
Topology of an incident investigation
Trace how unhandled production crashes traverse deterministic filters, git blame pinning, and bounded reasoning before touching human review.
Release SHA Blame Correlator
BuiltGraphQL blame pinned at deployed SHA
Executes GitHub GraphQL blame pinned at the deployed release commit SHA to identify the last author and commit touching the line.
Constraint: Blame locates the line author, not the root cause. Large refactors and squash merges can mislead blame.
Authenticates project public keys, DSN tokens, and HMAC-SHA256 signatures on inbound telemetry webhooks.
Executes GitHub GraphQL blame pinned at the deployed release commit SHA to identify the last author and commit touching the line.
Issues remain in draft state until reviewed and approved by an engineer.
Investigate incidents in a real-time console
Experience the actual triage environment engineers use during an outage: pinned release blame, deterministic diff isolation, and human sign-off.
ZeroDivisionError: division by zero
Live telemetry benchmarking
Production duplicate ratio
Bounded read-only model (explain.py:44)
Draft state required by default
Watch an incident reconstruct itself.
Step-by-step forensic execution of the real payment bug (KeyError: 'stripe_token' at services/payment.py:84). Code gathers evidence; bounded AI explains.
Error Ingestion
Sentry error webhook received for checkout-service.
Three bands of authority. No autonomous decisions.
Deterministic code finds the facts. A bounded LLM only synthesizes the evidence. A human engineer retains final authority before anything is published.
Band 1: Evidence (Deterministic Code)
HMAC-SHA256 signature verification
SHA-256 crash grouping
GraphQL blame @ deployed commit
Commit diff bounded to ~6 KB
Deterministic P1/P2/P3 rules
Regex token scrubbing (<25ms)
confidence: low. It never invents an author.Band 2: Reasoning (Bounded AI)
- •Domain classification: backend, frontend, database, or infra.
- •Root-cause hypothesis: 2-sentence explanation from diff and traceback.
- •Exactly 3 verification checks: actionable checklist for on-call triage.
- •Suspect commit SHA: Selected strictly by git blame at release.
- •Author / Assignee: Selected strictly by version history.
- •Severity level: Evaluated strictly by deterministic user/event rules.
- •Code modification: Zero file write or branch push permissions.
Band 3: Human Control (Authority Gate)
Examine suspect commit diff and failing frame context in cockpit.
Optionally modify suggested fix or re-assign to another team member.
Sign off with single click or discard non-actionable incident.
Issue dispatched with fingerprint label and notification card.
Discrete, verifiable engine modules
Every step of incident correlation is handled by a standalone, testable component backed by repo source code or explicitly marked as planned.
Stack parser
Parses Sentry error payloads, strips container prefix paths, and isolates the failing in-app frame.
strip_prefix("/app/", frame.abs_path) -> "src/payments/service.py:142"Duplicate filter
Computes a deterministic SHA-256 fingerprint from exception type, path, and line number to suppress duplicate events.
sha256(f"{type}:{path}:{line}").hexdigest()Blame correlator
Executes GitHub GraphQL blame pinned at the deployed release commit SHA to identify the last author and commit touching the line.
query BlameAtSHA($sha: GitObjectID!) { commit(oid: $sha) }Bounded explainer
Queries an LLM with deterministic evidence to produce a domain label, root-cause hypothesis, and exactly 3 verification checks.
class ExplanationOutput(BaseModel):
domain: str
hypothesis: str
checks: list[str]Secret redactor
Scrubs credentials, bearer tokens, AWS keys, Stripe secrets, and passwords from payloads before prompt construction.
re.sub(r"(sk_live_[a-zA-Z0-9]+)", "[REDACTED_STRIPE]", raw)Severity rules
Determines P1, P2, or P3 incident priority using fixed deterministic arithmetic on user and event counts.
if users >= 50 and events >= 100:
return "P1-Critical"Issue publisher
Drafts or publishes structured GitHub issues containing evidence, suspect commit, diff, and assignee tag via GitHub App installation tokens or PAT.
gh.issues.create(title=f"[{sev}] {title}", body=markdown)Offline replay CLI
Replays bundled or captured incident payloads end-to-end with zero external network or LLM API calls.
python -m ichnoscope replay --fixture samples/zerodiv.jsonPost-mortem draft
Automated timeline and retrospective document drafting from resolved triage runs.
Coding-agent hand-off
Direct hand-off of triage evidence, suspect diff, and checklist to local coding agents.
Safety by construction, not by prompt
Production environments cannot tolerate autonomous bots executing arbitrary commands. Ichnois enforces strict architectural boundaries at every layer.
Autonomous "Auto-Fix" Bots
- Requires full repo write access, commit push rights, and CI trigger tokens.
- Sends entire unredacted crash payloads and environment dumps directly to commercial LLM APIs.
- Hallucinates false fix branches or opens unreviewed PRs directly touching production code.
- Zero deterministic verification: relies purely on non-deterministic model confidence.
Ichnois Bounded Architecture
- Zero file or code modification: Ichnois never writes code, pushes commits, or touches branches.
- Regex credential redaction: Bearer tokens, Stripe keys, and secrets scrubbed in < 0.1ms (measured: 0.011 - 0.022 ms in bench/results.md).
- Release SHA pinning: GraphQL blame query locked strictly to the deployed commit SHA.
- Mandatory human sign-off: Issues remain in draft state until reviewed and approved by an on-call engineer.
Verifiable Security Guarantees (claims.ts registry)
Honesty Rule: All features backed by repo code or tagged PlannedAuthenticates project public keys, DSN tokens, and HMAC-SHA256 signatures on inbound telemetry webhooks.
Automatic stripping of bearer tokens, API keys, and connection strings prior to LLM analysis.
The LLM has zero file system, git push, or deploy access; it outputs only an explanation and checklist.
Issues remain in draft state until reviewed and approved by an engineer.
Operates with read-only repository access and minimal issue creation permissions.
Granular permissions (owner, admin, member, viewer) with team action audit logging.
Server A DMZ microservice with TLS termination, Slowloris protection, 50KB payload ceiling, and synchronous PII scrubbing.
FastAPI bearer token verification middleware decoding Supabase HS256 JWTs with edge session guards.
Enterprise SSO via Okta, Google Workspace, and SAML 2.0.
Exportable compliance log of all webhook receipts, LLM prompts, and approval decisions.
Directly measured latency from backend benchmarks
No invented numbers or extrapolated estimates. All values are sourced directly from backend/bench/results.md and codebase definitions.
Synchronous PII scrubbing verified at 0.011 - 0.022 ms in benchmark suite.
Average parse, dedupe, redact, and severity rating latency across fixtures.
MAX_DIFF_CHARS context limit enforced to protect prompt context budgets.
Full CLI execution timing actively being benchmarked across platforms.
Fixture Latency Breakdown (backend/bench/results.md)
Evaluated across 3 reference crash fixtures under the automated benchmark suite.
| Fixture | Exception Type | Severity | Parse | Redact | Severity Rate | Total Core |
|---|---|---|---|---|---|---|
| bug1_keyerror_payment.json | KeyError | P1-Critical | 0.050 ms | 0.011 ms | 0.005 ms | 0.084 ms |
| bug2_attributeerror_cart.json | AttributeError | P2-High | 0.103 ms | 0.022 ms | 0.003 ms | 0.133 ms |
| edge_library_frame_last.json | ValueError | P3-Medium | 0.041 ms | 0.011 ms | 0.001 ms | 0.058 ms |
Stop guessing root causes in production
Correlate crashes to suspect commits at release SHA. Route your microservice telemetry webhook or run offline replay locally.