◆ MyrmexIQ

Investigating an incident across five tools without opening five tabs

· Rasshad Khan · 8 min read

Here's the shape of a small incident. A monitor fires: error rate on the checkout endpoint is up from 0.2% to 4%. It's not an outage, but it's not nothing. Someone has to figure out what changed.

The information you need to answer that exists. It's just spread across five places:

  • CloudWatch has the actual errors — the stack traces, the timestamps, which requests failed.
  • Datadog has the shape of it — when it started, whether it's steady or climbing, which region.
  • GitHub has the deploys — what shipped in the last few hours, and what was in each one.
  • Slack has the context — someone probably mentioned they were changing something related, in a thread you weren't in.
  • Jira has the intent — the ticket for whatever feature the risky change belongs to.

The traditional move is to open all five, become a human join operation, and hold the timeline in your head while you tab between them. It works. It's also the slowest part of most investigations, and it's worse at 2am.

What the investigation tool does

You ask one question — "error rate on checkout jumped around 14:20, what happened" — and it runs an investigation across every connected source instead of searching one at a time.

Concretely, for that question it:

  1. Pulls checkout errors from the logs around 14:20 and reads the actual exception — say, a KeyError on a field called promo_code.
  2. Checks the monitor history to pin the start time precisely: 14:18, climbing for six minutes, then flat.
  3. Lists deploys in the two hours before 14:18. One went out at 14:15.
  4. Reads that deploy's diff and PR description. It touches the checkout serializer.
  5. Searches Slack and Jira for promo_code and the PR number. Finds a thread from that morning where someone said they were "making promo codes optional" and a ticket, SEQ-4412, describing exactly that.
  6. Assembles a timeline and a conclusion: the 14:15 deploy (PR #904, SEQ-4412) made promo_code optional in the request but the serializer still assumes it's present; carts without a promo code now 500. Started 14:18, matches deploy time plus warm-up.

Every step links to its source. You can click into the log line, the PR, the Slack message, and check the reasoning yourself.

Why this is different from search

Regular search finds documents that match your words. This is doing something more specific: it's treating "what happened" as an investigation with a method — establish the timeline, find what changed near the start of the timeline, connect the change to the symptom, find the intent behind the change — and running that method across sources that don't share a schema.

The join it's doing — this log error relates to this deploy relates to this Slack thread relates to this ticket — is exactly the join a person does by hand, and it's the part that takes the time.

Where it still needs a human

It proposes a cause; it doesn't confirm one. In the example the conclusion is well-supported, but "well-supported" isn't "verified." A person still has to look at the diff and agree that yes, that's the bug. The tool is good at narrowing five hours of tab-switching down to one PR to check. It's not good at being certain.

Correlation near a deploy is seductive. If two things shipped at 14:15, the investigation will lean toward whichever one better matches the error text, and it can be wrong. Deploy timing is a strong signal and a fallible one.

It only knows what's connected and synced. If your logs aren't in one of the connected sources, or the incident-relevant Slack channel isn't indexed, that thread of the investigation is just missing, and the conclusion is built on what's left.

Novel failures are hard. "The error rate is up and the logs show a timeout we've never seen, no recent deploy, no relevant chatter" is a real situation, and there the tool can tell you what it ruled out but not what it is. That's still useful — knowing it wasn't a deploy narrows things — but it's not an answer.

The agent side: drafting the follow-up

Once you have a conclusion, there's paperwork: an incident ticket, a summary in the channel, maybe a revert PR. The agent can draft all of that — a ticket pre-filled with the timeline and links, a Slack summary, the revert — but nothing is created or posted without a person approving it. You see exactly what it's about to do, edit it, and then it acts. We were deliberate about that gate. An investigation tool that can also take actions is only comfortable to use if the actions are always the last, explicit, human step.

For the promo_code bug, the real fix was a two-line serializer change and it shipped twenty minutes after the monitor fired. Most of those twenty minutes used to be the investigation. Now most of them are the fix.

Connect your logs, monitors, and repos and point an investigation at your last incident — the timeline it builds is the fastest way to see whether it's useful for yours.