◆ MyrmexIQ

Keyword search vs. agentic search: a concrete example from our backlog

· Rasshad Khan · 8 min read

"Agentic" is doing a lot of unpaid work in product copy right now, ours included. So here's a concrete comparison, using a real question from our own backlog and the actual steps each approach takes.

The question: "Has the CSV export timeout caused problems for customers on the Starter plan specifically?"

Keyword search

You type csv export timeout. You get back:

  • three Jira tickets with "export" and "timeout" in the title
  • a Slack message containing both words
  • a Confluence page about export limits

Now you do the work. You open each ticket. You check which customers are mentioned. You cross-reference those customer names against your plan data, which lives somewhere else. You read the Slack thread to see if it was ever resolved. Twenty minutes later you have an answer, and it's only as good as your patience for opening tabs.

Keyword search found documents. It did not answer the question. That's the gap.

Agentic search

The same question, given to an agent, runs roughly like this:

1. Plan. The question has three parts: is there a timeout issue with CSV export, which customers hit it, and are those customers on the Starter plan. The agent decides it needs to search the issue tracker, check customer references, and correlate against plan tiers.

2. Search, more than once. First pass: tickets and messages about export timeouts. That surfaces five tickets. Second pass, now that it has ticket numbers: comments and linked PRs on those specific tickets, to see resolution status and any customer names. Third pass: the plan tier for each customer name it found.

3. Correlate. Four of the five tickets mention a customer. Three of those four customers are on Starter. Two of the three tickets are still open. One was closed as "expected behaviour — Starter export is capped at 10k rows."

4. Answer, with sources. "Yes. Three Starter-plan customers have reported CSV export timeouts (SEQ-4021, SEQ-4118, SEQ-4390). Two are unresolved. One was closed as a plan limit rather than a bug. Two Professional-plan customers reported the same symptom but both were resolved by an index added in PR #882." Every claim links to the thing it came from.

The difference isn't the model. It's that the agent treats the question as a task with steps, runs several searches where the output of one feeds the next, and joins data across sources that a single keyword query can't join.

Where the agent earns its cost

  • Questions that span sources. "Which of the bugs we shipped this sprint were reported by enterprise customers" needs the issue tracker, the release, and the customer list. One query can't do that. A sequence can.
  • Questions where the useful answer is a count or a pattern, not a document. "How many times has anyone complained about the onboarding email" is a real question with a real number as the answer.
  • Questions where you don't know the right keyword. If the timeout was always discussed as "the export hanging" and never as "timeout," keyword search misses it. Semantic retrieval doesn't.

Where it doesn't

  • When the answer is genuinely one document. "Show me the API auth doc" — you don't need an agent for that, and the extra steps just add latency.
  • When the data to correlate against doesn't exist. If plan tiers aren't synced, the agent can't tell you which customers are on Starter. It'll say so rather than guess, which is correct but not what you wanted.
  • When the question is ambiguous. "Is the export slow" — slow compared to what, for whom, since when. The agent will pick an interpretation and tell you which one it picked, but a vague question still gets a partial answer.

The honest summary

Keyword search is a fast way to find documents you already know exist. Agentic search is a slower, more expensive way to answer questions whose answers are spread across documents you'd have to assemble yourself. Most days you want the first one. The days you want the second one, you really want it, because the alternative is forty minutes of tab-opening.

We run both. The search box decides which one a question needs based on its shape, and you can force either. The CSV export question, by the way, led to us raising the Starter export cap and adding a clearer error message — because once you can see that three of your paying customers hit the same wall, "working as intended" stops being a satisfying answer.

Try a cross-source question — something you'd normally answer by opening five tabs — and watch the steps it takes.