RAG on real Jira data: the chunking decisions that actually mattered

Every RAG tutorial uses clean input. A tidy PDF, a well-formed wiki, a set of blog posts. Real engineering data is not that. A Jira ticket is a title, a description someone wrote in ninety seconds, forty comments over six months, three of which are just "bump," a wall of pasted stack trace, and a resolution field that says "Done."

We learned most of what we know about RAG by getting it wrong on data that looks like that. Here are the decisions that moved the needle.

Chunk by meaning, not by length

The naive approach is to split every document into fixed-size windows — say 500 tokens — with some overlap. It's simple and it's bad for tickets, because it slices a comment in half and pairs the front of one person's reasoning with the back of someone else's.

We chunk on the natural boundaries instead. The title-plus-description is one chunk. Each comment is its own chunk, tagged with who wrote it and when. A pasted stack trace is its own chunk and gets flagged as code so it isn't treated as prose. The resolution and the final state are a chunk.

The effect is that when retrieval pulls "the comment where Priya explained why we can't just increase the timeout," it pulls that comment, whole, with her name and the date attached, not a 500-token slice that starts mid-sentence.

Keep the metadata, use it as a filter

Every chunk carries structured fields alongside the text: project, component, status, issue type, labels, created and updated dates, assignee.

This matters because a lot of real questions have a hard filter buried in them. "What bugs did we close in the payments component last quarter" is a semantic search and a component = payments AND status = closed AND closed_date in Q1 filter. Doing the filter in metadata first and the semantic search second is both faster and much more accurate than hoping the embedding captures "last quarter."

Dedupe before you index, not after

Our sync appends. A ticket that gets touched in five syncs can show up five times in the raw export. Early on we indexed all five, and retrieval would return the same ticket three times in a top-ten, crowding out other results.

Now we collapse to the latest version of each ticket by key before anything gets embedded. Obvious in hindsight. Cost us a week of wondering why results felt thin.

Recency is a real signal, and it's easy to get wrong

If someone asks "what's the current thinking on the caching layer," a comment from last week should outrank a comment from two years ago, even if the old one is a slightly better semantic match.

We don't do this by throwing away old data. We do it by letting the model see the dates on the chunks it retrieves and instructing it to weigh recency when the question is about current state. Questions about history — "why did we originally choose Redis" — should surface the old chunk, and do.

The retrieval budget is smaller than you think

You can stuff a lot into a modern context window. You shouldn't. When we retrieved thirty chunks per question, answers got worse — the model would latch onto a tangentially-relevant chunk and build the answer around it.

Eight to twelve well-chosen chunks, with the metadata visible, produced tighter answers than thirty. Retrieval quality beats retrieval quantity, and past a point, quantity actively hurts.

What we still haven't solved well

Images and attachments. A ticket whose entire content is a screenshot of an error is nearly invisible to text retrieval. We index the filename and any surrounding comments and hope.

Cross-ticket threads. When a discussion spans a Jira ticket, a linked PR, and a Slack thread, each is indexed well on its own but the connection is only as good as the links people remembered to add.

Sarcasm and shorthand. "Oh that's definitely not going to cause problems later" is, semantically, a positive statement. A human reads the thread and knows it's a warning. Retrieval does not.

The short version

If you're pointing RAG at your own engineering data: chunk on human boundaries, keep every structured field and filter on it before you search, collapse duplicates before you embed, let the model see dates, and retrieve less than feels right. None of that is exotic. All of it was the difference between answers we trusted and answers we double-checked every time.

Index your own tickets and ask it something with a filter in it — "closed bugs in <component> last month" — that's the query that shows whether the metadata handling is working.