Building Flo: an agentic RAG assistant for Kubeflow, end to end

For Google Summer of Code 2026 I worked on kubeflow/docs-agent, the assistant behind the Kubeflow website. We call it Flo. The brief was to turn a documentation chatbot into an agentic RAG system that runs entirely on Kubernetes, and that other cloud native projects could copy.

This post walks through the whole project in the order the problems appeared. The first half is about getting good answers: what to index, how to retrieve it, which model to run, and how to expose it safely. The second half is about everything that happens after the answer: deploying changes, measuring them, and learning from real users. That second half turned out to be most of the work.

Contents

1. The problem: where answers live
2. Architecture
3. Ingestion: index the sources, not just the docs
4. Retrieval: hybrid search and an intent router
5. Agents and tools
6. Choosing the model
7. The public edge
8. Delivery: GitOps on one cluster
9. An eval on every merge
10. Tracing two environments
11. Feedback that becomes work
12. Challenges, in one table
13. Results and what is next

1. The problem: where answers live

The original system did the standard thing: chunk the Markdown docs, embed them, retrieve the top results, generate. It answered documentation questions from documentation. Most of the questions people bring to a Kubeflow assistant are not documentation questions:

My pipeline pods are stuck in Pending with FailedScheduling.

What does this field default to, and where do I override it?

My InferenceService shows Ready=False. What now?

The first has its fix in a closed GitHub issue. The second is a default value in source code or a CRD schema. The third needs the behaviour of a controller. None of them is organised around an error string or a field name in the prose docs.

Retrieval quality was not the bottleneck. The corpus was. No ranking trick recovers an answer that was never indexed.

2. Architecture

Everything runs on one Oracle Kubernetes Engine cluster with two NVIDIA A10 GPUs.

3. Ingestion: index the sources, not just the docs

The first big change was ingestion, built as Kubeflow Pipelines that write into Milvus:

• Documentation from kubeflow/website. The Hugo pages are parsed with their front matter, split on headings with a token-aware chunker, and tagged with section, version and release date.
• GitHub issues across the Kubeflow repositories. An issue is indexed with its title, problem and resolution together, plus state and labels. A closed issue with a fix is worth more than an open one describing the same symptom.
• Code and manifests, chunked on structure: YAML resources stay whole, so a CRD field and its default are retrieved together.

The rule underneath all three: chunk according to the structure of the thing. Prose splits on headings, issues split into threads, manifests do not split at all. Uniform windows make the indexer’s life easy and the retriever’s life hard.

Ingestion is idempotent (delete by id, then insert), so a pipeline can be re-run after a fix without duplicating anything.

4. Retrieval: hybrid search and an intent router

Pure vector search is good at “how does X work” and bad at “what changed in 1.10”. Version numbers, flag names and error strings are exactly the tokens embeddings blur. The docs collection therefore stores both a dense vector and a native BM25 sparse vector in Milvus, and an intent router chooses per query:

• Version, date and “latest” questions use BM25 with a deeper candidate pool, re-ranked by release date.
• Conceptual questions use hybrid search, dense and BM25 fused with reciprocal rank fusion.
• Older collections without BM25 fall back to dense search.

Results come back as evidence with a strict citation contract. The text the model sees has its URLs stripped, so it cannot invent links. The URLs travel separately as structured citations that the widget renders in a Sources panel. That one rule removed a whole class of hallucinated links.

5. Agents and tools

The MCP server exposes three tools rather than one search endpoint:

search_kubeflow_docs    how-to and conceptual content
search_github_issues    error messages, symptoms, "has anyone hit this"
search_kubeflow_code    defaults, CRD fields, manifests, controller behaviour

A single endpoint is tempting, because the model cannot pick the wrong tool. It also cannot pick the right one. “Why is my pod Pending” and “how does the pipeline DAG execute” need different sources, and a unified index decides between them by similarity alone, which is the signal that does not tell them apart. With separate tools the agent makes an explicit routing decision. Models are good at that when each tool’s description says what it is for and what it is not for.

The agents are kagent resources: a docs agent, the same agent with thinking enabled, and a debug agent that also searches code. The system prompt is in the Helm chart, so a prompt change is reviewed and deployed like code.

6. Choosing the model

We started with Qwen2.5-7B. Two evaluations decided the switch to Gemma 4 E4B: a retrieval and answer-quality eval I ran over the indexed docs, and Santhosh’s tool-use eval of 97 cases, each repeated three times.

Answer quality was level, tool use was far better, and answers came back about four times faster. The model sits behind a stable served name, flo-llm, so switching models is one value in one Helm release, and Qwen stays in the chart’s catalogue as the rollback.

7. The public edge

A public chatbot backed by an LLM is an open invitation to spend someone else’s GPU. The edge is a Helm chart, gateway-guardrails, so the protections ship with every deployment rather than living in a runbook:

• TLS from cert-manager, a CORS allow-list, and global and per-IP rate limits.
• Anonymous session tokens. The widget asks for a short-lived signed token and sends it on every request. Istio checks the signature and expiry at the gateway, so there are no user accounts, but every client has a bounded identity.
• No catch-all route. The agent UI behind the gateway also serves an admin API. The gateway routes only the agent stream, with a strict path regex, because the UI decodes %2F after routing and /a2a/..%2fapi/sessions would otherwise escape to the admin API.

One bug from this layer is worth repeating. Long answers were cut off mid-sentence. The cause was a 30-second route timeout: it applied to the whole streamed response, not just to the first byte, and a 7B model regularly streams for longer than that. Streaming changes what a timeout means.

8. Delivery: GitOps on one cluster

By late September Flo worked, and nobody could safely change it. When I compared Git with the cluster object by object, the model had been patched by hand, the MCP server ran an image built from a fork, and an OpenTelemetry collector, a proxy and several agents existed only in the cluster. Each was a reasonable fix made under time pressure. Together they meant the running system could not be rebuilt from the repository.

One difference was invisible to the usual tools. Tracing keys had been added to the MCP server with kubectl set env. Neither kubectl diff nor Argo CD reported it, because both compare what the chart renders and ignore fields that were added by hand. A clean redeploy from Git would have silently switched off tracing.

The fix was in stages:

1. Argo CD in observe-only mode. An Application per Helm release, pointing at main, with no sync policy. Argo CD reports drift and changes nothing. The list of hand-made changes became a dashboard, and we fixed them one pull request at a time.
2. Staging as a namespace, not a branch. docs-agent-staging runs the same chart. Argo CD syncs it from main automatically and self-heals it. Branch pairs drift and need merge-backs, and in an open-source repository every contributor would have to know which branch to target.
3. Production pinned and synced by hand. Its values live in deploy/envs/prod/. A release is a small pull request that changes a pin, and a rollback is reverting it. GPU releases are always synced manually, because a sync reloads a model.
4. Configuration in Git, secrets in the cluster. Environment values are public and reviewed. Secrets are Kubernetes Secrets referenced by name and never committed.

Staging shares the model and Milvus with production. Retrieval only reads, and the model is the most expensive thing in the cluster. What staging must isolate is the code and configuration we are about to ship, and a namespace does that.

9. An eval on every merge

After every staging sync, Argo CD runs a Job that sends 58 real questions through the deployed stack: agent, tools, Milvus and model, exactly the path a user’s question takes. It checks:

If a gate fails, the Job fails and so does the Argo CD sync. The current baseline is 100% routing, about 73% fully grounded, and a 2.6-second median answer.

Groundedness needs a judge, and the answer model does not judge itself. The second GPU, which had been idle, runs Qwen2.5-7B as a separate judge. A different model family grades Gemma’s answers, and grading never takes GPU time from users. Qwen is a small judge, so its scores are best for comparing two commits on the same questions, not as an absolute grade.

Each run is stored in Langfuse as an experiment named after the commit, the pull request and its author. A contributor finds their PR by number or GitHub username and compares it with the previous run, question by question. That makes answer quality visible to someone who has never had access to the cluster.

10. Tracing two environments

Two environments in one cluster created a new problem: their traces landed in the same Langfuse project, so eval runs polluted the project that holds real conversations.

The standard fix is a resource attribute, deployment.environment=staging, set through OTEL_RESOURCE_ATTRIBUTES. I set it on the staging agents, and their traces kept arriving in production. kagent’s agent runtime builds its OpenTelemetry resource without reading that variable.

So the collector decides instead. Its k8sattributes processor looks up the namespace of the pod that sent each span, and a routing connector sends staging to the evaluation project and everything else to the live one, tagging each with a Langfuse environment. The agents need to know nothing about it. Don’t trust a workload to label itself when the platform already knows where it runs.

The MCP server’s own tracing existed only in that forked image. We ported it into the repository as a small decorator, so each tool call, retrieval and embedding is a span with the search mode, hit count and top scores. Porting it, rather than copying files out of the image, mattered: the image predated several fixes on main.

11. Feedback that becomes work

The eval covers the 58 questions we chose. Users ask the other ten thousand.

Every answer in the widget now has a small Helpful? bar, and a thumbs-down can carry a comment. The vote goes to a small service that finds the trace for that exact answer, from the chat’s session ID and turn number, and records a score on it. Changing a vote overwrites the score.

Each night a triage job reads the day’s thumbs-down. For each one it takes the question, tool calls, retrieved chunks and answer from the trace, searches the index again independently, and asks the judge two questions. Did the agent see a passage that answers this? Does such a passage exist at all?

That last split matters for an open-source project. A missing Katib example is not fixed by tuning our prompt; it is fixed by a docs pull request in Katib. The job writes its verdict onto the trace and prints a report grouped by owner, and a person files the issues. Automatic filing can wait until the judge is calibrated. The first real vote showed why: a user commented that a “trainer” question was really about Pipelines, and the keyword-based component guess said Katib.

12. Challenges, in one table

13. Results and what is next

Where Flo stands after the project:

• Answers from docs, issues and code, with citations rendered from structured data rather than generated links.
• A self-hosted model chosen by evaluation, about four times faster than its predecessor.
• A public edge with session tokens, rate limits and no route to anything but the agent stream.
• Every component described in Git, deployed by Argo CD, with drift visible.
• A staging environment and an end-to-end eval on every merge, visible to any contributor.
• Live and evaluation traces kept apart, and a feedback loop that sorts failures by owner.

What comes next:

• Calibrate the judge against human labels before trusting its absolute scores.
• Enforce in-cluster access with sidecars, starting with the namespace that holds the models and data.
• Automate image promotion: staging picks up new builds automatically, and production moves through a promotion pull request.
• Rewrite the MCP unit tests that the retrieval refactor left behind.

Flo is now a subproject of the new Kubeflow Agents working group, and it is being discussed as a reference implementation for CNCF work on cloud native agent architecture. The project documentation covers the details, and I am walking through all of it at this Saturday’s Kubeflow community meeting.

Lessons

• Find out where the answers live before tuning retrieval.
• Chunk by structure, and keep different sources in different tools.
• Make drift visible before you fix it, and give the cluster one writer.
• Never let a model grade itself, and compare runs rather than reading one score.
• Let the platform label telemetry instead of each workload.
• Attach feedback to the exact trace, and end the loop with a person and an owner.

Thanks to my mentors Chase Christensen, Tarek Abouzeid and Santhosh Toorpu. Santhosh built the hybrid retrieval and the Gemma migration this post leans on, and all three reviewed and questioned everything else.