Building a Production RAG Service in Java: A Complete Guide
Your manager's exact words: "We're launching the docs assistant to customers next month. It needs to be reliable, observable, and safe." Not "build a chatbot." Not "demo something with an LLM." A service — one that strangers will rely on, that breaks at 2 AM, that answers questions about your product with your company's name on every response.
Everything in this track was preparation for this moment. Calling LLMs from Java gave you the client. Prompt engineering gave you the answer-shaping. Embeddings and vector search gave you retrieval. RAG from scratch gave you the grounded-answer loop. LangChain4j and Spring AI showed you the framework options. Function calling taught you structured outputs. Evals taught you how to measure. This post assembles all of it into one coherent service skeleton — the ingestion pipeline, the query path, and the production concerns that separate a demo from a service: configuration, guardrails, observability, evals wired into deploys, rate limiting, and cost awareness.
Here is the principle this whole post hangs on: a demo answers questions; a service answers for its answers. Every answer your service produces must be traceable (which documents? which prompt version? which model?), measurable (how long? how many tokens? what did it cost?), and accountable (did the eval set pass before this deployed?). If you can't answer those questions about an answer, you don't have a service yet — you have a demo with uptime.
The architecture
The service has two pipelines and three operational rails. The ingestion pipeline runs offline: load documents, chunk them, embed them, store the vectors. The query path runs online: validate and rate-limit the request, retrieve passages, build a versioned prompt, generate, check the citations, respond. The rails — telemetry, evals, and real health checks — run across both and are what make the whole thing production-shaped.
The ingestion pipeline: load, chunk, embed, store
Ingestion runs offline — on a schedule, on document change, or in CI — so it must be deterministic, re-runnable, and boring. Four stages, each one testable in isolation:
- Load. Pull the source documents (markdown files, help-center exports, PDFs) and record each one's identity: a stable
documentId, a title, a version. ThedocumentIdis what citations point back to, so it must be stable across re-ingestions — never use a row number or a timestamp as an identity. - Chunk. Split documents into retrievable units. A sliding window with overlap is the honest default: small enough that each chunk is about one idea, overlapping enough that a sentence split across a boundary still appears whole somewhere. This is pure string work — no network, no model — so it belongs in plain, unit-tested Java:
package com.docsbot.rag.chunk;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
/** One retrievable unit of a source document. */
public record Chunk(String documentId, int chunkIndex, String text) {
public Chunk {
Objects.requireNonNull(documentId, "documentId");
Objects.requireNonNull(text, "text");
if (chunkIndex < 0) {
throw new IllegalArgumentException("chunkIndex must be >= 0");
}
}
}
public final class SlidingWindowChunker {
public List<Chunk> chunk(String documentId, String text,
int chunkSizeChars, int overlapChars) {
Objects.requireNonNull(documentId, "documentId");
Objects.requireNonNull(text, "text");
if (chunkSizeChars <= 0) {
throw new IllegalArgumentException("chunkSizeChars must be positive");
}
if (overlapChars < 0 || overlapChars >= chunkSizeChars) {
throw new IllegalArgumentException(
"overlapChars must be in [0, chunkSizeChars)");
}
List<Chunk> chunks = new ArrayList<>();
int index = 0;
int pos = 0;
while (pos < text.length()) {
int end = Math.min(pos + chunkSizeChars, text.length());
chunks.add(new Chunk(documentId, index++, text.substring(pos, end)));
if (end == text.length()) {
break;
}
pos = end - overlapChars; // overlap < chunkSize, so pos always advances
}
return List.copyOf(chunks);
}
}
- Embed. Convert each chunk to a vector through the embedding service. This is the expensive step — batch it, retry with backoff on transient failures, and persist the mapping of
(documentId, chunkIndex) → vectorso a re-ingestion that only changed one document doesn't re-embed the world. - Store. Upsert vectors plus their chunk text and metadata into the vector store. Upsert (not insert) is the key verb: re-running ingestion over unchanged documents must be a no-op, and re-running over an edited document must replace the old chunks, not duplicate them.
One decision that pays off for months: keep the chunker, the prompt builder, and the eval runner as pure Java with no framework dependencies. The orchestration core below compiles with plain javac — every class in this post's core was compiled with javac --release 21 and smoke-tested with stub adapters before a single line was written here. When the LLM SDK changes its API next year, you rewrite one adapter, not the service.
The query path: one request, end to end
The query path is the class customers feel. Its job list, in order: validate the input, enforce the rate limit, retrieve evidence, build a versioned prompt, generate, verify the citations, and log everything. Here is the full orchestration core — pure Java, interfaces at every external seam:
package com.docsbot.rag.service;
import com.docsbot.rag.cost.CostEstimator;
import com.docsbot.rag.guard.InputGuard;
import com.docsbot.rag.llm.LlmPorts.LlmAnswer;
import com.docsbot.rag.llm.LlmPorts.LlmClient;
import com.docsbot.rag.llm.LlmPorts.LlmException;
import com.docsbot.rag.llm.LlmPorts.LlmRequest;
import com.docsbot.rag.limit.RateLimitRegistry;
import com.docsbot.rag.ports.Ports.Embedding;
import com.docsbot.rag.ports.Ports.EmbeddingException;
import com.docsbot.rag.ports.Ports.EmbeddingService;
import com.docsbot.rag.ports.Ports.RetrievedPassage;
import com.docsbot.rag.ports.Ports.VectorStore;
import com.docsbot.rag.service.QueryTypes.Citation;
import com.docsbot.rag.service.QueryTypes.QueryRecord;
import com.docsbot.rag.service.QueryTypes.QueryResult;
import com.docsbot.rag.service.QueryTypes.Telemetry;
import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.UUID;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public final class QueryService {
private static final Pattern CITATION_MARKER = Pattern.compile("\\[(\\d+)\\]");
private final EmbeddingService embeddings;
private final VectorStore store;
private final LlmClient llm;
private final InputGuard guard;
private final RateLimitRegistry rateLimits;
private final CostEstimator costs;
private final Telemetry telemetry;
private final String modelName;
private final int topK;
public QueryService(EmbeddingService embeddings, VectorStore store, LlmClient llm,
InputGuard guard, RateLimitRegistry rateLimits, CostEstimator costs,
Telemetry telemetry, String modelName, int topK) {
this.embeddings = Objects.requireNonNull(embeddings, "embeddings");
this.store = Objects.requireNonNull(store, "store");
this.llm = Objects.requireNonNull(llm, "llm");
this.guard = Objects.requireNonNull(guard, "guard");
this.rateLimits = Objects.requireNonNull(rateLimits, "rateLimits");
this.costs = Objects.requireNonNull(costs, "costs");
this.telemetry = Objects.requireNonNull(telemetry, "telemetry");
this.modelName = Objects.requireNonNull(modelName, "modelName");
if (topK <= 0) {
throw new IllegalArgumentException("topK must be positive");
}
this.topK = topK;
}
public QueryResult ask(String clientId, String question) {
String requestId = UUID.randomUUID().toString();
long startNanos = System.nanoTime();
try {
InputGuard.GuardResult gate = guard.check(question);
if (!gate.allowed()) {
log(requestId, clientId, gate.reason(),
elapsedMs(startNanos), 0, 0, BigDecimal.ZERO);
return QueryResult.refused(requestId,
"I can't help with that.", gate.reason());
}
if (!rateLimits.forClient(clientId).tryAcquire()) {
log(requestId, clientId, "rate-limited",
elapsedMs(startNanos), 0, 0, BigDecimal.ZERO);
return QueryResult.refused(requestId,
"Too many requests — please wait a moment and retry.",
"rate-limited");
}
Embedding query = embeddings.embed(question);
List<RetrievedPassage> passages = store.search(query.vector(), topK);
if (passages.isEmpty()) {
log(requestId, clientId, "no-evidence",
elapsedMs(startNanos), 0, 0, BigDecimal.ZERO);
return QueryResult.refused(requestId,
"I don't have documentation covering that question.",
"no-evidence");
}
String prompt = PromptTemplates.buildAnswerPrompt(question, passages);
LlmAnswer answer = llm.complete(new LlmRequest(
PromptTemplates.PROMPT_VERSION, modelName, prompt, 512, 0.2));
List<Citation> citations = extractCitations(answer.text(), passages);
if (citations.isEmpty()) {
// Answered without grounding: safer to refuse than to hand the
// user an uncited claim. This does NOT prove the answer is
// wrong — only that it cannot be traced.
log(requestId, clientId, "no-citations",
elapsedMs(startNanos),
answer.promptTokens(), answer.completionTokens(), BigDecimal.ZERO);
return QueryResult.refused(requestId,
"I don't have documentation covering that question.",
"no-citations");
}
BigDecimal cost =
costs.estimateUsd(answer.promptTokens(), answer.completionTokens());
QueryResult result = QueryResult.answered(requestId, answer.text(),
citations, elapsedMs(startNanos),
answer.promptTokens(), answer.completionTokens(), cost);
log(requestId, clientId, "ok", result.latencyMs(),
result.promptTokens(), result.completionTokens(), result.costUsd());
return result;
} catch (EmbeddingException | LlmException e) {
// Dependency failure: safe fallback, never leak internals to caller.
log(requestId, clientId, "dependency-error",
elapsedMs(startNanos), 0, 0, BigDecimal.ZERO);
return QueryResult.refused(requestId,
"The assistant is temporarily unavailable. Please try again.",
"dependency-error");
}
}
static List<Citation> extractCitations(String answerText,
List<RetrievedPassage> passages) {
List<Citation> citations = new ArrayList<>();
Matcher m = CITATION_MARKER.matcher(answerText);
while (m.find()) {
int n = Integer.parseInt(m.group(1));
if (n >= 1 && n <= passages.size()
&& citations.stream().noneMatch(c -> c.number() == n)) {
RetrievedPassage p = passages.get(n - 1);
citations.add(new Citation(n, p.documentId(), p.title()));
}
}
return List.copyOf(citations);
}
private void log(String requestId, String clientId, String outcome, long latencyMs,
int promptTokens, int completionTokens, BigDecimal costUsd) {
telemetry.logQuery(new QueryRecord(requestId, clientId,
PromptTemplates.PROMPT_VERSION, modelName, outcome,
latencyMs, promptTokens, completionTokens, costUsd));
}
private static long elapsedMs(long startNanos) {
return (System.nanoTime() - startNanos) / 1_000_000;
}
}
Notice what this class doesn't do: no HTTP, no JSON parsing, no SDK calls. EmbeddingService, VectorStore, LlmClient, and Telemetry are interfaces. The Spring Boot controller below is a thin adapter over this core — and because the core is pure Java, the eval harness in the last section can exercise the real query path without starting a server.
The prompt it builds is versioned, on purpose. The prompt version is a first-class operational fact — logged with every request, pinned at deploy time — so a bad prompt edit can be identified in the logs and rolled back like any other config change:
package com.docsbot.rag.service;
import com.docsbot.rag.ports.Ports.RetrievedPassage;
import java.util.List;
public final class PromptTemplates {
/** Bump this whenever the prompt text changes; never silently. */
public static final String PROMPT_VERSION = "rag-answer-v3";
private PromptTemplates() {
}
public static String buildAnswerPrompt(String question,
List<RetrievedPassage> passages) {
StringBuilder sb = new StringBuilder(4096);
sb.append("You are a documentation assistant. Answer ONLY from the numbered ")
.append("sources below. Cite every factual claim as [n], where n matches ")
.append("the source number. If the sources do not contain the answer, ")
.append("say so plainly and do not guess.\n\n")
.append("Sources:\n");
for (int i = 0; i < passages.size(); i++) {
RetrievedPassage p = passages.get(i);
sb.append('[').append(i + 1).append("] ")
.append(p.title()).append(" — ").append(p.text()).append('\n');
}
sb.append("\nQuestion: ").append(question).append("\nAnswer:");
return sb.toString();
}
}
The HTTP adapter (structural sketch)
The web layer is deliberately thin — it translates HTTP into ask(clientId, question) and QueryResult into HTTP. This sketch assumes Spring Boot on the classpath; it was not compile-verified here (no framework jars in this environment), but every type it touches is the real core above:
// Structural sketch — Spring Boot adapter over the pure-Java core.
@RestController
@RequestMapping("/api/ask")
public class AskController {
private final QueryService queryService; // the compiled core above
public AskController(QueryService queryService) {
this.queryService = queryService;
}
@PostMapping
public ResponseEntity<AnswerDto> ask(@RequestBody AskRequest body,
@RequestHeader("X-Client-Id") String clientId) {
QueryResult result = queryService.ask(clientId, body.question());
HttpStatus status =
result.refused() && "rate-limited".equals(result.refusalReason())
? HttpStatus.TOO_MANY_REQUESTS
: HttpStatus.OK;
return ResponseEntity.status(status).body(AnswerDto.from(result));
}
}
Streaming the answer token-by-token (server-sent events) is a UX improvement you add at this layer — the adapter subscribes to the LLM client's streaming API and forwards tokens — but it changes nothing in the core: the citation check still runs on the complete answer before it is presented as final. Sketch it here only after the non-streaming path is measured and working.
Configuration: no hardcoded keys, models, or prices
A demo hardcodes the model name and reads the API key from a string literal. A service treats every one of those as configuration, because all of them change: keys rotate, models get replaced, prices get updated, topK gets tuned. In Spring Boot that means @ConfigurationProperties backed by environment variables — with a fail-fast rule for secrets:
# application.properties — values come from the environment, never from this file.
docsbot.llm.model=${LLM_MODEL}
docsbot.llm.api-key=${LLM_API_KEY}
docsbot.retrieval.top-k=${RETRIEVAL_TOP_K:5}
docsbot.pricing.input-per-mtok-usd=${PRICE_INPUT_PER_MTOK_USD}
docsbot.pricing.output-per-mtok-usd=${PRICE_OUTPUT_PER_MTOK_USD}
// Structural sketch — binds the properties above; resolved at startup.
@ConfigurationProperties(prefix = "docsbot")
public record DocsBotProperties(Llm llm, Retrieval retrieval, Pricing pricing) {
public record Llm(String model, String apiKey) {}
public record Retrieval(int topK) {}
public record Pricing(BigDecimal inputPerMtokUsd, BigDecimal outputPerMtokUsd) {}
}
And the fail-fast secret rule, in compiled pure Java — call it at startup, before the server accepts traffic:
package com.docsbot.rag.config;
/**
* Secrets come from the environment (or a secret manager that populates it),
* never from source code, and never from a committed properties file.
* Missing secrets fail fast at startup with the NAME of the missing variable —
* values are never logged.
*/
public final class Secrets {
private Secrets() {
}
public static String requireEnv(String name) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
throw new IllegalStateException("Missing required environment variable: "
+ name + " — set it in the environment or secret manager; "
+ "never commit secrets to source control.");
}
return value;
}
}
Guardrails: what each control actually does
Guardrails are the most over-claimed part of every RAG demo. Be precise about what each control does and doesn't do:
- Input validation (length caps, non-blank, schema checks) does one honest job: it bounds cost and keeps garbage out of the embedding call. It does not make the input safe.
- A blocklist is a rule-of-thumb first layer: cheap, fast, and effective against obvious, known-bad inputs — and trivially bypassed by paraphrase or obfuscation. It is a tripwire, not a wall. Say that out loud in your design doc, or someone will assume the wall exists.
- Output checks (citation presence, refusal on no-evidence) do one honest job: they keep untraceable answers from reaching the user. They do not prove the answer is correct — a model can cite a real source while misquoting it. Citation existence ≠ support; support is what the eval set and human review measure.
The input guard itself is compiled, testable Java:
package com.docsbot.rag.guard;
import java.util.Locale;
import java.util.Objects;
import java.util.Set;
public final class InputGuard {
private final Set<String> blocklist;
private final int maxChars;
public InputGuard(Set<String> blocklist, int maxChars) {
this.blocklist = Set.copyOf(Objects.requireNonNull(blocklist, "blocklist"));
if (maxChars <= 0) {
throw new IllegalArgumentException("maxChars must be positive");
}
this.maxChars = maxChars;
}
public record GuardResult(boolean allowed, String reason) {}
public GuardResult check(String question) {
if (question == null || question.isBlank()) {
return new GuardResult(false, "empty-question");
}
if (question.length() > maxChars) {
return new GuardResult(false, "exceeds-max-length");
}
String lowered = question.toLowerCase(Locale.ROOT);
for (String term : blocklist) {
if (!term.isBlank()
&& lowered.contains(term.toLowerCase(Locale.ROOT))) {
return new GuardResult(false, "blocklist-match");
}
}
return new GuardResult(true, "ok");
}
}
Observability: log what you'd need at 2 AM
Every request emits one structured record with exactly the fields a future incident needs: requestId, clientId, promptVersion, model, outcome (ok, refused-reason, or error class), latencyMs, promptTokens, completionTokens, and costUsd. The QueryService already logs this on every path — including refusals and dependency failures — through the Telemetry seam; the production adapter ships it to your log aggregator as JSON.
Why these fields, specifically: promptVersion + model let you attribute a quality regression to the exact deploy that caused it. latencyMs + token counts let you separate "the model is slow" from "our prompts got fat." costUsd turns the next section's arithmetic into a dashboard. And logging refusals with their reasons is how you discover that your guardrails are blocking legitimate users — the failure mode nobody notices until support tickets pile up.
The second half of observability is health. A /health endpoint that unconditionally returns 200 is theater — it tells the load balancer the process is alive while the vector store is down. This one actually exercises each dependency and reports 503 when anything is unhealthy:
package com.docsbot.rag.health;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
public final class DependencyHealth {
public interface HealthCheck {
String name();
CheckResult check();
}
public record CheckResult(String name, boolean healthy,
String detail, long latencyMs) {}
public record OverallStatus(boolean healthy, List<CheckResult> results) {}
private final List<HealthCheck> checks;
public DependencyHealth(List<HealthCheck> checks) {
this.checks = List.copyOf(Objects.requireNonNull(checks, "checks"));
if (this.checks.isEmpty()) {
throw new IllegalArgumentException(
"at least one health check is required");
}
}
public OverallStatus checkAll() {
List<CheckResult> results = new ArrayList<>();
boolean allHealthy = true;
for (HealthCheck check : checks) {
long start = System.nanoTime();
try {
CheckResult raw = check.check();
long latencyMs = (System.nanoTime() - start) / 1_000_000;
results.add(new CheckResult(check.name(), raw.healthy(),
raw.detail(), latencyMs));
allHealthy &= raw.healthy();
} catch (RuntimeException e) {
long latencyMs = (System.nanoTime() - start) / 1_000_000;
results.add(new CheckResult(check.name(), false,
e.getClass().getSimpleName() + ": " + e.getMessage(),
latencyMs));
allHealthy = false;
}
}
return new OverallStatus(allHealthy, List.copyOf(results));
}
}
// Structural sketch — the HTTP adapter for the health aggregate above.
@RestController
public class HealthController {
private final DependencyHealth health;
public HealthController(DependencyHealth health) {
this.health = health;
}
@GetMapping("/health")
public ResponseEntity<DependencyHealth.OverallStatus> health() {
DependencyHealth.OverallStatus status = health.checkAll();
return ResponseEntity
.status(status.healthy() ? 200 : 503)
.body(status);
}
}
Each HealthCheck implementation does something real: the vector-store check runs a trivial nearest-neighbor query, the embedder check embeds a fixed probe string, the LLM check sends a minimal prompt. Expensive? Slightly. Worth it? The first time your load balancer routes traffic away from a node whose LLM credentials expired — while a static 200 would have kept serving errors — you'll think so.
Evals wired in: the regression set is a release gate
This is the eval-harness idea from the evals lesson, now bolted to the release pipeline. A fixed set of question/answer expectations runs against the real query path — not a mock of the prompt — before every deploy. Any failure blocks the release. The harness is compiled pure Java and runs in CI in seconds:
package com.docsbot.rag.eval;
import com.docsbot.rag.service.QueryTypes.QueryResult;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;
import java.util.Objects;
import java.util.function.Function;
public final class RegressionRunner {
/** One regression case: the question plus machine-checkable expectations. */
public record EvalCase(String id, String question,
List<String> mustContainPhrases, int minCitations) {
public EvalCase {
Objects.requireNonNull(id, "id");
Objects.requireNonNull(question, "question");
mustContainPhrases = List.copyOf(mustContainPhrases);
if (minCitations < 0) {
throw new IllegalArgumentException("minCitations must be non-negative");
}
}
}
public record CaseVerdict(String caseId, boolean passed,
List<String> failures, long latencyMs) {}
public record EvalReport(int passed, int failed, List<CaseVerdict> verdicts) {
/** The release gate: deploy only when every case passes. */
public boolean deployAllowed() {
return failed == 0 && passed > 0;
}
}
public EvalReport run(List<EvalCase> cases, Function<String, QueryResult> ask) {
List<CaseVerdict> verdicts = new ArrayList<>();
int passed = 0;
for (EvalCase c : cases) {
long start = System.nanoTime();
List<String> failures = new ArrayList<>();
try {
QueryResult result = ask.apply(c.question());
String lowered = result.answer().toLowerCase(Locale.ROOT);
for (String phrase : c.mustContainPhrases()) {
if (!lowered.contains(phrase.toLowerCase(Locale.ROOT))) {
failures.add("missing expected phrase: \"" + phrase + "\"");
}
}
if (result.citations().size() < c.minCitations()) {
failures.add("expected at least " + c.minCitations()
+ " citation(s), got " + result.citations().size());
}
if (result.refused()) {
failures.add("query was refused (" + result.refusalReason() + ")");
}
} catch (RuntimeException e) {
failures.add("threw " + e.getClass().getSimpleName()
+ ": " + e.getMessage());
}
long latencyMs = (System.nanoTime() - start) / 1_000_000;
boolean ok = failures.isEmpty();
if (ok) {
passed++;
}
verdicts.add(new CaseVerdict(c.id(), ok,
List.copyOf(failures), latencyMs));
}
return new EvalReport(passed, verdicts.size() - passed,
List.copyOf(verdicts));
}
}
Two honesty notes that belong in your runbook, not just this post. First: the regression set detects regressions — it proves the new build still does what the old build did. It does not prove the answers are good; that's what the broader eval set and human review are for. Second: the cases must run against the real path with the real prompt version and model, or you're testing a system that doesn't exist. A green eval run against a mock is worse than no eval run, because it manufactures confidence.
Rate limiting: a token bucket per client
LLM calls cost money and have provider-side quotas; one misbehaving client — or one retry loop with no backoff — can burn budget and starve everyone else. A token bucket per client key is the honest, boring answer: bursts up to capacity are allowed, sustained rate is capped, and excess requests get a clean 429 instead of a slow, expensive failure.
package com.docsbot.rag.limit;
/**
* Thread-safe token-bucket rate limiter. Bursts up to {@code capacity} are
* allowed; sustained rate is capped at {@code refillPerSecond}.
*/
public final class TokenBucketRateLimiter {
private final double capacity;
private final double refillPerSecond;
private double tokens;
private long lastRefillNanos;
public TokenBucketRateLimiter(double capacity, double refillPerSecond) {
if (capacity <= 0 || refillPerSecond <= 0) {
throw new IllegalArgumentException(
"capacity and refillPerSecond must be positive");
}
this.capacity = capacity;
this.refillPerSecond = refillPerSecond;
this.tokens = capacity;
this.lastRefillNanos = System.nanoTime();
}
public synchronized boolean tryAcquire() {
refill();
if (tokens >= 1.0) {
tokens -= 1.0;
return true;
}
return false;
}
private void refill() {
long now = System.nanoTime();
double elapsedSeconds = (now - lastRefillNanos) / 1_000_000_000.0;
if (elapsedSeconds > 0) {
tokens = Math.min(capacity, tokens + elapsedSeconds * refillPerSecond);
lastRefillNanos = now;
}
}
}
The QueryService holds one bucket per client via a small registry (RateLimitRegistry, also compiled) — per-tenant fairness without a shared global bucket that lets one tenant crowd out the rest. Set capacity and refill from measured traffic, not from intuition: start from your provider's rate limits and your own load tests, then tune from the 429 rate in telemetry. And return 429 Too Many Requests with a Retry-After hint — a client that knows when to come back is a client that doesn't hammer you.
Cost awareness: log the arithmetic, plug in real prices
Every LLM request has a price shaped like this: (promptTokens × inputPrice + completionTokens × outputPrice) ÷ 1,000,000. The shape is stable; the prices are not — providers change them, and different models price differently. So the estimator takes prices as injected configuration, and the post you are reading invents no numbers:
package com.docsbot.rag.cost;
import java.math.BigDecimal;
import java.math.RoundingMode;
import java.util.Objects;
/**
* Estimated per-request cost in USD from token counts.
*
* <p>Pricing is INJECTED, never invented here: copy the current per-token
* (or per-million-token) prices from your provider's pricing page into
* configuration. Provider prices change; code constants do not track them.
*/
public final class CostEstimator {
/** Prices per million tokens, in USD, for one model. */
public record ModelPricing(String modelName,
BigDecimal inputPerMillionTokens,
BigDecimal outputPerMillionTokens) {
public ModelPricing {
Objects.requireNonNull(modelName, "modelName");
Objects.requireNonNull(inputPerMillionTokens, "inputPerMillionTokens");
Objects.requireNonNull(outputPerMillionTokens, "outputPerMillionTokens");
if (inputPerMillionTokens.signum() < 0
|| outputPerMillionTokens.signum() < 0) {
throw new IllegalArgumentException("prices must be non-negative");
}
}
}
private static final BigDecimal MILLION = BigDecimal.valueOf(1_000_000);
private final ModelPricing pricing;
public CostEstimator(ModelPricing pricing) {
this.pricing = Objects.requireNonNull(pricing, "pricing");
}
public BigDecimal estimateUsd(int promptTokens, int completionTokens) {
if (promptTokens < 0 || completionTokens < 0) {
throw new IllegalArgumentException("token counts must be non-negative");
}
BigDecimal inputCost = pricing.inputPerMillionTokens()
.multiply(BigDecimal.valueOf(promptTokens));
BigDecimal outputCost = pricing.outputPerMillionTokens()
.multiply(BigDecimal.valueOf(completionTokens));
return inputCost.add(outputCost).divide(MILLION, 6, RoundingMode.HALF_UP);
}
}
Because QueryService logs costUsd on every request, cost becomes a dashboard, not a surprise: per-tenant spend, cost per successful answer, and — the one that matters at 2 AM — a cost spike that fires before the invoice does. Note what the estimate isn't: it doesn't include embedding costs, vector-store costs, or retries. Extend the record when those matter to you; the point is that the arithmetic lives in one tested place, fed by configured prices, instead of scattered guesswork.
What an answer looks like (illustrative)
Putting it together, a healthy request flows like this — illustrative example output, not a measured run:
POST /api/ask X-Client-Id: acme-corp
{"question": "How do I rotate the API key for the billing service?"}
200 OK
{
"answer": "Rotate billing-service API keys from the admin console under
Settings → API Keys, then revoke the old key [1]. The old key
remains valid for a 24-hour grace period after rotation [2].",
"citations": [
{"n": 1, "document": "runbook-api-keys", "title": "API key rotation"},
{"n": 2, "document": "runbook-api-keys", "title": "API key rotation"}
],
"requestId": "3f9a…",
"promptVersion": "rag-answer-v3",
"latencyMs": 1840,
"tokens": {"prompt": 1213, "completion": 64},
"estimatedCostUsd": 0.0041
}
Every field the principle demands is there: the answer, what it cites, which prompt version and model produced it, how long it took, what it cost. Answering for its answers.
The deployment checklist
Before the docs assistant goes to customers, walk this list. Each item exists because a demo skipped it and paid later:
- Regression set green in CI.
RegressionRunnerruns against the real query path;deployAllowed()is false → the pipeline stops. No manual override without a recorded reason. - Secrets via environment or secret manager.
Secrets.requireEnvfails fast at startup. No keys in properties files, no keys in the repo, no keys in logs — ever. - Prompt version pinned and logged. The deploy manifest names the prompt version; every request logs it. Rolling back a bad prompt is a config change, not a code deploy.
/healthwired to the load balancer. Dependency checks exercise the vector store, embedder, and LLM. Unhealthy → 503 → traffic routes away.- Rate limits set from measured traffic. Capacity and refill per tenant, tuned from load tests and the 429 rate in telemetry.
Retry-Afteron every 429. - Cost dashboards and alerts. Per-request
costUsdis logged; alert on spikes per tenant before the invoice arrives. Prices copied from the provider's pricing page into config — re-check them on a schedule, because they change. - Data classification decided up front. Which documents are approved for embedding, what gets minimized or redacted before it leaves your boundary, and how long query logs are retained. Decide this before the first customer question, not after the first incident.
- Blocklist reviewed as a tripwire. It catches the cheap, obvious cases. It is documented as a tripwire — not a security boundary — in the design doc, so nobody builds on the assumption of a wall.
- Incident runbook for dependency outage. When the LLM or vector store is down, the service degrades to the safe fallback in
QueryService— never a stack trace, never a half-answer. The runbook says who gets paged and what "degraded" means to customers. - Rollback plan rehearsed. Previous prompt version and previous model config remain deployable. You will need this the week a provider deprecates a model with thirty days' notice.
Where to go from here
You now have the full arc of the track in one service: calling LLMs from Java, shaping answers with prompts, retrieving with embeddings and vector search, grounding with RAG, structuring with frameworks, constraining with function calling, and measuring with evals — assembled into something you could actually operate. The honest next steps are the ones no single post can give you: run the eval set against your own documents and watch where it fails, tune topK and chunk sizes against measured quality instead of defaults, and let a month of telemetry teach you what your users actually ask.
If you followed the track in order, revisit the earlier lessons with operator eyes — the prompt-engineering lesson reads differently once a prompt version is a deploy artifact, and the evals lesson reads differently once a failing case blocks a release. That second reading is the real capstone.
Continue: Java Learning Roadmap 2026
Comments
Post a Comment