AI for Java Developers: The Landscape

Picture a standup not unlike yours. The product manager says: "Customers keep asking whether we have AI. Can we add AI to the support dashboard by next quarter?" Three developers hear three different projects. One hears a chatbot widget. One hears automatically summarized tickets. One hears "replace the search box with something smarter." All three nod. None of them means the same thing.

That gap — everyone saying "AI" while nobody shares a vocabulary — is the real blocker on most teams. Not the math. Not the model weights. The words. This post fixes that. It gives you a map of the landscape: what a large language model actually does, the handful of building blocks every AI feature is assembled from, and exactly where your Java code sits in the picture. Theory is kept to the minimum you need to navigate the rest of this track — everything else is grounded in one concrete feature we will carry through all ten posts.

The running example. We will ground every term in a single feature: a support-ticket assistant for an online store. When a customer writes "Where is my order #48291?", the assistant drafts a one-sentence reply for a human agent to approve. Drafts — not sends. A human approves every reply before it goes out. That approval step matters, and we will keep coming back to it.

What an LLM actually is

A large language model is a program trained on vast amounts of text to do exactly one thing: predict the next token — a token being a chunk of text, roughly a word fragment — given the tokens that came before it. Feed it "The capital of France is" and the most likely next token is " Paris". Ask it a question and it predicts a plausible continuation, one token at a time, until it decides to stop. There is no magic step beyond that loop; everything impressive about these models is what falls out of doing that loop extremely well.

You reach a model over HTTPS. Your code sends a JSON request containing a prompt, the provider runs the model, and a JSON response comes back with generated text. Two consequences of that shape matter enormously:

• The model is stateless. It remembers nothing between calls. If you want a conversation, your code resends the earlier messages with every request. Every call is a fresh, self-contained request — and the conversation history is your code's responsibility, not the model's.
• HTTP doesn't care what language called it. The model sits behind an ordinary web API. Your Java service calls it the same way it calls a payment gateway. That single fact is why this entire track exists.

What it isn't

Three misconceptions cause most of the expensive mistakes. Clear them now and half the track is just details.

It is not a database. A model doesn't look things up; it generates plausible text. Ask it for the status of order #48291 and it will cheerfully invent one. Decision rule: if a fact lives in your database, the model must never be its source of truth. Fetch the fact yourself and hand it to the model inside the prompt. Our ticket assistant will never ask the model "where is the order?" — our code will look up the order and tell the model "the order shipped yesterday; draft a reply saying so."

It is not deterministic. The same prompt can produce different wording on different calls. There is a temperature knob — lower values make output more repeatable, higher values more varied — but treat it as a rule of thumb: temperature is a sampling control, not a reliability dial. If your feature needs the exact same answer every time, that exactness has to come from your code: parse the output, validate it against what you expect, and retry or fall back when it doesn't fit.

It is not an agent by itself. Left alone, a model produces text and stops. It doesn't click buttons, run queries, or send emails. Anything that touches the world happens because your code ran. That sentence is the load-bearing wall of this track.

The vocabulary map: nine terms that cover 90% of conversations

With the model understood, here are the building blocks. Each one gets a plain definition and the sentence you'll use to recognize it in the wild.

1. Chat / completions API. The HTTPS interface itself: you send messages, you get text back. Providers differ in URL paths and field names — check the current docs for the one you use — but the shape is always the same, as the worked example below shows.

2. Prompt. The full input text you send: developer instructions ("You draft support replies; never promise refunds"), plus any context you retrieved, plus the user's actual question. Prompt engineering is the unglamorous craft of writing this text well — being specific, showing an example of good output, and stating what to do when information is missing.

3. Tokens. The model's unit of text. Requests are measured — and usually priced — in tokens, so "how many tokens is my prompt?" is really "how long and how expensive is this call?" Tokenizers differ between models, so treat any tokens-to-words conversion as approximate. The context window is how many tokens fit in a single call; limits change often enough that you should check the provider's current docs rather than memorize a number.

4. Embeddings. A function that turns text into a list of numbers (a vector), with the property that texts with similar meaning land near each other. Rule of thumb: vector similarity ≈ meaning similarity — roughly, and worth verifying on your own data before you bet a feature on it. Embeddings power search, deduplication, and clustering: anywhere you need "find me text like this."

5. Vector search / vector database. Nearest-neighbor lookup over stored embeddings: "find the 5 passages most similar to this question." A vector database is just a store optimized for that lookup at scale. This is the retrieval half of the next item.

6. RAG — retrieval-augmented generation. Don't ask the model to remember your documentation; fetch the relevant passages and paste them into the prompt, then ask. The answer is grounded in text you supplied, which directly attacks the "model invents facts" problem: it is much harder to hallucinate when the answer is sitting in the prompt. Our ticket assistant is a RAG system in miniature: look up the order, paste the facts into the prompt, ask for a draft.

7. Tools / function calling. The model can request that your code run a function — "look up order #48291", "check inventory for SKU-77" — by emitting a structured request naming the function and its arguments. Your code then runs an ordinary method call and feeds the result back. The model never touches your systems directly; every tool call is code you wrote, executing with permissions you granted.

8. Agents. A loop your code controls: ask the model, run the tools it requests, feed the results back, repeat until the task is done or a limit you set is hit. "Agent" is a heavy word for a simple idea — a while loop around the model with tools. Your code owns the loop, the iteration cap, and every approval gate inside it.

9. Evals. Tests for behavior you can't assert with ==. A small labeled set of inputs with expected properties ("mentions the tracking number", "stays professional"), re-run whenever you change the prompt, the model, or the retrieval setup. If you wouldn't ship a payment feature without tests, don't ship an AI feature without evals.

One more term you'll hear constantly: guardrails — validation on the way in and the way out. Strip secrets and PII from prompts before sending them; check the model's output before acting on it. Guardrails live in your code, not in the model — which brings us to the principle this whole track is built on.

Where Java fits

Nearly every LLM provider exposes an HTTPS API, and Java's built-in java.net.http.HttpClient speaks HTTPS. You need no special AI framework to make your first call — the example below uses only the JDK. When the boilerplate grows, two libraries earn their place:

• LangChain4j — Java-native building blocks (chat models, embeddings, vector stores, tool calling) with a plain-Java feel. The pick when you want AI primitives without a framework.
• Spring AI — model access through Spring's portable abstractions. The natural pick if your service is already on Spring Boot.

Both call the same provider APIs under the hood; they remove plumbing, they don't change the vocabulary. Decision rule: learn the wire shape first with plain HttpClient — that's this post's example. Reach for LangChain4j or Spring AI when you catch yourself hand-rolling retries, streaming, or tool-calling plumbing for the third time.

A note on the ecosystem: most AI tutorials and sample code you'll find online are in Python. That doesn't make Python required. The model sits behind an HTTP API, and HTTP doesn't care what language called it. If your system is Java, Java is a first-class way to call it — with your existing logging, metrics, connection pools, and deployment story coming along for free.

Worked example: your first LLM call in plain Java

Here is the complete shape of an LLM API call — no frameworks, just the JDK. It builds the ticket-assistant prompt from the last section, sends it, and prints what comes back. Save it as AiClient.java:

import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;

public class AiClient {

    private static final String ENDPOINT = "https://api.openai.com/v1/responses";

    public static void main(String[] args) {
        String apiKey = System.getenv("OPENAI_API_KEY");
        if (apiKey == null || apiKey.isBlank()) {
            System.err.println("Set the OPENAI_API_KEY environment variable to a real key first.");
            System.exit(1);
        }

        // The prompt: developer instructions + the facts OUR code looked up + the question.
        // The model never sees the database; it only sees this text.
        String body = """
                {
                  "model": "gpt-4o-mini",
                  "instructions": "You draft support replies. One sentence, friendly tone. Never promise refunds.",
                  "input": "Order #48291 shipped yesterday and arrives Thursday. Customer asks: Where is my order #48291?"
                }
                """;

        HttpRequest request = HttpRequest.newBuilder()
                .uri(URI.create(ENDPOINT))
                .timeout(Duration.ofSeconds(30))
                .header("Content-Type", "application/json")
                .header("Authorization", "Bearer " + apiKey)
                .POST(HttpRequest.BodyPublishers.ofString(body))
                .build();

        // HttpClient is AutoCloseable since Java 21, so try-with-resources applies.
        try (HttpClient client = HttpClient.newBuilder()
                .connectTimeout(Duration.ofSeconds(10))
                .build()) {
            HttpResponse<String> response =
                    client.send(request, HttpResponse.BodyHandlers.ofString());
            System.out.println("Status: " + response.statusCode());
            System.out.println(response.body());
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            System.err.println("Request was interrupted.");
            System.exit(1);
        } catch (IOException e) {
            System.err.println("Request failed: " + e.getMessage());
            System.exit(1);
        }
    }
}

Four things to notice, because each one is a lesson in miniature:

1. The key comes from the environment, never from source code. System.getenv("OPENAI_API_KEY") with an explicit guard is the whole secrets story at this level: no key in the repo, no key in the post, a clear error when it's missing. You need a real key from the provider to run this — the snippet compiles without one, but the call won't.
2. The JSON body is a text block. Java 15+ text blocks make request payloads readable. The "model" value is just an example — model names change, so check the provider's current docs rather than memorizing this one.
3. The timeout is a placeholder, not a recommendation. Thirty seconds here stands in for a number you should derive from your feature's actual latency budget: how long may a support agent wait for a draft before the UI should give up? Timeouts come from your workload's SLO, never from a tutorial's magic number.
4. The response is a JSON string your code must parse. This example prints it raw to keep the focus on the request shape. A production version would parse it with a JSON library such as Jackson and map the reply text into a record — ordinary Java, no AI involved.

Compile and run it:

javac --release 21 AiClient.java
java AiClient

Without a key set, the guard fires and the program exits before touching the network — which is exactly the behavior you want to verify first. With a real key, you get a 200 and a JSON body whose shape looks like this (illustrative example output — the real response is longer, and the wording differs on every call):

Status: 200
{"id":"resp_9f3...","output":[{"type":"message","content":[{"type":"output_text",
"text":"Your order #48291 shipped yesterday and is arriving Thursday — thanks for your patience!"}]}]}

Read that reply once more and notice what happened: the model predicted fluent text, but every fact in it — the order number, the ship date, Thursday — came from the prompt your code built. That is the whole game.

A small builder: records and text blocks doing real work

Hand-writing JSON strings gets old fast. Here is a tiny, honest builder that keeps the request shape in one place, using a record and a text block — the modern-Java pairing you'll see throughout this track. Save it as ChatRequest.java:

public record ChatRequest(String model, String instructions, String input) {

    public String toJson() {
        return """
                {
                  "model": %s,
                  "instructions": %s,
                  "input": %s
                }
                """.formatted(jsonString(model), jsonString(instructions), jsonString(input));
    }

    // Minimal JSON string escaping: enough for learning the shape.
    // Real code should use a library such as Jackson.
    private static String jsonString(String value) {
        return "\"" + value
                .replace("\\", "\\\\")
                .replace("\"", "\\\"")
                .replace("\n", "\\n") + "\"";
    }

    public static void main(String[] args) {
        ChatRequest req = new ChatRequest(
                "gpt-4o-mini",
                "You draft support replies. One sentence, friendly tone.",
                "Order #48291 shipped yesterday.\nCustomer asks: Where is my order?");
        System.out.println(req.toJson());
    }
}

Compile and run it — no key, no network, pure string building:

javac --release 21 ChatRequest.java
java ChatRequest

The escaping helper is deliberately naive: it handles quotes, backslashes, and newlines, and nothing else. That limitation is stated in the comment because it matters — a tutorial serializer that silently mangles a customer's message would be a bug factory. When your prompts carry real user text, reach for Jackson. Knowing where the naive version stops being enough is the skill.

The one picture: anatomy of an AI-powered answer

Every feature you will build in this track is a variation on one loop. Read it left to right, and notice who does what — and who doesn't:

The model appears exactly once in that picture, doing exactly one thing. Everything before it (instructions, retrieved facts, the question) and everything after it (parsing, validation, the decision about what the text means) is your code. Which is the principle to carry out of this post:

The model predicts text; your code decides what it means. The model never sees your database, never clicks your buttons, never approves a refund. Every fact it uses, every action it triggers, every output your users see — your code put it there. Treat the model as a brilliant, confident, occasionally wrong colleague: give it clear instructions, hand it the facts it needs, and check its work before anything leaves the building.

The route ahead: what this track covers

This post is the map; the next nine walk the territory, each adding a working piece to the ticket assistant until it's a service you could actually run:

1. The Landscape — this post: the map, the vocabulary, your first call.
2. Calling LLMs from Java — the full client: configuration, retries, timeouts, error handling, and parsing responses into records.
3. Prompt Engineering for Developers — instructions that survive contact with real users: structure, examples, and the failure modes to design against.
4. Embeddings & Vector Search — turning text into vectors and finding similar passages, in Java.
5. RAG from Scratch — grounding answers in your own documents: chunk, embed, retrieve, generate.
6. LangChain4j — the Java-native toolkit: chat models, embeddings, and tool calling without hand-rolled plumbing.
7. Spring AI — the same ideas through Spring Boot: portable model clients and Boot conventions.
8. Function Calling — letting the model ask your code to act: tools, approvals, and the trust boundary.
9. Evals — testing what you can't assert: labeled sets, judging outputs, and regression-proofing your prompts.
10. Capstone: a production RAG service — the ticket assistant from this post, hardened: configuration, observability, guardrails, and the human approval step.

What's next

You now have the shared vocabulary that the standup in the opening was missing: tokens, prompts, embeddings, vector search, RAG, tools, agents, evals — and the one principle that organizes them all. The next step is to make the wire shape second nature: configuration, retries, timeouts, and turning raw JSON into records you can trust.

Field check before you move on: compile both snippets from this post with javac --release 21 and run the ChatRequest builder. Read the JSON it prints and point at the three parts of the prompt — instructions, retrieved context, question. Then explain to a teammate which part of a model response your code would trust and which part it would validate. If you can draw that line crisply, the map has done its job.

Continue: Java Learning Roadmap 2026