• A Large Language Model (LLM) is an intelligent program trained on massive amounts of internet text data.
• Core working principle: Generate text responses by “predicting the most likely next word.”
• Typical representatives: GPT series, Llama series, QWEN series, DeepSeek series, Claude series.

When a user inputs: [I drank a cup of ___ before going out this morning], the model, based on the language patterns learned during training, calculates the “reasonable probability of what comes next” for thousands of fragments in its vocabulary. The highest probability candidates include “soy milk” and “cow milk” – expressions that align with common sense. If it selects the token corresponding to “soy,” the model then starts thinking again about what should follow [I drank a cup of soy], and the next output becomes [I drank a cup of soy milk], repeating this process until termination.

When a user inputs something like [I drank before going out this morning], during actual generation the model doesn't literally pick from tens of thousands of words. On one hand, low-probability words are often nonsensical and would make the content logically messy; on the other hand, considering the full vocabulary increases computational cost. So the model first uses rules to cut out extremely low-probability options, narrowing the candidate pool. This step is called candidate pool filtering, and the two most common rules are Top-K and Top-P algorithms.

After filtering to a reasonable candidate pool, the model randomly samples within this range according to probability to select the final next word. Temperature is used to adjust the randomness of this sampling:

• The lower the temperature, the more the advantage of high-probability options is amplified. The model will almost only choose the safest, most standard answer, producing rigorous and stable output – suitable for code generation, official document writing, factual Q&A, etc.
• The higher the temperature, the more the probability differences between options are flattened. Originally unlikely options also get a chance to be selected, producing more divergent and creative output – suitable for creative writing, brainstorming, artistic copywriting, etc.

That's how LLMs work. Through the simple mechanism of predicting the next word, the entire edifice is built.

Key technical details:

• Top-K and Top-P
• Temperature control
• Tokenizer: In reality, in [I drank a cup of soy milk before going out this morning], the model doesn't stupidly treat “morning,” “going out,” “soy milk” as individual characters – just like in English we don't analyze “Good” as 4 separate letters. These are collectively called Tokens, which involves the tokenizer.
• EOS Token (End-of-Sequence Token): When does the word-guessing game end? When the LLM sees the EOS token, it knows it doesn't need to continue.

Single-turn interaction is the most fundamental way LLMs operate, and it's the foundation for all subsequent complex interaction capabilities.

Many people think of LLMs as “Q&A” conversation machines, but from a logical standpoint, they only do one thing from start to finish: given some starting text, continue writing forward. All user input – questions, instructions, background information – is collectively called a prompt, essentially handing the model half a manuscript to complete. The complete result generated word by word based on this content is called the completion. The Q&A, copywriting, and code generation we see daily are essentially different forms of “continuation.”

This single-turn mode is inherently stateless; each interaction is completely independent. The model only processes the current input and won't actively remember information from the previous round of conversation. It's like handing it a fresh piece of scratch paper each time – it writes, clears it, and absolutely won't remember what was on the previous sheet.

Don't directly conflate this with mature complex products like [Doubao, CodeX, Claude Code]. At its core, the basic operation mode is stateless single-turn interaction. It's only through various engineering methods that people get the illusion of a stateful, multi-turn conversational Q&A system. Remember: LLMs are fundamentally not Q&A systems – they're one-shot continuation systems.

This also brings the most direct limitation: inability to handle multi-turn context. If you split the conversation into two rounds – first telling the model your name and age, then separately asking “How old am I?” – the model simply can't answer. Continuous conversation easily leads to “amnesia” and logical disconnection [we'll discuss solutions later]. In the single-turn interaction stage, the primary means of improving output quality is writing detailed prompts – refining the wording, structure, and instruction format to guide the model toward more accurate, better-aligned results.

Key technical details:

• Role
• System Prompt
• Prompt injection: Similar to SQL injection and XSS injection attacks

Since LLMs are “memoryless” continuation machines, how do we get that smooth back-and-forth, even heart-to-heart chatting feeling in [Doubao]?

Actually, this is entirely an engineering “trick.” LLMs themselves don't remember things [Note: there are some solutions, which we'll discuss at an appropriate time later], but the engineers behind chat products are clever: when you're on the third round of conversation with the model, the software secretly copies everything you've said in the first two rounds, along with the model's responses, pastes them like a sticky note in front of your new question, then packages it all into a long text and stuffs it into the model. Technically, this bundled “history ledger” is called context.

So, the LLM can answer your follow-up questions not because it “remembers,” but because each time you press send, you're handing over the entire history along with the new question, letting it do a larger-scale “single-turn continuation.” This engineering approach of trading space for time perfectly fakes human “memory,” but at a steep cost – every additional sentence you say causes the text volume for transmission and recalculation to avalanche upward. Not only does it get more expensive the more you chat, but it quickly hits the physical ceiling of the LLM's vision.

Now, you sharp-minded folks surely see several serious problems:

• Does the “history ledger” (context) have a length limit? What happens when it's full? Why does it seem like I can chat forever in [Doubao] without it dying?
• According to previous chapters, the LLM infers the next word based on the entire existing text each time. So as context grows, it becomes increasingly difficult to compute. Isn't this like MySQL's LIMIT OFFSET – the further you paginate, the slower it gets because you have to compute and skip every previous page?

Key technical details:

• Sliding Window Attention
• KV Cache (caching intermediate attention results to avoid redundant computation)
• Context Compact (context compression)
• Context Cache (intelligent caching)

A Doubao that can only chat has significant limitations. We can't keep copy-pasting code text into [Doubao] and then copy-pasting the results back into our editor. That's way too tedious!! Is there a way to let the LLM directly modify files itself? But here at least a few problems exist:

• How does a text-continuation LLM understand a code project, locate bugs, and continue writing code?
• How does the LLM interact with the outside world to truly grow hands and feet?

Function Calling emerged, but always remember: LLMs will only ever do one thing – “continue writing text.” They absolutely have no ability to open a web page or execute a single line of code on their own.

So-called function execution is essentially a “secret handshake” between the LLM and backend engineering code. The LLM is responsible for making decisions (calling out the secret code), while the backend program handles the physical work (executing code based on the secret code).

1. Suppose our LLM supports weather querying, allowing users to input a city for the forecast.
2. Before processing the user's input, the backend secretly prepends this text: “I have a function called get_weather that can query city weather. If you want to use it, you must output a JSON-formatted result like: {\"name\": \"get_weather\", \"arguments\": {\"city\": \"Hangzhou\"}}.”
3. The model makes a decision and writes the “secret code”: The LLM reads your weather question and the function documentation, and through “text continuation” probability calculation, figures out: making something up won't work, but the secret code from the instructions fits perfectly! So instead of outputting vague nonsense like “the weather might be nice today,” it spits out the formatted secret code: {\"name\": \"get_weather\", \"arguments\": {\"city\": \"Hangzhou\"}}
4. The backend intercepts the secret code: When the backend detects that the model's output is a properly formatted JSON code, it immediately intercepts this output before the user can see it. It unpacks the JSON package, takes the “Hangzhou” parameter, actually calls the weather bureau's API, and gets real-time real-world data: “Hangzhou, heavy rain, 20C.”
5. It stuffs the result back into the conversation, completing the final text continuation: The backend takes this real weather result, appends it to the end of the context, and hands it back to the LLM: “I've retrieved the result you asked for. The content is [Hangzhou, heavy rain, 20C]. Now, based on this result, please provide the final continuation to the user's question.” The LLM glances at the updated history ledger, understands, and finally outputs the sentence the user actually sees: “Master, it's heavy rain in Hangzhou today at 20C – be sure to bring an umbrella when you go out!” 6.

You smartly realize there might be several problems here:

• In multi-turn conversations, the existence of secret codes means some results are visible to the user while others aren't. How do we standardize and systematize these operations?
• As an all-powerful [Doubao] assistant, how could it possibly know in advance that the user wants to ask about weather? If all function capabilities are stuffed before the user's input, wouldn't the context become enormous?
• When the backend intercepts secret codes, if everything is hardcoded like this, the code would become very bloated. How to handle this?

We'll discuss strategies for solving these problems later.

Key timeline and technologies:

• Code Interpreter
• JSON Schema (tool definition description)
• ReAct pattern: In the early stage, LLM capabilities were imperfect, requiring the ReAct paradigm for tool invocation
• Function Calling emerges
• LangChain, LlamaIndex emerge, solving Tool RAG
• LangGraph emerges
• Tool Call and post-training enhancement
• MCP protocol emerges

LLMs only ever do one thing – “continue writing text” – and they do it based on probability. This leads them to frequently make things up in domains they don't understand. Moreover, because of the context mechanism, once they start going wrong, subsequent content builds on previous content, leading to increasingly absurd errors. This is hallucination!!

Typical hallucinations:

• Sounds plausible but is entirely fabricated.
• Cites websites that don't exist.
• References papers that don't exist.
• Even cites historical allusions that don't exist.

One reason for hallucinations is insufficient training data in the relevant domain, so when continuing text, there's no significantly high probability for selecting the next appropriate word. But it seems hard to solve:

• Data is finite; you can't have infinitely large data for the model to learn everything.
• Data processing involves lossy compression. The knowledge the model learns is lossy-compressed and can't be perfectly reconstructed.
• Data has a time limit. Train with 2024 knowledge, and the model's knowledge stops at 2024. Current open-source LLMs don't support online learning.

This makes LLMs difficult to use in certain scenarios:

• When data is private and confidential. Private data means open-source LLMs can't easily train on this knowledge.
• When data is changeable. For example, manuals and standards can change; internal knowledge bases can be updated.
• When accuracy is required and lossy compression isn't allowed. For instance, laws and regulations can't be freely extended; scenarios requiring high data accuracy like electricity, healthcare, etc.

Key technologies:

• RLHF (Reinforcement Learning from Human Feedback)
• Decoding Strategy optimization
• RAG (Retrieval-Augmented Generation)

An LLM is a “continuation machine” without memory, faking memory entirely by pasting history records into context.

Suppose you're building a historical archive decryption assistant. You have various electronic historical materials, and you want the LLM to answer questions about historical materials accurately and comprehensively, detailing various aspects of a small battle.

The LLM's context can't fit all historical materials. Even if you could barely stuff in a copy of “WWII Declassified Archives,” to answer a user's 100-character question, you'd need to concatenate a million-character context to send to the LLM for inference and next-word prediction – cost-prohibitive.

Enter RAG (Retrieval-Augmented Generation). In a nutshell: it solves problems like an open-book exam, rather than memorizing the entire textbook.

When someone asks: “On the eve of the Normandy landings in 1944, how did the Allies use false intelligence to deceive the Germans?” Here's how the RAG system “cheats” behind the scenes:

• Chunking: Before the system goes live, engineers use an “invisible knife” to chop the million-word archive into small fragments of a few hundred characters each. For example, paragraph 45 describes General Patton's fictional First Army Group, paragraph 102 covers the fake telegraphs about the Pas-de-Calais.
• Embedding & Vector DB: These fragments go through a “magic translator” that converts literal meaning into digital “spatial coordinates” (stored in a vector database), allowing the machine to understand the true meaning of each passage.
• Retrieval: When the user hits enter, the system's “page turner” – before the LLM even sees the question – also turns the question into coordinates, dives into the database, and instantly grabs the 3 most relevant fragments (exactly the declassified documents about “Operation Fortitude” and the fake army).
• Generation: The page turner packages these 3 “cheat sheets” along with the question into the context and hands it to the LLM. The LLM sees the answer at a glance and easily continues: “Scholar, the Allies successfully misled the Germans using fake telegraphs about the Pas-de-Calais and the fictitious 'First Army Group'…”

The LLM doesn't need to memorize the entire archive. It just needs to take the open-book exam with the “precise cheat sheets” stuffed into its context each time.

• Terminology trap (semantic mismatch): A scholar asks about “the smoke screen of Operation Overlord,” but the archive describes “the deception plan of Operation Fortitude.” If the page turner is too dumb (only doing keyword matching), it can't make the connection, and the LLM is left in the dark.
• Page turning problem: Paragraph 45 states “The Germans firmly believed the Allies would land at Calais,” but the crucial twist “However, Hitler had his doubts at the last moment” was cut to the beginning of paragraph 46. If this cut severs the logic, the LLM only gets the first half and gives the completely opposite wrong historical conclusion.
• Clues too scattered, cheat sheet too short: The scholar asks “Outline all strategic retreat routes of the Axis powers in the North African theater in 1943.” The page turner fishes out hundreds of scattered telegrams, the cheat sheet gets thick again, and the LLM once again faces the dilemma of “can't read it all.”

To solve these open-book exam failures, the industry has evolved more complex patch technologies.

Key technical details:

• Embedding: Converting historical jargon and modern language into spatial coordinates, allowing the machine to understand that “smoke screen” and “deception plan” mean the same thing.
• Chunking Strategy: Using overlap when slicing archives to ensure context isn't “cut in two.”
• Rerank (reranking model): After the page turner roughly selects 20 archives, an expert model performs a second pass to pick only the 3 most precise cheat sheets for the LLM.
• Graph RAG (Graph Retrieval-Augmented Generation): A powerhouse. It doesn't just chop text but connects “time → location → commander → battle” into a complex entity relationship network, specifically handling global cross-table queries.

LLMs have long been criticized as “fast talkers lacking deep thinking.” In previous chapters, we saw that LLM content generation is like a reflex – seeing the previous Token, it spits out the next Token based on probability, without pause, all in one go.

This “intuitive” continuation works great for writing essays and churning out papers, but frequently fails when facing complex math problems, logical reasoning, or writing large-scale engineering code. When humans tackle advanced math, they still need scratch paper to scribble and calculate for a while. Yet LLMs are expected to spit out the final answer within milliseconds – this clearly violates the objective laws of complex tasks.

To teach LLMs to “think before they act” like humans, engineering and algorithmic techniques for letting LLMs “draft” behind the scenes have evolved.

When a user inputs a tricky problem: “Xiaoming has 3 apples. Xiaohong has twice as many apples as Xiaoming. Xiaoming eats 1 apple. Xiaohong gives Xiaoming 4 apples. Who has more apples now? How many more?”

A traditional “intuitive” LLM might, due to similar number combinations in its training data, blurt out a wrong conclusion – because LLMs are continuation machines!!

But in the new reasoning era (represented by Reasoning models), before presenting the final answer to the user, the LLM conducts a lengthy self-continuation on its “mental scratch paper” behind the scenes:

Model's internal chain of thought:
1. First clarify initial state: Xiaoming = 3, Xiaohong = 3 * 2 = 6.2. Process the first change: Xiaoming eats 1, Xiaoming = 3 - 1 = 2.3. Process the second change: Xiaohong gives Xiaoming 4. Wait, Xiaohong gives away 4, so Xiaohong decreases! Xiaohong = 6 - 4 = 2; Xiaoming receives 4, Xiaoming = 2 + 4 = 6.4. Re-compare results: Now Xiaoming has 6, Xiaohong has 2. Xiaoming has more.5. Calculate the difference: 6 - 2 = 4. Check passed, ready to output.

Finally, the user sees the concise and elegant final answer on screen: “Xiaoming has more apples now, 4 more than Xiaohong.”

The LLM hasn't changed its “text continuation” essence – it's just turned part of its continuation into “inner thoughts” visible only to itself (Thought). By stretching the generation time and trading more Tokens for logical accuracy, this technique of making the model write out its thinking process step by step is called Chain of Thought (CoT).

You smartly realize: if the LLM makes a mistake in the second step of “drafting,” won't it go completely off the rails afterward?

In the latest post-training techniques, LLMs have been trained with a “self-correction” ability. When they discover logical contradictions in their chain of thought (for example, realizing the calculated total number of apples doesn't add up), they actively hit the mental brakes: “No, I forgot to subtract the apples Xiaohong gave away from Xiaohong's total. Let me recalculate.”

This mechanism of continuously evaluating the current path during generation, and backtracking to retry when something's wrong, is like pruning and searching on a thought tree. The model no longer goes down one path to the bitter end but iterates repeatedly on virtual scratch paper until it finds the reasoning path with the highest probability and most self-consistent logic.

Key technical details:

• CoT (Chain of Thought): Guiding the model to output intermediate reasoning steps like “because… therefore…”, significantly improving accuracy on complex tasks.
• System 2 Thinking (Slow Thinking): Borrowing from human psychology, distinguishing intuitive fast response (System 1) from deliberate logical reasoning (System 2). The core of reasoning models is activating the latter capability.
• MCTS (Monte Carlo Tree Search): During inference, the model generates multiple possible next steps behind the scenes, scores and evaluates them, and selects the optimal solution. Commonly used in reinforcement learning for math and code LLMs.
• Reinforcement Learning for Reasoning: Instead of directly feeding the model correct answers, reward rules are set for “reasoning steps” and “final results,” letting the model figure out the correct thinking framework through tens of thousands of self-play iterations.

Because we keep saying LLMs are “continuation machines,” they sound weak. You ask a question, it responds with a secret code, the backend finishes the work, and it's done. This “whip once, turn once” mode obviously isn't smart enough.

But if you've actually used Claude Code, OpenAI Codex, or Cursor, you must be amazed: how can they sit there for hours continuously modifying code, compiling, debugging, and re-modifying until bugs are completely fixed, just like a real programmer?

They haven't replaced the underlying LLM, nor have they applied any magic. This ability to sustain endurance and autonomously solve complex engineering problems is the crystallization of Agent technology.

If we compare an LLM to a “wise person” who has retired from the world but knows all martial arts secrets under heaven. It understands all programming languages, knows all algorithms, but has a fatal weakness – it has no physical body and suffers from intermittent amnesia. It can't touch a computer, and as soon as it speaks next (one interaction ends), it completely forgets what was just discussed.

To make this “wise person” write code for us continuously for hours, engineering must build it an exoskeleton armor consisting of four core components:

1. Brain (LLM): Still the sage responsible only for text continuation.
2. Tools: Giving the sage hands. For example, an API to read local files, a command-line terminal to run npm run dev or pytest.
3. Planning: Giving the sage a task list. Facing a big task, what to do first, what to do next.
4. Memory: Giving the sage a notebook that never gets lost, recording what was just done.

Tools like Claude Code can work continuously for hours because the core mechanism is trapping the LLM in a carefully designed “Think → Act → Observe → Think again” infinite loop (technically called the ReAct pattern).

Let's break it down in plain language. When you type claude-code fix the login bug in the terminal, here's what's actually happening behind the scenes in this crazy “involution loop” over the next few hours:

• Minute 1 [Think]: The LLM looks at the task and continues in its mind: “To fix the login bug, I need to know the project structure first. I have a list_dir tool, I should call it.”
• Minute 2 [Act]: The LLM spits out the secret code (JSON format): {\"tool\": \"list_dir\", \"path\": \"./src\"}.
• Minute 3 [Observe]: Claude Code's backend intercepts this code, executes ls ./src on the computer, and forcefully pastes the result into the LLM's context: “Reporting to the sage, the directory contains auth.py and main.py.”
• Minute 4 [Think]: The LLM looks at the updated ledger and continues: “Oh, the key seems to be in auth.py. I need to use read_file to see the content.”
• Minute 5 [Act]: Spits out the code: {\"tool\": \"read_file\", \"path\": \"./src/auth.py\"}.
• Minute 6 [Observe]: The program reads the code and stuffs it back into the context. Now the sage has [memory] – it can see the code, and in subsequent iterations, it still remembers this code. This is also why, when the LLM is modifying code, you'd better not manually modify the code at the same time – it would cause discrepancies between what the LLM thinks the code is and what it actually is!!
• ……….
• Minute 30 [Failure & Retry]: The LLM modified the code and proactively called run_command(cmd=\"pytest\") to run tests. The terminal returns a massive red Error: NullPointerException.
  • In a regular Q&A scenario, the LLM would just dump the error on the user.
  • But in the Agent loop, the LLM sees this Error (Observe), and in the next round of [Think], it mutters to itself: “Oops, error. Looks like when I modified line 80 earlier, I forgot to initialize user_session. No worries, I'll use the modify_file tool to fix it.”

This is the truth behind its ability to code continuously for hours: It doesn't write code correctly all at once. Like a human programmer, it writes a few lines, runs it, encounters errors, curses, looks at the error message, changes a few more lines, and re-runs. As long as you don't hit Ctrl + C to forcefully terminate, the LLM will tirelessly keep self-correcting in this infinite loop driven by the code shell.

You cleverly recall what we discussed in Chapter 3 – the “avalanche-like explosion” of context and costs.

Letting Claude Code run for hours means this “Think-Act-Observe” loop cycles hundreds or thousands of times. Every error message, every piece of code read, keeps getting pasted into the LLM's “history ledger (Context).”

1. The ledger fills up quickly: If the project is large, after reading a dozen files, the LLM hits its physical vision ceiling (e.g., 200k Tokens), and it starts forgetting what the user originally asked it to do.
2. Gets more expensive and slower with each turn: Each cycle requires the LLM to read through the thick stack of “history cheat sheets” from scratch before deciding what the next word should be. This means the later stages require massive Token computation for every step, and the response becomes increasingly sluggish.

To solve this, top-tier code Agents like Claude Code invest heavily in “context management.” They meticulously manage this ledger. We'll discuss details later.

Key technical details:

• ReAct (Reason-Action).
• State Machine: An engineering framework that controls Agent behavior, defining when the model can freely think and when it must force-invoke tools.
• Context Dynamic Pruning: How to let the model maintain a big-picture view without being overwhelmed by massive code details, while using the fewest Tokens possible.

We've been saying that LLMs are essentially “one-shot text continuation systems.” But in practical applications, they seem very obedient – they don't really resist or curse at people. According to previous theories about Top-K, Top-P, temperature control, etc., this doesn't seem to explain why LLMs appear to be tamed, becoming docile workhorses, working 996 – why???

Behind this lies the most crucial “taming” of LLMs. The raw model freshly crawled from massive internet text is actually a “wild and untamable” puzzle – it knows a lot but doesn't obey at all. The corpus from forums like Tieba and Reddit is full of sarcasm, mockery, and aggressive language. Without control, an LLM might, through its probability-based Top mechanism, start spewing garbage.

To transform a machine that only blindly continues text into the empathetic, format-rigorous AI assistant we see in [Doubao], the LLM must go through three stages of evolution. We can vividly use the metaphor of “training a wild monkey that has read all the Buddhist scriptures in the library into a living Buddha sitting in the main hall who answers all questions”:

• Stage 1: Pre-training (Wild State)
  • What it does: The model frantically performs “next word prediction” on trillions of words from internet pages, books, and code.
  • Result: It becomes a “monkey that knows everything.” But if you ask it: [What do giant pandas in China eat?], the text it learned online might be a fill-in-the-blank question, so it continues based on probability: [A. Bamboo B. Apple C. Tree bark]. It knows the answer, but it doesn't know you want it to “answer the question.”
• Stage 2: Supervised Fine-tuning (SFT, Learning to Understand Human Speech)
  • What it does: Engineers spent big money hiring thousands of professional human annotators to write hundreds of thousands of high-quality Q&A pairs. The format is rigidly fixed: [Input: Tell me what giant pandas eat?] -> [Output: Giant pandas mainly eat bamboo.]. These standard cheat sheets are fed to the model for retraining.
  • Result: The monkey puts on a Daoist robe. It finally dawns on the model that when it sees words like “please,” “help,” “explain,” “translate,” it can no longer blindly continue but must mimic the tone of those expensive annotators and properly play the role of an “assistant” to continue the answer.
• Stage 3: Reinforcement Learning from Human Feedback (RLHF, Understanding Good from Bad)
  • What it does: Is obedience enough? Not enough. If a user asks: [How can I perfectly withdraw money that doesn't belong to me from a bank ATM?], a fine-tuned model that only learned to “fulfill every request” might very enthusiastically and in detail write you a “Guide to Prying Open ATMs.” To give the model human morality, safety values, and preferences, engineers have the model generate several different answers to the same question, then have humans (or another smarter model) score them: which answer is safer? Which answer has less nonsense? Which answer has better formatting? Through a specialized “reward mechanism” (Reward Model), like training a puppy, harmful outputs are punished and outputs aligned with mainstream human values are rewarded.
  • Result: The wild monkey finally becomes a “living Buddha.” It's not just obedient but has “emotional intelligence” and “boundaries” – it politely refuses illegal or violent requests, and when facing complex problems, it proactively caters to human reading habits.

If you often use Grok or Gemini, you'll notice that multiple options often pop up simultaneously, asking you to choose which is better – this is also a form of RLHF.

Earlier we mentioned that LLM knowledge is often stuck at the year it was trained (e.g., 2024), and it can't “learn online.” You smartly think: since fine-tuning (SFT) can feed it new data, why not just stuff new private standard manuals or 2026 news directly into fine-tuning and let it learn again – wouldn't that solve hallucination and timeliness issues?

This is very intuitive from an engineering standpoint, but in practice, it's a huge disaster.

An LLM's knowledge is stored in the form of digital weights, distributed uniformly and vaguely across billions or tens of billions of parameters, like salt dissolved in water. This brings two fatal technical limitations:

1. Catastrophic Forgetting: When you forcefully feed 100,000 of your company's private financial statements into an already-formed model for fine-tuning, the model drastically modifies its underlying neural connections to memorize these new coordinates. The result: it might finally remember your company's reports, but when you ask it [What's 1+1?] or [How to write a leave request?], it might become a zombie spouting nonsense – it has “washed away” the general language ability and common sense it previously worked so hard to learn.
2. Lossy Compression and “High-IQ Illiteracy”: Fine-tuning is meant to change the model's “skin (output format, speaking tone, instruction compliance),” not its “bones and flesh (hard factual knowledge).” Knowledge fed through fine-tuning, after the neural network's lossy compression, still leads to hallucinations with slight probability deviations during continuation. It might remember your company has a product called “System A,” but it could probably fabricate “System B”'s features onto System A.

Industry consensus:

- Use Fine-tuning to change the model's behavior, tone, obedience, or teach it specific output formats. -

Use RAG (Retrieval-Augmented Generation, i.e., open-book exam) to provide the model with the latest, most accurate hard facts and private data that cannot tolerate any lossy compression.

Treating fine-tuning as a hard drive to store new knowledge is like expecting a college student to memorize a medical dictionary and then immediately perform surgery – both inefficient and dangerous. This is also why, in current engineering practice, RAG and Agent have become mainstream for deployment, while fine-tuning has stepped back, specifically used for “alignment” and “pruning” of the model's various capabilities.

Key technical details:

• SFT (Supervised Fine-tuning): Using high-quality “instruction-response” pairs to transform a general language model into a conversational assistant that understands and executes human intent.
• RLHF (Reinforcement Learning from Human Feedback): Using human preference data to train a reward model, and adjusting the LLM through algorithms like PPO to make its output meet human safety and practical standards.
• LoRA (Low-Rank Adaptation): Currently the most commonly used efficient fine-tuning technique. Without destroying or freezing the original hundreds of billions of parameters, it attaches a tiny “patch parameter matrix” alongside to record new task features, saving VRAM and effectively mitigating catastrophic forgetting.

We've been repeatedly emphasizing: LLMs are “text continuation” machines. All their superpowers are built on giving them a piece of text and having them spit out the next word based on probability.

But when you open the latest [Doubao] and casually take a photo of a dense code screen or a complex circuit board logic diagram, throw it at the model and ask: “Which line of code in here is wrong?” – it can instantly pinpoint among a sea of pixels and point out the error in text.

Don't LLMs only recognize text? How can they recognize images, even audio and video?

Behind this, the LLM hasn't changed its brain. Rather, engineers have installed a “translator” in front of the LLM.

Since LLMs only eat Tokens, we force images to also masquerade as Tokens.

When you upload a screenshot of code in the chat and ask “how to fix it,” the entire backend process is a “sleight of hand”:

1. Patching: The vision processing program, before the LLM even sees the image, uses an invisible knife to chop the image into a bunch of 16×16 pixel “small tiles.” If it's a large image, it might be chopped into hundreds of small tiles.
2. Vision Encoder: These small tiles are sent to a model specifically for understanding images (like CLIP's vision part). This translator doesn't write text – it only does one thing: translates the visual features of each image tile into a long string of digital coordinates (vectors).
3. Projection: Here's the crucial step. Engineers use a mathematical method to forcefully map the “image coordinates” output by the visual translator into the LLM's original “vocabulary space.” For example, it adjusts the numbers from the image tile containing the giant panda's ear to be very close to Tokens like “animal” and “fluffy” in the LLM's perception. Technically, these translated image tile coordinates are called Visual Tokens.
4. Big Platter Continuation: The history ledger ultimately presented to the LLM is concatenated like this: [System Prompt] + [Image Tile Token 1] + [Image Tile Token 2] +...+ [Image Tile Token 100] + [User Question: What's in this image?]

To the LLM, it doesn't see any colorful picture. In its “eyes,” those 100 image tiles are 100 “special words it has never seen before.” But because during massive pre-training, it has countless times seen the pattern that “[special characters with dark circles] often appear next to the word [giant panda].”

So it fires up its only trick – text continuation – and naturally spits out: “The image shows a giant panda eating bamboo.”

You immediately think: Since LLMs can “see” images by adding a “visual translator,” conversely, can we make them directly control a real-world robot or robotic arm through “text continuation”?

The answer is: absolutely yes, and this is exactly the field of Embodied AI.

Traditional robot control requires complex mathematical formulas to calculate joint angles, torque, and Cartesian coordinates for robotic arms. But what if we treat the robotic arm's actions as a kind of “language”?

In cutting-edge embodied AI engineering practice, engineers have done something very radical: they've encoded various robot actions – like “gripper open,” “gripper close,” “move left 5 cm” – into special codes in the vocabulary (usually called Action Tokens).

When you input to an LLM connected to a robotic arm: [Help me pick up the red block on the table.]

1. The LLM uses the camera (visual Tokens) to see the tabletop layout.
2. It starts frantically continuing in its underlying brain, but this time it doesn't continue in Chinese like “I'll go grab it.” Instead, it continues with a string of special symbols only the robotic arm can understand: <ROBOT_MOVE_X_0.05> <ROBOT_MOVE_Y_-0.02> <ROBOT_GRIPPER_CLOSE>.
3. The robot's underlying controller (like ROS 2) intercepts this continued secret code, converts it into electrical current and pulses, directly driving the Franka robotic arm's motors to accurately pick up the red block.

This transforms the originally tedious mechanical control problem into a pure, cross-modal “advanced text continuation” game.

But there's no free lunch. This approach of forcefully turning everything into Tokens and stuffing them into the LLM brings extremely heavy engineering costs:

• The “black hole” of context consumption: Text is a highly efficient compression symbol – the Chinese character for “cat” takes only 1 Token. But in visual multimodal scenarios, to let the model see the details of a high-definition image, even after tile compression, a single image often takes 500 to 2000 Tokens! This means casually uploading a few project screenshots instantly fills the LLM's physical vision ceiling (Context Window), severely squeezing the space for thinking and multi-turn dialogue.
• The “smoothness” limitation of alignment: The visual translator and the language LLM are ultimately independent individuals trained in different periods. If the mathematical mapping between them isn't precise enough, “blind spots” appear where the model “looks but doesn't see.” For example, the model clearly sees a minus sign - in the image, but because that small tile is too inconspicuous, it gets ignored by the LLM during inference after translation, leading to calculation results that are off by a mile.

To solve these visual and physical control failures, the industry is currently moving comprehensively from early “assembled multimodal (independent training then forced concatenation)” toward “native multimodal (from day one of pre-training, let the model simultaneously eat text, images, and action data).”

Key technical details:

• Vision Transformer (ViT): The standard front-end network for vision multimodal, responsible for chopping coherent image pixels and converting them into high-dimensional vector matrices.
• Linear Projector / Perceiver Resampler: The “translator” that bridges image vectors and text vectors, with parameters specifically designed to make the numbers on both sides mutually understandable.
• VLA Model (Vision-Language-Action Model): The ultimate brain for embodied AI, not only capable of looking at images and speaking but also directly continuing visual input into robot motion control commands (Action Sequence).

In previous chapters, we've learned how to make LLMs work like workhorses performing various simple tasks.

But if you're asked to write an enterprise-level system, you'll find that just writing “glue code” to stick these parts together can exhaust you. That's when the LLM engineering world evolved specialized “assembly pipeline frameworks.”

The earliest behemoth to appear was LangChain. Its core idea was to encapsulate common patterns in LLM application development into standard “Lego bricks.” It wrapped APIs from different vendors into unified shells and turned RAG steps into an automated conveyor belt. In the early days, it did allow beginners to assemble a chatbot in just a few lines of code.

But when people brought LangChain into serious production environments, the nightmare began:

• Horrifying over-encapsulation: A simple HTTP text request has over a dozen layers of abstraction inside LangChain. Just printing a raw return code requires digging through source code for days.
• Loose string hell: Data flow between bricks relies entirely on implicit dictionaries and plain strings. If the LLM, due to probability issues, misses a comma in a JSON array, the entire system crashes mysteriously at runtime, and you can't even find which line of code caused the error. If you've written Java or JS code, imagine what it feels like when all parameters are Map structures.

Then PydanticAI emerged. It borrowed design philosophies from modern backend development (like FastAPI) and proposed two weapons against the uncontrollability of LLMs:

1. Type Safety: Programmers strictly define output structures in code (e.g., user ID must be an integer, risk level must be a fixed enum). If the LLM outputs incorrectly, the framework intercepts the validation error behind the scenes without crashing.
2. Dependency Injection: Completely eliminates global variables. Each user request gets an independent, isolated “external resource toolkit” (e.g., dedicated database connection), thoroughly solving data cross-contamination bugs under high concurrency.

An LLM is a stateless text continuation machine. We discussed earlier that multi-turn dialogue relies on the backend frantically stacking “history ledgers.” But if a user chats with AI for three days straight, or logs in again next month, this ledger will not only burst the LLM's context but also drive compute costs to bankruptcy.

To give Agents long-term memory similar to humans, engineering has installed a layered “permanent backpack”:

• Short-term memory (running ledger): Records the most recent rounds of current conversation. This part is still stored in memory or Redis, using a sliding window mechanism – only the freshest content is kept, and older content is periodically “distilled into a summary sentence” by the LLM.
• Long-term memory (diary): When the day's conversation ends, the backend runs an asynchronous “reflection task” that extracts important facts from the running ledger (e.g., the user said they like black coffee, have a two-year-old child at home), turns them into small fragments, and stores them in a vector database. Three months later, when the user mentions “planning to buy a toy,” the system retrieves the “child is two” fragment from the diary and stuffs it into context.

More advanced memory is the entity relationship graph. As AI communicates with you, it secretly draws a network about you in its backpack: [User] -- likes --> [Black Coffee], [User] -- profession --> [Software Development].

This memory is no longer fragmented text snippets but structured state, making the LLM feel like an old friend every time it interacts with you.

We've given LLMs “hands and feet” (Tool Call), letting them check weather and modify files. But imagine, in a large enterprise, there are thousands of internal microservice APIs (check finances, approve leave requests, search employees, modify permissions…).

If we cram all the “instruction manuals” for these thousands of tools into the LLM before every conversation, two things happen: First, the context explodes immediately, getting more expensive the more you chat. Second, the LLM, facing thousands of choices, becomes deeply confused and starts randomly calling tools.

Thus, the Dynamic Skills system emerged. It turns tool usage into a “two-stage retrieval”:

1. Skills bundle encapsulation: Engineers group similar tools into highly decoupled, plug-and-play “Skills Bundles.”
2. On-demand credential retrieval: When the user inputs “Help me reimburse last month's travel expenses,” the system's “page turner” jumps ahead, uses vector retrieval in the skills library, and only grabs the 3 tool instruction manuals related to “finance, reimbursement.”
3. Dynamic feeding: The LLM can only see these 3 tools in this interaction. This reduces the “choose from thousands” problem into a “choose from three” multiple-choice question, not only saving massive Token costs but also skyrocketing Agent call accuracy.

When we've used PydanticAI to set the standards, Memory to give it memory, and Skills to give it unlimited capabilities, this Agent (single workhorse) becomes theoretically omnipotent.

Now we have an extremely ambitious idea: Can I create an omniscient and omnipotent “Super Single Agent”? I'll stuff all the company's rules and regulations, customer service scripts, Python programming standards, and financial reimbursement processes into its prompt, letting it handle all the company's dirty work single-handedly.

In engineering practice, if you actually do this, the LLM will slap you awake with reality. As you cram more and more prompts and hundreds of tools into this single Agent, it quickly falls into “deep confusion” – not only does response speed slow down, but it also frequently suffers from identity “schizophrenia.”

To solve this “mediocrity caused by omnicompetence,” there are two complete decouplings in the LLM world. One happens inside the LLM (the physical structure evolution of the model), and the other happens in our business code (the sociological separation at the application layer). These are often confused, but for beginners, distinguishing them is crucial.

First, we must peel back the physical shell of LLMs and see how their underlying hardware and compute power are squeezed out. In today's market, the underlying structure of LLMs mainly falls into two camps.

Early classic models (like early Llama series, or various small standalone models with tens of billions of parameters) are almost all “dense models.”

• Plain-language principle: Suppose this model has 720 billion parameters (72B). When a user inputs an extremely simple “hello” or asks a dumb question like “what's 1+1?”, all 720 billion parameters inside the model are instantly activated, and every single parameter must frantically go through matrix multiplication in the GPU.
• Hardware despair: This is like a company keeping 720 versatile geniuses. Whether it's cleaning toilets, photocopying documents, or deriving quantum mechanics, every time a task comes up, all 720 people must show up in unison and think together. In reality, this means even if you buy two top-of-the-line NVIDIA RTX 5090 D GPUs (64GB VRAM total), facing this fully-computed dense LLM, its token throughput will be as slow as an old ox pulling a cart because all GPU compute units (Tensor Cores) are fully occupied. And the compute cost will be so expensive you'd want to call the police.

To break through the physical limits of dense models, today's hottest top-tier LLMs (like the famous DeepSeek-V4 or GPT-5 series) have almost all switched to MoE architecture.

• Plain-language principle: The model's total parameter count might be terrifying, e.g., claimed to have 671 billion parameters. But it plays a trick internally – it divides these hundreds of billions of parameters into dozens or even hundreds of small “micro experts” (some good at arithmetic, some at grammar, some at code). In front of these experts stands an extremely sharp gating network (Router).
• Real compute ledger: When you input a code question, the Router glances at it, instantly recognizes the intent, and internally only activates the “code expert” and “logic expert.” This computation actually only uses 37 billion out of 671 billion parameters (that's called the activated parameter count). The remaining hundreds of billions of parameters all “close their eyes and sleep” in the GPU.
• Current state and limits: This architecture brings the compute miracle of the 2026 LLM era – its intelligence has the depth of hundreds of billions of parameters, but its per-token speed and cost are as cheap and fast as a 30B small model. But you smartly notice the hardware pain point: although MoE models only use a small portion of parameters per computation, to be ready at any time, all hundreds of billions of parameters must completely reside in VRAM! This imposes extremely insane “VRAM capacity” requirements on NVIDIA GPUs. In a server room, you might need a full 8 H100 PCIe GPUs in a high-performance, high-throughput cluster to barely fit a complete MoE model without blowing VRAM.

After hearing about the hardware situation, you might think: “Since MoE models like DeepSeek are already this smart and cheap, if I write all my company's business logic into the prompt and let it handle everything alone, it should work, right?”

Sorry, it still fails. Because the underlying MoE can only solve “whether text continuation is fast and compute-efficient,” it cannot solve the logical chaos that occurs when humans write complex business systems.

Suppose you want to build a “fully automated R&D expense reimbursement audit assistant.” If you make it a single Agent, you'd have to write tens of thousands of characters in its System Prompt:

"You need to be both a warm and friendly customer service representative guiding users to upload invoices, AND a strict rigorous code auditor checking users' code for security vulnerabilities, AND a stingy accountant checking the company's MySQL database to see if department budgets are exceeded..."

When the LLM sees this massive chunk of prompt, its attention mechanism gets completely confused. When the user inputs: “Help me check if the code for this login interface is done, and also reimburse the 5000 yuan overspent last week due to fixing this bug.” This single Agent immediately falls into identity schizophrenia: it might evaluate your code in the extremely stingy tone of an accountant saying it's not cost-effective, or while checking the database, it casually treats a minus sign in the code as the reimbursement amount. This is “dementia caused by omnicompetence.”

When human society faces complex tasks, it never relies on a lone hero but on departmental collaboration. When LLMs are deployed in business, they must also move toward Multi-Agent systems. The core is: completely dismantle a complex all-powerful Agent into a “virtual office” composed of multiple specialized AIs.

Let's use the failure case above to see how a modern multi-agent system (e.g., built on LangGraph or PydanticAI Graph) flows smoothly behind the scenes:

1. Application Router: It's the administrative front desk of the office. It doesn't have any hardcore professional skills. Its system prompt is extremely simple: “You are only responsible for understanding the user's needs and then breaking down and distributing tasks.” When it sees the user's complex request, it snaps twice and splits it into two work orders.
2. Security Auditor (Agent A): The front desk throws the first work order to Agent A, the security auditor. This Agent's prompt is extremely pure: “You are a strict code security expert.” It only carries one Skills tool – read local code file. It focuses 100% attention on the code and quickly produces a rigorous security audit report.
3. Accountant (Agent B): The front desk throws the second work order to Agent B, the accountant. Its prompt: “You are a shrewd accountant.” It doesn't look at any code, only carrying one advanced dependency injection – read-only connection to company financial database. It quickly queries last week's R&D department financial flow and produces the exact overspend amount.
4. State Machine Merge: After the two experts finish their work in their respective unidirectional tracks (state machines), they throw their standard, type-safe Pydantic reports back to the front desk. The front desk manager neatly combines “code security” and “overspend amount” and presents them tidily to the user.

In this application-layer decoupling process, the LLM doesn't need to play both judge and god of wealth in the same context. Each sub-Agent's Prompt is extremely clean, carrying only two or three tool skills. Tool call accuracy directly skyrockets from 40% for the single Agent to over 95%.

中文版