AI 如何運作：引導而非祈禱，一張駕馭 AI 的完整地圖

"please review this code"
   ↓ tokenization
["please", " review", " this", " code"]
   ↓ token IDs
[7847, 12043, 5521, 9982]
   ↓ embeddings
[[0.23, -0.81, ...], [0.11, 0.42, ...], ...]
   ↓
what the model actually computes on

// In practice, use the model provider's tokenizer or API usage response.
// Different models use different tokenizers, so character-based formulas drift fast.
const usage = await client.messages.countTokens({
  model: "claude-opus-4-8",
  system: systemPrompt,
  messages: [
    {
      role: "user",
      content: `${documentContent}\n\nQuestion: ${userQuestion}`,
    },
  ],
});

console.log(`This input is about ${usage.input_tokens} tokens`);

Context Window (all tokens the model can see in this API call)

System Prompt · fixed settings
Conversation history · accumulated
Uploaded files · specific to this call
Your question this turn · latest input
─────────────────────────
Remaining space · how many tokens left?

// The model has no memory. Send the full conversation history on every call.
interface Message {
  role: "system" | "user" | "assistant";
  content: string;
}

async function chat(history: Message[], newMessage: string) {
  const messages: Message[] = [
    ...history, // ← full past conversation, resent every time
    { role: "user", content: newMessage },
  ];

  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    messages,
  });

  return response;
}

// Omit history and the model has no idea what you talked about before

// Simplified truncation: when over budget, drop from the oldest messages
function truncateToFit(
  messages: Message[],
  maxTokens: number,
  countTokens: (text: string) => number,
): Message[] {
  let total = messages.reduce((sum, m) => sum + countTokens(m.content), 0);

  // Keep system prompt and latest message; drop second-oldest onward
  while (total > maxTokens && messages.length > 2) {
    const removed = messages.splice(1, 1)[0]; // remove oldest non-system message
    total -= countTokens(removed.content);
  }

  return messages;
}

// In real apps, context is assembled — not dumped in randomly
interface ContextLayers {
  systemPrompt: string;      // fixed role definition (cache-friendly)
  retrievedDocs: string[];   // RAG chunks for this turn
  conversationHistory: Message[]; // history (length-managed)
  currentQuery: string;      // user's question this turn
}

function assembleContext(layers: ContextLayers, budget: number): Message[] {
  const messages: Message[] = [];

  // In practice systemPrompt usually goes in the API `system` field (see §8);
  // here we focus on assembling `messages`.
  // 1. Fixed system prompt — highest priority, always kept
  // 2. Retrieved docs — only the most relevant chunks, within budget
  const docs = layers.retrievedDocs.slice(0, 3).join("\n\n");

  // 3. Conversation history — keep recent turns until budget is tight
  const history = trimHistoryToBudget(layers.conversationHistory, budget);

  messages.push(...history);
  messages.push({
    role: "user",
    content: `Reference material:\n${docs}\n\nQuestion: ${layers.currentQuery}`,
  });

  return messages;
}

it → animal   ████████████   high relevance
it → tired    ███████        medium (semantic)
it → street   ██             low
it → cross    █              very low

Q (Query)  → "What am I looking for?"     current token's query
K (Key)    → "What can I offer?"          each token's outward "label"
V (Value)  → "What do I actually carry?"  each token's real contribution

1. Take the current token's Q
2. Dot product Q with every token's K → relevance scores
3. Softmax the scores → weights that sum to 1
4. Weighted average of every token's V
5. Result = this token's vector after "seeing" the full context

// Simplified single-head attention to illustrate the flow
type Vector = number[];

function dotProduct(a: Vector, b: Vector): number {
  return a.reduce((sum, val, i) => sum + val * b[i], 0);
}

function softmax(scores: number[]): number[] {
  const max = Math.max(...scores);
  const exps = scores.map((s) => Math.exp(s - max)); // subtract max to avoid overflow
  const sum = exps.reduce((a, b) => a + b, 0);
  return exps.map((e) => e / sum);
}

function attention(
  query: Vector, // current token's Q
  keys: Vector[], // all tokens' K
  values: Vector[], // all tokens' V
): Vector {
  // Step 2: dot Q with each K → relevance scores
  const scores = keys.map((k) => dotProduct(query, k));

  // Step 3: Softmax → weights that sum to 1
  const weights = softmax(scores);

  // Step 4: weighted average of all V
  const dim = values[0].length;
  const output: Vector = new Array(dim).fill(0);

  values.forEach((v, i) => {
    for (let d = 0; d < dim; d++) {
      output[d] += weights[i] * v[d];
    }
  });

  return output; // Step 5: vector after "seeing" full context
}

Head 1 → syntax (subject ↔ verb)
Head 2 → semantics (synonyms, near-meaning)
Head 3 → reference (it / they / this → what?)
Head 4 → position (distance in the sequence)
   ⋮

Attention weights (simplified)

high │ ████                              ████
     │ ████                              ████
 low │      ████  ████  ████  ████  ████
     └──────────────────────────────────────
       start         middle              end
       (System       (easily ignored)    (latest question)
        Prompt)

Without KV Cache:
Every call
  → System Prompt (1000 tokens) recompute K, V
  → Conversation history recompute K, V
  → New message compute K, V
  → Generate answer

With KV Cache:
First call
  → System Prompt (1000 tokens) compute K, V → store in cache

Second call (System Prompt unchanged)
  → System Prompt → read cache ✓ (no recompute!)
  → New message compute K, V
  → Generate answer

// ✅ Cache hit: stable system prompt; different user messages still hit cache
const systemPrompt = "You are a professional support agent. Be friendly and concise.";

await chat(systemPrompt, "Where is my order?"); // system prompt KV computed once
await chat(systemPrompt, "How do I request a refund?"); // system prompt read from cache ✓

// ❌ Cache miss: one word changed in system prompt → full cache invalidation
const v1 = "You are a professional support agent. Be friendly and concise.";
const v2 = "You are a professional support helper. Be friendly and concise."; // "agent" → "helper"

await chat(v1, "Where is my order?"); // compute once
await chat(v2, "Where is my order?"); // prefix diverges → cache miss → full recompute ✗

Context order (top to bottom; prefix matched from the top)

System Prompt · stable · front, easiest to cache
Few-shot examples · stable · same
Uploaded documents · call-specific, often large
Conversation history · grows over time
User message this turn · changes every time · last

const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: longStableSystemPrompt, // long, stable system prompt
      cache_control: { type: "ephemeral" }, // ← mark for caching
    },
  ],
  messages: [{ role: "user", content: userMessage }],
});

Your question → AI "looks up" a giant knowledge base → finds answer → returns it

Your question → predict "the most likely next token" → append it
        → predict again → append → repeat until done

Step 1: Context = "The capital of Taiwan is"
        → predict next token distribution
        → candidates: " Taipei" 88%, " Ta" 8%, other 4%
        → pick " Taipei"

Step 2: Context = "The capital of Taiwan is Taipei"  ← note: output becomes input
        → predict next token
        → candidates: "." 70%, "," 15%, other 15%
        → pick "."

Step 3: Context = "The capital of Taiwan is Taipei."
        → predict next token
        → candidates: <EOS> 92%, other 8%
        → pick <EOS> → stop generation

// Autoregressive generation is fundamentally a loop
async function generate(prompt: string, maxTokens: number): Promise<string> {
  let context = prompt;

  for (let i = 0; i < maxTokens; i++) {
    // Each step: predict next-token distribution from current context
    const distribution = await predictNextToken(context);

    // Pick one token from the distribution (§6 covers "how to pick")
    const nextToken = sample(distribution);

    if (nextToken === END_OF_SEQUENCE) break;

    // Key: append output back into context for the next round
    context += nextToken;
  }

  return context;
}

// The "typing" you see in a stream is autoregressive generation itself
const stream = await client.messages.stream({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  messages: [{ role: "user", content: "What is the capital of Taiwan?" }],
});

for await (const event of stream) {
  if (event.type === "content_block_delta") {
    process.stdout.write(event.delta.text); // each token is its own forward pass
  }
}

// ❌ Ask for the answer directly: model must get it right in one shot
await chat("", "A store sold 23 units on day one. Day two was 3× day one. How many total?");

// ✅ Ask for step-by-step reasoning: intermediate steps scaffold later tokens
await chat(
  "",
  "A store sold 23 units on day one. Day two was 3× day one. How many total? Walk through each step, then give the answer.",
);

Next token candidates:
" Paris"  → 80%
" Lyon"   → 15%
" Mars"   →  3%
" Rome"   →  1%
other     →  1%

Temperature = 0 (approaching 0): sharpest
" Paris" → 99.9%   " Lyon" → 0.1%   other → ~0%
→ almost always picks the top token; stable, predictable

Temperature = 1.0 (default): original distribution
" Paris" → 80%   " Lyon" → 15%   " Mars" → 3%
→ sample by original probabilities; some variation

Temperature = 2.0 (high): flattest
" Paris" → 40%   " Lyon" → 30%   " Mars" → 20%   " Rome" → 10%
→ probabilities level out; low-probability tokens get a real chance; more random

function applyTemperature(logits: number[], temperature: number): number[] {
  if (temperature === 0) {
    // Temperature 0: greedy — fully deterministic
    const maxIndex = logits.indexOf(Math.max(...logits));
    return logits.map((_, i) => (i === maxIndex ? 1 : 0));
  }

  // Divide by temperature: lower → sharper; higher → flatter
  const scaled = logits.map((l) => l / temperature);
  return softmax(scaled);
}

Code generation: Temperature = 0
  → you want the correct answer, not a creative one
  → the highest-probability token is usually syntactically right

Creative writing: Temperature = 0.9 ~ 1.2
  → you want surprising phrasing
  → let low-probability tokens through for variety

Top-P = 0.9, candidate list:
" Paris" → 80%   ← cumulative 80%
" Lyon"  → 15%   ← cumulative 95% > 90%, include it and stop
" Mars"  →  3%   ← excluded
" Rome"  →  1%   ← excluded

→ sample only between " Paris" and " Lyon"

Top-K = 2:
→ sample only from " Paris" (80%) and " Lyon" (15%); everything else excluded

Task type              Temperature   Top-P    Notes
─────────────────────────────────────────────────────────
Code generation         0 ~ 0.2      0.9      stable, correct
Extraction / classify   0            -        fully deterministic
Q&A / summarization     0.3 ~ 0.7    0.9      accuracy + fluency
General chat            0.7 ~ 1.0    0.95     natural variation
Creative writing        1.0 ~ 1.2    0.95     encourage surprise
Brainstorming           1.2+         1.0      maximum diversity

Wrong mental model:
AI → query knowledge base → find fact → output

Right mental model:
AI → given context, predict "the most likely next token" → output

Intrinsic Hallucination
  → output directly contradicts context you provided
  → you gave a doc saying A; AI answers B

Extrinsic Hallucination
  → output cannot be verified from your context
  → AI "fills in" information you never provided
  → more common, harder to spot

// ❌ Rely on "memory": prone to hallucination
const bad = await chat(
  "You are an assistant",
  "What is our company's return policy window in days?",
);

// ✅ Provide context: put facts in the prompt
const good = await chat(
  "You are an assistant. Answer only from the user's provided material. If it's not there, say \"not mentioned in the material.\"",
  `Return policy document:\n${policyDocument}\n\nQuestion: How many days for returns?`,
);

const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  system: "You are a professional technical writing assistant...", // ← System Prompt
  messages: [
    { role: "user", content: "Draft API documentation for me" }, // ← User Prompt
    { role: "assistant", content: "Sure — please share the API spec..." },
    { role: "user", content: "Here is the spec: ..." }, // ← User Prompt
  ],
});

Define role     → "You are a professional legal assistant"
Set constraints → "Only answer contract-related questions"
Specify format  → "Use bullet points; stay under 200 words"
Set tone        → "Formal tone; no emoji"
Inject knowledge → "Here is our product documentation: ..."

Assign task      → "Draft an apology email to a customer"
Provide material → "Here is the contract text: ..."
Give feedback    → "This version is too formal — make it conversational"
Add constraints  → "Also keep it under 100 words"

System Prompt              User Prompt
────────────────────────────────────────────────
"Reply in English"    +    "what is AI?"
  → AI explains AI in English

"Only legal questions" +   "write me a poem"
  → AI should refuse (not guaranteed — depends how firm the system prompt is)

"Keep tone formal"    +    "explain this casually"
  → user prompt overrides system; AI usually goes casual
    (when they conflict, the nearer, clearer instruction tends to win)

const systemPrompt = `
You are the technical writing assistant for Lumen Tech.

# Hard constraints (up front — high Attention weight at the start)
- Answer only from material the user provides. Do not add outside information.
- If unsure, say "I'm not sure" — do not guess.

# Output format (structure narrows the token space)
1. Conclusion (one sentence)
2. Explanation (bullets, max three)
3. Code example if needed

# Long threads (respect Context Window limits)
If the conversation grows, prioritize the latest task; older detail can drop.
`.trim();

// Stable system prompt → good KV Cache candidate → lower cost

Job description (System Prompt) covers:
  → scope of this role
  → how to handle uncertainty
  → output format and tone standards
  → what to decide alone vs. when to say "I don't know"

Work assignment (User Prompt) covers:
  → the specific task this turn
  → materials or constraints for this turn only
  → feedback on the last output

const systemPrompt = `
You are the support assistant for Lumen Tech.

# Core constraints
- Answer only from "reference material." If it's not there, say "I'll connect you with a specialist."
- Never guess or invent specs, prices, or return policies.
- If unsure, say so — don't sound plausible without evidence.

# Tone
- Professional, friendly, concise.
- No emoji.

# Answer format
- Answer the question first, then add necessary detail.
- Use numbered steps when describing procedures.
`.trim();

async function answerCustomerQuestion(question: string) {
  // 1. Retrieve relevant chunks from the knowledge base
  const relevantDocs = await vectorSearch(question, { limit: 3 });

  // 2. Assemble retrieved material into context
  const userContent = `
Reference material:
${relevantDocs.map((d) => d.content).join("\n---\n")}

Customer question: ${question}
  `.trim();

  // 3. Call the model
  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    temperature: 0.3, // low: support needs stable, predictable answers
    system: [
      {
        type: "text",
        text: systemPrompt,
        cache_control: { type: "ephemeral" },
      },
    ],
    messages: [{ role: "user", content: userContent }],
  });

  return response;
}

async function handleConversation(
  history: Message[],
  newQuestion: string,
): Promise<Message[]> {
  // Cap history length — avoid blowing context budget or losing Attention focus
  const trimmedHistory = history.slice(-6); // keep last 3 Q&A pairs

  const relevantDocs = await vectorSearch(newQuestion, { limit: 3 });

  const messages: Message[] = [
    ...trimmedHistory,
    {
      role: "user",
      content: `Reference material:\n${formatDocs(relevantDocs)}\n\nQuestion: ${newQuestion}`,
    },
  ];

  const response = await client.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 1024,
    temperature: 0.3,
    system: [{ type: "text", text: systemPrompt, cache_control: { type: "ephemeral" } }],
    messages,
  });

  return [...messages, { role: "assistant", content: extractText(response) }];
}

Token
  → AI's basic unit. All computation starts here; capacity and billing are token-based.

Context Window
  → AI's working memory. Finite like RAM; cleared every call.

Attention
  → How AI decides what matters. Each token gets a different weight via Q, K, V.
     This is where Lost in the Middle comes from.

KV Cache
  → Optimization built from Attention's K and V. Stable prefixes cache;
     that's why system prompts should be stable and up front.

Autoregressive Generation
  → How AI produces answers. One token at a time, no global plan;
     errors compound — source of many odd behaviors.

Temperature & Sampling
  → Knobs for how to pick from the distribution. Controls determinism,
     not intelligence. Low = stable; high = varied.

Hallucination
  → Structural outcome of generation, not a bug. AI optimizes for plausible;
     factual correctness needs extra tooling. Use it wisely; don't trust blindly.

System Prompt vs User Prompt
  → Turns all of the above into levers you actually pull.
     System defines the role (stable, front); user assigns the task (dynamic, can override).