An AI system design interview asks you to turn an ambiguous product request into an implementable, measurable system. Explain what the system must do, how data moves through it, what can fail, and why the proposed tradeoffs fit the requirements. Model selection is one decision within that design.

This practice section is part of Learnastra's The Design Room. Use it to rehearse explanations and decisions, then test whether you can adapt them when the interviewer changes a constraint. The questions are authored practice material; they are not claimed to be confidential or verified questions from particular employers.

Start with one observable baseline

Choose one prompt from the question bank. Work without looking at its answer for 30–45 minutes, depending on the format you are preparing for. Save your diagram and notes. Mark where you guessed, where you could not explain a component, and where you ran out of time.

There is no required number of weeks before beginning practice. When a gap blocks your explanation, study that concept and return to the same prompt.

Use a repeatable practice loop

flowchart LR
    P[Choose a prompt] --> A[Answer without notes]
    A --> R[Review diagram and reasoning]
    R --> G[Study the specific gaps]
    G --> T[Retry with a changed constraint]
    T --> P

A useful retry changes something material: the tenant model, deadline, data freshness, access policy or consequence of a wrong action. Repeating a memorized diagram without adapting its decisions does not establish understanding.

1. Read: identify the concept or failure you need to understand.
2. Recall: define it and work a small example without the page open.
3. Draw: show actors, services, stores and labeled data flows.
4. Critique: find an unmet requirement and explain how you would detect it.
5. Repair: propose a change and account for its new cost and failure modes.
6. Retry: answer a related prompt after a break and compare your reasoning.

Keep this interview sheet beside the whiteboard

These are conventional system design activities, not a mandatory acronym or a universal hiring rubric. Adapt their order to the interviewer and the problem. If the interviewer asks for an implementation detail, answer it directly and reconnect it to the design.

Match practice to the responsibilities

Job titles are inconsistent across employers. Use the actual role description and recruiter guidance to choose depth.

An experienced candidate may be new to one subsystem. Say where your experience ends, state assumptions, and reason from the contract instead of inventing a production story.

Example practice schedules

Treat these as adjustable plans, not promises of readiness.

For a 60-minute practice session, one possible split is 35 minutes answering, 15 minutes reviewing and 10 minutes rewriting the weakest explanation. A real interview may use a different length or include coding; confirm its format.

Review your work with evidence

Use this local practice scale: 0 = missing, 1 = asserted, 2 = explained, 3 = tested against a counterexample. It is a learning aid, not an employer's scoring system or an offer predictor.

Record the specific weakness and your repair, not just a score. For example: “I treated a timeout as a failed payment. On retry I will distinguish unknown outcome, reconcile the provider operation and preserve the original idempotency key.”

Practice questions to ask yourself

Final summary and notes

• Define unfamiliar terms before relying on them.
• Number requirements; put comparisons in tables; label diagram arrows.
• Begin with a complete baseline and improve it for a demonstrated reason.
• Separate facts, assumptions and measurements.
• Include quality, recovery and full operating cost in your decision.
• Finish with the main compromise and the next test you would run.

For complete worked designs, start with enterprise RAG, customer support automation or evaluation-gated delivery. Use the transition guide to connect existing experience to the preparation plan.

Reviewed 24 September 2026. Numerical examples are illustrative unless explicitly sourced. This bank contains original practice prompts and answer guides, not claims about questions asked by particular employers.

Remember: Answer first. Check the key. Defend a changed requirement.

Learn the explanation, then practice the answer

This page has three levels: 40 quick checks for recall, 128 developed answers for understanding, and five worked design scenarios for synthesis. The ten leadership follow-ups apply when the role includes those responsibilities; answer them from your own experience.

For each topic, first explain the mechanism in ordinary language. Then walk through an example, name the failure you must prevent, and defend the tradeoff. A short answer is the last step of learning, not a substitute for understanding. If you cannot explain why a control works, follow the linked chapter and return to the question afterward.

1. Understand: read the linked concept and redraw the worked mechanism.
2. Recall: keep the answer closed and explain it aloud, then compare the missing points.
3. Transfer: change the deadline, permission boundary or failed dependency and adapt the answer.

Return to missed questions after a gap. The short recall cues below organize an answer; they do not replace its explanation. Repeated subjects test different skills: Q15 defines MCP, Q50 addresses production operations, Q117 plans migration and Q127 enforces tenant isolation.

Navigate the full answers

• Retrieval and agents: Q1–Q17
• Models, inference, and evaluation: Q18–Q32
• Security, tooling, and choices: Q33–Q49
• Production diagnosis: Q50–Q65
• Model controls and judgment: Q66–Q80
• Emerging techniques and risks: Q81–Q96
• Infrastructure and leadership: Q97–Q112
• Agent reliability and protocols: Q113–Q128
• Five worked system-design scenarios
• Ten manager answer guides

How to use this bank

Use the quick checks as a warm-up, then work through the developed answers and design scenarios. Use the model and pricing chapters when an answer depends on current product capabilities or costs.

Give a 60–90 second answer before revealing the check. Then ask “what would change my decision?” Practice one question from each family before repeating comfortable topics. The checks are answer criteria, not complete scripts.

Retrieval and data

Quick check 1: Explain a production RAG system

Quick check 2: RAG, long context, or fine-tuning?

Quick check 3: How do you choose chunks?

Quick check 4: Why hybrid search?

Quick check 5: What does a reranker fix?

Quick check 6: A faithful answer is wrong. Why?

Quick check 7: How do permissions affect retrieval?

Quick check 8: What can an embedding tell you?

Quick check 9: How do you migrate embeddings?

Quick check 10: What causes data leakage in evaluation?

Agents, state, and safety

Quick check 11: Agent or workflow?

Quick check 12: What makes a tool contract safe?

Quick check 13: When would you use multiple agents?

Quick check 14: What is durable execution?

Quick check 15: A payment times out. Retry?

Quick check 16: How does approval survive a restart?

Quick check 17: How do you stop runaway agents?

Quick check 18: Does restoring a checkpoint undo an action?

Quick check 19: How do you defend against prompt injection?

Quick check 20: Is a sandbox enough?

Evaluation and release

Quick check 21: How do you evaluate open-ended answers?

Quick check 22: What can go wrong with LLM judges?

Quick check 23: Why is 95% accuracy insufficient?

Quick check 24: pass@k versus pass^k?

Quick check 25: Should an agent match a reference trajectory?

Quick check 26: Offline quality improves. Ship?

Quick check 27: How do you shadow an action agent?

Quick check 28: What belongs in a release manifest?

Quick check 29: What do you observe in production?

Quick check 30: How do you handle model drift or alias changes?

Economics and system tradeoffs

Quick check 31: How do you choose a model?

Quick check 32: API or self-hosting?

Quick check 33: What changes capacity planning for LLMs?

Quick check 34: When does caching help?

Quick check 35: Why can a cheap model cost more?

Quick check 36: How do you respond to a doubled bill?

Quick check 37: What do retries, breakers, and bulkheads do?

Quick check 38: How would you design a recommender?

Quick check 39: What changes for moderation or fraud?

Quick check 40: What does governance add beyond filters?

Ten manager follow-ups

For each, use your own evidence and the behavioral guide:

1. What project did you stop, and how did you communicate the decision?
2. How did you hire for a capability your team lacked?
3. How did you coach someone into broader ownership?
4. How did you handle sustained underperformance fairly?
5. When did you disagree with research or product, and what resolved it?
6. How did you delegate a consequential technical decision?
7. How did you turn evaluation into recurring team practice?
8. What did you do when a launch harmed users or missed expectations?
9. How did you choose between platform investment and near-term delivery?
10. What evidence would change your current roadmap or staffing plan?

Readiness test: you can give a clear answer, draw one example, identify a failure, and respond to a changed constraint. Recognizing the words on this page is not the same as being ready to explain them.

Full practice: retrieval and agent fundamentals (Q1–Q17)

The answers explain a defensible approach. Practice expressing the reasoning in your own words, then test it against the follow-up constraint. Hypothetical incidents and numerical targets are interview scenarios, not claims about named companies.

Q1: Walk me through the architecture of a production RAG system

Recall: Ingest → authorize → retrieve → ground → verify.

flowchart TD
    S[Source version and ACL] --> I[Parse and index]
    U[Authenticated question] --> R[Permitted retrieval]
    I --> R
    R --> P[Pack evidence and provenance]
    P --> G[Generate and check support]
    G --> O[Answer or abstain]

Q2: When would you choose RAG over fine-tuning, and vice versa?

Recall: Knowledge source versus learned behavior.

Q3: How do you handle the “lost in the middle” problem?

Recall: Position tests → evidence selection → measured answer quality.

Q4: Explain chunking strategies and when to use each

Recall: Evidence units → structure → overlap → evaluation.

Q5: How would you evaluate a RAG system?

Recall: Retrieval → support → correctness → useful coverage.

Q6: Describe hybrid search and when you would use it

Recall: Exact terms + semantics → rank fusion → slice tests.

Q7: How do you handle multi-tenant RAG systems?

Recall: Trusted scope → every data path → revocation → quotas.

def retrieve_for_user(auth_context, query, search):
    scope = {"tenant_id": auth_context.tenant_id,
             "principal_id": auth_context.principal_id,
             "acl_version": auth_context.acl_version}
    return search(query=query, enforced_scope=scope)

Q8: What is reranking, and when would you skip it?

Recall: Candidate coverage first; richer scoring second.

Q9: How would you handle documents with tables, charts, and images?

Recall: Structure → visual evidence → provenance → review.

Q10: Explain vector database indexing algorithms

Recall: Exact baseline → ANN recall → filters → resource cost.

flowchart TD
    Q[Query vector and filters] --> C{Index approach}
    C -->|Exact| A[Compare every eligible vector]
    C -->|HNSW| H[Navigate graph with bounded search breadth]
    C -->|IVF| I[Choose coarse partitions and scan candidates]
    H --> E[Compare top-k against exact neighbors]
    I --> E
    A --> E
    E --> R[Measure relevance, recall, and latency separately]

Q11: What is the difference between an agent and a workflow?

Recall: Prescribed flow versus adaptive next steps.

Q12: Explain the ReAct pattern

Recall: Reason → act → observe, inside enforced limits.

Q13: How do you implement tool use or function calling?

Recall: Schema → meaning → authority → effect → receipt.

{
  "name": "lookup_order",
  "description": "Read the status of an order accessible to the current user.",
  "input_schema": {
    "type": "object",
    "properties": {"order_id": {"type": "string", "minLength": 1}},
    "required": ["order_id"],
    "additionalProperties": false
  }
}

Q14: How would you design a multi-agent system?

Recall: Useful decomposition → contracts → integration → evidence.

Q15: Explain the Model Context Protocol

Recall: Host → client → server; protocol is not permission.

Q16: How do you handle long-running agent tasks?

Recall: Persist progress; reconcile uncertain effects.

{
  "job_id": "refund-job-42",
  "step": "payment",
  "proposal_version": 3,
  "operation_id": "refund-42-1",
  "status": "unknown",
  "receipt": null
}

Q17: What is flow engineering?

Recall: Explicit stages → branches → checks → bounded repair.

flowchart TD
    I[Classify input] --> R[Retrieve required evidence]
    R --> V{Evidence sufficient}
    V -->|No| C[Clarify or hand off]
    V -->|Yes| D[Draft bounded answer]
    D --> K{Contract and support checks}
    K -->|Pass| O[Return]
    K -->|Fail| H[One repair or explicit failure]

Full practice: models, inference, and evaluation (Q18–Q32)

Q18: How do you choose between frontier models for a production workload?

Recall: Hard constraints → task evidence → full economics.

Q19: When would you use a small language model instead of a frontier model?

Recall: Bounded competence → hard cases → operating cost.

Q20: Explain reasoning models and controllable thinking. When are they worth the cost?

Recall: Extra inference work must improve the outcome.

Q21: How do you evaluate and compare embedding models?

Recall: Task slices → compatible encoding → exact/ANN comparison.

Q22: Explain the KV cache and why it matters

Recall: Reuse keys/values; size the actual architecture.

Q23: What is speculative decoding, and when would you use it?

Recall: Draft → verify → accept/correct → benchmark.

Q24: Compare batching strategies for LLM serving

Recall: Batch policy → queueing → KV capacity → fairness.

Q25: How do you optimize LLM inference costs?

Recall: Cost per accepted task, including failed work.

Q26: Explain quantization techniques for LLM deployment

Recall: Representation → kernels → quality → actual memory.

Q27: How do you evaluate LLM outputs when there is no single ground-truth answer?

Recall: Rubric → evidence → calibrated review → uncertainty.

Q28: Explain the Ragas evaluation framework

Recall: Metric definition → required inputs → calibrated meaning.

def claim_support_score(verdicts):
    allowed = {"supported", "contradicted", "insufficient_evidence"}
    if not isinstance(verdicts, (list, tuple)):
        raise ValueError("Expected a sequence of claim verdicts")
    if any(not isinstance(v, str) or v not in allowed for v in verdicts):
        raise ValueError("Invalid or unjudgeable verdict")
    return sum(v == "supported" for v in verdicts) / len(verdicts) if verdicts else None

assert claim_support_score(["supported", "supported", "contradicted",
                            "insufficient_evidence"]) == 0.5

Q29: How do you detect and handle hallucinations?

Recall: Check claims against the right evidence.

Q30: How do you implement observability for LLM applications?

Recall: Task trace → versions → outcomes → protected evidence.

Q31: Describe CI/CD for LLM applications

Recall: Exact release → gates → canary → compatible rollback.

candidate: support-release-b
baseline: support-release-a
contracts: required
critical_risk_failures_allowed: 0
quality_comparison: paired_cases_with_predeclared_bounds
operating_checks: [p95_latency, total_cost, dependency_failure]
on_pass: small_canary
on_failure: retain_baseline
rollback: restore_compatible_release_a

Q32: How do you handle rate limits and quotas?

Recall: Admission → queue → deadline → shared retry budget.

from math import isfinite

def token_bucket_admit(tokens, elapsed_s, rate_per_s, capacity, estimated_cost):
    values = (tokens, elapsed_s, rate_per_s, capacity, estimated_cost)
    if any(type(v) not in (int, float) or (type(v) is float and not isfinite(v)) or v < 0 for v in values):
        raise ValueError("Expected finite nonnegative numbers")
    if tokens > capacity:
        raise ValueError("Stored balance exceeds capacity")
    available = min(capacity, tokens + elapsed_s * rate_per_s)
    allowed = estimated_cost <= available
    return allowed, available - estimated_cost if allowed else available

assert token_bucket_admit(100, 2, 50, 200, 150) == (True, 50)

Full practice: security, tooling, and design choices (Q33–Q49)

Q33: Describe strategies for LLM application security

Recall: Trust boundaries and permitted effects.

Q34: Explain tradeoffs between vector database options

Recall: Representative filtered workload beats a generic ranking.

Q35: How do you handle model updates and provider deprecations?

Recall: Inventory → compatibility tests → migration → supported fallback.

release = {
    "model_snapshot": "immutable-revision-a",
    "prompt_digest": "example-prompt-digest",
    "tool_schema": "orders-v3",
    "retrieval_snapshot": "policy-19",
    "grader": "support-rubric-4",
}

Q36: What is DSPy, and when would you use it?

Recall: Program → metric → optimization → protected evaluation.

import dspy

class GroundedAnswer(dspy.Signature):
    """Answer only from supplied evidence; state when it is insufficient."""
    context: str = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

answer_module = dspy.Predict(GroundedAnswer)

Q37: How do you design a feedback loop for continuous improvement?

Recall: Observed failure → hypothesis → controlled fix → outcome.

Q38: Explain token counting and why it matters

Recall: Count the assembled request; reserve actual output capacity.

def available_evidence_tokens(context_limit, fixed_input, output, margin):
    values = (context_limit, fixed_input, output, margin)
    if any(type(v) is not int or v < 0 for v in values) or context_limit == 0:
        raise ValueError("Expected nonnegative integer counts and a positive limit")
    available = context_limit - fixed_input - output - margin
    if available < 0:
        raise ValueError("Fixed request cannot fit")
    return available

assert available_evidence_tokens(8192, 1500, 1500, 512) == 4680

Q39: How do you compare two RAG systems objectively?

Recall: Same conditions → paired outcomes → uncertainty → operations.

Q40: When would you use self-consistency versus best-of-N sampling?

Recall: Agreement versus selection; include selector cost.

Q41: How do you prevent reward hacking in best-of-N?

Recall: Score exploits → hard gates → independent checks.

Q42: Design an evaluation system for comparing two LLMs on open-ended tasks

Recall: Blind paired review → calibrated rubric → slices.

Q43: What is the difference between ensemble learning and model arbitration?

Recall: Combine predictions versus select a route or candidate.

Q44: When would you use multi-agent debate versus mixture-of-agents synthesis?

Recall: Critique versus synthesis; consensus is not proof.

Q45: When should you use LangChain or another framework versus building directly?

Recall: Hardest requirement → prototype → ownership → exit cost.

Q46: How do you manage context limits in long conversations?

Recall: Working context ≠ facts ≠ execution authority.

{
  "summary_version": 4,
  "active_goal": "Compare two permitted vendors",
  "constraints": ["No purchase without explicit approval", "Budget USD 500"],
  "confirmed_facts": [{"value": "Vendor A quote is USD 420", "source": "quote-17"}],
  "open_questions": ["Does quote include shipping?"],
  "superseded_facts": ["Earlier USD 600 budget"]
}

Q47: How do you defend against prompt injection?

Recall: Untrusted instructions cannot grant capabilities.

Q48: When would you choose fine-tuning over prompt engineering?

Recall: Stable behavioral gap → suitable data → held-out benefit.

Q49: How do you optimize latency for real-time LLM applications?

Recall: Critical path → TTFT → token intervals → deadline.

async def forward_text(stream):
    try:
        async for event in stream:
            if event.kind == "text":
                yield event.text
    finally:
        await stream.aclose()

Full practice: production diagnosis (Q50–Q65)

Q50: Why does MCP matter for production agents beyond a demo?

Recall: Compatibility plus operational and security contracts.

Q51: Your agent uses 47 calls for a task expected to take five. How do you debug it?

Recall: Classify calls → find cause → contain → retest completion.

Q52: When would you choose a reasoning-oriented model over a standard low-latency option?

Recall: Reasoning benefit under the actual deadline.

Q53: How do you prevent direct prompt injection in user input?

Recall: User request stays within authenticated authority.

Q54: Explain agentic RAG versus traditional RAG

Recall: Fixed retrieval versus bounded adaptive evidence gathering.

Q55: A RAG system works on test data but fails in production. What do you check?

Recall: Distribution → source → retrieval → packing → generation.

Q56: How do you implement guardrails for an autonomous agent taking real-world actions?

Recall: Consequence → current authority → exact approval → effect.

Q57: How would you optimize KV-cache pressure in a serving system?

Recall: Occupancy → scheduler → layout → precision → lifecycle.

Q58: Design a system where one user's prompt cannot leak to another user

Recall: Every data, artifact, cache and output path.

Q59: Your LLM costs are ten times higher than expected. Walk through the investigation

Recall: Reconcile bill → attribute usage → contain driver.

Q60: How would you evaluate whether an LLM is hallucinating?

Recall: Supported, correct, incomplete and unknown are distinct.

Q61: What tradeoffs matter when choosing embeddings for RAG?

Recall: Representation quality plus migration economics.

Q62: Relevant search results are ignored and the model answers from prior knowledge. What do you do?

Recall: Inspect the actual context before rewriting instructions.

Q63: How do you version prompts in production?

Recall: Recipe + compiled artifact + compatible release.

Q64: Design a semantic cache that works in production

Recall: Valid equivalence → scope → freshness → false reuse.

Q65: Your agent can execute arbitrary Python. How do you make this safe?

Recall: Untrusted code → isolation → no ambient authority.

Full practice: model controls and production judgment (Q66–Q80)

Q66: When would you use extended or adaptive thinking, and how do you control cost?

Recall: Effort guidance is not a hard task budget.

def deliberate_answer(client, supported_model_id, question):
    return client.messages.create(
        model=supported_model_id,
        max_tokens=4096,
        thinking={"type": "adaptive"},
        output_config={"effort": "medium"},
        messages=[{"role": "user", "content": question}],
    )

Q67: How do you compare reasoning-effort controls across providers?

Recall: Compare outcomes, not provider effort labels.

Q68: How would you use a coding agent in CI for automated bug fixing?

Recall: Pinned task → isolated patch → protected checks → review.

job: propose_bugfix
input: issue_and_pinned_commit
workspace: disposable_isolated_checkout
secrets: none
network: approved_package_mirror_only
limits: {attempts: 3, minutes: 15, cost_usd: 5}
protected: [release_configuration, approval_policy, regression_harness]
checks: [reproduce_bug, syntax, types, tests, security_review]
output: patch_and_check_evidence
next_step: human_code_review

Q69: A strong, inexpensive open-weight model becomes available. How does it affect architecture?

Recall: Rights → quality → serving → lifecycle → total cost.

Q70: Explain provider-level prompt caching and how to improve eligible reuse

Recall: Eligible prefix → actual usage → expiry → economics.

stable_system = [{"type": "text", "text": approved_policy_text,
                  "cache_control": {"type": "ephemeral"}}]
# Pass stable_system as the system content; put changing request data in messages.

Q71: How do you build an LLM-as-judge evaluation pipeline?

Recall: Calibrate judge → protect holdout → inspect false decisions.

{
  "case_id": "c17",
  "rubric_version": "grounding-v3",
  "verdict": "fail",
  "reason_code": "unsupported_eligibility",
  "evidence_ids": ["policy-19-p2"],
  "unjudgeable": false
}

Q72: How do you explain current MCP versions and production security risks?

Recall: Protocol revision ≠ SDK version ≠ application policy.

Q73: Design a router that chooses the cheapest adequate model

Recall: Eligible candidates → outcome labels → routing → audit.

Q74: Someone claims 95% accuracy. What do you ask?

Recall: Definition → denominator → sampling → severity → uncertainty.

Q75: How do SWE-bench Verified and LiveCodeBench differ?

Recall: Repository repair versus time-indexed coding tasks.

Q76: Quality drops after a possible provider update. How do you respond?

Recall: Confirm decline → contain → isolate cause → regressions.

Q77: Design a multi-provider architecture targeting 99.9% availability

Recall: Task SLI → eligible spare capacity → failover drills.

Q78: Should a huge context window replace the entire RAG pipeline?

Recall: Bounded direct context versus selective current evidence.

Q79: How do you defend a multi-tenant browsing agent against indirect injection?

Recall: External facts never become tenant authority.

def execute_proposal(proposal, principal, policy, tools):
    name = proposal["tool"]
    if name not in tools:
        raise PermissionError("Tool unavailable")
    validated_args = tools[name].validate(proposal["arguments"])
    policy.require_allowed(principal, name, validated_args)
    return tools[name].execute(principal, validated_args)

Q80: What is the difference between error analysis and automated evaluation?

Recall: Explain failures, then automate meaningful measurement.

Full practice: emerging techniques and risk decisions (Q81–Q96)

Q81: Pick a frontier model for a production agent and defend the choice

Recall: Defend one workload-specific, measured configuration.

Q82: How would you exploit cache discounts or off-peak pricing?

Recall: Eligibility → timing → realized discount → full overhead.

Q83: A model advertises an enormous context window. How do you assess practical usefulness?

Recall: Accepted length is not reliable evidence use.

Q84: When would you deploy a latent or continuous-space reasoning approach?

Recall: Research mechanism → compute comparison → serving evidence.

Q85: When does an agent need a memory layer beyond a long context window?

Recall: Purpose → provenance → recall → correction/deletion.

Q86: How should an AI manager hire as prompting becomes part of broader engineering work?

Recall: Hire for observed work and missing capability.

Q87: An agent calls a broken tool 400 times. What prevents this?

Recall: Terminal errors → runtime limits → independent stop.

Q88: When does an agent-as-judge outperform a single model judge?

Recall: Additional evidence versus new evaluator failure modes.

Q89: Design a process reward model for a customer-support agent

Recall: Observable step quality plus independent outcome gates.

Q90: When do you use A2A versus MCP, and how do they compose?

Recall: Agent task delegation versus tool/context integration.

Q91: A critical vulnerability is disclosed in an MCP server or transport implementation. What do you do?

Recall: Affected deployment → contain → patch → investigate → recover.

Q92: How does AI-assisted exploitation change your threat model?

Recall: Changed attacker economics; concrete boundaries still matter.

Q93: How do you coordinate AI impact and privacy assessments for an EU product?

Recall: Use → actor → data → applicability → release evidence.

Q94: Design a computer-use agent's sandbox and confirmation process

Recall: Observe → validate proposal → authorize → act → verify.

Q95: What does model signing buy you, and what gaps remain?

Recall: Expected signer and digest do not prove safe behavior.

Q96: Design layered defense against indirect prompt injection in RAG

Recall: Source provenance → narrow capability → enforced boundary.

Full practice: infrastructure and AI leadership (Q97–Q112)

Q97: What changes when serving a mixture-of-experts model?

Recall: Total weights → expert dispatch → imbalance → tail latency.

Q98: Quote a distillation project intended to reduce a $50,000 monthly model bill

Recall: Initial investment + recurring costs → measured payback.

Q99: How would you choose between vLLM, SGLang, and TensorRT-LLM?

Recall: Exact model/hardware/workload → usable capacity → operations.

Q100: How would you size an accelerator fleet for the next six months?

Recall: Traffic → measured replica → failure headroom → commitment.

Q101: Compare namespaces, per-tenant shards, and row-level policies for multi-tenant RAG

Recall: Access enforcement ≠ resource isolation ≠ operating scale.

Q102: When should a company hire forward-deployed engineers?

Recall: Customer integration bottleneck → reuse → supported handover.

Q103: What does a provider policy change teach about vendor lock-in?

Recall: Identify dependency → evaluate exit path → preserve useful options.

Q104: What would an autonomous shop-management experiment teach about agency limits?

Recall: Business invariants across many locally plausible actions.

Q105: How should vendor strategy changes affect an open-weight strategy?

Recall: Available rights and artifacts, not future vendor promises.

Q106: How do you establish an evaluation culture without encouraging metric gaming?

Recall: Useful decisions → protected cases → honest incentives.

Q107: What belongs in a PRD for a generative AI feature?

Recall: User outcome → scope → failure behavior → acceptance evidence.

Q108: Design fraud detection with a p99 below 500 ms and an LLM layer

Recall: Percentile target ≠ per-request deadline.

Q109: How should code review change when many pull requests are agent-generated?

Recall: Useful patches → review capacity → defects → accountability.

Q110: Your legal-research assistant fabricates a citation. How do you respond?

Recall: Contain reliance → qualified owners → corrections → verified repair.

Q111: Critique classifier-gated fallback to a stronger model

Recall: Policy eligibility before quality and cost routing.

Q112: An agent degrades after 30 minutes. How do you diagnose it?

Recall: Lost state, expired authority or failing dependencies?

Full practice: agent reliability and current protocol changes (Q113–Q128)

Q113: A computer-use agent fails 30% of real workflows. What is your reliability plan?

Recall: Failure taxonomy → verified state → bounded recovery.

Q114: How do agent skills differ from tools and fine-tuning?

Recall: Guidance versus executable capability versus learned behavior.

Q115: Evaluation scores improve but production complaints stay flat. What do you change?

Recall: Compare complaints with what the metric actually measures.

Q116: Design cost-aware multi-provider routing using current prices

Recall: Actual billing categories and outcomes per route.

Q117: Plan a migration from older stateful MCP deployments to the 2026-07-28 revision

Recall: Compatibility matrix → self-contained requests → durable effects.

Q118: A provider fails during a 40-step task. How do you recover?

Recall: Persisted progress → unknown effects → eligible continuation.

Q119: Design telemetry for a coding agent that avoids covert repository or secret uploads

Recall: Task context and optional telemetry have separate purposes.

Q120: How do you train and evaluate a request-level router without feedback loops?

Recall: Counterfactual labels → protected test → safe exploration.

Q121: Design an environment for evaluating dangerous agent capabilities safely

Recall: Protected grading plus an independently enforced outer boundary.

Q122: Design a browsing agent with payment authority that resists data injection

Recall: Trusted recipient → exact approval → current checks → receipt.

Q123: A provider raises prices sharply. How do you rebuild the cost model?

Recall: New rates × actual usage; normalize the comparison.

Q124: How do you control access to a powerful dual-use capability?

Recall: Specific purpose → scoped capability → revocation → oversight.

Q125: How do you defend coding agents against poisoned repository or package hooks?

Recall: Opening and indexing can execute code; inspect that boundary.

Q126: Design review and distribution for internal agent plugins

Recall: Artifact inventory → isolated review → controlled release → revoke.

Q127: Design a secure multi-tenant MCP server using stateless requests

Recall: Authenticate each request; reauthorize every handle use.

Q128: What transparency features would you build for generated content in multiple markets?

Recall: Jurisdiction and role → precise duty → testable artifact.

Five worked system-design scenarios

Use these as complete mock prompts. A possible 35-minute allocation is five minutes for scope, eight for the baseline, ten for the deep dive, seven for failures and economics, and five for closing. Adapt it to the actual interview. All numbers below are independent hypothetical worksheets, not measured deployments or vendor quotes. Valued human time represents capacity and economic cost; reducing it does not automatically reduce payroll.

Scenario 1: Design a customer-support chatbot

Prompt: Handle 10,000 tickets/day in five languages using product documentation, customer order history and a ticketing system. Support human handoff, and decide how to handle refunds.

Functional requirements

1. Answer product and policy questions with applicable evidence in the customer's language.
2. Read only orders belonging to the authenticated customer.
3. Prepare an eligible refund proposal; execute only within current authority and required approval.
4. Transfer unresolved cases with evidence, attempted actions and current status.

Nonfunctional requirements

1. For the supported short-answer class, target p95 complete informational replies within five seconds; external approval and handoff have separate clocks.
2. Define correct resolution, recontact and harmful-action rates by language and issue type. Treat 70% resolution without handoff as a hypothesis to validate.
3. Preserve current policy/customer scope and prevent duplicate refunds across retries or restarts.
4. Bound queue age, calls and spend; provide an explicit pending state during dependency failures.

Basic design

flowchart TD
    U[Authenticated customer] --> A[Support application]
    A --> D[Current product and policy search]
    A --> O[Authorized order read]
    D --> M[Evidence packet for support representative]
    O --> M
    M --> H[Human reply or handoff]

Start with a human-led support application and read-only order access. Measure its evidence quality, handling time and resolution rate before adding generated answers or automated financial effects. Source ingestion must track policy versions and applicable product, region and dates.

Find flaws and justify changes

Detailed design

flowchart TD
    P[Versioned policy ingestion] --> R[Search index and applicability catalog]
    U[Customer request] --> G[Identity, scope and deadline]
    G --> C[Current permitted order and policy context]
    R --> C
    C --> M[Answer or typed action proposal]
    M --> V[Evidence and business validation]
    V -->|supported answer| A[Reply]
    V -->|refund| H[Approval of exact proposal if required]
    H --> J[Durable operation record and balance reservation]
    J --> X[Payment service with stable operation key]
    X -->|confirmed| T[Receipt and ticket update]
    X -->|unknown| Q[Reconciliation without a new operation]
    V -->|unresolved| S[Human handoff with evidence]
    Q --> S

A proposal includes customer/order, amount in minor units, currency, policy version and digest. Bind approval to that digest and expiry. Execution rechecks current authority and refundable balance at the effect boundary; two concurrent valid proposals must not over-refund the order. Store the logical operation separately from attempt IDs. A timeout is an unknown outcome, not permission to create a new transfer. Ticket updates need their own deduplication.

Capacity: 10,000 cases/day averages 0.116 cases/s over 24 hours; measure peak arrivals and calls/case. If 30% require eight human minutes, review requires 3,000 × 8 / 60 = 400 hours/day. At six productive hours per staffed shift, that is about 67 shifts before coverage and absence allowance.

Full cost and benefit

Assume human time at $60/hour. Compare the same daily case volume and accepted resolution quality.

The projected $33,000/day improvement depends on correct automated resolution and the assumed remaining review time. A lower handoff rate obtained by silently closing unresolved cases is not a saving. Include recontacts, complaints and payment correction work in measured results.

Closing: Launch a bounded informational scope, then add each action only with an enforceable state and recovery contract. Measure complete resolution and human capacity, not chatbot containment alone.

Changed constraint: Refunds are prohibited. Remove the payment capability and produce a reviewable request for an authorized person; do not merely change the prompt.

Deep dive: Customer-support design and refund whiteboard walkthrough.

Scenario 2: Design a document-processing pipeline

Prompt: Process 100,000 invoices, contracts and forms/day from PDFs, images and scans. Leadership requests “99% accuracy.” Establish what that means before choosing an extraction model.

Functional requirements

1. Accept declared document types and retain private versioned source bytes.
2. Extract typed fields with page/region provenance and missing-value states.
3. Validate schema and applicable cross-field business relationships.
4. Let authorized reviewers correct an exact extraction version and publish an accepted record.

Nonfunctional requirements

1. Assume ten pages/document for sizing; report field accuracy, complete-document correctness and false acceptance separately.
2. For this mock, target automated drafts within five minutes at p95 for documents of at most 20 pages; review has a same-day service target.
3. Preserve tenant scope, retention and actual assurance obligations. Document format or financial-services use alone does not establish HIPAA applicability.
4. Resume partial processing, expose missing pages and prevent duplicate downstream business effects.

Basic design

flowchart TD
    U[Validated upload] --> O[Private source object]
    O --> P[Parser or OCR]
    P --> E[Field extraction]
    E --> H[Human checks source and fields]
    H --> A[Accepted record]

Keep structured text, tables, units and reading order where available. OCR is a text-recognition stage; it does not by itself establish the meaning or business correctness of a value.

Find flaws and justify changes

Detailed design

flowchart TD
    U[Authenticated source upload] --> S[Immutable source and document job]
    S --> Q[Bounded page-work queue]
    Q --> P[Parser, OCR and layout workers]
    P --> A[Expected-page aggregation]
    A --> E[Typed fields with source regions]
    E --> V[Schema, arithmetic and business checks]
    V -->|eligible| R[Accepted extraction revision]
    V -->|uncertain or incomplete| H[Prioritized reviewer queue]
    H -->|exact reviewed revision| R
    R --> B[Separately authorized business posting]
    B --> C[Receipt or unknown-outcome reconciliation]

A record carries tenant, source digest, parser/extractor version, field value/type, coordinate convention, evidence region and status. An unreadable amount stays unresolved. If line items, discounts and tax do not reconcile, show the conflicting evidence; do not invent the missing charge. A changed source or material correction invalidates earlier approval. Repeated parsing may be reusable within scope, but two byte-different uploads can still represent the same business invoice.

Capacity: One million pages/day at an illustrative two worker-seconds/page requires 2,000,000 / 86,400 = 23.15 fully occupied worker equivalents. At 70% utilization, use at least 34 comparable workers for average load, then size peaks and failures. A 5% review rate at three minutes/document adds 250 human-hours/day. Under the illustrative assumption of independent field errors, 100 fields each 99% correct give only 0.99^100 ≈ 36.6% complete records; actual errors are correlated, so measure the document result directly.

Full cost and benefit

Assume $60/hour human time and the same 100,000 daily documents.

The projected $39,000/day reduction requires the lower review rate to retain acceptable false-acceptance and whole-record quality. A backlog that delays important invoices can erase savings even when extraction is inexpensive.

Closing: Optimize for an accurate, traceable accepted record. Keep extraction, review and posting as separate states and size human work alongside page processing.

Changed constraint: Same-day completion is mandatory and reviewers are saturated. Bound intake, prioritize consequential cases and agree on deferred scope; do not lower review thresholds without quality evidence.

Deep dive: Document-intelligence design.

Scenario 3: Design enterprise RAG search

Prompt: Serve 50,000 employees over ten million changing documents with document-level permissions. Explain new content, corrections, deletion and revocation separately.

Functional requirements

1. Search authorized company documents and answer questions with supporting citations.
2. Ingest versioned source changes, deletions and access changes.
3. Apply policy region, date and other relevant applicability rules.
4. Clarify, abstain or escalate when evidence is missing or conflicting.

Nonfunctional requirements

1. For this mock, target complete short answers within three seconds at p95 under the declared peak load.
2. Target normal source-update freshness within 15 minutes; current authorization checks govern sensitive disclosure.
3. Prevent cross-user/tenant leakage through retrieval, saved answers, caches and traces.
4. Define supported-answer correctness, useful coverage and critical-case gates; latency alone is insufficient.

Basic design

flowchart TD
    D[Source documents and access metadata] --> P[Parse and lexical index]
    U[Authenticated question] --> R[Permitted search]
    P --> R
    R --> C[Selected evidence and citations]
    C --> M[Employee reads sources or asks helpdesk]

Begin with permission-aware search and human-assisted answers as the cost baseline. Add evidence-grounded generation, then vector retrieval, reranking or adaptive search when measured failures justify them. A travel-policy answer must include the current applicable exception, not simply a textually similar global policy.

Find flaws and justify changes

Detailed design

flowchart TD
    D[Source connectors and durable cursors] --> Q[Versioned change queue]
    Q --> P[Structure-preserving parsing]
    P --> I[Lexical and vector index generation]
    Q --> A[Current access and source-version catalog]
    U[Authenticated request and deadline] --> R[Scoped hybrid retrieval]
    I --> R
    A --> R
    R --> K[Optional rerank and evidence budget]
    K --> M[Answer generation]
    M --> V[Claim support, citation and current-access checks]
    A --> V
    V --> O[Answer, clarification or abstention]
    V --> T[Privacy-scoped traces and slice evaluation]

Bind an evidence item to source, revision, passage, access decision and applicability metadata. An access check cannot precede the source change it has not learned about: declare connector detection delay and use an authoritative read where the requirement demands immediate current access. Observed revocations invalidate serving, caches and affected in-flight work. Index migrations bind the query encoder to the matching index and retain a compatible rollback version.

Capacity: Ten chunks/document implies 100 million records. At 768 float32 dimensions, vectors alone occupy 307.2 decimal GB per full copy. Two replicas and two simultaneous model/index generations require 1.2288 TB of raw vectors before index graphs, metadata and backups. Assume 50,000 queries/day over eight busy hours: average 1.736 requests/s; a provisional fivefold peak is 8.68 requests/s. Measure input/output token distributions rather than sizing from employee count.

Full cost and benefit

Compare a search-and-human-help baseline with the candidate for 1.1 million monthly questions, assuming 22 days and $2 per human assistance case.

The projected $93,000/month improvement depends mainly on reduced assistance without worse decisions. Citations and a fluent answer do not establish that reduction. Include source administration, recontact and correction work when measuring actual costs.

Closing: Current authorized evidence is the core contract. Improve retrieval selectively, track source and permission changes explicitly, and validate complete answers and operating costs before expansion.

Changed constraint: Permissions change frequently. Measure detection and enforcement separately and show what happens when the authoritative permission service is unavailable.

Deep dive: Enterprise-RAG design.

Scenario 4: Design a code assistant

Prompt: Build IDE completion, explanation and multi-file editing using repository context. Support streaming and a clear source-code privacy policy.

Functional requirements

1. Suggest code for a specific file revision, cursor position and surrounding prefix/suffix.
2. Explain code using permitted repository definitions and references.
3. Produce reviewable multi-file patches and run permitted checks.
4. Report evidence, incomplete checks and changes requiring human review.

Nonfunctional requirements

1. For this mock, target inline first suggestions within 300 ms at p95; use a separate visible progress contract for multi-file work.
2. Prevent stale suggestions after the file or cursor changes and preserve user edits.
3. Keep repository text out of optional telemetry by default; enforce permitted model destinations.
4. Isolate untrusted execution, bound resources and preserve merge/deploy authority outside the agent.

Basic design

flowchart TD
    E[IDE file, revision and cursor] --> C[Authorized local context selection]
    C --> M[Completion model]
    M --> V[Current revision and cursor check]
    V --> S[Stream suggestion]
    S --> U[Developer accepts or rejects]

The baseline handles local completion. Multi-file editing requires a separate durable workflow because it has different latency, context, execution and recovery needs.

Find flaws and justify changes

Detailed design

flowchart TD
    R[Versioned repository events] --> I[Symbols, lexical and optional vector index]
    U[IDE request] --> G[Identity, file revision and request class]
    G -->|inline| C[Small context and tight deadline]
    C --> M[Completion model]
    M --> V[Still-current cursor and revision]
    V --> S[Suggestion stream]
    G -->|multi-file| Q[Durable bounded edit job]
    I --> Q
    Q --> X[Isolated checkout and scoped context]
    X --> P[Patch proposal]
    P --> T[Permitted checks against protected contracts]
    T --> H[Human patch and evidence review]
    H --> B[Normal repository merge controls]

Use (repository, base_commit, document_version, cursor, request_id) for interactive work, and a separate job/attempt identity for edits. On disconnect or new typing, cancel upstream where supported and reject late output regardless of cancellation success. Track test PASS, FAIL, UNAVAILABLE and NOT_RUN distinctly. A passing unit suite cannot establish compatibility with a consumer it never exercises. Rebase or apply a patch against the current owner state with conflicts surfaced; never overwrite concurrent edits silently.

Capacity: Suppose 10,000 active developers issue 100 inline requests/day: one million requests over eight busy hours, or 34.72/s average. A provisional threefold peak is 104.17/s. At 1,500 input and 80 output tokens/request, peak demand is about 156,250 input and 8,333 output tokens/s. Separately, 5,000 edit jobs/day occupying a worker for two minutes require 6.94 continuously occupied workers over 24 hours before peaks and headroom.

Full cost and benefit

Compare 30,000 weekly coding tasks, with human time valued at $90/hour ($1.50/minute). The candidate's 11 minutes includes suggestion review and fixes; do not add the same review time twice.

The projected $23,000/week improvement requires the one-minute average reduction with comparable defect outcomes. If task time remains 12 minutes, the candidate costs $565,000/week. Acceptance rate alone cannot prove time saved or code correctness.

Closing: Separate fast completion from durable editing. Bind every result to source state, retain developer ownership and measure useful task time and escaped defects alongside model cost.

Changed constraint: Execution is unavailable. Keep the patch reviewable and label checks unavailable; do not present it as a tested fix.

Deep dive: Code-assistant design and autonomous coding recovery.

Scenario 5: Design AI-assisted content moderation

Prompt: Handle one million text, image and video posts/day with an initial visibility decision below 500 ms and an appeal process. Define what happens to media that cannot be completely assessed that quickly.

Functional requirements

1. Assess supported content under versioned category and language policies.
2. Record explicit allow, restrict or pending-review treatment.
3. Perform deeper media analysis and authorized human review when necessary.
4. Support appeals, revised decisions and enforcement against the correct content revision.

Nonfunctional requirements

1. For this mock, target p99 initial-state latency below 500 ms; separately measure pending fraction and final-review delay.
2. Set category/language-specific false-positive, false-negative and severe-harm criteria.
3. Bound media duration/size, queues and retry work; long videos cannot be assumed fully analyzed within the short request budget.
4. Restrict review evidence and preserve an auditable decision history without treating appeals as unbiased labels.

Basic design

flowchart TD
    U[Content submission] --> F[Fast rules and classifier]
    F --> D[Policy decision]
    D --> V[Allow or restrict]
    V --> A[Appeal queue]

Separate category detection from the policy mapping to enforcement. One score cannot express every jurisdiction, context and action rule.

Find flaws and justify changes

Detailed design

flowchart TD
    U[Validated post and media revision] --> S[Private staging]
    S --> F[Fast checks within request budget]
    F --> D{Policy action determined?}
    D -->|yes| E[Version-bound enforcement]
    D -->|no| P[Explicit pending visibility]
    P --> Q[Bounded deep-analysis queue]
    Q --> M[Media evidence and policy checks]
    M --> H[Reviewer or validated decision rule]
    H --> E
    E --> O[Visible or restricted content]
    O -->|appeal| A[Independent reassessment]
    A -->|new decision revision| E
    A --> L[Adjudicated feedback dataset]

Bind each decision to content revision, policy version, category, evidence and action ID. Recheck the revision atomically when enforcing. If a provider fails, apply the agreed temporary treatment and record it. Measure user impact and queue age; reporting only the fast path hides work waiting for hours. Include reviewer wellbeing, escalation and access controls in operations.

Capacity and quality: One million posts/day averages 11.57/s over 24 hours, but video bytes and duration determine much of the workload. At 0.1% harmful prevalence, 90% recall and 1% false-positive rate yield 900 true positives and 9,990 false positives: only 8.26% of 10,890 flags are truly positive. If every flag takes two minutes to review, that is 363 hours/day. The false-positive rate uses benign items as its denominator; it is not the fraction of flags that are false.

Full cost and benefit

Assume a candidate reaches a 0.1% false-positive rate at the same 90% recall, verified on a representative sample. It produces 999 false positives plus 900 true positives. At $60/hour, each two-minute review costs $2.

The potential $16,882/day reduction depends on the quality improvement, review time and stable workload. Both versions still miss 100 harmful posts under the assumptions. A threshold change does not guarantee lower false positives while retaining recall; measure the operating point and harm independently of savings.

Closing: Define policy and temporary visibility first, then measure error costs at real prevalence. Preserve a reliable enforcement/appeal lifecycle and enough review capacity to meet the actual service promise.

Changed constraint: Every accepted video must be fully reviewed before visibility. Bound duration and processing capacity or change the visibility deadline; an asynchronous queue cannot remove that conflict.

Deep dive: Moderation design.

Answer guides for the ten manager follow-ups

Use these to organize a real experience. The answers must come from your work; do not adopt an invented success story or metric.

Manager Q1: What project did you stop, and how did you communicate the decision?

Answer guide: Explain the intended outcome, the evidence that the investment was not working, and the options you considered before stopping it. Describe your own decision responsibility and how you involved stakeholders. Then explain how you handled commitments, customer impact, and the team's morale, including what useful work was retained. End with the observed consequence and what would justify revisiting the idea. A strong answer shows judgment under uncertainty rather than celebrating cancellation as a success in itself.

Probe: What did you know at the time, and what did you learn only afterward?

Manager Q2: How did you hire for a capability your team lacked?

Answer guide: Name the actual capability gap and how it affected delivery. Explain why hiring was appropriate compared with training, contracting, or changing scope. Describe the role definition, evidence-based interview criteria, and how you avoided selecting only for familiar backgrounds. Include onboarding and the support needed for the person to become effective. Finish with what changed in team capability and how you assessed it. Hiring is a team-design decision, not just the story of filling an open position.

Probe: How did you distinguish a specialist's knowledge from their ability to apply it in your environment?

Manager Q3: How did you coach someone into broader ownership?

Answer guide: Describe the person's starting strengths, the growth opportunity, and the responsibility you agreed they would take on. Explain the support, feedback cadence, and boundaries you provided without taking the work back at the first difficulty. Use an example of a decision they learned to make independently. Report the outcome with evidence and acknowledge where your approach changed. The interviewer should see both development of the person and protection of delivery commitments.

Probe: What did you stop doing so the person could actually own the work?

Manager Q4: How did you handle sustained underperformance fairly?

Answer guide: Separate observable behavior and outcomes from assumptions about motivation. Explain how you clarified expectations, checked for role or support problems, gave specific feedback, and agreed on measurable improvement and time frames. Describe the help offered and how you documented progress consistently with company process. If improvement did not occur, explain the decision and communication respectfully without revealing private details. A strong answer balances fairness to the individual with responsibility to the team.

Probe: How did you know the expectations were reasonable and understood?

Manager Q5: When did you disagree with research or product, and what resolved it?

Answer guide: Explain the shared goal and the competing assumptions. Give the other side's argument fairly, then describe how you made the tradeoff concrete through evidence, a bounded experiment, or an explicit accountable decision. State whether you changed your mind and why. After the decision, explain how the team committed and how results were reviewed. Avoid a story whose only lesson is that you persuaded everyone you were right.

Probe: What evidence would have changed your original position?

Manager Q6: How did you delegate a consequential technical decision?

Answer guide: Describe why the delegate was suited to the decision and what authority you gave them. Explain the constraints, success criteria, stakeholders, and escalation triggers, then show how you remained informed without requiring approval for every small step. Discuss a difficult point where you coached or intervened and why. End with the decision's outcome and how it changed the person's future scope. Delegation is credible when responsibility and authority move together.

Probe: Which decisions remained yours, and did everyone understand that boundary?

Manager Q7: How did you make evaluation a recurring team practice?

Answer guide: Start with a decision or failure that existing measurement could not explain. Describe how the team built representative cases and a rubric, assigned ownership, and integrated results into iteration and release. Show how people investigated errors rather than chasing a score. Include the cost of maintaining the suite and how fresh production evidence entered it safely. The outcome should show better decisions or reduced failures, not merely a larger test dashboard.

Probe: Tell me about a time the evaluation disagreed with users and you changed it.

Manager Q8: What did you do when a launch harmed users or missed expectations?

Answer guide: Explain how you recognized the problem, contained impact, and assigned incident roles. Describe communication, rollback or remediation, and how you supported affected users. Then identify the contributing system and process causes, including your own responsibility, and explain the changes with owners and follow-through. Separate facts known during the incident from later findings. A useful answer shows calm accountability and learning without blaming an individual for a systemic failure.

Probe: Which corrective action changed the likelihood or impact of recurrence, and how did you verify that?

Manager Q9: How did you choose between platform investment and near-term delivery?

Answer guide: Quantify the repeated pain or risk the platform would address and compare it with the opportunity cost of delaying product work. Explain who would use the capability, how much adoption was realistic, and whether a smaller increment could test the value. Describe the decision, ownership, and milestones, including a condition for stopping or changing direction. Report what happened to delivery speed, reliability, or cost rather than assuming that building a platform is inherently strategic.

Probe: How did you avoid creating infrastructure for hypothetical future customers?

Manager Q10: What evidence would change your roadmap or staffing plan?

Answer guide: State the assumptions behind the current plan, such as customer demand, task quality, delivery capacity, or unit economics. Identify observable signals and decision thresholds, then explain how you would collect evidence without waiting indefinitely for certainty. Discuss the consequences of being early or late and how reversible commitments affect the decision. A strong answer combines conviction about the goal with willingness to change the approach when the evidence changes.

Probe: Name a decision you would make now and one you would deliberately keep open.

Technical references for further explanation

The subject chapters contain fuller diagrams, mechanisms, and worked examples. For specific framework and protocol behavior, use versioned primary documentation:

• Speculative decoding research explains the exact sampling construction discussed in Q23.
• vLLM documentation, SGLang documentation, and TensorRT-LLM documentation are the starting points for Q99's version-specific support comparison.
• Durable execution, LLM internals, and embeddings provide the deeper mechanism explanations behind the shorter spoken answers.

Final summary and notes

I prepare by explaining decisions aloud, not by memorizing long answers or product-release trivia. For each topic, I give a plain definition, work through a concrete example, explain the main failure mode, and defend a tradeoff. I check my answer against evidence and retry the questions I cannot explain clearly. Leadership preparation, when relevant to the role, also needs real stories about hiring, coaching, delivery, conflict, and stopping weak investments. Technical fluency and leadership evidence reinforce each other.

Practice tip: Revisit one missed definition, one calculation and one changed-constraint scenario in the next session. Explain them without looking before reviewing the answer. This is a self-check, not a hiring guarantee.

An answer framework is a way to organize reasoning so another person can follow it. For system design, use the conventional sequence: clarify requirements, estimate scale, propose a baseline, define contracts, investigate bottlenecks, compare changes, and explain evaluation and operations.

Remember: requirements → baseline → evidence → improvements → closing decision.

The practice mnemonics SPIDER and ETA on this page are optional memory aids used in this guide, not globally standardized interview methods. STAR is an established structure for behavioral answers; “L” makes reflection explicit. A framework should support the conversation rather than dictate every minute.

Choose the structure that fits the question

If a term is unfamiliar, say so. Explain the capability you need and reason from the known contract. Do not guess SDK behavior from a product name or present a hypothetical deployment as experience.

System design framework: requirements first

If you use SPIDER, read it as Scope, Prioritize, Initial design, Deep dive, Evaluation, Reliability. Reliability, access control and evaluation also influence earlier decisions; they are not deferred until the final letter.

1. Scope the outcome

For “design an employee assistant,” clarify:

1. Who uses it, and what counts as successful completion?
2. Does it answer questions, create drafts or change records?
3. Which sources are authoritative, and who may access them?
4. Which mistakes are tolerable, and which must block a release?
5. What workload and operating constraints should the design support?

A public handbook search box and a payroll-changing agent need different permissions and recovery. Summarize the scope before drawing, and do not spend the whole session gathering every possible requirement.

2. Write functional and nonfunctional requirements separately

Functional requirements describe required behavior. Nonfunctional requirements describe quality attributes or operating constraints on that behavior. Teams sometimes classify a security rule differently; make the requirement explicit rather than arguing about the heading.

Distinguish a hard constraint, which rules out an option if violated, from a preference, which can be traded against another benefit. A weighted quality/cost score cannot compensate for failing an access requirement.

3. Estimate only what changes a decision

Write the units and assumptions. Calculate request rate, token throughput, storage or concurrency where they constrain the architecture. Average traffic does not determine peak capacity. Inference sizing and cost models provide detailed examples.

4. Draw a complete baseline

Show the actor, input, services, stores and output. Separate data preparation from live requests, and mark authorization and state ownership. A generic “AI layer” box is not enough to explain where evidence comes from or which operation may be retried.

Then trace one request end to end. Choose one or two deep dives from the dominant risks or the interviewer's direction. Add components when a requirement or measured failure justifies them.

5. Close the design with evidence and operations

Good offline results do not by themselves authorize broad exposure. Name the on-call owner, dashboard signals, fallback behavior and unresolved assumptions.

Worked interview: employee policy assistant

This is an illustrative practice design, not a report of an actual deployment. A 45-minute session might allocate 5 minutes to requirements, 5 to scale, 8 to the baseline, 15 to deep dives, 7 to evaluation/cost and 5 to closing discussion. Adapt to the interviewer's direction.

Functional requirements

1. Answer employee policy questions using authorized, current source material.
2. Attach passage-level citations with source and edition identifiers.
3. Return an explicit limitation when evidence is missing or contradictory.
4. Ingest updates and deletions from an owned policy corpus.
5. Provide an escalation route to the policy owner.

Out of scope: payroll writes, benefit enrollment, legal advice and autonomous HR decisions. This first release answers questions only.

Nonfunctional requirements

1. No cross-user disclosure through retrieval, model prompts, citations, logs or caches.
2. Updates visible within 15 minutes under normal operation; stale or unknown freshness must be visible to the serving policy.
3. Revoked source access must take effect before new disclosure; an unavailable permission check must not allow access.
4. p95 time to a verified final answer below 6 seconds at the agreed peak workload; first-token latency is measured separately.
5. At least 95% supported correct answers on the predefined answerable evaluation population, with uncertainty and slice results reported separately. Unanswerable cases and critical access failures have separate checks.
6. A 99.9% monthly service-availability target for the defined query operation; an error or dependency-unavailable response does not count as a successful answer merely because it uses HTTP 200.

The numerical targets are interview assumptions. Define the final acceptance criteria with the stakeholder rather than suggesting that these are universal defaults. A finite test suite cannot prove the absence of all security failures.

Scale and initial estimates

Confirm model quotas, queueing and realistic output lengths in load tests. A concurrency estimate is not a provider quota or capacity commitment.

Start with a baseline

flowchart LR
    S[Versioned policy sources] --> I[Parse and index]
    I --> X[Search index with source IDs]
    U[Employee query] --> A[Authenticate and check access]
    A --> R[Retrieve permitted passages]
    X --> R
    R --> P[Pack evidence]
    P --> M[Model drafts answer]
    M --> V[Validate citations and support]
    V --> O[Answer or limitation]

Begin with one corpus, lexical retrieval or a simple retrieval method appropriate to its content, and a model deployment that meets data constraints. Compare to the existing employee search experience. Do not assume a vector database, reranker and agent loop are all necessary.

Find the flaws before adding components

Detailed architecture and contracts

flowchart TD
    S[Source change or deletion] --> Q[Durable ingestion queue]
    Q --> W[Versioned parser and index worker]
    W --> X[Search index and source revision catalog]
    S --> P[Authoritative permissions and revocations]
    U[Authenticated employee] --> G[Gateway: identity, admission and deadline]
    G --> R[Retrieval with current access enforcement]
    X --> R
    P --> R
    R --> K[Evidence packing and optional reranking]
    K --> M[Approved model deployment]
    M --> V[Output and evidence checks]
    P --> V
    V --> A[Answer with citation dependencies]
    V --> H[Limitation or policy-owner escalation]
    G --> T[Restricted traces, metrics and usage ledger]
    W --> T
    V --> T

Explain these records before selecting a storage product:

A request-supplied employee ID does not establish identity. A cached group list does not prove that access is current. If source APIs cannot provide the promised revocation semantics, narrow the requirement or redesign the access boundary explicitly; do not promise instantaneous source truth from stale tags.

For this first release, avoid shared generated-answer caching. If later added, key by the relevant security scope and versions, record source dependencies, and invalidate/recheck on revocation. Do not disclose an old answer before its current authorization is established. See access control.

Recovery and degradation

1. Ingestion worker crash: reclaim an expired lease and retry the same source revision. Publish only a complete, current revision; a late worker cannot overwrite a newer one.
2. Source deletion: record a tombstone, remove searchable content and invalidate dependent artifacts. Serving checks the authoritative revision state during propagation.
3. Model timeout: classify the outcome and remaining deadline. A read-only inference retry still consumes cost; any future tool write would require operation reconciliation before replay.
4. Permission service outage: return unavailable for protected content. An older cached answer is not a safe shortcut around the failed check.
5. Insufficient evidence: report the limitation and offer the escalation path. Do not manufacture an answer to improve the completion metric.
6. Overload: bound the queue, apply per-user limits and shed work explicitly rather than allowing unbounded latency.

See reliability patterns and durable execution.

Evaluate the decision

Use the same held-out tasks for the search baseline and candidate. Include authoritative answers, unanswerable questions, stale editions, conflicting passages, access changes, injection attempts and relevant language slices. Measure retrieval separately from the final answer.

Report supported correct answers over the predefined eligible population, explicit limitations, incorrect answers and unresolved measurement cases. Do not silently exclude timeouts or bad answers from the denominator. Calibrate any LLM judge against appropriate human labels. See capability assessment.

Load-test end-to-end latency, including queueing and validation, and test degraded dependencies. Adding component p95 values does not calculate the system's p95. A pilot should also measure whether employees finish their task and how much human escalation work remains.

Calculate full cost and benefit

Assume 22 workdays/month: 1.1 million questions. The following figures are illustrative budget assumptions, not provider quotes or promised savings. An escalated question takes two minutes at a loaded rate of $60/hour, so each escalation costs $2.

The proposal saves $49,200/month only if the assumed escalation reduction occurs without unacceptable quality loss. If both systems produce 1 million independently verified accepted outcomes, the cost per 1,000 accepted outcomes is $116 versus $66.80. Count ordinary employee time separately if it differs; it is not included in this worksheet.

The candidate's non-escalation costs are $22,800. Its break-even escalation rate is (116,000 − 22,800) / (1,100,000 × 2) ≈ 4.24%. At 5%, the candidate costs $132,800 and is more expensive than the baseline. Measure the time per escalation and its tail, not just the rate.

Closing remarks

“I would start with a read-only assistant over one owned policy corpus. The main decisions are versioned evidence, current access checks and measurable answer support. Retrieval improvements are experiments against the baseline. I would pilot the system only after permission and recovery checks pass, then expand if final-answer quality, latency and full operating cost meet the agreed criteria. The largest unproven economic assumption is the reduction in human escalations.”

Explain a concept: definition, example, limits

ETA is a local memory aid for Explain simply, Technical details, Applications and tradeoffs. Start with the standard definition rather than an analogy alone.

For a KV cache:

1. Definition: a KV cache retains previously computed attention keys and values for reuse during compatible autoregressive decoding.
2. Mechanism: a new token produces its own states; attention can reuse earlier keys and values. Other computations and access to prior cache entries remain necessary.
3. Decision: cache memory may limit concurrent requests even when the model weights fit in memory.

For uniform full-attention layers, an uncompressed cache estimate is:

bytes = 2 × batch × cached_tokens × layers × KV_heads × head_width × bytes_per_element

For one sequence, 8,192 cached tokens, 32 layers, 8 KV heads, head width 128 and BF16 (2 bytes), that is 1 GiB. With 32 KV heads, it is 4 GiB. Hybrid, sliding-window or compressed architectures require their own formulas; parameter count alone does not specify cache size.

Paging reduces allocation waste; grouped-query attention changes the KV-head count. Neither makes all decoding work free. Provider prompt-cache pricing and eligibility are separate contracts from the mathematical cache estimate.

Defend a tradeoff with comparable evidence

Use options → hard gates → measurements → recommendation → reversal trigger.

An interface abstraction can reduce integration work, but cannot remove an embedding migration, tokenization difference or behavior change. Compare actual current deployments in the model-selection guide. Avoid arbitrary star scores and unexplained weights.

Debug with hypotheses and discriminating evidence

First define the user impact and contain it. Then compare failing and known-good cases with versions and timestamps. A recent change is a hypothesis, not automatic proof of cause.

flowchart TD
    A[Wrong or unsupported answer] --> B{Authoritative source correct?}
    B -->|No| C[Repair source quality or coverage]
    B -->|Yes| D{Current source parsed and indexed?}
    D -->|No| E[Repair ingestion and replay version]
    D -->|Yes| F{Required evidence reaches context?}
    F -->|No| G[Inspect permissions, retrieval and packing]
    F -->|Yes| H[Inspect interpretation, output and grading]
    C --> I[Verify repair and adjacent regression cases]
    E --> I
    G --> I
    H --> I

Avoid changing the model and retrieval pipeline together when trying to attribute the failure. In an incident, containment can precede complete diagnosis; record what changed so you can still investigate.

Behavioral answers: STAR with reflection

STAR means Situation, Task, Action and Result. It helps organize an actual experience. The National Careers Service also includes learning in its guidance; this guide's STAR-L makes that reflection explicit. STAR guidance.

A concrete qualitative result is valid when a trustworthy metric was not available. Say what was measured, what was observed and what remains uncertain. See behavioral preparation.

Handle interruptions and unknowns

An interruption changes the next explanation. It does not erase the requirements already agreed.

Interview questions with developed answers

Final summary and notes

Practice with the question bank, whiteboard exercises and common pitfalls.

A weak design answer often contains correct terms but omits the reasoning that connects them. This chapter helps you identify the missing requirement, mechanism or evidence, then repair the explanation. It is a practice checklist, not a universal employer scoring rubric.

Remember: requirement → mechanism → failure → evidence → decision.

For example, a policy assistant returns an obsolete leave rule. If ingestion missed the new edition, changing embeddings cannot retrieve it. If the current edition was retrieved but its exception was removed during context packing, a larger model still lacks the qualification. If the answer exposes restricted content, factual correctness does not make the disclosure acceptable.

Architecture pitfalls

1. Skipping the data pipeline

Gap: the diagram begins with a populated database and never explains how useful information arrives.

Repair: show ingestion, parsing, content boundaries, metadata, permissions, versioned indexing, updates and deletions. Name the source owner and the behavior when a source is stale or unavailable. Preserve evidence through each transformation. See data engineering for AI.

Check: trace one corrected policy from the source to a new answer and to any previously cached answer that depends on it.

2. Choosing models by brand or adding routing automatically

Gap: one model is declared best for all tasks, or a router is added merely to mention several models.

Repair: compare eligible deployments against the workload, quality, data, latency and complete-cost requirements. A single model can be an appropriate baseline. Routing is worthwhile only when its measured benefit exceeds classification mistakes, fallback calls and operating complexity. See model selection.

Check: identify the evidence that would make you remove the router.

3. Leaving evaluation until after the design

Gap: “we will monitor accuracy” does not say what a correct result is or how failures are counted.

Repair: define the task, target population, acceptance criteria and important slices. Evaluate the baseline and proposal on comparable held-out cases; report missing outcomes and uncertainty. Monitor production behavior and business outcomes separately from offline scores. See evaluation.

Check: could a system that abstains on every difficult question improve your metric without helping users?

4. Treating a tenant field as isolation

Gap: a client-supplied tenant_id is trusted, or all candidate text reaches the model before filtering.

Repair: derive identity and scope from authenticated server context. Enforce policy before protected data reaches unauthorized consumers, including models, rerankers, logs and caches. Include current user/group permissions within a tenant, resource ownership and noisy-neighbor limits. See access control.

A database's internal candidate filtering strategy is not by itself the authorization boundary. Explain which trusted service may access which data and where enforcement occurs. Tenant-specific embedding spaces do not replace access control.

Check: use two users in the same tenant with different document access, then revoke one permission.

5. Calling any fallback graceful degradation

Gap: retrieval fails, so the system generates an unsupported answer with a disclaimer; a protected service fails, so it returns a stale cached answer.

Repair: keep the original constraints during degradation. Valid options may be authorized source passages without synthesis, a permitted alternative model, an explicit unavailable response or human escalation. Classify errors, share the deadline across attempts and prevent retry storms. See reliability patterns.

Check: which fallback is still acceptable when current authorization cannot be established?

Technical knowledge pitfalls

6. Confusing representations with evidence

Gap: an embedding is treated as verified knowledge, or similarity as factual support.

Repair: an embedding represents an input as a vector useful for learned comparisons. In a typical text RAG pipeline, the generator receives selected passages, not the search vectors as a substitute for those passages. Related passages may contradict the proposed answer. See embeddings.

Check: explain how a retrieved passage supports the particular claim, including exceptions and edition.

7. Confusing context capacity with comprehension

Gap: a context limit is treated as proof that every fact inside that limit is used correctly, or a short prompt is claimed to eliminate all position effects.

Repair: count the full request using the model/API's rules, including tools, history and output allowance where applicable. Test relevant evidence among distractors and conflicting sources. The useful context depends on the task; there is no universal safe number of chunks. See context engineering.

Check: can the answer still be recovered when its evidence appears in a different position?

8. Reporting token savings as business savings

Gap: a rate is quoted without input/output units, retries, cache writes, tool charges or review work.

Repair: calculate the same workload and accepted outcomes, with explicit rate conditions and all incremental costs. Streaming changes delivery; it is not automatically a discount. Current rates belong in the dated pricing guide.

Worked example, using hypothetical rates: 10,000 requests/day, each with 2,000 input and 500 output tokens. At $2/$10 per million input/output tokens, model cost is $0.009/request = $90/day = $2,700 per 30-day month.

If 1,500 input tokens actually hit a cache priced at $0.20/million and the remaining 500 cost $2/million, each request costs $0.0003 + $0.001 + $0.005 = $0.0063. That is $1,890/month in this idealized steady state, a $810 or 30% call-cost reduction. Cache writes, misses and provider eligibility are excluded and must be added for a real estimate.

If the cache change adds $900/month of engineering and operations, total incremental comparison is $2,700 versus at least $2,790. Even before write/miss costs, it does not save money under these assumptions. Quality and authorization must remain acceptable too.

9. Listing RAG stages without explaining choices

Gap: “chunk, embed, retrieve, rerank” is recited without saying what each stage fixes.

Chunk overlap may preserve boundary context but also duplicates input and does not guarantee complete evidence. See chunking and reranking.

10. Treating prompts or schemas as control boundaries

Gap: “use only the evidence” is treated as guaranteed grounding, or valid JSON as an authorized action.

Repair: make instructions clear and preserve source provenance, then enforce permissions and business rules outside the model. Structured generation can constrain supported output shapes; application validation must still handle refusals, truncation, unsupported schemas and invalid domain values. See structured generation.

Check: a syntactically valid refund for the wrong customer must fail authorization.

Communication pitfalls

11. Monologuing or asking questions without making progress

Give a short roadmap, state bounded assumptions and draw the baseline. Pause at natural decision points so the interviewer can steer. Do not impose an exact interval for checking in or ask for approval after every sentence.

Repair phrase: “The complete request path is here. The main risk is revoked access; I will trace that next unless you want another area first.”

12. Using essay form for parallel requirements

A long paragraph hides missing constraints. Put functional and nonfunctional requirements in separate numbered lists. Use tables for alternatives and failure/repair comparisons, and diagrams for data movement.

Check: can the interviewer point to the requirement that justifies a new queue or model call? See answer frameworks.

13. Explaining one acronym with several more

Start with a standard definition, then a concrete example and the relevant limit. For idempotency, repeated identical requests have the same intended server effect as one request; the responses need not be identical. A refund API can implement this behavior through a scoped operation key and receiver-side deduplication.

A bulkhead reserves separate resources so one workload cannot exhaust all shared capacity. It trades some utilization flexibility for isolation. These definitions are more useful than a sequence of product names.

14. Defending a mistake or agreeing without analysis

If a counterexample exposes an error, state its consequence and repair the relevant decision. If you disagree with a suggestion, explain the requirement or evidence behind the disagreement. Collaboration does not require accepting every proposed component without analysis.

Check: preserve the parts of the design that still satisfy the requirements after a correction.

Interview strategy pitfalls

15. Solving a different problem

A read-only Q&A request does not automatically need autonomous planning or multiple agents. Explain the smallest complete baseline, then add capabilities justified by the scope. Simplicity is useful only while it satisfies the actual requirements.

Check: remove one component and explain which requirement fails. If none does, investigate whether it belongs.

16. Spending the entire session on one deep dive

Use the expected interview length as a planning constraint, while following the interviewer's direction. Reserve room to explain evaluation, recovery, economics and closing decisions. If time becomes short, summarize the remaining risks explicitly rather than claiming they are solved.

Check: can you give a two-minute closing assessment of the design already drawn?

17. Drawing decorative boxes without contracts

Label arrows with the request, event or data they carry. Show state ownership, trust boundaries and the path back to the user. Separate synchronous and asynchronous work. A diagram can be simple visually and still explain sophisticated behavior.

flowchart LR
    R[Numbered requirement] --> B[Baseline component and contract]
    B --> F[Specific failure under a changed condition]
    F --> E[Evidence that locates the failure]
    E --> C[Repair and added operating cost]
    C --> V[Verify requirement and remaining limits]

Check: trace a timeout and a duplicate request using only the diagram and its contracts.

AI-specific pitfalls

18. Treating inference as a uniform black box

At a sustained 40 output tokens/second after the first token, the remaining 199 tokens of a 200-token answer take about 4.975 seconds, in addition to the first-token delay. This is a simplified constant-rate example; real output intervals vary. Define your rate boundary before doing the arithmetic.

See inference fundamentals, KV cache and batching.

19. Treating fluency, citations or confidence as truth

A fluent answer can be wrong. A citation can point to an irrelevant, stale or contradictory passage. A model's stated confidence is not automatically a calibrated probability of correctness.

Use task-specific evidence checks, suitable human review and abstention/escalation behavior. Evaluate these mechanisms rather than promising zero hallucinations. A faithfulness score can pass an answer grounded in an incorrect source. See capability assessment.

20. Leaving security to a final content filter

Authorization, data handling and action permissions belong throughout the architecture. Delimiters and injection detectors can help, but they are not sufficient isolation mechanisms. Tool access, network access and secret exposure determine the consequence of a failure. OWASP prevention guidance.

Check: retrieved content that says “send this file outside the company” must not gain authority from being included in the prompt. See LLM security.

Recovery distinctions worth memorizing

A worker can crash after a payment commits but before its receipt is saved. Durable execution alone cannot resolve that ambiguity. Reuse the logical operation identity and reconcile with the receiver; distinguish confirmed success, confirmed failure and unknown outcome. See durable execution.

Manager and technical-lead pitfalls

Keep technical depth while explaining delivery:

1. Name source, application, evaluation, security and incident owners.
2. Define contracts and service expectations for dependencies owned by other teams.
3. Stage work around uncertainty: baseline, critical feasibility test, limited rollout, expansion.
4. State what will be narrowed, delayed or stopped if quality or economics fail.
5. Describe delegation, coaching and decisions accurately; writing the hardest code is not the only evidence of leadership.

A source owner validates policy content; engineering owns its propagation and enforcement. A release owner makes the exposure decision with evidence. None of these labels excuses an undefined handoff.

Five-minute post-mock review

1. Write the user outcome and three deciding requirements.
2. Trace one complete request and one data update.
3. Identify one severe failure and its detection/recovery path.
4. Compare your main choice with a feasible alternative.
5. Check one calculation, including units and missing costs.
6. State the largest unresolved assumption and who would validate it.

Score each as missing, mentioned or defended with evidence. This is a local practice aid. Then double the load or change a permission and retry the affected explanation.

Interview questions with developed answers

Final summary and notes

• Define the task before selecting products.
• Preserve data meaning, versions and permissions throughout the pipeline.
• Measure the complete outcome, including failures and uncertainty.
• Separate correctness, authorization, durability and protocol success.
• Count human and operating work when comparing cost.
• Use numbered lists for requirements, tables for choices and labeled diagrams for flows.
• Finish with the main compromise, owner and next validation.

Continue with answer frameworks and whiteboard exercises.

A system design diagram explains how a defined user outcome is produced under stated constraints. Use these nine authored exercises to practice the full interview: requirements → baseline → flaws → detailed design → cost/benefit → closing remarks.

All workload numbers, targets, staffing assumptions and financial worksheets below are hypothetical unless a source is explicitly cited. They are inputs to reason about, not benchmark results or reports of actual company deployments. Valued human time is an economic cost; releasing capacity does not automatically reduce payroll or produce cash savings. The proposed design is one defensible option; revise it when the interviewer changes a requirement.

How to run a mock

1. Read the prompt and state the key assumptions before reading the worked sections.
2. Write functional and nonfunctional requirements as separate numbered lists.
3. Trace data preparation and one live request through a complete baseline.
4. Find the dominant failure, then add the minimum justified repair.
5. Work through scale, quality, recovery and the complete cost comparison.
6. Close with the main compromise, release evidence and remaining uncertainty.

Use a 35–45 minute practice session if that matches the interview you are preparing for. The duration and this guide's self-checks are rehearsal aids, not employer scoring rules. Label arrows with the information they carry, separate synchronous/asynchronous work, and mark state ownership and trust boundaries.

For concepts behind these designs, use RAG, access control, capability assessment, reliability and current cost accounting.

Exercise 1: Enterprise RAG System

Prompt: Design a knowledge assistant for 50,000 employees and 10 million continuously changing documents from SharePoint, Confluence, Google Drive, and internal wikis. Enforce document permissions at query time, support English, Spanish, and Mandarin, and target a complete response in under three seconds for 95% of queries.

Functional requirements

1. Answer employee questions with evidence from the permitted document corpus.
2. Ingest source creations, edits, permission changes and deletions.
3. Provide passage-level citations and an explicit limitation when evidence is insufficient.
4. Support English, Spanish and Mandarin, including evidence in a different language from the question.
5. Keep the initial release read-only; do not change HR or business records.

Nonfunctional requirements

1. Serve 50,000 employees and a corpus of 10 million documents; validate document-size and update distributions.
2. Target p95 complete-answer latency below three seconds under the agreed peak and output-length limits.
3. Enforce current source access before disclosure; an unavailable check cannot grant access.
4. For this exercise, assume a 15-minute normal content-update target, with separate immediate-serving policy for observed revocations and deletions.
5. Define supported-answer quality, unanswerable behavior and language slices before choosing models; report availability and stale-source responses separately.

Start with the baseline

flowchart TD
    S[Sources and ACL changes] --> I[Parse and version chunks]
    I --> X[Keyword and vector indexes]
    U[Authenticated question] --> R[Authorized retrieval]
    X --> R
    R --> C[Evidence and token budget]
    C --> G[Generate and verify citations]
    G --> A[Answer or abstain]

Scale and initial state contracts

Assume 50,000 employees, 20% daily adoption, and five questions per active employee: 50,000 × 0.20 × 5 = 50,000 requests/day. Over eight hours, that is 50,000 / 28,800 = 1.74 requests/s; a fivefold peak is 8.68/s. At 6,000 input and 400 output tokens per request, the peak demand is about 52,083 input and 3,472 output tokens/s. This is a capacity input; benchmark the selected serving stack before converting it into GPUs.

An index record might be {doc_id: "leave", revision: 19, chunk: 4, acl_version: 8, deleted: false}. When revision 20 arrives, stage its chunks, validate completeness, and switch the active revision. At request time enforce current access, including cached answers. An old citation must not resurrect a deleted or forbidden source. Ten million documents at an assumed ten chunks each means 100 million chunks; storage and freshness are separate from request sizing.

Make the RAG latency and access checks concrete

For the three-second complete-response target, an illustrative planning allocation is 50 ms identity/permissions, 100 ms embedding, 100 ms retrieval, 150 ms reranking, 1,500 ms generation, and 100 ms network/serialization: 2,000 ms plus 1,000 ms of contingency. These are stage allocations, not measured stage p95 values whose sum proves the overall p95. Output length, queueing, and cache misses can consume the headroom. Measure the complete path under load; time to first token is a separate experience metric.

The retrieval predicate is same_tenant AND current_version AND not_deleted AND (tenant_public OR explicitly_granted_user OR permitted_group). The server obtains tenant, user, and group membership from trusted identity and policy state. Apply the predicate in both keyword and vector retrieval, then revalidate before returning evidence or a cached answer. “Public” here means public inside the authorized tenant unless a separate cross-tenant publication policy exists. A model-supplied tenant ID cannot broaden this scope. Test permission revocation as well as initial access.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    S[Source APIs and change cursors] -->|edition and permission events| Q[Durable ingestion queue]
    Q -->|source revision| W[Parser and completeness checks]
    W -->|versioned chunks| X[Lexical and vector indexes]
    S -->|current edition and tombstones| C[Revision and access catalog]
    U[Employee] -->|authenticated question| G[Admission and deadline]
    G --> R[Authorized retrieval]
    C -->|current policy| R
    X -->|candidates| R
    R -->|permitted passages| P[Optional rerank and evidence packing]
    P --> M[Approved model deployment]
    M --> V[Evidence and citation checks]
    C -->|revalidate dependencies| V
    V --> O[Answer or explicit limitation]
    W --> T[Restricted traces and freshness metrics]
    V --> T

Use durable source cursors plus reconciliation appropriate to each source API; a notification alone need not contain the full change. A language preference is a ranking choice, not a reason to exclude the only authoritative English passage from a Spanish question. Validate multilingual retrieval and answer support.

If a source cannot provide the promised permission freshness, make that limitation explicit and change the access architecture or scope. A five-minute cached group list cannot support a zero-staleness access promise. Check access before evidence reaches the model and before a saved or cached answer is disclosed.

Full cost and benefit

At 50,000 questions per workday and 22 workdays, compare 1.1 million monthly questions. These are illustrative full incremental budgets, with a human escalation taking one minute at $60/hour ($1 each).

The $12,800/month projected saving depends on the escalation reduction and comparable accepted outcomes. The candidate's non-escalation cost is $35,200; its break-even escalation rate is about 4.16%. Validate source/index rebuild costs at this corpus size and count any different employee handling time before claiming savings.

Closing remarks

Start with one owned corpus and a read-only evidence path. The dominant risk is stale or unauthorized evidence, not choosing a fashionable database. Expand after quality, freshness, access, latency and escalation-cost measurements meet the agreed criteria.

Interviewer changes the requirement: a user's access is revoked while their previous answer remains cached. Trace how you prevent exposure from index, cache, memory, and saved artifacts. Name the owner and measurable revocation policy.

Continue the deep dive: Complete enterprise RAG interview.

Exercise 2: Customer Support Chatbot

Prompt: Design e-commerce support for 10,000 conversations/day, a one-million-product catalog, order history, and FAQs, with three languages and integration into the existing Zendesk workflow. The business proposes resolving 70% of tickets without handoff; define correct resolution and recontact before accepting that target. Support order lookup, product questions, and eligible returns/refunds, with approval for policy exceptions.

Functional requirements

1. Answer product and policy questions across the three required languages.
2. Look up orders through authenticated customer-scoped tools.
3. Create eligible return/refund proposals and execute only permitted actions.
4. Obtain approval for exact policy exceptions and preserve the decision record.
5. Handoff to the existing Zendesk workflow with context, status and unresolved effects; support a customer's request for human help.

Nonfunctional requirements

1. Support 10,000 conversations/day and a one-million-product catalog; size turns and tool calls separately.
2. Treat 70% no-handoff as a proposed business target, conditional on correct resolution and measured later recontact.
3. Prevent cross-customer access and refunds beyond the remaining refundable balance.
4. For this exercise, target p95 read-only answers below four seconds; represent slow actions asynchronously with visible status.
5. Preserve operation identity across retries and recover known or unknown action outcomes after worker failure.

Start with the baseline

flowchart TD
    U[Identity and request] --> P[Policy and order lookup]
    P --> V[Validate exact proposed refund]
    V --> H{Approval needed}
    H -->|Yes| W[Durable approval wait]
    W --> R[Revalidate and execute with operation ID]
    H -->|No| R
    R --> C[Confirm or reconcile receipt]

Scale and initial state contracts

At 10,000 conversations/day, a 70% no-handoff target leaves 3,000 human-handled conversations if all remaining cases escalate. At six minutes each, this is 300 reviewer-hours/day. At six productive review-hours per person, plan 300 / 6 = 50 daily staffed equivalents before coverage, leave, and surge allowances. If a narrower 5% subset needs a specialist, size that queue separately: 500 × 6 / 60 = 50 specialist-hours/day. Deflection is not the same as saved labor if difficult cases become longer.

Persist {case_id, order_id, amount_cents, currency, policy_version, approval_digest, operation_id, status}. A timeout after submitting the refund moves SUBMITTED → UNKNOWN, not FAILED → NEW_REFUND. Query the receiver using the same operation ID; revalidate approval if amount or policy changes. Recall: propose, authorize, persist, execute, reconcile.

Validate a proposed refund before execution

A tool proposal can contain {order_id: "o17", amount_cents: 40000, currency: "USD"}. A strict schema requires an integer amount, known currency format, and no unexpected fields. The service then checks the authenticated principal's ownership, the order's currency, refundable balance, policy version, and required approval. Neither a valid schema nor a model-provided approved: true establishes authority. The server creates or retrieves the stable business operation ID and persists it before sending the payment request.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    U[Customer or ticket webhook] -->|identity and event ID| G[Authenticated intake and deduplication]
    G --> R[Intent and bounded workflow router]
    R -->|policy question| K[Authorized catalog and policy retrieval]
    K --> A[Evidence-grounded answer]
    R -->|order operation| O[Scoped order service]
    O --> P[Typed return or refund proposal]
    P --> V[Ownership, balance and policy checks]
    V --> H{Exact approval required?}
    H -->|yes| W[Durable approval wait]
    H -->|no| J[Persist business operation]
    W -->|revalidate approved digest| J
    J -->|same operation key| F[Refund receiver]
    F -->|receipt or uncertain result| C[Reconciler and status store]
    C -->|confirmed outcome| A
    R -->|request, exception or no progress| T[Human queue with full handoff]
    C -->|unresolved| T

Bind an approval to customer, order, currency, amount, policy/version and expiry. Recheck the order and refundable balance immediately before execution. Concurrent refunds require an atomic reservation or equivalent receiver-enforced balance rule; checking a balance in two independent requests is insufficient.

A ticket webhook may be delivered more than once. Deduplicate its event and the business action independently. Do not close the case while a consequential operation remains unknown. Sentiment or model self-confidence can inform routing, but neither alone proves that an action is allowed or a case is resolved.

Full cost and benefit

The earlier capacity calculation assumes six minutes per escalated case. Stress the economics with eight minutes for the harder remaining cases, at $60/hour.

The projected saving is $34,000/day, subject to verified resolution, recontact and staffing outcomes. The candidate requires 400 productive human-hours/day, or about 67 staffed equivalents at six productive hours each, before coverage allowances. The specialist subset is part of this total unless explicitly additional. Deflection alone cannot substantiate the saving.

Closing remarks

Separate informational answers from state-changing actions. Launch a narrow authorized workflow, reconcile payment uncertainty, and expand only when correct resolution and full human workload support the business target.

Interviewer changes the requirement: the payment API times out after it may have committed. Distinguish unknown from failed. Show reconciliation rather than a fresh refund request. Measure correct resolution and recontact, not only chatbot deflection.

Continue the deep dive: Complete support-automation interview.

Exercise 3: Code Review Assistant

Prompt: Review 50,000 pull requests/day across GitHub/GitLab, respect repository conventions, and provide specific, actionable feedback and suggested fixes without disrupting developers.

Functional requirements

1. Review pull-request diffs and relevant repository context after GitHub/GitLab events.
2. Produce specific findings with file/line, consequence, severity and supporting evidence.
3. Respect repository conventions without treating repository text as new authority.
4. Suggest patches and report which checks actually ran.
5. Supersede stale findings when the PR head changes; keep publishing, merging and deployment as separate permissions.

Nonfunctional requirements

1. Handle 50,000 PRs/day, with measured size and burst distributions.
2. For this exercise, target p95 completed review below two minutes for the supported PR-size class; show status for larger work.
3. Run untrusted code/configuration in appropriately isolated workers without production credentials.
4. Define finding precision, severe-defect recall and developer review-time objectives before rollout.
5. Prevent duplicate or stale comments and preserve enough versioned evidence to reproduce a finding.

Start with the baseline

flowchart TD
    E[PR event and commit SHA] --> Q[Deduplicated queue]
    Q --> S[Isolated checkout and static checks]
    S --> M[Diff and scoped context to model]
    M --> V[Verify findings and severity]
    V --> H[Human review of suggestions]

Scale and initial state contracts

At 50,000 PRs/day, assume 20% require model review and two model calls per reviewed PR: 20,000 calls/day. At 10,000 input plus 1,000 output tokens per call, that is 200 million input and 20 million output tokens/day. If each selected PR occupies a sandbox for 90 seconds, the mean concurrent sandboxes over 24 hours is 10,000 × 90 / 86,400 = 10.42; a sixfold burst requires about 63 before headroom. Peak webhook bursts and repository sizes need measurement.

Key work by (repository_id, head_sha, check_version). Before posting, verify the PR still has that head SHA. A finding includes file, lines, reason, evidence, severity, and test status. An unavailable test runner is UNAVAILABLE, never PASS. Running repository scripts requires isolation and a deliberate network/secret policy.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    E[PR webhook] -->|repo, head SHA, event ID| Q[Deduplicated job queue]
    Q --> S[Isolated checkout and bounded checks]
    S -->|diff, contracts and test status| C[Scoped context builder]
    C --> M[Model proposes findings]
    M --> V[Evidence, severity and duplicate checks]
    V --> H{PR head still matches?}
    H -->|yes| P[Scoped comment publisher]
    H -->|no| N[Supersede and schedule current revision]
    P --> U[Developer review]
    U -->|accepted, rejected or unclear| F[Adjudicated evaluation data]
    S --> T[Job trace and resource accounting]
    V --> T

Store the logical job identity, attempt lease, head SHA, check/model/prompt versions and finding IDs. A stale worker cannot publish after a newer attempt has superseded it. API retries need idempotent comment publication or reconciliation of an uncertain post, not simply another comment.

A suggested patch is untrusted until checked. Label tests as passed, failed, unavailable or not run. Do not infer that a renamed public field is safe from a passing unit suite that never exercises consumers. Source comments can express coding conventions, but cannot grant access to secrets or authorize a merge.

Full cost and benefit

Compare all 50,000 daily PRs, with human time valued at $60/hour. The baseline averages five minutes of review. The candidate averages 4.5 minutes of ordinary review plus separate investigation of 2,000 AI comments at two minutes each; this extra time is not included in the 4.5-minute figure.

The projected $20,000/day reduction requires a measured time saving with no unacceptable increase in missed defects. If ordinary review stays at five minutes, the candidate instead costs $256,000/day. A noisy reviewer can be a net burden despite inexpensive inference.

Closing remarks

Start with high-confidence, evidence-backed findings on a bounded PR class. Treat repository content and code as untrusted inputs, bind results to the current commit, and measure developer time and defect outcomes before expanding.

Interviewer changes the requirement: tests pass but a public API contract changed. Explain additional contract evidence and human review. Define whether the assistant can run builds, access secrets, or publish changes.

Continue the deep dive: Code-assistant design and autonomous-coding recovery.

Exercise 4: Document Processing Pipeline

Prompt: Process 100,000 financial-services documents/day: invoices, contracts, and forms, including PDFs, scans, and handwritten notes. Leadership asks for 99% extraction accuracy; define the field/document denominator and critical-field acceptance criteria. Provide traceable evidence and human review for uncertainty. Clarify the requested SOC 2 assurance scope and whether any health-data processing actually brings HIPAA obligations into scope.

Functional requirements

1. Accept supported invoices, contracts and forms, including scans and handwriting within a declared quality range.
2. Extract typed fields with source page/region evidence.
3. Validate field relationships and business identifiers without guessing missing values.
4. Route uncertain or inconsistent results to an authorized reviewer.
5. Track processing, review and downstream posting separately; prevent duplicate business records.

Nonfunctional requirements

1. Process 100,000 documents/day; assume ten pages/document for initial sizing and test the actual distribution.
2. Define the proposed 99% accuracy target by field type and whole-document correctness; critical fields need their own acceptance criteria.
3. For this exercise, target p95 automated draft completion within five minutes for documents of at most 20 pages; review has a separate service target.
4. Restrict source and extracted data by tenant and role; apply the actual retention and assurance requirements.
5. Bound queues and retries, preserve progress across crashes, and expose incomplete pages instead of marking the document complete.

Start with the baseline

flowchart TD
    U[Validated upload] --> Q[Job queue and content hash]
    Q --> O[OCR and layout]
    O --> E[Typed extraction with page evidence]
    E --> V{Arithmetic and evidence checks}
    V -->|Pass| A[Accepted extraction]
    V -->|Uncertain| H[Reviewer queue]
    H --> A

Scale and initial state contracts

Assume 100,000 documents/day and ten pages/document: one million pages/day. At an illustrative two worker-seconds/page, work totals two million worker-seconds/day, or 23.15 continuously busy worker equivalents. Targeting 70% utilization gives 23.15 / 0.70 = 33.1, so at least 34 comparable workers for average load, then add measured peak/deadline capacity. If 2% of documents need four minutes of review, plan 2,000 × 4 / 60 = 133.3 reviewer-hours/day separately.

Represent a field as {amount_minor: 125000, currency: "USD", scale: 2, page: 2, bbox: [40, 90, 180, 120], source_hash, status: "needs_review"}. A document can be extracted but not approved. A repeated upload can reuse parsing only when its tenant, source bytes, parser version, and retention policy allow it; a content hash alone is not permission.

Validate relationships as well as individual fields

Suppose line items total 100,000 cents, the documented discount is 5,000 cents, and tax is 7,600 cents. Under an invoice rule with no other charges, the expected total is 100000 − 5000 + 7600 = 102600 cents. An extracted total of 120,600 cents fails this check even if every field has a high extraction confidence. Use decimal arithmetic or integer minor units with the currency's correct scale. Confirm discounts, shipping, tax basis, and rounding from the document before applying the equation; a missing charge should route to review rather than be “corrected” by guessing. Show the source regions for the conflicting values.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    U[Authenticated upload] -->|validated bytes and tenant| O[Private immutable source object]
    O --> J[Durable document job and revision]
    J --> Q[Bounded page-work queue]
    Q --> P[Parser, OCR and layout workers]
    P -->|page results and failures| A[Complete-page aggregation]
    A --> E[Typed extraction with source regions]
    E --> V[Schema, arithmetic and business checks]
    V -->|eligible auto-accept| D[Accepted extraction version]
    V -->|uncertain or conflicting| H[Authorized review queue]
    H -->|reviewed exact version| D
    D --> B[Separately authorized business posting]
    B --> R[Receipt or reconciliation state]
    J --> T[Progress, queue age and audit metrics]
    H --> T

Bind review to the source digest and extraction revision. A corrected extraction invalidates an earlier approval. Represent money using integer minor units or decimal values with the currency's scale; document bounding boxes must declare the coordinate system and page dimensions.

A parser may return partial pages. Mark missing/unreadable pages explicitly and avoid claiming a complete contract extraction. Parsing reuse is scoped to authorized source bytes and parser configuration; a matching content hash alone cannot authorize access or identify a duplicate invoice business transaction.

For illustration only, if 100 required fields each independently had 99% correctness, whole-document correctness would be 0.99^100 ≈ 36.6%. Real field errors are often correlated, so measure complete-document accuracy directly. Financial-services documents are not automatically subject to HIPAA, and SOC 2 assurance is not obtained simply by selecting encryption algorithms.

Full cost and benefit

At $60/hour, a four-minute review costs $4. Compare the same 100,000 daily documents with a current parser/review baseline and an additional extraction stage.

The projected $25,000/day saving depends on validated auto-accept quality and review time. At tenfold volume with unchanged page work, average-load capacity at 70% utilization rises to at least 331 comparable page workers; review work at 2% becomes about 1,333 hours/day, before coverage. Better model confidence alone does not eliminate that staffing requirement.

Closing remarks

Optimize for a correct, traceable business record, not merely parsed JSON. Keep extraction acceptance separate from posting, make partial failures visible, and size the reviewer queue as deliberately as the OCR workers.

Interviewer changes the requirement: volume rises tenfold and reviewers are saturated. Prioritize queues, protect high-impact fields, and define latency/quality tradeoffs. Do not simply lower review thresholds without measuring harm.

Continue the deep dive: Document-intelligence interview.

Exercise 5: Real-Time Content Moderation

Prompt: Moderate one million posts/day across text, images, and video in ten languages. Detect policy categories such as hate, violence, adult content, and spam; support appeals. Posts should become visible within 500 ms when permitted. Define the percentile, temporary treatment of uncertain posts, and whether every media analysis must finish before publication.

Functional requirements

1. Evaluate text, image and video posts against versioned policy categories in ten languages.
2. Record an allow, restrict, pending-review or other explicitly defined policy action.
3. Provide slower analysis for cases outside the fast path without hiding their temporary state.
4. Support appeals and authorized correction of earlier decisions.
5. Preserve the content revision and evidence associated with enforcement.

Nonfunctional requirements

1. Handle one million posts/day; size video duration and media bytes separately from post count.
2. For this exercise, target p99 initial decision/state below 500 ms; content needing deep review remains explicitly pending under the agreed visibility policy.
3. Define category/language precision and recall with false-positive and false-negative consequences.
4. Report the pending fraction and final-decision delay so moving work off the fast path cannot disguise poor completion.
5. Bound review queues, protect reviewer access/wellbeing and preserve appeal/reversal history.

Start with the baseline

flowchart TD
    U[Content and policy version] --> F[Fast deterministic and classifier checks]
    F --> D{Confident decision}
    D -->|Yes| A[Durable allow or restrict action]
    D -->|No| Q[Deep model or reviewer queue]
    Q --> A
    A --> P[Appeal and outcome feedback]

Scale and initial state contracts

For one million items/day at 0.1% harmful prevalence, there are 1,000 harmful and 999,000 benign items. A detector with 90% recall and 1% false-positive rate produces 900 true positives and 9,990 false positives. Precision is 900 / 10,890 = 8.26%. This explains why impressive recall can still overwhelm review. Do not confuse false-positive rate with the fraction of flagged items that are wrong.

A decision record binds content_revision, policy_version, category, evidence, reviewer/model version, and action_id. If the content changes before enforcement, reconsider the decision. For the 500 ms publication budget, measure stage tails and reserve time for the response; a slow deep review may need a provisional restricted state rather than blocking the request indefinitely.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    U[Post and media upload] -->|content revision| S[Private staging and admission]
    S --> F[Fast rules and supported classifiers]
    F --> D{Policy action determined?}
    D -->|yes| A[Version-bound enforcement record]
    D -->|no| P[Explicit pending visibility state]
    P --> Q[Bounded deep-analysis queue]
    Q --> M[Media analysis and policy evidence]
    M --> H[Reviewer or validated decision rule]
    H --> A
    A --> V[Visible or restricted post]
    V -->|appeal| R[Independent reassessment]
    R -->|new versioned decision| A
    R --> L[Adjudicated labels, not automatic ground truth]
    F --> T[Latency, slices, prevalence and queue metrics]
    H --> T

Thresholds must be measured for the policy and error costs; a score of 0.95 is not inherently a 95% probability of correctness. Fast-path allow/restrict and pending decisions need distinct metrics. A provider outage should follow the agreed temporary visibility policy, not silently allow everything.

The no-delay requirement may be incompatible with mandatory complete analysis of long video. State that conflict, bound accepted media or change the visibility/review contract. Use an atomic content-revision check when applying enforcement; a delayed result cannot automatically apply to edited content.

Full cost and benefit

Use the same hypothetical prevalence and recall from the sizing example. If a candidate reduces false-positive rate from 1% to 0.1% while retaining 90% recall, it yields 900 true positives and 999 false positives: 1,899 flags, with precision about 47.39%. This is a target scenario, not a promised threshold adjustment.

Assume every flag needs two minutes of review at $60/hour ($2), and compare the same million daily posts.

The potential $16,882/day saving comes mainly from reduced false flags. Review work falls from 363 to 63.3 hours/day. Both detectors still miss 100 harmful posts under these assumptions; decide whether that is acceptable independently of savings. Measure appeals and reviewer time rather than assuming they stay constant.

Closing remarks

Define policy and temporary visibility before selecting models. Measure error costs at the actual prevalence, bind enforcement to content versions, and treat appeals as a governed correction path.

Interviewer changes the requirement: harmful content becomes rarer and false positives become more costly. Reassess threshold economics and precision, even if recall stays constant. Keep slower deep analysis off a hard synchronous path when it cannot meet the deadline.

Continue the deep dive: Content-moderation interview.

Exercise 6: Multi-Tenant AI Platform

Prompt: Serve 500+ enterprise customers, each with its own documents and models, strict tenant data isolation, usage-based billing, and pricing tiers with different capabilities. Support interactive apps and background jobs, and produce evidence for the agreed SOC 2 control scope.

Functional requirements

1. Configure each enterprise tenant's permitted models, regions, credentials and capabilities.
2. Serve authenticated interactive requests and durable background jobs.
3. Apply tenant/user authorization to data, tools, caches, jobs and artifacts.
4. Meter usage with task/attempt attribution and produce reconciled billing records.
5. Support tier changes, revocation and administrative audit without trusting client-supplied tier labels.

Nonfunctional requirements

1. Support at least 500 enterprise customers with declared traffic and service classes.
2. Prevent cross-tenant and within-tenant unauthorized disclosure, including asynchronous execution and observability.
3. For this exercise, preserve a reserved interactive share of a shared quota while bounding background queues.
4. Keep provider/model/region hard constraints during fallback and quota exhaustion.
5. Track spend reservations and actual charges atomically; do not treat an unknown remote charge as zero.

Start with the baseline

flowchart TD
    U[Authenticated tenant] --> A[Quota and token admission]
    A --> I[Interactive reserved pool]
    A --> B[Background bounded pool]
    I --> P[Provider or runtime]
    B --> P
    P --> M[Per-tenant usage and outcomes]

Scale and initial state contracts

Assume a usable limit of 600,000 tokens/minute and 6,000 total tokens/request. The theoretical ceiling is 100 requests/minute before separate input/output or request limits. Reserve 60% for interactive traffic: 60 requests/minute at this assumed size. A background tenant asking for 10,000-token jobs cannot be treated as consuming the same capacity as a 500-token chat. Use token-aware admission and measure output uncertainty.

A scheduler item holds {tenant_id, class, estimated_tokens, deadline, retry_count}. Charge actual usage afterward; reject or defer work that cannot finish before its deadline. A failed provider must not cause each tenant to retry without a shared budget. Record retries against the originating tenant while enforcing an overall limit.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    C[Admin control plane] -->|versioned policy and entitlements| P[Tenant configuration store]
    U[User or job submitter] --> G[Authenticated gateway]
    G --> A[Authorization and atomic quota reservation]
    P --> A
    A -->|interactive| I[Reserved interactive scheduler]
    A -->|background| B[Bounded fair job scheduler]
    I --> W[Scoped provider/runtime workers]
    B --> W
    P -->|recheck model, region and permission| W
    W --> D[Permitted data and tool services]
    W -->|attempt usage and status| L[Append-only usage ledger]
    L --> R[Provider reconciliation and bill adjustments]
    W --> O[Scoped results and artifacts]
    L --> M[Tenant metrics and operational alerts]

The 600,000-token/minute example above assumes one usable combined quota for teaching. Real services may separately limit input tokens, output tokens, requests, concurrency, regions or model deployments. Enforce all applicable limits; do not divide a model's advertised quota into a fictional uniform capacity pool.

Bind usage to tenant, task, attempt, model/region/tier, rate version and disjoint billed categories. Deduplicate events by their provider/event identity and append corrections rather than silently rewriting settled bills. A queue item retains the originating tenant but must revalidate current capabilities when it executes. Model memory or prompt caches require their own documented isolation guarantees.

Full cost and benefit

For the same monthly useful workload, suppose unavoidable model/serving work costs $20,000. Compare existing retry waste with a controlled scheduler. Figures are illustrative and include incremental implementation and operations.

The projected $2,000/month saving requires more than $4,000 of avoided waste to cover the new fixed costs, while maintaining the required completed outcomes. Reserved capacity can also increase idle cost. A necessary isolation control may still be justified when it does not save money; state the requirement rather than inventing savings.

Closing remarks

Make tenant scope and entitlements enforceable across synchronous and asynchronous work. Separate admission, scheduling, execution and billing state, and use observed workload and failure behavior to justify shared versus dedicated capacity.

Interviewer changes the requirement: one tenant floods the queue during a provider outage. Show how retry budgets, bulkheads, queue limits, and reserved capacity preserve other tenants' objectives.

Continue the deep dive: Multi-tenant application interview and gateway design.

Exercise 7: Semantic Search at Scale

Prompt: Search 50 million e-commerce products at 100 million queries/day with p99 latency below 100 ms. Support price, category, brand, and rating filters, personalization from permitted user history, and real-time inventory updates. Include exact SKU and paraphrase queries in the relevance tests.

Functional requirements

1. Search 50 million products using exact identifiers and natural-language queries.
2. Apply price, category, brand and rating filters with defined semantics.
3. Rank eligible results using permitted personalization data.
4. Propagate inventory, price and description updates with versioned state.
5. Migrate embedding/index versions with evaluation, cutover and rollback.

Nonfunctional requirements

1. Support 100 million queries/day; test a provisional fivefold average peak and mixed filter selectivity.
2. Target end-to-end p99 search latency below 100 ms for the defined region/network boundary.
3. For this exercise, target normal event-to-search metadata freshness within five seconds and revalidate stock/price at purchase.
4. Preserve exact-SKU and other critical relevance slices during model changes.
5. Keep index/query encoders compatible and avoid exposing personalized results through shared caches.

Start with the baseline

flowchart TD
    D[Product source of truth] -->|versioned changes| I[Lexical index and searchable metadata]
    Q[Query and explicit filters] --> S[Identifier or keyword search]
    I --> S
    S --> V[Current eligibility checks]
    V --> O[Ranked search results]

The lexical baseline supports exact identifiers and ordinary keyword queries. Add semantic retrieval only when measured paraphrase failures justify it; the following sizing estimates that candidate.

Scale and initial state contracts

With 50 million products and an assumed two vectors per product, plan for 100 million vectors, 768 dimensions, and float32 storage. Raw vectors require 100,000,000 × 768 × 4 = 307.2 GB in decimal units. Two copies require 614.4 GB before graph edges, IDs, metadata, allocator overhead, and backups. An 8-bit representation is 76.8 GB of raw codes, but quality, index overhead, and any retained full-precision vectors still matter. Measure recall against exact search on a representative subset while tuning search effort.

Never mix vector spaces during migration. Store {embedding_model, revision, dimensions, normalization, index_generation} with the index and query encoder. Shadow queries against B, compare exact-ID and language slices, then switch an alias atomically with a rollback. RRF combines ranks, not incompatible raw BM25 and cosine scores.

Separate query throughput, freshness, and the 100 ms deadline

100,000,000 / 86,400 = 1,157.4 queries/s on average. A hypothetical fivefold peak is about 5,787 queries/s, distinct from the number of stored vectors. Load-test filtered queries at that peak, including cold queries and inventory churn. One 100 ms planning budget is 5 ms admission/cache lookup, 15 ms query encoding, 30 ms parallel keyword/vector retrieval, 15 ms fusion/filter validation, 10 ms personalization, and 25 ms response/network. Concurrent retrieval paths contribute their critical-path duration, not their sum. This allocation leaves no spare time: measured tails may require caching, a faster encoder, less reranking, or a changed requirement. Verify the end-to-end p99 instead of adding component percentiles.

Inventory and price updates can update authoritative metadata without recomputing an unchanged description embedding. Description changes need new embeddings. Stream versioned updates, measure event-to-query freshness, and check authoritative stock before checkout. Personalization may reorder eligible results; it must not override price filters, current availability, or permission checks.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    D[Product source of truth] -->|versioned changes| Q[Update stream and replay cursor]
    Q -->|price and inventory| C[Current searchable metadata]
    Q -->|description changed| E[Versioned embedding workers]
    E --> A[Index A and encoder contract]
    E --> B[Staged index B and encoder contract]
    U[Query and permitted user context] --> G[Admission and scope-aware cache]
    G --> R[Release router: matching encoder and index]
    R --> K[Parallel lexical and vector retrieval]
    A --> K
    B --> K
    C -->|filter and freshness checks| K
    K --> F[Rank fusion and bounded personalization]
    F --> V[Eligible result validation]
    V --> O[Search results]
    R --> T[Isolated shadow comparison and slice metrics]

The latency allocation in the sizing section spends the full 100 ms. A revised planning allocation might be 5 ms admission, 10 ms encoding, 25 ms parallel retrieval, 10 ms fusion/filtering, 10 ms personalization and 20 ms serialization/network: 80 ms, leaving 20 ms of planning headroom. This is still not measured p99 performance. Validate cold queries, selective filters, churn and regional network delay.

Partitioning only by category can create hot shards or miss cross-category intent. Choose partition/routing behavior using the distribution and query semantics. A description edit re-embeds affected products; it does not inherently require rebuilding every product. During a whole-model migration, replay updates to both versions through cutover and preserve a rollback-ready version.

Two full index versions each with two replicas require 1.2288 TB of raw float32 vectors under the example's dimensions, before auxiliary data. The earlier two-copy estimate is not a complete replicated migration footprint.

Full cost and benefit

A better search experience may cost more. Compare the same 100 million daily queries with a lexical baseline and a candidate hybrid design.

The candidate costs $1,800/day more. If a verified incremental purchase contributes $6 after its non-search variable costs, the candidate needs 300 additional purchases/day to cover that increase. Measure causal purchase contribution in an appropriate experiment; clicks or a benchmark gain alone do not establish it. Include temporary duplicate-index capacity in the migration budget.

Closing remarks

Preserve exact-match behavior, current eligibility and the tail-latency contract. Use compatible index releases and measured relevance/business outcomes to justify the extra serving and migration cost.

Interviewer changes the requirement: an embedding upgrade improves a benchmark but harms exact SKU lookup. Compare slices and hybrid retrieval before replacing everything. Design a dual-index migration and rollback.

Continue the deep dive: Recommendation and retrieval design and production retrieval.

Exercise 8: Evaluation Pipeline for a Production LLM Product

Prompt: Support an assistant with 50,000 daily users and weekly model/prompt releases without unacceptable quality regressions. Design offline evaluation, CI gates, judge calibration, and production monitoring within a proposed evaluation budget of 2% of inference spend. State whether that budget includes human review and engineering.

Functional requirements

1. Evaluate an exact candidate release against a versioned baseline and datasets.
2. Run deterministic checks and appropriate semantic/human grading.
3. Produce per-case outcomes, slice summaries and a reproducible gate decision.
4. Calibrate judges and retain protected holdouts separate from development iteration.
5. Connect offline evidence to isolated shadowing, canaries, production outcomes and rollback.

Nonfunctional requirements

1. Support an assistant with 50,000 daily users and weekly release decisions.
2. Keep candidate execution isolated from protected labels and production write capabilities.
3. Preserve all scheduled outcomes, including execution failures and unknown grades; a missing result cannot silently pass.
4. Apply hard requirements separately from average quality, and report uncertainty using the actual sampling/repetition structure.
5. Treat 2% of inference spend as a proposed budget; define whether it covers human review, operations and engineering before claiming feasibility.

Start with the baseline

flowchart TD
    C[Candidate release manifest] --> T[Deterministic contracts]
    T --> D[Versioned labeled cases]
    D --> J[Calibrated grading and paired comparison]
    J --> G{Hard gates and quality bounds}
    G -->|Pass| N[Canary with rollback]
    G -->|Fail| F[Failure analysis and new regression cases]

Scale and initial state contracts

Assume 2,000 cases, two configurations, and three repetitions: 12,000 generations. At a hypothetical $0.004/generation and one $0.001 judge call per generation, the run costs $48 + $12 = $60, before labeling and infrastructure. Twenty percent expert review at 30 seconds/result adds 20 human-hours, often the dominant expense. Repetitions of one case are correlated; do not treat 12,000 generations as 12,000 independent user scenarios.

Persist {case_id, dataset_hash, candidate_digest, output, rubric_version, grade, severity, grader_version}. Block on a reproduced unauthorized action even when the mean improves. Use paired cases for quality comparisons and keep a protected holdout; inspect failures on development data without repeatedly selecting against the final test set.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    C[Exact candidate and baseline manifests] --> O[Evaluation orchestrator]
    D[Versioned cases without protected labels] --> O
    O --> Q[Bounded isolated execution queue]
    Q --> W[Candidate and baseline workers]
    W --> R[Immutable outputs and execution statuses]
    L[Restricted references and rubrics] --> G[Deterministic or calibrated semantic grading]
    R --> G
    G --> H[Human disagreement and critical-case review]
    G --> A[Complete aggregation and paired analysis]
    H --> A
    A --> P{Evidence meets all release criteria?}
    P -->|yes| V[Approved limited canary]
    P -->|no or inconclusive| F[Failure analysis and new candidate]
    V --> M[Live outcomes, stop conditions and rollback]

Bind the decision to code, model/endpoint, prompt, tools, retrieval, policy, dataset and grader versions. A passing report for a nearby commit is not approval for a changed release. Reconcile the expected case/attempt manifest with actual results and reject duplicates, foreign run IDs or missing evidence.

Binary criteria can be easier to operationalize for specific failures, but are not inherently unbiased or perfectly reproducible. Ordinal rubrics can also be useful when defined and validated. User thumbs, repeated queries and citation clicks are behavioral signals; they are not automatic correctness labels.

Keep development failures available for diagnosis while limiting selection against the protected final holdout. A canary or shadow that can issue real external writes needs explicit isolation or authorized effect controls; duplicating live requests is not automatically harmless.

Full cost and benefit

Complete the run-cost example above: 12,000 generations and judge calls cost $60, and 20% human review at 30 seconds/result requires 20 hours. At $60/hour, plus $15 orchestration/storage, that is $1,275/run.

Assume four release runs/month, $100,000/month inference spend, $500 evaluation maintenance and $500 production monitoring. These are illustrative budgets.

Neither plan meets the proposed $2,000/month ceiling. The lower-review design saves $3,600 but is acceptable only if its calibrated sampling still supplies the required evidence; critical cases may need additional review. Negotiate budget, scope or cadence. Do not silently remove required checks or omit labor to report compliance with 2%.

Closing remarks

Design the decision and evidence contract first. Version the complete experiment, preserve missing outcomes and critical failures, and show explicitly when the requested budget cannot support the needed confidence.

Interviewer changes the requirement: average quality rises while a rare high-severity failure worsens. Explain why the average cannot override the hard gate, who decides, and what evidence is needed next.

Continue the deep dive: Complete evaluation-gated CI/CD interview.

Exercise 9: Memory and State for a Long-Running Agent

Prompt: Support one million users whose assistant remembers selected facts and preferences over months, with sessions that may last hours. Design relevant recall, correction, forgetting, and per-user isolation, alongside execution state that can wait for approvals and resume across days.

Functional requirements

1. Retain selected user facts/preferences and relevant episode summaries for later sessions.
2. Retrieve useful memories under current user/tenant scope.
3. Correct, supersede and delete memories and derived artifacts.
4. Keep durable task progress, approval state and operation receipts separate from recalled facts.
5. Resume long-running work across days with compatible state and current authorization.

Nonfunctional requirements

1. Support one million users with declared active-user and session distributions.
2. For this exercise, target p95 memory retrieval below 150 ms for the defined per-user record limit; model time is separate.
3. Prevent memory poisoning from promoting untrusted content into authority.
4. Before confirmed deletion completes, establish how current-serving state, derived records and in-flight work are invalidated; storage/backups follow an explicit retention/deletion policy.
5. Bound per-call context, memory writes and background jobs; evaluate false, stale and missing recall separately.

Start with the baseline

flowchart TD
    R[Recent raw turns] --> S[Episode summaries with provenance]
    S --> F[Approved durable facts]
    F --> C[Relevance and current-access checks]
    C --> P[Bounded prompt]
    W[Workflow checkpoints and operation receipts] --> P
    D[Deletion or permission event] --> F
    D --> S
    D --> R

Scale and initial state contracts

Assume a 16,000-token context budget: 2,000 instructions/tools, 3,000 recent conversation, 5,000 retrieved evidence, 1,000 memory, 3,000 reserved output, and 2,000 safety margin. These sum to 16,000. When evidence grows, select and compress with provenance instead of silently truncating the newest user constraint. Durable storage capacity is a different budget: one million users with 20 approved 500-byte facts each is 10 GB raw, before indexes, history, and backups.

A fact has {subject, value, source_event, valid_from, expires_at, consent_scope, tombstone_version}. A checkpoint says where execution resumes; a fact says what may be recalled. On resume, recheck deletion and permissions against current stores. A summary is a lossy view of evidence and must not override a later correction or resurrect a deleted preference.

Find flaws and compare repairs

Detailed design and recovery

flowchart TD
    U[User events and permitted observations] --> P[Provenance and purpose checks]
    P --> C[Candidate extraction and episode summary]
    C --> V[Conflict, authority and retention validation]
    V --> F[Versioned approved facts and summaries]
    Q[Authenticated current task] --> R[Scoped relevant recall]
    F --> R
    R --> B[Context budget and evidence packing]
    B --> M[Model invocation]
    D[Correction or deletion request] --> E[Authoritative epoch and tombstones]
    E --> F
    E --> R
    E --> I[Invalidate derived caches and affected in-flight work]
    W[Durable workflow, approvals and receipts] --> A[Resume with current-policy validation]
    E --> A
    A --> B
    M --> O[Validate result dependencies before disclosure or action]

Working, episodic, semantic and procedural memory are useful conceptual categories, not a universal L1–L4 architecture. Semantic memory may contain beliefs or claims; storage does not make them true. A model's context can contain selected working state while the complete task record remains external.

Valid time records when a fact applies in the domain. Recorded/transaction time records when the system learned or stored it. valid_from and valid_to alone describe valid-time history, not a complete bitemporal model. Preserve both histories when the application needs that distinction. See long-term memory.

A deletion must invalidate derived summaries, embeddings and caches as required, not merely remove one visible fact row. A paused task rechecks current state. An in-flight provider request cannot be made to “forget” by deleting a local row: cancel where supported, prevent stale disclosure/actions, and apply the provider's actual retention contract. An external payment receipt remains business execution state rather than a removable preference.

Full cost and benefit

Assume 10% of the million users have one session/day: 100,000 sessions/day or 3 million per 30-day month. Compare a permitted full-history baseline with selective memory, assuming equal accepted task outcomes.

The projected saving is $11,000/month, not the $24,000 reduction in context calls alone. Measure changed false recall, correction workload and missed information. The full-history baseline is eligible only when retaining and sending that history is permitted; an ineligible privacy design cannot win on price.

Closing remarks

Retain information for an explicit purpose, with provenance and correction/deletion semantics. Keep factual recall separate from execution authority, and validate current state on every resumed action and affected result.

Interviewer changes the requirement: a user deletes a remembered preference while a workflow is paused. Trace deletion and revalidation. Explain why restoring a checkpoint must not restore revoked authority or undo an external action.

Continue the deep dive: Memory architectures, state management and durable execution.

Add the leadership round

For any exercise, assume a four-engineer team as a rehearsal constraint, then explain:

1. Which useful scope the team delivers first.
2. Which uncertainty the first experiment resolves.
3. Who owns source correctness, evaluation labels, interfaces and incidents.
4. What is purchased, built, deferred or removed.
5. What evidence permits expansion and what result would stop the project.

The staffing number is not a claim that every system needs four people. Tie the plan to actual skills, dependencies and operating work. For individual-contributor interviews, keep your design technically concrete without inventing management responsibility.

Final summary and notes

Change one constraint on your second attempt: latency, data freshness, permissions, budget, reviewer availability or workload mix. Explain which decisions change and which remain valid. Use answer frameworks for the conversation structure and the question bank for further practice.

Behavioral interviews ask for evidence from past experience. Explain the situation, your responsibility, the action you took and the result. Match the depth to the role; individual contribution, technical leadership and people management require different evidence.

Remember: Your decision. Your people. Your evidence. Your learning.

Understand what a behavioral answer must demonstrate

A behavioral interview asks for evidence of how you act when the answer is uncertain or people disagree. Knowing the right vocabulary is not enough. The interviewer needs a situation in which you had responsibility, a decision you made, how other people were involved, and what happened afterward.

Start with an actual event, then reconstruct the decision. Suppose a pilot performed poorly after an impressive demo. “We improved the evaluation” describes a team activity. A useful account explains that you noticed the demo set excluded difficult cases, agreed with product on a narrower launch, delegated failure analysis, and changed the release process. Those details show judgment, delegation, and organizational learning. Only use them if they describe your real experience; the example here is a rehearsal illustration.

The established STAR structure means Situation, Task, Action and Result. Reflection can be included in the result; this guide uses STAR-L to make learning explicit. It is an optional extension, not a separate universal hiring standard. National Careers Service: STAR.

Build one story at three levels

Write a one-sentence headline first: “I narrowed a launch after discovering that our evaluation missed changing policies.” Then prepare a two-minute account with the essential context, decision, actions, and result. Finally prepare the details an interviewer may request: rejected alternatives, who disagreed, what you delegated, how you measured progress, and what you got wrong.

This layered preparation makes interruptions easier. If asked why you delayed launch, discuss the failure consequence and alternatives rather than restarting from the project's beginning. If asked about leadership, explain how you created ownership and alignment. The same facts can support different questions, but do not force one favorite story into every topic.

Distinguish evidence from attribution. If support workload fell after a launch, ask whether staffing, seasonality, or process changes also contributed. You can credibly say what changed and what evidence connects it to your work without claiming that you alone caused every benefit. When exact metrics are unavailable, use a concrete observation such as an adopted operating process or independent ownership by an engineer you coached.

Practice the full range of behavioral questions

Prepare stories about ambiguity, unrealistic expectations, failure, cross-functional work, responsible deployment, influence without authority, technical decisions, collaboration styles, and learning. For management roles, add people-management depth instead of letting technical project stories replace it. Prepare a coaching story that shows changed behavior, a hiring story that connects the role to a team gap, and a delegation story in which somebody else became more capable. For individual-contributor roles, emphasize your implementation, debugging, collaboration and technical decisions without claiming management authority you did not hold.

For conflict, describe the other person's strongest legitimate concern. Product may be protecting a customer commitment; research may be protecting experimental validity. Your job is not to portray one side as irrational. Explain the shared decision criteria, the disagreement that remained, and who was accountable for the final call. For a failure story, spend more time on your missed signals and corrective actions than on external excuses.

After rehearsing, ask a listener to repeat your decision and contribution. If they remember only the model name or the project's impressive scale, the leadership evidence is still buried.

Build a small, reusable story bank

Select real stories from these areas according to the role. Hiring and performance-management examples are relevant when those responsibilities are part of your experience and target role. One story can cover several areas, but avoid forcing the same launch into every question.

Use confidential details only at an appropriate level of abstraction. Do not present a hypothetical example from this guide as your own experience.

Shape the answer with STAR-L

Use “I” for your actions and “we” for the team's work. A credible answer can show both without claiming sole credit.

For a two-to-three-minute practice answer, spend most of the time on actions and results. Timing is a rehearsal aid, not an interview rule. Prepare a short version and details for interruptions.

Worked hypothetical: a promising pilot misses production needs

The following is a teaching scenario, not Anup Rai's or the reader's employment history.

Follow-up: explain the legitimate pressure to launch, the alternative you considered, what remained unresolved and the evidence that would allow expansion. In a real answer, use the actual decision and outcome, including a mixed or unsuccessful result.

People leadership needs its own detail

For coaching, explain how you diagnosed the gap, listened to the person, agreed on expectations, provided opportunities and feedback, and checked progress. Do not equate a skill gap with low motivation without evidence.

For performance management, describe fair, specific expectations and timely feedback, support, and follow-through using your organization's process. Avoid disclosing personal details or portraying yourself as a heroic rescuer. Show how you protected team delivery while treating the person respectfully.

For hiring, begin with team needs and an assessment rubric. Avoid hiring for model-name trivia alone. Evaluate judgment, learning, engineering fundamentals, collaboration, and role-specific depth. Explain how onboarding closes gaps after hiring.

AI-specific judgment

AI projects have uncertainty in labels, feasibility, operating cost, and user acceptance. Strong examples show staged investment: a cheap feasibility test, a representative pilot, explicit continuation/stop criteria, and a route to production ownership.

If research wants more experiments and product wants a date, define the decision each experiment changes, time-box uncertain work, and agree on a shippable fallback. If security raises a concern, translate it into an exposure path and mitigation evidence rather than dismissing it as bureaucracy.

Common weak answers

“We worked hard” hides your decisions. “I built everything myself” can signal weak delegation. “The metric improved because of me” may overclaim causality. “The other team was unreasonable” avoids your part in the conflict. “I learned communication matters” is too vague without a changed practice.

Numbers help when grounded. If exact figures are unavailable, describe verifiable qualitative outcomes, a range, or the evidence limitation. Never invent revenue, accuracy, team size, or savings.

Follow-up rehearsal

After each story, answer: What alternative did you reject? Who disagreed? What did you delegate? What would the other person say? What did you personally get wrong? How do you know the outcome was better? What would you do differently with half the team?

Ask interviewers how the team defines success, divides research/product/platform ownership, handles on-call and evaluation work, grows managers and engineers, and resolves launch-risk disagreements.

Recall test: tell one coaching story and one stopped-project story without mentioning any model brand. If the leadership contribution disappears, add the missing decisions and people work.

Hypothetical STAR-L stories to practice

These invented situations are teaching examples. Their results belong to the examples, not to your career. Replace them with truthful events and evidence when rehearsing; do not adopt the wording as personal testimony. Together with the pilot story above, they demonstrate six different situations.

Being wrong about the technical direction

Being overruled and escalating responsibly

Responsible deployment and an ethical concern

Cross-functional incentives and influence without authority

Coaching and delegation after a missed milestone

Questions that help you assess the role

Use the answers to evaluate mutual fit. Avoid turning the final minutes into a checklist; select the unresolved questions that matter most to your decision.

Interview questions with developed answer guidance

Behavioral answers must come from your experience. These fifteen prompts describe evidence to prepare; they are not sample achievements to claim.

Check the facts before rehearsing

For a hypothetical measurement worksheet, 26 correct cases out of 40 is 65%; 30 out of 40 is 75%. That is a 10-percentage-point increase, or about 15.4% relative to the initial rate. Those counts alone do not establish statistical significance, representative production quality or causation. Do not substitute a percentage that sounds more impressive. See capability assessment.

Practice aloud and adapt

flowchart LR
    E[Choose an actual event] --> F[Verify facts and ownership]
    F --> S[Prepare a short account]
    S --> M[Rehearse with follow-up questions]
    M --> R[Review clarity and unsupported claims]
    R --> S

1. Prepare a brief headline and a two-to-three-minute version; the times are practice aids.
2. Rehearse explaining your decision without naming a model or framework.
3. Record yourself if useful and review whether context crowds out actions and results.
4. Ask a practice partner to interrupt with “why?”, “what alternative?” and “what did you get wrong?”
5. Check that the same facts survive the follow-up; change the emphasis to answer the actual question.
6. Prepare different events for learning, conflict, failure and people leadership where relevant.

Follow the employer's rules on AI assistance in applications and interviews. You can practice with tools, but live assistance requires permission under that interview's stated rules. Confirm the format and accessibility arrangements with the recruiter.

Final summary and notes

A strong account can include uncertainty, disagreement and an unsuccessful outcome. Its value comes from clear evidence of how you acted and learned.

60-second interview answer

For a behavioral question, I choose a real example and explain the situation briefly, then focus on my decisions, actions and collaboration. I describe the tradeoff, disagreement, or uncertainty, what I did, and what evidence shows the result. I distinguish my contribution from the team's work and avoid overstating causality. I end with what I learned and changed. My examples should match the role. For management, that includes people leadership and delivery as well as technical judgment; a successful model launch alone does not demonstrate all those responsibilities.

Source check: September 24, 2026. Use current employer postings to identify the work, then prepare evidence that you can do it. A title, salary headline or model launch does not establish what every employer wants.

This chapter combines a small set of dated primary labor-market reports with specific employer examples. It is not a survey of all AI jobs and does not claim to have analyzed hundreds of listings. The role and preparation tables are a practical classification, not an official or exhaustive occupational standard.

Read the evidence before the headline

Different sources measure different things:

Three current observations

These studies cover different populations and outcomes. They support careful, role-specific research; they do not prove that one occupation is safe, that a title has disappeared or that every experienced AI specialist can command an offer.

Role taxonomy: classify responsibilities

Titles overlap. Research is not confined to frontier labs, and a research-engineer title does not universally require a PhD or publication. “Member of technical staff” does not specify a level by itself. Model prompting, distillation and MCP can be responsibilities without being standalone job titles.

What current employer examples actually show

These examples were accessible on the source-check date. Openings and requirements can change or close.

The interpretation in the last column is this guide's preparation advice. Three examples demonstrate variation; they cannot establish market-wide framework frequencies, demand rankings or a universal required stack.

Skills by scope, not a universal level number

An L5 or L6 at one employer may not mean the same scope as that label elsewhere. Years of experience can be an eligibility criterion, but do not alone determine level.

Prepare technical depth relevant to the work. A serving role may need GPU memory and scheduling; an application role may need authorization, retrieval and product evaluation. Neither needs to pretend expertise in every layer of AI.

Turn a job description into a study plan

flowchart LR
    J[Current role description] --> R[Separate requirements from preferences]
    R --> E[Map each responsibility to your evidence]
    E --> G[Identify gaps and clarify scope]
    G --> P[Practice relevant concepts and designs]
    P --> M[Mock the confirmed interview format]

1. Save the title, employer, location, posting URL and date checked.
2. Separate explicitly required experience from preferred experience and vague marketing language.
3. Identify the actual outputs: a service, trained model, evaluation system, customer deployment or team outcome.
4. Match each responsibility to a real project, experiment or skill demonstration.
5. Choose the two or three gaps most likely to block performance in that role.
6. Confirm the interview format and permitted tools before rehearsing it.

Interview tip: a small complete project with clear evidence can be more informative than several unfinished demos. This is a preparation principle, not a claim that a portfolio always outweighs a degree or research paper in hiring.

Compensation: compare like with like

Use a posting's own pay label. Base salary, total compensation, sales on-target earnings and contractor revenue are different quantities.

Dated examples, not market medians

All three rows were checked September 24, 2026. They are selected employer examples, not a representative salary sample. Do not compare the GBP and USD figures without an explicit exchange-rate date and an understanding of taxes, benefits and location. Do not infer an offer at the top of a published range.

Build an offer-comparison worksheet

For a hypothetical arithmetic example, $180,000 base plus a 10% target bonus is $198,000 in base plus target bonus. A $20,000 sign-on makes the first-year figure $218,000 if paid in that year; it does not make recurring annual pay $218,000. Equity is additional and must be described under its actual terms. This is a comparison worksheet, not personal tax or investment advice.

It is reasonable to ask a recruiter about the budgeted range and level early enough to establish mutual fit. Detailed negotiation timing varies; there is no universal rule that compensation may only be discussed after an offer.

Geography and industry change the role

Do not infer worldwide opportunity from a few US technology-company postings. Research the location and sector you can actually work in.

Regulatory requirements depend on the application and jurisdiction. SOC 2, HIPAA, FedRAMP and the EU AI Act are not interchangeable credentials or rules that every AI engineer must personally “certify.” Use the governance chapter for current distinctions.

Interview process patterns: confirm the actual loop

A process may include recruiter discussion, coding, system design, a project deep dive, experiments, a take-home, behavioral questions or a practical build/review. Employers combine these differently; whiteboards and ordinary programming fundamentals have not universally disappeared.

A documented example is Sierra's April 22, 2026 account. It describes a Plan → Build → Review onsite with a two-hour AI-assisted build, and a system design interview replacing its coding phone screen. It also describes a debugging format being piloted, with the degree of AI use still under consideration. That is one employer's published process, not permission to use AI in another company's interview.

For a permitted AI-assisted task, be able to explain and test the resulting code, identify failures, and separate working behavior from unverified assumptions. A generated artifact is not evidence that you understand it.

Specialized responsibilities to watch

A new label can identify useful work, but its long-term prevalence needs evidence. Learn the underlying contract so your expertise survives framework and title changes.

For hiring managers and engineering leaders

1. Define the outcomes and authority of the role before listing tools.
2. Separate necessary day-one skills from skills the team can teach.
3. Use consistent, relevant assessment criteria and make permitted assistance clear.
4. Size the assessment to the actual responsibility and avoid requesting confidential prior work.
5. Give evaluation and release decisions appropriate independence without assuming every team needs a separate evaluation department.
6. Account for operating work, incident coverage, source ownership and onboarding in staffing.
7. Calibrate compensation to the actual level, location and package; do not apply an invented universal “AI premium.”

A practical exercise can be useful when it measures job-relevant skills, but this chapter does not establish that one interview format is universally more predictive than all others.

Interview and career-research questions

Final summary and notes

Use the transition guide to map existing skills, the practice hub to plan study, and behavioral preparation to present your actual experience.

Use this page for a direct definition and a practical distinction. Follow the linked lesson for diagrams, calculations and interview practice. Product availability changes; the model, pricing and framework chapters carry dated checks. There is no universal best stack independent of workload and constraints.

General

What is an AI engineer?

An AI engineer builds software systems that use AI models to perform useful tasks. Work can include data preparation, model integration, evaluation, serving, security and operations. A role may focus on generative AI, traditional machine learning, or both.

Interview example: a support assistant needs identity, authorized retrieval, answer evaluation and failure handling in addition to a model call. See the role preparation guide.

What is the difference between an AI engineer and an ML engineer?

The titles overlap. “AI engineer” often emphasizes applications and model integration; “ML engineer” often emphasizes training, data pipelines or model deployment. Neither title has a universal boundary, and either role can own the full lifecycle.

Read the responsibilities rather than assuming one role never trains models or the other never builds applications. See role and market research.

How do I become an AI engineer?

Build the missing skills for a target responsibility and demonstrate them in a small, complete project:

1. Implement the ordinary software and data path.
2. Add a justified model capability.
3. Evaluate it against a simpler baseline.
4. Test permissions, failure recovery and cost.
5. Explain the results and limitations.

Existing backend, frontend, data or testing experience can transfer, but does not remove the need to learn model behavior and evaluation. Use the transition guide.

What programming language should I learn for AI engineering?

Choose the language required by the work and its supported libraries. Python is useful for model experiments and data workflows; TypeScript is useful for web applications and supported agent SDKs. Serving infrastructure may also involve C++, Rust, Go or Java.

A language choice alone is not a career strategy. Learn to inspect API contracts, test failures and operate the resulting system. Check a framework's current SDK support before choosing it. See framework selection.

Is AI engineering a good career?

That depends on your interests, local opportunities and the responsibilities you want. Investigate actual postings, interview expectations, team stability and the work behind the title. No guide can promise demand, compensation or an offer for an individual.

A useful interview question for an employer is: “Which production outcomes will this role own, and how are they measured?” See job-market research.

RAG

What is RAG?

Retrieval-augmented generation combines retrieval of external information with generation conditioned on that information. A policy assistant might retrieve the applicable policy edition and use its passages to answer with citations.

RAG can improve access to relevant information. It does not guarantee correct retrieval, faithful interpretation or a supported answer. The original research combines parametric and retrieved knowledge; today's applications use several retrieval architectures. RAG paper; RAG fundamentals.

How does RAG work?

Separate the data preparation path from the request path:

flowchart LR
    S[Source documents and permissions] --> I[Parse, version and index]
    I --> X[Search index]
    Q[Authenticated question] --> R[Authorized retrieval]
    X --> R
    R --> C[Select evidence within budget]
    C --> G[Generate answer]
    G --> V[Validate support and citations]
    V --> A[Answer or explicit limitation]

Dense embeddings are one retrieval method; lexical or structured retrieval can also supply evidence. Preserve document versions and permissions through ingestion, caches and citations. See production RAG.

Is RAG dead because of long context windows?

No. A larger context window permits more input but does not decide which material is relevant, current or authorized. Retrieval can reduce unnecessary input and target evidence. Directly providing a small, stable document set may be simpler than operating an index.

Compare answer quality, omitted evidence, latency and total cost on the same task. Long context and retrieval can be combined. See context engineering.

What is the difference between RAG and fine-tuning?

Fine-tuning can affect knowledge and behavior; it is not merely a formatting tool. RAG does not update weights. Neither guarantees strict format, correctness or lower latency. They can be used together. See fine-tuning strategies.

What is the best vector database?

There is no universal winner. Test the actual embedding dimension, dataset size, filters, update/delete load, recall target, latency and deployment requirements. Include operational ownership and the option of adding vector search to an existing database.

A published latency number without its hardware, recall, filters and concurrency is not a sizing guarantee. See vector databases.

What is contextual retrieval?

In this retrieval technique, a short explanation of a chunk's document context is added before indexing it. For example, a passage saying “the limit is 30 days” needs context about the product and policy to which the limit applies.

The added context can help retrieval and can also introduce incorrect information. Keep the original passage, version the derived context, and measure against a plain-chunk baseline. See contextual retrieval.

What is hybrid search?

Hybrid search combines retrieval signals, commonly lexical matching and dense-vector similarity. An exact error code may benefit from lexical matching, while a paraphrased question may benefit from semantic retrieval.

Fusion methods such as reciprocal rank fusion combine ranked results; they do not make unrelated raw scores directly comparable. Product support and filtering behavior vary. See hybrid search.

What is GraphRAG?

GraphRAG is a family of approaches that use graph structure in retrieval-augmented generation. Nodes and edges may represent entities, relationships, documents or extracted claims. Some implementations also summarize communities for broader questions.

A graph query can reliably follow stored edges while those edges remain wrong, stale or incomplete. Graph construction and permissions add costs. Use a graph when the task needs relationships that simpler retrieval does not adequately provide. See GraphRAG.

What is the best chunk size for RAG?

Choose it by measuring evidence retrieval and answer quality for the content. A table, legal clause and source-code function have different boundaries. Small chunks can lose context; large chunks can dilute relevance and consume the input budget.

Test a few sensible structural strategies, including how headings, tables and neighboring passages are retained. Count chunks and their overlap when estimating indexing cost. See chunking strategies.

Agents

What is an AI agent?

An AI agent is a system that selects actions based on observations to pursue an objective. In an LLM-based agent, the model helps choose actions; application code executes them within defined permissions, budgets and stopping rules.

An agent does not require a specific model brand, framework or vector database. In a predefined workflow, code determines the sequence and branches; an agentic workflow gives the model some control over the path. Agent fundamentals; architectural distinction.

What is the difference between an agent and a chatbot?

“Chatbot” describes a conversational interface. “Agent” describes how actions are selected. A chatbot can answer directly, run a fixed workflow or expose an agent. An agent can operate through a chat interface or a background task.

Ask what the system can read or change, how it selects actions, and which results require verification. See agent fundamentals.

What is MCP (Model Context Protocol)?

MCP is an open protocol for connecting AI applications with tools and context provided by servers. It standardizes integration contracts; it does not itself establish that a tool is trustworthy or that the user is authorized to perform a business action.

Protocol versions and SDK releases are separate. Choose compatible versions, enforce authorization at the service boundary and treat returned content as data. Official introduction; MCP lesson.

What is the best agent framework?

Choose by the behavior required: typed inputs, resumable execution, human approval, tracing, testing, supported runtimes and deployment ownership. A small workflow may need only ordinary application code and an SDK.

Framework popularity does not prove correct recovery or tenant isolation. Test an interrupted run and a duplicate tool request before relying on the abstraction. See framework selection.

What is agentic RAG?

Agentic RAG lets a model direct some retrieval decisions, such as rewriting a query, choosing a source or deciding that more evidence is needed. A fixed retrieve-then-generate pipeline does not make those decisions dynamically.

Bound the search and generation work. More iterations can increase cost without finding better evidence; measure stopping quality and failed searches. Different research methods use different control loops. See agentic RAG.

How do computer-use agents work?

They observe an interface, propose actions such as clicking or typing, execute permitted actions, and inspect the result. Observations may use screenshots, accessibility information or browser structure, depending on the system.

A clicked “Submit” button is not proof of a completed transaction. Reconcile the resulting application state, and separate observation permissions from consequential writes. See computer-use agents.

What is context engineering?

Context engineering is the design of the information supplied to model invocations: instructions, task state, retrieved evidence, conversation, tools and their results. It includes selection, formatting, budgeting and updating that information.

The term overlaps with prompt engineering and does not replace a precise explanation of what changed. A larger prompt is not automatically better context. See context engineering.

What are Agent Skills?

In the Agent Skills format, a skill packages instructions and optional resources that an agent can load for a task. Metadata can support discovery before fuller instructions or resources are loaded.

A skill is not an authorization boundary, and an instruction file does not sandbox scripts. Review its provenance and execution requirements under the application's permission model. See building tools and skills.

Models

What is the best LLM right now?

Select a deployment that passes the task's hard constraints and performs well on representative held-out cases. Evaluate quality, feature compatibility, latency, reliability and complete operating cost. A public benchmark is useful evidence for its stated task, not a universal ranking for your application.

Record the model identifier, endpoint, region and settings. Re-evaluate material changes. See model selection.

How much does Claude / GPT / Gemini / DeepSeek cost?

The answer depends on the exact model, input/output usage, cache categories, service tier, context thresholds, region and separately billed tools. Subscription prices for a chat application are not API rates.

Use the dated pricing tables and worked cost models. Count retries and review work as well as model calls. Do not copy a single price into a business case without its unit and conditions.

What is the difference between Claude Opus and Claude Sonnet?

They are separate model families with different product positioning, pricing and model-specific capabilities. The names alone do not establish a fixed quality percentage, latency ratio or feature contract.

Compare the exact current models on your task. Check endpoint, tool, reasoning and context support independently of the family name. See model taxonomy and selection.

Should I use an open-source model?

First distinguish open weights from a model released under terms that meet an open-source definition. Inspect the actual license, allowed uses and available artifacts.

Self-hosting can provide deployment control, but you then own capacity, security, upgrades and operations. It does not automatically provide lower cost or stronger privacy than every managed alternative. See model taxonomy and serving infrastructure.

What is prompt caching?

Provider prompt caching reuses processing for a compatible repeated input prefix, subject to the provider's rules. Cached input may still be billed. Writes, reads, minimum sizes, expiry and routing differ by model and provider.

If ordinary input costs 2 units, a cache write costs 2.5 and a later read costs 0.2, two uses cost 2.7 rather than 4 units. That example assumes an actual cache hit and excludes unrelated costs. See cache semantics and cache economics.

Evaluation

How do you evaluate an LLM?

Evaluate the behavior required by the application, not just whether the response sounds plausible.

1. Define success, critical failures and the target population.
2. Create representative held-out cases and important slices.
3. Run comparable configurations with controlled versions.
4. Use suitable deterministic checks, human labels or calibrated judges.
5. Report uncertainty, missing outcomes, latency and cost.
6. Validate the complete application under limited production exposure.

For RAG, evidence support is different from relevance and retrieval quality. “Reference-free” grading may still require the supplied context. See LLM evaluation.

What is LLM-as-judge?

An LLM judge grades an output against a rubric, reference or evidence. It is a measurement instrument with possible bias and errors, not independent ground truth.

Validate it against appropriate human judgments, inspect disagreement, and test sensitivity to answer order, style and injected instructions. A more expensive judge is not automatically reliable for every domain. See capability assessment.

What is the best LLM observability tool?

Choose by the required traces, metrics, evaluations, privacy controls, retention, deployment model and export/interoperability support. Confirm that instrumentation captures the application boundaries and versions you need.

A trace can reveal latency or a failed tool call; it does not by itself prove factual correctness. Avoid recording raw sensitive prompts by default. See observability.

What is RAGAS?

Ragas is an evaluation framework with metrics and workflows for AI applications, including RAG. Select metrics according to the question being measured and the required inputs.

A faithfulness score assesses support relative to supplied context; that context may itself be stale or wrong. Version the metric, prompts and judge configuration, and calibrate against human review. See RAG evaluation. Do not treat one framework as a mandatory standard.

How do you detect and handle model drift in production?

Monitor input distributions, task outcomes, slice performance, cost and latency; compare a stable regression set and newly labeled production cases. Changes can come from the model, data, retrieval, prompts, tools or the user population.

A falling score signals a problem but does not identify its cause. Trace versions, reproduce failures, and use a tested rollback or constrained fallback if it meets the same requirements. See evaluation-gated delivery.

Inference

What is vLLM?

vLLM is an open-source model inference and serving engine. Its features include mechanisms for scheduling requests and managing attention memory; supported models, hardware and APIs depend on the release.

Compatibility with an API shape does not guarantee identical behavior to another provider. Test the exact model/runtime/hardware combination, load and failure paths. See serving infrastructure.

What is the difference between vLLM and SGLang?

They are distinct inference/serving projects with overlapping capabilities and different implementations and release support. Neither is universally faster or safer for every model and workload.

Compare quality compatibility, time to first token, inter-token latency, throughput, memory use and operational requirements under the same conditions. Check supported versions and security advisories when deploying. See inference fundamentals.

What is TensorRT-LLM?

TensorRT-LLM provides NVIDIA-focused components for optimizing and serving language-model inference. Model support, precision formats and execution paths depend on the release and GPU architecture.

Evaluate the deployment and build workflow as well as measured performance. There is no universal speed multiplier or fixed setup duration relative to another engine. See serving infrastructure.

How do you optimize LLM inference cost?

Measure cost per accepted outcome first. Then test reductions in unnecessary input, output, retries, model size, redundant calls or idle serving capacity. Caching, batching, routing and quantization help under different conditions.

See cost optimization.

What is speculative decoding?

A draft mechanism proposes tokens that the target model verifies, potentially reducing expensive sequential target steps. Correct exact speculative sampling can preserve the target distribution under its algorithmic assumptions.

Speed depends on acceptance rate, draft overhead, hardware and workload. Heuristic variants need separate quality analysis; there is no guaranteed speedup. See speculative decoding.

What is a token budget and how do you enforce it?

A token budget limits input or generated token usage for an operation. A spending budget also needs model rates and other billable work, such as tools and retries.

Count with the correct tokenizer or provider method, reserve estimated spend atomically before admission, cap supported outputs, and reconcile actual usage. Hold uncertain in-flight charges until resolved. A prompt asking the model to “stay under budget” is not enforcement. See FinOps and token economics.

How do you design fallbacks across multiple LLM providers?

Select alternatives that satisfy the same data, feature and quality constraints. Normalize interfaces while preserving provider-specific limits and errors. Classify the failure before retrying.

flowchart TD
    E[Primary call did not complete normally] --> O{Outcome known?}
    O -->|No, side effect possible| R[Reconcile operation before replay]
    O -->|Yes| C{Alternative meets task constraints?}
    C -->|Yes, retry permitted| F[Bounded fallback with shared deadline]
    C -->|No| H[Return limitation or review path]
    R --> S[Record confirmed or unresolved status]

A fallback must not bypass an authorization denial or safety requirement. Independent providers can still share correlated failures or dependencies. See AI gateways.

Memory

What is the best AI agent memory framework?

Start with the memory contract: what may be retained, who owns it, how it is retrieved, how claims are corrected, and when deletion must take effect. Then choose storage and tooling that implement those rules.

Evaluate usefulness, false memories, access isolation and lifecycle behavior. A vector database or published benchmark score does not prove that remembered information is true. See memory architectures.

What is the difference between short-term and long-term memory in agents?

Short-term memory commonly refers to working information for the current task or conversation. Long-term memory retains selected information across tasks or sessions. These are application concepts, not a required mapping to one storage technology.

Working state can live outside the current model context. A KV cache is computation reuse, not durable semantic memory. See short-term context and long-term memory.

How does a knowledge graph help an AI agent?

It represents entities and relationships that can support structured lookup, relationship traversal and provenance. For example, a graph can record which document edition supports a particular claim.

The graph is only as reliable as its data and update process. Exact traversal does not imply correct or complete facts. Apply access rules to nodes, edges and derived answers. See GraphRAG.

Security

What is prompt injection?

Prompt injection occurs when instructions embedded in input influence a model in ways that conflict with the application's intended instruction or trust boundaries. An indirect injection may arrive in a retrieved page or tool result.

For example, a document may tell a support agent to send private account data to an outside URL. The document is task data; it cannot authorize that action. See prompt injection.

How do you prevent prompt injection?

No prompt format or detector guarantees prevention in an arbitrary application. Reduce exposure and consequences through enforceable boundaries:

1. Keep authorization and allowed actions in application code.
2. Use least-privilege tools and scoped credentials.
3. Preserve the origin of untrusted content.
4. Restrict execution, file access and network destinations as appropriate.
5. Validate outputs and bind required approval to the exact action.
6. Test attacks, monitor failures and limit recovery costs.

Delimiters and model-based classifiers can help but do not replace these controls. See LLM security.

What is OWASP LLM Top 10?

It is OWASP's community guidance on major risks in LLM applications, with explanations and mitigation approaches. It is not a certification or proof that a system is secure.

The 2026 edition was published August 3, 2026. Always name the edition when using an identifier because categories and rankings change. Use the official 2026 resource and the security lesson's mapped examples.

What is sandboxing in AI agents?

Sandboxing restricts code or tool execution within a controlled environment. Isolation mechanisms may include operating-system controls, containers or virtual machines; these provide different boundaries.

A disposable workspace can still expose mounted secrets, send data over the network or consume resources. Define filesystem, credential, network, tenant and lifetime restrictions, then test them. A container is not automatically equivalent to a microVM. See agent sandboxing.

Final summary and notes

Interview tip: define the term in one sentence, give one concrete example, and state the limitation that matters to the proposed design. If asked for a product choice, connect it to a measurable requirement rather than repeating a popularity ranking.

Related reading

• Interview practice hub
• Question bank
• Answer frameworks
• Complete enterprise RAG interview
• Model assessment

Three ways to study the same subject: a full, paced explanation; a self-contained rapid revision handbook; and questions with concealed answers.

Choose the part for your current task. Within rapid revision, the topic links stay in that section.

Part I — Detailed understanding

A large language model (LLM) is a neural network trained on large amounts of language data to model language and perform tasks such as text generation and understanding. Its parameters are numerical values learned during training. There is no universal parameter-count threshold that makes a model “large.”

This chapter follows an autoregressive, decoder-only Transformer: it predicts a probability distribution over the next token, selects a token, and repeats using the growing sequence. That is the scope of the generation examples, not the definition of every language model. Encoder models, encoder–decoder models and hybrid architectures appear later with their own information flow.

Start with the standard definitions

Technical example: given The backup is stored in the archive. Where is the backup stored?, a supported response is In the archive. We use this short evidence-based question to track changes to the prompt and generated sequence. The separate The cat sat examples isolate matrix arithmetic. All hand-chosen vectors, probabilities and workload figures are teaching assumptions, not measurements from a commercial model.

The six learning objectives are:

1. Convert text into token IDs and vectors.
2. Calculate how allowed positions contribute through attention.
3. Assemble the complete Transformer and output-selection path.
4. Derive a training loss and parameter update.
5. Estimate serving memory, latency and cost.
6. Defend an application decision using quality and operational evidence.

Choose a route and a stopping point

You need ordinary arithmetic and the idea of a percentage to start. When a new kind of calculation is needed, we will explain its purpose and work an example before using the compact mathematical notation. Our main example writes from left to right: it chooses the next token using the input and the tokens it has already generated. The technical name for this design is a causal decoder. Here, “causal” means that a calculation for a token position can use that token and earlier text, but cannot look ahead to later text. “Decoder” refers to the part that produces the output. Section 17 compares it with other designs.

Use these six stages to organize your study. A stage can take several sittings, especially when the mathematics or programming is new. At each stopping point, close the explanation and do the check aloud. Move on when you can explain the steps, rather than when a timer expires.

First pass: follow the stages in order. Section 9 uses one numerical coordinate per query, key, and value to explain how attention combines information. Section 10 repeats that calculation with vectors containing multiple coordinates. Work the first example before the second. The optional disclosures and section 23's more advanced attention-cache compression method can be a second pass; the full explanations remain on the page.

Implementation pass: after following the explanations, use the complete tiny decoder to see how the steps become code. Check the shapes of the matrices, which positions can read which others, how word order enters the calculation, and where one result is added to another. Predict the effect of a change before running it. The later debugging exercises ask you to find which of these rules a program has broken.

Interview revision: use Part II’s self-contained revision handbook for a concise review. The eighteen anchors also remain at the end of these detailed lessons. Then use Part III’s questions and concealed answers, the recall coverage checklist, and the timed mock. Open an answer only after committing to your own. Being able to recognize an explanation is easier than producing it independently.

A stage is complete when you can explain the purpose, work a new example, and state one limitation without notes. If only one of those fails, revisit that component rather than rereading the entire chapter. The page's reading-progress indicator measures position, not understanding.

Keep the whole model in view

The diagram shows the order of work in our main example. Read it from top to bottom. You do not need to know how each box works yet: the following sections open those boxes. The return arrow means “process the selected token to predict the following token.” Tokenization supplies token IDs, embedding lookup retrieves the corresponding vectors, and Transformer blocks update those vectors using the permitted context. A vector is an ordered collection of numerical coordinates; the following lessons explain each transformation in detail.

flowchart TB
    P["Input text"] --> T["Tokenization: convert text into token IDs"]
    T --> E["Embedding lookup: retrieve a learned vector for each token ID"]
    E --> B["Transformer blocks: update token representations with attention and feed-forward networks"]
    Pos["Position information"] -.-> B
    B --> L["Language-model head: calculate a logit for each vocabulary token"]
    L --> S["Decoding: select the next token from the vocabulary scores"]
    S --> A["Append the selected token to the generated response"]
    A --> C{"Stopping token or output limit reached?"}
    C -->|"Yes"| F["Return the generated response"]
    C -->|"No: process the selected token"| E

The first pass processes the supplied request. Each later pass processes the selected token and reuses earlier tokens’ attention keys and values from the key/value cache. This avoids starting every calculation again from the beginning. Section 22 explains exactly what is saved. The model uses the same parameters throughout ordinary answer generation; choosing a new token does not itself train the model.

1. What the model is actually trying to do

For the backup-location question, the model builds an answer through repeated choices. It chooses a token—a unit of text from its vocabulary, such as a word or part of a word—adds it to the response, then chooses the next token using both the request and what it has written so far.

For an easy illustration, imagine that the tokens selected are In, the, archive, and .. Actual token boundaries depend on the tokenizer, which the next section explains. The important point is the sequence of decisions:

The model has a fixed set of possible tokens, called its vocabulary. At each step it calculates one number—a score, also called a logit—for every token in that vocabulary. Software converts the scores into probabilities: numbers between 0 and 1 whose total is 1. A probability of 0.7 corresponds to 70% of that total. A selection rule then chooses one token. It can always take the most probable token, or make a random choice in which higher-probability tokens are more likely to be chosen. Section 18 works through these choices with numbers.

Why earlier text matters

Change the request to “The backup is stored in the vault.” A useful model should now favor vault over archive in the answer. The question is nearly identical, but the preceding evidence changes what it should say.

The information available for a particular choice is the model's context. In our text example, this includes the request and any tokens already generated. An application may also include instructions, earlier conversation, documents, or the output of another program. For example, it could put a weather service's result into the text the model receives. The model calculates the next choice using the information actually supplied.

Each vocabulary entry has an identifying number, called a token ID. A unit represented by one of these entries is a token; section 2 shows how text is split into tokens. Calculating probabilities for the next token is next-token prediction. Repeating the choice and giving each chosen token back as input for the next step is autoregressive generation. To remember that longer name, picture the growing answer becoming part of the next input.

How can predicting the next token support a useful answer?

Suppose many training examples contain questions followed by answers that use information from the question. A model that learns to use those clues can predict the answers better than one that ignores them. The same pressure can help it learn spelling, grammar, common facts, and ways to combine information. Additional training examples can demonstrate how to follow instructions. These abilities arise through changes to the model’s parameters, the stored numerical values used in its calculations; programmers do not write a separate rule for every possible question.

Prediction can still be wrong. If the request says “vault” but the model has a strong learned association with “archive,” it may write the wrong location. Its probabilities describe the choices favored by its calculation. They do not show that someone checked the answer against the request or an outside source.

Several next-token choices define a probability for a sequence

The probability of several successive choices comes from multiplying the probabilities at their respective steps. Suppose our teaching model assigns 0.5 to choosing In first, 0.4 to choosing the after In, and 0.25 to choosing archive after In the. It assigns 0.5 × 0.4 × 0.25 = 0.05, or 5%, to that particular three-token continuation. Each step uses the text available at that step. We are not assuming that the choices are independent of earlier text.

Here is how to read the compact notation for that multiplication. P means probability. A vertical bar means “given”: P(B | A) is the probability of B when A is already known. Let c be the prompt and T the total number of answer tokens being scored. The tokens are named x₁, x₂, and so on up to x_T, in order. The lowercase t counts which step we are on; x_t is the token at that step. This formula, called the probability chain rule, repeats the multiplication we just did:

$$P(x_1,\ldots,x_T\mid c)=\prod_{t=1}^{T}P(x_t\mid c,x_1,\ldots,x_{t-1})$$

The large ∏ sign means “multiply the following expression for every step from 1 through T.” At each step, the expression to its right uses the prompt and all earlier answer tokens. For step 1 there are no earlier answer tokens. If we want the probability of an answer that ends at a particular place, we must also include the probability of choosing its ending marker, a special token explained next. Section 19 shows how training uses these probabilities to measure mistakes. A high probability still does not establish that the answer is true.

Interview question: What does a language model produce at one generation step?

2. Why text is split into tokens and given IDs

The model needs a list of candidates so it can calculate one score for each. Making that list contain only whole words causes a problem: what happens when someone types a new name, a spelling mistake, or a word the list does not contain? A practical vocabulary therefore includes smaller pieces that can be combined to represent unfamiliar words, as well as frequently used whole words.

A tokenizer is the software that splits text into these pieces and looks up their identifying numbers, or token IDs. For example, an invented tokenizer might split unbelievable! into un, believ, able, and !, then return the four corresponding IDs. A different tokenizer can use different pieces and IDs for the same text.

Spaces and punctuation also need representation. Some tokens include a leading space, which is why the and the can have different IDs. Some tokenizers can fall back to pieces of the underlying stored bytes when a larger piece is unavailable. A bit is a stored 0 or 1; a byte groups eight bits. The characters you see on screen are encoded using bytes, sometimes several bytes per character. A tokenizer with suitable byte handling can represent unfamiliar text using those smaller units. Its exact rules still matter: some tokenizers first change capitalization or accents, which can lose information.

For a small teaching vocabulary, suppose these entries exist:

ID 9821 identifies the cat entry in this invented vocabulary. Treat it like a book's catalogue number: a nearby number need not identify a book on a similar topic. Likewise, token ID 9822 need not mean something similar to cat. You could relabel the vocabulary if you also relabelled every model table that uses those IDs consistently. The identity of each piece would be preserved. This table illustrates the lookup; it is not a measurement from a named tokenizer.

Keep two counts separate. Vocabulary size is the number of entries the model can choose from. Sequence length is the number of tokens in this particular input. If our input consists of cat, cat, its sequence length is two. Both positions use ID 9821; repeating an entry does not add a new entry to the vocabulary.

Where the vocabulary comes from

People building a tokenizer use a collection of text to decide which pieces deserve vocabulary entries. One method, Byte Pair Encoding (BPE), starts with small units and repeatedly combines frequently adjacent pairs. For a tiny illustration, if a followed by t occurs often, the method can add at as a combined piece. It then counts pairs again and continues. The resulting vocabulary and merge rules are saved and used to split new text; the tokenizer does not normally rebuild its vocabulary for every request.

Other methods construct or use their vocabularies differently. WordPiece, once its vocabulary is built, splits a word by taking the longest available piece at the current position, then continuing with the remainder. In an invented vocabulary containing un, unbeliev, ##able, and ##believable, it would split unbelievable as unbeliev plus ##able. The ## marker identifies a continuation inside a word; it is not text the user typed. If this procedure cannot finish the word, it normally returns an unknown-word marker instead of backtracking to try a shorter earlier piece. WordPiece walkthrough.

A unigram tokenizer assigns probabilities to candidate pieces and can compare complete alternative splits by multiplying their piece probabilities. Suppose abc can be split as a + bc or ab + c. With invented probabilities 0.2, 0.1, 0.3, and 0.2 respectively, the scores are 0.2 × 0.1 = 0.02 and 0.3 × 0.2 = 0.06. Choosing the higher-scoring split gives ab + c. When building the vocabulary, the training procedure starts with many candidate pieces and removes pieces whose removal hurts its scoring of training text least. The next chapter explains those training steps in depth. Unigram walkthrough.

These choices can produce different pieces and token counts for identical text. Once a split is selected, all these methods look up the pieces' vocabulary IDs and pass that list of IDs to the model.

A vocabulary can reserve entries for instructions about the format rather than ordinary text. These special tokens can mark where text starts or ends, who is speaking, or where data from an image or another program begins. A padding token can fill unused positions when examples of different lengths are processed together; later calculations must know which positions are padding.

For a conversation, software also needs to distinguish your message from the assistant's reply. A chat template supplies the model's expected format for these roles and their text. An application might display the label “user,” while the model's actual input uses a special marker and other formatting. Sending the right words in the wrong format can therefore give a different input from the one the model was trained to follow.

When estimating an input limit or a bill charged per token, run the text through the tokenizer for that model. Counting words or characters gives a different quantity. A new name usually becomes a sequence of existing pieces. The system can therefore represent it without creating new vocabulary entries or retraining the model during your request.

Interview question: Is tokenization already understanding the sentence?

python3 -m venv /tmp/llm-tokenizer-lab-env
/tmp/llm-tokenizer-lab-env/bin/python -m pip install tokenizers==0.22.2
/tmp/llm-tokenizer-lab-env/bin/python - <<'PY'
import hashlib
import urllib.request
import tokenizers
from tokenizers import Tokenizer

revision = "86b5e0934494bd15c9632b12f734a8a67f723594"
url = ("https://huggingface.co/google-bert/bert-base-uncased/resolve/"
       + revision + "/tokenizer.json")
with urllib.request.urlopen(url, timeout=30) as response:
    payload = response.read()
assert hashlib.sha256(payload).hexdigest() == (
    "ce64fce797c24f68df90b40a3f74f579b336a493db14bd583fd520ea0d8c9a98")
tokenizer = Tokenizer.from_str(payload.decode("utf-8"))
text = "Hello, world!"
plain = tokenizer.encode(text, add_special_tokens=False)
framed = tokenizer.encode(text, add_special_tokens=True)
print("tokenizers:", tokenizers.__version__)
print("input:", repr(text))
print("pieces:", plain.tokens)
print("IDs:", plain.ids)
print("decoded:", repr(tokenizer.decode(plain.ids)))
print("special pieces:", framed.tokens)
print("special IDs:", framed.ids)
print("decoded with specials:",
      repr(tokenizer.decode(framed.ids, skip_special_tokens=False)))
print("decoded skipping specials:",
      repr(tokenizer.decode(framed.ids, skip_special_tokens=True)))
PY

tokenizers: 0.22.2
input: 'Hello, world!'
pieces: ['hello', ',', 'world', '!']
IDs: [7592, 1010, 2088, 999]
decoded: 'hello, world!'
special pieces: ['[CLS]', 'hello', ',', 'world', '!', '[SEP]']
special IDs: [101, 7592, 1010, 2088, 999, 102]
decoded with specials: '[CLS] hello, world! [SEP]'
decoded skipping specials: 'hello, world!'

3. Vectors, embeddings, and tensor shapes

The ID tells the software which vocabulary entry appeared. The next step is to look up an ordered list of numbers stored for that entry. This numerical representation is called the token's embedding; it supplies input to the model's calculations. During training, software changes the embedding values so that the model's later predictions can improve.

A list such as [0.72, 0.15, −0.33, 0.90] is a vector. Order matters: the first number has a different place in the later calculation from the second. Each slot is a coordinate; the value in the first coordinate here is 0.72. This list contains four numbers, so we say its width is four.

A familiar vector is a student's marks in Math, English, and Science: [92, 81, 95]. You know what each slot means because we assigned those subjects in advance. A model's embedding is different in this respect. Training adjusts the numbers for their usefulness in later calculations, without assigning each slot a subject-like label such as “animal” or “location.” A useful distinction may depend on several numbers together. This spreading of information across multiple coordinates is called a distributed representation. The school-marks analogy explains the ordered list, not the meaning of each model coordinate.

How the initial vector is obtained

The embedding vectors are saved in an embedding table. Each vocabulary entry has one row, and the row contains that entry's vector. A vocabulary of 10,000 entries with four numbers per entry needs 10,000 rows and four columns. We use the letter E as a short name for this table. The word shape below asks how many rows and columns it contains:

$$\operatorname{shape}(E) = 10{,}000 \times 4$$

For ID 9821, the computer selects the row labelled 9821. The subscript in E₉₈₂₁ means “that row of E”:

$$E_{9821} = \left[0.72,0.15,-0.33,0.90\right]$$

No similarity search is needed: the ID specifies exactly which row to read. If the same ID appears twice in the input, both occurrences initially select the same embedding vector. Later calculations can transform the two occurrences into different vectors because they occur at different places and have different surrounding text. Section 6 follows that change.

Width is not depth or sequence length

Models perform several stages of calculation on these vectors. A stage is often called a layer. Keep three counts separate: width is how many coordinates are in one position's vector; depth is how many layers the model uses; and sequence length is how many token positions are being processed. For example, five tokens, four numbers per token, and three successive layers give length 5, width 4, and depth 3. Each count answers a different question.

The symbol $d_{\text{model}}$ is the usual name for the model's main working width. The d stands for dimension: the number of coordinates. If the width is 4,096, each position carries a vector of 4,096 coordinates. It does not mean 4,096 words or 4,096 separately labelled facts.

A vector's magnitude measures the size of its values taken together. This is another quantity, distinct from the number of slots. One common measure, Euclidean magnitude, squares each value, adds the squares, and takes the square root. Squaring means multiplying a number by itself: 3² is 3 × 3 = 9. The square root symbol √ asks for the nonnegative number whose square gives the number inside it: √25 is 5 because 5 × 5 = 25. For [3, 4], the magnitude is therefore $\sqrt{3^2+4^2}=\sqrt{9+16}=5$. The vector has two slots but magnitude five. You will use this distinction when comparing vectors in section 5.

Several positions make a table

We can arrange several vectors in a table. Such a table is called a matrix. Put one token position's vector in each row. For this separate invented example, use three numbers per position instead of the earlier four. Four positions then give four rows and three columns. The letter X names the following example table; “feature” here is just a label for a coordinate:

$$X = \begin{array}{c|ccc} & \text{feature 1} & \text{feature 2} & \text{feature 3} \\\\ \hline \text{The} & 0.2 & 0.4 & -0.1 \\\\ \text{cat} & 0.8 & -0.3 & 0.6 \\\\ \text{sat} & -0.2 & 0.9 & 0.1 \\\\ \text{.} & 0.0 & 0.1 & -0.5 \end{array}$$

The shape of this matrix is written as rows × columns. Counting the displayed rows from one, row 2 contains cat's three values. Column 2 contains the second value from each row. Thus shape(X) = 4 × 3 describes a table containing twelve numbers:

$$\operatorname{shape}(X) = 4 \times 3$$

A computer can process several inputs as a group, called a batch. Picture a separate matrix of token vectors for each input. Grouping them helps the computer work on many numbers together; it does not let one person's text read another person's text. The calculations keep the inputs separate.

Suppose a batch contains two inputs, each with three token positions and four numbers per position. It contains 2 × 3 × 4 = 24 numbers. To identify one number you must answer three questions: which input, which token position, and which coordinate? These three choices are its three axes. An organized collection of numbers described by such axes is called a tensor. A single number, called a scalar, needs no indexing axis; a vector needs one; a matrix needs two. A vector with 4,096 entries still needs only one index to select an entry, so it has one axis containing 4,096 positions.

Interview question: What do the numbers in an embedding mean?

4. What is learned, and what changes while answering

There are three different kinds of numbers to distinguish: numbers saved in the model, numbers calculated while processing an input, and settings that control how the model is built or trained. We will identify each in a small example before using their technical names.

Imagine one small step inside a model multiplies an incoming value by a stored number. The incoming value is 2 and the stored multiplier is 3, so the result is 2 × 3 = 6. The result 6 goes into a later calculation; it is not yet the answer shown to the user. This is an invented example of one operation, not a complete language model.

A parameter is a stored number that training can adjust

The multiplier 3 is a parameter, also called a weight. It is saved as part of the model. Training can change it to make the model's predictions better. Numbers in the embedding table are parameters too. “Coefficient” is a mathematical word you will encounter for a multiplying number such as this 3.

To see why training would change a number, return to The backup is stored in the archive. Suppose the training software supplies The backup is stored in the and asks for the next token. In this simplified example, that next token is archive. The training software has the complete sentence, so it knows this target even though the model must predict it from the earlier text. The software compares the model's probabilities with that known next token and calculates how to adjust the parameters. Section 19 works through that adjustment. The software repeats the process with many examples; whether the resulting model is useful depends on the examples and on checks using new data.

An activation is a working result calculated for an input

The result 6 is an activation: a number produced while the model is doing its work on this input. Now supply 5 to the same multiplication step. The stored multiplier is still 3, but the result is 5 × 3 = 15. The activation changed because the input changed. We did not have to change the parameter.

Real model steps usually calculate whole vectors or matrices of values at once. We also call those calculated vectors or matrices activations. A vector carried for one token through the model is often called a hidden state, which section 6 will illustrate. Both terms describe values being calculated, rather than a separate kind of stored knowledge.

During inference—using the trained model to calculate an answer—the model normally keeps its parameters fixed and calculates activations from the supplied input. Some working results are kept temporarily to avoid calculating them again. Saving a result for reuse does not turn it into a trained parameter. During training, software also calculates activations, then uses information from those calculations to work out parameter changes.

flowchart TB
    A["First input: 2"] --> C["Multiply by stored parameter: 3"]
    C --> D["Calculated activation: 6"]
    E["New input: 5"] --> F["Multiply by the same parameter: 3"]
    F --> G["New calculated activation: 15"]

The two paths use the same rule and stored multiplier. Different inputs produce different working results. That is how a trained model can respond differently to different requests while keeping its stored parameters unchanged.

A hyperparameter is a setting used to build or train the model

Before running the example, someone had to choose what calculation to build and how training would change its parameters. Choices of that kind are hyperparameters. The person building or training the model can choose them directly, or set up software to compare candidate settings. They are not the individual numbers that the model's usual training update learns from each example.

Consider two different choices. A layer count chooses how many stages of calculation the model has: for example, 12 stages rather than 24. A learning rate controls how large a training update is. In a simple update rule, suppose the software calculates a proposed correction of −0.4 to our parameter 3. With learning rate 0.1, it applies 0.1 × (−0.4) = −0.04, producing a new parameter of 3 − 0.04 = 2.96. The learning rate scales the update; it is not the parameter being updated.

The software rule that calculates and applies parameter updates is called an optimizer. You do not need its internal algorithm to distinguish these quantities: 3 is the current stored parameter, 6 is a result calculated for input 2, and 0.1 is a chosen setting controlling an update. Training settings can follow a planned schedule or an automatic selection procedure; “hyperparameter” does not mean “a number that can never change.”

The same distinction applies to larger examples. A matrix of learned multipliers is a parameter matrix. A matrix of values just calculated for the current text is an activation matrix. Later, attention will calculate numbers describing how much different positions contribute. Those are called attention weights, but despite the shared word “weights,” they are calculated activations, not the stored parameters that training adjusts. Section 9 calculates them explicitly.

A checkpoint is a saved model that can be loaded later

After training has improved the stored numbers, we need to keep them. A checkpoint is a saved copy of those parameters, together with the information needed to reconstruct the model's layout—for example, how many layers it has and the vector width at each layer. Loading it lets a program use the saved model without training again from the beginning.

To continue an interrupted training run closely, saving parameters alone may not be enough. The update rule may keep records of earlier updates; a learning-rate schedule must know which step it has reached; and random choices depend on a generator's saved state. A resumable training checkpoint can save these records too. This kind of saved training state is different from keeping a request's temporary working results.

Remember it through the example: save the multiplier, calculate the result, choose the training settings. Training can revise the saved multiplier. Answering a new request normally calculates new results with the saved multipliers. Consequently, a model can follow a new instruction placed in a request without permanently learning that instruction.

Practice aloud: our model now receives 4 and calculates 4 × 3 = 12. Which value is a parameter, which is an activation, and did this new input train the model?

5. What a learned matrix actually does

We now have a vector for each token. The next problem is to transform that vector into a new one. A weight matrix, a table of learned multipliers, lets the computer do this: each column specifies how to calculate one output coordinate. We will choose simple multipliers so you can check every step. Training would adjust multipliers of this kind in a real model.

Take the input [2, 3, 4]. We want two results. To get the first, add the first and third input values: 2 + 4 = 6. To get the second, add the second and third: 3 + 4 = 7. We can express both rules using multiplications: multiply an included value by 1 and an excluded value by 0. The symbols y₁ and y₂ name the first and second results:

$$y_1 = 2(1) + 3(0) + 4(1) = 6$$

$$y_2 = 2(0) + 3(1) + 4(1) = 7$$

Put the first rule's multipliers [1, 0, 1] in the first column of a table. Put the second rule's [0, 1, 1] in the second column. The table has three rows because each rule needs a multiplier for each of the three input values. It has two columns because we want two output values. Call the table W:

$$W = \begin{bmatrix} 1 & 0 \\\\ 0 & 1 \\\\ 1 & 1 \end{bmatrix}$$