AI Tutorial: A Guide to AI LLMs, from Basics to Depth

#

This article takes you deep into the core concepts of large language models — from basic principles to vector representation — building a complete knowledge system step by step.

1. Foundations of AI Application Development

#

1.1 Basic Principles and Concepts

#

In Plain Words

#

• Core mechanism: predict the next word from the previous one — like a word-chain game
• How it works: output is generated token by token

A Deeper Look

#

A large AI model involves two key stages:

Core Technical Components

#

• Transformer architecture

  • Made of an encoder and a decoder
  • The core is the attention mechanism, which enables efficient information processing

• Embedding and positional encoding

  • Convert text into numeric vectors a computer can process
  • Add ordering information so the model understands sequence in language

• Multi-head attention

  • The core computational step
  • Decides which content matters more, which in turn shapes the output

2. Core Concepts

#

2.1 Basic Terms

#

• AGI (Artificial General Intelligence): the end goal of large models — human-level intelligence
• LLM (Large Language Model): short for large language model
• Chat products vs LLMs: the difference between the application layer and the model layer

2.2 How Models and Products Relate

#

3. Tokens: The Smallest Unit of AI Language

#

3.1 What Is a Token?

#

A token = the smallest unit of text a model processes. It is neither strictly a character nor a fixed word — it is a fragment cut out of the text by a compression scheme.

How tokens are split

#

• English: usually split into word fragments

  • "I love apples" → ["I", " love", " apple", "s"]

• Chinese: usually split by character or short word

  • "我喜欢苹果" → ["我", "喜欢", "苹果"]
  • (the exact granularity depends on the tokenizer)

• Special tokens: start/end markers, newlines, tool-call boundaries, and so on

💡 Intuition: tokens are like the particles of AI language — a model reads and generates one token at a time.

3.2 How Are Tokens Cut?

#

Most LLMs use algorithms such as BPE/Unigram:

• Find the most frequent character combinations in the text and assign each a vocabulary ID
• This way both single characters and common words or word fragments can be represented
• It balances efficiency (fewer tokens) against generalization (rare words can still be split apart)

⚠️ Important: the same sentence may produce a different token count under different models or vocabularies.

3.3 How Tokens Affect Products

#

📊 Rules of Thumb for Token Estimation

#

• English: ~3-4 words ≈ 1 token (100 tokens ≈ 75 English words)
• Chinese: 1 character/word ≈ 0.6 tokens (varies by vocabulary)
• Note: for real counts, always defer to the specific model’s tokenizer

4. Vectors: The Foundation of AI Understanding

#

4.1 What Is a Vector?

#

In mathematics, a vector is a quantity with both magnitude and direction — or, more generally, an ordered list of numbers.

The simplest vector can be written as:

1

(2, 3)

This means:

• Move 2 units along the x axis
• Move 3 units along the y axis

It can represent the position of a point (an offset relative to the origin), or an arrow starting at the origin (direction + length).

4.2 🧭 A Geometric Example

#

Imagine walking around on a plane:

• The vector (2, 3) means “walk right 2, walk up 3”
• The vector (-1, 4) means “walk left 1, walk up 4”

These numbers act like coordinates, telling you where to go in space.

📊 Draw it out:

• Origin at (0, 0)
• Endpoint at (2, 3) → the arrow pointing there is the “vector”

4.3 💡 Vectors as Features

#

When we apply this concept to AI, a vector is no longer just a position — it can represent features or meaning.

Example 1: color vectors

#

Suppose we describe a color’s red, green, and blue components with 3 numbers:

1
2
3

Red:   (255, 0, 0)
Green: (0, 255, 0)
Blue:  (0, 0, 255)

This is a 3-dimensional vector space. Every color is a 3-D point in that space, which lets us compute similarity between colors.

Example 2: human feature vectors

#

Suppose we want to describe a person with numbers:

A person then becomes (25, 180, 70) — again a 3-dimensional vector. To compare how alike two people are, we compute the distance between their vectors.

For example:

1
2

A(25, 180, 70)
B(26, 178, 72)

Their vector distance is small → the two have similar features.

Example 3: semantic vectors for words

#

In natural language processing (NLP), models turn every word into a high-dimensional vector (say, 768 dimensions).

The model then discovers:

king - man + woman ≈ queen

In other words, mathematical relations between vectors can express semantic relations. That is why we say:

Vectors let machines understand meaning, not merely see text.

5. Token Management in the LLM Workflow

#

5.1 The End-to-End Workflow

#

Here is the main pipeline of a conversational Q&A application (with each step’s relationship to tokens):

1. User input

#

• Raw text: for example, “help me write a post-interview thank-you email”
• ✅ Key point: length is uncontrollable — clean up and cap it downstream

2. Preprocessing (cleaning / structuring)

#

• Strip meaningless whitespace and normalize the text format
• Inject role/tone requirements (prompt templating)
• ✅ Key point: fewer dirty tokens — convey intent more clearly with fewer tokens

3. Retrieval (optional: RAG)

#

• Vectorize the user’s question → find relevant documents in a vector store → bring back several passages
• Splice those passages into the prompt as context
• ✅ Key point: retrieved passages must be trimmed and summarized, or you risk blowing the context window

4. Assemble the final prompt (input sequence)

#

• Composition: system instructions + tool/function definitions + retrieved evidence + conversation history + the current user question
• Then the tokenizer cuts all of it into tokens
• ✅ Key point: count input tokens; when close to the limit:
  • keep the most relevant evidence first
  • summarize / sliding-window the history
  • cap generation (max_tokens)

5. Model forward pass and the generation loop (decoding)

#

• The model reads the input tokens → outputs a probability distribution over the next token

• A sampling strategy (greedy/temperature/top-p…) picks the next token

• The new token is appended to the context, and the next one is predicted — round and round

• Until a stop condition is met: an end token appears / max_tokens is reached / a stop word is hit

• ✅ Key point:

  • Output tokens are produced as a stream
  • The more divergent the sampling (high temperature), the more tokens and the livelier the style may get
  • A sensible max_tokens controls cost and latency

6. Detokenization

#

• The model outputs a token sequence that must be turned back into a text string
• ✅ Key point: seemingly trivial spaces and indentation are actually part of tokens

7. Post-processing

#

• Structured extraction, formatting into Markdown/JSON
• Sensitive-information / compliance filtering
• Summarizing results or multi-round tool calls
• ✅ Key point: cutting wasted output tokens lowers cost and speeds things up

8. Logging and billing

#

• Record input/output token counts, latency, failures, and retries
• Combine with quality metrics to iterate on prompts and retrieval strategy

🔄 Flow diagram:

5.2 🎯 Real-World Case Studies

#

Case 1: why long context does not mean high quality

#

• Problem: stuff all 20 pages of a document into the prompt, token count explodes → forced to truncate
• Result: the 2 most relevant passages get dropped
• Fix: retrieval + passage scoring + summarization — keep the most critical information with fewer tokens

Case 2: controlling cost and latency

#

• Need: the user only wants a bullet list — there is no reason to let the model write 1,000 tokens of prose
• Strategy: set max_tokens=120 + prompt it with “answer in 6 bullets, each 20 words or fewer”
• Effect: cost and latency drop immediately, and the output matches the need

Case 3: how Chinese and English tokens differ in practice

#

• Observation: 100 Chinese characters and 100 English words usually produce different token counts
• Advice: at the product level, do rate limiting and budgeting based on real token counts

5.3 🛠️ Product and Engineering Playbook

#

Core strategies

#

1. Count tokens in real time: count once after assembling the prompt and before calling the model; near the limit, trigger a trimming strategy
2. Layered context: system instructions (short and stable) + highly relevant evidence (short and sharp) + recent dialogue turns (after summarization)
3. Output caps and stop words: configure max_tokens and stop words per scenario to keep answers from rambling
4. Cap retrieved passage length: set a maximum token count per passage and trim within sentences (keep only a few words around each hit)
5. Metrics loop: log input_tokens/output_tokens/latency/success_rate and A/B test prompts and retrieval strategies
6. Multilingual scenarios: token efficiency differs across languages; when necessary, do language detection + translation into a single language before the model sees the input

6. 🧠 Key Takeaways

#

6.1 Concepts at a Glance

#

6.2 Recap

#

1. Basic principle: predict the next word, generate token by token
2. Core architecture: Transformer + attention mechanism
3. Key concept: vector representation lets machines understand semantics
4. Real-world use: the complete chain from model to product
5. Token management: the key to controlling length, cost, and quality

6.3 💡 Study Advice

#

• Understand the token concept: a crucial step into AI — tokens underpin how modern models process language
• Practice token optimization: in product development, good token management markedly improves results and lowers cost
• Master vector representation: understand how human language becomes math a machine can digest

🚀 Next step: if you’d like, I can draw you a diagram of the LLM workflow and its token interaction points, or write a small script that counts tokens for a given text across different models and estimates cost and latency.

📚 Further Reading

#

🔗 AI LLM Systematic Tutorials Series

#

1. [This post] A Complete Guide to AI LLMs — tokens and vectors explained in depth, from the ground up
2. Transformer Architecture Deep Dive — the attention mechanism and the core technology behind LLMs
3. Prompt Engineering Complete Guide — a hands-on guide from prompt engineering to context engineering
4. AI Technical Glossary — 270+ terms and a dictionary of the AI technology landscape

🎯 Suggested Learning Path

#

• Beginners: read this post first for the fundamentals, then use the glossary to consolidate terminology
• Developers: after this post, focus on the Prompt Engineering hands-on guide
• Researchers: go deep on the Transformer architecture to master the core principles of AI