🌼 Lexical Analysis

Before a compiler can understand a program, it faces a simple but annoying problem:

The program is just a long stream of characters.

To the compiler, this looks like someone dumped a bucket of letters, digits, and symbols with no structure.
So the first question is:

“How do I separate this mess into meaningful pieces?”

This job belongs to the lexical analyzer.

---

🌟 What Is Lexical Analysis?

Lexical analysis is the stage where the compiler scans the source code from left to right, grouping characters into small chunks that actually mean something.

These chunks are called tokens.

A token is like a sticker that tells the parser:

• “This word is a variable name.”
• “This thing is a number.”
• “Here comes a plus sign.”
• “This is like a keyword, so don’t confused it with variable.”

If you’ve ever highlighted words in a sentence to understand grammar, you’ve already done something similar.

---

💡 Why Do We Need Tokens at All?

Let’s say someone writes:

count=total+50//update value

Humans can read it easily.
But the compiler sees one giant sequence of characters.

The lexical analyzer steps in and sorts everything out:

• count → identifier
• = → assignment symbol
• total → identifier
• + → operator
• 50 → number
• //update value → comment (ignored)

It’s like taking a blurry photo and sharpening it so every detail becomes clear.

---

🧠 Responsibilities

Here are its main tasks, written in a warm, beginner-friendly way:

1. Splitting characters into proper tokens

It doesn’t guess blindly; it uses rules to decide what counts as an identifier, number, operator, etc.

2. Skipping stuff that humans need but compilers don’t

Things like:

• spaces
• line breaks
• indentation
• comments

These are helpful for the programmer, but completely unnecessary for the parser.

3. Reporting invalid or strange character sequences

If your program contains something odd like:

abc@12

the lexical analyzer is the first to say,
“Hold on… this doesn’t fit any token pattern!”

4. Helping create the symbol table

When it meets a new variable, it helps record its name for later stages.

5. Passing clean, well-organized tokens to the parser

The parser depends on the lexical analyzer just like a reader depends on spaces between words.

---

🍃 Simple Example

Take this small Ompass-style line:

x = y * 25;

The lexical analyzer gently breaks it into:

• x → identifier
• = → operator
• y → identifier
• * → operator
• 25 → number
• ; → special symbol

What started as a plain row of characters becomes a sequence of labeled pieces — ready for parsing.

---

🌳 Diagram — Where the Lexical Analyzer Fits

Here’s a brand-new diagram, drawn plainly so anyone can understand:

   ┌────────────────────────────┐
   │      Raw Source Code       │
   │ (just characters typed in) │
   └───────────────┬────────────┘
                   │
                   ▼
        ┌──────────────────────┐
        │    Lexical Analyzer  │
        └──────────────────────┘
                   │
          produces tokens
                   │
                   ▼
        ┌──────────────────────┐
        │        Parser        │
        └──────────────────────┘

You can think of it like this:

text → [lexical analyzer] → tokens → [parser]

---

About the Author

examhopeinfo@gmail.com

Administrator

Visit Website View All Posts