Build a String Tokenizer in JavaScript

The problem

Write a tokenize(input) that splits source text like name = "John Doe", age = 30 into a list of typed tokens — identifiers, strings (with escape support), numbers, and punctuation — skipping whitespace.

This question is a deliberate staircase: it's the first half of every parser question (JSON.parse is tokenizer + grammar), and it tests whether you can manage a cursor over a string without off-by-ones — the same skill as trim, scaled up.

The trap in the naive framing: input.split(/[\s,=]+/) can't work, because delimiters inside quotes must not split, and escaped quotes inside strings must not terminate them. Once quoting exists, you need a real scanner.

Design: emit typed tokens, not strings

A split-style tokenizer returns raw fragments and forces every consumer to re-guess what each fragment was. A real lexer classifies as it scans:

{ type: "identifier", value: "name" }
{ type: "punct",      value: "="    }
{ type: "string",     value: "John Doe" }   // quotes consumed, escapes decoded
{ type: "number",     value: 30    }        // parsed, not the raw digits

Saying "I'll emit typed tokens so the parser downstream doesn't re-parse" is the design sentence interviewers wait for.

Implementation

function tokenize(input) {
  const tokens = [];
  let i = 0; // the cursor — all state lives here
 
  const isDigit = (c) => c >= "0" && c <= "9";
  const isIdentStart = (c) => /[A-Za-z_$]/.test(c);
  const isIdentChar = (c) => /[A-Za-z0-9_$]/.test(c);
 
  while (i < input.length) {
    const char = input[i];
 
    // 1. Skip whitespace — it separates tokens but produces none
    if (/\s/.test(char)) {
      i++;
      continue;
    }
 
    // 2. Quoted strings: consume quotes, decode escapes
    if (char === '"' || char === "'") {
      const quote = char;
      i++; // past the opening quote
      let value = "";
      while (i < input.length && input[i] !== quote) {
        if (input[i] === "\\") {
          i++; // past the backslash
          if (i >= input.length) throw new SyntaxError("Unterminated escape");
          const esc = input[i];
          value +=
            esc === "n" ? "\n" :
            esc === "t" ? "\t" :
            esc === "r" ? "\r" :
            esc; // \" \' \\ and unknown escapes: the char itself
        } else {
          value += input[i];
        }
        i++;
      }
      if (i >= input.length) {
        throw new SyntaxError(`Unterminated string starting with ${quote}`);
      }
      i++; // past the closing quote
      tokens.push({ type: "string", value });
      continue;
    }
 
    // 3. Numbers (integer, decimal, leading minus)
    if (isDigit(char) || (char === "-" && isDigit(input[i + 1]))) {
      let start = i;
      if (char === "-") i++;
      while (i < input.length && isDigit(input[i])) i++;
      if (input[i] === "." && isDigit(input[i + 1])) {
        i++;
        while (i < input.length && isDigit(input[i])) i++;
      }
      tokens.push({ type: "number", value: Number(input.slice(start, i)) });
      continue;
    }
 
    // 4. Identifiers / keywords
    if (isIdentStart(char)) {
      let start = i;
      while (i < input.length && isIdentChar(input[i])) i++;
      tokens.push({ type: "identifier", value: input.slice(start, i) });
      continue;
    }
 
    // 5. Single-character punctuation
    if (",=:;(){}[]".includes(char)) {
      tokens.push({ type: "punct", value: char });
      i++;
      continue;
    }
 
    throw new SyntaxError(`Unexpected character '${char}' at position ${i}`);
  }
 
  return tokens;
}

Dry run

tokenize('name = "John \\"JD\\" Doe", age = -30.5');
// [
//   { type: "identifier", value: "name" },
//   { type: "punct",      value: "=" },
//   { type: "string",     value: 'John "JD" Doe' },  // escapes DECODED
//   { type: "punct",      value: "," },
//   { type: "identifier", value: "age" },
//   { type: "punct",      value: "=" },
//   { type: "number",     value: -30.5 },             // one token, parsed
// ]

Walk the interesting part aloud: at the \\ inside the string, the cursor advances past the backslash, reads ", appends a literal quote, and — critically — does not treat it as the string terminator. That interplay (escape consumption before terminator checking) is the heart of the question.

The invariants that keep you honest

Cursor-based scanners live or die on two rules; state them and your implementation debugs itself:

1. Every loop iteration makes progress — each branch either advances i or throws. A branch that forgets to advance is the classic infinite loop.
2. Each branch owns its token completely — it consumes the entire token (including closing quotes) and continues. No branch leaves the cursor mid-token for someone else.

Edge cases interviewers probe

• Unterminated string ('"abc') — must throw with a useful message, not loop forever or return a garbage token. This is the #1 planted input.
• Escaped backslash before a closing quote ("a\\\\") — the string is a\ followed by a real terminator; ordering of the escape check handles it.
• Adjacent delimiters (a,,b) — two punct tokens, no phantom empty tokens between them. (Typed emission gives you this for free; split approaches emit "".)
• Numbers vs minus signs — a-3 vs -3: is - part of the number or an operator? State your policy (here: minus binds to a digit only when it starts a token). Real lexers resolve this at the parser level; knowing the ambiguity exists is the point.
• Empty input → [], no special-casing needed — a sign the loop structure is right.

Common mistakes

• Reaching for split + regex, then patching quote-awareness with flags until it collapses — the question is designed to make that path fail.
• Including the quotes in the string token's value (forces every consumer to strip and re-decode).
• Checking the terminator before handling escapes (breaks \").
• Forgetting i++ in a branch → infinite loop under pressure. (State invariant #1 aloud and you'll catch it.)
• No position information in errors — Unexpected character at position 17 versus a shrug is a production-readiness signal.

Follow-up questions

• "Extend it to true/false/null keywords." — check identifier tokens against a keyword set after scanning; this is exactly the step toward JSON.parse.
• "Add line/column tracking for error messages." — track line and lineStart alongside i; every real compiler does.
• "Make it lazy." — turn it into a generator (function* tokenize) yielding one token at a time; connects directly to iterator helpers.
• "What's the complexity?" — O(n): the cursor only moves forward (invariant #1 is the proof).
• "Where would you use this instead of a regex?" — the moment the format has nesting or quoting: query filters, search DSLs, template syntaxes. Regexes can't count; scanners can — that one line is a strong close.