InterviewsVector
Original Academy

Legacy mirror · noindex · upstream phase 5

NLP: Foundations to Advanced

Language is the interface to intelligence.

Provenance: this phase outline and its lesson readings are preserved from ai-engineering-from-scratch by Rohit Ghumare under the MIT License. InterviewsVector does not claim authorship. These archive pages remain available for old links and progress, but are excluded from indexing.

Attributed readings

  1. 01Text Processing: Tokenization, Stemming, LemmatizationLanguage is continuous. Models are discrete. Preprocessing is the bridge.
  2. 02Bag of Words, TF-IDF & Text RepresentationCount first, think later. TF-IDF still beats embeddings on well-defined tasks in 2026.
  3. 03Word Embeddings: Word2Vec from ScratchA word is the company it keeps. Train a shallow net on that idea and geometry falls out.
  4. 04GloVe, FastText & Subword EmbeddingsWord2Vec trained one embedding per word. GloVe factorized the co-occurrence matrix. FastText embedded the pieces. BPE bridged to transformers.
  5. 05Sentiment AnalysisThe canonical NLP task. Most of what you need to know about classical text classification shows up here.
  6. 06Named Entity Recognition (NER)Pull the names out. Sounds easy until you deal with ambiguous boundaries, nested entities, and domain jargon.
  7. 07POS Tagging & Syntactic ParsingGrammar was unfashionable for a while. Then every LLM pipeline needed to validate structured extraction, and it came back.
  8. 08Text Classification — CNNs & RNNs for TextConvolutions learn n-grams. Recurrences remember. Both are superseded by attention. Both still matter on constrained hardware.
  9. 09Sequence-to-Sequence ModelsTwo RNNs pretending to be a translator. The bottleneck they hit is the reason attention exists.
  10. 10Attention Mechanism — The BreakthroughThe decoder stops squinting at a compressed summary and starts looking at the whole source. Everything after this is attention plus engineering.
  11. 11Machine TranslationTranslation is the task that paid for NLP research for thirty years and keeps paying now.
  12. 12Text SummarizationExtractive systems tell you what the document said. Abstractive systems tell you what the author meant. Different tasks, different pitfalls.
  13. 13Question Answering SystemsThree systems shaped modern QA. Extractive found spans. Retrieval-augmented grounded them in documents. Generative produced answers. Every modern AI assistant is a mix of the th…
  14. 14Information Retrieval & SearchBM25 is precise but brittle. Dense casts a wide net but misses keywords. Hybrid is the 2026 default. Everything else is tuning.
  15. 15Topic Modeling: LDA, BERTopicLDA: documents are mixtures of topics, topics are distributions over words. BERTopic: documents cluster in embedding space, clusters are topics. Same goal, different decompositi…
  16. 16Text GenerationIf a word is surprising, the model is bad. Perplexity makes surprise a number. Smoothing keeps it finite.
  17. 17Chatbots: Rule-Based to NeuralELIZA replied with pattern matches. DialogFlow mapped intents. GPT answered from weights. Claude runs tools and verifies. Each era solved the previous one's worst failure.
  18. 18Multilingual NLPOne model, 100+ languages, zero training data for most of them. Cross-lingual transfer is the practical miracle of the 2020s.
  19. 19Subword Tokenization: BPE, WordPiece, Unigram, SentencePieceWord tokenizers choke on unseen words. Character tokenizers blow up sequence length. Subword tokenizers split the difference. Every modern LLM ships on one.
  20. 20Structured Outputs & Constrained DecodingAsk an LLM for JSON. Get JSON most of the time. In production, "most" is the problem. Constrained decoding turns "most" into "always" by editing the logits before sampling.
  21. 21NLI & Textual Entailment"t entails h" means a human reading t would conclude h is true. NLI is the task of predicting entailment / contradiction / neutral. Boring on the surface, load-bearing in produc…
  22. 22Embedding Models Deep DiveWord2Vec gave you a vector per word. Modern embedding models give you a vector per passage, cross-lingual, with sparse, dense, and multi-vector views, sized to fit your index. P…
  23. 23Chunking Strategies for RAGChunking configuration influences retrieval quality as much as the choice of embedding model (Vectara NAACL 2025). Get chunking wrong and no amount of reranking saves you.
  24. 24Coreference Resolution"She called him. He did not answer. The doctor was at lunch." Three references to two people and nobody is named. Coreference resolution figures out who is who.
  25. 25Entity Linking & DisambiguationNER found "Paris." Entity linking decides: Paris, France? Paris Hilton? Paris, Texas? Paris (the Trojan prince)? Without linking, your knowledge graph stays ambiguous.
  26. 26Relation Extraction & Knowledge Graph ConstructionNER found the entities. Entity linking anchored them. Relation extraction finds the edges between them. A knowledge graph is the sum of nodes, edges, and their provenance.
  27. 27LLM Evaluation: RAGAS, DeepEval, G-EvalExact-match and F1 miss semantic equivalence. Human review does not scale. LLM-as-judge is the production answer — with enough calibration to trust the number.
  28. 28Long-Context Evaluation: NIAH, RULER, LongBench, MRCRGemini 3 Pro advertises 10M tokens of context. At 1M tokens, 8-needle MRCR drops to 26.3%. Advertised ≠ usable. Long-context evaluation tells you the actual capacity of the mode…
  29. 29Dialogue State Tracking"I want a cheap restaurant in the north... actually make it moderate... and add Italian." Three turns, three state updates. DST keeps the slot-value dict in sync so the booking …