Original Academy
Legacy mirror · noindex · upstream phase 3
Deep Learning Core
Neural networks from first principles. No frameworks until you build one.
Provenance: this phase outline and its lesson readings are preserved from ai-engineering-from-scratch by Rohit Ghumare under the MIT License. InterviewsVector does not claim authorship. These archive pages remain available for old links and progress, but are excluded from indexing.
Attributed readings
- 01The Perceptron: Where It All StartedThe perceptron is the atom of neural networks. Split it open and you find weights, a bias, and a decision.
- 02Multi-Layer Networks & Forward PassOne neuron draws a line. Stack them, and you can draw anything.
- 03Backpropagation from ScratchBackpropagation is the algorithm that makes learning possible. Without it, neural networks are just expensive random number generators.
- 04Activation Functions: ReLU, Sigmoid, GELU & WhyWithout nonlinearity, your 100-layer network is a fancy matrix multiply. Activations are the gates that let neural networks think in curves.
- 05Loss Functions: MSE, Cross-Entropy, ContrastiveYour network makes a prediction. The ground truth says otherwise. How wrong is it? That number is the loss. Pick the wrong loss function and your model optimizes for the wrong t…
- 06Optimizers: SGD, Momentum, Adam, AdamWGradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data.
- 07Regularization: Dropout, Weight Decay, BatchNormYour model gets 99% on training data and 60% on test data. It memorized instead of learning. Regularization is the tax you impose on complexity to force generalization.
- 08Weight Initialization & Training StabilityInitialize wrong and training never starts. Initialize right and 50 layers train as smoothly as 3.
- 09Learning Rate Schedules & WarmupThe learning rate is the single most important hyperparameter. Not the architecture. Not the dataset size. Not the activation function. The learning rate. If you tune nothing el…
- 10Build Your Own Mini FrameworkYou have built neurons, layers, networks, backprop, activations, loss functions, optimizers, regularization, initialization, and LR schedules. All as separate pieces. Now wire t…
- 11Introduction to PyTorchYou built the engine from pistons and crankshafts. Now learn the one everyone actually drives.
- 12Introduction to JAXPyTorch mutates tensors. TensorFlow builds graphs. JAX compiles pure functions. That last one changes how you think about deep learning.
- 13Debugging Neural NetworksYour network compiled. It ran. It produced a number. The number is wrong and nothing crashed. Welcome to the hardest kind of debugging -- the kind where there is no error message.