Tensor + Autograd Warm-up
Create a tensor `x` with `requires_grad=True`, compute `y = (x**2).sum()`, call `backward()`, and print `x.grad`.
Data / Knowledge library
PyTorch
python-first deep learning and tensor framework
Why learn PyTorch?
PyTorch helps you build, train, and deploy ML models using tensors, automatic differentiation, GPU acceleration, and distributed tooling, so you can focus on model design instead of manual gradient math and low-level kernels.
Where it came from
2016: Open-sourced by Facebook AI Research (FAIR). Early adoption grew from dynamic eager execution and Pythonic APIs. 2018: Caffe2 and PyTorch development paths were unified. 2022: PyTorch Foundation launched under the Linux Foundation. 2023+: PyTorch 2.x compiler stack (`torch.compile`) became a major performance focus.
- 2016: Open-sourced by Facebook AI Research (FAIR).
- Early adoption grew from dynamic eager execution and Pythonic APIs.
- 2018: Caffe2 and PyTorch development paths were unified.
- 2022: PyTorch Foundation launched under the Linux Foundation.
- 2023+: PyTorch 2.x compiler stack (`torch.compile`) became a major performance focus.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Create a tensor `x` with `requires_grad=True`, compute `y = (x**2).sum()`, call `backward()`, and print `x.grad`.
- Train `nn.Linear(1,1)` for one batch using MSE loss and SGD. Log loss before and after one optimizer step.
- Move model and batch tensors to `cuda` when available, train for 3 epochs, and report average epoch loss.
- Run inference with `model.eval()` and `torch.no_grad()`. Compare outputs to train mode with dropout enabled.
- Prepare a small model for deployment using `torch.export` (or equivalent in your installed version) and note unsupported ops.
More ways to practice
Single-Batch Training Step
Train `nn.Linear(1,1)` for one batch using MSE loss and SGD. Log loss before and after one optimizer step.
GPU-Aware Mini Loop
Move model and batch tensors to `cuda` when available, train for 3 epochs, and report average epoch loss.
Evaluation Safety Check
Run inference with `model.eval()` and `torch.no_grad()`. Compare outputs to train mode with dropout enabled.
Export Readiness
Prepare a small model for deployment using `torch.export` (or equivalent in your installed version) and note unsupported ops.
Documentation and sources
- PyTorch Docs (pytorch.org)
- PyTorch Tutorials (pytorch.org)
- Get Started (pytorch.org)
- PyTorch GitHub (github.com)
Catalog research notes
- PyTorch 2.x made `torch.compile` a central path for speeding up eager models.
- `torch.export` improved stable graph capture for deployment.
- ExecuTorch expanded mobile/edge inference options.
- Distributed training features like FSDP and DTensor continued maturing for large models.
Connect the ideas
Enables
Explore next
NumPyONNXTensorFlowJAXHugging Face TransformersCNN (Convolutional Neural Network)Diffusion ModelsDVCFederated LearningGraph Neural Networks (GNNs)Hugging Face HubKerasKnowledge DistillationLlamaMLflowMoE (Mixture of Experts)Reinforcement Learning (RL)scikit-learnNVIDIA TensorRTVision Transformer (ViT)vLLM
Real World Example
Optimizing High-Throughput Image Classification Pipelines
Retail logistics firms using automated visual inspection systems to categorize incoming inventory.
The business problem
High latency in data ingestion from storage to GPU memory causes significant underutilization of compute resources, leading to bottlenecked training cycles.
How this topic solves it
Utilizing PyTorch DataLoader optimizations to overlap data transfer with model computation, ensuring the GPU remains saturated with data.
- Configure the DataLoader with num_workers to enable multi-process data loading.
- Enable pin_memory to allow faster data transfer from CPU host memory to GPU device memory.
- Implement persistent_workers to avoid the overhead of re-creating worker processes between epochs.
- Profile the training loop using the PyTorch Profiler to identify remaining I/O wait times.
Visualizes the flow of data from storage to the GPU, highlighting the optimization points.
Raw image dataset
Preprocessing and augmentation
Host-side staging area
Model training execution
- Storage → Parallel Fetch → CPU Workers
- CPU Workers → pin_memory transfer → Pinned Memory
- Pinned Memory → High-speed copy → GPU
Implementation example
This configuration maps directly to the solution by utilizing num_workers for parallel fetching and pin_memory for optimized GPU transfer.
loader = DataLoader(dataset, batch_size=32, num_workers=4, pin_memory=True, persistent_workers=True) # Enables multi-process loading and memory pinningExpected business outcome
Improved training throughput and reduced idle time for GPU hardware, allowing for faster model iteration cycles.
Limitations and checks
Excessive worker processes can lead to high CPU memory consumption; monitor system RAM usage to ensure it does not exceed available capacity.