/ writing
70 technical notes across AI infrastructure, LLM algorithms, CS336, key concepts, and engineering reviews.
A request-path walkthrough of the complete LLM cloud stack, covering inference engines, model serving, orchestration, gateways, observability, security, billing, and the boundaries between them.
A systems-oriented derivation of Kimi Delta Attention, covering channel-wise forgetting, the specialized DPLR transition, chunkwise kernels, the 3:1 KDA/MLA hybrid, controlled evaluations, and the quality-throughput trade-off at 1M context.
A systems-oriented guide to how vLLM-Omni unifies autoregressive, diffusion, encoder, and audio-decoder workloads through stages, orchestration, transport, hierarchical scheduling, and disaggregated deployment.
A deep dive into vLLM PR #51113: how MTP/EAGLE, concurrent chunked prefill, and prefix caching could advertise an incomplete Mamba recurrent state as a complete cache block, and how scheduler alignment restores correctness.
A systems-oriented reading of Kimi k1.5, covering verifiable RL prompts, Long-CoT SFT, online mirror-descent policy optimization, length penalties, partial rollouts, hybrid training and inference deployment, multimodal training, and long2short.
A systems-oriented reading of the DeepSeek-V4 preview report, covering the 1.6T/284B MoE models, CSA+HCA hybrid attention, mHC, Muon, 32T-token pretraining, specialist+OPD post-training, FP4 QAT, KV-cache management, million-token evaluation, and limitations.
A systems-oriented deep dive into DSpark, covering semi-autoregressive drafting, Markov/RNN sequential heads, confidence heads, STS calibration, hardware-aware prefix scheduling, training objectives, DeepSpec implementation, and DeepSeek-V4 deployment tradeoffs.
A shape-by-shape guide to standard Multi-Head Attention, covering Q/K/V projections, per-head computation, output concatenation, prefill, decode, KV cache, and the differences between MHA, GQA, and MQA.
A systems-oriented comparison of GQA, MLA, Linear Attention, and Sparse Attention, and why hybrid memory architectures are emerging as the practical direction.
A derivation of Gated DeltaNet from linear attention, covering error-driven Delta Rule updates, global decay, parallel training, recurrent inference, memory capacity, and comparisons with Transformers and Mamba.
Distilling four deprecated skills and tracing how their useful ideas moved into broader, composable workflows.
Distilling four miscellaneous and two personal skills for Git safety, hooks, migrations, exercise scaffolds, editing, and Obsidian.
A systems-oriented deep dive into DeepEP V2, covering MoE dispatch and combine semantics, rank-level deduplication, ElasticBuffer, EPHandle, direct and hybrid communication, FP8 transport, SM/QP modeling, and communication-compute overlap.
Distilling handoff, teach, and seven experimental skills for persistent sessions, interviews, teaching, and writing.
Distilling codebase-design, architecture improvement, research, triage, and TypeScript module boundaries.
A feedback-loop-centered synthesis of prototype, tdd, implement, diagnosing-bugs, code-review, and resolving-merge-conflicts.
A systems-oriented deep dive into FlashMLA, from weight absorption and its 576/512 layout to paged KV cache, Split-KV, online softmax, seesaw scheduling, sparse attention, FP8 KV cache, and crossover.
Distilling to-spec, to-tickets, wayfinder, and to-questionnaire into durable planning artifacts.
A systems-oriented reading of DeepSeek-R1 and R1-Zero, covering GRPO, outcome rewards, multi-stage post-training, test-time compute, distillation, RL infrastructure, costs, failures, and limitations.
Distilling grilling, grill-me, grill-with-docs, and domain-modeling into a disciplined alignment workflow.
A systems-oriented reading of DeepSeek-V3, covering MLA, DeepSeekMoE, auxiliary-loss-free balancing, MTP, FP8, DualPipe, long-context training, deployment, evaluation, and cost accounting.
A map of the 41 skills in mattpocock/skills: the main flow, on-ramps, routing, project setup, and principles for predictable skills.
A technical guide to Beam Search: sequence scoring, expansion and pruning, beam width, length normalization, stopping criteria, and inference costs.
A systems guide to RL infrastructure for LLM post-training, covering rollout, environments, rewards, trajectories, training, weight synchronization, deployment models, open-source frameworks, and the path from inference infrastructure to RL systems.
An engineering-oriented introduction to compiler passes: how they analyze and rewrite graphs and IR, and how fusion, lowering, pattern rewriting, and kernels fit together.
A systems-level guide to tensor strides: address calculation, zero-copy views, kernel layout contracts, memory coalescing, attention and paged KV cache layouts, quantization, communication packing, compilation, and debugging.
A systems guide to All-to-All optimization in MoE, covering dispatch and combine, uneven routing, small messages, topology, permutation overhead, hierarchical communication, kernel fusion, overlap, expert placement, and profiling.
A systems view of FlexKV: prefix matching, RadixTree indexing, asynchronous transfer graphs, and distributed KV reuse with Redis and Mooncake.
A serving-oriented guide to Multi-Token Prediction: its training objective, DeepSeek-V3 sequential MTP modules, speculative verification, KV cache commit, and scheduler integration.
Notes for lesson seven: Relax ASTs, visitors, operator pattern rewriting, Linear + ReLU fusion, and mapping graph optimization to TensorIR calls.
Notes for lesson six: thread blocks, local/shared-memory tiling, hardware specialization, and tensorization for mapping tensor programs to GPUs.
Notes for lesson five: TE, BlockBuilder, and PyTorch FX integration for importing framework models into IRModule.
Notes for lesson four: turning schedules into a search space, stochastic schedule transformations, candidate measurement, and the Meta-Schedule optimization loop.
Where xLLM fits in LLM infrastructure: not a model, but an inference engine and clustered serving system for diverse AI accelerators, with Service-Engine decoupling, PD/EPD disaggregation, global KV cache, dynamic scheduling, and MoE optimizations.
A unified guide to DP, TP, PP, SP, CP, EP, and prefill-decode disaggregation in LLM inference: what each dimension partitions, how they compose, and where MoE and KV cache create practical coupling.
A mathematical comparison of PPO, DPO, and GRPO through policy gradients, KL regularization, critics, rewards, old policies, reference policies, and practical tradeoffs.
Notes for lesson three: using Relax graphs, call_tir, dataflow blocks, and parameter binding to assemble TensorIR functions into an executable model.
Notes for lesson two: primitive tensor functions, loops, buffers, block axes, and why TensorIR schedules make tensor programs optimizable.
A practical walkthrough of where Kubernetes fits in AI infrastructure: deployment, scheduling, scaling, recovery, and its relationship with Docker, Ray, Slurm, and LLM serving engines.
Notes for lesson one of Machine Learning Compilation: development and deployment forms, tensor functions, and the abstraction-implementation boundary.
A practical explanation of N-Gram language models: chain rule, Markov assumption, count-based estimation, sparsity, smoothing, backoff, interpolation, and how they differ from Transformers.
A systems note on why prefix caching can turn hardware compute benchmarks into measurements of cache hit rate, serving policy, and workload repetition.
A narrative note on what reasoning changes and does not change, from training and reasoning tokens to vLLM serving, PD disaggregation, and production benchmarking.
A practical explanation of checkpoints in LLMs: what they store, how they enable resume training, how they relate to model weights and .safetensors, and how serving systems load them.
A practical walkthrough of prefill-decode disaggregation: why prefill and decode conflict, what is actually disaggregated, how it affects TTFT and TPOT, and why KV cache transfer becomes the new systems bottleneck.
A practical explanation of CUDA Graph: how it captures stable CUDA workloads for replay, why it reduces launch overhead, and how it is used in PyTorch and LLM serving.
A practical explanation of Linear Attention: how kernel feature maps and recurrent states reduce long-context cost, and why Gamma, Delta, Gate, and Conv variants matter.
An intuitive guide to Maximal Update Parametrization, covering its motivation, relation to NTK and standard parametrization, Transformer-specific changes, and hyperparameter transfer.
A practical comparison of distributed training and distributed inference across goals, dataflow, memory, communication, performance metrics, scheduling, and vLLM deployment.
A practical explanation of skills in AI agents: reusable workflow packages that combine instructions, metadata, optional scripts, and resources.
A first-person walkthrough of NCCL: its role in distributed GPU systems, common collectives, topology-aware communication, multi-node paths, and practical debugging.
A practical guide to link bandwidth, injection bandwidth, bisection bandwidth, algbw, and busbw across PCIe, ring, NVSwitch, and fat-tree topologies.
From outgrowing a resume to building a dynamic personal showcase with Next.js, MDX, Tailwind, and GitHub Pages.
Notes for Stanford CS336 Spring 2025 lecture 17: Policy Gradient, GRPO.
Notes for Stanford CS336 Spring 2025 lecture 16: RLVR, GRPO.
Notes for Stanford CS336 Spring 2025 lecture 15: Alignment, RLHF, SFT.
Notes for Stanford CS336 Spring 2025 lecture 14: Data Filtering, Dedup.
Notes for Stanford CS336 Spring 2025 lecture 13: Data, Pretraining.
Notes for Stanford CS336 Spring 2025 lecture 12: Evaluation, Benchmarks.
Notes for Stanford CS336 Spring 2025 lecture 11: Scaling, muP.
Notes for Stanford CS336 Spring 2025 lecture 10: Inference, KV Cache.
Notes for Stanford CS336 Spring 2025 lecture 9: Scaling Laws, Chinchilla.
Notes for Stanford CS336 Spring 2025 lecture 8: torch.distributed, NCCL.
Notes for Stanford CS336 Spring 2025 lecture 7: Parallelism, ZeRO.
Notes for Stanford CS336 Spring 2025 lecture 6: Triton, Kernels.
Notes for Stanford CS336 Spring 2025 lecture 5: GPU, CUDA, FlashAttention.
Notes for Stanford CS336 Spring 2025 lecture 4: MoE, Routing.
Notes for Stanford CS336 Spring 2025 lecture 3: Architecture, Transformer.
Notes for Stanford CS336 Spring 2025 lecture 2: PyTorch, FLOPs, Memory.
Notes for Stanford CS336 Spring 2025 lecture 1: Overview, Tokenization.