II - Calculus to Optimization & Analysis

Jump to Topic

Part 1: The Derivative of f:ℝⁿ → ℝ Part 2: The Derivative of f:ℝⁿ → ℝⁿ Part 3: The Derivative of f:ℝⁿˣⁿ → ℝⁿˣⁿ Part 4: Intro to Numerical Computation Part 5: The Derivative of Scalar Functions of Matrices Part 6: The Mean Value Theorem Part 7: Gradient Descent (First-order Method) Part 8: Newton's method (Second-order Method) Part 9: Constrained Optimization Part 10: Riemann Integration Part 11: Measure Theory with Probability Part 12: Intro to Lebesgue Integration Part 13: Duality in Optimization & Analysis

Calculus is essential in various branches of mathematics. This section is designed to take you from fundamental calculus concepts to the advanced techniques essential for optimization. Building on the foundation laid in Linear Algebra, this section explores how calculus is applied to analyze and optimize complex systems. We begin with the classical notion of derivatives—ranging from scalar functions to those of vectors and matrices—and gradually introduce numerical methods. In doing so, we not only deepen your understanding of calculus but also provide the essential background and analytical foundations that lie behind machine learning.

∇

Part 1: The Derivative of \(f:\mathbb{R}^n \rightarrow \mathbb{R}\)

Linear approximation Linearization Differentials Product rule Gradient Quadratic form \(L_2\) norm
J

Part 2: The Derivative of \(f:\mathbb{R}^n \rightarrow \mathbb{R}^n\)

Jacobian matrix Chain rule Backpropagation reverse(forward) mode automatic differentiation
M

Part 3: The Derivative of \(f:\mathbb{R}^{n \times n} \rightarrow \mathbb{R}^{n \times n}\)

Powers of a matrix Inverse of a matrix LU decomposition
≈

Part 4: Intro to Numerical Computation

Code Included Finite-difference approximation Relative error Roundoff error
det

Part 5: The Derivative of Scalar Functions of Matrices

Frobenius inner product Frobenius norm Trace Determinant Cofactor Adjugate
μ

Part 6: The Mean Value Theorem

Rolle's Theorem Lagrange's Mean Value Theorem Cauchy's Mean Value Theorem Taylor's Theorem Taylor polynomial little-o notation Higher-dimensional MVT
↓

Part 7: Gradient Descent (First-order Method)

Code Included Optimization problems Convexity Gradient Descent (SD) Steepest Descent Stochastic Gradient Descent (SGD) Mini-batch SGD Sub-gradient Sub-differentiable
N

Part 8: Newton's method (Second-order Method)

Code Included Line search Armijo condition Curvature condition Wolfe conditions Newton's method Quasi-Newton methods BFGS Secant condition Inverse Hessian approximation Limited memory BFGS Rosenbrock function
λ

Part 9: Constrained Optimization

Code Included Constrained optimization problems Penalty terms Lagrange Multipliers Lagrangian Karush-Kuhn-Tucker (KKT) conditions Active set Slack variables
∫

Part 10: Riemann Integration

Riemann integral Riemann integrable Improper Riemann integration Dirichlet function
σ

Part 11: Measure Theory with Probability

Sample space \(\sigma\)-algebra Measurable set Measurable space Measure Probability measure Probability space Countable additivity Borel \(\sigma\)-algebra Borel set Lebesgue measure
a.e.

Part 12: Intro to Lebesgue Integration

Lebesgue integral Characteristic function Almost everywhere(a.e.) Simple function
P⟷D

Part 13: Duality in Optimization & Analysis

Interactive Demo Duality Weak duality Strong duality Duality gap Smoothness Lipschitz continuity Contraction mapping Convergence rate