Optimization

Theorem 4.1 (First Derivative Test). If $f$ has a local extremum at an interior point $\mathbf{a}$ And $\nabla f(\mathbf{a})$ exists, then $\nabla f(\mathbf{a}) = \mathbf{0}$.

Points where $\nabla f = \mathbf{0}$ are called critical points (or stationary points).

Remark. Not all critical points are extrema. A critical point can be a local minimum, local maximum, Or saddle point. The second derivative test (Section 4.2) distinguishes these cases.

Theorem 4.2 (Second Derivative Test). Let $f$ have continuous second partial derivatives near a Critical point $(a,b)$ with $f_x(a,b) = f_y(a,b) = 0$. Let

$D = f_{xx}(a,b) f_{yy}(a,b) - [f_{xy}(a,b)]^2$

Be the Hessian determinant. Then:

• If $D \gt 0$ and $f_{xx}(a,b) \gt 0$: local minimum.
• If $D \gt 0$ and $f_{xx}(a,b) \lt 0$: local maximum.
• If $D \lt 0$: saddle point.
• If $D = 0$: the test is inconclusive.

Proof. By Taylor”s theorem to second order, for small $h, k$:

$f(a+h, b+k) - f(a,b) = \frac{1}{2}\left[f_{xx} h^2 + 2f_{xy} hk + f_{yy} k^2\right] + R_2$

Where the remainder $R_2 = o(h^2 + k^2)$ and all partials are evaluated at $(a,b)$. The sign of the Right-hand side is determined by the quadratic form

$Q(h,k) = f_{xx} h^2 + 2f_{xy} hk + f_{yy} k^2 = \begin{pmatrix} h & k \end{pmatrix} H \begin{pmatrix} h \\ k \end{pmatrix}$

Where $H = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{xy} & f_{yy} \end{pmatrix}$ is the Hessian matrix.

By Sylvester’s criterion for $2 \times 2$ symmetric matrices:

• If $\det(H) = D \gt 0$ and $f_{xx} \gt 0$Then $H$ is positive definite, so $Q \gt 0$ for all $(h,k) \neq (0,0)$Giving a local minimum.
• If $\det(H) = D \gt 0$ and $f_{xx} \lt 0$Then $H$ is negative definite, so $Q \lt 0$ for all $(h,k) \neq (0,0)$Giving a local maximum.
• If $\det(H) = D \lt 0$Then $H$ is indefinite, so $Q$ takes both positive and negative values, giving a saddle point.

When $D = 0$The quadratic form is degenerate and the sign is determined by higher-order terms. $\blacksquare$

Theorem 4.3 (Method of Lagrange Multipliers). To find the extrema of $f(x,y,z)$ subject to the Constraint $g(x,y,z) = 0$Solve the system:

$\nabla f = \lambda \nabla g, \quad g = 0$

More generally, for $k$ constraints $g_1 = 0, \ldots, g_k = 0$:

$\nabla f = \lambda_1 \nabla g_1 + \cdots + \lambda_k \nabla g_k$

Proof (single constraint, geometric justification). Let $M = \\{(x,y,z) : g(x,y,z) = 0\\}$ be the constraint surface. If $f$ has a local extremum on $M$ at $\mathbf{p}$Then the directional derivative $D_{\mathbf{v}} f(\mathbf{p}) = 0$ for every tangent Vector $\mathbf{v}$ to $M$ at $\mathbf{p}$. Since $\nabla f(\mathbf{p}) \cdot \mathbf{v} = 0$ for all Such $\mathbf{v}$The gradient $\nabla f(\mathbf{p})$ must be orthogonal to the tangent space of $M$ At $\mathbf{p}$. But the tangent space of $M$ is orthogonal to $\nabla g(\mathbf{p})$ (by the implicit Function theorem). Therefore $\nabla f(\mathbf{p})$ must be parallel to $\nabla g(\mathbf{p})$I.e., $\nabla f(\mathbf{p}) = \lambda\, \nabla g(\mathbf{p})$ for some scalar $\lambda$. $\blacksquare$

Problem. Find the maximum of $f(x,y) = xy$ subject to $x^2 + y^2 = 1$.

Solution. Set $g(x,y) = x^2 + y^2 - 1$. The Lagrange multiplier equations:

$\nabla f = \lambda \nabla g \implies (y, x) = \lambda(2x, 2y)$

This gives $y = 2\lambda x$ and $x = 2\lambda y$. Multiplying: $xy = 4\lambda^2 xy$.

Case 1: $xy \neq 0$. Then $4\lambda^2 = 1$So $\lambda = \pm 1/2$.

• $\lambda = 1/2$: $y = x$And $x^2 + x^2 = 1$So $x = \pm 1/\sqrt{2}$. Points: $(1/\sqrt{2}, 1/\sqrt{2})$ and $(-1/\sqrt{2}, -1/\sqrt{2})$ with $f = 1/2$.
• $\lambda = -1/2$: $y = -x$And $x^2 + x^2 = 1$So $x = \pm 1/\sqrt{2}$. Points: $(1/\sqrt{2}, -1/\sqrt{2})$ and $(-1/\sqrt{2}, 1/\sqrt{2})$ with $f = -1/2$.

Case 2: $xy = 0$. Then either $x = 0$ or $y = 0$. From the constraint: $(0, \pm 1)$ or $(\pm 1, 0)$ with $f = 0$.

Maximum: $f = 1/2$ at $(\pm 1/\sqrt{2}, \pm 1/\sqrt{2})$. Minimum: $f = -1/2$ at $(\pm 1/\sqrt{2}, \mp 1/\sqrt{2})$. $\blacksquare$

Problem. Find and classify all critical points of $f(x,y) = x^4 + y^4 - 4xy$.

Problem. Find and classify all critical points of $f(x,y) = x^3 + y^3 - 3xy$.

Problem. Find the point on the plane $x + 2y + 3z = 6$ closest to the origin.

Problem. Maximise $f(x,y,z) = xyz$ subject to $x + y + z = 1$ and $x^2 + y^2 + z^2 = 1/3$.

:::caution Common Pitfalls

• Lagrange multipliers find candidates only. The method produces candidates for constrained extrema but does not guarantee they are extrema. Always evaluate $f$ at all candidates and use additional reasoning (e.g., compactness of the constraint set via the extreme value theorem) to determine which gives the max/min.
• Boundary vs. Interior. For unconstrained problems on a closed, bounded domain, check both interior critical points and boundary points separately.
• Degenerate Hessian. When the Hessian determinant $D = 0$The second derivative test is inconclusive. Use higher-order Taylor expansions or direct analysis of the function near the critical point.
• Non-normalised constraint gradients. Ensure the constraint functions are written in the form $g = 0$; multiplying $g$ by a constant changes $\lambda$ but not the critical points.

:::