Poor conditioning

Conditioning measures how rapidly the output changed with tiny changes in input.

For example, in a linear equation, we can use the inverse matrix to solve .

Nevertheless it is not commonly done in machine learning because is slow to compute, and worse, may amplify input errors rapidly.

For the function:

Condition number is defined as:

Poorly conditioned matrix is a matrix with a high condition number. amplifies input errors. Small errors in can change the output of rapidly .

Other methods including matrix factorization can replace the matrix inversion method to avoid the poor conditioning to improve the numerical stability.

Underflow or overflow

Softmax functions:

can be very large or very small. To avoid overflow or underflow, we can transform with :


Jacobian matrix


Hessian matrix

The first derivative measures the gradient and the second derivative measures the curvature.

When , curves down and when , curves up.

The second derivative indicates whether a gradient step drops the cost as much as the gradient alone may imply. For example, at (the orange dot on the right below), the gradient is positive and the cost drops towards direction. Since the second derivative is positive, the function curves upwards towards zero. i.e. the cost drops less than one predicted by the gradient alone.

With the second derivative, we may take advantage of the curvature information to create a better gradient descent method to reduce overshoot. For example, instead of descending to a local minimum from , we may overshoot to in the left diagram below. In some NLP problem, the gradient is so steep that we may bound upward to much higher cost.

Hessian matrix is defined as:

Eigenvalues for H

is symmetrical:

And it is real. Any real symmetrical matrices can be decomposed into eigenvalues and eigenvectors. One more observation for the later use: the maximum value of for vector happens when aligns with the eigenvector that has the maximum eigenvalue , i.e.

Learning rate

With Taylor series in 2nd order:

If is negative or 0, decreases as increases. However, we cannot drop too far as the accuracy of the Taylor series drops as increases. If is positive, it may cause to go up again. The optimal step for is (assume ):

is the maximum eigenvalue for . Hence, Hessian matrix establishes a lower bound of the optimal learning rate.

If the Hessian matrix has a poor condition number, the gradient along the eigenvector with the largest eigenvalue is much smaller than the one with the smallest eigenvalue. Gradient descent methods work poorly if the gradients in different directions are in different order of magnitude. The gradient descent methods will either learn too slow in the low gradient direction and/or overshoot the solution in the high gradient direction. We may use Newton’s method to control the gradient descent better.

Newton’s Method

With Newton’s method:

To find the critical point, we set :


Extend it to multiple variables with :

Apply the gradient descent with Newton’s method:

Saddle point

alone cannot tell whether is a local optimal point or a saddle point. With the second derivative test, when and , is a local minimum. When and , is a local maximum. However, if , it will be in-conclusive (saddle point or local optimal point).

For multiple dimension, when is positive definite (all the eigenvalues are positive), is a local minimum. If is negative definite, is a local maximum. If at least one eigenvalue is positive and at least one is negative, the point is a saddle point because one direction is a local minimum and the other direction is a local maximum. If at least one eigenvalue is zero and the rest have the same sign, it will be in-conclusive again.

Constrained Optimization

In deep learning, we may want to find an optimal point under certain constraints. For example, we want to maximize subject to . We will construct a new Lagrangian function from and which the original optimal solution is the same as the optimal solution for the Lagrangian function. i.e. .

Lagrange multiplier

To maximize subject to , we plot the contour plot of for different (). The solution lies on the red line with the largest .

(Source Wikipedia)

Geometrically, the optimal point lies where the gradient at , the blue arrow, aligned with the gradient at , the red arrow.


where is the Lagrange multiplier and it can be positive or negative. We can now solve a constrained optimization problem using unconstrained optimization of the generalized Lagrangian.

We can have multiple constraints (). ie. we want to maximize subject to . The Lagrangian is generalized as:

And the optimal solution is


Maximize subject to

The Lagrangian is:

To optimize , we need to solve:


By simply plugin the values, we can determine which one is the max or min.

Karush–Kuhn–Tucker (KKT)

KKT expands the constraints in the Lagrange multiplier to inequality also:

The Lagrangian is generalized to:


KKT conditions

The required KKT conditions to solve the optimization problems are:

and the solution needs to be verified with the constraints again.

Let’s go through the meaning of each KKT conditions. Same as Lagrange multiplier, the optimal points happen when the derivative , i.e.

In the Lagrange multiplier, can be positive, negative or zero. In KKT. must be greater or equal to 0. This guarantees the in-equality such that the solution is within the constrained area.


it indicates either the KKT multiplier or the . If , we do not care about the constraint. The in-equality constrain is not necessary like the diagram below because the optimal point is guarantee to be inside the constrained area. We can simply ignore the constraint.

Otherwise, the in-equality constrain becomes the equality constraint .


We sample the outputs of a few small values and select that output the best optimal value.