Optimization

All econ problems are optimization problems

Optimization

All econ problems are optimization problems

• Min costs

Optimization

All econ problems are optimization problems

• Min costs

• Max PV E[welfare]

Optimization

All econ problems are optimization problems

• Min costs

• Max PV E[welfare]

Some are harder than others:

Optimization

All econ problems are optimization problems

• Min costs

• Max PV E[welfare]

Some are harder than others:

• Individual utility max: easy

Optimization

All econ problems are optimization problems

• Min costs

• Max PV E[welfare]

Some are harder than others:

• Individual utility max: easy

• Decentralized electricity market with nodal pricing and market power: hard

Optimization

All econ problems are optimization problems

• Min costs

• Max PV E[welfare]

Some are harder than others:

• Individual utility max: easy

• Decentralized electricity market with nodal pricing and market power: hard

• One input profit maximization problem: easy

Linear rootfinding

How do we solve these?

Linear rootfinding

How do we solve these?

Consider a simple generic problem:

$Ax = b$

Linear rootfinding

How do we solve these?

Consider a simple generic problem:

$Ax = b$

Invert $A$

$x = A^{-1}b$

Non-linear rootfinding

With non-linear rootfinding problems we want to solve:

$f(x) = 0, f:\mathbb{R}\rightarrow\mathbb{R}^n$

Non-linear rootfinding

With non-linear rootfinding problems we want to solve:

$f(x) = 0, f:\mathbb{R}\rightarrow\mathbb{R}^n$

What's a common rootfinding problem?

Non-linear rootfinding

With non-linear rootfinding problems we want to solve:

$f(x) = 0, f:\mathbb{R}\rightarrow\mathbb{R}^n$

What's a common rootfinding problem?

Can we reframe a common economic problem as rootfinding?

Non-linear rootfinding

With non-linear rootfinding problems we want to solve:

$f(x) = 0, f:\mathbb{R}\rightarrow\mathbb{R}^n$

What's a common rootfinding problem?

Can we reframe a common economic problem as rootfinding?

Yes!

Non-linear rootfinding

With non-linear rootfinding problems we want to solve:

$f(x) = 0, f:\mathbb{R}\rightarrow\mathbb{R}^n$

What's a common rootfinding problem?

Can we reframe a common economic problem as rootfinding?

Yes!

Fixed point problems are rootfinding problems:

Rootfinding with constraints

Economic problems virtually always involve constraints

Rootfinding with constraints

Economic problems virtually always involve constraints

These constraints may or may not be binding

Rootfinding with constraints

Economic problems virtually always involve constraints

These constraints may or may not be binding

e.g. a firm profit max problem: a control vector $x$ may be constrained to be $x \in [a,b]$

Rootfinding with constraints

Economic problems virtually always involve constraints

These constraints may or may not be binding

e.g. a firm profit max problem: a control vector $x$ may be constrained to be $x \in [a,b]$

This results in a complementarity problem:

Rootfinding with constraints

Economic problems virtually always involve constraints

These constraints may or may not be binding

e.g. a firm profit max problem: a control vector $x$ may be constrained to be $x \in [a,b]$

This results in a complementarity problem:

$$\begin{align} x_i > a_i \Rightarrow f_i(x) \geq 0 \,\,\, \forall \, i=1,...,n \\ x_i < b_i \Rightarrow f_i(x) \leq 0 \,\,\, \forall \, i=1,...,n \end{align}$$

Rootfinding with constraints

$$\begin{align} x_i > a_i \Rightarrow f_i(x) \geq 0 \,\,\, \forall \, i=1,...,n \\ x_i < b_i \Rightarrow f_i(x) \leq 0 \,\,\, \forall \, i=1,...,n \end{align}$$

If the constraints on $x_i$ do not bind, $a_i < x_i < b_i$, then the first-order condition is precisely zero

Rootfinding with constraints

$$\begin{align} x_i > a_i \Rightarrow f_i(x) \geq 0 \,\,\, \forall \, i=1,...,n \\ x_i < b_i \Rightarrow f_i(x) \leq 0 \,\,\, \forall \, i=1,...,n \end{align}$$

If the constraints on $x_i$ do not bind, $a_i < x_i < b_i$, then the first-order condition is precisely zero

Suppose the upper bound binds, $x_i = b_i$

Basic non-linear rootfinders: Bisection method

What does the intermediate value theorem tell us?

Basic non-linear rootfinders: Bisection method

What does the intermediate value theorem tell us?

If a continuous real-valued function on a given interval takes on two values $a$ and $b$,
it achieves all values in the set $[a,b]$ somewhere in its domain

Basic non-linear rootfinders: Bisection method

What does the intermediate value theorem tell us?

If a continuous real-valued function on a given interval takes on two values $a$ and $b$,
it achieves all values in the set $[a,b]$ somewhere in its domain

How can this motivate an algorithm to find the root of a function?

Basic non-linear rootfinders: Bisection method

What does the intermediate value theorem tell us?

If a continuous real-valued function on a given interval takes on two values $a$ and $b$,
it achieves all values in the set $[a,b]$ somewhere in its domain

How can this motivate an algorithm to find the root of a function?

If we have a continuous, 1 variable function that is positive at some value and negative at another, a root must fall in between those values

Basic non-linear rootfinders: Bisection method

What does the intermediate value theorem tell us?

If a continuous real-valued function on a given interval takes on two values $a$ and $b$,
it achieves all values in the set $[a,b]$ somewhere in its domain

How can this motivate an algorithm to find the root of a function?

If we have a continuous, 1 variable function that is positive at some value and negative at another, a root must fall in between those values

We know a root exists by IVT, what's an efficient way to find it?

Basic non-linear rootfinders: Bisection method

What does the intermediate value theorem tell us?

If a continuous real-valued function on a given interval takes on two values $a$ and $b$,
it achieves all values in the set $[a,b]$ somewhere in its domain

How can this motivate an algorithm to find the root of a function?

If we have a continuous, 1 variable function that is positive at some value and negative at another, a root must fall in between those values

We know a root exists by IVT, what's an efficient way to find it?

Continually bisect the interval!

The bisection method

The bisection method is incredibly robust: if a function $f$ satisfies the IVT, it is guaranteed to converge in a specific number of iterations

Function iteration

Fixed points can be computed using function iteration

Function iteration

Fixed points can be computed using function iteration

Since we can recast fixed points as rootfinding problems we can use function iteration to find roots too

Newton's method

Newton's method and variants are the workhorses of solving n-dimensional non-linear problems

Newton's method

Newton's method and variants are the workhorses of solving n-dimensional non-linear problems

What's the idea?

Newton's method

Newton's method and variants are the workhorses of solving n-dimensional non-linear problems

What's the idea?

Take a hard non-linear problem and replace it with a sequence of linear problems

Newton's method

Start with an initial guess of the root at $x^{(0)}$

Newton's method

Start with an initial guess of the root at $x^{(0)}$

Approximate the non-linear function with its first-order Taylor expansion about $x^{(0)}$

Newton's method

Start with an initial guess of the root at $x^{(0)}$

Approximate the non-linear function with its first-order Taylor expansion about $x^{(0)}$

This is just the tangent line at $x^0$, solve for the root of this linear approximation, call it $x^{(1)}$

Newton's method

Start with an initial guess of the root at $x^{(0)}$

Approximate the non-linear function with its first-order Taylor expansion about $x^{(0)}$

This is just the tangent line at $x^0$, solve for the root of this linear approximation, call it $x^{(1)}$

Repeat starting at $x^{(1)}$

Newton's method

Begin with some initial guess of the root vector $\mathbf{x^{(0)}}$

Newton's method

Code up Newton's method

Newton's method

f(x) = x^3;
f_prime(x) = 3x^2;
newtons_method(f, f_prime, 1.)

## The root of f(x) is at 1.231347218094855e-6.

Newton's method

Newton's method has nice properties regarding convergence and speed:

If $f(x)$ is continuously differentiable, the initial guess is "sufficiently close" to the root, and $f(x)$ is invertible near the root, then Newton's method converges to the root

Newton's method

Newton's method has nice properties regarding convergence and speed:

If $f(x)$ is continuously differentiable, the initial guess is "sufficiently close" to the root, and $f(x)$ is invertible near the root, then Newton's method converges to the root

What is "sufficiently close"?

Newton's method

Newton's method has nice properties regarding convergence and speed:

If $f(x)$ is continuously differentiable, the initial guess is "sufficiently close" to the root, and $f(x)$ is invertible near the root, then Newton's method converges to the root

What is "sufficiently close"?

We need $f(x)$ to be invertible so the algorithm above is well defined

Quasi-Newton: Secant method

We usually don't want to deal with analytic derivatives unless we have access to autodifferentiation

Quasi-Newton: Secant method

We usually don't want to deal with analytic derivatives unless we have access to autodifferentiation

Why?

Quasi-Newton: Secant method

We usually don't want to deal with analytic derivatives unless we have access to autodifferentiation

Why?

1. Coding error / time
2. Can actually be slower to evaluate than finite differences for a nonlinear problem, see Ken Judd's notes

Quasi-Newton: Secant method

We usually don't want to deal with analytic derivatives unless we have access to autodifferentiation

Why?

1. Coding error / time
2. Can actually be slower to evaluate than finite differences for a nonlinear problem, see Ken Judd's notes

Quasi-Newton: Secant method

Our new iteration rule then becomes

Quasi-Newton: Secant method

Our new iteration rule then becomes

$x^{(k+1)} = x^{(k)} - \frac{x^{(k)}-x^{(k-1)}}{f(x^{(k)})-f(x^{(k-1)})}f(x^{(k)})$

Quasi-Newton: Broyden's method

Broyden's method is the most widely used rootfinding method for n-dimensional problems

Quasi-Newton: Broyden's method

Broyden's method is the most widely used rootfinding method for n-dimensional problems

It is a generalization of the secant method where have a sequence of guesses of the Jacobian at the root

Quasi-Newton: Broyden's method

Root guess update is the same as before but with our guess of the Jacobian substituted in for the actual Jacobian or the finite difference approximation

$\mathbf{x^{(k+1)}} = \mathbf{x^{(k)}} - A_{(k)}^{-1} \, f(\mathbf{x^{(k)}}).$

Quasi-Newton: Broyden's method

Root guess update is the same as before but with our guess of the Jacobian substituted in for the actual Jacobian or the finite difference approximation

$\mathbf{x^{(k+1)}} = \mathbf{x^{(k)}} - A_{(k)}^{-1} \, f(\mathbf{x^{(k)}}).$

we still need to update $A_{(k)}$: we do this update is performed by making the smallest change, in terms of the Frobenius matrix norm, that satisfies what is called the secant condition (under determined if $n>1$): $f(\mathbf{x^{(k+1)}}) - f(\mathbf{x^{(k)}}) = A_{(k+1)}\left( \mathbf{x^{(k+1)}} - \mathbf{x^{(k)}} \right)$

Quasi-Newton: Broyden's method

Root guess update is the same as before but with our guess of the Jacobian substituted in for the actual Jacobian or the finite difference approximation

$\mathbf{x^{(k+1)}} = \mathbf{x^{(k)}} - A_{(k)}^{-1} \, f(\mathbf{x^{(k)}}).$

we still need to update $A_{(k)}$: we do this update is performed by making the smallest change, in terms of the Frobenius matrix norm, that satisfies what is called the secant condition (under determined if $n>1$): $f(\mathbf{x^{(k+1)}}) - f(\mathbf{x^{(k)}}) = A_{(k+1)}\left( \mathbf{x^{(k+1)}} - \mathbf{x^{(k)}} \right)$

The updated differences in root guesses, and the function value at those root guesses, should align with our estimate of the Jacobian at that point

Accelerating Broyden

Why update the Jacobian and then invert when we can just update an inverted Jacobian $B = A^{-1}$

$B_{(k+1)} = B_{(k)} + \frac{[d^{(k)} - u^{(k)}]{d^{(k)}}^TB_{(k)}}{{d^{(k)}}^T u^{(k)}}$

where $d^{(k)} = (\mathbf{x^{(k+1)}} - \mathbf{x^{(k)}})$, and $u^{(k)} = B_{(k)}\left[f(\mathbf{x^{(k+1)}})-f(\mathbf{x^{(k)}})\right]$.

Accelerating Broyden

Why update the Jacobian and then invert when we can just update an inverted Jacobian $B = A^{-1}$

$B_{(k+1)} = B_{(k)} + \frac{[d^{(k)} - u^{(k)}]{d^{(k)}}^TB_{(k)}}{{d^{(k)}}^T u^{(k)}}$

where $d^{(k)} = (\mathbf{x^{(k+1)}} - \mathbf{x^{(k)}})$, and $u^{(k)} = B_{(k)}\left[f(\mathbf{x^{(k+1)}})-f(\mathbf{x^{(k)}})\right]$.

Broyden converges under relatively weak conditions:

Convergence speed

Rootfinding algorithms will converge at different speeds in terms of the number of operations

Convergence speed

Rootfinding algorithms will converge at different speeds in terms of the number of operations

A sequence of iterates $x^{(k)}$ is said to converge to $x^*$ at a rate of order $p$ if there is a constant $C$ such that

$||x^{(k+1)} - x^*|| \leq C||x^{(k)} - x^*||^p$

for sufficiently large $k$

Convergence speed

How fast do the methods we've seen converge?

Convergence speed

How fast do the methods we've seen converge?

• Bisection: linear rate with $C = 0.5$ (kind of obvious once you see it)

Convergence speed

How fast do the methods we've seen converge?

• Bisection: linear rate with $C = 0.5$ (kind of obvious once you see it)

• Function iteration: linear rate with $C = ||f'(x^*)||$

Convergence speed

How fast do the methods we've seen converge?

• Bisection: linear rate with $C = 0.5$ (kind of obvious once you see it)

• Function iteration: linear rate with $C = ||f'(x^*)||$

• Secant and Broyden: superlinear rate with $p \approx 1.62$

Convergence speed

Ex:

• Bisection method only requires a single function evaluation during each iteration
• Function iteration only requires a single function evaluation during each iteration
• Broyden's method requires both a function evaluation and matrix multiplication
• Newton's method requires a function evaluation, a derivative evaluation, and solving a linear system

Complementarity problems: rootfinding

Let:

• $x$ be an n-dimensional vector of some economic action
• $a_i$ denotes a lower bound on action $i$, and $b_i$ denotes the upper bound on action $i$
• $f_i(x)$ denotes the marginal arbitrage profit of action $i$

Complementarity problems

We obtain a no-arbitrage equilibrium if and only if $x$ solves the complementary problem $CP(f,a,b)$

Complementarity problems

We obtain a no-arbitrage equilibrium if and only if $x$ solves the complementary problem $CP(f,a,b)$

We can write out the problem as finding a vector $x \in [a,b]$ that solves $$\begin{align} x_i > a_i \Rightarrow f_i(x) \geq 0 \,\,\, \forall i = 1,...,n \notag\\ x_i < b_i \Rightarrow f_i(x) \leq 0 \,\,\, \forall i = 1,...,n \notag \end{align}$$

Complementarity problems

Example: single commodity competitive spatial price equilibrium model

• $n$ regions of the world
• excess demand for the commodity in region $i$ is $E_i(p_i)$

Complementarity problems

Example: single commodity competitive spatial price equilibrium model

• $n$ regions of the world
• excess demand for the commodity in region $i$ is $E_i(p_i)$

If no trade $\rightarrow$ equilibrium condition is $E_i(p_i) = 0$ in all regions of the world: a simple rootfinding problem

Complementarity problems

Marginal arbitrage profit from shipping a unit of the good from $i$ to $j$ is $p_j - p_i - c_{ij}$

Complementarity problems

Marginal arbitrage profit from shipping a unit of the good from $i$ to $j$ is $p_j - p_i - c_{ij}$

If it's positive: incentive to ship more goods to region $i$ from region $j$

If it's negative: incentive to decrease shipments

Complementarity problems

How do we formulate this as a complementarity problem?

Complementarity problems

How do we formulate this as a complementarity problem?

Market clearing in each region $i$ requires that net imports = excess demand

Complementarity problems

How do we formulate this as a complementarity problem?

Market clearing in each region $i$ requires that net imports = excess demand

$$\sum_k [x_{ki} - x_{ik}] = E_i(p_i)$$

How do we solve them?

We can then solve the problem with conventional rootfinding techniques
or approximate this with a smooth function to make it easier on a rootfinder

Maximization (minimization) methods

How we solve maximization problems has many similarities to rootfinding and complementarity problems

Maximization (minimization) methods

How we solve maximization problems has many similarities to rootfinding and complementarity problems

FOCs of an unconstrained maximization problem are just a rootfinding problem

Maximization (minimization) methods

How we solve maximization problems has many similarities to rootfinding and complementarity problems

FOCs of an unconstrained maximization problem are just a rootfinding problem

FOCs of a constrained maximization problem are just a complementarity problem

Maximization (minimization) methods

How we solve maximization problems has many similarities to rootfinding and complementarity problems

FOCs of an unconstrained maximization problem are just a rootfinding problem

FOCs of a constrained maximization problem are just a complementarity problem

I'll tend to frame problems as minimization problems because it is the convention the optimization literature

Maximization (minimization) methods

How we solve maximization problems has many similarities to rootfinding and complementarity problems

FOCs of an unconstrained maximization problem are just a rootfinding problem

FOCs of a constrained maximization problem are just a complementarity problem

I'll tend to frame problems as minimization problems because it is the convention the optimization literature

We make two distinctions:

Maximization (minimization) methods

How we solve maximization problems has many similarities to rootfinding and complementarity problems

FOCs of an unconstrained maximization problem are just a rootfinding problem

FOCs of a constrained maximization problem are just a complementarity problem

I'll tend to frame problems as minimization problems because it is the convention the optimization literature

We make two distinctions:

Local vs global: are we finding an optimum in a local region, or globally?

Derivative free optimization: Golden search

Similar to bisection, golden search looks for a solution of a one-dimensional problem over smaller and smaller brackets

Derivative free optimization: Golden search

Similar to bisection, golden search looks for a solution of a one-dimensional problem over smaller and smaller brackets

If we have a continuous one dimensional function, $f(x)$,
and we want to find a local minimum in some interval $[a,b]$

Derivative free optimization: Golden search

Similar to bisection, golden search looks for a solution of a one-dimensional problem over smaller and smaller brackets

If we have a continuous one dimensional function, $f(x)$,
and we want to find a local minimum in some interval $[a,b]$

1. Select points $x_1,x_2 \in [a,b]$ where $x_2 > x_1$

Derivative free optimization: Golden search

Similar to bisection, golden search looks for a solution of a one-dimensional problem over smaller and smaller brackets

If we have a continuous one dimensional function, $f(x)$,
and we want to find a local minimum in some interval $[a,b]$

1. Select points $x_1,x_2 \in [a,b]$ where $x_2 > x_1$

2. If $f(x_1) < f(x_2)$ replace $[a,b]$ with $[a,x_2]$, else replace $[a,b]$ with $[x_1,b]$

Derivative free optimization: Golden search

Similar to bisection, golden search looks for a solution of a one-dimensional problem over smaller and smaller brackets

If we have a continuous one dimensional function, $f(x)$,
and we want to find a local minimum in some interval $[a,b]$

1. Select points $x_1,x_2 \in [a,b]$ where $x_2 > x_1$

2. If $f(x_1) < f(x_2)$ replace $[a,b]$ with $[a,x_2]$, else replace $[a,b]$ with $[x_1,b]$

3. Repeat until convergence criterion is met

Derivative free optimization: Golden search

How do we pick $x_1$ and $x_2$?

Derivative free optimization: Golden search

How do we pick $x_1$ and $x_2$?

Achievable goal for selection process:

• New interval is independent of whether the upper or lower bound is replaced
• Only requires one function evaluation per iteration

Derivative free optimization: Golden search

Golden search algorithm for point selection:

$$\begin{gather} x_i = a + \alpha_i (b-a) \notag \\ \alpha_1 = \frac{3-\sqrt{5}}{2} \qquad \alpha_2 = \frac{\sqrt{5} - 1}{2} \end{gather}$$

Derivative free optimization: Golden search

Golden search algorithm for point selection:

$$\begin{gather} x_i = a + \alpha_i (b-a) \notag \\ \alpha_1 = \frac{3-\sqrt{5}}{2} \qquad \alpha_2 = \frac{\sqrt{5} - 1}{2} \end{gather}$$

The value of $\alpha_2$ is called the golden ratio and is where the algorithm gets its name

Golden search

function golden_search(f, lower_bound, upper_bound)
    alpha_1 = (3 - sqrt(5))/2  # GS parameter 1
    alpha_2 = (sqrt(5) - 1)/2  # GS parameter 2
    tolerance = 1e-2           # tolerance for convergence
    difference = 1e10
    while difference > tolerance
        x_1 = lower_bound + alpha_1*(upper_bound - lower_bound)  # new x_1
        x_2 = lower_bound + alpha_2*(upper_bound - lower_bound)  # new x_2
        if f(x_1) < f(x_2)     # reset bounds
            upper_bound = x_2
        else
            lower_bound = x_1
        end
        difference = x_2 - x_1
    end
    println("Minimum is at x = $((lower_bound+upper_bound)/2).")
end;

Nelder-Mead: Simplex

Golden search is nice and simple but only works in one dimension

There are several derivative free methods for minimization that work in multiple dimensions, the most commonly used one is Nelder-Mead (NM)

Nelder-Mead: Simplex

There are six operations:

Nelder-Mead: Simplex

There are six operations:

• Order: order the value at the vertices of the simplex $f(x_1)\leq...\leq f(x_{n+1})$

Nelder-Mead: Simplex

There are six operations:

• Order: order the value at the vertices of the simplex $f(x_1)\leq...\leq f(x_{n+1})$
• Centroid: calculate $x_0$, the centroid of the non - $x_{n+1}$ points

Nelder-Mead: Simplex

• Expansion: push the reflected point further in the same direction

Nelder-Mead: Simplex

• Expansion: push the reflected point further in the same direction
• Contract: Contract the highest value point toward the middle
  • Compute $x_c = x_0 + \gamma(x_0 - x_{n+1})$, $0 < \gamma \leq 0.5$
  • If $x_c$ is better than the worst point replace $x_{n+1}$ with $x_c$ and restart
  • Else go to step 6

Nelder-Mead: Simplex

• Expansion: push the reflected point further in the same direction
• Contract: Contract the highest value point toward the middle
  • Compute $x_c = x_0 + \gamma(x_0 - x_{n+1})$, $0 < \gamma \leq 0.5$
  • If $x_c$ is better than the worst point replace $x_{n+1}$ with $x_c$ and restart
  • Else go to step 6
• Shrink: shrink the simplex toward the best point
  • Replace all points but the best one with $x_i = x_1 + \sigma(x_i - x_1)$

What is a solution?

We typically want to find a global extremum, here a minimum,
of our objective function $f$

What is a solution?

We typically want to find a global extremum, here a minimum,
of our objective function $f$

$x^*$ is a global minimizer if $f(x^*) \leq f(x)$ for all $x$ over the domain of the function

What is a solution?

We typically want to find a global extremum, here a minimum,
of our objective function $f$

$x^*$ is a global minimizer if $f(x^*) \leq f(x)$ for all $x$ over the domain of the function

Problem: most algorithms are local minimizers that find a point $x^*$
such that $f(x^*) \leq f(x)$ for all $x \in N$, where $N$ is a neighborhood of $x^*$

What is a solution?

Lots of problems have properties that don't satisfy the typical sufficiency conditions for a unique minimum (strictly decreasing and convex), like

What is a solution?

Lots of problems have properties that don't satisfy the typical sufficiency conditions for a unique minimum (strictly decreasing and convex), like

• Concave transitions
• Games with multiple equilibria
• Etc

What is a solution?

Lots of problems have properties that don't satisfy the typical sufficiency conditions for a unique minimum (strictly decreasing and convex), like

• Concave transitions
• Games with multiple equilibria
• Etc

How do we find a local minimum?

The general unconstrained approach

Optimization algorithms typically have the following set up:

1. Start at some $x_0$
2. Work through a series of iterates $\{x_k\}_{k=1}^\infty$ until we have "converged" with sufficient accuracy

The general unconstrained approach

Optimization algorithms typically have the following set up:

1. Start at some $x_0$
2. Work through a series of iterates $\{x_k\}_{k=1}^\infty$ until we have "converged" with sufficient accuracy

If the function is smooth, we can take advantage of that information
about the function's shape to figure out which direction to move in next

The general unconstrained approach

Taylor's Theorem tells us that if $f$ is twice differentiable, then there exists a $t \in (0,1)$ such that

$$f(x^* + p) = f(x^*) + \nabla\,f(x^*)^T \, p + \frac{1}{2!}\,p^T\,\nabla^2\,f(x^*+tp)\,p$$

This is an exact equality

Two large classes of algorithms

All modern algorithms have that general set up but may go about it in different ways

Two large classes of algorithms

All modern algorithms have that general set up but may go about it in different ways

Most modern optimization problems fall into one of two classes:

1. Line search
2. Trust region

Line search algorithms

General idea:

1. Start at some current iterate $x_k$
2. Select a direction to move in $p_k$
3. Figure out how far along $p_k$ to move

Line search algorithms

General idea:

1. Start at some current iterate $x_k$
2. Select a direction to move in $p_k$
3. Figure out how far along $p_k$ to move

How do we figure out how far to move?

Line search algorithms

General idea:

1. Start at some current iterate $x_k$
2. Select a direction to move in $p_k$
3. Figure out how far along $p_k$ to move

How do we figure out how far to move?

"Approximately" solve this problem to figure out the step length $\alpha$ $$\min_{\alpha > 0} f(x_k + \alpha p_k)$$

Line search algorithms

Typically do not perform the full minimization problem since it is costly

We only try a limited number of step lengths $\alpha$ before picking the best one and moving onto our next iterate $x_{k+1}$

Line search: direction choice

What's an obvious choice for $p_k$?

Line search: direction choice

What's an obvious choice for $p_k$?

The direction that yields the steepest descent

Line search: direction choice

What's an obvious choice for $p_k$?

The direction that yields the steepest descent

$-\nabla \, f_k$ is the direction that makes $f$ decrease most rapidly, $k$ indicates we are evaluating $f$ at iteration $k$

Line search: steepest descent

We can verify this is the direction of steepest descent by referring to Taylor's theorem

Line search: steepest descent

We can verify this is the direction of steepest descent by referring to Taylor's theorem

For any direction $p$ and step length $\alpha$, we have that $$f(x_k + \alpha p) = f(x_k) + \alpha\,p^T\,\nabla\,f_k + \frac{1}{2!}\,\alpha^2p^T\,\nabla^2\,f(x_k+tp)\,p$$

Line search: steepest descent

The the unit vector of quickest descent solves $$\min_p p^T\,\nabla\,f_k \,\,\,\,\, \text{subject to: }||p|| = 1$$

Line search: steepest descent

The the unit vector of quickest descent solves $$\min_p p^T\,\nabla\,f_k \,\,\,\,\, \text{subject to: }||p|| = 1$$

Re-express the objective as $\min_{\theta,p} ||p||\,||\nabla\,f_k||cos\,\theta$, where $\theta$ is the angle between $p$ and $\nabla\,f_k$

Line search: alternative directions

We can always use search directions other than the steepest descent

Line search: alternative directions

We can actually verify this with Taylor's theorem

Line search: alternative directions

We can actually verify this with Taylor's theorem

$$f(x_k + \epsilon p_k) = f(x_k) + \epsilon\,p_k^T\,\nabla\,f_k + O(\epsilon^2)$$

Line search: alternative directions

We can actually verify this with Taylor's theorem

$$f(x_k + \epsilon p_k) = f(x_k) + \epsilon\,p_k^T\,\nabla\,f_k + O(\epsilon^2)$$

If $p_k$ is in a descending direction, $\theta_k$ will be of an angle such that $cos\,\theta_k < 0$

This gives us

Line search: alternative directions

We can actually verify this with Taylor's theorem

$$f(x_k + \epsilon p_k) = f(x_k) + \epsilon\,p_k^T\,\nabla\,f_k + O(\epsilon^2)$$

If $p_k$ is in a descending direction, $\theta_k$ will be of an angle such that $cos\,\theta_k < 0$

This gives us

$$p_k^T\,\nabla f_k = ||p_k||\,||\nabla\,f_k||cos\,\theta_k < 0$$

Line search: alternative directions

We can actually verify this with Taylor's theorem

$$f(x_k + \epsilon p_k) = f(x_k) + \epsilon\,p_k^T\,\nabla\,f_k + O(\epsilon^2)$$

If $p_k$ is in a descending direction, $\theta_k$ will be of an angle such that $cos\,\theta_k < 0$

This gives us

$$p_k^T\,\nabla f_k = ||p_k||\,||\nabla\,f_k||cos\,\theta_k < 0$$

Therefore $f(x_k + \epsilon p_k) < f(x_k)$ for positive but sufficiently small $\epsilon$

Newton's method

The most important search direction is not steepest descent but Newton's direction

Newton's method

The most important search direction is not steepest descent but Newton's direction

Newton's direction comes out of the second order Taylor series approximation to $f(x_k + p)$ $$f(x_k + p) \approx f_k + p^T\,\nabla\,f_k + \frac{1}{2!}\,p^T\,\nabla^2f_k\,p$$ Define this as $\mathbf{m_k(p)}$

Newton's method

The most important search direction is not steepest descent but Newton's direction

Newton's direction comes out of the second order Taylor series approximation to $f(x_k + p)$ $$f(x_k + p) \approx f_k + p^T\,\nabla\,f_k + \frac{1}{2!}\,p^T\,\nabla^2f_k\,p$$ Define this as $\mathbf{m_k(p)}$

We find the Newton direction by selecting the vector $p$ that minimizes $f(x_k + p)$

Newton's method

Newton's method

This approximation to the function we are trying to solve has error of $O(||p||^3)$,
so if $p$ is small, the quadratic approximation is very accurate

Trust region methods

Trust region methods construct an approximating model, $m_k$
whose behavior near the current iterate $x_k$ is close to that of the actual function $f$

Trust region methods

Trust region methods construct an approximating model, $m_k$
whose behavior near the current iterate $x_k$ is close to that of the actual function $f$

We then search for a minimizer of $m_k$

Trust region methods

Trust region methods construct an approximating model, $m_k$
whose behavior near the current iterate $x_k$ is close to that of the actual function $f$

We then search for a minimizer of $m_k$

Issue: $m_k$ may not represent $f$ well when far away from the current iterate $x_k$

Line search vs trust region

Whats the fundamental difference between line search and trust region?

Problem scaling

The scaling of a problem matters for optimization performance

Problem scaling

The scaling of a problem matters for optimization performance

A problem is poorly scaled if changes to $x$ in a certain direction
produce much bigger changes in $f$ than changes to in $x$ in another direction

Problem scaling

Ex: $f(x) = 10^9 x_1^2 + x_2^2$ is poorly scaled

Problem scaling

Ex: $f(x) = 10^9 x_1^2 + x_2^2$ is poorly scaled

This happens when things change at different rates:

• Investment rates are between 0 and 1
• Consumption can be in trillions of dollars

Problem scaling

Ex: $f(x) = 10^9 x_1^2 + x_2^2$ is poorly scaled

This happens when things change at different rates:

• Investment rates are between 0 and 1
• Consumption can be in trillions of dollars

How do we solve this issue?

Constrained optimization

How do we solve constrained optimization problems?

Constrained optimization

How do we solve constrained optimization problems?

Typically as a variant of unconstrained optimization techniques

Constrained optimization: Penalty methods

Suppose we wish to minimize some function subject to equality constraints (easily generalizes to inequality) $$\min_x f(x) \,\,\, \text{subject to:} \,\, c_i(x) = 0$$

Constrained optimization: Penalty methods

Suppose we wish to minimize some function subject to equality constraints (easily generalizes to inequality) $$\min_x f(x) \,\,\, \text{subject to:} \,\, c_i(x) = 0$$

How does an algorithm know to not violate the constraint?

Constrained optimization: Penalty methods

$$Q(x;\mu) = f(x) + \frac{\mu}{2} \sum_i c_i^2(x)$$

Constrained optimization: Penalty methods

$$Q(x;\mu) = f(x) + \frac{\mu}{2} \sum_i c_i^2(x)$$

The second term increases the value of the function, bigger $\mu \rightarrow$ bigger penalty from violating the constraint

Constrained optimization: Penalty methods

$$Q(x;\mu) = f(x) + \frac{\mu}{2} \sum_i c_i^2(x)$$

The second term increases the value of the function, bigger $\mu \rightarrow$ bigger penalty from violating the constraint

The penalty terms are smooth $\rightarrow$ use unconstrained optimization techniques
to solve the problem by searching for iterates of $x_k$

Constrained optimization: Penalty methods

$$Q(x;\mu) = f(x) + \frac{\mu}{2} \sum_i c_i^2(x)$$

The second term increases the value of the function, bigger $\mu \rightarrow$ bigger penalty from violating the constraint

The penalty terms are smooth $\rightarrow$ use unconstrained optimization techniques
to solve the problem by searching for iterates of $x_k$

Also generally iterate on sequences of $\mu_k \rightarrow \infty$ as $k \rightarrow \infty$, to require satisfying the constraints as we close in

Constrained optimization: Penalty method example

Example: $$\min x_1 + x_2 \,\,\,\,\,\text{ subject to: } \,\,\, x_1^2 + x_2^2 - 2 = 0$$

Constrained optimization: Penalty method example

Example: $$\min x_1 + x_2 \,\,\,\,\,\text{ subject to: } \,\,\, x_1^2 + x_2^2 - 2 = 0$$

Solution is pretty easy to show to be $(-1, -1)$

Constrained optimization: Penalty method example

Example: $$\min x_1 + x_2 \,\,\,\,\,\text{ subject to: } \,\,\, x_1^2 + x_2^2 - 2 = 0$$

Solution is pretty easy to show to be $(-1, -1)$

The penalty method function $Q(x_1, x_2; \mu)$ is $$Q(x_1, x_2; \mu) = x_1 + x_2 + \frac{\mu}{2} (x_1^2 + x_2^2 - 2)^2$$

Constrained optimization: Active set methods

Active set methods encapsulate sequential quadratic programming (SQP) methods

Constrained optimization: Active set methods

Active set methods encapsulate sequential quadratic programming (SQP) methods

Main idea:

1. Replace the large non-linear constrained problem with a constrained quadratic programming problem
2. Use Newton's method to solve the sequence of simpler quadratic problems

Constrained optimization: Active set methods

Active set methods encapsulate sequential quadratic programming (SQP) methods

Main idea:

1. Replace the large non-linear constrained problem with a constrained quadratic programming problem
2. Use Newton's method to solve the sequence of simpler quadratic problems

Tthe Lagrangian is $$L(x,\lambda) = f(x) - \lambda^T\,c(x)$$

Constrained optimization: Active set methods

The first-order conditions $F(x,\lambda)$ can be written as, $$\begin{gather} \nabla\,f(x) - A(x)^T\,\lambda = 0 \notag\\ c(x) = 0 \notag \end{gather}$$

Any solution to the equality constrained problem, where $A(x^*)$ has full rank also satisfies the first-order necessary conditions

Constrained optimization: Active set methods

Issue: if we have many constraints, keeping track of all of them can be expensive

Constrained optimization: Active set methods

Issue: if we have many constraints, keeping track of all of them can be expensive

Main idea: recognize that if an inequality constraint is not binding, or active, then it has no influence on the solution

$\rightarrow$ in the iteration procedure we can effectively ignore it

Constrained optimization: Interior point methods

Interior point methods are also called barrier methods

Constrained optimization: Interior point methods

Interior point methods are also called barrier methods

These are typically used for inequality constrained problems

Constrained optimization: Interior point methods

Interior point methods are also called barrier methods

These are typically used for inequality constrained problems

The name interior point comes from the algorithm traversing the domain along the interior of the inequality constraints

Constrained optimization: Interior point methods

Interior point methods are also called barrier methods

These are typically used for inequality constrained problems

The name interior point comes from the algorithm traversing the domain along the interior of the inequality constraints

Issue: how do we ensure we are on the interior of the feasible set?

Constrained optimization: Interior point methods

Consider the following constrained optimization problem $$\begin{gather} \min_{x} f(x) \notag\\ \text{subject to: } c_E(x) = 0, c_I(x) \geq 0 \end{gather}$$

Constrained optimization: Interior point methods

Final step: introduce a barrier function to eliminate the inequality constraint, $$\begin{gather} \min_{x,s} f(x) - \mu \sum_{i=1}^m log(s_i) \notag\\ \text{subject to: } c_E(x) = 0, c_I(x) - s = 0 \end{gather}$$

where $\mu$ is a positive barrier parameter

Constrained optimization: Interior point methods

Final step: introduce a barrier function to eliminate the inequality constraint, $$\begin{gather} \min_{x,s} f(x) - \mu \sum_{i=1}^m log(s_i) \notag\\ \text{subject to: } c_E(x) = 0, c_I(x) - s = 0 \end{gather}$$

where $\mu$ is a positive barrier parameter

The barrier function prevents the components of $s$ from approaching
zero by imposing a logarithmic barrier $\rightarrow$ it maintains slack in the constraints

Constrained optimization: Interior point methods

Final step: introduce a barrier function to eliminate the inequality constraint, $$\begin{gather} \min_{x,s} f(x) - \mu \sum_{i=1}^m log(s_i) \notag\\ \text{subject to: } c_E(x) = 0, c_I(x) - s = 0 \end{gather}$$

where $\mu$ is a positive barrier parameter

The barrier function prevents the components of $s$ from approaching
zero by imposing a logarithmic barrier $\rightarrow$ it maintains slack in the constraints

Interior point methods solve a sequence of barrier problems until $\{\mu_k\}$ converges to zero

Best practices for optimization

Plug in your guess, let the solver go, and you're done right?

Best practices for optimization

Plug in your guess, let the solver go, and you're done right?

WRONG

Try alternative algorithms

Optimization is approximately 53% art

Try alternative algorithms

Optimization is approximately 53% art

Not all algorithms are suited for every problem $\rightarrow$ it is useful to check how different algorithms perform

Be aware of tolerances

Two main tolerances in optimization:

1. ftol is the tolerance for the change in the function value (absolute and relative)
2. xtol is the tolerance for the change in the input values (absolute and relative)

Be aware of tolerances

Two main tolerances in optimization:

1. ftol is the tolerance for the change in the function value (absolute and relative)
2. xtol is the tolerance for the change in the input values (absolute and relative)

What is a suitable tolerance?

Be aware of tolerances

Two main tolerances in optimization:

1. ftol is the tolerance for the change in the function value (absolute and relative)
2. xtol is the tolerance for the change in the input values (absolute and relative)

What is a suitable tolerance?

It depends

Be aware of tolerances

May be a substantial tradeoff between accuracy of your solution and speed

Be aware of tolerances

May be a substantial tradeoff between accuracy of your solution and speed

Common bad practice is to pick a larger tolerance (e.g. 1e-3) so the problem "works" (e.g. so your big MLE converges)

Perturb your initial guesses

Initial guesses matter

Perturb your initial guesses

Initial guesses matter

Good ones can improve performance

• e.g. initial guess for next iteration of coefficient estimates should be current iteration estimates