Problems with Equality Constraints

• Realistic optimization problems have constraints.

• If these constraints are equality constraints, then the optimization methods developed in the previous lecture can be modified to solve the problem with constraints.

Problem Formulation

minimize f(x_{1}, x_{2}, ..., x_{n}, u_{1}, u_{2}, ..., u_{m})  (1)

subject to

\begin{aligned}g_{1}(x_{1}, x_{2}, ..., x_{n}, u_{1}, u_{2}, ..., u_{m}) & = & 0 \\ g_{2}(x_{1}, x_{2}, ..., x_{n}, u_{1}, u_{2}, ..., u_{m}) & = & 0 \\ & . \\ & . \\ g_{n}(x_{1}, x_{2}, ..., x_{n}, u_{1}, u_{2}, ..., u_{m}) & = & 0 \end{aligned}  (2)

Note that if m variables u_{1}, u_{2}, ..., u_{m} could be found, the remaining n variables x_{1}, x_{2}, ..., x_{n} are fixed by the n constraints.

In compact form, the above problem can be stated as:

minimize f(x, u)  (3)

subject to

g(x, u) = 0  (4)

where x ∈ R^{n} and u ∈ R^{m} \underline{Approach # 1}:

• Eliminate the constraint equations by substituting x in terms of u in the function to the optimized.

• Use unconstrained optimization methods to

optimize the function in u. \underline{Approach # 2}:

• Develop necesary and sufficient conditions of optimality from “scratch” for problems with equality constraints.

\underline{Example 1}:

min f(x, u) = 4x^{2} + 5u^{2}  (5)

subject to

2x + 3u = 6  (6)

1. Write x in terms of u in the constraint equation.

x = \frac{6 − 3u}{2}  (7)

2. Substitute x in terms of u in the function to be minimized.

\begin{aligned}min f(u) & = & 4 (\frac{6 − 3u}{2})^{2} + 5u^{2} \\ & = & 14u^{2} − 36u + 36 \end{aligned}  (8)

3. Use method developed for minimization of unconstrained functions.

\frac{∂f}{∂u} = 0  (9)

Solving for u, we get:

u^{∗} = 1.286  (10)

Substitute this value of u in the constraint equation to get the optimal value of x.

x^{∗} = 1.071  (11)

The minimum value of the function is

f(x^{∗}, u^{∗}) = 12.857  (12)

Note that if there was no equality constraint, the minimum value of the function would have been ZERO.

\underline{Example 2}:

min f(x, u) = (u_{1}−x_{1})^{2}+(x_{1}−u_{2})^{2}+(u_{2}−x_{2})^{4}+(u_{2}−x_{3})^{4}

(13)

subject to

\begin{aligned}x_{1} + 3x_{2} + 2u_{1} − 6 & = & 0 \\ 2x_{2} + u_{1} + 3u_{2} − 6 & = & 0 \\ x_{2} + 3x_{3} + 2u_{2} − 6 & = & 0 \end{aligned}  (14)

It is not so easy to use Approach # 1 here.

Approach # 1

\underline{Advantages}:

• Utilizes methods developed for unconstrainted

optimization. \underline{Disdvantages}:

• It is tedious to eliminate constraints if there are many constraint equations.

Approach # 2: Necessary Conditions

minimize f(x, u)  (15)

subject to

g(x, u) = 0  (16)

where x ∈ R^{n} and u ∈ R^{m} A stationary point is one where df = 0 for arbitrary du while holding dg = 0 letting dx change as it will.

df = \frac{∂f}{∂x} .dx + \frac{∂f}{∂u} .du  (17)

and

dg = \frac{∂g}{∂x}.dx + \frac{∂g}{∂u}.du  (18)

Since dg = 0, this implies that

dx = − [\frac{∂g}{∂x}]^{−1} \frac{∂g}{∂u}.du  (19)

Substituting in the expression for df, we get

df = [−\frac{∂f}{∂x} [\frac{∂g}{∂x}]^{−1} \frac{∂g}{∂u} + \frac{∂f}{∂u} ] du  (20)

If df has to be zero for arbitrary du, it is necessary that

\frac{∂f}{∂u} − \frac{∂f}{∂x} [\frac{∂g}{∂x}]^{−1} \frac{∂g}{∂u} = 0  (21)

Thus, we need to solve simultaneously, the following equations:

\begin{aligned}g(x, u) & = & 0 & (n equations) \\ \frac{∂f}{∂u} − \frac{∂f}{∂x} [\frac{∂g}{∂x}]^{−1} \frac{∂g}{∂u} & = & 0 & (m equations) \end{aligned} (22)

Can you solve Example 2 now?

\underline{Example 3}: Find the stationary value of

f(x, u) = \frac{1}{2} (\frac{x^{2}}{a^{2}} + \frac{u^{2}}{b^{2}} )  (23)

subject to

g(x, u) = x + mu − c = 0  (24)

\underline{Example 4}: A chemical company owns an elliptic piece of land whose principle axes are of lengths 2a and 2b meters. It is desired to build a rectangular tank that has the largest possible perimeter that fits in this land. What are the dimensions of this tank? Hint: This problem can be mathematically formulated as follows:

Maximize P = 4(x + y)  (25)

with the constraint

\frac{x^{2}}{a^{2}} + \frac{y^{2}}{b^{2}} = 1  (26)

Approach # 2

\underline{Advantages}:

• It is not necessary to eliminate the constraint

equations \underline{Disdvantages}:

• A large number (n + m) of nonlinear equations have to solved simultaneously.

The Case for Using Numerical Methods

• For both unconstrained as well as equality constrained optimization problems, a set of nonlinear algebraic equations have to solved simultaneously.

• For all but the simplest problems, these equations are tedious to solve by “hand calculations”.

Numerical Problem Formulation

Given a set of n equations in n variables:

\begin{aligned}p_{1}(x_{1}, x_{2}, ..., x_{n}) & = & 0 \\ p_{2}(x_{1}, x_{2}, ..., x_{n}) & = & 0 \\ & . \\ & . \\ p_{n}(x_{1}, x_{2}, ..., x_{n}) & = & 0 \end{aligned}  (27)

to find a numerical solution for x starting from

x_{0} = [x_{10} x_{20} ... x_{n0}]^{T}  (28)

In compact form, we need to find the solution of

p(x) = 0  (29)

starting from x = x_{0}

Newton’s Method

Define H = \frac{∂p}{∂x}|_{x=x_{k}} where x_{k} is the value of x at the k^{th} iteration.

p(x) = p(x_{k}) + H.(x − x_{k}) + higher order terms When x = x^{∗}, p(x^{∗}) = 0. Substituting above:

0 ≈ p(x_{k}) + H(x^{∗} − x_{k})

Solving for x^{∗}, we get:

x^{∗} ≈ x_{k} − (H)^{−1} p(x_{k})

The above expression is used to develop the Newton’s method:

x_{k+1} = x_{k} − (H)^{−1} p(x_{k})  (30)

Can you solve Example 2 now?

\underline{Example 5}: Use Newton’s method to solve the following system of nonlinear equations:

\begin{aligned}3x_{1} − cos(x_{2}x_{3}) − \frac{1}{2} & = & 0 \\ x_{1}^{2} − 81(x_{2} + 0.1)^{2} + sin(x_{3}) + 1.06 & = & 0 \\ e^{−x_{1}x_{2}} + 20x_{3} + \frac{10π − 3}{3} & = & 0 \end{aligned}  (31)

starting from the initial condition

x_{0} = [0.1 0.1 − 0.1]^{T}  (32)

Using the formula

x_{k+1} = x_{k} − (H)^{−1} p(x_{k})  (33)

we get the following:

p(x_{k}) = [ \begin{matrix}3x_{1} − cos(x_{2}x_{3}) − \frac{1}{2} \\ x_{1}^{2} − 81(x_{2} + 0.1)^{2} + sin(x_{3}) + 1.06 \\ e^{−x_{1}x_{2}} + 20x_{3} + \frac{10π − 3}{3} \end{matrix} ]_{x=x_{k}}

H = [ \begin{matrix}3 & x_{3}sin(x_{2}x_{3}) & x_{2}sin(x_{2}x_{3}) \\ 2x_{1} & −162(x_{2} + 0.1) & cos(x_{3}) \\ −x_{2}e^{−x_{1}x_{2}} & −x_{1}e^{−x_{1}x_{2}} & 20 \end{matrix} ]_{x=x_{k}}

The results using Newton’s method are as follows:

\begin{matrix}k & x_{1_{k}} & x_{2_{k}} & x_{1_{k}} & ||x_{k+1} − x_{k}|| \\ 0 & 0.10000000 & 0.10000000 & -0.10000000 & - \\ 1 & 0.50003702 & 0.01946686 & -0.52152047 & 0.422 \\ 2 & 0.50004593 & 0.00158859 & -0.52355711 & 1.79 × 10^{−2} \\ 3 & 0.50000034 & 0.00001244 & -0.52359845 & 1.58 × 10^{−3} \\ 4 & 0.50000000 & 0.00000000 & -0.52359877 & 1.24 × 10^{−5} \end{matrix}