Warmup:Differentiating on the unit sphere in n dimensions

Geometrically, we all know that velocity vectors (equivalently tangents) on the sphere are orthogonal to radii. Our differentials say this algebraically:

Since $x^{T}x=1$, we have that
$2x^{T}dx=d(1)=0$, which says that at the point $x$ on the sphere (a radius, if you will), $dx$, the linearization of the constraint of moving along the sphere satisfies $dx\perp x$ (dot product is 0).

This is our first example where we have seen the infinitesimal perturbation $dx$ being constrained.

Special case: a circle

Let us consider simply the circle in the plane.

$x=(\cos \theta ,\sin \theta )$
$x^{T}dx=(\cos \theta ,\sin \theta )\cdot (-\sin \theta ,\cos \theta )d\theta =0$

We can think of $x$ as "extrinsic" coordinates, in that it is a vector in $R^2.$ On the other hand $\theta$ is an "intrinsic" coordinate, every point on the circle is specified by one $\theta$.

Suppose $A$ is symmetric. We then know that if we allow general $dx$ then
$d(\frac{1}{2}{x}^{T}Ax)=(Ax{)}^{T}dx$ and we would conclude $Ax$ is the gradient.
Now we wish to restrict to the sphere.

On the sphere

You may remember that $I-x{x}^T$ is a Projection Matrix ( meaning that its equal to its square and it is symmetric). Geometrically the matrix removes components in the $x$ direction.
In particular if $x^{T}dx=0$, $(I-x{x}^T)dx=dx$.

It follows that if $x^{T}dx=0$ then $x^{T}A(dx)=x^{T}A(I-x{x}^T)dx=((I-x{x}^T)Ax{)}^{T}dx$ so that $(I-x{x}^T)Ax$ is the gradient of $\frac{1}{2}{x}^{T}Ax$ on the sphere.

What did we just do?

To get the gradient we needed two things:

• A linearization of the function that is correct on tangents and

• A direction that is tangent (satisifes the linearized constraint)

Gradient of a general scalar function on the sphere:

$df=g(x{)}^{T}dx=((I-x{x}^T)g(x){)}^{T}dx$

Project the unconstrainted gradient to the sphere to get the constrained gradient. It is the direction of maximal increase on the sphere.

Differentiating nxn orthogonal matrices (the orthogonal group)

Do you see the structure?

Q^TdQ is anti-symmetric (sometimes called skew-symmetric).

(If $M=-M^T$, we say that $M$ is anti-symmetric. Note all anti-symmetric have 0 diagonally)

Proof: The constraint of being orthogonal is $Q^{T}Q=I$ so differentiating, $Q^{T}dQ+d{Q}^{T}Q=0$ which is the same as saying $(Q^{T}dQ)+(Q^{T}dQ{)}^T=0$, but this is the equation for being antisymmetric.

What is the dimension of the "surface" of orthogonal matrices in the $n^2$ dimensional , n by n matrix space?

For example when n=2 we have rotations (and reflections). Rotations have the form
$Q=\left(\begin{array}{cc}\cos \theta & \sin \theta \\ -\sin \theta & \cos \theta \end{array}\right)$

When n=2 we have one parameter.

When n=3, airplane pilots know about "roll, pitch, and yaw" and these are three parameters.

For general $n$ the answer is $n(n-1)/2.$

A few ways to see that:

• n^2 free parameters, orthogonality $Q^{T}Q=I$ imposes n(n+1)/2 constraints leaving

$n(n-1)/2$ free parameters.

• When we do QR, the R "eats" up n(n+1)/2 parameters leaving n(n-1)/2 for Q.

• Think about the symmetric eigenvalue problem: S = QΛQᵀ.

S has n(n+1)/2 and Λ has n, leaving n(n-1)/2 for Q.

• Think about the singular value decomposition. A = UΣVᵀ

A has n^2, and Σ has n, leaving n(n-1) to be split evenly for the orthogonal matrices U and V.

Differentiating the Symmetric Eigendecomposition

$S=Q\mathrm{\Lambda }{Q}^T$ is the eigendecomposition of a symmetric $S$ with $\mathrm{\Lambda }$ diagonal containing eigenvalues, and $Q$ othogonal with columns as eigenvectors.

$dS=dQ\mathrm{\Lambda }{Q}^T+Qd\mathrm{\Lambda }{Q}^T+Q\Lambda d{Q}^T$ which may be written $Q^{T}dSQ=Q^{T}dQ\mathrm{\Lambda }-\mathrm{\Lambda }{Q}^{T}dQ+d\mathrm{\Lambda }$

Exercise: Check that the left and right side of the above are both symmetric.

Maybe easier if one looks at the diagonal entries on their own:

$(Q^{T}dSQ{)}_{ii}=q_i^{T}dS{q}_i,$ where $q_i$ is the ith eigenvector.
Hence $q_i^{T}dS{q}_i=d{\lambda }_i.$

Sometimes we think of a curve of matrices $S(t)$ depending on a parameter such as time. If we ask for $\frac{d{\lambda }_i}{dt}$ we have that it equals $q_i^T\frac{dS(t)}{dt}{q}_i$.

How do we get the gradient $\mathrm{\nabla }{\lambda }_i$ of one eigenvalue ${\lambda }_i$?

trace($(q_i{q}_i^T{)}^{T}dS$) $=d{\lambda }_i$, thus we instantly see that $\mathrm{\nabla }{\lambda }_i=q_i{q}_i^T$

What about the eigenvectors? Those come from the off-diagonal elements:
$(Q^{T}dSQ{)}_{ij}=(Q^T\frac{dQ}{dt}{)}_{ij}({\lambda }_j-{\lambda }_i),$ if $(i\ne j)$, so we can form the elements of $Q^T\frac{dQ}{dt}$ (remember the diagonal is 0), and left multiply by $Q$ to obtain $\frac{dQ}{dt}.$

It is interesting to get the second derivative of eigenvalues when moving along a line in symmetric matrix space. For simplicity we'll start at a diagonal matrix $\mathrm{\Lambda }$.

Let $S(t)=\mathrm{\Lambda }+tE$.

Differentiating $\frac{d\mathrm{\Lambda }}{dt}=diag(Q^T\frac{dS}{dt}Q)$ we get $\frac{d^2\mathrm{\Lambda }}{d{t}^2}=diag(Q^T\frac{d^{2}S}{d{t}^2}Q)+2diag(Q^T\frac{dS}{dt}\frac{dQ}{dt}).$

Evaluating at $Q=I$ and recognizing that the first term is $0$ since we are on a line, we have

$\frac{d^2\mathrm{\Lambda }}{d{t}^2}=2diag(E\frac{dQ}{dt})$

or

$\frac{d^2{\lambda }_i}{d{t}^2}=2\sum _{k\ne i}{E}_{ik}^2/({\lambda }_i-{\lambda }_k).$

We can write this as a Taylor series.

${\lambda }_i(ϵ)={\lambda }_i+ϵE_{ii}+{ϵ}^2\sum _{k\ne i}{E}_{ik}^2/({\lambda }_i-{\lambda }_k)+\dots$