Discriminative latent subspace learning with adaptive metric learning

Abstract

Least squares regression (LSR) has been widely used in the field of pattern recognition. However, LSR-based classifier still suffers from the following issues. One is that it focuses only on the dependency between the input data and the output targets, while overlooking the local structure of instances. Another one is that using binary labels as the regression targets is too strict to fully exploit the discriminative information of the data. To address these issues, we propose a novel multiclass classification method called discriminative latent subspace learning with adaptive metric learning (DLSAML). Specifically, DLSAML adaptively learns a metric matrix for the residuals between inputs and outputs, driving smaller distances between instances of the same class and larger distances between instances of different classes in the output space. To solve the second problem, latent representations are learnt guided by the pairwise label relations as the regression targets, allowing for more flexible use of discriminative information in the data. As a combination of these two techniques, the interactive optimization of the projection matrix and metric matrix allows DLSAML to fully exploit the structural and supervised information of the data to obtain a more discriminative latent subspace for multiclass classification. Extensive experiments on several benchmark datasets have demonstrated the effectiveness of the proposed method.

Appendix 1 Proof of the Theorem 1

Proof

For simplicity, let $\mathcal {L}$ denotes the optimization problem (11). The KKT conditions for (12) are derived as follows (note that the process of solving $\textbf{M}$ and normalization constraint of $\textbf{T}$ does not involve in the Lagrange multipliers, thus we do not proof the KKT condition for them):

$$\begin{aligned} \textbf{U}&=\textbf{W},~~~~~\textbf{V}=\textbf{T}. \end{aligned}$$

(25)

$$\begin{aligned} \frac{\partial {\mathcal {L}}}{\partial {\textbf{W}}}&=\textbf{X}^{\top }\widetilde{\textbf{L}}(\textbf{X} \textbf{W}\!-\textbf{T})\textbf{M}+\textbf{P}=0. \end{aligned}$$

(26)

$$\begin{aligned} \frac{\partial {\mathcal {L}}}{\partial {\textbf{U}}}&=\textbf{X}^{\top } \widetilde{\textbf{D}}(\textbf{X} \textbf{U}\!-\!\textbf{V})\textbf{M}^{-1}\!+\!\lambda _1\textbf{U}\!-\textbf{P}=0 \end{aligned}$$

(27)

$$\begin{aligned} \frac{\partial {\mathcal {L}}}{\partial {\textbf{T}}}&=\widetilde{\textbf{L}}(\textbf{T} \!-\!\textbf{X}\textbf{W})\textbf{M}\!+\!\lambda _2(\textbf{T}\textbf{V}^{\top }\!-\!\textbf{Y} \textbf{Y}^{\top })\textbf{V} \!+\textbf{Z}=0 \end{aligned}$$

(28)

$$\begin{aligned} \frac{\partial {\mathcal {L}}}{\partial {\textbf{V}}}&=\widetilde{\textbf{D}}(\textbf{V} \!-\!\textbf{X}\textbf{U})\textbf{M}^{-1}\!+\lambda _2(\textbf{V}\textbf{T}^{\top }\! -\!\textbf{Y}\textbf{Y}^{\top })\textbf{T} -\textbf{Z}=0 \end{aligned}$$

(29)

First, the Lagrangian multiplier $\textbf{P}$ and $\textbf{Z}$ can be obtained from Algorithm 1, as follows

$$\begin{aligned} \textbf{P}^{k}=\textbf{P}+\mu (\textbf{W}-\textbf{U}), \textbf{Z}^{k}=\textbf{Z}+\sigma (\textbf{T}-\textbf{V}). \end{aligned}$$

(30)

If sequence $\{\textbf{P}^{k}\}_{k=1}^{\infty }$ and $\{\textbf{Z}^{k}\}_{k=1}^{\infty }$ converge to the stationary points, i.e., $(\textbf{P}^{k}-\textbf{P})\rightarrow 0$ and $(\textbf{Z}^{k}-\textbf{Z})\rightarrow 0$, then $(\textbf{U}-\textbf{W})\rightarrow 0$ and $(\textbf{V}-\textbf{T})\rightarrow 0$. Thus, the first two KKT conditions (25) is proved.

The third KKT condition can also be derived by utilizing the result of $\textbf{W}$ in Algorithm 1. We first rewrite (14) as follows:

$$\begin{aligned} \!\mu \textbf{W}\!=\!-(\textbf{X}^{\top }\widetilde{\textbf{L}}(\textbf{X} \textbf{W}\!-\textbf{T})\textbf{M}\!+\textbf{P})+\!\mu \textbf{U}\!\! \end{aligned}$$

(31)

Then, we have

$$\begin{aligned} \mu (\textbf{W}^{k}-\textbf{W})=\!-(\textbf{X}^{\top }\widetilde{\textbf{L}}(\textbf{X} \textbf{W}\!-\textbf{T})\textbf{M}\!+\textbf{P})+\mu (\textbf{U}-\textbf{W}) \end{aligned}$$

(32)

Based on the first condition $\textbf{U}-\textbf{W}=0$, we can infer $\textbf{X}^{\top }\widetilde{\textbf{L}}(\textbf{X} \textbf{W}\!-\textbf{T})\textbf{M}+\textbf{P}=0$, when $(\textbf{W}^{k}-\textbf{W})\rightarrow 0$.

Likewise, we can get the following equation using $\textbf{U}$ from Algorithm 1:

$$\begin{aligned} (\lambda _1+\mu )(\textbf{U}^{k}-\textbf{U})&=-(\!\textbf{X}^{\top }\widetilde{\textbf{D}}(\textbf{X} \textbf{U}\!-\!\textbf{V})\textbf{M}^{-1}\nonumber \\ {}&\quad +\lambda _1\textbf{U} -\textbf{P}+\mu (\textbf{U}-\textbf{W})). \end{aligned}$$

(33)

Since $\textbf{U}-\textbf{W}$ converges to 0, we obtain $\textbf{X}^{\top }\widetilde{\textbf{D}}(\textbf{X} \textbf{U}\!-\!\textbf{V})\textbf{M}^{-1}\!+\!\lambda _1\textbf{U}\!-\textbf{P}=0$ whenever $(\textbf{U}^{k}-\textbf{U})\rightarrow 0$.

For the fifth condition, from (18), we have the following equation:

$$\begin{aligned} \widetilde{\textbf{L}}(\textbf{T}^{k}-\textbf{T})\textbf{M} =\;&\widetilde{\textbf{L}}(\textbf{X}\textbf{W}-\textbf{T}) \textbf{M}-\lambda _2(\textbf{T}\textbf{V}^{\top }\nonumber \\ {}&\quad -\textbf{Y}\textbf{Y}^{\top })\textbf{V} -\textbf{Z}-\delta (\textbf{T}-\textbf{V}) \end{aligned}$$

(34)

On the left-hand side of Eq. (34), $\widetilde{\textbf{L}}$ is a bounded constant matrix and $\textbf{M}$ is a constant SPD matrix defined as in Eq. (22). Since $\textbf{T}-\textbf{V}$ converges to 0, we have $\widetilde{\textbf{L}}(\textbf{T} \!-\!\textbf{X}\textbf{W})\textbf{M}\!+\!\lambda _2(\textbf{T} \textbf{V}^{\top }\!-\!\textbf{Y}\textbf{Y}^{\top })\textbf{V} \!+\textbf{Z}\rightarrow 0$ whenever $(\textbf{T}^{k}-\textbf{T})\rightarrow 0$.

For the last condition, from (20), we have the following equation:

$$\begin{aligned} \widetilde{\textbf{D}}(\textbf{V}^{k}-\textbf{V})\textbf{M}^{-1}&=\widetilde{\textbf{D}}(\textbf{X}\textbf{U}-\textbf{V}) \textbf{M}^{-1}\!-\!\lambda _2(\textbf{V}\textbf{T}^{\top }\nonumber \\&\quad -\textbf{Y}\textbf{Y}^{\top })\textbf{T}\!+\!\textbf{Z}\!-\!\delta (\textbf{V}-\textbf{T}) \end{aligned}$$

(35)

On the left-hand side of Eq. (35), $\widetilde{\textbf{D}}$ is a bounded constant matrix and $\textbf{M}^{-1}$ is a constant SPD matrix, since $\textbf{M}$ is an SPD matrix as defined in Eq. (22). If $(\textbf{V}^{k}-\textbf{V})\rightarrow 0$, then $\widetilde{\textbf{D}}(\textbf{V} \!-\!\textbf{X}\textbf{U})\textbf{M}^{-1}\!+\lambda _2(\textbf{V} \textbf{T}^{\top }\!-\!\textbf{Y}\textbf{Y}^{\top })\textbf{T} -\textbf{Z}\rightarrow 0$ $(\textbf{V}-\textbf{T}=0)$ as well.

Since the solution sequence $\{\Theta ^k\}_{k=1}^{\infty }$ is assumed to satisfy the condition of $lim_{k\rightarrow \infty }(\Theta ^{k+1}-\Theta ^{k})=0$, the value of sequence $\{\Theta ^k\}_{k=1}^{\infty }$ asymptotically satisfies the KKT condition for objective function (11). $\square$