支持矩阵机：综述

为了使用传统技术解决矩阵分类问题，一种常用的启发式方法涉及将每个矩阵数据实例 ${\mathbf{X}}_i$${\mathbf{X}}_i$X_(i)\mathbf{X}_{i} 转换为向量格式。随后，使用这些向量化数据训练模型。在传统分类器中，一个高效的方法是软间隔支持向量机模型（Cortes & Vapnik, 1995），其优化问题如下所示：
$\underset{\mathbf{w},b}{min}\frac{1}{2}{\mathbf{w}}^T\mathbf{w}+\zeta \sum _{i=1}^N{\{1-y_i({\mathbf{w}}^T{\mathbf{x}}_i+b)\}}_{+}$$\underset{\mathbf{w},b}{min} \frac{1}{2}{\mathbf{w}}^T\mathbf{w}+\zeta \sum _{i=1}^N {\left\{1-y_i\left({\mathbf{w}}^T{\mathbf{x}}_i+b\right)\right\}}_{+}$min_(w,b)(1)/(2)w^(T)w+zetasum_(i=1)^(N){1-y_(i)(w^(T)x_(i)+b)}_(+)\min _{\mathbf{w}, b} \frac{1}{2} \mathbf{w}^{T} \mathbf{w}+\zeta \sum_{i=1}^{N}\left\{1-y_{i}\left(\mathbf{w}^{T} \mathbf{x}_{i}+b\right)\right\}_{+},
其中 $\mathbf{w}\in {\mathbb{R}}^{p\times q};b\in \mathbb{R};{\mathbf{x}}_i=\mathrm{vec}\left({\mathbf{X}}_i\right)$$\mathbf{w}\in {\mathbb{R}}^{p\times q};b\in \mathbb{R};{\mathbf{x}}_i=\mathrm{vec}\left({\mathbf{X}}_i\right)$winR^(p xx q);b inR;x_(i)=vec(X_(i))\mathbf{w} \in \mathbb{R}^{p \times q} ; b \in \mathbb{R} ; \mathbf{x}_{i}=\operatorname{vec}\left(\mathbf{X}_{i}\right) 表示矩阵 ${\mathbf{X}}_i,\{h{\}}_{+}:=max\{0,h\}$${\mathbf{X}}_i,\{h{\}}_{+}:=max\{0,h\}$X_(i),{h}_(+):=max{0,h}\mathbf{X}_{i},\{h\}_{+}:=\max \{0, h\} 的向量化格式，${\mathbf{X}}_i,\{h{\}}_{+}:=max\{0,h\}$${\mathbf{X}}_i,\{h{\}}_{+}:=max\{0,h\}$X_(i),{h}_(+):=max{0,h}\mathbf{X}_{i},\{h\}_{+}:=\max \{0, h\} 表示经典的 hinge 损失函数， $\zeta >0$$\zeta >0$zeta > 0\zeta>0 。

considerations, Eq. (1) is identical to the subsequent formulation for performing matrix classification directly: 
$\underset{\mathbf{W},b}{min}\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)+\zeta \sum _{i=1}^N{\{1-y_i[\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{X}}_i\right)+b]\}}_{+}$$\underset{\mathbf{W},b}{min} \frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)+\zeta \sum _{i=1}^N {\left\{1-y_i\left[\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{X}}_i\right)+b\right]\right\}}_{+}$min_(W,b)(1)/(2)tr(W^(T)W)+zetasum_(i=1)^(N){1-y_(i)[tr(W^(T)X_(i))+b]}_(+)\min _{\mathbf{W}, b} \frac{1}{2} \operatorname{tr}\left(\mathbf{W}^{T} \mathbf{W}\right)+\zeta \sum_{i=1}^{N}\left\{1-y_{i}\left[\operatorname{tr}\left(\mathbf{W}^{T} \mathbf{X}_{i}\right)+b\right]\right\}_{+}.
This illustrates that directly employing Eq. (2) for classification is insufficient to capture the inherent structure present within each input matrix. As a consequence, this approach leads to a loss of information. 

In order to consider the structural characteristics, an intuitive strategy involves capturing the correlations inherent within each input matrix by introducing a low-rank restriction on the matrix $\mathbf{W}$$\mathbf{W}$W\mathbf{W}. To tackle this issue, a number of approaches are suggested such as the low-rank SVM (Wolf et al., 2007) and the bi-linear SVM (Pirsiavash et al., 2009). However, these approaches have limitations as they demand manual pre-specification of the latent rank of $\mathbf{W}$$\mathbf{W}$W\mathbf{W} tailored to various applications. To address the aforementioned issues, Luo et al. (2015) proposed an advancement in the realm of supervised learning, SMM, which overcomes the pre-specified rank criteria, preserves the structural information of the input matrix by employing the spectral elastic net penalty for the regression matrix, and produces a better result for matrix-form data. 

SMM is a proficient matrix-based adaptation of SVM, capitalizing on the strengths of SVM, which include robust generalization capabilities. Further, it has the ability to comprehensively harness the structural insights embedded within matrix data. The objective function of SMM, 
在最大边距原则下，表达式如下：

Here, the initial term $\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)+\lambda \Vert \mathbf{W}{\Vert }_{\ast }$$\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)+\lambda \Vert \mathbf{W}{\Vert }_{\ast }$(1)/(2)tr(W^(T)W)+lambda||W||_(**)\frac{1}{2} \operatorname{tr}\left(\mathbf{W}^{T} \mathbf{W}\right)+\lambda\|\mathbf{W}\|_{*} pertains to the utilization of spectral elastic net regularization, which serves the purpose of capturing correlations inherent within individual matrices. On the other hand, the last summation term represents the hinge loss function. The term $\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)$$\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)$(1)/(2)tr(W^(T)W)\frac{1}{2} \operatorname{tr}\left(\mathbf{W}^{T} \mathbf{W}\right) can also be written as $\frac{1}{2}\Vert \mathbf{W}{\Vert }_F^2$$\frac{1}{2}\Vert \mathbf{W}{\Vert }_F^2$(1)/(2)||W||_(F)^(2)\frac{1}{2}\|\mathbf{W}\|_{F}^{2}, which represents the square Frobenius norm. 

The Frobenius norm of matrix $\mathbf{W}$$\mathbf{W}$W\mathbf{W} serves as a regularization term, aiming to find a weight matrix with a reduced rank. It is also crucial to highlight that the nuclear norm serves as a regularization factor to ascertain the rank of matrix $\mathbf{W}$$\mathbf{W}$W\mathbf{W}. Estimating the rank of a matrix can be a complex problem with NP-hard characteristics (Wang, Wang, Hu, & Yan, 2015), however, the nuclear norm is widely acknowledged as the optimal convex approximation method for assessing the rank of the matrix (Candes & Recht, 2012; Zhou & Li, 2014). Additionally, the low-rank parameter $\lambda$$\lambda$lambda\lambda governs the level of structure information incorporated for constructing the classification hyperplane. The presence of the term $\Vert \mathbf{W}{\Vert }_{\ast }$$\Vert \mathbf{W}{\Vert }_{\ast }$||W||_(**)\|\mathbf{W}\|_{*} introduces non-smoothness to the objective function of SMM. This characteristic poses a challenge when attempting to directly solve Eq. (3). Consequently, the solution for SMM is derived through the application of the alternating direction method of multipliers (ADMM) (Goldstein, O’Donoghue, Setzer, & Baraniuk, 2014). Now, by introducing an auxiliary matrix variable $\mathbf{Q}$$\mathbf{Q}$Q\mathbf{Q}, the objective function of SMM can be reformulated in the following manner: 

Table 5 
Robust and sparse models of SMM. 

s.t. $\mathbf{W}-\mathbf{Q}=0$$\mathbf{W}-\mathbf{Q}=0$W-Q=0\mathbf{W}-\mathbf{Q}=0.

这里， $F(\mathbf{W},b)=\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)+\zeta \sum _i^N{\{1-y_i[\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{X}}_i\right)+b_i]\}}_{+}$$F(\mathbf{W},b)=\frac{1}{2}\mathrm{tr}\left({\mathbf{W}}^T\mathbf{W}\right)+\zeta \sum _i^N {\left\{1-y_i\left[\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{X}}_i\right)+b_i\right]\right\}}_{+}$F(W,b)=(1)/(2)tr(W^(T)W)+zetasum_(i)^(N){1-y_(i)[tr(W^(T)X_(i))+b_(i)]}_(+)F(\mathbf{W}, b)=\frac{1}{2} \operatorname{tr}\left(\mathbf{W}^{T} \mathbf{W}\right)+\zeta \sum_{i}^{N}\left\{1-y_{i}\left[\operatorname{tr}\left(\mathbf{W}^{T} \mathbf{X}_{i}\right)+b_{i}\right]\right\}_{+} ，和 $G(\mathbf{Q})=\lambda \Vert \mathbf{Q}{\Vert }_{\ast }$$G(\mathbf{Q})=\lambda \Vert \mathbf{Q}{\Vert }_{\ast }$G(Q)=lambda||Q||_(**)G(\mathbf{Q})=\lambda\|\mathbf{Q}\|_{*} 。然后，使用增广拉格朗日乘子 $\beta$$\beta$beta\beta ，方程(4)被重写为如下形式：
$L(\mathbf{Q},\mathit{\beta },(\mathbf{W},b))=F(\mathbf{W},b)+G(\mathbf{Q})+\frac{\rho }{2}\Vert \mathbf{Q}-\mathbf{W}{\Vert }_F^2+\mathrm{tr}[{\mathit{\beta }}^T(\mathbf{Q}-\mathbf{W})]$$L(\mathbf{Q},\mathit{\beta },(\mathbf{W},b))=F(\mathbf{W},b)+G(\mathbf{Q})+\frac{\rho }{2}\Vert \mathbf{Q}-\mathbf{W}{\Vert }_F^2+\mathrm{tr}\left[{\mathit{\beta }}^T(\mathbf{Q}-\mathbf{W})\right]$L(Q,beta,(W,b))=F(W,b)+G(Q)+(rho)/(2)||Q-W||_(F)^(2)+tr[beta^(T)(Q-W)]L(\mathbf{Q}, \boldsymbol{\beta},(\mathbf{W}, b))=F(\mathbf{W}, b)+G(\mathbf{Q})+\frac{\rho}{2}\|\mathbf{Q}-\mathbf{W}\|_{F}^{2}+\operatorname{tr}\left[\boldsymbol{\beta}^{T}(\mathbf{Q}-\mathbf{W})\right].
此处， $\rho$$\rho$rho\rho 表示 ADMM 方法的固有系数。在此框架内，我们需要确定三个变量矩阵： $\mathbf{Q},\mathit{\beta }$$\mathbf{Q},\mathit{\beta }$Q,beta\mathbf{Q}, \boldsymbol{\beta} 和 $(\mathbf{W},b)$$(\mathbf{W},b)$(W,b)(\mathbf{W}, b) 。获取这些矩阵的最优解涉及迭代方法。其一般步骤为
updating these variable matrices are outlined as follows: 

The fundamental steps for solving within this context involve calculating ${\mathbf{Q}}^{(t)}$${\mathbf{Q}}^{(t)}$Q^((t))\mathbf{Q}^{(t)} and the pair ( ${\mathbf{W}}^{(t)},b^{(t)}$${\mathbf{W}}^{(t)},b^{(t)}$W^((t)),b^((t))\mathbf{W}^{(t)}, b^{(t)} ) during each iteration. To update $\mathbf{Q}$$\mathbf{Q}$Q\mathbf{Q} (for the sake of ease, we omit the superscripts in the subsequent explanation), suppose that $\mathit{\beta }$$\mathit{\beta }$beta\boldsymbol{\beta} and ( $\mathbf{W},b$$\mathbf{W},b$W,b\mathbf{W}, b ) remain constant, then the 
solution of $\mathbf{Q}$$\mathbf{Q}$Q\mathbf{Q} can be determined using the following equation: 
$\underset{\mathbf{Q}}{\arg min}G(\mathbf{Q})+\frac{\rho }{2}\Vert \mathbf{Q}-\mathbf{W}{\Vert }_F^2+\mathrm{tr}[{\mathit{\beta }}^T(\mathbf{Q}-\mathbf{W})]$$\underset{\mathbf{Q}}{\arg min}G(\mathbf{Q})+\frac{\rho }{2}\Vert \mathbf{Q}-\mathbf{W}{\Vert }_F^2+\mathrm{tr}\left[{\mathit{\beta }}^T(\mathbf{Q}-\mathbf{W})\right]$arg min_(Q)G(Q)+(rho)/(2)||Q-W||_(F)^(2)+tr[beta^(T)(Q-W)]\underset{\mathbf{Q}}{\arg \min } G(\mathbf{Q})+\frac{\rho}{2}\|\mathbf{Q}-\mathbf{W}\|_{F}^{2}+\operatorname{tr}\left[\boldsymbol{\beta}^{T}(\mathbf{Q}-\mathbf{W})\right].
By solving Eq. (5), we can derive the updating formula for the matrix $\mathbf{Q}$$\mathbf{Q}$Q\mathbf{Q} during each iteration as follows: 
$\mathbf{Q}=\frac{1}{\rho }{S}_{\lambda }(\rho \mathbf{W}-\mathit{\beta })$$\mathbf{Q}=\frac{1}{\rho }{S}_{\lambda }(\rho \mathbf{W}-\mathit{\beta })$Q=(1)/(rho)S_(lambda)(rhoW-beta)\mathbf{Q}=\frac{1}{\rho} S_{\lambda}(\rho \mathbf{W}-\boldsymbol{\beta}),
其中 $S_{\lambda }$$S_{\lambda }$S_(lambda)S_{\lambda} 是用于奇异值的阈值算子（Cai, Candès, & Shen, 2010）。类似地，通过求解以下方程式获得对 ( $\mathbf{W},b$$\mathbf{W},b$W,b\mathbf{W}, b ) 的解：
$\underset{(\mathbf{W},b)}{\arg min}F(\mathbf{W},b)-\mathrm{tr}\left({\mathit{\beta }}^T\mathbf{W}\right)+\frac{\rho }{2}\Vert \mathbf{W}-\mathbf{Q}{\Vert }_F^2$$\underset{(\mathbf{W},b)}{\arg min}F(\mathbf{W},b)-\mathrm{tr}\left({\mathit{\beta }}^T\mathbf{W}\right)+\frac{\rho }{2}\Vert \mathbf{W}-\mathbf{Q}{\Vert }_F^2$arg min_((W,b))F(W,b)-tr(beta^(T)W)+(rho)/(2)||W-Q||_(F)^(2)\underset{(\mathbf{W}, b)}{\arg \min } F(\mathbf{W}, b)-\operatorname{tr}\left(\boldsymbol{\beta}^{T} \mathbf{W}\right)+\frac{\rho}{2}\|\mathbf{W}-\mathbf{Q}\|_{F}^{2}.
随后，通过式（6）的解，更新 ( $\mathbf{W},b$$\mathbf{W},b$W,b\mathbf{W}, b ) 的公式可以表示如下：
$\mathbf{W}=\frac{1}{\rho +1}(\rho \mathbf{Q}+\mathit{\beta }+\sum _{i=1}^N{\gamma }_i{y}_i{\mathbf{X}}_i)$$\mathbf{W}=\frac{1}{\rho +1}\left(\rho \mathbf{Q}+\mathit{\beta }+\sum _{i=1}^N {\gamma }_i{y}_i{\mathbf{X}}_i\right)$W=(1)/(rho+1)(rhoQ+beta+sum_(i=1)^(N)gamma_(i)y_(i)X_(i))\mathbf{W}=\frac{1}{\rho+1}\left(\rho \mathbf{Q}+\boldsymbol{\beta}+\sum_{i=1}^{N} \gamma_{i} y_{i} \mathbf{X}_{i}\right),
$b=\frac{1}{|\mathbf{Q}|}\sum _{i=1}^N(y_i-\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{X}}_i\right))$$b=\frac{1}{|\mathbf{Q}|}\sum _{i=1}^N \left(y_i-\mathrm{tr}\left({\mathbf{W}}^T{\mathbf{X}}_i\right)\right)$b=(1)/(|Q|)sum_(i=1)^(N)(y_(i)-tr(W^(T)X_(i)))b=\frac{1}{|\mathbf{Q}|} \sum_{i=1}^{N}\left(y_{i}-\operatorname{tr}\left(\mathbf{W}^{T} \mathbf{X}_{i}\right)\right).
当达到指定的最大迭代次数时，参数更新过程终止。此时，获得 $\mathbf{Q},\mathit{\beta }$$\mathbf{Q},\mathit{\beta }$Q,beta\mathbf{Q}, \boldsymbol{\beta} 和 ( $\mathbf{W},b$$\mathbf{W},b$W,b\mathbf{W}, b ) 的最优解。然后，使用以下决策函数预测新矩阵数据 $\tilde{\mathbf{X}}$$\tilde{\mathbf{X}}$tilde(X)\tilde{\mathbf{X}} 的标签：
$\tilde{y}=\mathrm{sign}(\mathrm{tr}\left({\tilde{\mathbf{X}}}^T\mathbf{W}\right)+b)$$\tilde{y}=\mathrm{sign}\left(\mathrm{tr}\left({\tilde{\mathbf{X}}}^T\mathbf{W}\right)+b\right)$tilde(y)=sign(tr( tilde(X)^(T)W)+b)\tilde{y}=\operatorname{sign}\left(\operatorname{tr}\left(\tilde{\mathbf{X}}^{T} \mathbf{W}\right)+b\right).

SMM outperforms SVM in terms of performance due to the preservation of the structural information in the input matrix. However, it considers the hinge loss function and $l_2$$l_2$l_(2)l_{2} norm in the objective function which reduces the robustness (Wu & Liu, 2007) and sparsity (Tanveer, Sharma, Rastogi, & Anand, 2021), respectively. Moreover, the input data often contains distortions from measurement artifacts, outliers, and unconventional sources of noise. Consequently, the obtained classifier may have poor performance. Thus, in order to tackle the aforementioned challenges, various variants of SMM have been proposed in the literature. To counter intra-sample outliers, Zheng, Zhu, Heng (2018) proposed a robust SMM (RSMM). It decomposed the input matrix into a latent low-rank clean matrix plus a sparse noise matrix and used only the clean matrix for training, which makes it robust to intra-sample outliers. Also, to enhance the sparseness, it employs the $l_1$$l_1$l_(1)l_{1} norm instead of the $l_2$$l_2$l_(2)l_{2} norm. The $l_1$$l_1$l_(1)l_{1} norm optimization problems tend to drive some of the coefficients to exactly zero and encourage sparse solutions (Tanveer et al., 2021). The time complexity of RSMM is $O\left(N^{2}pq\right)\times s$$O\left(N^{2}pq\right)\times s$O(N^(2)pq)xx sO\left(N^{2} p q\right) \times s. Following the same concept, Razzak et al. (2023) proposed SMM that simultaneously performs matrix Recovery (SMMRe), a variant of SMM via joint $l_{2,1}$$l_{2,1}$l_(2,1)l_{2,1} and nuclear norm minimization. The objective function of SMMRe combines the property of matrix recovery along with low rank and joint sparsity to deal with complex high-dimensional noisy data. 

Other approach to build a robust model is by incorporating a robust classification loss function. In light of this, Qian et al. (2019) proposed robust multicategory SMM (RMSMM) which makes SMM robust by using the truncated hinge loss function (Wu & Liu, 2007) rather than hinge loss. The hinge loss is unbounded and can grow indefinitely for outliers away from the optimal hyperplane. In contrast, the truncated hinge loss limits the impact of such outliers by capping the loss at a predefined value. As a result, the truncated hinge loss is resistant to outliers. In a similar way, Gu et al. (2021) proposed ramp sparse SMM model (RSSMM) to improve the robustness of SMM. RSSMM uses 
smooth ramp loss function instead of hinge loss, which also limits the maximum loss and weakens the sensitivity to outliers. 

Here, we delved into a comprehensive analysis of robust and sparse variants within the realm of SMM models. By examining these nuanced approaches, we gained valuable insights into enhancing model robustness and promoting sparsity. To enhance robustness against outliers or noise, two primary strategies are considered in the literature. First, decompose the input matrix into clean and noise matrices and use only the clean matrix for training (Razzak et al., 2023; Zheng, Zhu, Heng, 
2018). Second, to integrate robust loss functions, such as truncated hinge/pinball loss, to limit the impact of outliers and enhance noise resistance (Gu et al., 2021; Qian et al., 2019). On the other hand, to enhance the sparsity, a key approach involves introducing regularization terms that combine the $l_1$$l_1$l_(1)l_{1} norm (Wang et al., 2022; Zheng, Zhu, Qin, Chen et al., 2018). Further, introducing novel regularization terms that encourage sparse solutions by driving some coefficients to zero may be incorporated to enhance sparsity. Table 5 presents an overview of the various robust and sparse variants of SMM.