Advancing BCI with a transformerbased model for motor imagery classification
使用基於 transformer 的模型推進 BCI 運動圖像分類

Wangdan Liao $^{1}$ , Hongyun Liu $^{2, 3}$ & Weidong Wang $^{1, 2, 3} ⊠$
廖旺丹 $^{1}$ ，劉 $^{2, 3}$ 洪雲 & 王 $^{1, 2, 3} ⊠$ 衛東Brain-computer interfaces (BCIs) harness electroencephalographic signals for direct neural control of devices, offering significant benefits for individuals with motor impairments. Traditional machine learning methods for EEG-based motor imagery (MI) classification encounter challenges such as manual feature extraction and susceptibility to noise. This paper introduces EEGEncoder, a deep learning framework that employs modified transformers and Temporal Convolutional Networks (TCNs) to surmount these limitations. We propose a novel fusion architecture, named Dual-Stream TemporalSpatial Block (DSTS), to capture temporal and spatial features, improving the accuracy of Motor Imagery classification task. Additionally, we use multiple parallel structures to enhance the model's performance. When tested on the BCI Competition IV-2a dataset, our proposed model achieved an average accuracy of $86.46 %$ for subject dependent and average $74.48 %$ for subject independent.
腦機介面（BCI）利用腦電圖信號對設備進行直接神經控制，為運動障礙者提供顯著的好處。用於基於 EEG 的運動圖像（MI）分類的傳統機器學習方法會遇到諸如手動特徵提取和對雜訊的敏感性等挑戰。本文介紹了 EEGEncoder，這是一個深度學習框架，它採用改進的轉換器和時間卷積網路（TCN）來克服這些限制。我們提出了一種新的融合架構，命名為 Dual-Stream TemporalSpatial Block （DSTS），用於捕獲時間和空間特徵，提高運動圖像分類任務的準確性。此外，我們使用多個並行結構來提高模型的性能。當在 BCI 競賽 IV-2a 數據集上進行測試時，我們提出的模型實現了受試者依賴的平均準確性和受試者獨立 $74.48 %$ 的平均準確性 $86.46 %$ 。

Keywords Motor imagery (MI), Electroencephalography (EEG), Classification, Transformer, Temporal Convolutional Networks (TCNs)
關鍵詞運動意象（MI）、腦電圖（EEG）、分類、變壓器、時間卷積網路（TCN）

Brain-computer interfaces (BCIs) represent an emerging technological field, offering an innovative approach to human-computer interaction. By enabling direct neural communication, BCIs allow individuals to control external devices or systems solely through brain activity, bypassing traditional motor pathways. BCIs hold significant promise for applications in healthcare, rehabilitation, entertainment, and education. In the medical sector, they provide hope for individuals with motor impairments, enabling the restoration of control over bodily functions. For instance, BCIs have been crucial in helping individuals with spinal cord injuries operate prosthetic limbs and have supported stroke survivors in regaining mobility

^{1, 2}

.
腦機介面（BCI）代表了一個新興的技術領域，為人機交互提供了一種創新的方法。通過實現直接神經通信，BCI 允許個體僅通過大腦活動控制外部設備或系統，繞過傳統的運動通路。BCI 在醫療保健、康復、娛樂和教育領域的應用前景廣闊。在醫療領域，它們為運動障礙者提供了希望，使他們能夠恢復對身體機能的控制。例如，BCI 在説明脊髓損傷患者作假肢和支援中風倖存者恢復活動

^{1, 2}

能力方面發揮了關鍵作用。

A critical BCI modality is EEG-based motor imagery (MI), which utilizes electroencephalographic (EEG) signals to deduce a user’s intent for limb movement. MI signals, which are the brain’s response to the mental rehearsal of motor actions, are essential for a BCI to identify the intended limb movement and control external devices accordingly.
一種關鍵的 BCI 模式是基於 EEG 的運動圖像（MI），它利用腦電圖（EEG）信號來推斷使用者的肢體運動意圖。MI 信號是大腦對運動動作的心理排練的反應，對於 BCI 識別預期的肢體運動並相應地控制外部設備至關重要。

Researchers have traditionally relied on pattern recognition and machine learning methods, using handcrafted features to classify EEG data. These approaches have proven highly effective, enabling the development of communication aids for stroke and epilepsy patients, brainwave-controlled devices like wheelchairs and robots for individuals with mobility impairments, and remote pathology detection systems based on

{EEG}^{3 - 5}

. Despite these advancements, creating effective BCI systems remains a significant challenge. The limited spatial resolution, low signal-to-noise ratio (SNR), and dynamic nature of MI signals complicate the extraction of reliable features. Additionally, the substantial inherent noise in EEG data adds another layer of complexity to the analysis of brain dynamics and the precise classification of EEG signals.
研究人員傳統上依賴模式識別和機器學習方法，使用手工製作的特徵對 EEG 數據進行分類。這些方法已被證明非常有效，使中風和癲癇患者的通信輔助設備、行動不便人士的腦電波控制設備（如輪椅和機器人）以及基於

{EEG}^{3 - 5}

的遠端病理檢測系統成為可能。儘管取得了這些進步，但創建有效的 BCI 系統仍然是一項重大挑戰。MI 信號的空間解析度有限、信噪比（SNR）低和動態特性使可靠特徵的提取變得複雜。此外，腦電圖數據中大量的固有雜訊為大腦動力學的分析和腦電圖信號的精確分類增加了另一層複雜性。

Traditional BCIs generally encompass five main processing stages: data acquisition, signal processing, feature extraction, classification, and feedback

^{6}

. Each stage often relies on manually specified signal processing

^{7}

, feature extraction

^{8}

, and classification methods

^{9}

, requiring significant expertise and prior knowledge of the expected EEG signals. For instance, preprocessing steps are typically tailored to specific EEG features of interest, such as band-pass filtering for certain frequency ranges, which might exclude other potentially relevant EEG features outside the band-pass range. As BCI technology expands into new application areas, the demand for robust feature extraction techniques continues to grow

^{10 - 13}

.
傳統的 BCI 通常包括五個主要處理階段：數據採集、信號處理、特徵提取、分類和反饋

^{6}

。每個階段通常依賴於手動指定的信號處理

^{7}

、特徵提取

^{8}

和分類方法

^{9}

，需要大量的專業知識和預期腦電圖信號的先驗知識。例如，預處理步驟通常是針對感興趣的特定 EEG 特徵量身定製的，例如某些頻率範圍的帶通濾波，這可能會排除帶通範圍之外的其他可能相關的 EEG 特徵。隨著 BCI 技術擴展到新的應用領域，對穩健特徵提取技術的需求不斷增長

^{10 - 13}

。

Early research in EEG signal classification has significantly contributed to our understanding of MI and other cognitive tasks. For example, the study on the classification of MI BCI using multivariate empirical mode
腦電圖信號分類的早期研究為我們對 MI 和其他認知任務的理解做出了重大貢獻。例如，使用多元經驗模式對 MI BCI 進行分類的研究

decomposition (MEMD) demonstrated the effectiveness of MEMD in dealing with data nonstationarity, low SNR, and closely spaced frequency bands of interest. This approach allows for enhanced localization of frequency information in EEG, providing a highly localized time-frequency representation

^{14}

. Another study focused on emotional state classification from EEG data using machine learning approaches, highlighting the importance of power spectrum features and feature smoothing methods in improving classification accuracy

^{15}

. Additionally, research on mu rhythm (de)synchronization and EEG single-trial classification illustrated the importance of event-related desynchronization (ERD) and synchronization (ERS) patterns in discriminating between different MI tasks

^{16}

.
分解（MEMD）證明瞭 MEMD 在處理數據不穩定、低 SNR 和緊密間隔的感興趣頻段方面的有效性。這種方法允許增強 EEG 中頻率資訊的定位，提供高度本地化的時頻表示

^{14}

。另一項研究側重於使用機器學習方法從腦電圖數據中對情緒狀態進行分類，強調了功率譜特徵和特徵平滑方法在提高分類準確性

^{15}

方面的重要性。此外，對 mu 節律（de）同步和 EEG 單次試驗分類的研究說明了事件相關不同步（ERD）和同步（ERS）模式在區分不同 MI 任務

^{16}

中的重要性。

These early studies have laid a solid foundation for the field, deepening our understanding of EEG signals and MI classification. The insights gained from these traditional methods have significantly influenced the development of subsequent technologies, which continue to benefit from the robust feature extraction techniques and classification strategies established by prior research. As a result, contemporary models are better equipped to handle the complexities of EEG data, leveraging the advancements made by earlier studies to achieve improved performance in various BCI applications.
這些早期研究為該領域奠定了堅實的基礎，加深了我們對腦電信號和 MI 分類的理解。從這些傳統方法中獲得的見解對後續技術的發展產生了重大影響，這些技術繼續受益於先前研究建立的穩健特徵提取技術和分類策略。因此，現代模型能夠更好地處理複雜的腦電圖數據，利用早期研究取得的進步來提高各種 BCI 應用的性能。

In recent years, BCI technology has gained significant attention in the classification of MI tasks. Traditional MI classification methods mainly rely on manual feature extraction and machine learning algorithms. While these methods have achieved some success, they also have certain limitations, such as the cumbersome feature extraction process and the high demand for domain expertise

^{17 - 19}

. The advent of deep learning has brought new possibilities for MI classification by learning discriminative features directly from raw EEG data, thereby reducing the need for manual feature extraction. Among deep learning methods, Convolutional Neural Networks (CNNs) have become foundational due to their layered feature extraction capabilities and end-to-end learning potential. Various CNN architectures, such as Inception-CNN, Residual CNN, 3D-CNN, and Multi-scale CNN, have been widely applied in MI classification

^{20 - 25}

.
近年來，BCI 技術在 MI 任務的分類中受到了極大的關注。傳統的 MI 分類方法主要依賴於人工特徵提取和機器學習演算法。雖然這些方法取得了一些成功，但它們也有一定的局限性，例如繁瑣的特徵提取過程和對領域專業知識

^{17 - 19}

的高要求。深度學習的出現為 MI 分類帶來了新的可能性，它直接從原始腦電圖數據中學習判別性特徵，從而減少了手動特徵提取的需求。在深度學習方法中，捲積神經網路（CNN）因其分層特徵提取能力和端到端學習潛力而成為基礎。各種 CNN 架構，如 Inception-CNN、Residual CNN、3D-CNN 和 Multi-scale CNN，已在 MI 分類

^{20 - 25}

中得到廣泛應用。

In addition to CNNs, Recurrent Neural Networks (RNNs) and Temporal Convolutional Networks (TCNs) have been used to capture the temporal dynamics in EEG signals

^{26, 27}

. For instance, Kumar et al.

^{28}

proposed an LSTM model combined with FBCSP features and an SVM classifier, while Luo and Chao

^{26}

utilized FBCSP features as inputs to a Gated Recurrent Unit (GRU) model, which showed superior performance over LSTM.To overcome the limitations of individual models, researchers have attempted to combine different deep learning models. For example, integrating CNNs with LSTM to leverage their respective strengths

^{23}

. Additionally, TCNs, a novel variant of CNNs designed for time-series modeling and classification, can exponentially expand the receptive field size by linearly increasing the number of parameters, thereby avoiding the gradient issues faced by RNNs

^{29}

.
除了 CNN 之外，迴圈神經網路（RNN）和時間捲積網路（TCN）也被用來捕獲腦電圖信號

^{26, 27}

中的時間動態。例如，Kumar 等人

^{28}

提出了一種結合了 FBCSP 特徵和 SVM 分類器的 LSTM 模型，而 Luo 和 Chao

^{26}

利用 FBCSP 特徵作為門控循環單元（GRU）模型的輸入，該模型顯示出優於 LSTM.To 克服單個模型的局限性的性能，研究人員試圖結合不同的深度學習模型。例如，將 CNN 與 LSTM 集成以利用它們各自的優勢

^{23}

。此外，TCN 是專為時間序列建模和分類而設計的 CNN 的新變體，可以通過線性增加參數數量來呈指數級擴展感受野大小，從而避免 RNN 面臨的梯度問題

^{29}

。

The emergence of attention mechanisms has further advanced EEG signal decoding. Since Bahdanau et al.

^{30}

introduced attention-based models, these mechanisms have been widely applied in various fields, such as Natural Language Processing (NLP) and Computer Vision (CV)

^{30, 31}

. Recent efforts in MI classification have begun to harness the potential of transformer models, yielding promising results

^{24, 32}

.
注意力機制的出現進一步推動了腦電信號解碼。自從 Bahdanau 等人

^{30}

引入基於注意力的模型以來，這些機制已廣泛應用於各個領域，例如自然語言處理（NLP）和電腦視覺（CV）。

^{30, 31}

MI 分類的最新努力已開始利用 transformer 模型的潛力，併產生了有希望的結果

^{24, 32}

。

Despite the impressive capabilities of deep learning, it also faces significant challenges. For instance, RNNs, while adept at capturing temporal dynamics, are difficult to train, computationally costly, and susceptible to gradient vanishing problems. Similarly, CNNs excel in local feature extraction but may struggle with capturing global information. Transformer models, although effective with sequential data, often require large datasets to converge, posing a limitation with the typically scarce EEG data. In the realm of motor imagery (MI) classification, these challenges are compounded by the limited availability of publicly accessible EEG MI datasets, leading to overfitting in models with extensive parameter spaces

^{2, 33}

. Unlike more mature fields like computer vision and natural language processing, where deep learning has benefited from abundant data, EEG data presents unique hurdles such as high variability, low signal-to-noise ratio, and non-stationarity, complicating model training and generalization

^{34}

. While transfer learning offers potential, the distinct characteristics of EEG signals necessitate customized approaches

^{35, 36}

. Thus, there is a pressing need for deep learning models specifically optimized for EEG data, alongside further research to improve data understanding and model robustness in this emerging field.
儘管深度學習的功能令人印象深刻，但它也面臨著重大挑戰。例如，RNN 雖然擅長捕獲時間動態，但難以訓練，計算成本高昂，並且容易受到梯度消失問題的影響。同樣，CNN 擅長局部特徵提取，但可能難以捕獲全域資訊。Transformer 模型雖然對序列數據有效，但通常需要大型數據集才能收斂，這對通常稀缺的 EEG 數據構成了限制。在運動圖像（MI）分類領域，這些挑戰因可公開訪問的 EEG MI 數據集的可用性有限而變得更加複雜，導致具有廣泛參數空間

^{2, 33}

的模型過度擬合。與計算機視覺和自然語言處理等更成熟的領域不同，深度學習受益於豐富的數據，而 EEG 數據存在獨特的障礙，例如高可變性、低信噪比和非平穩性，使模型訓練和泛化

^{34}

複雜化。雖然遷移學習提供了潛力，但 EEG 信號的獨特特徵需要定製方法

^{35, 36}

。因此，迫切需要專門針對 EEG 數據優化的深度學習模型，同時進一步研究以提高這一新興領域的數據理解和模型穩健性。

Our proposed model amalgamates the contextual processing prowess of transformers with the nuanced temporal dynamics captured by temporal convolutional networks (TCNs). This amalgamation is meticulously engineered to discern both the global and local dependencies that are characteristic of EEG signals. In our pursuit, we have also integrated cutting-edge developments from transformer architectures to bolster our model’s efficacy. Our methodology represents a concerted effort to refine the interplay between transformers and TCNs, with the objective of bolstering the robustness and precision of EEG signal classification in a systematic and empirical fashion.
我們提出的模型將轉換器的上下文處理能力與時間捲積網路（TCN）捕獲的細微時間動態相結合。這種融合經過精心設計，以識別 EEG 信號特徵的全域和局部依賴關係。在我們的追求中，我們還整合了 transformer 架構的前沿開發成果，以提高我們模型的效率。我們的方法代表了改進變壓器和 TCN 之間相互作用的共同努力，目的是以系統和實證的方式加強 EEG 信號分類的穩健性和準確性。

Our contribution: In this paper, we introduce EEGEncoder, a novel model for EEG-based MI classification that effectively combines the temporal dynamics captured by TCNs with the advanced attention mechanisms of Transformers. This integration is further augmented by incorporating recent technical enhancements in Transformer architectures. Moreover, we have developed a new parallel structure within EEGEncoder to bolster its robustness. Our work aims to provide a robust and efficient tool to the MI classification community, thereby facilitating progress in brain-computer interface technology. Notably, our model has demonstrated outstanding performance on the BCI Competition IV dataset

2 a^{37}

, highlighting its potential and effectiveness in real-world applications.
我們的貢獻：在本文中，我們介紹了 EEGEncoder，這是一種基於 EEG 的 MI 分類的新模型，它有效地將 TCN 捕獲的時間動態與 Transformer 的高級注意力機制相結合。通過在 Transformer 架構中整合最新的技術增強功能，進一步增強了這種集成。此外，我們在 EEGEncoder 中開發了一種新的並行結構，以增強其穩健性。我們的工作旨在為 MI 分類社區提供一個強大而高效的工具，從而促進腦機介面技術的進步。值得注意的是，我們的模型在 BCI Competition IV 數據集

2 a^{37}

上表現出了出色的性能，突出了它在實際應用中的潛力和有效性。

Methods 方法

The input to the EEGEncoder model consists of segmented EEG data recorded during motor imagery tasks. These segments are preprocessed through the Downsampling Projector, which employs multiple layers of
EEGEncoder 模型的輸入由運動圖像任務期間記錄的分段 EEG 數據組成。這些段落通過 Downsampling Projector（縮減採樣投影儀）進行預處理，該投影儀採用多層
convolution to reduce the dimensionality and noise of the input signals. The processed signals are then fed into the DSTS blocks for feature extraction.
convolution 來降低輸入信號的維度和雜訊。然後將處理后的信號饋送到 DSTS 模組中以進行特徵提取。

The output of the model is a classification of the EEG segments into one of several categories, which correspond to the intended movements as labeled in the training dataset. The number of categories is determined by the specific dataset used. For instance, in the BCIC IV 2a dataset, there are four categories: left hand, right hand, feet, and tongue.
該模型的輸出是將 EEG 分段分為幾個類別之一，這些類別對應於訓練數據集中標記的預期運動。類別的數量由使用的特定數據集決定。例如，在 BCIC IV 2a 數據集中，有四類：左手、右手、腳和舌頭。

The proposed EEGEncoder model, as depicted in Fig. 1, is designed to classify motor imagery (MI) EEG signals into specific movement categories. The architecture of EEGEncoder primarily consists of a Downsampling Projector and multiple parallel Dual-Stream Temporal-Spatial (DSTS) blocks. Each DSTS block integrates Temporal Convolutional Networks (TCN) and stable transformers to capture both temporal and spatial features of EEG signals. To enhance the model’s robustness, dropout layers are introduced before each parallel DSTS branch. The following sections provide a detailed description of the structure and function of each module.
如圖 1 所示，提出的 EEGEncoder 模型旨在將運動圖像（MI） EEG 信號分類為特定的運動類別。EEGEncoder 的架構主要由一個下採樣投影儀和多個並行雙流時間空間（DSTS）模組組成。每個 DSTS 模組都整合了時間捲積網路（TCN）和穩定的變壓器，以捕獲 EEG 信號的時間和空間特徵。為了增強模型的魯棒性，在每個並行 DSTS 分支之前引入了 dropout 層。以下部分提供了每個模組的結構和功能的詳細說明。

Downsampling projector for EEG signal preprocessing
用於 EEG 信號預處理的下採樣投影儀

The Downsampling projector module within our EEG-based deep learning framework is designed to preprocess Motor Imagery EEG data, preparing it for intricate analysis by subsequent Transformer and Temporal Convolutional Network (TCN) layers. This module adeptly reshapes high-dimensional EEG sequences, characterized by a temporal resolution of 1125 and 22 channels, into a format that is conducive to convolutional processing. The main purpose of this process is to reduce the length of the sequence by passing continuous EEG signals through simple convolutional layers and average pooling layers.
我們基於 EEG 的深度學習框架中的下採樣投影儀模組旨在預處理 Motor Imagery EEG 數據，為後續 Transformer 和 Temporal Convolutional Network （TCN）層的複雜分析做好準備。該模組巧妙地將高維腦電圖序列（其特徵為時間解析度為 1125 和 22 個通道）重塑為有利於卷積處理的格式。這個過程的主要目的是通過簡單的捲積層和平均池化層傳遞連續的腦電圖信號來減少序列的長度。

Considering the EEG data analogous to an image with dimensions (1125, 22, 1), our approach involves the application of convolutional layers to extract spatial-temporal features, while concurrently mitigating noise and reducing inter-channel latency effects.
考慮到腦電數據類似於維度為（1125， 22， 1）的圖像，我們的方法涉及應用卷積層來提取時空特徵，同時減輕雜訊並減少通道間延遲效應。

The core architecture of the Downsampling projector, as illustrated in Fig. 2, comprises three convolutional layers. The first convolutional layer is designed to initiate the feature extraction process without the application of an activation function. In contrast, the second and third convolutional layers are each followed by a batch
如圖 2 所示，Downsampling 投影儀的核心架構包括三個捲積層。第一個捲積層旨在在不應用啟動函數的情況下啟動特徵提取過程。相比之下，第二個和第三個捲積層後面各跟一個 batch

Fig. 1. Architecture of the EEGEncoder. The figure illustrates the data processing pipeline within the EEGEncoder, highlighting the novel application of parallel dropout layers to enrich the diversity of the hidden state representations.
圖 1.EEGEncoder 的架構。該圖說明瞭 EEGEncoder 中的數據處理管道，突出了並行 dropout 層的新應用，以豐富隱藏狀態表示的多樣性。

$^{1}$ School of Biological Science and Medical Engineering, Beihang University, Beijing 100191, China. $^{2}$ Medical Innovation Research Division, Chinese PLA General Hospital, Beijing 100853, China. $^{3}$ Key Laboratory of Biomedical Engineering and Translational Medicine, Chinese PLA General Hospital, Beijing 100853, China. $^{⊠}$ email: wangwd301@126.com
$^{1}$ 北京航空航太大學生物科學與醫學工程學院，中國北京 100191。 $^{2}$ 中國人民解放軍總醫院醫學創新研究部，中國 100853 $^{3}$ 中國人民解放軍總醫院生物醫學工程與轉化醫學重點實驗室，中國北京市 100853. $^{⊠}$ 電子郵件：wangwd301@126.com

Advancing BCI with a transformerbased model for motor imagery classification 使用基於 transformer 的模型推進 BCI 運動圖像分類

Methods 方法

Downsampling projector for EEG signal preprocessing用於 EEG 信號預處理的下採樣投影儀

Advancing BCI with a transformerbased model for motor imagery classification
使用基於 transformer 的模型推進 BCI 運動圖像分類

Downsampling projector for EEG signal preprocessing
用於 EEG 信號預處理的下採樣投影儀