9、EEGNex：可靠的EEG信号解码模型

1、介绍：

文章：《Toward reliable signals decoding for electroencephalogram: A benchmark study to EEGNeX 》，Biomedical Signal Processing and Control ，IF：5.1\JCR：Q1

作者：Xia Chen a,*, Xiangbin Teng b, Han Chen c, Yafeng Pan c, Philipp Geyer a

单位：Department of Sustainable Building Systems, Leibniz University Hannover, Germany b Department of Psychology, The Chinese University of Hong Kong, Hong Kong Special Administrative Region, China c Department of Psychology and Behavioral Sciences, Zhejiang University, China

水平很高的一篇文章，是莱布尼兹大学、香港中文大学和浙江大学共同发表的，主要做的工作是对2018年的EEGNet模型进行改进。作者对该模型的改进，进行了多种尝试，甚至在原来的EEGNet中加入CBAM注意力机制，但是在文中作者说无效。其中，作者说明有4处改进非常成功，使得EEGNex成为媲美EEGNet的优秀的、先进的、专门处理EEG信号的深度学习轻量级模型。下面专门对该模型相对于EEGNet的改进部分进行解读：

2、四处改进：

EEGNex模型以EEGNet作为网络主干，和EEGNet一样，排除模型输入、输出两个层，还有3个层：block1、block2、block3：

1.加强对脑电输入信号的空间表征信息的提取(block1)

2.用2个空洞卷积替代深度可分离卷积(block3)

3.EEGNex模型使用逆瓶颈结构(block1、2、3)

4.在卷积层中使用填充和通过减少激活层来增加模型感受野，以扩充网络(block3)

这里给出EEGNet、EEGNex模型图：

EEGNet模型图

EEGNet解析：

第一列：输入层第二列：一层普通Conv2d卷积，提取频率信息，block1

第三列：深度卷积层，提取空间信息，block2

第四列：深度可分离卷积：先深度卷积后点向卷积，学习每个通道特征映射的时间特征，block3

第五列：输出层

EEGNex模型图

EEGNex解析：

(a) EEGNeX-8、32的总体结构可视化。EEGNeX由五个单元组成。8表示临时过滤器的数量，32表示内核大小。

(b)单元结构，由卷积层、BatchNormalization层和ELU激活层组成，block1

(c)深度卷积的说明过程。在EEGNeX中，它充当频率空间学习器/过滤器,block2

(d)膨胀机制可视化:核扫描卷积层，中间留有空隙，增加核接受野,block3

2.1 加强对脑电输入信号的空间表征信息的提取

在block1中，EEGNex比EEGNet多设置一层普通Conv2d卷积:

block1的目的是从EEG输入中提取浅层的频谱信息(kernel_size=(1，sample_rate/2))，EEGNet滤波器数量是16，但在前人研究中[1]，普通卷积深度不应过深，但是为了多提取频谱信息，复制一层普通Conv2d，两层的Conv2d的Filter=8（2层卷积提取了充分的频谱信息，filter数量也是EEGNet原来的一半）

[1].R.T. Schirrmeister, J.T. Springenberg, L.D.J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, et al., Deep learning with convolutional neural networks for EEG decoding and visualization, Hum. Brain Mapp. 38 (11) (2017) 5391–5420.

2.2 用2个空洞卷积替代深度可分离卷积

去掉深度可分离卷积层的原因：

深度可分离层由（深度卷积层+点向卷积）组成，本质是将一个卷积核分解成2个较小的卷积核[1]。因为在上一层block2中，数据经过深度卷积(channels,1)已经把通道压缩为1，所以深度可分离在这里变得多余。

添加空洞卷积原因：

1.为了增强每个提取的特征映射的时间特征学习能力

2.为了增强网络的参数空间

在第四处改进，我再对空洞卷积进行详说。

[1].A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 2017.

2.3 EEGNex模型使用逆瓶颈结构

逆瓶颈结构：根据先进的CNN构架设计建议[1、2、3]，使用扩展比为4的逆瓶颈结构，EEGNex自上到下的输出filter:8-32-64-32-8

[1].M. Tan, Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning; 2019, p. 6105–6114.

[2]. Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie. A ConvNet for the 2020s. arXiv preprint arXiv:2201.03545 2022.

[3].A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand et al. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 2017.

2.4 在卷积层中使用填充和通过减少激活层来增加模型感受野，以扩充网络

用更少的激活函数扩张网络：

EEGNex中的block3的2个Conv2d块中添加dilation_rate=1*2、1*4，使得普通2维卷积变为2维空洞卷积，来捕获全局时间特征，并减少卷积块之间的激活函数[1]，与普通卷积相比，空洞卷积有着更大的感受野，

例如：

dilation=1,感受野=3*3=9（dilation=1也叫普通卷积）

dilation=2,感受野=5*5=25

dilation=4,感受野=9*9=81

图例：

普通卷积：3*3=9

空洞卷积：5*5=25

[1].Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, S. Xie. A ConvNet for the 2020s. arXiv preprint arXiv:2201.03545 2022.

3、模型结构：

这里给出EEGNet、EEGNex详细的结构图：

EEGNet模型结构图

EEGNex模型结构图

4、模型代码：

4.1 EEGNex模型源码：keras实现

from keras.models import Model, Sequential

from keras.layers import Input, GRU, ConvLSTM2D, TimeDistributed, AvgPool2D, AvgPool1D, SeparableConv2D, DepthwiseConv2D, AveragePooling1D, Add, AveragePooling2D
from keras.layers import Dense, Flatten, Dropout, LSTM
# from keras.layers.merge import concatenate
from keras.layers import concatenate
from tensorflow.python.keras.layers.convolutional import Conv1D, MaxPooling1D, Conv2D, MaxPooling2D
from tensorflow.python.keras.layers import BatchNormalization, Activation, LayerNormalization, Reshape
# import tensorflow_addons as tfa

from keras.constraints import max_norm
from keras.regularizers import l2
from tensorflow.python.keras.utils.vis_utils import plot_model


from tensorflow.python.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint
from tensorflow import Tensor

import csv
from datetime import datetime

## 改图片名称

##### Basic models
def Single_LSTM(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(LSTM(100, input_shape=(n_features, n_timesteps), return_sequences=True, kernel_regularizer=l2(0.0001)))
    model.add(LSTM(100, return_sequences=True, kernel_regularizer=l2(0.0001)))
    model.add(LSTM(100, kernel_regularizer=l2(0.0001)))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model


def Single_GRU(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(GRU(100, input_shape=(n_features, n_timesteps), return_sequences=True, kernel_regularizer=l2(0.0001)))
    model.add(GRU(100, return_sequences=True, kernel_regularizer=l2(0.0001)))
    model.add(GRU(100, kernel_regularizer=l2(0.0001)))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='Single_GRU.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model


def OneD_CNN(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, input_shape=(n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    
    model.add(Conv1D(filters=64, kernel_size=3))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv1D(filters=64, kernel_size=3))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))

    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='OneD_CNN.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 

def OneD_CNN_Dilated(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, input_shape=(n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    
    model.add(Conv1D(filters=64, kernel_size=3, dilation_rate=2))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv1D(filters=64, kernel_size=3))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))

    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='OneD_CNN_Dilated.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 

def OneD_CNN_Causal(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, padding='causal', input_shape=(n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv1D(filters=64, kernel_size=3, padding='causal'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv1D(filters=64, kernel_size=3, padding='causal'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Dropout(0.5))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='OneD_CNN_Causal.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 

    
def OneD_CNN_CausalDilated(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(Conv1D(filters=64, kernel_size=3, padding='causal', input_shape=(n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv1D(filters=64, kernel_size=3, padding='causal', dilation_rate=2))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv1D(filters=64, kernel_size=3, padding='causal'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Dropout(0.5))
    model.add(MaxPooling1D(pool_size=2))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='OneD_CNN_CausalDilated.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 


def TwoD_CNN(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(Conv2D(filters=64, kernel_size=(1, 3), input_shape=(1, n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv2D(filters=64, kernel_size=(1, 3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv2D(filters=64, kernel_size=(1, 3)))
    model.add(Dropout(0.5))
    model.add(AvgPool2D(pool_size=(2, 1), padding='same'))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='TwoD_CNN.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model

def TwoD_CNN_Dilated(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(Conv2D(filters=64, kernel_size=(1, 3), input_shape=(1, n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv2D(filters=64, kernel_size=(1, 3), dilation_rate=2))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Conv2D(filters=64, kernel_size=(1, 3)))
    model.add(Dropout(0.5))
    model.add(AvgPool2D(pool_size=(2, 1), padding='same'))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='TwoD_CNN_Dilated.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model


def TwoD_CNN_Separable(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(SeparableConv2D(filters=64, kernel_size=(1, 3), input_shape=(1, n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(SeparableConv2D(filters=64, kernel_size=(1, 3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(SeparableConv2D(filters=64, kernel_size=(1, 3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Dropout(0.5))
    model.add(AvgPool2D(pool_size=(2, 1), padding='same'))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='TwoD_CNN.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model

def TwoD_CNN_Depthwise(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(DepthwiseConv2D(kernel_size=(1, 3), input_shape=(1, n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(SeparableConv2D(filters=64, kernel_size=(1, 3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(SeparableConv2D(filters=64, kernel_size=(1, 3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Dropout(0.5))
    model.add(AvgPool2D(pool_size=(2, 1), padding='same'))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='TwoD_CNN.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

    return model

##### 
##### 
##### 
##### Combined models
##### 
##### 
##### 

def CNN_LSTM(n_timesteps, n_features, n_outputs):
    # define model
    model = Sequential()
    model.add(TimeDistributed(Conv1D(filters=64, kernel_size=3), input_shape=(1, n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(TimeDistributed(Conv1D(filters=64, kernel_size=3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(TimeDistributed(Conv1D(filters=64, kernel_size=3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(TimeDistributed(Dropout(0.5)))
    model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
    model.add(TimeDistributed(Flatten()))
    model.add(LSTM(480, kernel_regularizer=l2(0.0001)))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='CNN_LSTM.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model

def CNN_GRU(n_timesteps, n_features, n_outputs):
    
    # define model
    model = Sequential()
    model.add(TimeDistributed(Conv1D(filters=64, kernel_size=3), input_shape=(1, n_features, n_timesteps)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(TimeDistributed(Conv1D(filters=64, kernel_size=3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(TimeDistributed(Conv1D(filters=64, kernel_size=3)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(TimeDistributed(Dropout(0.5)))
    model.add(TimeDistributed(MaxPooling1D(pool_size=2)))
    model.add(TimeDistributed(Flatten()))
    model.add(GRU(480, kernel_regularizer=l2(0.0001)))
    model.add(Dropout(0.5))
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='CNN_GRU.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model

def Single_ConvLSTM2D(n_timesteps, n_features, n_outputs):
    # kernel regulate good
    # define model
    model = Sequential()
    model.add(ConvLSTM2D(filters=64, kernel_size=(1,3), input_shape=(1, 1, n_features, n_timesteps), kernel_regularizer=l2(0.0001)))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))

    model.add(Dropout(0.5))
    model.add(Flatten())
    model.add(Dense(100, activation='elu'))
    model.add(Dense(n_outputs, activation='softmax'))
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='Single_ConvLSTM2D.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model

##### 
##### 
##### 
##### Advanced models
##### 
##### 
##### 

def EEGNet_8_2(n_timesteps, n_features, n_outputs):
    # define model 原版 EEGNet_8,2
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    
    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2,  use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(SeparableConv2D(filters=16, kernel_size=(1, 16),  use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 8), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNet.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 

def EEGNet_4_2(n_timesteps, n_features, n_outputs):
    # define model 原版 EEGNet_8,2
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=4, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    
    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2,  use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(SeparableConv2D(filters=8, kernel_size=(1, 16),  use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 8), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNet.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 


# add conv more for feature extraction
def test1(n_timesteps, n_features, n_outputs):
    # define model 原版 EEGNet_8,2
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    
    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2,  use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(SeparableConv2D(filters=16, kernel_size=(1, 16),  use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 8), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNet.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 


# # wider output
# def test2(n_timesteps, n_features, n_outputs):
#     # define model 原版
#     model = Sequential()
#     model.add(Input(shape=(1, n_features, n_timesteps)))
#     # 可以分为三个kernelsize
#     model.add(Conv2D(filters=8, kernel_size=(1, 32), use_bias = False, padding='same', data_format="channels_first"))
#     model.add(BatchNormalization())
#     model.add(Activation(activation='elu'))
#     model.add(Conv2D(filters=8, kernel_size=(1, 32), use_bias = False, padding='same', data_format="channels_first"))
#     model.add(BatchNormalization())
#     model.add(Activation(activation='elu'))

#     model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2, use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
#     model.add(BatchNormalization())
#     model.add(Activation(activation='elu'))
#     model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
#     model.add(Dropout(0.5))
    
#     model.add(SeparableConv2D(filters=16, kernel_size=(1, 16),  use_bias = False, padding='same', data_format="channels_first"))
#     model.add(BatchNormalization())
#     model.add(Activation(activation='elu'))
#     model.add(AvgPool2D(pool_size=(1, 8), padding='same', data_format="channels_first"))
#     model.add(Dropout(0.5))
    
#     model.add(Flatten())
#     model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
#     model.add(Activation(activation='softmax'))
    
#     # save a plot of the model
#     plot_model(model, show_shapes=True, to_file='EEGNet.png')
#     model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
#     return model 



# no SeparableConv2D and pooling, 2 Conv 
def test2(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    
    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2,  use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    # model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))


    model.add(Conv2D(filters=16, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 2), data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNet.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 


# wider output 
def test3(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    
    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2,  use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    # model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))


    model.add(Conv2D(filters=32, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 2), data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNet.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 




# wider output
def test4(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Conv2D(filters=32, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    
    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2,  use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    # model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))


    model.add(Conv2D(filters=32, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 2), data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNet.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 

# add pool
def EEGNeX_8_32(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(Conv2D(filters=32, kernel_size=(1, 64), use_bias = False, padding='same', data_format="channels_first"))
    model.add(BatchNormalization())

    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2, use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))

    
    model.add(Conv2D(filters=32, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 2), data_format='channels_first'))
    model.add(BatchNormalization())
    # model.add(Activation(activation='elu'))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(BatchNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNeX_8_32.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 


def test7(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 32), use_bias = False, padding='same', data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    model.add(Conv2D(filters=32, kernel_size=(1, 32), use_bias = False, padding='same', data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))

    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2, use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))

    
    model.add(Conv2D(filters=32, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 2), data_format='channels_first'))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNeX_8_32.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    
    return model 


def test8(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 32), use_bias = False, padding='same', data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))

    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2, use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    # model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))

    model.add(SeparableConv2D(filters=16, kernel_size=(1, 16),  use_bias = False, padding='same', data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    # model.add(AvgPool2D(pool_size=(1, 8), padding='same', data_format="channels_first"))
    # model.add(Dropout(0.5))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNeX_8_32.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model 


def test9(n_timesteps, n_features, n_outputs):
    # define model 原版
    model = Sequential()
    model.add(Input(shape=(1, n_features, n_timesteps)))
    # 可以分为三个kernelsize
    model.add(Conv2D(filters=8, kernel_size=(1, 32), use_bias = False, padding='same', data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))

    model.add(DepthwiseConv2D(kernel_size=(n_features, 1), depth_multiplier=2, use_bias = False, depthwise_constraint=max_norm(1.), data_format="channels_first"))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    # model.add(AvgPool2D(pool_size=(1, 4), padding='same', data_format="channels_first"))
    model.add(Dropout(0.5))

    # model.add(SeparableConv2D(filters=8, kernel_size=(1, 16),  use_bias = False, padding='same', data_format="channels_first"))
    # model.add(BatchNormalization())
    # model.add(Activation(activation='elu'))
    # # model.add(AvgPool2D(pool_size=(1, 8), padding='same', data_format="channels_first"))
    # model.add(Dropout(0.5))
    
    model.add(Conv2D(filters=8, kernel_size=(1, 16), use_bias = False, padding='same', dilation_rate=(1, 4),  data_format='channels_first'))
    model.add(LayerNormalization())
    model.add(Activation(activation='elu'))
    model.add(Dropout(0.5))
    
    model.add(Flatten())
    model.add(Dense(n_outputs, kernel_constraint=max_norm(0.25)))
    model.add(Activation(activation='softmax'))
    
    # save a plot of the model
    plot_model(model, show_shapes=True, to_file='EEGNeX_8_32.png')
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model

4.2 EEGNex模型训练测试

import numpy as np
from numpy import mean, std, dstack
import pandas as pd
from pandas import read_csv

from matplotlib import pyplot

import tensorflow
import torch
import os, gc
from time import time

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

from keras.models import Sequential
from keras.layers import Dense, Flatten, Dropout, LSTM
from tensorflow.keras.utils import to_categorical

# Selfbuilded functions
from BenchmarkModels import *


'''
This notebook loads dataset in YourDataName.npy data format and run benchmarkmodels:
- x_{}.npy (Input, format as #Samples, Channels, Lengths) 
- y_{}.npy (Label, num)

'''

DATA_LIST = ['YourDataName']
channel_last=False

# load a dataset group, such as train or test
def load_dataset_group(data):
    print('Loading data', data)
    X = np.load('x_{}.npy'.format(data))
    y = np.load('y_{}.npy'.format(data))
    # load input data
    
    return X, y

# load the dataset, returns train and test X and y elements
def load_dataset(prefix='', channel_last=False):
    # X_SHAPE:  samples, channels, lengths
    X, y = load_dataset_group(DATA)
    # One hot encode y
    y = to_categorical(y)
    # TRAIN - 0.75. VALIDATION - 0.125, TEST - 0.125
    trainX, testX, trainy, testy = train_test_split(X, y, test_size=0.25)
    valX, testX, valy, testy = train_test_split(testX, testy, test_size=0.5)
    
    print(trainX.shape, trainy.shape, valX.shape, valy.shape, testX.shape, testy.shape)
    return trainX, trainy, valX, valy, testX, testy


# summarize scores
def summarize_results(scores):
    print(scores)
    m, s = mean(scores), std(scores)
    print('Accuracy: %.3f%% (+/-%.3f)' % (m, s))
    return  m, s 


# Model traning/evaluating
def model_evaluation(trainX, trainy, valX, valy, testX, testy, Model_name, Data_name, channel_last=False, plot_model=False):
    
    # Callbacks
    callback_es = tensorflow.keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)
    callback_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5)
    
    checkpointDir = './model_weights/'
    os.makedirs(checkpointDir, exist_ok=True)
    checkpointPath = os.path.join(checkpointDir, "{}_best_{}_weights.h5".format(Data_name, Model_name))
    checkpointer = ModelCheckpoint(filepath=checkpointPath, verbose=1, save_best_only=True)

    # Params
    verbose, epochs, batch_size = 0, 200, 128
    n_timesteps, n_features, n_outputs = trainX.shape[2], trainX.shape[1], trainy.shape[1]

    trainX = trainX.reshape((trainX.shape[0], 1, n_features, n_timesteps))
    valX = valX.reshape((valX.shape[0], 1, n_features, n_timesteps))
    testX = testX.reshape((testX.shape[0], 1, n_features, n_timesteps))

    ### Model structure        
    if(Model_name == 'Single_LSTM'):
        model = Single_LSTM(n_timesteps, n_features, n_outputs)

    if(Model_name == 'Single_GRU'):
        model = Single_GRU(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'OneD_CNN'):
        model = OneD_CNN(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'OneD_CNN_Dilated'):
        model = OneD_CNN_Dilated(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'OneD_CNN_Causal'):
        model = OneD_CNN_Causal(n_timesteps, n_features, n_outputs)   
        
    if(Model_name == 'OneD_CNN_CausalDilated'):
        model = OneD_CNN_CausalDilated(n_timesteps, n_features, n_outputs)   
        
    if(Model_name == 'TwoD_CNN'):
        model = TwoD_CNN(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'TwoD_CNN_Separable'):
        model = TwoD_CNN_Separable(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'TwoD_CNN_Dilated'):
        model = TwoD_CNN_Dilated(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'TwoD_CNN_Depthwise'):
        model = TwoD_CNN_Depthwise(n_timesteps, n_features, n_outputs)
        
    ###  
    if(Model_name == 'Single_ConvLSTM2D'):
        # reshape into subsequences (samples, time steps, rows, cols, channels)
        trainX = trainX.reshape((trainX.shape[0], 1, 1, n_features, n_timesteps))
        valX = valX.reshape((valX.shape[0], 1, 1, n_features, n_timesteps))
        testX = testX.reshape((testX.shape[0], 1, 1, n_features, n_timesteps))
    
        model = Single_ConvLSTM2D(n_timesteps, n_features, n_outputs)
    
    if(Model_name == 'CNN_LSTM'):
        model = CNN_LSTM(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'CNN_GRU'):
        model = CNN_GRU(n_timesteps, n_features, n_outputs)
        
    ###

    if(Model_name == 'EEGNet_8_2'):
        model = EEGNet_8_2(n_timesteps, n_features, n_outputs)
        
    if(Model_name == 'test1'):
        model = test1(n_timesteps, n_features, n_outputs) 
    if(Model_name == 'test2'):
        model = test2(n_timesteps, n_features, n_outputs) 
    if(Model_name == 'test3'):
        model = test3(n_timesteps, n_features, n_outputs) 
    if(Model_name == 'test4'):
        model = test4(n_timesteps, n_features, n_outputs) 
        
    if(Model_name == 'EEGNeX_8_32'):
        model = EEGNeX_8_32(n_timesteps, n_features, n_outputs) 

    
    if(plot_model==True):
        print('The current running model is:', Model_name)
        print(model.summary())
    
    
    # Fit / Evaluate model
    # Fit network callback_es, callback_lr
    model.fit(trainX, trainy, epochs=epochs, batch_size=batch_size, verbose=verbose, 
              validation_data=(valX, valy), callbacks=[callback_es, callback_lr])

    # Load optimal weights
    # model.load_weights(checkpointPath)
    # Evaluate model
    _, accuracy = model.evaluate(testX, testy, batch_size=batch_size, verbose=0)
    pred_i = model.predict(testX, batch_size=batch_size)
    
    return accuracy, pred_i

for DATA in DATA_LIST:
    # Log
    path = 'ResultLog_{}.csv'.format(DATA)
    now = datetime.now()
    with open(path,'a', newline='') as f:
        csv_write = csv.writer(f)
        # Write a row of date dd/mm/YY H:M:S
        dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
        csv_write.writerow([dt_string])


    Model_list = [
                  'Single_LSTM', 'Single_GRU',
                  'OneD_CNN', 'OneD_CNN_Dilated', 
                  'OneD_CNN_Causal', 'OneD_CNN_CausalDilated',
                  'TwoD_CNN' ,'TwoD_CNN_Dilated', 
                  'TwoD_CNN_Separable','TwoD_CNN_Depthwise', 
                  'CNN_LSTM', 'CNN_GRU', 
                  'Single_ConvLSTM2D',
                  'EEGNet_8_2',
                  'EEGNeX_8_32',
                  ]
    
    
    # Clear cache
    gc.collect()
    torch.cuda.empty_cache()
    print('='*20)        
    # Load data
    trainX, trainy, valX, valy, testX, testy = load_dataset(channel_last=channel_last)

    
    for Model_name in Model_list:
        
        scores = list()
        pred = list()
        true = list()
        run_time = list()
        
        # Repeat experiment
        for r in range(5):
            if(r == 0):
                score, pred_i = model_evaluation(trainX, trainy, valX, valy, testX, testy, Model_name, DATA, plot_model=True)
            else:
                score, pred_i = model_evaluation(trainX, trainy, valX, valy, testX, testy, Model_name, DATA)
            score = score * 100.0
            print('>#%d: %.3f' % (r+1, score))
            scores.append(score)
            pred.append(pred_i)
            true.append(testy)
        # Summarize results
        mean_, std_ = summarize_results(scores)
        print("Classification Report:\n", classification_report(np.argmax(testy, axis=1), np.argmax(pred_i, axis=1), digits=3))
        
        with open(path,'a', newline='') as f:
            csv_write = csv.writer(f)
            # Write data
            csv_head = [Model_name, mean_, std_]
            csv_write.writerow(csv_head)