A Review of Speaker Diarization- Recent Advances with Deep Learning

A Review of Speaker Diarization- Recent Advances with Deep Learning

论文：A Review of Speaker Diarization- Recent Advances with Deep Learning

Speaker Diarization：SD

1. Definition: The goal (task) of SD is to label audio or video recordings with categories corresponding to speaker identities, in short, a task of identifying " who spoke when "
2. This article:
   1. Review the historical development of SD technology and the latest progress of SD based on neural network
   2. Discuss SD and downstream tasks such as ASR for integration

Introduction

In the process of Diarization, audio data will be segmented and clustered into groups of speech segments with the same speaker identity/label, usually this process does not require any prior knowledge about the speaker.

A traditional SD consists of several independent submodules:

Front-end processing includes speech enhancement, reverberation, speech separation, target speaker extraction, etc. VAD is used to separate silent segments, and then converts the original audio into acoustic features or embedding vectors. In the clustering stage, the converted speech segments are used for Grouping and labeling, followed by fine-tuning, each sub-module above is independently trained and optimized.

Historical development of SD

1. Early use of generalized likelihood ratio GRL, Bayesian Information Criterion II (BIC) for speaker change point detection and clustering distances between speech segments
2. Improved methods include beamforming, information bottleneck clustering (IBC), variational Bayesian (VB), joint factor analysis (JFA), etc.
3. After i-vector was proposed, it successfully replaced MFCC and other features, and combined with PCA, VB-GMM, PLDA, etc.
4. In the future, the neural network will be used to extract speaker embedding, such as d-vector, x-vector, etc.
5. End-to-end neural diarization replaces each module of the traditional SD with a neural network, which is very promising

motivation

The paper An overview of automatic speaker diarization systems (2016), gives an overview of different SD systems and subtasks in broadcast news and CTS data, covering the time period of the 1990s and early 2000s.

The paper Speaker diarization: A review of recent research (2012) focuses more on the SD of conference presentations and their respective challenges.

This article focuses more on techniques for mitigating the problem in a conferencing environment. In conference settings, there are often more participants than broadcast news or CTS data, and multimodal data is often available.

SD Overview and Classification

Four classifications are based on two criteria:

Criterion 1: Whether the SD model is trained based on the SD objective function
Criterion 2: Whether multiple modules are jointly optimized for an objective function

Diarization evaluation metrics

1. Diarization Error Rate：$$THER\_=\frac{\text{ FA }+\text{ Missed }+\text{ Speaker-Confusion }}{\text{ Total Duration of Time }}$$In order to realize the one-to-one mapping, the Hungarian algorithm is adopted.
2. Jaccard Error Rate：$$JER=\frac{1}{N}\sum _i^{N_{\text{ref }}}\frac{{\mathrm{F}\mathrm{A}}_i+{\mathrm{M}\mathrm{I}\mathrm{S}\mathrm{S}}_i}{{\mathrm{T}\mathrm{O}\mathrm{T}\mathrm{A}\mathrm{L}}_{\mathrm{i}}}$$The error rate for each speaker is first calculated and then averaged to calculate JER.

DER may exceed 100%. JER Impossible.

3. Word-level Diarization Error Rate: Measure errors caused by vocabulary

Individual modules of the SD system

This section gives an overview of SD algorithms that are trained under non-SD targets. That is the principle of each independent module~

front-end processing

First set $s_{i,f,t}\in C$ is speaker$i$ at frequency bin$f$ and timettThe STFT feature under$t$$h_{i,f,t}\in C$ and additive noise$n_{t,f}\in C$ 's combination:$$x_{t,f}=\sum _{i=1}^K\sum _t{h}_{i,f,}{s}_{i,f,t-\tau }+n_{t,f}$$Among them, $K$ denotes the number of speakers in the audio.

The goal of front-end processing is, given the observed signal X = ( { xt , f } f ) t \mathbf{X}=\left(\left\{x_{t, f}\right\}_f\right)_tX=({ xt,f​}f​)t​To estimate the original signal ${\hat{\mathbf{x}}}_{i,t}$ ：$${\hat{x}}_{i,t}=\mathrm{FrontEnd}(\mathbf{X}),i=1,\dots ,K$$其中，${\hat{\mathbf{x}}}_{i,t}\in {\mathbb{C}}^D$ means the$Estimated\; STFT\; features\; for\; i$ speakers.

Voice Enhancement and Noise Reduction

Speech enhancement is used to suppress the noise component of noisy speech. The speech enhancement method based on LSTM is: $${\hat{x}}_t=LSTM\left(X\right)$$ This is a regression based method by minimizing${\mathcal{L}}_{\mathrm{M}\mathrm{S}\mathrm{E}}={\Vert {\mathbf{s}}_t-{\hat{x}}_t\Vert }^2$ to achieve,${\mathbf{s}}_t$Usually the log-likelihood spectrum.

de-reverb

De-reverberation is implemented based on statistical signal processing methods. The most widely used method is weighted prediction error (WPE) based dereverberation by decomposing the original signal into two parts: early response and late reverberation: xt , $$x_{t,f}=\sum _t{h}_{f,}{s}_{f,t-}=x_{t,f}^{\text{early }}+x_{t,f}^{\text{late }}$$WPE estimated filter coefficient ${\hat{h}}_{f,t}^{\mathrm{w}\mathrm{p}\mathrm{e}}\in C$ , based on MLE to preserve the early response while compressing the late reverberation:$${\hat{x}}_{t,f}^{\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{l}\mathrm{y}}=x_{t,f}-\sum _{\tau =D}^L{\hat{h}}_{f,t}^{\mathrm{w}\mathrm{p}\mathrm{e}}{x}_{f,t-}$$Among them, $\Delta$ denotes the number of frames to be separated,$L$ represents the size of the filter.

The advantage of WPE is that it is based on linear filtering, does not introduce signal distortion, and can be safely used in downstream tasks.

voice separation

The effectiveness of beamforming-based multi-channel speech separation has been extensively demonstrated. For example, in the CHiME-6 challenge, the guided source separation (GSS) based multi-channel speech extraction technique achieves the best results.

While single-channel speech separation usually does not show any significant effect in realistic multi-speaker scenarios. Monophonic speech separation systems typically produce redundant non-speech or repetitive speech signals for non-overlapping areas, so audio "leakage" leads to many false alarms.

Some papers proposed a leakage filtering method to solve this problem, and after adopting this method, the performance of SD was observed to improve significantly.

WHAT

VAD: Distinguishes speech from non-speech such as background noise.

The SAD system mainly consists of two main parts. One is front-end feature extraction, commonly used acoustic features, such as zero-crossing rate, pitch, signal energy, high-order statistics in the residual domain of linear predictive coding or MFCC, etc. The second is a classifier, which predicts whether an input frame contains speech or not. Classifiers include traditional models based on GMM and HMM, as well as models based on DNN, CNN and LSTM.

The performance of VAD greatly affects the performance of SD, because it may generate a large number of false positive silence segments (that is, it is clearly silent segments but not identified) or lose some speech segments.

It is common practice in SD tasks to report DER with a setting of "oracle VAD", indicating that the system output is the output using the same VAD as GT.

Split

In the context of SD, speech segmentation is the splitting of input audio into segments to obtain speaker-unified segments. Therefore, the output unit of the SD system is determined by a segmentation process, and speech segmentation for SD falls into two categories:

• Segmentation by speaker-change point detection Segmentation based on speaker change point detection
• uniform segmentation Uniform segmentation

Segmentation based on speaker change point detection was standard in early SD systems. Speaker change points are detected by two hypotheses: H0 assumes that the left and right speech windows are from the same speaker, while H1 assumes that they are from different speakers. A measure-based method is used for hypothesis testing. First, the speech features are considered to conform to the Gaussian distribution $N(\mu ,\Sigma )$ ，则两个假设可以表示为：$$\begin{array}{rl}{H}_0:& {\mathbf{x}}_1\cdots {\mathbf{x}}_N\sim \mathcal{N}(\mu ,\Sigma )\\ H_1:& {\mathbf{x}}_1\cdots {\mathbf{x}}_i\sim \mathcal{N}\left({\mu }_1,{\Sigma }_1\right)\\ & {\mathbf{x}}_{i+1}\cdots {\mathbf{x}}_N\sim \mathcal{N}\left({\mu }_2,{\Sigma }_2\right)\end{array}$$其中，$\left({\mathbf{x}}_i\mid i=1,\cdots ,N\right)$ is the speech feature sequence. Measures available then include Kullback-Leibler (KL) distance, generalized likelihood ratio (GLR), and BIC.

Where BIC is the most commonly used, the BIC value between two models from two hypotheses is expressed as follows: $$BIC(i)=N\log |\Sigma |-N_1\log |{\Sigma }_1|-N_2\log |{\Sigma }_2|-\lambda P$$ where,$P$ is the penalty term,$P=\frac{1}{2}\left(d+\frac{1}{2}d(d+1)\right)\log N$ ，$d$ represents the dimension of the feature,$N=No.1+N2$ represent frame length respectively,$l=1$ , when the present formula is established, the position$i$ is considered a change point:$$\left\{\underset{i}{\max }BIC(i)\right\}>0$$
In general, the segment lengths of segments based on speaker change point detection are inconsistent.

With the introduction of x-vector and DNN-based embedding, the uniform segmentation method is gradually adopted. This difference may reduce the "fidelity" of the speaker representations because of the different segmentation lengths.

In uniform splitting, a given audio input is split with a fixed window length and overlap length. However, the selection of the length of different segments is a problem that requires trade-off consideration:

• Needs to be short enough so as not to contain multiple speakers
• But not too short to capture enough acoustic features to extract reliable speaker representations

Speaker Representation and Similarity Measurement

Speaker representation plays a crucial role for SD systems to measure similarity between speech segments

Measure-based similarity measure

The methods used for speech segmentation are also used to measure the similarity of speech segments, such as KL distance, GLR, BIC, etc.

As in the case of segmentation, BIC-based methods, where the similarity between two segments is also computed by the above equation, is one of the most widely used measures due to its effectiveness and ease of implementation.

Measure-based similarity measures are often used together with segmentation methods based on speaker change point detection.

Joint Factor Analysis (JFA) i-vector and PLDA

GMM-UBM has been successfully applied to speaker verification tasks before. The UBM consists of a large GMM trained to represent the speaker-dependent distribution of acoustic features. But the speaker verification system based on GMM-UBM has the problem of inter-session variability.

JFA compensates for the variability problem by modeling inter-speaker variability and channel variability separately. JFA uses the GMM super vector, which is the concatenated mean of the adaptive GMM, for example, the speaker-independent GMM mean vector $m_c\in {\mathbb{R}}^{F\times 1}$ , then the supervector (supervector)$M$ 为：$$\mathbf{M}={\left[m_1^t,m_2^t,\dots ,m_C^t\right]}^{tIn}$$
JFA, the GMM supervector is decomposed into four parts: speaker-independent, speaker-dependent, channel-dependent and residual components:$${\mathbf{M}}_J={\mathbf{m}}_J+Vy+\mathbf{U}\mathbf{x}+Dz$$ where$m_J$is a speaker-independent supervector, $V$ is the speaker correlation matrix,$U$ is the channel correlation matrix,$D$ is the speaker-related residual matrix, vector$y$ represents the speaker factor,$x$ represents the channel factor,$z$ represents the residual factor of a specific speaker, and all vectors satisfy$N(0,1)$ prior distribution.

Dehak et al. proposed a method to combine the channel and speaker spaces into a combined change space through the total change matrix, and denote it as $T$ , the corresponding weight vector$w$ is called i-vector and is considered to be the speaker representation vector, at this time$\mathbf{M}$ 可以表示为：$${\mathbf{M}}_I={\mathbf{m}}_I+\mathbf{T}\mathbf{w}$$其中，$m_I$Supervectors for speaker independence and channel independence.

The process of extracting the i-vector is a MAP estimation problem, using UBM, average supervector and the total change matrix $T$ extracted Baum–Welch statistics as parameters.

i-vector has been used not only in speaker recognition research but also in many SD studies and has shown superior performance compared to measure-based methods such as BIC, GLR and KL.

The intersession variability in the i-vector method is further compensated using back-end processing such as linear discriminant analysis (LDA) and within-class covariance normalization (WCCN), and then the cosine similarity score is calculated, which is later replaced by probabilistic LDA (PLDA ) model to replace.

G-PLDA applies i-vector Gaussianization and generates Gaussian hypotheses in PLDA, which was originally used for speaker verification. Usually, PLDA for the jjthiiin the$j\; speech\; (conversation)$$Representation\; of\; i$ speakers${\varphi }_{ij}$Definition: $${\varphi }_{ij}=m+{\mathbf{F}\mathbf{h}}_i+{\mathbf{G}\mathbf{w}}_{ij}+{ϵ}_{ij}$$where $\mu$ is the mean vector,$F$ is the speaker change matrix,$G$ is the channel change matrix,${ϵ}_{ij}$represents the residual component. ${\mathbf{h}}_i,{\mathbf{w}}_{ij}$As a hidden variable conforming to Gaussian distribution, during training, $m,S,\mathbf{F},G$ is estimated using the EM algorithm, testing two hypotheses simultaneously:

• $H_0$Both samples come from the same speaker
• $H_1$The two samples come from different speakers.
  In the hypothesis $H_0$Next, the speaker representations of the two scripts are ${\varphi }_1,{\varphi }_2$Through the public hidden variable ${\mathbf{h}}_{12}$ 建模为：$$\left[\begin{array}{l}{\varphi }_1\\ {\varphi }_2\end{array}\right]=\left[\begin{array}{l}\mathbf{\mu }\\ \mathbf{\mu }\end{array}\right]+\left[\begin{array}{lll}\mathbf{F}& \mathbf{G}& 0\\ \mathbf{F}& 0& \mathbf{G}\end{array}\right]\left[\begin{array}{l}{\mathbf{h}}_{12}\\ {\mathbf{w}}_1\\ {\mathbf{w}}_2\end{array}\right]+\left[\begin{array}{l}{ϵ}_1\\ {ϵ}_2\end{array}\right]$$
  在假设 $H_1$ 下，则通过不同的隐变量 ${\mathbf{h}}_1,{\mathbf{h}}_2$ 建模：$$\left[\begin{array}{l}{\varphi }_1\\ {\varphi }_2\end{array}\right]=\left[\begin{array}{l}\mathbf{\mu }\\ \mathbf{\mu }\end{array}\right]+\left[\begin{array}{cccc}\mathbf{F}& \mathbf{G}& 0& 0\\ 0& 0& \mathbf{F}& \mathbf{G}\end{array}\right]\left[\begin{array}{c}{\mathbf{h}}_1\\ {\mathbf{w}}_1\\ {\mathbf{h}}_2\\ {\mathbf{w}}_2\end{array}\right]+\left[\begin{array}{c}{ϵ}_1\\ {ϵ}_2\end{array}\right]$$
  In G-PLDA, suppose$\varphi$ comes from a Gaussian distribution, thus:$$p(\varphi \mid \mathbf{h},\mathbf{w})=N(\varphi |m+\mathbf{F}\mathbf{h}+\mathbf{G}\mathbf{w},\Sigma )$$
  Based on the above formulas, the log likelihood ratio is calculated as:$$s\left({\varphi }_1,{\varphi }_2\right)=\log p\left({\varphi }_1,{\varphi }_2|H_0\right)-\log p\left({\varphi }_1,{\varphi }_2|H_1\right)$$
  In speaker verification, choose to accept the hypothesis$H_0$Still $H_1$. In SD, $s\left({\varphi }_1,{\varphi }_2\right)$ to check the similarity between two clusters.

Speaker Representation Based on Neural Network

DNN automatically learns the mapping without specifying any factors, which is not as interpretable as JFA.

Extracting speaker representations based on DNN does not require a predefined model (such as GMM-UBM).

It is more efficient than JFA in the inference stage and does not involve matrix inversion operations.

The most classic is the d-vector method, which uses stacked filter bank features that include context frames as input features, uses cross-entropy loss to train multiple fully connected layers, and obtains d-vector from the last fully connected layer:

x-vector further improves speaker representation. Latency architecture and statistics pooling layer distinguish x-vector from d-vector. The statistics pooling layer aggregates the frame-level output of the previous layer and calculates its mean and standard deviation, which are passed to the next layer. Thus, it can allow x-vectors to be extracted from variable-length inputs:

clustering

The clustering algorithm clusters speech segments based on the speaker representations and similarity measures obtained in the previous section

Aggregative Hierarchical Clustering (AHC)

It is a bottom-up clustering, starting from a single data, merging the clusters with the highest similarity at each step, and merging iteratively:

For the SD task, the iteration termination condition of AHC can use the similarity threshold or the target number of clusters.

If PLDA is used as the distance metric, the AHC process should be at $s({\varphi }_1,{\varphi }_2)=0$ , but when the number of speakers is known, then the number of clusters of AHC is equal to the number of speakers to stop iterations.

spectral clustering

Spectral clustering follows these steps:

1. Calculate the similarity matrix: the original similarity value $d$ through kernel$\exp \left(-d^2/p^2\right)$to calculate, where$\sigma$ is the scaling parameter.
2. Calculating the Laplacian matrix: There are two calculation methods for the graph Laplacian matrix: normalization and non-normalization. degree matrix $D$ is a diagonal matrix, and${\mathbf{d}}_{\mathbf{i}}={\sum }_{j=1}^n$, among which $a_{ij}$is the similarity matrix AAelements of $A.$
   1. Normalization: $\mathrm{L}={\mathrm{D}}^{-1/2}{AD}^{-1/2}$
   2. Unnormalized: $L=D-\mathbf{A}$
3. Definition: $L=\mathbf{X}\mathbf{\Lambda }{\mathbf{X}}^{\top }$
4. Quadratic normalization (optional): for $X$ is row normalized so that$y_{ij}=x_{ij}/{\left({\sum }_j{x}_{ij}^2\right)}^{1/2}$
5. Number of speakers: Estimate the number of speakers by finding the largest eigengap
6. Spectral embedding clustering: $The\; k$ smallest eigenvalues ​​and corresponding eigenvectors are spliced ​​to construct a matrix$U\in {\mathbf{R}}^{n\times k}$ ， U U The row vector of$U$$k-$ dimensional spectral embedding, and finally use a clustering algorithm to cluster the spectral embeddings (such as k-means clustering).
   There are many variants of spectral clustering, and NJW is often used for SD tasks. Different from AHC, spectral clustering mainly uses cosine distance. In addition, LSTM-based similarity measures and spectral clustering are also competitive. Depending on the data set, the method of cosine distance + spectral clustering is better than LPDA + AHC.

Other clustering algorithms

1. k-means
2. mean-shift clustering, used with KL distance in the SD task

Post-processing

heavy division

Re-segmentation is a process of refining speaker boundaries.

Viterbi rearrangement method based on BaumWelch's algorithm: Alternately applying an estimate of the GMM corresponding to each speaker and a Viterbi-based re-segmentation using the estimated speaker GMM

Hidden Markov Model (VB-HMM) based on Variational Bayesian: The effect is better than that of Viter proportion segmentation, see the following chapters for details.

system integration

Fusion of multiple SD systems can improve accuracy.

In SD system fusion, there are some special problems:

1. Speaker labels are not standardized across different SD systems (i.e. the same speaker may be labeled differently)

2. The estimated number of speakers may vary

3. Estimated time boundaries may also differ across multiple SD systems
   . Solutions are:

4. The recorded result sequence from each SD system is regarded as an object to be clustered, and AHC is used to cluster the result set, and finally clustered into two clusters. In a large cluster, the one closest to other records is the final record.

5. Two SD systems are combined by finding matches between two speaker clusters and then rearranging based on the matching results.

6. The DOVER method: Combining the results of multiple SDs in a voting-based scheme while aligning the labels:

7. The DOVER method has an implicit assumption that there are no overlapping speeches (at most one speaker per time index), in order to combine this method with overlapping speakers, there are two approaches:

   • Align speaker labels from different diarization results with root hypotheses and estimate each speaker's speech activity based on their weighted vote scores for each subsegment.
   • Multiple hypothetical speakers are aligned by weighted k-part graph matching, and the number of speakers k for each subsegment is estimated based on the weighted average of multiple systems to select the top-k voted speaker labels.

Joint Optimization of Segmentation and Clustering

This section presents a VB-HMM-based SD technique that can be viewed as a joint optimization of segmentation and clustering.

VB-HMM is an extension of VB-based speaker clustering, which introduces HMM to constrain speaker switching. Suppose the speech features $\mathbf{X}=({\mathbf{x}}_t|t=1,\dots ,T)$ is the speech feature, where each HMM state corresponds to$One\; of\; the\; K$ possible speakers. Suppose there is$M$ states, introduce$MvariableZ$ = ( zt ∣ 1 , … , T ) \mathbf{Z}=(\mathbf{z}_t|1,\dots,T)$\mathbf{Z}=({\mathbf{z}}_t|1,\dots ,T)$ means the$k$ speakers at time$When\; T$ speaks,${\mathbf{z}}_t$The $k$ elements are 1 and the rest are 0. At the same time based on the hidden variable$\mathbf{Y}=({\mathbf{y}}_k|k=1,\dots ,K)$ 对 ${\mathbf{x}}_t$The distribution of is modeled, where ${\mathbf{y}}_k$Indicates the $The\; low-dimensional\; vector\; of\; k$ speakers, then$\mathbf{X},\mathbf{Y},\mathbf{Z}$ 的联合分布为：$$P(\mathbf{X},\mathbf{Z},\mathbf{Y})=P(\mathbf{X}\mid \mathbf{Z},\mathbf{Y})P(\mathbf{Z})P(\mathbf{Y})$$其中，$P(\mathbf{X}\mid \mathbf{Z},\mathbf{Y})$ 是 GMM 建模的发射概率（也就是 HMM 的观测概率 $B$），其平均向量由 $\mathbf{Y}$ 表示，$P\left(Z\right)$ is the transition probability of HMM,$P\left(Y\right)$是Y \mathbf{Y}The prior distribution of$Y.$

Then the SD problem can be expressed as, maximize the posterior distribution $P(\mathbf{Z}\mid \mathbf{X})=\int P(\mathbf{Z},\mathbf{Y}|X)dY$ .

But it is difficult to solve this problem directly, so use variational Bayesian to approximate $P(Z,Y|X)$ Model parameters.

A simplified VB-HMM based on x-vector, called VBx, uses the x-vector based on the PLDA model to calculate $P(X\mid Z,Y)$ . The original VB-HMM is frame-level, while VBx is based on x-vector, so it can be regarded as a joint clustering method.

VB-HMM is usually used in the last step of SD system.

Advances in Deep Learning-Based SD

We first introduce the method of using DNN for some of the previous independent modules, and then introduce the method of unifying several parts of SD into a single network.
Overview of DNN-based SD:

Single Module Optimization

Speaker clustering enhancement based on deep learning

Speaker diarization with session-level speaker embedding refinement
using graph neural networks paper proposes a GNN-based method:

This method aims to refine the similarity matrix in spectral clustering. Suppose the speaker embedding sequence { e 1 , … e N } \left\{\mathbf{e}_1, \ldots \mathbf{e}_N\right\}{ e1​,…eN​} where$N$ is the sequence length, the input xi of GNN${\mathbf{x}}_i^0$ 为：$\left\{{\mathbf{x}}_i^0={\mathbf{e}}_i\mid i=1,\dots ,N\right\}$ , theppThe output of layer$p$$$x_i^{(p)}=p\left(\mathbf{W}\sum _j{\mathbf{L}}_{i,j}{x}_j^{(p-1)}\right)$$ where,$L$ is a normalized similarity matrix (with self-loop connection),$W$ is theppthThe weight matrix to be trained in layer$p$$\sigma (\cdot )$ is a non-linear function that is trained by minimizing the distance between the reference and estimated similarity matrices. And a combination of histogram loss and kernel norm is used to calculate the distance.

The paper Self-attentive similarity measurement strategies in speaker diarization introduces a self-attention-based neural network model to generate similarity matrices directly from speaker embedding sequences. Multi-scale speaker diarization with neural affinity score fusion fuses several similarity matrices with different temporal resolutions into a single similarity matrix based on neural networks.

Unsupervised deep embedding for clustering analysis proposes deep embedding clustering (DEC), whose goal is to transform embeddings to make them easier to separate. To make clusters separable, for each embedding $i$ calculates "belongs to" the$The\; probability\; of\; j$ speaker clusters$q_{i,j}$：$$q_{ij}=\frac{{\left(1+{\Vert z_i-m_j\Vert }^2/a\right)}^{-\frac{a+1}{a}}}{{\sum }_l{\left(1+{\Vert z_i-m_l\Vert }^2/a\right)}^{-\frac{a+1}{a}}},p_{ij}=\frac{q_{ij}^2/f_i}{{\sum }_l{q}_{il}^2/f_l}$$Among them, $z_i$is the bottleneck feature, $a$ is the degrees of freedom of the t-distribution. $m_i$for secondThe centroid of$i$$f_i=\sum q_{ij}$, using an autoencoder to estimate the bottleneck feature, iteratively based on the target distribution.

Improved DEC (IDEC) improves the accuracy of SD by adding a reconstruction loss between the output of the autoencoder and the input features to preserve the local structure of the data. IDEC's loss function consists of four parts:

• Clustering error L c L_c in the original DEC$L_c$
• Reconstruction error $L_r$
• uniform “speaker airtime” distribution loss $L_u$
• Loss LMSE L_{MSE} for measuring the distance of the bottleneck feature from the centroid$L_{MSE}$
  总这些的：$$L=\alpha L_c+\beta L_r+\gamma L_u+LOST{\_}_{MSE}$$The preceding coefficients represent weights.

Learning distance estimator

This section introduces a new approach using a trainable distance function

Based on relational RNN (RRNN), the model learns the relationship between a series of input features (far-near relationship), and the SD problem can be roughly regarded as this type of problem, because the final result of dairization depends on the relationship between the speech segment and the speaker's centroid. distance.

Problems that limit the accuracy of the SD system:

• The duration of the speaker embedding segment, which requires a trade-off between temporal resolution and robustness
• The speaker embedding extractor is not explicitly trained to provide the best representation for SD
• Distance metrics are often based on heuristics and/or rely on certain assumptions that do not necessarily hold
• Context information is ignored during audio processing.
  The above problems can be attributed to the distance metric function, and most of the problems can be solved with RRNN.

A paper proposes a method for learning the relationship between the speaker cluster centroid and embedding. SD is considered as a classification task for the segmented audio, that is, for each extracted embedding xj $x_j$Compared with the centroids of all speakers, a distance-based loss function is minimized to assign a label to this audio segment.

Post-processing based on deep learning

Medennikov proposed Targeted Speaker VAD (TS-VAD) to achieve accurate SD also under noisy conditions with many speakers overlapping.

TS-VAD assumes that the i-vector of each speaker is known, $\mathcal{E}=\left\{{\mathbf{e}}_k\in {\mathbb{R}}^f\mid k=1,\dots ,K\right\}$ , where$f$ is the dimension of i-vector,$K$ is the number of speakers, as shown in the figure:

The input of TS-VAD is MFCC feature $\mathbf{X}$+ i-vector $E$ , the output$k-$ dimensional vector sequence$\mathbf{O}\models \left({\mathbf{o}}_t\in {\mathbb{R}}^K\mid t=1,\dots ,T\right)$ ,except${\mathbf{o}}_t$The $k$ elements represent the time frame$t$ corresponds to$e_k$The probability of the speaker's speech activity (that is, if the speaker $e_k$at time $t$ speaks, then$0_t$The $k$ elements are 1, others are 0).

The flow of the whole model is:

1. Implement cluster-based diarization
2. Given the result of diarization above, estimate the i-vector for each speaker
3. Repeat the following:
   1. Execute TS-VAD according to i-vector
   2. The downside of updating the i-vector
      TS-VAD is that the maximum number of speakers the model can handle is limited by the dimensionality of the output vector.

A different approach was proposed by Horiguchi et al., employing the EEND model to update cluster-based SD results. Clustering-based SD methods can handle a large number of speakers, but cannot handle overlapping speech. EEND has the opposite property. The two can be complementary: first adopt traditional clustering methods, and then iteratively apply the EEND model to each pair of detected speakers to refine the temporal boundaries of overlapping regions.

SD joint optimization

Joint Segmentation and Clustering

The unbounded interleaved-state recurrent neural networks (UIS-RNN) model replaces segmentation and clustering methods with a single trainable model. Given an input embedding sequence $\mathbf{X}=\left({\mathbf{x}}_t\in {\mathbb{R}}^d\mid t=1,\dots ,T\right)$ , UIS-RNN produces diarization results$\mathbf{Y}=\left(y_t\in \mathbb{N}\mid t=1,\dots ,T\right)$ as the speaker index for each time frame. $\mathbf{X},The\; joint\; probability\; of\; Y$ can be decomposed by the chain rule:$$P(\mathbf{X},\mathbf{Y})=P\left({\mathbf{x}}_1,y_1\right)\prod _{t=2}^{T}P\left(x_t,y_t|{\mathbf{x}}_{1:t-1},y_{1:t-1}\right)$$
In order to model the distribution of speaker change, UIS-RNN introduces a hidden variableZ = ( zt ∈ { 0 , 1 } ∣ t = 2 , … , T ) \mathbf{Z}=\left(z_t \in\{0 ,1\} \mid t=2, \ldots, T\right)Z=(zt​∈{ 0,1}∣t=2,…,T ) , when the speaker is at time$t-1$ and time$When\; t$ is different,$z_t$for $1$ , otherwise$0$ Determine:$$P(\mathbf{X},\mathbf{Y},\mathbf{Z})=P\left({\mathbf{x}}_1,y_1\right)\prod _{t=2}^{T}P\left(x_t,y_t,z_t\mid {\mathbf{x}}_{1:t-1},y_{1:t-1},z_{1:t-1}\right)$$
式 $P\left(x_t,y_t,z_t|x_{1:t-1},y_{1:t-1},z_{1:t-1}\right)$ 可以进一步分解成三个部分：$$P\left({\mathbf{x}}_t,y_t,z_t\mid {\mathbf{x}}_{1:t-1},y_{1:t-1},z_{1:t-1}\right)=P\left(x_t|x_{1:t-1},y_{1:t}\right)P\left(y_t|z_t,y_{1:t-1}\right)P\left(z_t|z_{1:t-1}\right)$$ If,$P\left(x_t|x_{1:t-1},y_{1:t}\right)$ represents the sequence generation probability, modeled by a GRU-based RNN. $P\left(y_t\mid z_t,y_{1:t-1}\right)$ represents the speaker assignment probability,
modeled by a distance-dependent Chinese restaurant process. $P\left(z_t|z_{1:t-1}\right)$ represents the speaker change probability, modeled by a Bernoulli distribution. UIS-RNN can maximize$\log P(\mathbf{X},\mathbf{Y},Z)$ to train the optimal parameters. The inference process can be found by a beam search based method for a given$X$ in$\log P(X,Y)$ the largest$Y$ to achieve.

Joint segmentation, embedding extraction and re-segmentation

RPN-based methods can jointly perform segmentation, speaker embedding extraction, and re-segmentation.

The RPN network is shown on the left side of the above figure. The STFT feature is first converted into a feature map (that is, the channel dimension is increased), and then three neural networks are used to apply sliding windows (anchor) of different sizes on the time axis. For each anchor, three The neural network performs VAD, embedding extraction and region refinement respectively.

• VAD is used to estimate the probability of speech activity in anchor presence
• Embedding extracts speaker features in anchors
• region refinement is used to estimate the duration and center position of anchors.
  The inference process is shown on the right. RPN first obtains the anchors whose voice activity is higher than the threshold, and then calculates the speaker embedding of each anchor. The traditional clustering method is used to cluster the anchors. Finally, the anchors that overlap too much after region refinement are removed.

RPN-based SD systems have the advantage of handling overlapping speech that may have an arbitrary number of speakers.

Joint Speech Separation and Diarization

Kounades-Bastian proposed an NFM-based spatial covariance model to incorporate the voice activity detection model into voice separation, using the EM algorithm to estimate the separated voice and each speaker's voice activity from multi-channel overlapping voices. This method is completely based on statistical construction. mold.

Neumann proposed the online Recurrent Selective Attention Network (online RSAN) model, which combines speech separation, speaker counting, and SD based on a single model. Network input spectrum ${\mathbf{X}}_b\in {\mathbb{R}}^{T\times F}$ , a residual mask matrix$R_{b,i-1}\in R^{T\times F}$ and a speaker embedding${\mathbf{e}}_{b-1,i}\in R^d$ , of which$b,i,T,F$ are the index of the audio block, the speaker index, the length of the audio block, and the maximum frequency bin. Output speech mask${\mathbf{M}}_{b,i}\in R^{T\times F}$ and corresponding to the speaker$e_{b,i}$The updated speaker embedding of . For each audio block $b$ and speaker$i$ , the neural network proceeds iteratively:
Repeat (a) and (b) for$b=1,2,\dots$
(a) ${\mathbf{R}}_{b,0}=1$
(b) Repeat (i)-(iii) for $i=1,2,\dots$ until being stopped at (iii).
i. ${\mathbf{M}}_{b,i},{\mathbf{e}}_{b,i}=\mathrm{N}\mathrm{N}\left({\mathbf{X}}_b,R_{b,i-1},{\mathbf{e}}_{b-1,i}\right)$
(e $e_{b-1,i}$ is set to 0 if it was not calculated previously)
ii. ${\mathbf{R}}_{b,i}=\max \left({\mathbf{R}}_{b,i-1}-{\mathbf{M}}_{b,i},0\right)$
iii. If $\frac{1}{TF}{\sum }_{t,f}{R}_{b,i}(t,f)<$ threshold, stop iteration.

speaker $i$ in speech block$The\; separated\; speech\; of\; b$ can be passed through${\mathbf{M}}_{\mathbf{b},i}\odot {\mathbf{X}}_{\mathbf{b}}$obtained, where $\odot$ is element-wise multiplication. Speaker embedding${\mathbf{e}}_{b,i}$is used to track the blocks adjacent to the speaker.

Thanks to an iterative approach, neural networks can jointly perform speech separation and SD while handling a variable number of speakers.

Full end-to-end neural Diarization

The EEND framework implements the entire SD process based on a single network. The network structure is shown in the figure:

The input of EEND is long as $T$ 的声学特征 $X=\left({\mathbf{x}}_t\in {\mathbb{R}}^F\mid t=1,\dots ,T\right)$ , the speaker label sequence$\mathbf{Y}=\left({\mathbf{y}}_t\mid t=1,\dots ,T\right)$，其中 y t = [ y t , k ∈ { 0 , 1 } ∣ k = 1 , … , K ] \mathbf{y}_t=\left[y_{t, k} \in\{0,1\} \mid k=1, \ldots, K\right] yt​=[yt,k​∈{ 0,1}∣k=1,…,K] 当 $y_{t,k}=When\; 1$ , it means that in the time frame$t$ speaker$k$ has speech activity, and for different$k,k^{\prime }$，$y_{t,k}$和$y_{t,k^{\prime }}$can be both $1$ , indicating that two speakers spoke at the same time (that is, overlapping).

Suppose the output $y_{t,k}$Independently, the network maximizes the conditional distribution $\log P(Y)\mid \mathbf{X})\sim {\sum }_t{\sum }_k\log P\left(y_{t,k}|X\right)$ for training. Swapping speaker indices produces multiple label candidates, so a loss function is computed for all possible reference labels, and the reference label with the smallest loss is used for error backpropagation.

EEND initially used BLSTM, and later expanded the attention network.

The advantages of EEND are:

• Overlapping speech can be handled audibly
• Optimize directly by maximizing diarization accuracy
• Input labels can be retrained from real data.
  The constraints of EEND are:
• The model architecture limits the maximum number of speakers it can handle
• Unable to process online
• easy to overfit

Therefore, the extension of EEND has:

• Horiguchi proposed to extend EEND with an encoder-based attractor (EDA). The method applies an LSTM-based codec on the output of EEND to generate multiple attractors until their existence probability is less than a threshold. Then, each attractor is multiplied by the embedding generated by EEND to calculate the voice activity of each speaker.

Fujita et al. output speech activity by using the conditional speaker chain rule. In this method, the neural network is used to generate the posterior probability $P\left({\mathbf{y}}_k\mid {\mathbf{y}}_1,\dots ,y_{k-1},\mathbf{X}\right)$，其中 y k = ( y t , k ∈ { 0 , 1 } ∣ t = 1 , … , T ) y_k=\left(y_{t, k} \in\{0,1\} \mid t=1, \ldots, T\right) yk​=(yt,k​∈{ 0,1}∣t=1,…,T ) is the$Speech\; activity\; vectors\; for\; k$ speakers. Then use the chain rule to calculate the joint probability:$$P\left({\mathbf{y}}_1,\dots ,{\mathbf{y}}_K\mid \mathbf{X}\right)=\prod _{k=1}^{K}P\left(y_k|y_1,\dots ,y_{k-1},X\right)$$ In the inference phase, y_k is
estimated by repeated calculations until the final estimate of$y_k$close to $0$。

Kinoshita proposed a method combining EEND and speaker clustering. The neural network is trained to generate speaker embedding and speech activity probabilities, and speaker clustering constrained by the speech activity estimated by EEND is applied between different processing blocks. Align the estimated speakers.

There are also some ways to extend EEND to realize online processing.

SD in the context of ASR

It is generally believed that SD is a preprocessing step of ASR. This section discusses how to develop SD systems in the context of ASR.

early work

The lexical information from ASR can be made available to SD systems in the following ways:

1. RT03 Evaluation: Segmentation Using Word Boundary Information (First attempt to use the output of ASR to improve SD performance)
2. RT07 Evaluation: Use the results of ASR to improve VAD to reduce false alarms and improve the clustering performance of SD systems.
3. Silovsky et al: used ASR word alignment during segmentation, the word output by ASR was truncated by the segmentation result, because the segmentation result and the decoded word sequence were not aligned
4. Canseco-Rodriguez et al: create a dictionary (for broadcast news data) whose phrases provide the identity of who is speaking, who will speak, and who has spoken in a broadcast news scene

Although the early SD research did not make full use of lexical information to greatly improve DER, many studies integrated the output information of ASR to improve the output of SD.

Use lexical information from ASR

In a recent study, SD systems used DNNs to capture the output of ASR.

1. Flemotomos et al.: propose a method for using linguistic information for SD tasks:

The system employs neural text-based speaker change detection and text-based character recognition, using both linguistic and acoustic information to improve DER.
2. Park et al.: By using the Seq2Seq model to output speaker turn tokens, the lexical information of ASR is used for speaker segmentation, and then segmented based on speaker turn 3. Park et al.: Using an integrated
adjacency matrix to combine lexical information and speech segment clustering . The adjacency matrix is ​​the max of acoustic information and lexical information:

Joint ASR and SD based on deep learning

Method 1: Introduce speaker labels in end-to-end ASR:

1. Shafey et al. insert speaker role labels in the output of an RNN-T ASR system
2. Mao et al. propose to insert speaker identity labels into the output of an attention-based encoder-decoder ASR system

These two studies show that inserting speaker labels is a simple and promising approach to joint ASR and SD. But superior to speaker persona or speaker identity labels need to be determined and fixed during training. This approach is difficult to handle with an arbitrary number of speakers.

Method 2: MAP-based joint decoding.

Kanda et al. jointly decode ASR and SD, as shown in the figure:

Suppose the observation sequence X = { X 1 , … , XU } X=\left\{\mathbf{X}_1, \ldots, \mathbf{X}_U\ right\}X={ X1​,…,XU​} , among which$U$ represents the number of segments,$X_u$means $Acoustic\; features\; of\; segment\; u$ , words with temporal boundaries are assumed to beW = { W 1 , … , WU } W=\left\{\mathbf{W}_1, \ldots, \mathbf{W}_U\right\}W={ W1​,…,WU​} Among them,$W_u$uu _The speech recognition hypothesis corresponding to$u\; .$And${\mathbf{W}}_u=\left({\mathbf{W}}_{1,u},\dots ,W_{K,u}\right)$ contains the segmentuuAll speaker hypotheses of$u$$K$ represents the number of speakers), speaker embedding$\mathcal{E}=\left({\mathbf{e}}_1,\dots ,{\mathbf{e}}_{\mathbf{K}}\right)$ , among them$e_j\in {\mathbb{R}}^d$ is thekkthdd$for\; k$ speakers$d-$ dimensional vector. Then the multi-speaker ASR and SD joint decoding problem becomes to solve the most probable$\hat{W}$：$$\begin{array}{rl}\hat{W}& =\underset{W}{\mathrm{argmax}}P(\mathcal{W}\mid X)\\ & =\underset{W}{\mathrm{argmax}}\left\{\sum _{\mathcal{E}}P(\mathcal{W},E|X)\right\}\\ & \approx \underset{W}{\mathrm{argmax}}\left\{\underset{E}{\max }P(\mathcal{W},\mathcal{E}\mid X)\right\}\end{array}$$The last equation uses the Viterbi approximation. The equation can be further decomposed into two iterative problems: $$\begin{array}{c}{\hat{\mathcal{W}}}^{(i)}=\underset{W}{\mathrm{argmax}}P\left(\mathbf{W}|{\hat{E}}^{(i-1)},\mathcal{X}\right),\\ {\hat{E}}^{\left(i\right)}=\underset{E}{\mathrm{argmax}}P\left(\mathcal{E}\mid {\hat{\mathcal{W}}}^{(i)},\mathcal{X}\right),\end{array}$$Among them, $i$ represents the iteration index.

Approach 3: End-to-end Speaker Attribute Modeling (SA-ASR) for joint speaker counting, multi-speaker ASR and speaker recognition. Contrary to the previous two methods, the end-to-end SA-ASR model additionally inputs the speaker profile and identifies the speaker profile based on the attention mechanism:

Due to the attention mechanism trained on serial output, there is no limit to the number of speakers the model can handle. During inference, providing the relevant speaker profile, end-to-end SA-ASR can automatically transcribe speech and identify the speaker of each utterance.

Evaluation Series and Datasets

This section describes the Evaluation Series and datasets used for SD evaluation, and the datasets are summarized as follows:

• CALLHOME: The most extensive dataset used in SD papers
• AMI: A Suitable Dataset for Evaluating Speaker Classification Systems Integrated with ASR Modules
• ICSI: Conference Audio
• CHiME-5/6 Challenge and Dataset: ASR Challenge for Multi-Speaker Everyday Conversation
• VoxSRC Challenge and VoxConverse Corpus: Speaker Identification Challenge, Contains Overlapping Speeches
• LibriCSS: designed for studying speech separation, ASR and SD, including overlapping speech
• DIHARD Challenge and Dataset: Focusing on the Performance Gap of State-of-the-Art SD Systems, Challenging
• Rich Transcription Evaluation Series: RT Evaluation aims to conduct a more in-depth study of SD associated with ASR
• Other datasets: slightly

Application Scenario

1. Minutes, some of the technical challenges to overcome include
   1. Overlapped speech ASR, background noise, reverberation, etc.
   2. Modular framework capable of handling multi-modal and variable-channel-count scenarios without loss of performance
   3. When implementing online or streaming ASR, many audio preprocessing operations are required, resulting in low efficiency of the entire process
   4. Multi-device audio capture to improve meeting transcription quality
2. Dialogue interaction analysis and behavior modeling: the main thing is to extract the voice information of a specific speaker
3. Audio indexing: content-based audio indexing, understanding speech content with ASR transcripts, speaker summarization, and more
4. Conversational AI

SD's challenges and future

1. SD online (online) processing: focus on real-time
2. Domain Mismatch: A model trained on data in one domain does not perform well on data from another domain
3. Speaker Overlap: Overlap is unavoidable in multi-speaker scenarios, but traditional methods (especially cluster-based systems) only focus on non-overlapping regions.
4. Integration with ASR: Many applications require ASR results as well as SD results, and determining the optimal system architecture for both SD and ASR tasks remains an open problem
5. Audiovisual modeling: Visual information provides powerful cues for speaker recognition. For example, video captured by a fisheye camera is used to improve the accuracy of SD in a meeting transcription task