Big data algorithm focus

1 Sublinear Space Algorithms for Big Data

Scenario: storing a number N in binary requires log(N) space

Question: What should I do if N is very large and there are many such N?

Idea: reduce some accuracy and thus save more space.

Solution: use the approximate counting algorithm , the storage of each number only needs loglog ( N ) loglog(N)The space complexity of$l$$o$$g\; log$$($$N$$)$$will$$do$$.$

1.1 The counting problem of the flow model

problem definition

Define a data flow $<ai>,i\in [1,m],a_i\in [1,n]$ , frequency vector$<fi>，i\in [1,n],f_i\in [1,m]$。

The design space complexity is required to be $loglog\left(N\right)$ algorithm, record how many ai a_iappear in it$a_i$

morris algorithm

1. Initialize X to 0

2. loop: if $a_i$Appears once, with $1/(2^X)$with probability of increasing X by 1

3. returns $\hat{f}=(2^X-1)$

Using Chebyshev's inequality $P[|X-\mu |\ge ϵ]⩽p^2/{ϵ}^2$ proof

Expected $E[2^{X_N}-1]=N$

Variance $var[2^{X_N}-1]=\frac{1}{2}{N}^2-\frac{1}{2}N$

Let X = Y = 2 XN − 1 in Chebyshev's inequality X=Y=2^{X_N}−1$X=Y=2^{X_N}-1$。

Finally, the formula $P[|Y-N|\ge ϵ]⩽\frac{N^2-N}{2{ϵ}^2}$。

morris+algorithm

1. Run the morris algorithm k times
2. Record record results $(X_1,...,X_k)$
3. 返回$\gamma =\frac{1}{k}{\sum }_{i=1}^k(2^{X_i}-1)$

证明

$E[\gamma ]=N$

$D[\gamma ]=\frac{N^2-N}{2k}$。

$P[|\gamma -N|\ge ϵ]⩽\frac{N^2-N}{2k{ϵ}^2}$

morris++算法

1. 重复morris+算法$m=O(log(\frac{1}{d}))$ times
2. take mmMedian of$m\; results$

prove

$E[X_i]=0.9$
$m=0.9m$
$P[\sum X_i<0.5m]<d$

1.2 Number of non-repeating elements

problem definition

Define a data flow $<a_i>,i\in [1,m],a_i\in [1,n]$ , frequency vector$<f_i>,i\in [1,n],f_i\in [1,m]$。计算不等于0并且不重复的元素个数

FM算法

1. 随机选取一个哈希函数（$[0,1]$上的均匀分布）
   $h:[n]\mapsto [0,1]$

2. $z=1$

3. 当一个数字 i 出现的时候： z = m i n { z , h ( i ) } z=min\{z,h(i)\} z=min{ z,h(i)}

4. 返回$\frac{1}{z}-1$

证明

期望$E[z]=\frac{1}{d+1}$

$var\left[z\right]⩽\frac{2}{(d+1)(d+2)}<\frac{2}{(d+1)(d+1)}$

$P[|z-\frac{1}{d+1}|>ϵ\frac{1}{d+1}]<\frac{var[z}{{\frac{}{d+1}}^2}<\frac{2}{{ϵ}^2}$

FM+ algorithm

1. Run the FM algorithm q times in total
2. Randomly pick a hash function hj for each run
   $h_j:[n]\mapsto [0,1]$
3. Initialize $z_j=1$
4. Start counting: whenever i occurs, update $z_j=min(z_j,h_j(i))$
5. $Z=\frac{1}{q}{\sum }_{j=1}^q{z}_j$
6. returns $\frac{1}{Z}-1$

prove

$E[Z]=\frac{1}{d+1}$

$var\left[Z\right]⩽\frac{2}{(d+1)(d+2)}\frac{1}{q}<\frac{2}{(d+1)(d+1)}\frac{1}{q}$

$P[|X-d|>{ϵ}^{\prime }d]<\frac{2}{q}(\frac{2}{{ϵ}^{\prime }}+1{)}^2$

The calculation cost is reduced to $O(\frac{1}{{ϵ}^2}log\frac{1}{d})$

FM'+ algorithm

1. Randomly select a hash function
   $h:[n]\mapsto [0,1]$
2. $(z_1,z_2,...,z_k)=1$ means that the initial value of all z is set to 1
3. Maintain the smallest k hash values ​​currently seen
4. return $\frac{k}{z_k}$

prove

$P[|\frac{k}{z_k}-d|>ϵd]=P[\frac{k}{(1+ϵ)d}>z_k]+P[\frac{k}{(1-ϵ)d}<z_k]$

$P<\frac{2}{{ϵ}^{2}k}$

Preparatory knowledge of PracticalFM algorithm and BJKST algorithm

若我们无法存储实数，则采用PracticalFM算法 和 BJKST算法
$$\begin{gathered} 若\forall j_1,...,j_k\in [b],\forall j_1,...,j_k\in [a], \\ p[h(i_1)=j_1\wedge ...\wedge h(i_k)=j_k]=\frac{1}{b^k} \\ 则:一个从[a]映射到[b]的哈希函数是k-wise\_\_ \end{gathered}$$

$$\begin{gathered} zeros(h(j))=max(i:p\%2^i=0) \\ That\; is,\; the\; number\; of0s\; at\; the\; end\; after\; being\; expanded\; into\; binary, \\ For\; example,the\; binary\; value\; corresponding\; to8\; is1000,\; thenzeros\left(8\right)=3。 \end{gathered}$$

PracticalFM algorithm

1. 从$2-wiseindependent$ Randomly select hfrom the hash function family$h:[n]\mapsto \left[n\right]$

2. $z=0$

3. If $zeros(h(j))>z$

   $z=zeros\left(h\right(j))$

4. return $\hat{d}=2^{z+\frac{1}{2}}$

algorithm explanation

$1-\frac{2\sqrt{2}}{C}$The probability satisfies $d/C\le \hat{d}\le Cd$。

prove

$E[{AND}_r]=\frac{d}{2^r}$

$var\left[Y_r\right]\le \frac{d}{2^r}$。

The final correct probability should be greater than $1-\frac{2\sqrt{2}}{C}$。

BJKST algorithm

1. Randomly select $2-wiseindependenthash$ function$h:[n]\to \left[n\right]$

2. Randomly select $2-wiseindependentHash$ function$g:[n]\to [bϵ-4log2n]$

3. $z=0,B=\varnothing$

   若$zeros(h(j))>z$

   1. $B=B\cup (g(j),zeros(h(j)))$
   2. Then $|B|>\frac{c}{{ϵ}^2}$
      1. $z=z+1$
      2. remove (α,β) from B (α,β)$(a,b)$，wherein$b<z$

4. return $\hat{d}=|B|2^z$

algorithm explanation

To explain this algorithm clearly, only two points need to be clear.

• when a new element appears
  • If $zeros(h(j))>z$
    • Then insert the corresponding binary group into B;
    • If the number of elements in B exceeds a certain size:
      • The value of z increases by 1 accordingly
      • Just delete the elements whose second item is less than z.

prove

The algorithm can achieve at least 2/3 probability guaranteed (1+ϵ) approximation.

$E[{AND}_r]=\frac{d}{2^r}$

$var[Y_r]\le \frac{d}{2^r}$

Final $P[FAIL]<1/6$。

Coupled with the error probability caused by the previous algorithm assumptions, the final total error probability is within 1/3.

evaluate

The space complexity is $O(logn+\frac{1}{{ϵ}^2}(log\frac{1}{ϵ}+loglogn))$。

1.3 point query

problem definition

Define a data stream $<ai>，i\in [1,m],a_i\in [1,n]$ , frequency vector$<fi>,i\in [1,n],f_i\in [1,m]$ . Count the number of occurrences of all elements in the stream

knowledge preparation

• 范数：$l_p={\Vert x\Vert }_p={({\sum }_i|x_i{|}^p)}^{\frac{1}{p}}$
• $l_p$Point query (frequency estimation)
  for a given data stream $\sigma$和$a_i$输出$\widehat{f_i}$满足$\widehat{f_i}=f_i\pm ϵ\left||\text{f}\right|{|}_p$
  $\left||x\right|{|}_1\ge \left||x\right|{|}_2\ge ...\ge \left||x\right|{|}_{\infty }$，p越大，估计越准确
  $\Vert x{\Vert }_0$是不同元素的数目
  $\Vert x{\Vert }_1$是流的长度
  $\Vert x{\Vert }_{\infty }$是最大频度

Misra_Gries算法

Maintain a set A whose elements are $(i,\widehat{f_i})$

1. $A\leftarrow \varnothing$

2. For each element e in the data stream

   if e∈A，令$(e,\widehat{f_e})\to (e,\widehat{f_e}+1)$

   else if $|A|<\frac{1}{ϵ}$: Insert (e,1) into A

   else

   1. Decrement all counts in A by 1
   2. if $\widehat{f_j}=0$ : remove (j,0) from A

3. For query i, if $i\in A$ , returns$\widehat{f_i}$, otherwise return 0

prove

For any query i, return $\widehat{f_i}$ 满足$f_i-ϵm\le \widehat{f_i}\le f_i$。

证明

结合算法过程，总共有两种情况。

如果不发生减1的情况，那么$\widehat{f_i}=f_i$
如果发生了减1的情况，有$\widehat{f_i}<f_i$
假设发生了c次减1的情况，总数减少$\frac{c}{ϵ}\le m$，每个计数至多减少c，$\widehat{f_i}\ge f_i-c\ge f_i-ϵm$。

算法的空间代价是$O({ϵ}^{-1}logn)$

Metwally算法

Maintain a set A, the elements in the set are $(i,\widehat{f_i})$

1. A←∅
2. For each element e in the data stream
   1. if e∈A：令$(e,\widehat{f_i})\leftarrow (e,\widehat{f_i}+1)$
   2. else if $|A|<\frac{1}{ϵ}$: Insert (e,1) into A
   3. else Insert (e,MIN+1) into A, and delete one that satisfies $\widehat{f_e}=MIN$
3. query i if $i\in A$ , returns$\widehat{f_i}$, otherwise return MIN

proof goal

For any query i, return $f_i\le \widehat{f_i}\le f_i+ϵm$

证明

① 如果不发生删除的情况，那么$\widehat{f_i}=f_i$。

② 如果删除，计数一定不大于删除后的MIN，有$\widehat{f_i}\ge f_i$，A中元素总是m，$MIN\frac{1}{ϵ}\le m\Rightarrow MIN\le ϵm$，每个元素至多超出真实值MIN，$\widehat{f_i}\le f_i+ϵm$。

算法的空间代价是$O({ϵ}^{-1}logn)$

新的定义

Sketch

定义在数据流σ上的数据结构DS(σ)是一个Sketch

如果存在一个Space−Efficient的合并算法COMB使得$COMB(DS({\sigma }_1),DS({\sigma }_2))=DS({\sigma }_1\circ {\sigma }_2)$，其中∘是数据流的连接操作。

Linear Sketch

定义在[n]上的数据流σ上的sketching输出sk(σ)，如果sk(σ)取值为维度l=l(n)的向量，并且是f(σ)的线性函数，那么sk(σ)是一个Linear Sketch，l是这个sketch的维度。

Count-Min算法

1. $C[\mathrm{1...}t][\mathrm{1...}k]\leftarrow \text{0},k=\frac{2}{ϵ},t=\lceil log\frac{1}{d}\rceil$

2. Randomly select t 2−wise independent hash functions $h_i:[n]\to [k]$

3. For each update (j,c) that occurs, do the following

   for i=1 to t

   ​ $C[i][h_i(j)]=C\left[i\right]\left[h_i\right(j)]+c$

4. For a query for a, return $\widehat{f_a}={\min }_{1\le i\le t}C\left[i\right][h_i(a)]$

algorithm explanation

在算法开始时，构造一个 t 行 k 列 的空数组，可以认为每一行是独立的，算法在运行时同时记录了t个这样的数组。在每出现一个流数据的时候，对每一个数组进行一次更新，注意元素的第二个下标用的是数据的哈希值。

算法在运行的过程中可能产生冲突，也就是两个不同的流数据的哈希值可能相同，这个时候就会导致结果偏大，但是因为有相当于t次的重复计算，通过取最小值的方法来进行一些弥补

证明

该算法以1−δ概率给出$l_1$点查询问题的(1+ϵ)近似

评价

算法的空间代价为$O(\frac{1}{ϵ}log\frac{1}{\delta }(logn+logm))$

Count-Median算法

1. $C[\mathrm{1...}t][\mathrm{1...}k]\leftarrow \text{0},k=\frac{2}{ϵ},t=\lceil log\frac{1}{\delta }\rceil$

2. 随机选择t个2−wise独立哈希函数$h_i:[n]\to [k]$

3. 对每一个出现的更新(j,c)进行如下操作

   for i=1 to t

   ​ $C[i][h_i(j)]=C[i][h_i(j)]+c$

4. 针对对于a的查询，令$|C[x][h_x(a)]|=media{n}_{1\le i\le t}|C[i][h_i(a)]|$

5. 返回$\widehat{f_a}=|C[x\left]\right[h_x(a)]|$

algorithm explanation

The counting method of the Count−MinSketch algorithm is exactly the same, the difference lies in the acquisition of the return value, which returns the original value corresponding to the median of the absolute values ​​of all t array values.

Count Sketch Algorithm

1. $C[\mathrm{1...}k]\leftarrow 0,k=\frac{3}{{ϵ}^2}$

2. 随机选择1个2−wise独立哈希函数$h:[n]\to [k]$

3. 随机选择1个2−wise独立哈希函数$g:[n]\to -1,1$

4. 对于每一个更新(j,c)

   $C[h(j)]=C[h(j)]+c\ast g(j)$

5. 针对查询a，返回$\hat{f}=g(a)\ast C[h(j)]$

Count Sketch+算法

1. $C[\mathrm{1...}t][\mathrm{1...}k]\leftarrow \text{0},k=\frac{3}{{ϵ}^2},t=O(log\frac{1}{d})$

2. Randomly select a 2−wise independent hash function $h_i:[n]\to [k]$

3. Randomly select a 2−wise independent hash function gi : [ n ] → { − 1 , 1 } g_i:[n] \rightarrow \{-1,1\}gi​:[n]→{ −1,1}

   For each update (j,c)

   ​ For $i:1\to t$

   ​ $C[h_i(j)]=C\left[h_i\right(j)]+c\ast g_i(j)$

4. 返回$\hat{f}=median1\le i\le tgi(a)C[i][h_i(a)]$

算法解释

相当于是将Count Sketch算法运行了t次，最后取了中值。利用齐尔诺夫不等式可以解决，令Xi=1⇔第i次运行成功，成功概率是2/3，最后只要成功的个数超过一半即可。最终是通过t控制了δ。

1.4 频度矩估计

Basic AMS算法

1. $(m,r,a)\leftarrow (0,0,0)$
2. 对于每一个更新 j
   1. $m\leftarrow m+1$
   2. $b\leftarrow randombitwithP[\beta =1]=\frac{1}{m}$
   3. if β==1： a=j，r=0
   4. if j==a： $r\leftarrow r+1$
3. returns $X=m(r^k-(r-1{)}^k)$

Analysis of Algorithms

$E\left[X\right]=F_k$

$Yes\left[X\right]\_\le k{n}^{1-\frac{1}{k}}{F}_k^2$

利用切比雪夫不等式对结果进行检验$P[|X-E[x]|>ϵE[X]]<\frac{k{n}^{1-\frac{1}{k}}}{{ϵ}^2}$

评价

存储 m 和 r 需要 log n 位
存储 a 需要 log m 位
算法的方差太大，需要进一步的改进，但是其相应的存储代价为$O(logm+logn)$

Final AMS 算法

1. 利用Median‑of‑Mean技术调用Basic AMS算法;
2. 计算$t=clog\frac{1}{d}$averages, each of which is $r=\frac{3}{{ϵ}^2}{n}^{1-\frac{1}{k}}$average of calls
3. Returns the median of t values

prove

记 { X i j } i ∈ [ t ] , j ∈ [ r ] \{X_{ij}\}_{i \in [t],j \in [r]} { Xij​}i∈[t],j∈[r]​is a set of random variables that are independent of the same part as X.

$Y_i=\frac{1}{r}{\sum }_{j=1}^r{X}_{ij}$

$Z={\sum }_{i=1}^t{Y}_i$

计算$E[Y_i]=E[X],Var[Y_i]=\frac{Var[X]}{k}$

根据切比雪夫不等式有$P[|Y_i-E[Y_i]|\ge ϵE[Y_i]]\le \frac{Var[X]}{k{ϵ}^{2}E[X{]}^2}$

取值$r=\frac{3Var[X]}{k{ϵ}^{2}E[X{]}^2}$，将期望和方差带入可以计算得到算法中给出的结果

现在$P[|Y_i-E[Y_i]|\ge ϵE[Y_i]]\le \frac{1}{3}$

最后利用Median技术进行处理即可。

评价

算法的空间代价为$O(\frac{1}{{ϵ}^2}log\frac{1}{\delta }k{n}^{1-\frac{1}{k}}(logm+logn))$ , when k≥2, the cost is too high.

Basic F2 AMS algorithm

1. Randomly select 4−wise independent hash functions $h:[n]\to -1,1$

2. x←0

3. For each update (j,c)

   $x\leftarrow x+c\ast h(j)$

4. returns $x^2$

analyze

This algorithm itself does not have a better effect, but after optimization of the median−of−mean technique, a (ϵ,δ) algorithm can be obtained. So in the proof here, just calculate the expectation and variance of the result.

prove

记$Y_j=h(j),j\in [1,n],X={\sum }_{j=1}^n{f}_j{Y}_j$

$E[X^2]={\sum }_{i=1}^n{f}_i{Y}_i\times {\sum }_{j=1}^n{f}_j{Y}_j$

$Var[X^2]=E[X^4]-E[X^2{]}^2$

评价

算法的空间代价是O(logm+logn)，下面先使用mean将犯错概率限制在1/3，再使用median技术对结果进行优化。

1.5 固定大小采样

水库抽样算法

1. m←0
2. 使用数据流的前s个元素对抽样数组进行初始化
   $A[1,...,s],m\leftarrow s$
3. 对于每一个更新x
   1. x以$\frac{s}{m+1}$概率随机替换A中的一个元素
   2. m++

证明

假定已经流过的数据量为n，采样池大小为s

考虑最普通的情况，第j个元素进了采样池，之后再也没有被选出去，那么在第n个元素流过之后，这个元素还在采样池中的概率是s/n

计算方法：被选进去的概率是s/j，为保证不被选出去，两种情况：新的元素没有选进来，新的元素选进来了但是该元素没有被替换掉。这两种情况对应着$(1-\frac{s}{j+1})+\frac{s}{j+1}\ast \frac{s-1}{s}=\frac{j}{j+1}$。依次类推最终可以计算得到结果s/n。

1.6 Bloom Filter

给定一个数据集U，从中抽取一个子集S，给定一个数q∈U，判定q∈S是否成立。

近似哈希的方法

1. 令H是一族通用哈希函数：$[U]\to [m]，m=\frac{n}{\delta }$
2. 随机选择 h∈H，并维护数组A[m]，S的大小是n
3. 对每一个 i∈S，$A[h(i)]=1$
4. Given a query q, return yes if and only if $A\left[h\right(i)]=1$

prove

If q∈S, yes is returned. If q∉S, then no should be returned, but yes is returned with a certain probability. This is the wrong situation. The element is not in S, but its hash value is the same as An element of has the same hash value. ${\sum }_{j\in S}P[h(q)=h(j)]\le \frac{n}{m}=\delta$ , which computes values ​​other than m and solves the approximation problem.

Bloom Filter method

1. Let H be a family of independent ideal hash functions: [U]→[m]

2. Randomly select $h_1,...,h_d\in H$ , and maintain the array A[m]

3. For every i∈S

   ​ For each j∈[1,d]

   ​ $A[h_j(i)]=1$

4. Given a query q, return yes if and only if $\forall j\in [d],A[h_j(q)]=1$

prove

The probability of failure is $P\le (\frac{n}{m}{)}^d=\delta$ , so the final cost is$m=O\left(nlog\frac{1}{d}\right)$

2 Sublinear Time Algorithms for Big Data

2.1 Vertex Cover 1, the average degree algorithm for computing graphs

definition

Known : $G=(V,E)$
求：平均度$\bar{d}=\frac{{\sum }_{u\in V}d(u)}{n}$
假设：G是简单图，没有平行边和自环

分析

将具有相似或者相同度数的节点分组，然后估算每个分组的平均度数。

首先将所有的点进行分桶，分成t个桶，第i个桶里的点集合为 B i = { v ∣ ( 1 + β ) ( i − 1 ) < d ( v ) < ( 1 + β ) i } , 0 < i ≤ t − 1 B_i=\{v|(1+\beta)^{(i-1)} < d(v) < (1+\beta)^{i}\},0<i\leq t-1 Bi​={ v∣(1+β)(i−1)<d(v)<(1+β)i},0<i≤t−1，其中β是超参数。

于是$B_i$中的点的总度数有上下界如公式所示：$(1+\beta {)}^{(i-1)}|B_i|<d(B_i)<(1+\beta {)}^i|B_i|$

进一步的G的总度数可以表示为：${\sum }_{i=0}^{t-1}(1+\beta {)}^{(i-1)}|B_i|<{\sum }_{u\in V}d(u)<{\sum }_{i=0}^{t-1}(1+b{)}^i|B_i|$

于是我们可以得到：$\frac{{\sum }_{i=0}^{t-1}(1+b{)}^{(i-1)}|B_i|}{n}<\bar{d}<\frac{{\sum }_{i=0}^{t-1}(1+b{)}^i|B_i|}{n}$

So the problem is transformed into $\frac{B_i}{n}$estimate

algorithm

1. Take the sample set S from V
2. $S_i\leftarrow S\cap B_i$
3. $r_i\leftarrow \frac{S_i}{S}$
4. 返回 $\hat{\bar{d}}={\sum }_{i=0}^{t-1}{r}_i(1+b{)}^i$

The idea of ​​the algorithm is actually very simple, the $\frac{B_i}{n}$It is understood as a kind of probability, which is to randomly select a point, which belongs to the probability of Bi, so it is very simple to understand the algorithm.

evaluate

The idea and calculation of the algorithm are very simple, but a very clever transformation has been carried out, but this algorithm is still problematic

2.2 Calculation graph average degree algorithm 2

improve algorithm

ρ i \rho_i for the smaller bucket$r_i$, assuming a $d$A lower orderα \alpha$\stackrel{of\; ˉ}{}$$a$ .

1. Draw samples S from V
2. $S_i\leftarrow S\cap B_i$
3. f o r   i ∈ { 0 , … , t − 1 }   d o \boldsymbol{for}\ i \in \{0,\dots,t-1\}\ \boldsymbol{do} for i∈{ 0,…,t−1} do
   1. $\mathbf{if}|S_i|\ge {\theta }_{\rho }\mathbf{then}$
   2. ${\rho }_i\leftarrow \frac{|S_i|}{|S|}$
   3. $\mathbf{else}$
      1. ${\rho }_i\leftarrow 0$
4. $\mathbf{return}\hat{\bar{d}}={\sum }_{i=0}^{t-1}{\rho }_i(1+\beta {)}^i$

我们将算法的结果调整为以2/3的概率有$(0.5-ϵ)\bar{d}<\hat{\bar{d}}<(1+ϵ)\bar{d}$。

这里给出一组参数，使得能够满足上述结果：

1. $\beta =\frac{ϵ}{4}$
2. $|S|=\Theta (\sqrt{\frac{n}{\alpha }}\cdot poly(logn,1/ϵ))$
3. $t=\lceil lo{g}_{(1+\beta )}n\rceil +1$
4. ${\theta }_{\rho }=\frac{1}{t}\sqrt{\frac{3}{8}\cdot \frac{ϵ}{n}}|S|$

2.3 Calculation graph average degree algorithm 3

Thoughts on Algorithm Improvement

We attribute the errors of the algorithm to the edges, let us see which edges lead to such errors. Divide the nodes into two parts U, V/U, where U is a node with a small degree, V/U is a node with a large degree, and E(U,V/U) represents the set of edges connecting the two sets. Therefore, we assert that the error occurs because we only calculate the edge in E(U,V/U) once, and it is easy to understand this point if we recall the previous example. So we only need to find the proportion of this part of the side at each sampling time.

improve algorithm

Use $E\left[D_i\right]$的$1+ϵ$估计，可以得到$D_i{r}_i(1+b{)}^{iorT}$ in$\frac{T_i}{n}of(1+ϵ)(1+\beta )$ estimate. So, the modified algorithm is as follows:

1. 从V中抽取样本S，$|S|=\tilde{O}(\frac{L}{\rho {ϵ}^2}),L=poly(\frac{logn}{ϵ}),\rho =\frac{1}{t}\sqrt{\frac{ϵ}{4}\cdot \frac{\alpha }{n}}$
2. $S_i\leftarrow S\cap B_i$
3. f o r   i ∈ { 0 , … , t − 1 }   d o \boldsymbol{for}\ i \in \{0,\dots,t-1\}\ \boldsymbol{do} for i∈{ 0,…,t−1} do
   1. $\mathbf{if}|S_i|\ge {\theta }_{\rho }\mathbf{then}$
      1. ${\rho }_i\leftarrow \frac{|S_i|}{|S|}$
      2. $estimate{\Delta }_i$
   2. $\mathbf{else}$
      1. ${\rho }_i\leftarrow 0$
4. $\mathbf{return}\hat{\bar{d}}={\sum }_{i=0}^{t-1}(1+{\Delta }_i){\rho }_i(1+\beta {)}^i$

2.4 计算图的平均度算法四

Alg III

Use $E\left[D_i\right]$的$1+ϵ$估计，可以得到$D_i{r}_i(1+b{)}^{iorT}$ in$\frac{T_i}{n}of(1+ϵ)(1+\beta )$ estimate. So, the modified algorithm is as follows:

1. Let V be an independent variable S, $|S|=\tilde{O}(\frac{L}{\rho {ϵ}^2}),L=poly(\frac{logn}{ϵ}),\rho =\frac{1}{t}\sqrt{\frac{ϵ}{4}\cdot \frac{\alpha }{n}}$
2. $S_i\leftarrow S\cap B_i$
3. f o r   i ∈ { 0 , … , t − 1 }   d o \boldsymbol{for}\ i \in \{0,\dots,t-1\}\ \boldsymbol{do} for i∈{ 0,…,t−1} do
   1. $\mathbf{if}|S_i|\ge {\theta }_{\rho }\mathbf{then}$
      1. ${\rho }_i\leftarrow \frac{|S_i|}{|S|}$
      2. $estimate{\Delta }_i$
   2. $\mathbf{else}$
      1. ${\rho }_i\leftarrow 0$
4. $\mathbf{return}\hat{\bar{d}}={\sum }_{i=0}^{t-1}(1+{\Delta }_i){\rho }_i(1+\beta {)}^i$

Alg IV

1. $\alpha \leftarrow n$
2. $\hat{\bar{d}}<-\infty$
3. $\mathbf{while}\hat{\bar{d}}<\alpha \mathbf{do}$
   1. $\alpha \leftarrow \alpha /2$
   2. $\mathbf{if}\alpha <\frac{1}{n}\mathbf{then}$
      1. $\mathbf{return}0;$
   3. $\hat{\bar{d}}\leftarrow AlgII{I}_{\sim \alpha }$
4. $\mathbf{return}\hat{\bar{d}}$

算法相关指标

近似比：$(1+ϵ)$

运行时间：$\tilde{O}(\sqrt{n})\cdot poly({ϵ}^{-1}logn)\sqrt{n/\bar{d}}$

2.5 Approximate minimum spanning tree

definition

known：$G=(V,E),ϵ,d=deg\left(G\right)$ , the weight of edge (u,v) iswuv ∈ { 1 , 2 , … , w } ∪ { ∞ } w_{uv}∈\{1,2,…,w\}∪\{ ∞\}wuv​∈{ 1,2,…,w}∪{ ∞}

Find : $\hat{M}$ satisfy$(1-ϵ)M\le \hat{M}\le (1+ϵ)M$，令M为$mi{n}_{TspansG}W(T)$

分析

定义：

G的子图$G^{(i)}=(V,E^{(i)})$
$E(i)=\{(u,v)|w_{uv}\le i\}$
连通分量的个数为C(i)

Theorem : If M is the sum of the number of connected components of all such subgraphs, $M=n-w+{\sum }_{i=1}^{w-1}{C}^{(i)}$

Proof : Let a minimum spanning tree of graph G be $MST=(V,E^{\prime })$，MST的子图$MS{T}^{\left(i\right)}=(V^{\prime },E^{\prime (i)}),V^{\prime }=V,E^{\prime (i)}=\{(u,v)|(u,v)\in E^{(i)}\&(u,v)\in E^{\prime }\}$

ảọa $a_i$is the number of edges with weight i in MST

${\sum }_{i>1}{a}_i$Indicates the number of edges with edge weights greater than 1 in the MST. Adding 1 to this value should be the MST obtained after removing these edges from the MST in the MST ( l $MS{T}^{}$The number of connected components in${}^{(}$${}^l$${}^{)}$

$MS{T}^{The\; number\; of\; connected\; components\; of\; (l)}$ should be consistent with G(i), so we get the following formula:${\sum }_{i>1}{a}_i=C^{(l)}-1$。

于是我们就可以用如下的方法计算M了：
$$\begin{array}{rl}M& =\sum _{i=1}^{w}i\cdot {\alpha }_i=\sum _{i=1}^w{\alpha }_i+\sum _{i=2}^w{\alpha }_i+\cdots +\sum _{i=w}^w{\alpha }_i\\ & =C^{(0)}-1+C^{(1)}-1+\cdots +C^{(w-1)}-1\\ & =n-1+C^{(1)}-1+\cdots +C^{(w-1)}-1\\ & =n-w+\sum _{i=1}^{w-1}{C}^{(i)}\end{array}$$

time complexity

The time complexity of running w times to solve the number of connected components: $w\cdot O(d/{ϵ}^3)=O(dw{\_}^4/{ϵ}^{\prime 3})$。

2.6 Find the diameter of the point set

Definition :

It is known that there are m points, and the distance between points is represented by an adjacency matrix, then Dij represents the distance from point i to point j, D is a symmetric matrix, and satisfies the triangle inequality D ij ≤ D ik + D kj $D_{ij}\le D_{}+D_{kj}$。

Find out: the point pair (i, j) makes Dij the largest, then Dij is the diameter of the set of m points.

Solving algorithm: The Indyk's Algorithm

1. Optional $k\in [1,m]$
2. select $l$，使得$\forall i,D_{}\le D_{kl}$
3. returns $(k,l),D_{kl}$

Approximation Ratio Analysis of Algorithm Analysis

The optimal solution is denoted as opt, which is the distance between point i and point j

Then there are $\frac{opt}{2}\le D_{kl}\le opt$

Approximate Ratio of Algorithm Analysis to Prove Inequality

Regarding the easy proof of the right side of the inequality, the proof of the left side is as follows:
$$\begin{array}{rl}opt=& D_{ij}\\ \le & D_{}+D_{kj}\\ \le & D_{kl}+D_{kl}\\ \le & 2D_{kl}\end{array}$$

Algorithm evaluation

The time complexity of the algorithm is $O(m)=O(\sqrt{n})$。

2.7 Find the number of connected components

definition

known：$G=(V,E),ϵ,d=deg\left(G\right)$ , the graph G is represented by an adjacency list, where d represents the degree of the node with the largest degree among all nodes,$|V|=n,|E|=m\le d\cdot n$

求出：一个y，make $C-ϵ\cdot n\le y\le C+ϵ\cdot n$ , where C is the standard solution obtained by using the linear algorithm

problem analysis

Record the number of nodes in the connected component to which vertex v belongs as $n_v$, A∈V is a point set of connected components, then there is the following equation relationship: ${\sum }_{u\in A}\frac{1}{n_u}={\sum }_{u\in A}\frac{1}{|A|}=1$ , this is easy to prove, because the nv n_vof each point in the same connected component$n_v$It is the same, both are the reciprocal of the number of midpoints in the components. In this way, the final result C can be expressed as ${\sum }_{u\in V}\frac{1}{n_u}$. So further, the estimation of C can be transformed into $\frac{1}{n_u}$estimate.

problem-solving thinking

$n_u$It is very large, and it is difficult to calculate accurately, but at this time $\frac{1}{n_u}$Very small, can be replaced by a small constant $\frac{1}{n_u}$(0 $\frac{}{2}$), if we set $\widehat{n_u}=my\{n_u,\frac{2}{ϵ}\}$，则$\hat{C}={\sum }_{u\in V}\frac{1}{\widehat{n_u}}$, using $\hat{C}$ to estimate C can get very good results.

Algorithm - Calculate $\widehat{n_u}$

The idea is very simple, it is a small search, if the number of searched points is less than $\frac{2}{ϵ}$Just continue to search, otherwise return directly to $\frac{2}{ϵ}$。

The time complexity is $O(d\cdot \frac{1}{ϵ})$ , the larger d is, the longer it takes,$\frac{1}{ϵ}$The bigger it is, the longer it takes.

Algorithm - Calculate $n_u$

Randomly select r = b / ϵ 2 r = b/{\epsilon}^2 from the set of nodes$r=b/{ϵ}^2$ nodes constitute node U, apply the previous algorithm to each node, so the final$\hat{C}=\frac{n}{r}{\sum }_{u\in U}\frac{1}{\widehat{n_u}}$, the time complexity is $O(d/{ϵ}^3)$

3 Parallel Computing Algorithms

3.1 Basic Questions (1)

MapReduce

There are several points for calculations using the MR model:

• The calculation is carried out in rounds, and the input and output data of each round are in the form of <key, value>
• Each round is divided into: Map, Shuffle, Reduce
• Map: Each data is mapprocessed by a function, and a new data set is output, that is, <key, value> is changed
• Shuffle: Transparent to programmers, all the data output in the Map are grouped key, and the same keydata is assigned to the samereducer
• Reduce: Input <k,v1,v2,…>, output a new data set
• For such a computing framework, our design goals are: fewer rounds, less memory, and greater parallelism

problem instance

Let's look at some specific problem examples. These examples can be parallelized. Of course, there are more than one logic to implement, and there are many possibilities. Here is just one method.

Build an inverted index

Definition : Given a set of documents, count which documents each word appears in

• Map函数：$<docID,content>\to <word,docID>$ , split the content when the map function is processing, and convert the separated words into words plus the corresponding docID output.
• Reduce函数：$<word,docID>\to <word,listofdocID>$ , after the map function ends, shuffle will automatically output the key-value pairs with the same key to a machine, and we can directly organize the data in this batch together.

word count

Definition : Given a set of documents, count the number of occurrences of each word

• Map function: <docID, content> → <word, 1>
• Reduce function: <word,1>→<word,count>

retrieve

Definition : Given a line number and corresponding document content, count the position where the specified word appears (*line number)

• Map function: slightly
• Reduce function: slightly

3.2 Basic Questions (2)

Introduction

Problem Example: Matrix Multiplication

It is simple matrix multiplication. We know that the time complexity of ordinary matrix multiplication is O(m⋅n⋅d), for a matrix multiplication of m⋅d and d⋅n. Through the parallel computing framework, we greatly speed up the execution of this process. The specific two methods are as follows

Matrix multiplication 1

Definition : matrix A and matrix B, (A,i,j) points to the element of row i and column j of matrix A, but it is not the real element, but represents an index, and aij represents this element.

• Map：
  • ((A,i,j),aij)→(j,(A,i,aij))
  • ((B,j,k),bjk)→(j,(B,k,bjk))
• Reduce：(j,(A,i,aij)),(j,(B,k,bjk))→((i,k),aij∗bjk)
• Map：nothing（identity）
• Reduce：((i,k),(v1,v2,…))→((i,k),∑vi)

Analysis of ideas : Each element on the target matrix is ​​obtained by accumulating d intermediate elements (aij∗bjk). We first find out all such elements and then add them up. This is true for each element on the target matrix.

Matrix multiplication 2

• Map function:

  • ((A,i,j),aij)→((i,x),(A,j,aij)) for all x∈[1,n]
  • ((B,j,k),bjk)→((y,k),(B,j,bjk)) for all y∈[1,m]

• Reduce function: ((i,k),(A,j,aij))∧((i,k),(B,j,bjk))→((i,k),∑aij∗bjk)

Analysis of ideas : Compared with the first method, method 2 does not find the intermediate elements first, but puts together the 2d basic elements used to solve each target matrix, and then performs the dot product operation.

3.3 Sorting algorithm

Introduction to the problem

algorithm

Using p processors, input $<i,A[i]>$

• Map：$<i,A\left[i\right]>\to <j,((i,A[i]),y)>$

  1. 输出$<i\%p,((i,A[i]),0)>$

  2. 以概率T/n为所有$j\in [0,p-1]$输出$<j,((i,A[i]),1)>$

     否则输出$<j,((i,A[i]),0)>$

• Reduce：

  • 将y=1的数据收集为S并排序
  • 构造$(s_1,s_2,...,s_{p-1})$，$s_k$For the kth in S $k\lceil \frac{|S|}{p}\rceil$
  • Collect data for y=0 as D
  • (i,x)∈D satisfies $s_k<x\le s_{k+1}$, output <k,(i,x)>

• Map：nothing（identity）

• Reduce：$ <j, ((i, A[i]), . . . )>$

  • general owned $(i,A[i])$ Sort and output according to $A[i]$

Analysis of ideas : The overall idea is to divide the entire data into different small segments, and the data between segments has a strict size relationship, so that the data inside each segment is some very close data, so we The data set inside each segment can be sorted with the help of some efficient sorting methods. Then one of the keys to solving the problem is whether we divide the data well or not. We can use theory to prove that there is a high probability that the division effect can be guaranteed. **But it is not absolute. It is possible that there is basically no data on a certain computing node, and excessive data appears on another computing node. Of course, this is an extremely low probability event.

3.4 Calculate the minimum spanning tree (spanning tree)

Introduction to the problem

Known graph G=(V, E), where V, E are the point set and edge set of the graph respectively. A subgraph $T^{\ast }=(V,E^{\prime })$is called a minimum spanning tree if and only if$T^{\ast }$是连通的并且 ∑ ( u , v ) ∈ T ∗ w ( u , v ) = min ⁡ T { ∑ ( u , v ) ∈ T w ( u , v ) } \sum_{(u,v)\in T^*}{w(u,v)} = \min_{T}\{\sum_{(u,v)\in T}{w(u,v)}\} ∑(u,v)∈T∗​w(u,v)=minT​{ ∑(u,v)∈T​w(u,v)}。

In the small data range, the Kruskal or Prim algorithm is usually used to solve it. When the scale of the graph becomes larger, due to the limitation of the complexity of these two algorithms, it will not be able to solve it within a limited time. At this time, MapReduce can be used to speed up the calculation.

Algorithm main idea

Using the graph partition algorithm, the graph G is divided into k subgraphs, and the minimum spanning tree is calculated in each subgraph, as follows.

1. Divide the node into k parts, for each $(i,j)\in [k{]}^2$，令$G_{ij}=(V_i\cup V_j,E_{ij})$ is node$V_i\cup V_j$The derived subgraph on
2. Solve M ij = MSF ( G ij ) M_{ij}=MSF(G_{ij}) separately on each Gij$M_{ij}=MSF(G_{ij})$
3. 令$H={\cup }_{i,j}{M}_{ij}$, calculate $M=MST\left(H\right)$

Note: MSF refers to the minimum spanning forest. Give an example to illustrate this concept. There are n−1 edges in a minimum spanning tree, which can be understood as a minimum spanning forest with only one tree. Similarly, there are also minimum spanning forests with two trees and three trees in the same graph, as long as the forest is min ⁡ { ∑ ( u , v ) ∈ Forestw ( u , v ) } \min\{\ sum_{(u,v)\in Forest}w(u,v)\}min{ ∑(u,v)∈Forest​w(u,v )} will do.

In essence, the essence of the algorithm is to first calculate the spanning tree locally, then use the remaining edges connecting these spanning trees to form a new graph, and find the minimum spanning tree of this new graph as the total result.

MR algorithm

• Map：input:<(u,v),NULL>
  • transformation<(h(u),h(v));(u,v)>
  • For the above conversion data, if h(u)=h(v), then for all j∈[1,k], output <(h(u),j);(u,v)>
• Reduce：input:<(i,j);Eij>
  • 令Me=MSF(Thou)
  • Output <NULL;(u,v)> for each edge e=(u,v) in Mij
• Map：nothing（identity）
• Reduce：M=MST(H)

Among them, h is a hash function, which can be realized by a random algorithm with uniform values ​​(ps: use a random algorithm to generate a hash table instead of randomly taking values ​​every time it runs).

Of course, the load balancing problem of each computing node should also be considered when dividing.

4 External memory model algorithm

4.1 External storage model

External memory model

So far, according to the storage model, the algorithm models we have learned should be divided into two types: one is the RAM model, which is the design model of our commonly used algorithms, and the other is the I/O model, the memory is smaller than the amount of data , the external memory is unlimited.

There are some differences between external memory access and memory access:

• RAM is faster than external storage
• Continuous access to external memory is less costly than random access, that is to say: access in units of blocks

Fundamental Problems Based on the External Storage Model

In the I/O model, the size of the memory is M, the page size is B, the size of the external memory is unlimited, and the page size is B.

Continuous reading of N data on the external storage requires O(N/B) I/O times

How to calculate matrix multiplication?

Input two matrices X and Y of size N×N

1. Divide the matrix into sizes $\sqrt{M}/2\times \sqrt{M}/2$ blocks
2. Considering each block in the X×Y matrix, there is obviously a total of $O((\frac{N}{\sqrt{M}}{)}^2)$blocks need to output
3. Each block needs to scan $\frac{N}{\sqrt{M}}$to the input block
4. Each memory calculation requires O(M/B) I/O times
5. Total $O((\frac{N}{\sqrt{M}}{)}^3\cdot M/B)$ Next I/O

linked list

Perform three operations: insert(x,p), remove§, traverse(p,k)

The time complexity of each operation under the memory model: updateO(1), traverseO(k)

Under the external memory model, the consecutive elements in a linked list are placed in a block of size B. Also, let each block be at least B/2 in size:

• remove: If it is less than B/2 after deletion, it will be merged with the adjacent block, if it is greater than B after the merger, it will be divided equally
• insert: If it is greater than B after insertion, it will be divided evenly
• traverse：O(2k/B)

search structure

Perform three operations: insert(x), remove(x), query(x)

$(a,b)-tree:2\le a\le (b+1)/2$

Similar to binary search tree $\Rightarrow (p_0,k_1,...,k_c,p_c)$

The root node has 0 or ≥ 2 children; except the root, the number of children of each non-leaf node ∈ [a,b]:

• remove: Find the corresponding leaf node. If it is smaller than a after deletion, it will be merged with the adjacent block. If it is greater than b after the merger, it will be divided evenly, and then recursively delete the node or adjust the key value in the upper layer.
• insert: Find the corresponding leaf node, if it is greater than b after insertion, divide it evenly, and recursively insert to the next level
• $query:O(lo{g}_a(N/a))$

4.2 External memory sorting

External memory sorting problem

When considering the external memory sorting algorithm, it should be closely combined with the external memory model.

algorithm

• Given N data, divide it into groups of size O(M)
• Each set of data can be sorted in memory
• Reading each set of data from external storage requires O(M/B) I/O times
• Perform the above operations on all groups, so each group contains sorted data
• Perform a multi-way merge sort on these sorted groups
• O(M/B) groups can be merged each time

process explanation

The first thing to understand is that when transferring data from external storage to internal memory, only the amount of data of B can be transferred at a time. Therefore, if you want to read the memory slowly at one time, the corresponding number of I/Os is O(M/B). In addition, when performing multi-way merge sort, how many groups can be merged at most. A page is read from each group, and then sorted, so it has nothing to do with the size of each group, only the size of the memory, so it is O(M/B).

icon

evaluate

The time complexity is divided into two parts, one is intra-group sorting, and the other is inter-group merge sorting.

1. For intra-group sorting, you only need to read the data of each group into memory, and the corresponding time complexity of this part is $O(N/B)$。
2. For merge sorting, the corresponding time cost should be the sum of the overhead of each merge, and each merge needs to import all the data into memory once, and this time cost is O ( N / B ) O(N $O(N/B)$ . The number of times to merge can be expressed as$O(lo{g}_{M/B}\frac{N}{B})$ . To sum up: the total time overhead is:$O(N/B\cdot lo{g}_{M/B}\frac{N}{B})$

4.3 List Ranking

List Ranking

definition

[List Ranking Problem] : Given an adjacency linked list L of size N, L is stored in an array (continuous external storage space), and the rank (serial number in the linked list) of each node is calculated.

[General List Ranking Problem] : Given an adjacency linked list L of size N, L is stored in an array (continuous external storage space), and each node v in L stores weight wv $w_v$, calculate the rank of each node v (the sum of weights from the head node to v)

analyze

• If the subsequence on the merge list is a node (that is, the page is merged internally, and a page is regarded as a node), the weight sum is used as the weight of the node, and the rank value of the preceding and following nodes is not affected.
• If the size of the linked list is at most M, then this problem can be solved by using O(M/B) I/O times, that is, all the data is read into the memory, and the memory algorithm is used to solve this problem.

Let's take a look at the intuitive understanding of this problem through a picture, as shown in the figure below, the relevant data is: N=10, B=2, M=4. In the worst case, the access cost is O(N) I/O times.

algorithm

Input the external memory linked list L of size N

1. Find an independent set of vertices X in L
2. "Skip" the nodes in X to build a new, smaller external memory linked list L'
3. Solve L' recursively
4. "Backfill" the nodes in X, and construct the rank of L according to the rank of L'

Step 1.1.4 can be done in $OR\left(sort\right)\_=O(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B})$ time I/O solution, construct the following recursive equation:

$T(N)=T((1-a\left)N\right)+O\left(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B}\right)$ , solving the equation:$T(N)=O(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B})$

step~1

According to the ID order of the nodes, it is divided into a forward (f) linked list and a backward (b) linked list. The f linked list is colored according to the interval of red and blue, and the b linked list is colored according to the interval of green and blue. Note that the f linked list and b linked list here refer to a continuous segment of the linked list, and the ID sequence relationship of the data in this segment is either all consistent with the location in the memory, or just the opposite. Obviously, there are multiple f linked lists and b linked lists in a linked list.

The realization of this step only needs to be implemented by one external memory sorting, so the time cost is $O(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B})$。

step~2,4

1. $\tilde{L}=copy(L)$
2. Sort the linked list L according to the address of the successor node, and perform the following operations while sorting
   1. In step~2, rewrite the node pointer and weight of the successor node belonging to X
   2. In step~4, rewrite the node pointer and weight of the successor node belonging to X, rewrite and store the weight of the node belonging to X,
      and use the memory algorithm to solve this problem.

Let's take a look at the intuitive understanding of this problem through a picture, as shown in the figure below, the relevant data is: N=10, B=2, M=4. In the worst case, the access cost is O(N) I/O times.

algorithm

Input the external memory linked list L of size N

1. Find an independent set of vertices X in L
2. "Skip" the nodes in X to build a new, smaller external memory linked list L'
3. Solve L' recursively
4. "Backfill" the nodes in X, and construct the rank of L according to the rank of L'

Step 1.1.4 can be done in $OR\left(sort\right)\_=O(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B})$ time I/O solution, construct the following recursive equation:

$T(N)=T((1-a\left)N\right)+O\left(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B}\right)$ , solving the equation:$T(N)=O(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B})$

step~1

According to the ID order of the nodes, it is divided into a forward (f) linked list and a backward (b) linked list. The f linked list is colored according to the interval of red and blue, and the b linked list is colored according to the interval of green and blue. Note that the f linked list and b linked list here refer to a continuous segment of the linked list, and the ID sequence relationship of the data in this segment is either all consistent with the location in the memory, or just the opposite. Obviously, there are multiple f linked lists and b linked lists in a linked list.

The realization of this step only needs to be implemented by one external memory sorting, so the time cost is $O(\frac{N}{B}lo{g}_{\frac{M}{B}}\frac{N}{B})$。

step~2,4

1. $\tilde{L}=copy(L)$
2. Sort the linked list L according to the address of the successor node, and perform the following operations while sorting
   1. In step~2, rewrite the node pointer and weight of the successor node belonging to X
   2. In step~4, rewrite the node pointer and weight of the successor node belonging to X, and rewrite the weight of the node belonging to X
3. Reorder L by address