Big data algorithm self-test

1 Sublinear Space Algorithms for Big Data

1.1 The counting problem of the flow model

problem definition? What algorithm to use? Algorithmic steps? (Hint: three layers of progression)

Chebyshev's inequality? how to prove? Expectation, variance, space complexity?

Very limited space to store huge numbers

morris，morris+，morris++

$1/(2^X)$ => $\hat{f}=(2^X-1)$

$E\left[c\right]=N$

$D\left[c\right]=\frac{N^2-N}{2k}$。

$P[|\gamma -N|\ge ϵ]⩽\frac{N^2-N}{2k{ϵ}^2}$

1.2 Number of non-repeating elements

problem definition? What algorithm to use? Algorithmic steps?
(Hint: store real numbers: three levels of progression + cannot store real numbers: 1+1)

how to prove? Expectation, variance, space complexity?

Count the number of unique elements in a data stream

1. FM，FM+

   $h:[n]\mapsto [0,1]$ => $z=min\{z,h(i)\}$ => $\frac{1}{z}-1$

   $E[Z]=\frac{1}{d+1}$

   $var\left[Z\right]⩽\frac{2}{(d+1)(d+2)}\frac{1}{q}<\frac{2}{(d+1)(d+1)}\frac{1}{q}$

   $P[|X-d|>{ϵ}^{\prime }d]<\frac{2}{q}(\frac{2}{{ϵ}^{\prime }}+1{)}^2$

2. FM’+

   Maintain the smallest k hash values ​​currently seen, return $\frac{k}{z_k}$

3. PracticalFM

   1. If $zeros(h(j))>z$：$z=zeros(h(j))$

   2. return $\hat{d}=2^{z+\frac{1}{2}}$

   $E[{AND}_r]=\frac{d}{2^r}$

   $var\left[Y_r\right]\le \frac{d}{2^r}$。

   The final correct probability should be greater than $1-\frac{2\sqrt{2}}{C}$。

4. BJKST

   1. 若$zeros\left(h\right(j))>z$

      1. $B=B\cup (g(j),zeros(h(j)))$
      2. Then $|B|>\frac{c}{{ϵ}^2}$
         1. $z=z+1$
         2. remove (α,β) from B (α,β)$(a,b)$，wherein$b<z$

   2. return $\hat{d}=|B|2^z$

1.3 point query

problem definition? What algorithm to use? Algorithmic steps?

Space complexity?

Count the number of occurrences of all elements in the stream

1. Misra_Gries

   Maintain a set A whose elements are $(i,\widehat{f_i})$

   1. $A\leftarrow \varnothing$

   2. For each element e in the data stream

      if e∈A，令$(e,\widehat{f_e})\to (e,\widehat{f_e}+1)$

      else if $|A|<\frac{1}{ϵ}$: Insert (e,1) into A

      else

      1. Decrement all counts in A by 1
      2. if $\widehat{f_j}=0$ : remove (j,0) from A

   3. For query i, if $i\in A$ , returns$\widehat{f_i}$, otherwise return 0

   The space cost is $O({ϵ}^{-1}logn)$

2. Metwally

   1. For each element e in the data stream
      1. if e∈A：令$(e,\widehat{f_i})\leftarrow (e,\widehat{f_i}+1)$
      2. else if $|A|<\frac{1}{ϵ}$: Insert (e,1) into A
      3. else Insert (e,MIN+1) into A, and delete one that satisfies $\widehat{f_e}=MIN$
   2. query i if $i\in A$ , returns$\widehat{f_i}$, otherwise return MIN

   The space cost is $O({ϵ}^{-1}logn)$

3. Count-Min

   1. Randomly select t 2−wise independent hash functions $h_i:[n]\to [k]$

   2. For each update (j,c) that occurs, do the following

      for i=1 to t

      ​ $C[i][h_i(j)]=C[i][h_i(j)]+c$

   3. For a query for a, return $\widehat{f_a}={\min }_{1\le i\le t}C\left[i\right][h_i(a)]$

4. Count-Median (min change median)

5. Count Sketch

   1. Randomly select a 2−wise independent hash function $h:[n]\to [k]$

   2. Randomly select a 2−wise independent hash function $g:\left[n\right]\to -1,1$

   3. For each update (j,c)

      $C[h(j)]=C\left[h\right(j)]+c\ast g(j)$

   4. For query a, return $\hat{f}=g(a)\ast C\left[h\right(j)]$

6. Count Sketch+
   (equivalent to running the Count Sketch algorithm t times, and finally taking the median value)

   1. Randomly select a 2−wise independent hash function $h_i:[n]\to [k]$

   2. Randomly select a 2−wise independent hash function gi : [ n ] → { − 1 , 1 } g_i:[n] \rightarrow \{-1,1\}gi​:[n]→{ −1,1}

      For each update (j,c)

      ​ For $i:1\to t$

      ​ $C[h_i(j)]=C\left[h_i\right(j)]+c\ast g_i(j)$

   3. 返回$\hat{f}=media{n}_{1\le i\le t}{g}_i(a)C[i][h_i(a)]$

1.4 Frequency moment estimation

problem definition? What algorithm to use? Algorithmic steps?

Expectation, variance, space complexity?

slightly

1.5 Fixed-Size Sampling

problem definition? What algorithm to use? Algorithmic steps?

Expectation, variance, space complexity?

Reservoir Sampling Algorithm

1. Initialize the sampling array
   $A[1,...,s],m\leftarrow s$
2. For each update x
   1. x by $\frac{s}{m+1}$Probabilistically replace an element in A at random
   2. m++

1.6 Bloom Filter

problem definition? What algorithm to use? Algorithmic steps?

Draw a small data set from the large data set, draw a number, and guess whether it belongs to the small data set

1. approximate hash

   1. Let H be a family of universal hash functions: $[U]\to [m]，m=\frac{n}{d}$
   2. Randomly select h∈H, and maintain the array A[m], the size of S is n
   3. For each i∈S, $A[h(i)]=1$
   4. Given a query q, return yes if and only if $A\left[h\right(i)]=1$

2. Bloom Filter

   1. Let H be a family of independent ideal hash functions: [U]→[m]

   2. Randomly select $h_1,...,h_d\in H$ , and maintain the array A[m]

   3. For every i∈S

      ​ For each j∈[1,d]

      ​ $A[h_j(i)]=1$

   4. Given a query q, return yes if and only if $\forall j\in [d],A[h_j(q)]=1$

2 Sublinear Time Algorithms for Big Data

2.1 Find the number of connected components

problem definition? What algorithm to use? Algorithmic steps?

Calculation formula? time complexity?

If the number of searched points is less than $\frac{2}{ϵ}$Just continue to search, otherwise return directly to $\frac{2}{ϵ}$。

Randomly select r = b / ϵ 2 r = b/{\epsilon}^2 from the set of nodes$r=b/{ϵ}^2$ nodes form a node U, apply this algorithm to each node

最终的$\hat{C}=\frac{n}{r}{\sum }_{u\in U}\frac{1}{\widehat{n_u}}$, the time complexity is $O(d/{ϵ}^3)$

2.2 Approximate minimum spanning tree

problem definition? What algorithm to use? Algorithmic steps?

Calculation formula? time complexity?

G-symbol $G^{(i)}=(V,E^{(i)})$，$E\left(i\right)=\{(u,v)|w_{uv}\le i\}$ , the number of connected components is C(i)

M is the sum of the number of connected components of all such subgraphs: $M=n-w+{\sum }_{i=1}^{w-1}{C}^{(i)}$
$$\begin{array}{rl}M& =\sum _{i=1}^{w}i\cdot a_i=\sum _{i=1}^w{a}_i+\sum _{i=2}^w{a}_i+\cdots +\sum _{i=w}^w{a}_i\\ & =C^{(0)}-1+C^{(1)}-1+\cdots +C^{(w-1)}-1\\ & =n-1+C^{\left(1\right)}-1+\cdots +C^{(w-1)}-1\\ & =n-w+\sum _{i=1}^{w-1}{C}^{(i)}\end{array}$$

2.3 Find the diameter of the point set

problem definition? What algorithm to use? Algorithmic steps?

Calculation formula? time complexity?

The Indyk’s Algorithm

1. Optional $k\in [1,m]$
2. select $l$，使得$\forall i,D_{}\le D_{kl}$
3. returns $(k,l),D_{kl}$

2.4 Average Degree Algorithm for Calculation Graph

Alg III

1. Let V be an independent variable S, $|S|=\tilde{O}(\frac{L}{\rho {ϵ}^2}),L=poly(\frac{log}{ϵ}),r=\frac{1}{t}\sqrt{\frac{}{4}\cdot \frac{}{n}}$
2. $S_i\leftarrow S\cap B_i$
3. f o r   i ∈ { 0 , … , t − 1 }   d o \boldsymbol{for}\ i \in \{0,\dots,t-1\}\ \boldsymbol{do} for i∈{ 0,…,t−1} do
   1. $\mathbf{if}|S_i|\ge i_{}\mathbf{then}$
      1. $r_i\leftarrow \frac{|S_i|}{|S|}$
      2. $estimate{\Delta }_i$
   2. $\mathbf{else}$
      1. $r_i\leftarrow 0$
4. $\mathbf{return}\hat{\bar{d}}={\sum }_{i=0}^{t-1}(1+D_i)r_i(1+b{)}^i$

Algae IV

1. $a\leftarrow n$
2. $\hat{\bar{d}}<-\infty$
3. $\mathbf{while}\hat{\bar{d}}<ado$
   1. $a\leftarrow a/2$
   2. $\mathbf{if}\alpha <\frac{1}{n}\mathbf{then}$
      1. $\mathbf{return}0;$
   3. $\hat{\bar{d}}\leftarrow AlgII{I}_{\sim \alpha }$
4. $\mathbf{return}\hat{\bar{d}}$

Algorithm related indicators

Approximate ratio: $(1+)\_$

Running time: $\tilde{O}(\sqrt{n})\cdot poly({ϵ}^{-1}logn)\sqrt{n/\bar{d}}$

3 Parallel Computing Algorithms

3.1 Building an inverted index

problem definition? What does the Map function do? What does the Reduce function do?

Given a set of documents, count which documents each word appears in

map：$<docID,content>\to <word,docID>$
reduce：$<word,docID>\to <word,listofdocID>$

3.2 Word Count

problem definition? What does the Map function do? What does the Reduce function do?

Given a set of documents, count the number of occurrences of each word

• Map function: <docID, content> → <word, 1>
• Reduce function: <word,1>→<word,count>

3.3 Search

problem definition?

Given a line number and corresponding document content, count the occurrences of a specified word

• Map函数：$<lineID,word>\to <word,lineID>$
• Reduce函数：$<word,lineID>\to <word,listoflineID>>$

3.4 Matrix multiplication

problem definition?

Two algorithms: What does the Map function do? What does the Reduce function do?

1. Matrix multiplication 1
   • Map：
     • ((A,i,j),aij)→(j,(A,i,aij))
     • ((B,j,k),bjk)→(j,(B,k,bjk))
   • Reduce：(j,(A,i,aij)),(j,(B,k,bjk))→((i,k),aij∗bjk)
   • Map：nothing（identity）
   • Reduce：((i,k),(v1,v2,…))→((i,k),∑vi)
2. Matrix multiplication 2

   • Map function:

     • ((A,i,j),aij)→((i,x),(A,j,aij)) for all x∈[1,n]
     • ((B,j,k),bjk)→((y,k),(B,j,bjk)) for all y∈[1,m]

   • Reduce function: ((i,k),(A,j,aij))∧((i,k),(B,j,bjk))→((i,k),∑aij∗bjk)

3.5 Sorting algorithm

What does the Map function do? What does the Reduce function do? A key to solving the problem?

Using p processors, input $<i,A[i]>$

• Map：$<i,A\left[i\right]>\to <j,((i,A[i]),y)>$

  1. 输出$<i\%p,((i,A[i]),0)>$

  2. With probability T/n for all $j\in [0,p-1]$输出$<j,((i,A[i]),1)>$

     Otherwise output $<j,((i,A[i]),0)>$

• Reduce：

  • Collect the data of y=1 as S and sort
  • Construct $(s_1,s_2,...,s_{p-1})$，$s_k$For the kth in S $k\lceil \frac{|S|}{p}\rceil$
  • Collect data for y=0 as D
  • Satisfy sk < x ≤ sk + 1 s_k < x \leq s_{k+1} for any (i,x)∈D$s_k<x\le s_{k+1}$, output <k,(i,x)>

• Map：nothing（identity）

• Reduce：$ <j, ((i, A[i]), . . . )>$

  • general owned $(i,A[i])$ Sort and output according to $A[i]$

3.6 Computing the minimum spanning tree (spanning tree)

The main idea? What does the Map function do? What does the Reduce function do?

Using the graph partition algorithm, the graph G is divided into k subgraphs, and the minimum spanning tree is calculated in each subgraph

The essence of the algorithm is to calculate the spanning tree locally first, then use the remaining edges connecting these spanning trees to form a new graph, and find the minimum spanning tree of this new graph as the total result

• Map：input:<(u,v),NULL>
  • transformation<(h(u),h(v));(u,v)>
  • For the above conversion data, if h(u)=h(v), then for all j∈[1,k], output <(h(u),j);(u,v)>
• Reduce：input:<(i,j);Eij>
  • 令Me=MSF(Thou)
  • Output <NULL;(u,v)> for each edge e=(u,v) in Mij
• Map：nothing（identity）
• Reduce：M=MST(H)

4 External memory model algorithm

4.1 External storage model

In the I/O model, the memory size is ___, the page size is ___, and the external storage size is ___. How many I/Os are required to continuously read N data from the external memory?

M, B, Unlimited, N/B

4.2 Computing matrix multiplication

Input two matrices X and Y of size N×N

1. Divide the matrix into blocks of size ___
2. Considering each block in the X×Y matrix, there are obviously ___ blocks to output
3. Each block needs to scan ___ pairs of input blocks
4. Each in-memory calculation requires ___ I/O
5. Total___times I/O

$\sqrt{M}/2\times \sqrt{M}/2$

$O((\frac{N}{\sqrt{M}}{)}^2)$

$\frac{N}{\sqrt{M}}$

O(M/B)

$O((\frac{N}{\sqrt{M}}{)}^3\cdot M/B)$

4.3 Data structure

4.3.1 External storage stack

The memory maintains an array of size ___, realizes the memory stack structure, and stores the rest of the data in the external memory

How to push the stack (push)?
How to pop the stack (pop)?

I/O cost analysis:
▷ Worst case cost: O(1) times I/O
▷ Amortized analysis: ___, optimal

2b

If it is not full, press it, if it is full, write it out and then press it

Play it if it’s not empty, read it when it’s empty

O(1/B)

4.3.2 External memory linked list

Queue (Queue)
▷ The memory maintains two arrays A and B of size B, one for dequeue and one for enqueue
▷ A and B are stored separately?
▷ The rest of the data is stored in external memory
How to handle queue operations?
▷ Insert?
▷ Remove?
I/O cost analysis:
▷ Worst case cost: O(1) I/O
▷ Amortized analysis: ___, optimal

k queue head data and k′ queue tail data

If B is not full, it will be stored in memory, if it is full, it will be written out and then stored in memory

Play when A is empty, read and play again when empty

O(1/B)

4.3.3 Linked list

Perform three operations: insert(x,p), remove§, traverse(p,k)

• Idea 2: block "half full" ⇒ data at least B/2;

  Under the external memory model, the consecutive elements in a linked list are placed in a block of size B. At the same time, make each block size at least B/2:
  ▷ remove: When to merge? Under what circumstances should it be equally divided?
  ▷ insert: under what circumstances should it be equally divided?
  ▷ traverse: ___, the worst case cost of insert and remove is O(1)

  ▷ Amortized cost: N consecutive insertions ___, consecutive deletions ___

  If it is less than B/2, it will be merged with the neighbor block, and if it is greater than B, it will be divided equally

  greater than B, evenly divided

  O(2k/B)

  N consecutive insertions O(2N/B), consecutive deletions $O(lo{g}_{2}B\cdot N/B)$

• Idea 3: Two consecutive blocks contain at least 2B/3 data;

  Two consecutive blocks contain at least 2B/3 data; the memory maintenance size is B buffer
  ▷ remove: when to merge? Under what circumstances?
  ▷ insert: When to merge? Under what circumstances is it equally divided?
  ▷ traverse: ___

  ▷ Amortized cost: N consecutive insertions ___, continuous deletions ___
  ▷ Amortized cost: N consecutive updates ___

Delete and check whether there are adjacent blocks such that the data volume is ≤ 2B/3, and if so, merge

If the current block is full, insert to the neighbor; if the neighbors are full, divide the current block equally

O(3k/B)

O(2N/B)，O(3N/B)

O(12N/B)

4.4 Search structure

Perform three operations: insert(x), remove(x), query(x)

$(a,b)-tree:$ the relationship between a and b

Similar to binary search tree $\Rightarrow (p_0,k_1,p_1,k_2,p_2,...,k_c,p_c)$

The root node has __ children; the number of children of each non-leaf node is __:

• remove: how to operate?
• insert: how to operate?
• query: time complexity?

2 ≤ a ≤ (b + 1)/2

The root node has 0 or ≥ 2 children, and the number of children of other non-leaf nodes ∈ [a, b]

If it is less than a after deletion, it will be merged with the adjacent block, if it is greater than b after the merger, it will be divided evenly

If it is greater than b after insertion, it will be divided equally

$ O(log_a(N/a))$

insert operation

Assuming the inserted key value is K, first find the corresponding leaf node L

• If there is free space in L, then insert directly, end;

• Otherwise, split the leaf node into two nodes, and divide the keys into these two new nodes, so that the number of keys meets the minimum requirement;

  • When splitting leaf node N:

    ▷Create a new node M, let M be the right brother of N, sort the keys, before $\lceil (n+1)/2\rceil$ stay in N, the other key‐pointers are put in M.

  • When splitting the non-leaf node N:
    ▷Sort the key-pointers, before $\lceil (n+2)/2\rceil$ pointers stay in N, leaving$\lfloor (n+2)/2\rfloor$ pointers into M.
    ▷ Front$\lceil n/2\rceil$ keys stay in N, after$\lfloor n/2\rfloor$ keys are put into M, and the key in the middle is reserved and inserted into the upper layer node, and the key pointer points to M.

delete operation

Assuming that the deleted key value is K, first find the corresponding leaf node L.

• If L still has the minimum number of keys after deleting K, stop

• Otherwise, you need to do the following:

  ▷Try to merge with one of L's adjacent sibling nodes (the same node can still be placed after merging). After merging, it is equivalent to deleting a key value in the upper node, then recursive processing;

  ▷ Otherwise, consider L's adjacent siblings

  • Assuming that one of them can provide L with a key-pointer, and after removing the key-pointer, the brother node still meets the minimum requirement for the number of keys, then L borrows a pair of key-pointers from the brother, and updates the corresponding key value;
  • If neither sibling can provide a key-pointer, then it must be the case that L has less than the minimum number of keys, and L's brother M has exactly the minimum number of keys, then the two nodes can be merged.

B+ tree – performance

Maximum number of pointers in a node: n, number of records: N
B+Tree insertion operation: $O(lo{g}_{\lceil n/2\rceil }(N))$
B+Tree delete operation:$O(lo{g}_{\lceil n/2\rceil }(N))$

4.5 External memory sorting

▷ Given __ data
▷ Divided into groups of size __, each group can be sorted in memory, requiring __ I/O times
▷ sorted groups, merge (Merge)
▷ __ can be merged each time Grouping
▷ I/O cost:
▷ Drawing comprehension

N

O(M)，O(M/B)

O(M/B)

$O(N/B\cdot lo{g}_{M/B}\frac{N}{B})$或$O(N/B\cdot lo{g}_{M/B}\frac{N}{M})$ (doubtful)

4.6 List Ranking

problem definition? (two questions)

algorithm? (four steps)

Given an adjacency linked list L of size N, L is stored in an array (contiguous external storage space), and the rank (serial number in the linked list) of each node is calculated

Input the external memory linked list L of size N

1. Find an independent set of vertices X in L
2. "Skip" the nodes in X to build a new, smaller external memory linked list L'
3. Solve L' recursively
4. "Backfill" the nodes in X, and construct the rank of L according to the rank of L'