14

Announcements: Start chapter 10 New problem set out

What is clustering and why do people want it?

Basic idea:

• you have a collection of `atoms'
• you have `relationships' between atoms
• highlight strong relationships
• highlight weak relationships

Bird's eye view:

• many samples consisting of instances of a few things
• reduce the data to something managable

A quadrant II task

Applications

• assessing function
• inferring relatedness
• match triage
• (theme discovery)
• (load balancing)

Methods

Kinds of clustering

• inputs: distances, similarities, high-D embeddings
• outputs: partitions, trees, representatives
• Does it optimize something?
  • Is it exact or heuristic?
• Is it consistent?

Heuristic styles

• Agglomerative Hierarchical

    P = trivial partition of n-blocks
    while P != trivial partition of 1-block
      pick two closest blocks B1,B2 from P
      combine B1,B2 into a new block B to get P'
  

• Divisive Hierarchical

    P = trivial partition of 1-block
    while P != trivial partition of n-blocks
      pick the weakest block B
      split B into two blocks B1,B2 to get P'
  

• Local optimization

    while 1
      find MIN_P' Score(P) over all P' near P
      if P' == P then stop!
  

Doing clustering for biologists

Science demands repeatability and robustness. Simpler methods often have an advantage.

Hierarchical methods produce trees as a bi-product

k-Means

k-means is simple and popular

   randomly generate k centers c_i
   while 1
     Assign x_i to block B_j and min_j d(x_i,c_j)
     c'_j = average of x_i in B_j
     if all c'_j == c_j then stop!

Equivalent to a local optimization of:

  F(P) = sum of Var(B_j) = Score(P)

Lloyd algorithm can be replaced by something else

Score(P) just penalizes for bad blocks

k-Means generalizations

Change the optimization score to another block-wise penalty

  F(P) = sum of cost(B_j) = Score(P)

Common variations

• k-Median
• k-Center

All variations are computationally intractable

Corrupted Clique Problems

Defn: A clique graph - exists a partition P so that every block of P is a clique

Can obtain a graph from the data by thresholding

Corrupted cliques I

• input: graph
• output: nearest clique graph which can be obtained by changing edges
• Optimizes F(P) = #(edges not in cluster) + #(edges between clusters)

Corrupted cliques II

• input: graph
• output: nearest clique graph which can be obtained by adding edges
• Optimizes F(P) = #(edges not in cluster) + (infinity) * #(edges between clusters)

Parallel Classification with Cores (PCC)

Model: G = clique graph + random noise

PCC is correct with high probability in the limit of large graphs

  Divide nodes up into S1, S2, S3
  for-each P1 on S1
    create P2 from P1 by extending each block B of P1
      x in S2 is added to B where max_B affinity(x,B)
    create P3 from P2 by extending each block of P2
      x in S3 is added to B where max_B affinity(x,B)
    P = better of P and P3

  affinity(x,B) = #(edges from x to B)/#(elements in B)

A more flexible heuristic called Cluster Affinity Search Technique

  P = empty partition (no blocks)
  while S not empty
    C = {s} where s is maximum degree
    while 1
      if v in S-C s.t. affinity(v,C)>=t
        C = C+v  [add highest such v]
      else v in C s.t. affinity(v,C)<t
        C = C-v  [remove lowest such v]
      else
        stop
    S = S-C
    add C to P

CAST: unbounded runtime, but seems to work well

Spectral methods

Spectral bisection

• input: similarity matrix
• output: divisive hierarchical clustering
• compute first eigenvector of the Laplacian of M
• order atoms according to eigenvector values
• divide into two parts and recurse

Corrupted cliques

• input: graph
• output: partition into approximate cliques
• compute largest eigenvalues and eigenvectors
• eigenvalues ~ clique sizes
• eigenvectors ~ approximate indicator functions
• a bit tricky when eigenvalues nearly equal
• generalizes to corrupted bi-partite cliques with SVD

Low dimensional projection

• input: atom data as points in high-D space
• output: low dimensional orthogonal projection maximizing variance
• Not clustering, but it is a powerful data reduction
• can be adapted to similarity and distance based inputs
• the Grassmann analogue of k-Means