AI Trends

SuperEMeans Combines SuperKMeans and e-means to Speed Up High-Dimensional Vector Clustering by 3×

Trending on Hacker NewsUpdated Nov 19 05:23

AI summary

SuperEMeans combines SuperKMeans with e-means, accelerating high-dimensional vector clustering by up to 3× without reducing clustering quality. The code is open-sourced at https://github.com/cwida/SuperKMeans. Tests on OpenAI and Cohere embedding datasets show speedups over e-means of up to 1.8× at 2 epochs and 3× to 3.2× at 5 epochs. The tests also found that cluster imbalance in e-means increases the number of vectors scanned; raising the revival threshold from 0.1 to 0.3 brings scanning costs on the Cohere dataset closer to those of Lloyd’s k-means.

Why read this

AI score 56

The tests present clustering time, recall, and the number of vectors scanned, revealing that WCSS alone is insufficient for measuring cluster balance and query costs in vector search.

Topics

Tags

Body

Only a summary is authorized here. Continue at the validated source link.

Source

Trending on Hacker News