An Enhanced Hierarchical Clustering Algorithm: Methodology and Application

Abstract

Hierarchical clustering is a widely used technique for data grouping in various fields, renowned for its ability to create a dendrogram that provides insights into data structure. However, traditional hierarchical methods often suffer from challenges such as computational inefficiency, sensitivity to noise, and difficulty in determining the optimal number of clusters. This paper introduces an improved hierarchical clustering algorithm that incorporates advanced distance metrics, the use of cluster representatives, and a robust agglomeration strategy designed to address these challenges. The proposed algorithm incorporates a mechanism to dynamically select the distance metric based on the underlying data characteristics. Unlike traditional methods that rely solely on a fixed distance metric, our approach determines the most suitable metric for the given dataset. For instance, a method like Cosine similarity could be employed for high-dimensional data or text data where orientation matters, while Euclidean distance can be effective for low-dimensional, continuous data. Our proposed method is evaluated through comprehensive experiments on synthetic and real-world datasets, showcasing significant enhancements in clustering performance, adaptability to noise, and computational efficiency. The findings indicate that the proposed algorithm outperforms traditional hierarchical methods, demonstrating its potential for broader applications across various domains.