This paper presents a multi-centroid diastolic duration model for the hidden semi-Markov model (HSMM) based heart sound segmentation. The centroids are calculated by hierarchical agglomerative clustering of the neighboring diastolic duration values using Ward’s method until center of clusters are found at least a systolic duration apart. The multiple peak distribution yields a sharper gradient of likelihood around the expected centroids and improves the discriminability of similar observations. The peak density at each centroid acts as a reference point for the HSMM to determine the origin of the hidden-state and adjust the corresponding state duration based on the maximum likelihood criterion. This model overcomes the limitation of the single peak mean value model that may overfit the duration distribution when the heart rate variation is relatively large. An extended logistic regression-HSMM algorithm using the proposed duration model is presented for the heart sound segmentation. In addition, the total variation filter is used to attenuate the effect of noises and emphasize the fundamental heart sounds, S1 and S2. The proposed method is evaluated on the training-set-a of 2016 Physionet/Computing in Cardiology Challenge and yields an average F1 score of 98.36 ± 0.43.