Despite continuous growth in dietary pattern research, the relative importance of each dietary component in the overall pattern and their joint effects on mortality risk have not been examined adequately. We explored the individual and joint associations of multiple food groups with all-cause and cause-specific mortality (cardiovascular disease (CVD) or cancer), by analyzing data from a cohort of 3995 Hong Kong Chinese older adults in the Mr. Osteoporosis (OS) and Ms. OS Study. Cox proportional hazards models were used to examine the associations of food groups with mortality risk. The individual and joint contribution of food groups to mortality risk has been quantified by a machine learning approach, i.e., the Quantile G-Computation. When comparing the highest with the lowest quartile of intake, dark green and leafy vegetables (hazard ratio (HR) = 0.82, 95% confidence interval (CI) = 0.70 to 0.96, Ptrend = 0.049), fruit (HR = 0.79, 95% CI = 0.68 to 0.93, Ptrend = 0.006), legumes (HR = 0.75, 95% CI = 0.63 to 0.87, Ptrend = 0.052), mushroom and fungi (HR = 0.76, 95% CI = 0.65 to 0.88, Ptrend = 0.023), soy and soy products (HR = 0.77, 95% CI = 0.66 to 0.90, Ptrend = 0.143), and whole grains (HR = 0.76, 95% CI = 0.65 to 0.89, Ptrend = 0.008) were inversely associated with all-cause mortality. Legume intake was associated with a lower risk of CVD mortality, while fruit, nuts, soy and soy products were associated with a lower risk of cancer mortality. From the Quantile G-Computation, whole grains, legumes, fruits, mushroom and fungi, soy and soy products had a higher relative weighting on mortality risk, and the joint effect of food groups was inversely associated with the mortality risk due to all-causes (HR = 0.39, 95% CI = 0.27 to 0.55), CVD (HR = 0.78, 95% CI = 0.67 to 0.91), and cancer (HR = 0.31, 95% CI = 0.15 to 0.65). From a sex-stratified analysis, most associations between food groups (whole grains, legumes, fruits, mushroom and fungi, soy and soy products) and mortality risk remained significant among men. In conclusion, whole grains, legumes, fruits, mushroom and fungi, soy and soy products were the main contributors to a reduction in mortality risk, and their joint effects were stronger than individual food groups. Moreover, the sex-specific association of sweets and desserts with cancer mortality may be worth further investigation.