The aim of this study was to investigate the relationship between heart rate and heart rate variability (HRV) with respect to individual characteristics and acute stressors. In particular, the relationship between heart rate, HRV, age, sex, body mass index (BMI), and physical activity level was analyzed cross-sectionally in a large sample of 28,175 individuals. Additionally, the change in heart rate and HRV in response to common acute stressors such as training of different intensities, alcohol intake, the menstrual cycle, and sickness was analyzed longitudinally. Acute stressors were analyzed over a period of 5 years for a total of 9 million measurements (320±374 measurements per person). HRV at the population level reduced with age (p < 0.05, r = −0.35, effect size = moderate) and was weakly associated with physical activity level (p < 0.05, r = 0.21, effect size = small) and not associated with sex (p = 0.35, d = 0.02, effect size = negligible). Heart rate was moderately associated with physical activity level (p < 0.05, r = 0.30, effect size = moderate) and sex (p < 0.05, d = 0.63, effect size = moderate) but not with age (p = 0.35, r = −0.01). Similar relationships between BMI, resting heart rate (p < 0.05, r = 0.19, effect size = small), and HRV (p < 0.05, r = −0.10, effect size = small) are shown. In response to acute stressors, we report a 4.6% change in HRV (p < 0.05, d = 0.36, effect size = small) and a 1.3% change in heart rate (p < 0.05, d = 0.38, effect size = small) in response to training, a 6% increase in heart rate (p < 0.05, d = 0.97, effect size = large) and a 12% reduction in HRV (p < 0.05, d = 0.55, effect size = moderate) after high alcohol intake, a 1.6% change in heart rate (p < 0.05, d = 1.41, effect size = large) and a 3.2% change in HRV (p < 0.05, d = 0.80, effect size = large) between the follicular and luteal phases of the menstrual cycle, and a 6% increase in heart rate (p < 0.05, d = 0.97, effect size = large) and 10% reduction in HRV (p < 0.05, d = 0.47, effect size = moderate) during sickness. Acute stressors analysis revealed how HRV is a more sensitive but not specific marker of stress. In conclusion, a short resting heart rate and HRV measurement upon waking using a smartphone app can effectively be used in free-living to quantify individual stress responses across a large range of individuals and stressors.