It is known that phonons have angular momentum, and when the time-reversal symmetry (TRS) is broken, the total phonon angular momentum in the whole system becomes nonzero. In this paper, we propose that as an angular momentum of phonons for a crystal without TRS, we need to consider the canonical angular momentum, as opposed to the kinetic angular momentum in previous works. Next, we show that the angular momentum of phonons without TRS exhibits universal behaviors near the $\Gamma$ point. We focus on in-plane oscillations in two-dimensional crystals as an example. By breaking the TRS, one of the acoustic phonon branches at the $\Gamma$ point acquires a gap. We show that the angular momentum of its acoustic phonon with a gap has a peak with the height $\pm \hbar$ regardless of the details of the system. From this, we find that this peak height changes discontinuously by changing the sign of the TRS-breaking parameter.