Human communication not only involves the need to switch between the modalities of speaking and listening, but for bilinguals, it can also involve switching between languages. It is unknown as to whether modality and language switching share underlying control mechanisms or whether one type of switching affects control processes involved in the other. The present study uses behavioral and fMRI measures to examine neural circuits of control during communicative situations that required Chinese-English bilinguals to switch between modalities and their two languages according to associated color cues. The results showed that for both language and modality control, similar brain regions were recruited during speech production and comprehension. For modality control, the specific control processes partly depended on the corresponding modality. Finally, switching between modalities appears to exert more influence on language control in production compared to comprehension. These findings offer a first detailed characterization of the neural bases involved in control mechanisms in bilingual communication.