Poxviruses (PXVs) are mostly known for the variola virus, being the cause of smallpox; however, re-emerging PXVs have also shown a great capacity to develop outbreaks of pox-like infections in humans. The situation is alarming; PXV outbreaks have been involving both endemic and non-endemic areas in recent decades. Stopped smallpox vaccination is a reason offered mainly for this changing epidemiology that implies the protective role of immunity in the pathology of PXV infections. The immune system recognizes PXVs and elicits responses, but PXVs can antagonize these responses. Here, we briefly review the immunology of PXV infections, with emphasis on the role of pattern-recognition receptors, macrophages, and natural killer cells in the early response to PXV infections and PXVs’ strategies influencing these responses, as well as taking a glance at other immune cells, which discussion over them mainly occurs in association with PXV immunization rather than PXV infection. Throughout the review, numerous evasion mechanisms are highlighted, which might have implications for designing specific immunotherapies for PXV in the future.