Massive military manpower is deployed on borders to keep a vigilant eye on possible infiltration from neighboring countries. This traditional approach is prone to errors because of human factors. To make border surveillance more effective, countries have installed CCTV cameras on borders but generally such systems are passive in nature and require human operators to keep an eye on the captured video footage. This chapter describes a cloud based approach for infiltration detection in border defense environments. The processing of video data, in cloud, and the real-time response are the factors that make system suitable for military purposes. As far as affordability is concerned, the governments can easily bear expense of establishing a private cloud for implementing visual surveillance. This chapter represents pertinent research of authors in the field of visual surveillance as well as other state-of-theart breakthroughs in this area. The chapter is of multi-disciplinary significance in the field of cloud computing, video & image processing, behavioral sciences, and defense studies.