The four core technologies of automatic driving have evolved environment perception, precise positioning, path planning. and line control execution. Perfect planning must establish a deep understanding of the surrounding environment for environmental perception, especially the dynamic environment. Visual environment perception has played a key role in the development of autonomous vehicles. It has been widely used in intelligent rearview mirror, reversing radar, 360° panorama, driving recorder, collision warning, traffic light recognition, lane departure, line-parallel assistance, automatic parking and etc. The traditional way to obtain environmental information is the narrow-angle pinhole camera, which has limited field of vision and blind area. Multiple cameras are often needed to be covered around the car body, which not only increases the cost, but also increases the information processing time. Fisheye lens perception can be an effective way to use for environmental information. The large field of view (FOV) can provide the entire hemisphere view of 180°. Theoretically, The capability to cover 360° to avoid visual blindness, reduce the occlusion of visual objects, provide more information for visual perception and greatly reduce the processing time with only two cameras. Based on deep learning, processing surrounded image has been mainly processed in two ways. First, the surrounded fisheye image is transformed into ordinary normal image based on the image correction and distortion. The corrected image has been processed via classical image processing algorithm. The disadvantage is that image distortion can damage image quality, especially the image edges, lead to important visual information missing, the closer the image edge, the more loss of information. Second, the distorted fisheye image has been modeled and processed directly. The complexity of the fisheye image geometric process (model) cannot make the algorithm to migrate to the surrounded fisheye image very well, which is determined by the imaging characteristics of ordinary image and fisheye image, there is no surround fisheye image modeling model with better effect. Finally, there is no representative public dataset to carry out unified evaluation of the vision algorithm, and there is also a lack of a large number of data for model training. The related research directions of the fisheye image including the correction processing of the fisheye image have been summarized. Subdivided into the fisheye image correction method based on calibration has been conducted and the fisheye image correction method based on the projection transformation model has been demonstrated; the target detection in the fisheye image has been mainly introduced to pedestrian detection as well. The city road environment semantic segmentation, pseudo fisheye image dataset generation method has mainly been introduced based on the semantic segmentation of fisheye images. The other fisheye image modeling methods have been used to list the approximate proportion of these research directions and analyze the application background and real-time characteristics of the environment of automatic driving vehicle. In addition, the general datasets of the fisheye image has included the size of these datasets, publishing time, annotation category and etc. The experimental results of object detection methods and semantic segmentation methods in the fisheye image have been compared and analyzed. The evaluation dataset of fisheye image, the construction of algorithm model of fisheye image and the efficiency of the model issues have been discussed. The fisheye image processing has been benefited from the development of weak supervised and unsupervised learning.