Identification of the seed varieties is essential in the quality control and high yield crop growth. The existing methods of varietal identification rely primarily on visual examination and DNA fingerprinting. Although the pattern of DNA fingerprinting allows precise classification of seed varieties but fraught with challenges such as low rate of polymorphism amongst closely related species, destructive method of analysis and a huge cost involved in identification of robust markers such as simple sequence repeat (SSR) and single nucleotide poly-morphisms. Here, we propose a fast, non-contact and non-invasive technique, deep learning assisted optical coherence tomography (OCT) for subsurface imaging in order to distinguish different seed varieties. The volu-metric dataset of, (a) four rice varieties (PUSA Basmati 1, PUSA 1509, PUSA 44 and IR 64) and, (b) seven morphologically similar seeds of rice landrace Pokkali was acquired using OCT technique. A feedforward deep neural network is implemented for deep feature extraction and to classify the OCT images into their relevant classes. The proposed method provides the classification accuracy of 89.6% for the dataset of total 158,421 OCT images and 82.5% in classifying the dataset of total 56,301 OCT images collected from Pokkali seeds. The current technique can accurately classify seed varieties irrespective of the morphological similarities and can be adopted for the removal of varietal duplication and assessment of the purity of the seeds.