Recognizing representative living patterns in population is extremely valuable for urban planning and decision making. Thanks to the growing popularity of location-based applications and check-ins on social networking sites, Point of Interest (POI) of a location is quite often available in the trajectory data, which expresses user living semantics. However, adopting trajectory semantics for living pattern recognition is an open and challenging research problem due to three major technical challenges: effective feature representation, suitable granularity selection for habit unit, and reliable habit distance measurement. In this paper, we propose a representation learning based system named habit2vec to represent user trajectory semantics in vector space, which preserves the original user living habit information. We evaluated our proposed system on a large-scale real-world dataset provided by a popular social network operator including 123,803 users for 1.5 months in Beijing. The results justify the representation ability of our system in preserving user habit pattern, and demonstrate the effectiveness of clustering users with similar living habits.