Liutong Xu’s research while affiliated with Beijing University of Posts and Telecommunications and other places

Knowledge discovery has often used as a background application to motivate many technical problems in ETL research. However, traditional ETL tools face new challenges include tremendous amount of data and limitation of computing ability and so on. Meanwhile, MapReduce parallel computing model has been widely used in recent years. In This paper, we first analyze the problems of existing ETL tools and propose a metadatadriven ETL service model, and then summarize the types of metadata and their application scopes. Based on this metadata-driven ETL service model, we put forward a concrete ETL framework combined ETL with MapReduce algorithm framework and provided as PaaS to meet the requirements. Afterwards, many significant services are also discussed. At last, we illustrate some strategies for advancing the flexibility, extensibility of the framework and promote the reusability of ETL components and ETL application. In conclusion, practices have proved that the model and the framework proposed in this paper have advantages that open-source or commercial ETL tools do not have and can deal the problem of processing large scale data.

Infrastructure-as-a-service (IaaS), it is typically a platform virtualization environment as a service, which deploys IT infrastructure as virtual machines in a provider’s data center. There are many open source cloud platforms such as Eucalyptus, Open Nebula, oVirt, Enomaly ECP etc, that can help to transform enterprises’ or organizations’ existing IT infrastructure into a private or hybrid cloud. With the growing popularity of IaaS, resource monitoring and VM (Virtual Machine) monitoring has became more and more important in cloud management. But all those open source cloud platforms cannot satisfy the requirement of cloud environment monitoring and management completely. In this paper, we present a novel IaaS management platform based on Eucalyptus, which uses Libvirt management library, SNMP (simple network management protocol) and VMM (Virtual Machine Monitor) to get more abilities to manage and monitor the cloud. This platform provides some new features which can manage and monitor resources and VM status in the cloud environment and give a good user experience.

As the volume of data increasing sharply and the relationship among different data sources becoming intricately, how to integrate mass data sources and how to find latent information from the integrated data is a matter of urgency. At present, industry tends to adopt distributed computing model to solve the integration of massive data. Aiming at getting the valuable and in-depth information, visualization is a critical step in data analysis and data mining. We design a tool called MRData for heterogeneous data integration which has two features: 1) parallel data processing based on Hadoop which is a distributed platform; 2) visual analysis. And at last, experiments verify the efficiency of MRData.

Urban pipe network pipelines are criss-crossing and have 3D (three-dimension) distribution. Plenty of attribute information and spatial data are involved. In order to display it in a more intuitive way, UPNVS (urban pipe network visualization system) is designed. The thematic maps service is a part of it. In this paper, we propose the thematic maps service, upload and edit data service, information attention service and 3D viewing service. While 3D display service concentrates on displaying simple map and unique value map. It implements the transit from 2D thematic maps to 3D and reflects the different property values of pipelines with different colors. This application enhances thematic maps service lively and visibility. Finally, we plug it to our UPNVS which can show the patterns and trends easily and provide the information for decision-making. At the same time, the results can also be applied to other visualization system and 3D thematic maps.

This paper studies the functional services of the 3D visualization of urban pipe network (UPN). In accordance with the characteristics of the UPN and its demands for 3D visualization. The division of the visualization services of the 3D visualization of UPN is researched, then a structure of services division named service stack is propose in this paper, which is applicable to the 3D visualization of UPN. The service stack provides a unified standard for the development of the 3D visualization of UPN, simplifies the process of the system development, and provides the system with good compatibility, scalability and reusability. Key technologies about the service stack are also discussed in this paper. Finally, based on the service stack, a prototype of urban pipe network 3D visualization system is implemented.

Since planar graphics can not meet the needs of precisely reflecting real spatial relationship between pipelines, it is necessary to build a 3D visualization system of urban pipe network. 3D spatial data model is the basis of data representation and spatial visualization. Based on the analysis of the features of urban pipe network and the requirements of the 3D model, a spatial data model for Urban Pipe Network Visualization System (UPNVS) is designed and organized by spatial metadata. The advantage of the spatial metadata is discussed in the paper. The application of the model in a real project shows that it is effective.

Efficient enumeration of all maximal cliques in a given graph has many applications in the filed of Graph Theory, Social Network Analysis, Bioinformatics and etc. Recent researches indicate that many networks in our world are complex networks involving massive data. Being as the complete sub-graph, a maximal clique can represent a group of friends who all hang around together. It can also be used to find common sub-topologies in a set of protein structures. However, the large scale of real networks and the exponentially increasing computation time of the clique enumeration problem make most of the existing algorithms unsuitable in the real-world scenarios. Therefore, we present a parallel algorithm Peamc (Parallel Enumeration of All Maximal Cliques) which exploits several new and effective techniques to enumerate all maximal cliques in large-scale complex networks. Experimental results on true-life networks with up to 20 million vertices and 50 million edges show that Peamc can find all the maximal cliques with high efficiency and scalability.

Researchers in the field of data mining now confront a common problem that data mining tasks are time-consuming in that these tasks have to process large-scale datasets. Grid computing focuses on integrating distributed, heterogeneous and idle computers from the Internet to be a service system with high performance. Thus, it is possible to take advantage of grid computing to provide high performance computation capability to effectively reduce task durations. Here, we have successfully developed DMGrid, a grid handling data mining applications. In DMGrid, it not only considers efficient parallel computing as a crucial aspect, but also takes into account dynamic resource configuration. Unlike many existing data mining grids, DMGrid also provides an engine to execute the algorithm flow specified in an application. Moreover, it offers application execution monitoring. At last, we perform experiments and design two applications: Customer Churning Analysis and Customer Value Analysis through which the feasibility of DMGrid is validated.

In the field of data mining, most of necessary resources, such as datasets, computing resources, might be distributed in different places of the world, which impedes scientists to do research successfully. Service science advocates every available resource as a service and focuses on their collaboration to settle a problem. Here, we propose a service-based architecture for data mining applications, including configuration service, service engine, monitor service, analysis service, visualization service, computing service and data & algorithm provision service. The first 5 services focus on 5 specific aspects of data mining requirements. A demo system has also been developed to validate the feasibility of our architecture.