Thomas Drugeon’s research while affiliated with Institut national de l'audiovisuel and other places

Around the world, billions of people use social media like Twitter and Facebook every day, to find, discuss and share information. Social media, which has transformed people from content readers to publishers, is not only an important data source for researchers in social science but also a “must archive” object for web archivists for future generations. In recent years, various communities have discussed the need to archive social media and have debated the issues related to its archiving. There are different ways of archiving social media data, including using traditional web crawlers and application programming interfaces (APIs) or purchasing from official company firehoses. It is important to note that the first two methods bring some issues related to capturing the dynamic and volatile nature of social media, in addition to the severe restrictions of APIs. These issues have an impact on the completeness of collections and in some cases return only a sample of the whole. In this chapter, we present these different methods and discuss the challenges in detail, using Twitter as a case study to better understand social media archiving and its challenges, from gathering data to long-term preservation.

In this paper we present the technical approach we developed at the National Audiovisual Institute (INA) for the forthcoming extension of the French legal deposit law to web contents. This paper covers crawling and storage aspects of the project. The INA being in charge of the archiving of websites which are related to the media and AV (Audiovisual), focused crawling strategies are used to discover and subsequently archive relevant websites. A two-level crawling architecture is described, carrying on focused and continuous crawling policies using the web site as the unit. Thereby, issues raised by the harvest of the internals of a website and those of managing the global crawl are handled independently, allowing for finer control and better comprehensiveness in data archiving. The lower layer of the storage model used to archive and preserve web contents relies on a custom, distributed file system, using the same networking solutions as the crawling system. Specific issues led by streamed AV contents are taken into account in both crawling and storage parts.

Abstract (EN) It is now widely acknowledged,that the Web has given rise to radical new forms of contents and