Chapter
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

The Dark Web, a part of the Deep Web that consists of several darknets (e.g. Tor, I2P, and Freenet), provides users with the opportunity of hiding their identity when surfing or publishing information. This anonymity facilitates the communication of sensitive data for legitimate purposes, but also provides the ideal environment for transferring information, goods, and services with potentially illegal intentions. Therefore, Law Enforcement Agencies (LEAs) are very much interested in gathering OSINT on the Dark Web that would allow them to successfully prosecute individuals involved in criminal and terrorist activities. To this end, LEAs need appropriate technologies that would allow them to discover darknet sites that facilitate such activities and identify the users involved. This chapter presents current efforts in this direction by first providing an overview of the most prevalent darknets, their underlying technologies, their size, and the type of information they contain. This is followed by a discussion of the LEAs’ perspective on OSINT on the Dark Web and the challenges they face towards discovering and de-anonymizing such information and by a review of the currently available techniques to this end. Finally, a case study on discovering terrorist-related information, such as home made explosive recipes, on the Dark Web is presented.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... The great part of the WWW contents are hosted on the so-called Deep Web [1]. These contents are not accessible by search engines mainly due to they are authentication protected contents or pages that are only reachable through the well known darknets [2], [3]. To browse through darknets websites a special authorization or specific software and configurations are needed. ...
... They are distributed in different geographical places running in different computation clusters too. On the one hand, 7 of the VMs are deployed on the facilities of the University of Granada (Spain): 6 named as i2pProjectM [1][2][3][4][5][6] Every VM executes an I2P router instance for accessing the I2P darknet, except for the i2pProjectBBDD machine. Machines i2pProjectM [1][2][3][4][5][6] were set up as floodfill I2P routers, while the rest i2pProjectM[7-10] run as normal I2P routers. ...
... On the one hand, 7 of the VMs are deployed on the facilities of the University of Granada (Spain): 6 named as i2pProjectM [1][2][3][4][5][6] Every VM executes an I2P router instance for accessing the I2P darknet, except for the i2pProjectBBDD machine. Machines i2pProjectM [1][2][3][4][5][6] were set up as floodfill I2P routers, while the rest i2pProjectM[7-10] run as normal I2P routers. Floodfill routers, as mentioned in Section III-A, allow to discover additional eepsites since they all are in charge of maintaining and managing the NetDB. ...
Preprint
Full-text available
Web is a primary and essential service to share information among users and organizations at present all over the world. Despite the current significance of such a kind of traffic on the Internet, the so-called Surface Web traffic has been estimated in just about 5% of the total. The rest of the volume of this type of traffic corresponds to the portion of Web known as Deep Web. These contents are not accessible by search engines because they are authentication protected contents or pages that are only reachable through the well known as darknets. To browse through darknets websites special authorization or specific software and configurations are needed. Despite TOR is the most used darknet nowadays, there are other alternatives such as I2P or Freenet, which offer different features for end users. In this work, we perform an analysis of the connectivity of websites in the I2P network (named eepsites) aimed to discover if different patterns and relationships from those used in legacy web are followed in I2P, and also to get insights about its dimension and structure. For that, a novel tool is specifically developed by the authors and deployed on a distributed scenario. Main results conclude the decentralized nature of the I2P network, where there is a structural part of interconnected eepsites while other several nodes are isolated probably due to their intermittent presence in the network.
... There is currently a large volume of worthwhile open source data to be analyzed, correlated and linked [32]. This includes social networks, public government documents and reports, online multimedia content, newspapers and even the Deep web and the Dark web [33], among others. Actually, both the Deep Web and the Dark Web (the latter circumscribed within the former) contain even more information than the Surface Web (i.e., the Internet known by most users) [34]. ...
Article
Full-text available
The amount of data generated by the current interconnected world is immeasurable, and a large part of such data is publicly available, which means that it is accessible by any user, at any time, from anywhere in the Internet. In this respect, Open Source Intelligence (OSINT) is a type of intelligence that actually benefits from that open natureby collecting, processing and correlating points of the whole cyberspace to generate knowledge. In fact, recent advances in technology are causing OSINT to currently evolve at a dizzying rate, providing innovative data-driven and AI-powered applications for politics, economy or society, but also offering new lines of action against cyberthreats and cybercrime. The paper at hand describes the current state of OSINT and makes a comprehensive review of the paradigm, focusing on the services and techniques enhancing the cybersecurity field. On the one hand, we analyze the strong points of this methodology and propose numerous ways to apply it to cybersecurity. On the other hand, we cover the limitations when adopting it. Considering there is a lot left to explore in this ample field, we also enumerate some open challenges to be addressed in the future. Additionally, we study the role of OSINT in the public sphere of governments, which constitute an ideal landscape to exploit open data.
... • Huge amount of worthwhile open source data to be analyzed, crossed and linked [23]. It includes social networks, public government documents and reports, online [13]. Both the Deep Web and the Dark Web (the latter circumscribed within the former) contain even more information than the Surface Web (the Internet known by most users). ...
Conference Paper
Full-text available
Phenomenons like Social Networks, Cloud Computing or Internet of Thing are unknowingly generating unimaginable quantities of data. In this context, Open Source Intelligence (OSINT) exploits such information to extract knowledge that is not easily appreciable beforehand by the human eye. Apart from the political, economic or social applications OSINT may bring, there are also serious global concerns that could be covered by this paradigm such as cyber crime and cyber threats. The paper at hand presents the current state of OSINT, the opportunities and limitations it poses, and the challenges to be faced in the future. Furthermore, we particularly study Spain as a potential beneficiary of this powerful methodology.
Article
The World Wide Web is the most widely used service on the Internet, although only a small part of it, the Surface Web, is indexed and accessible. The rest of the content, the Deep Web, is split between that unable to be indexed by usual search engines and content that needs to be accessed through specific methods and techniques. The latter is deployed in the so-called darknets, which have been the subject of much less study, where anonymity and privacy security services are preserved. Although there are several darknets, Tor is the most well-known and widely analyzed. Hence, the current work presents an analysis of web site connectivity, relationships and content of one of the less known and explored darknets: Freenet. Given the special features of this study, a new crawling tool, called c4darknet, was developed for the purpose of this work. This tool is, in turn, used in the experimentation that was carried out in a real distributed environment. Our results can be summarized as follows: there is great general availability of websites on Freenet; there are significant nodes within the network connectivity structure; and underage porn or child pornography is predominant among illegal content. Finally, the outcomes are compared against a similar study for the I2P darknet, showing special features and differences between both darknets.
Article
Web is a primary and essential service to share information among users and organizations at present all over the world. Despite the current significance of such a kind of traffic on the Internet, the so-called Surface Web traffic has been estimated in just about 5% of the total. The rest of the volume of this type of traffic corresponds to the portion of the Web known as Deep Web. These contents are not accessible by usual search engines because they are authentication protected contents or pages only reachable through technologies denoted as darknets. To browse through darknet websites, special authorization or specific software and configurations are needed. TOR is one of the most used darknet nowadays, but there are several other alternatives such as I2P or Freenet, which offer different features for end users. In this work, we perform a connectivity analysis of the websites in the I2P network (named eepsites) aimed to discover if different patterns and relationships from those used in legacy webs are followed in I2P, as well as to get insights about the dimension and structure of this darknet. For that, a novel tool is specifically developed by the authors and deployed on a distributed scenario. Main results conclude the decentralized nature of the I2P network, where there is a structural part of interconnected eepsites while other several nodes are isolated probably due to their intermittent presence in the network.
Article
Full-text available
In this paper, the authors examine the adequacy of the counter-terrorism concept, which does not envisage institutional responsibility for collecting, processing, and fixing traces of cyber-related terrorist activities. The starting point is the fact that today numerous human activities and communication take place in the cyberspace. Firstly, the focus is on the aspects of terrorism that present a generator of challenges to social stability and, in this context, the elements of the approach adopted by the current National Security Strategy of the Republic of Serbia. In this analysis, adequacy is evaluated from the point of view of functionality. In this sense, it is an attempt to present elements that influence the effectiveness of counter-terrorism in the information age. Related to this is the specification of the role that digital forensics can play in this area. The conclusion is that an effective counter-terrorism strategy must necessarily encompass the institutional incorporation of digital forensics since it alone can contribute to the timely detection or assertion of responsibility for terrorism in a networked computing environment.
Conference Paper
Full-text available
We perform a comprehensive measurement analysis of Silk Road, an anonymous, international online marketplace that operates as a Tor hidden service and uses Bitcoin as its exchange currency. We gather and analyze data over eight months between the end of 2011 and 2012, including daily crawls of the marketplace for nearly six months in 2012. We obtain a detailed picture of the type of goods sold on Silk Road, and of the revenues made both by sellers and Silk Road operators. Through examining over 24,400 separate items sold on the site, we show that Silk Road is overwhelmingly used as a market for controlled substances and narcotics, and that most items sold are available for less than three weeks. The majority of sellers disappears within roughly three months of their arrival, but a core of 112 sellers has been present throughout our measurement interval. We evaluate the total revenue made by all sellers, from public listings, to slightly over USD 1.2 million per month; this corresponds to about USD 92,000 per month in commissions for the Silk Road operators. We further show that the marketplace has been operating steadily, with daily sales and number of sellers overall increasing over our measurement interval. We discuss economic and policy implications of our analysis and results, including ethical considerations for future research in this area.
Conference Paper
Full-text available
Tor hidden services allow running Internet services while protecting the location of the servers. Their main purpose is to enable freedom of speech even in situations in which powerful adversaries try to suppress it. However, providing location privacy and client anonymity also makes Tor hidden services an attractive platform for every kind of imaginable shady service. The ease with which Tor hidden services can be set up has spurred a huge growth of anonymously provided Internet services of both types. In this paper we analyse the landscape of Tor hidden services. We have studied Tor hidden services after collecting 39824 hidden service descriptors on 4th of Feb 2013 by exploiting protocol and implementation flaws in Tor: we scanned them for open ports; in the case of HTTP services, we analysed and classified their content. We also estimated the popularity of hidden services by looking at the request rate for hidden service descriptors by clients. We found that while the content of Tor hidden services is rather varied, the most popular hidden services are related to botnets.
Article
Full-text available
Searching on the Internet today can be compared to dragging a net across the surfgace of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple. Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it.
Article
Full-text available
We perform a comprehensive measurement analysis of Silk Road, an anonymous, international online marketplace that operates as a Tor hidden service and uses Bitcoin as its exchange currency. We gather and analyze data over eight months between the end of 2011 and 2012, including daily crawls of the marketplace for nearly six months in 2012. We obtain a detailed picture of the type of goods being sold on Silk Road, and of the revenues made both by sellers and Silk Road operators. Through examining over 24,400 separate items sold on the site, we show that Silk Road is overwhelmingly used as a market for controlled substances and narcotics, and that most items sold are available for less than three weeks. The majority of sellers disappears within roughly three months of their arrival, but a core of 112 sellers has been present throughout our measurement interval. We evaluate the total revenue made by all sellers, from public listings, to slightly over USD 1.2 million per month; this corresponds to about USD 92,000 per month in commissions for the Silk Road operators. We further show that the marketplace has been operating steadily, with daily sales and number of sellers overall increasing over our measurement interval. We discuss economic and policy implications of our analysis and results, including ethical considerations for future research in this area.
Article
Full-text available
Freenet is a distributed, Internet-wide peer-to-peer overlay network de-signed to allow anonymized and censorship resistant publication and distribution of information. The system functions as a distributed hashtable, where participating com-puters store information which can be retrieved by any other participant. In this paper we describe a new architecture for the Freenet system, designed to strengthen the privacy of users, and to protect participating nodes from attack in situations where just running a node in the network could be a liability. While in previous incarnations of Freenet the edges of the overlay network were decided algo-rithmically in a manner intended to optimize routing in the network, in the new version we allow nodes to restrict their connections to nodes that they trust. In this type of network (sometimes called a Darknet, or a friend-to-friend network) every participant communicates directly only with his own trusted peers, and thus, under some assump-tions, reveals his identity only to peers he already trusts. While everybody has to trust somebody in the network, there is no central party whom everybody must trust. We offer participants to either stay hidden in a darknet, or to take part of a hybrid model with direct connections also to strangers. The novel algorithm we use for routing in the fixed mesh of trusted connections has been previously described in [29]. This paper focuses on the practical aspects of deploying a functioning data publication system based on the trusted connection model: we describe the system we have settled on and simulate it under realistic conditions.
Conference Paper
Full-text available
This paper analyzes the web browsing behaviour of Tor users. By collecting HTTP requests we show which websites are of interest to Tor users and we determined an upper bound on how vulnerable Tor users are to sophisticated de-anonymization attacks: up to 78 % of the Tor users do not use Tor as suggested by the Tor community, namely to browse the web with TorButton. They could thus fall victim to de- anonymization attacks by merely browsing the web. Around 1% of the requests could be used by an adversary for exploit piggybacking on vul- nerable le formats. Another 7 % of all requests were generated by social networking sites which leak plenty of sensitive and identifying informa- tion. Due to the design of HTTP and Tor, we argue that HTTPS is cur- rently the only eective countermeasure against de-anonymization and information leakage for HTTP over Tor.
Article
Full-text available
Topical crawling is a young and creative area of research that holds the promise of benefiting from several sophisticated data mining techniques. The use of classification algorithms to guide topical crawlers has been sporadically suggested in the literature. No systematic study, however, has been done on their relative merits. Using the lessons learned from our previous crawler evaluation studies, we experiment with multiple versions of different classification schemes. The crawling process is modeled as a parallel best-first search over a graph defined by the Web. The classifiers provide heuristics to the crawler thus biasing it towards certain portions of the Web graph. Our results show that Naive Bayes is a weak choice for guiding a topical crawler when compared with Support Vector Machine or Neural Network. Further, the weak performance of Naive Bayes can be partly explained by extreme skewness of posterior probabilities generated by it. We also observe that despite similar performances, different topical crawlers cover subspaces on the Web with low overlap.
Article
Full-text available
Anonymity systems such as Tor aim to enable users to communicate in a manner that is untraceable by adversaries that control a small number of machines. To provide efficient service to users, these anonymity systems make full use of forwarding capacity when sending traffic between intermediate relays. In this paper, we show that doing this leaks information about the set of Tor relays in a circuit (path). We present attacks that, with high confidence and based solely on throughput information, can (a) reduce the attacker's uncertainty about the bottleneck relay of any Tor circuit whose throughput can be observed, (b) exactly identify the guard relay(s) of a Tor user when circuit throughput can be observed over multiple connections, and (c) identify whether two concurrent TCP connections belong to the same Tor user, breaking unlinkability. Our attacks are stealthy, and cannot be readily detected by a user or by Tor relays. We validate our attacks using experiments over the live Tor network. We find that the attacker can substantially reduce the entropy of a bottleneck relay distribution of a Tor circuit whose throughput can be observed-the entropy gets reduced by a factor of 2 in the median case. Such information leaks from a single Tor circuit can be combined over multiple connections to exactly identify a user's guard relay(s). Finally, we are also able to link two connections from the same initiator with a crossover error rate of less than 1.5% in under 5 minutes. Our attacks are also more accurate and require fewer resources than previous attacks on Tor.
Article
Full-text available
We present Tor, a circuit-based low-latency anonymous communication service. This second-generation Onion Routing system addresses limitations in the original design by adding perfect forward secrecy, congestion control, directory servers, integrity checking, configurable exit policies, and a practical design for location-hidden services via rendezvous points. Tor works on the real-world Internet, requires no special privileges or kernel modifications, requires little synchronization or coordination between nodes, and provides a reasonable tradeoff between anonymity, usability, and efficiency. We briefly describe our experiences with an international network of more than 30 nodes. We close with a list of open problems in anonymous communication.
Article
Freenet is a popular peer to peer anonymous content-sharing network, with the objective to provide the anonymity of both content publishers and retrievers. Despite more than a decade of active development and deployment and the adoption of well-established cryptographic algorithms in Freenet, it remains unanswered how well the anonymity objective of the initial Freenet design has been met. In this paper we develop a traceback attack on Freenet, and show that the originating machine of a content request message in Freenet can be identified; that is, the anonymity of a content retriever can be broken, even if a single request message has been issued by the retriever. We present the design of the traceback attack, and perform both experimental and simulation studies to confirm the feasibility and effectiveness of the attack. For example, with randomly chosen content requesters (and random contents stored in the) Freenet testbed, Emulab-based experiments show that, for 24% to 43% of the content request messages, we can identify their originating machines. We also briefly discuss potential solutions to address the developed traceback attack. Despite being developed specifically on Freenet, the basic principles of the traceback attack and solutions have important security implications for similar anonymous content-sharing systems.
Conference Paper
This work investigates the effectiveness of a novel interactive search engine in the context of discovering and retrieving Web resources containing recipes for synthesizing Home Made Explosives (HMEs). The discovery of HME Web resources both on Surface and Dark Web is addressed as a domain-specific search problem; the architecture of the search engine is based on a hybrid infrastructure that combines two different approaches: (i) a Web crawler focused on the HME domain; (ii) the submission of HME domain-specific queries to general-purpose search engines. Both approaches are accompanied by a user-initiated post-processing classification for reducing the potential noise in the discovery results. The design of the application is built based on the distinctive nature of law enforcement agency user requirements, which dictate the interactive discovery and the accurate filtering of Web resources containing HME recipes. The experiments evaluating the effectiveness of our application demonstrate its satisfactory performance, which in turn indicates the significant potential of the adopted approaches on the HME domain.
Article
Encryption policy is becoming a crucial test of the values of liberal democracy in the twenty-first century.
Article
We created the Yahoo Flickr Creative Commons 100 Million Dataseta (YFCC100M) in 2014 as part of the Yahoo Webscope program, which is a reference library of interesting and scientifically useful datasets. The YFCC100M is the largest public multimedia collection ever released, with a total of 100 million media objects, of which approximately 99.2 million are photos and 0.8 million are videos, all uploaded to Flickr between 2004 and 2014 and published under a CC commercial or noncommercial license. The dataset is distributed through Amazon Web Services as a 12.5GB compressed archive containing only metadata. However, as with many datasets, the YFCC100M is constantly evolving; over time, we have released and will continue to release various expansion packs containing data not yet in the collection; for instance, the actual photos and videos, as well as several visual and aural features extracted from the data, have already been uploaded to the cloud, ensuring the dataset remains accessible and intact for years to come. The YFCC100M dataset overcomes many of the issues affecting existing multimedia datasets in terms of modalities, metadata, licensing, and, principally, volume.
Article
We lead a double life. We have relationships that enrich our lives, events we participate in, and places we go that become a part of who we are. We also have online friendships, which enrich our lives and make up part of who we are. There are points of intersection between our two lives: events we arrange online that take place offline, places that we have been (offline) that we photograph, tag, and share with our online social community. The amount of data being generated by people online, who share information about their offline lives, is unprecedented and growing. We are a lucky few. We are the "technorati": the people who have access to technology, and the knowledge to use it. We are wealthy, well-travelled, well-educated, and proud owners of a plethora of gadgets that allow us to weave a seamless tapestry between our online and offline lives. We are not most people. The data we generate does not represent most people.
Article
Communication privacy has been a growing concern, particularly with the Internet becoming a major hub of our daily interactions. Revelations of government tracking and corporate profiling have resulted in increasing interest in anonymous communication mechanisms. Several systems have been developed with the aim of preserving communication privacy via unlinkability within a public network environment such as Tor and I2P. As the anonymity networks cannot guarantee perfect anonymity, it is important for users to understand the risks they might face when utilizing such technologies. In this paper, we discuss potential attacks on the anonymity networks that can compromise user identities and communication links. We also summarize protection mechanisms against such attacks. Many attacks against anonymity networks are well studied, and most of the modern systems have built-in mechanisms to prevent these attacks. Additionally, some of the attacks require considerable resources to be effective and hence are very unlikely to succeed against modern anonymity networks.
Article
Instagram is a relatively new form of communication where users can instantly share their current status by taking pictures and tweaking them using filters. It has seen a rapid growth in the number of users as well as uploads since it was launched in October 2010. Inspite of the fact that it is the most popular photo sharing application, it has attracted relatively less attention from the web and social media research community. In this paper, we present a large-scale quantitative analysis on millions of users and pictures we crawled over 1 month from Instagram. Our analysis reveals several insights on Instagram which were never studied before: 1) its social network properties are quite different from other popular social media like Twitter and Flickr, 2) people typically post once a week, and 3) people like to share their locations with friends. To the best of our knowledge, this is the first in-depth analysis of user activities, demographics, social network structure and user-generated content on Instagram.
Conference Paper
Focussed crawlers enable the automatic discovery of Web resources about a given topic by automatically navigating the Web link structure and selecting the hyperlinks to follow by estimating their relevance to the topic based on evidence obtained from the already downloaded pages. This work proposes a classifier-guided focussed crawling approach that estimates the relevance of a hyperlink to an unvisited Web resource based on the combination of textual evidence representing its local context, namely the textual content appearing in its vicinity in the parent page, with visual evidence associated with its global context, namely the presence of images relevant to the topic within the parent page. The proposed focussed crawling approach is applied towards the discovery of environmental Web resources that provide air quality measurements and forecasts, since such measurements (and particularly the forecasts) are not only provided in textual form, but are also commonly encoded as multimedia, mainly in the form of heatmaps. Our evaluation experiments indicate the effectiveness of incorporating visual evidence in the link selection process applied by the focussed crawler over the use of textual features alone, particularly in conjunction with hyperlink exploration strategies that allow for the discovery of highly relevant pages that lie behind apparently irrelevant ones.
Conference Paper
Freenet is a popular peer to peer anonymous network, with the objective to provide the anonymity of both content publishers and retrievers. Despite more than a decade of active development and deployment and the adoption of well-established cryptographic algorithms in Freenet, it remains unanswered how well the anonymity objective of the initial Freenet design has been met. In this paper we develop a traceback attack on Freenet, and show that the originating machine of a content request message in Freenet can be identified; that is, the anonymity of a content retriever can be broken, even if a single request message has been issued by the retriever. We present the design of the traceback attack, and perform Emulab-based experiments to confirm the feasibility and effectiveness of the attack. With randomly chosen content requesters (and random contents stored in the Freenet testbed), the experiments show that, for 24% to 43% of the content request messages, we can identify their originating machines. We also briefly discuss potential solutions to address the developed traceback attack. Despite being developed specifically on Freenet, the basic principles of the traceback attack and solutions have important security implications for similar anonymous content sharing systems.
Conference Paper
This talk will review the emerging research in Terrorism Informatics based on a web mining perspective. Recent progress in the internationally renowned Dark Web project will be reviewed, including: deep/dark web spidering (web sites, forums, Youtube, virtual worlds), web metrics analysis, dark network analysis, web-based authorship analysis, and sentiment and affect analysis for terrorism tracking. In collaboration with selected international terrorism research centers and intelligence agencies, the Dark Web project has generated one of the largest databases in the world about extremist/terrorist-generated Internet contents (web sites, forums, blogs, and multimedia documents). Dark Web research has received significant international press coverage, including: Associated Press, USA Today, The Economist, NSF Press, Washington Post, Fox News, BBC, PBS, Business Week, Discover magazine, WIRED magazine, Government Computing Week, Second German TV (ZDF), Toronto Star, and Arizona Daily Star, among others. For more Dark Web project information, please see: http://ai.eller.arizona.edu/research/terror/ .
Article
This is a survey of the science and practice of web crawling. While at first glance web crawling may appear to be merely an application of breadth-first-search, the truth is that there are many challenges ranging from systems concerns such as managing very large data structures to theoretical questions such as how often to revisit evolving content sources. This survey outlines the fundamental challenges and describes the state-of-the-art models and solutions. It also highlights avenues for future work.
Article
We lead a double life. We have relationships that enrich our lives, events we participate in, and places we go that become a part of who we are. We also have online friendships, which enrich our lives and make up part of who we are. There are points of intersection between our two lives: events we arrange online that take place offline, places that we have been (offline) that we photograph, tag, and share with our online social community. The amount of data being generated by people online, who share information about their offline lives, is unprecedented and growing. We are a lucky few. We are the "technorati": the people who have access to technology, and the knowledge to use it. We are wealthy, well-travelled, well-educated, and proud owners of a plethora of gadgets that allow us to weave a seamless tapestry between our online and offline lives. We are not most people. The data we generate does not represent most people.
Article
THE PARADOX OF THE INVISIBLE WEB is that it's easy to understand why it exists, but it's very hard to actually define in concrete, specific terms. In a nutshell, the Invisible Web consists of content that's been excluded from general-purpose search engines and Web directories such as Lycos and LookSmart-and yes, even Google. There's nothing inherently "invisible" about this content. But since this content is not easily located with the information-seeking tools used by most Web users, it's effectively invisible because it's so difficult to find unless you know exactly where to look. In this paper, we define the Invisible Web and delve into the reasons search engines can't "see" its content. We also discuss the four different "types" of invisibility, ranging from the "opaque" Web which is relatively accessible to the searcher, to the truly invisible Web, which requires specialized finding aids to access effectively.
The Tor dark net.’ global commission on internet governance
  • G Owen
  • N Savage
PunkSPIDER, the crawler that scanned the Dark Web
  • P Paganini
Paganini P (2015) PunkSPIDER, the crawler that scanned the Dark Web. Retrieved 27 Jul 2016, from http://securityaffairs.co/wordpress/37632/hacking/punkspider-scanned-tor.html
Interactive discovery and retrieval of web resources containing home made explosive recipes International conference on human aspects of information security, privacy, and trust
  • G Kalpakis
  • T Tsikrika
  • C Iliou
  • T Mironidis
  • S Vrochidis
  • J Middleton
  • I Kompatsiaris
Web crawling: foundations and trends in information retrieval
  • C Olston
  • M Najork
Olston C, Najork M (2010) Web crawling: foundations and trends in information retrieval
White paper: the deep web: surfacing hidden value
  • J Bartlett
Bartlett J (2014) The Dark Net. Random House, London Bergman MK (2001) White paper: the deep web: surfacing hidden value. J Electron Pub 7(1)
The Dark Net. Random House
  • J Bartlett