Problem
The unabated growth of the Web has resulted in a situation in which more information is available to more people than ever in human history. Along with this unprecedented growth has come the inevitable problem of information overload. To counteract this information overload, users typically rely on search engines (like Google and AllTheWeb) or on manually-created categorization hierarchies (like Yahoo! and the Open Directory Project). Though excellent for accessing Web pages on the so-called "crawlable" web, these approaches overlook a much more massive and high-quality resource: the Deep Web. 数据挖掘实验室
The Deep Web (or Hidden Web) comprises all information that resides in autonomous databases behind portals and information providers' web front-ends. Web pages in the Deep Web are dynamically-generated in response to a query through a web site's search form and often contain rich content. A recent study has estimated the size of the Deep Web to be more than 500 billion pages, whereas the size of the "crawlable" web is only 1% of the Deep Web (i.e., less than 5 billion pages). Even those web sites with some static links that are "crawlable" by a search engine often have much more information available only through a query interface. Unlocking this vast deep web content presents a major research challenge. 数据挖掘研究院
