Keywords

fuzzy set model, web information retrieval, copy detection, HTML document, odds ratio

Abstract

Web documents that are either partially or completely duplicated in content are easily found on the Internet these days. Not only do these documents create redundant information on the Web, which take longer to filter unique information and cause additional storage space, but also they degrade the efficiency of Web information retrieval. In this paper, we present a new approach for detecting similar Web documents, especially HTML documents. Our detection approach determines the odd ratio of any two documents, which makes use of the degrees of resemblance of the documents, and graphically displays the locations of similar (not necessarily the same) sentences detected in the documents after (i) eliminating non-representative words in the sentences using the stopword-removal and stemming algorithms, (ii) computing the degree of similarity of sentences using a fuzzy set information retrieval approach, and (iii) matching the corresponding hierarchical content of the two documents using a simple tree matching algorithm. The proposed method for detecting similar documents handles wide range of Web pages of varying size and does not require static word lists and thus applicable to different Web (especially HTML) documents in different subject areas, such as sports, news, science, etc.

Original Publication Citation

Rajiv Yerra and Yiu-Kai Ng, "Detecting Similar HTML Documents Using a Fuzzy Set Information Retrieval Approach." In Proceedings of IEEE International Conference on Granular Computing (GrC'5), pp. 693-699, July 25, Beijing, China.