上海交通大学学报(自然版) ›› 2018, Vol. 52 ›› Issue (10): 1363-1369.doi: 10.16183/j.cnki.jsjtu.2018.10.027

• 学报(中文) • 上一篇    下一篇

基于文本对象模型的自动化网页内容提取方法

李桐宇,任锐,蔡鸿明,姜丽红   

  1. 上海交通大学 软件学院, 上海 200240
  • 通讯作者: 蔡鸿明,男,教授,博士生导师,电话(Tel.):021- 34205153;E-mail: hmcai@sjtu.edu.cn.
  • 作者简介:李桐宇(1995-),男,江苏省睢宁县人, 硕士生,主要研究方向为网页内容及语义标签提取.
  • 基金资助:
    国家自然科学基金资助项目(61373030)

Automated Web Page Content Extraction Method Based on Document Object Model

LI Tongyu,REN Rui,CAI Hongming,JIANG Lihong   

  1. School of Software, Shanghai Jiao Tong University, Shanghai 200240, China

摘要: 网页内容提取在信息检索、文本分析以及网络资源数据处理等领域具有重要的工程与应用价值.针对网页中的大量无关内容及网页结构的异构性所造成的网页内容提取难题,提出一种基于文本对象模型(DOM)的自动化网页内容提取方法.首先,在节点过滤后,对网页的DOM模型进行压缩,便于后续分析处理;然后,提出基于文本-链接密度的内容提取方法来识别网页内容;最后,基于节点熵来识别并去除网页内容中的噪声链接.实验结果表明,相比于传统的网页内容提取方法,该方法的准确率和F1分数均有明显提升,而召回率仅有轻微下降.

关键词: 文本对象模型, 网页内容提取, 文本密度, 节点熵

Abstract: Web content extraction has great engineering and application value in the fields of information retrieval, text analysis and network resource data processing. In view of the problem of web content extraction caused by useless information on web pages and the heterogeneity of web page structures, this paper proposes an automated web page content extraction method based on Document Object Model (DOM). Firstly, for DOMs generated from original web pages, we remove useless nodes from them and then compress the models, which facilitates subsequent processing. Then, we identify the web page content based on text and hyperlink density. Finally, we identify the noise hyperlinks based on node entropy and remove them from the content. The experimental results show that compared with the traditional methods of web page content extraction, the accuracy and F1 score of our method are obviously improved while there is only a slight decline on recall.

Key words: document object model (DOM), content extraction of web pages, text density, node entropy

中图分类号: