Web content extraction has great engineering and application value in the fields of information retrieval, text analysis and network resource data processing. In view of the problem of web content extraction caused by useless information on web pages and the heterogeneity of web page structures, this paper proposes an automated web page content extraction method based on Document Object Model (DOM). Firstly, for DOMs generated from original web pages, we remove useless nodes from them and then compress the models, which facilitates subsequent processing. Then, we identify the web page content based on text and hyperlink density. Finally, we identify the noise hyperlinks based on node entropy and remove them from the content. The experimental results show that compared with the traditional methods of web page content extraction, the accuracy and F1 score of our method are obviously improved while there is only a slight decline on recall.
LI Tongyu,REN Rui,CAI Hongming,JIANG Lihong
. Automated Web Page Content Extraction Method Based on
Document Object Model[J]. Journal of Shanghai Jiaotong University, 2018
, 52(10)
: 1363
-1369
.
DOI: 10.16183/j.cnki.jsjtu.2018.10.027
［1］WENINGER T, PALACIOS R, CRESCENZI V, et al. Web content extraction: A meta-analysis of its past and thoughts on its future［J］. ACM SIGKDD Explorations Newsletter, 2016, 17(2): 17-23.
［2］BORGOLTE K, KRUEGEL C, VIGNA G. Relevant change detection: A framework for the precise extraction of modified and novel web-based content as a filtering technique for analysis engines［C］//Proceedings of the 23rd International Conference on World Wide Web. Seoul: ACM, 2014: 595-598.
［3］PETPRASIT W, JAIYEN S. E-commerce web page classification based on automatic content extraction［C］//2015 12th International Joint Conference on Computer Science and Software Engineering (JCSSE). Songkhla: IEEE, 2015: 74-77.
［4］KADAM V, DEVALE P R. A methodology for template extraction from heterogeneous web pages［J］. Indian Journal of Computer Science and Engineering (IJCSE), 2012, 3(3): 449-452.
［5］WU S, LIU J, FAN J. Automatic web content extraction by combination of learning and grouping［C］//Proceedings of the 24th international conference on World Wide Web. Switzerland: International World Wide Web Conferences Steering Committee, 2015: 1264-1274.
［6］KIM M, KIM Y, SONG W, et al. Main content extraction from Web documents using text block context［C］//International Conference on Database and Expert Systems Applications, Prague. Berlin, Heidelberg: Springer, 2013: 81-93.
［7］REIS D C, GOLGHER P B, SILVA A S, et al. Automatic web news extraction using tree edit distance［C］//Proceedings of the 13th international conference on World Wide Web. New York: ACM, 2004: 502-511.
［8］杨柳青, 李晓东, 耿光刚.基于布局相似性的网页正文内容提取研究［J］.计算机应用研究, 2015, 32(9): 2581-2586.
YANG Liuqing, LI Xiaodong, GENG Guanggang. Study of web pages content extraction based on layout similarity［J］. Application Research of Computers, 2015, 32(9): 2581-2586.
［9］CAI D, YU S, WEN J R, et al. VIPS: A vision-based page segmentation algorithm［R］. Beijing: Microsoft, 2003.
［10］WANG P, ZHOU M, YOU Y, et al. A new vision-based method for extracting academic information from conference Web pages［C］//IEEE 24th International Conference on Tools with Artificial Intelligence (ICTAI). Athens: IEEE, 2012: 976-981.
［11］SUN F, SONG D, LIAO L. Dom based content extraction via text density［C］//Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. Beijing: ACM, 2011: 245-254.
［12］FU L, MENG Y, XIA Y, et al. Web content extraction based on webpage layout analysis［C］//Second International Conference on Information Technology and Computer Science (ITCS). Kiev: IEEE, 2010: 40-43.
［13］WENINGER T, HSU W H. Text extraction from the web via text-to-tag ratio［C］//19th International Workshop on Database and Expert Systems Application (DEXA). Turin: IEEE, 2008: 23-28.
［14］ZHENG X, GU Y, LI Y. Data extraction from web pages based on structural-semantic entropy［C］//Proceedings of the 21st International Conference on World Wide Web. Lyon: ACM, 2012: 93-102.
［15］LIU Q, SHAO M, WU L, et al. Main content extraction from web pages based on node characteristics［J］. Journal of Computing Science and Engineering, 2017, 11(2): 39-48.
