Fiche du document
22 juillet 2013
- handle: 10670/1.xf2zy8
- hal: hal-01091602
info:eu-repo/semantics/OpenAccess
Mots-clés
Corpus Construction Web Crawling Quality Assessment ACM: I.: Computing Methodologies/I.7: DOCUMENT AND TEXT PROCESSING/I.7.5: Document Capture ACM: H.: Information Systems/H.3: INFORMATION STORAGE AND RETRIEVAL/H.3.1: Content Analysis and Indexing/H.3.1.3: Linguistic processing ACM: I.: Computing Methodologies/I.2: ARTIFICIAL INTELLIGENCE/I.2.7: Natural Language Processing/I.2.7.6: Text analysisSujets proches
Construction Architectural engineering Construction Buildings--Design and construction Engineering, Architectural Western architecture (Western countries) Building design Construction Buildings--Design and construction Architecture, Western (Western countries)Citer ce document
Roland Schäfer et al., « The Good, the Bad, and the Hazy: Design Decisions in Web Corpus Construction », HAL-SHS : linguistique, ID : 10670/1.xf2zy8
Métriques
Partage / Export
Résumé
In this paper, we examine notions of text quality in the context of web corpus construction. Web documents often contain material which disqualifies them from inclusion in a corpus (tag clouds, lists of names or nouns, etc.). First, we look at the agreement between coders (especially corpus designers) given the task of rating text quality. Then, we evaluate a simple and fully unsupervised method of text quality assessment based on short and very frequent words. Finally, we describe our general approach to the construction of carefully cleansed and non-destructively normalized web corpora. Under this approach, we annotate documents with quality metrics instead of actually removing those documents classified as being of low quality.