Målfrid 2021 - Freely Available Documents from Norwegian State Institutions
- Datasets
- Public access
Publicly available to everyone. Access may still require registration and an API key request, as long as anyone can request such registration and/or API keys.
Description
This corpus consists of documents from 339 internet domains run by Norwegian state institutions, and comprises approximately 4.1 billion tokens (words and punctuation) in total, which makes it one of the largest freely available text resources for Norwegian Bokmål and Nynorsk. In addition to Norwegian, the corpus contains texts in Northern Sami, Lule Sami, Southern Sami and English.
The data were collected as part of the so-called Målfrid project, where the National Library of Norway on behalf of the Ministry of Culture and in collaboration with the The Language Council of Norway collects and aggregates data for mapping the usage of Norwegian Bokmål and Norwegian Nynorsk in Norwegian state institutions.
The corpus is the result of a focused crawl conducted between December 11th 2020 and January 18th 2021, recursively downloading text documents (HTML, DOC(X)/ODT and PDF) from a set of domains (down to and including level 12), while obeying robots.txt and politeness restrictions.
The crawled documents were further processed according to their format: text was extracted from HTML using the boilerplate removal system Justext (http://corpus.tools/wiki/Justext), from Word/ODT documents using Textract (https://textract.readthedocs.io/en/stable/) and from PDFs using Google Cloud Vision OCR.
The extracted text was classified using TextCat language identification (cf. https://www.let.rug.nl/~vannoord/TextCat/) at document level, provided as part of the metadata. The documents were deduplicated on domain level (exact duplicates).
The corpus is provided as gzipped JSON lines (jsonl), one document per line. There is one JSONL file per combination of domain, language and content type. The files are encoded as UTF-8, with ASCII escape sequences. Each document contains the following keys:
- lang: language of the document (detected using TextCat)
- url: the url of the document at crawl time
- date: crawl date
- mimetype: media type of the document (simplified): HTML, DOC or PDF
- fulltext: an array of strings, where each string represents one paragraph. An empty string denotes a new page in the PDF documents
Distributions1
Nameless distribution- gtar
- gtar
- Description:
- Not provided
- Access URL:
- https://hdl.handle.net/21.11146/69
- Status:
- Not provided
- Direct download:
- API:
- Not provided
- Documentation:
- Not provided
- License:
- Conforms to:
- Not provided
- Rights for use:
- Not provided
APIs providing this dataset0
Similar datasets
| Norwegian Parliamentary Speech Corpus 2.0 | Public access | |
| Norwegian Conversation Speech Corpus | Public access | |
| Translation memory from Nynorsk News Press Agency 2022 | Public access | |
| Norwegian Anaphora Resolution Corpus | Public access | |
| bokselskap.no 2023 | Public access |
Distributions1
Nameless distribution- gtar
- gtar
- Description:
- Not provided
- Access URL:
- https://hdl.handle.net/21.11146/69
- Status:
- Not provided
- Direct download:
- API:
- Not provided
- Documentation:
- Not provided
- License:
- Conforms to:
- Not provided
- Rights for use:
- Not provided
APIs providing this dataset0
Contact information
- Contact point:
- Not provided
- Website:
- https://www.nb.no/sprakbanken/
- Email:
- sprakbanken@nb.no
- Telephone:
- Not provided
About the data
- Language:
- Northern Sami, , Lule Sami, , , Southern Sami
- Content providers:
- Not provided
- Provenance:
- Not provided
- Update frequency:
- Not provided
- First issued:
This date indicates when the data in this dataset was first released. It may have happened before the dataset was published on data.norge.no.
- December 1, 2020
- Last updated:
- April 30, 2021
- Geographical scope:
- Not provided
- Temporal scope:
- Not provided
- Conforms to:
Reference to an implementation rule or other specification that forms the basis for the dataset.
- Not provided
- Quality annotations:
- Not provided
Applicable legislation
Concepts used in the dataset
References
Fees
About this dataset
- Publisher:
- Nasjonalbiblioteket
- Published:
This date indicates when the dataset was harvested by data.norge.no. It may have been available earlier elsewhere.
- May 2, 2022
- Last updated:
- March 13, 2026
- Landing page:
- Not provided
- Documentation:
- Not provided
- Dataset type:
- Not provided
- Metadata Quality:
Metadata quality is an indicator of how well the datasets are described using metadata.
- Good (59%)
- URI:
Themes
Keywords
Discussions on Datalandsbyen0
What is Datalandsbyen?
Datalandsbyen is our online forum where you can request data, share experiences, and ask for advice related to data sharing and information management.
Resource Description Framework (RDF)All URLs to resources on data.norge.no can provide RDF metadata in various formats, depending on the Accept header sent with the request.
All URLs to resources on data.norge.no can provide RDF metadata in various formats, depending on the Accept header sent with the request.