Fetch data via OAI-PMH

About OAI-PMH

OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting) is an international standard for retrieving descriptions of resources from a catalogue. It is widely used for harvesting between catalogues and data portals. The standard is described in OAI-PMH version 2.0.

What you get

Each record in the extract is described in RDF and delivered in RDF/XML format (selected with metadataPrefix=rdfxml in step 2). The descriptions follow a standard for each resource type, for example DCAT-AP-NO for datasets and SKOS-AP-NO-Begrep for concepts.

How it works

The OAI-PMH endpoint delivers the extract as a list of records. Each record describes one resource, for example one dataset or one concept, depending on which extract you fetch. Large lists are split across multiple pages. Each time you open an address you get one page with up to 50 records. You fetch the next page by changing the address. Every page except the last ends with a continuation key (resumptionToken) that you use to fetch the next page.

Step 1: Choose an extract

Find the ID of the extract you want to fetch. These extracts are available today:

An up-to-date overview with the number of records in each extract is available in the API documentation.

Step 2: Fetch the first page

Paste this address into the address bar. Replace <ID> with the ID of the extract you chose in step 1:

Copy
https://resource.api.fellesdatakatalog.digdir.no/v1/union-graphs/<ID>/oai-pmh?verb=ListRecords&metadataPrefix=rdfxml

You get XML with the first 50 records. Save the page with Ctrl+S (Mac: Cmd+S) if you want to keep it.

Step 3: Find the continuation key

Press Ctrl+F (Mac: Cmd+F) and search for resumptionToken. At the bottom of the page you will find a line similar to this. Example for an extract with 81 records:

Copy
<resumptionToken completeListSize="81" cursor="0"><ID>:rdfxml:50</resumptionToken>

completeListSize shows how many records the extract contains in total. cursor shows how many records came before this page. On the first page it is 0. If it says cursor="50", the page starts at record number 51. The text between the tags is the key to the next page.

If you do not find resumptionToken, the entire extract fit on one page, and you are done. You can then find the number of records by searching for <record> with Ctrl+F (Mac: Cmd+F). The number of matches is the number of records on the page.

Step 4: Fetch the next page

Replace everything after ? in the address with verb=ListRecords&resumptionToken= followed by the key you found:

Copy
https://resource.api.fellesdatakatalog.digdir.no/v1/union-graphs/<ID>/oai-pmh?verb=ListRecords&resumptionToken=<ID>:rdfxml:50

Note: metadataPrefix=rdfxml must not be included on this and the following pages. If you include it, you get the error message badArgument. The key remembers which format you chose in step 2.

Step 5: Repeat until you are done

Repeat steps 3 and 4 for each new page. The key only changes in the number at the end, which increases by 50 for each page (:50, :100, :150, and so on). You can therefore also fetch the next page by increasing the number yourself.

When resumptionToken is empty, you have fetched all records. Example for an extract with 81 records:

Copy
<resumptionToken completeListSize="81" cursor="50"/>

In the example, 50 records were delivered earlier, so the last page shows records 51 to 81 of 81.

If you increase the number past the end of the list, you get the error message badResumptionToken. Then you have also fetched everything.

Tips

• Firefox and Chrome display XML as a readable tree. If the page looks empty or only shows text, try Ctrl+U (Mac: Cmd+Option+U) to view the source code.
• The colons in the key usually work without change. If you get an error, try replacing : with %3A.
• If you only want to see which records exist, you can use verb=ListIdentifiers instead of verb=ListRecords in step 2. The pages will be much smaller, and the procedure is the same.