1. digitalnz
  2. Harvest-data-from-PapersPast.ipynb

Harvest data from Papers Past

This notebooks lets you harvest large amounts of data for Papers Past (via DigitalNZ) for further analysis. It saves the results as a CSV file that you can open in any spreadsheet program. It currently includes the OCRd text of all the newspaper articles, but I might make this optional in the future — thoughts?

You can edit this notebook to harvest other collections in DigitalNZ — see the notes below for pointers. However, this is currently only saving a small subset of the available metadata, so you'd probably want to adjust the fields as well. Add an issue on GitHub if you need help creating a custom harvester.

There's only two things you have to change — you need to enter your API key, and supply a search term. There are additional options for limiting your search results.

If you haven't used one of these notebooks before, they're basically web pages in which you can write, edit, and run live code. They're meant to encourage experimentation, so don't feel nervous. Just try running a few cells and see what happens!.

Some tips:

  • Code cells have boxes around them.
  • To run a code cell click on the cell and then hit Shift+Enter. The Shift+Enter combo will also move you to the next cell, so it's a quick way to work through the notebook.
  • While a cell is running a * appears in the square brackets next to the cell. Once the cell has finished running the asterix will be replaced with a number.
  • In most cases you'll want to start from the top of notebook and work your way down running each cell in turn. Later cells might depend on the results of earlier ones.
  • To edit a code cell, just click on it and type stuff. Remember to run the cell once you've finished editing.

Add your API key

Go get yourself a DigitalNZ API key, then paste it between the quotes below. You need a key to make API requests, but they're free and quick to obtain.

In [ ]:
# Past your API key between the quotes
# You might need to trim off any spaces at the beginning and end
API_KEY = '[YOUR API KEY]'
print('Your API key is: {}'.format(API_KEY))

Setting things up

Just run these cells to set up some things that we'll need later on.

In [ ]:
# This cell just sets up some stuff that we'll need later

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import pandas as pd
from tqdm.auto import tqdm
import time
import re
from slugify import slugify
from time import strftime
from IPython.display import display, FileLink
from pathlib import Path

s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('https://', HTTPAdapter(max_retries=retries))

API_URL = 'http://api.digitalnz.org/v3/records.json'

Set up some code

This is where all the serious harvesting work gets done. You shouldn't need to change anything unless you want to harvest something other than Papers Past. Just run the cell.

In [ ]:
def process_articles(results):
    articles = []
    for result in results:
        # If you're harvesting something other than Papers Past, you'd probably 
        # want to change the way results are processed.
        title = re.sub(r'(\([^)]*\))[^(]*$', '', result['title']).strip()
        articles.append({
            'id': result['id'],
            'title': title,
            'newspaper': result['publisher'][0],
            'date': result['date'][0][:10],
            'text': result['fulltext'],
            'paperspast_url': result['landing_url'],
            'source_url': result['source_url']
        })
    return articles

def get_total(params):
    np = params.copy()
    np['per_page'] = 0
    data = get_records(np)
    return data['search']['result_count']
    
def get_records(params):
    response = requests.get(API_URL, params=params)
    return response.json()

def harvest(params):
    '''
    Do the harvesting!
    '''
    more = True
    articles = []
    params['page'] = 1
    total = get_total(params)
    with tqdm(total=total) as pbar:
        while more:
            data = get_records(params)
            results = data['search']['results']
            if results:
                articles += process_articles(data['search']['results'])
                pbar.update(len(results))
                params['page'] += 1
                time.sleep(0.2)
            else:
                more = False 
    return articles

def start_harvest(query, start_year=None, end_year=None, **kwargs):
    '''
    Initiates a harvest.
    If you've specified start and end years it'll loop over them getting results for each.
    '''
    params = {
        'text': query,
        'and[primary_collection][]': 'Papers Past',
        'per_page': '100',
        'api_key': API_KEY
    }
    for key, value in kwargs.items():
        params[f'and[{key}][]'] = value
    if start_year and end_year:
        articles = []
        for year in tqdm(range(start_year, end_year+1), desc='Years'):
            current_year = year
            params['and[year][]'] = year
            articles += harvest(params)
    else:
        articles = harvest(params)
    return articles

def save_as_csv(articles, query_name):
    '''
    Save the results as a CSV file.
    Filename is constructed from the the supplied query_name and the current date/time.
    Displays a download link when finished.
    '''
    Path('data').mkdir(exist_ok=True)
    filename = f'{slugify(query_name)}-{strftime("%Y%m%d%H%M%S")}.csv'
    df = pd.DataFrame(articles)
    df.to_csv(Path('data', filename), index=False)
    display(FileLink(Path('data', filename)))

Harvest search results

At the very least, the harvesting code is expecting you to supply a search query, like 'possum'. You just feed this query to the start_harvest() function to kick things off.

In [ ]:
articles = start_harvest('possum')

Now we can save the results as a CSV file.

In [ ]:
save_as_csv(articles, 'possum')

Harvesting articles within a range of years

There's no direct way to search for a range of years, but we can get around this by issuing a request for each year separately and then combining the results. Just give start_harvest() a start_year and end_year value.

In [ ]:
articles = start_harvest('possum', start_year=1905, end_year=1910)
In [ ]:
save_as_csv(articles, 'possum-1905-1910')

Harvesting articles from a particular newspaper

You can add additional facets to the start_harvest() function to limit the results further. For example, you can use the collection facet to specify a particular newspaper:

  • collection='Evening Post'

Other facets you could use include:

  • century='1800'
  • decade='1850'
In [ ]:
articles = start_harvest('possum', collection='Evening Post')
In [ ]:
save_as_csv(articles, 'possum-evening-post')

Created by Tim Sherratt for the GLAM Workbench.