Get an random newspaper article from Trove

Changes to the Trove API mean that the techniques I've previously used to select resources at random will no longer work. This notebook provides one alternative.

I wanted something that would work efficiently, but would also expose as much of the content as possible. Applying multiple facets together with a randomly-generated query seems to do a good job of getting the result set below 100 (the maximum available from a single API call). This should mean that most of the newspaper articles are reachable, but it's a bit hard to quantify.

Thanks to Mitchell Harrop for suggesting I could use randomly selected stopwords as queries. I've supplemented the stopwords with letters and digits, and together they seem to do a good job of applying an initial filter and mixing up the relevance ranking.

As you can see from the examples below, you can supply any of the facets available in the newspapers zone – for example: state, title, year, illType, category.

In [1]:
import random
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import json

s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('https://', HTTPAdapter(max_retries=retries))
s.mount('http://', HTTPAdapter(max_retries=retries))

with open('stopwords.json', 'r') as json_file:
    STOPWORDS = json.load(json_file)
In [2]:
API_KEY = 'YOUR API KEY'
API_URL = 'http://api.trove.nla.gov.au/v2/result'
In [3]:
def get_random_facet_value(params, facet):
    '''
    Get values for the supplied facet and choose one at random.
    '''
    these_params = params.copy()
    these_params['facet'] = facet
    response = s.get(API_URL, params=these_params)
    data = response.json()
    try:
        values = [t['search'] for t in data['response']['zone'][0]['facets']['facet']['term']]
    except TypeError:
        return None
    return random.choice(values)

    
def get_total_results(params):
    response = s.get(API_URL, params=params)
    data = response.json()
    total = int(data['response']['zone'][0]['records']['total'])
    return total


def get_random_article(query=None, **kwargs):
    '''
    Get a random article.
    The kwargs can be any of the available facets, such as 'state', 'title', 'illtype', 'year'.
    '''
    total = 0
    applied_facets = []
    facets = ['month', 'year', 'decade', 'word', 'illustrated', 'category', 'title']
    tries = 0
    params = {
        'zone': 'newspaper',
        'encoding': 'json',
        # Note that keeping n at 0 until we've filtered the result set speeds things up considerably
        'n': '0',
        # Uncomment these if you need more than the basic data
        #'reclevel': 'full',
        #'include': 'articleText',
        'key': API_KEY
    }
    if query:
        params['q'] = query
    # If there's no query supplied then use a random stopword to mix up the results
    else:
        random_word = random.choice(STOPWORDS)
        params['q'] = f'"{random_word}"'
    # Apply any supplied factes
    for key, value in kwargs.items():
        params[f'l-{key}'] = value
        applied_facets.append(key)
    # Remove any facets that have already been applied from the list of available facets
    facets[:] = [f for f in facets if f not in applied_facets]
    total = get_total_results(params)
    # If our randomly selected stopword has produced no results
    # keep trying with new queries until we get some (give up after 10 tries)
    while total == 0 and tries <= 10:
        if not query:
            random_word = random.choice(STOPWORDS)
            params['q'] = f'"{random_word}"'
        tries += 1
    # Apply facets one at a time until we have less than 100 results, or we run out of facets
    while total > 100 and len(facets) > 0:
        # Get the next facet
        facet = facets.pop()
        # Set the facet to a randomly selected value
        params[f'l-{facet}'] = get_random_facet_value(params, facet)
        total = get_total_results(params)
        #print(total)
        #print(response.url)
    # If we've ended up with some results, then select one (of the first 100) at random
    if total > 0:
        params['n'] = '100'
        response = s.get(API_URL, params=params)
        data = response.json()
        article = random.choice(data['response']['zone'][0]['records']['article'])
        return article

Get any old article...

In [4]:
get_random_article()
Out[4]:
{'id': '166455502',
 'url': '/newspaper/166455502',
 'heading': 'THE SOCIETY OF ARTS EXHIBITION. A GLANCE AT THE PICTURES.',
 'category': 'Detailed lists, results, guides',
 'title': {'id': '824',
  'value': 'Quiz and the Lantern (Adelaide, SA : 1890 - 1900)'},
 'date': '1896-07-02',
 'page': 13,
 'pageSequence': 13,
 'relevance': {'score': '0.22012922', 'value': 'may have relevance'},
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/166455502?searchTerm=%22where%22'}

Get a random article about pademelons

In [5]:
get_random_article(query='pademelon')
Out[5]:
{'id': '155441736',
 'url': '/newspaper/155441736',
 'heading': 'UNDER THE SOUTHERN CROSS I.—A LIVELY PRIMITIVE: A MISSION ON THE MACLEAY RIVER.',
 'category': 'Article',
 'title': {'id': '647', 'value': 'The Methodist (Sydney, NSW : 1892 - 1954)'},
 'date': '1915-02-13',
 'page': 3,
 'pageSequence': 3,
 'relevance': {'score': '10.578832', 'value': 'very relevant'},
 'snippet': 'The Macleay River is situated about 250 miles to the north of Newcastle. The river is one of the most important between the Hunter and the Brisbane. One of our devoted local preachers from',
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/155441736?searchTerm=pademelon'}

Get a random article from Tasmania

In [6]:
get_random_article(state='Tasmania')
Out[6]:
{'id': '169044787',
 'url': '/newspaper/169044787',
 'heading': 'SUPREME COURT SITTING IN BANCO. FRIDAY, 11th MAY, 1866. (Before their Honor Sir Valentine Fleming, Knight, Chief Justice, and Sir F. Smith, Knight, Puisne Judge.)',
 'category': 'Detailed lists, results, guides',
 'title': {'id': '865',
  'value': 'Tasmanian Morning Herald (Hobart, Tas. : 1865 - 1866)'},
 'date': '1866-05-12',
 'page': 2,
 'pageSequence': 2,
 'relevance': {'score': '0.47627062', 'value': 'likely to be relevant'},
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/169044787?searchTerm=%22how%22'}

Get a random article from the Sydney Morning Herald

In [7]:
get_random_article(title='35', category='Article')
Out[7]:
{'id': '28366302',
 'url': '/newspaper/28366302',
 'heading': 'LONDON TELEGRAMS TO THE AMERICAN PRESS.',
 'category': 'Article',
 'title': {'id': '35',
  'value': 'The Sydney Morning Herald (NSW : 1842 - 1954)'},
 'date': '1885-04-10',
 'page': 5,
 'pageSequence': 5,
 'relevance': {'score': '1.1776975', 'value': 'likely to be relevant'},
 'snippet': 'London, February 27.—The Russian newspaper Svet, the organ of General Komaroff, asserts that there is a strong party at Herat who desire that Russian protection be extended over the city. The Journal de St. Petersburg',
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/28366302?searchTerm=%22ain%22'}

Get a random illustrated article

In [8]:
get_random_article(illustrated='true')
Out[8]:
{'id': '99665791',
 'url': '/newspaper/99665791',
 'heading': "IN A MENAGERIE. Stories of the Animals' Cleverness.",
 'category': 'Article',
 'title': {'id': '414',
  'value': 'Argyle Liberal and District Recorder (NSW : 1903 - 1930)'},
 'date': '1907-12-20',
 'page': 3,
 'pageSequence': 3,
 'relevance': {'score': '5.633662', 'value': 'very relevant'},
 'snippet': '"That elephant there," said George, the veteran head of a large menagerie, "hasn\'t got any more morals than—well, she\'s just naturally bad."',
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/99665791?searchTerm=%22doesn%22'}

Get a random illustrated advertisement from the Australian Womens Weekly

In [9]:
get_random_article(title='112', illustrated='true', category='Advertising')
Out[9]:
{'id': '53250867',
 'url': '/newspaper/53250867',
 'heading': 'Advertising',
 'category': 'Advertising',
 'title': {'id': '112',
  'value': "The Australian Women's Weekly (1933 - 1982)"},
 'date': '1980-06-18',
 'page': 101,
 'pageSequence': 101,
 'relevance': {'score': '0.24747185', 'value': 'may have relevance'},
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/53250867?searchTerm=%22had%22'}

Get a random cartoon

In [10]:
get_random_article(illtype='Cartoon')
Out[10]:
{'id': '75198942',
 'url': '/newspaper/75198942',
 'heading': "Don't Tell Auntie An Hilarious Christmas-time Episode",
 'category': 'Article',
 'title': {'id': '251', 'value': 'Sunshine Advocate (Vic. : 1924 - 1954)'},
 'date': '1937-12-17',
 'page': 1,
 'pageSequence': '1 S',
 'relevance': {'score': '1.9126042', 'value': 'likely to be relevant'},
 'snippet': "George Cunningham was not carrying his auntie's luggage; he was only directing the efforts of all the porters who were. At the head of the",
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/75198942?searchTerm=%22hadn%22'}

Get a random article from 1930

In [11]:
get_random_article(year='1930')
Out[11]:
{'id': '113686939',
 'url': '/newspaper/113686939',
 'heading': 'DID BUILDER ERR? President Townsend Outspoken',
 'category': 'Article',
 'title': {'id': '410',
  'value': 'Gilgandra Weekly and Castlereagh (NSW : 1929 - 1942)'},
 'date': '1930-11-20',
 'page': 7,
 'pageSequence': 7,
 'relevance': {'score': '0.9030686', 'value': 'likely to be relevant'},
 'snippet': '"In this case it appears that the Council has been deliberately flouted," declared the President (Cr. E. V. Townsend) at',
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/113686939?searchTerm=%22not%22'}

Get a random article tagged 'poem'

In [12]:
get_random_article(publictag='poem')
Out[12]:
{'id': '4759567',
 'url': '/newspaper/4759567',
 'heading': "MIDNIGHT LAMENTATION OF HEKI'S WIFE.",
 'category': 'Article',
 'title': {'id': '18', 'value': 'The Melbourne Argus (Vic. : 1846 - 1848)'},
 'date': '1846-07-10',
 'page': 4,
 'pageSequence': 4,
 'relevance': {'score': '0.59091616', 'value': 'likely to be relevant'},
 'snippet': 'WRITTEN ON READING, IN THE MELBOURNE ARGUS OF TUESDAY, 30TH JUNE, AN ACCOUNT OF A VISIT TO THE NEW ZEALAND CHIEFS, HEKI AND KAWITI.',
 'troveUrl': 'https://trove.nla.gov.au/ndp/del/article/4759567?searchTerm=%22g%22'}

Speed test

In [13]:
%%timeit
get_random_article()
2.49 s ± 742 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)