I was keen to play around with Omeka-S – the new, Linked Open Data powered version of Omeka. In particular I wanted to understand its API, its use of JSON-LD, and how I could create and use relationships between different types of entities. I decided that one way of exploring these things was to build a pipeline that would make it easy to upload Trove newspaper articles to an Omeka site. I recently figured out how to generate nice thumbnails from newspaper articles, so I could add images as well. But what about the 'Linked' part of LOD? I thought I'd not only create records for the newspaper articles, I'd create records for each cited newspaper and link them to the articles. Things not strings!
Of course, this is not the only way to get Trove records into Omeka. You can save them to Zotero and then use the Zotero Import module to upload to Omeka. My first Omeka 'how to' (written 8 years ago!) talked about using the Zotero Import plugin to upload records from the National Archives of Australia. While the import module works well, I wanted the flexibility to generate new images on the fly, modify the metadata, and unflatten the Zotero records to build newspaper-to-article relationships. I also wanted to be able to get newspaper articles from other sources, such as Trove lists.
As I started playing around I realised I could rejig my thumbnail generator to get an image of the whole article. This is useful because Trove's article download options tend to slice up images in unpleasant ways. So I decided to upload the complete image (or images if the article is published across multiple pages) and let Omeka create the derivative versions.
In summary, you can use this notebook to:
But what newspaper articles are we uploading? I've created four different ways of selecting articles for upload. There are further details and configuration options for each of these methods below:
There's a fair bit of configuration involved in getting this all working, so make sure you follow the instructions under Basic configuration and Preparing Omeka-S before proceeding.
Here's an example of a newspaper article in Trove:
Here's the same article after it's been imported into Omeka. Note that the article is linked to the newspaper that published it using the isPartOf
property. You can also see the images and PDF of the article that were automatically uploaded along with the metadata.
And here's the record for the linked newspaper:
First of all you need an Omeka-S site to upload to! If you don't have a site, a handy server, or a preferred web host, I'd suggest you set up an account with Reclaim Hosting – they're affordable, have excellent support, and provide a one-click installer for Omeka-S. You'll be up and running very quickly.
Once you have your Omeka-S site, you need to add some configuration values in the cell below:
API_URL
– the url (or endpoint) of your Omeka site's API. This is basically just the url of your site with /api
on the end.KEY_IDENTITY
and KEY_CREDENTIAL
these are the authentication keys you need to upload new records to Omeka-S.TROVE_API_KEY
– authentication key to access the Trove APISee below for instructions on generating your keys.
key_identity
and key_credential
values – the key_credential
is only ever displayed once, so copy them both now!To get your Trove API key, just follow these instructions.
Before we start uploading newspaper articles we need to set up the vocabularies and templates we need in Omeka to describe them.
Omeka uses defined vocabularies to describe items. A number of vocabularies are built-in, but we're going to add the widely-used Schema.org vocabularly. Amongst many other things, Schema.org includes classes for both Newspaper
and NewsArticle
. You can download the Schema.org definition files here. In the instructions below, I suggest installing the 'all layers' version of the definition file, as this includes the Bib extension as well as Pending changes such as ArchiveComponent
and ArchiveOrganization
.
To install Schema.org in Omeka:
Vocabularly url
field enter http://schema.org/version/latest/all-layers.rdfFile format
field select 'RDF/XML'Prefix
field enter 'schema'Namespace URI
field enter http://schema.org/Label
field enter 'Schema.org'This is what the import form should look like:
See the Omeka-S documentation for more information on managing vocabularies.
The Numeric Data Type module gives you more options in defining the data type of a field. In particular, you can identify certain values as ISO formatted dates so they can then be properly formatted, sorted, and searched. The template I've created for newspaper articles (see below) uses the 'timestamp' data type to import the dates of the newspaper articles, so you'll need to install this module before importing the templates.
See the Omeka-S documentation for more information on installing modules.
One of the powerful features of Omeka is the ability to define the types of items you're working with using 'resource templates'. These resource templates associate the items with specific vocabulary classes and list the expected properties. I've created resource templates for 'Newspaper' and 'Newspaper article' and exported them as JSON. The Trove upload code looks for these templates, so they need to be imported into your Omeka site.
Newspaper.json
file you downloaded.name
property and click on the pencil icon to edit it.description
property and click on the pencil icon to edit it.Newspaper_article.json
, but check at the 'Review import' stage to make sure that the selected data type for the datePublished
property is Timestamp
.Note that I've deliberately kept the templates fairly simple. You might want to add additional properties. However, if you change or remove any of the existing properties, you'll probably also have to edit the add_article()
function below.
See the Omeka-S documentation for more information on managing resource templates.
Hit Shift+Enter to run each of the cells below and set up all the basic configuration and functions we need.
import os
import re
from pathlib import Path
import arrow
import requests
from bs4 import BeautifulSoup
from omeka_s_tools.api import OmekaAPIClient
from pyzotero import zotero
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from tqdm.auto import tqdm
from trove_newspaper_images.articles import download_images
s = requests.Session()
retries = Retry(total=10, backoff_factor=1, status_forcelist=[502, 503, 504, 524])
s.mount("http://", HTTPAdapter(max_retries=retries))
s.mount("https://", HTTPAdapter(max_retries=retries))
%%capture
# Load variables from the .env file if it exists
# Use %%capture to suppress messages
%load_ext dotenv
%dotenv
Insert the values of your Trove API key and your Omeka API url and authentication keys below.
# CONFIGURATION
# Things you need to change!
# Paste your values in below!
# The url of your Omeka site's api (basically your Omeka site site url with '/api' on the end)
API_URL = "http://your.omeka.site/api"
# The keys to your Omeka site
KEY_IDENTITY = "YOUR OMEKA KEY IDENTITY"
KEY_CREDENTIAL = "YOUR OMEKA KEY CREDENTIAL"
# Your Trove API key
TROVE_API_KEY = "YOUR TROVE API KEY"
# Alternatively, use api keys and settings from environment variables if available
if os.getenv("TROVE_API_KEY"):
TROVE_API_KEY = os.getenv("TROVE_API_KEY")
if os.getenv("OMEKA_KEY_IDENTITY"):
KEY_IDENTITY = os.getenv("OMEKA_KEY_IDENTITY")
if os.getenv("OMEKA_KEY_CREDENTIAL"):
KEY_CREDENTIAL = os.getenv("OMEKA_KEY_CREDENTIAL")
if os.getenv("OMEKA_API_URL"):
API_URL = os.getenv("OMEKA_API_URL")
# Resize images so this is the max dimension -- the Trove page images are very big, so you might want to resize before uploading to Omeka
# Set this to None if you want them as big as possible (this might be useful if you're using the Omeka IIIF server & Universal viewer modules)
MAX_IMAGE_SIZE = 3000
omeka = OmekaAPIClient(
API_URL, key_identity=KEY_IDENTITY, key_credential=KEY_CREDENTIAL
)
def get_article(article_id):
"""
Retrieve an individual newspaper article from the Trove API.
Parameters:
* `article_id` - a Trove article identifier
Returns:
* a dict with article metadata from Trove API
"""
url = "http://api.trove.nla.gov.au/v2/newspaper/{}".format(article_id)
params = {"include": "articleText", "encoding": "json", "key": TROVE_API_KEY}
response = s.get(url, params=params)
return response.json()["article"]
def check_for_item(item_url, template_id):
"""
Check to see if an item exists in Omeka using schema:url values and template ids.
Parameters:
* `item_url` - a unique url saved in the schema:url field of an item
* `template_id` - the Omeka id of a template to filter item results
Returns:
* the JSON-LD item representation if found, None if not.
"""
# Filter items by the supplied parameters
results = omeka.filter_items_by_property(
filter_property="schema:url",
filter_value=item_url,
resource_template_id=template_id,
)
# Get the first record, or None if there are no matches
try:
item = results["results"][0]
except (KeyError, IndexError):
item = None
return item
def add_newspaper(newspaper):
"""
Check to see if the given newspaper has already been uploaded to Omeka.
If not, upload metadata to Omeka.
Parameters:
* `newspaper` - this is the dict identifying the newspaper from a Trove article record
Returns:
* the Omeka id of the newspaper record
"""
# Get details of the Newspaper template
newspaper_template = omeka.get_template_by_label("Newspaper")
template_id = newspaper_template["o:id"]
# Get the class used with the Newspaper template
class_id = newspaper_template["o:resource_class"]["o:id"]
# Construct a Trove persistent url for the newspaper
newspaper_url = f'http://nla.gov.au/nla.news-title{newspaper["id"]}'
# Check to see if the newspaper has already been uploaded to Omeka
newspaper_item = check_for_item(newspaper_url, template_id)
# If it hasn't been uploaded, upload it!
if not newspaper_item:
# Prepare the data
newspaper_data = {
"schema:name": [newspaper["value"]],
"schema:identifier": [newspaper["id"]],
"schema:url": [newspaper_url],
}
# Construct the payload for upload to omeka
payload = omeka.prepare_item_payload_using_template(newspaper_data, template_id)
# Upload the item
newspaper_item = omeka.add_item(
payload, template_id=template_id, class_id=class_id
)
return newspaper_item["o:id"]
def add_article(article_id):
"""
Check to see if the given article has already been uploaded to Omeka.
If not, retrieve information about a newspaper article from Trove and
upload the metadata, text, and images to Omeka.
Parameters:
* `article_id` - a Trove article identifier
Returns:
* a JSON-LD representation of the new Omeka item
"""
# Get details of the newspaper article template
article_template = omeka.get_template_by_label("Newspaper article")
template_id = article_template["o:id"]
# Get the resource class used with the newspaper article template
class_id = article_template["o:resource_class"]["o:id"]
# Construct a Trove article persistent url
article_url = f"http://nla.gov.au/nla.news-article{article_id}"
# Check to see if an article with this url has already been uploaded to Omeka
article_item = check_for_item(article_url, template_id)
# If the article hasn't been uploaded, we'll upload it!
if not article_item:
# Get article details from Trove
article = get_article(article_id)
# Format a description
formatted_date = arrow.get(article["date"], "YYYY-MM-DD").format("D MMM YYYY")
summary = (
f'{formatted_date}, {article["title"]["value"]}, page {article["page"]}'
)
# Get the Omeka id of the newspaper it was published in
newspaper_id = add_newspaper(article["title"])
# Remove html tags from article text
try:
soup = BeautifulSoup(article["articleText"])
article_text = soup.get_text()
except KeyError:
article_text = ""
# Prepare the article metadata
article_data = {
"schema:name": [article["heading"]],
"schema:description": [summary],
"schema:datePublished": [article["date"]],
"schema:isPartOf": [newspaper_id],
"schema:pagination": [article["page"]],
"schema:identifier": [article_id],
"schema:url": [article_url],
"schema:text": [article_text],
}
# Construct the payload for uploading to Omeka
payload = omeka.prepare_item_payload_using_template(article_data, template_id)
# Download images of the article
article_images = download_images(article_id, output_dir="temp")
image_paths = [Path("temp", i) for i in article_images]
# Upload the article
article_item = omeka.add_item(
payload, media_files=image_paths, template_id=template_id, class_id=class_id
)
return article_item
Select the method you want to use to select and process your articles and follow the associated instructions.
This will attempt to upload all the newspaper articles returned by a search to your Omeka site. Obviously you want to restrict your search to make sure you're only getting the articles you want – use things like facets and date ranges to keep the result set to a manageable size. You have been warned...
Once you've constructed your search, enter the various parameters below. You might want to add additional facets such as l-decade
or l-year
. Check out Trove's API documentation to see all the options.
Once you've edited the search details, hit Shift+Enter to run the cells below and start your upload.
def get_total_results(params):
"""
Get the total number of results for a Trove search.
"""
these_params = params.copy()
these_params["n"] = 0
response = s.get("https://api.trove.nla.gov.au/v2/result", params=these_params)
data = response.json()
return int(data["response"]["zone"][0]["records"]["total"])
def upload_trove_search(params):
start = "*"
total = get_total_results(params)
with tqdm(total=total) as pbar:
while start:
params["s"] = start
response = s.get("https://api.trove.nla.gov.au/v2/result", params=params)
data = response.json()
# The nextStart parameter is used to get the next page of results.
# If there's no nextStart then it means we're on the last page of results.
try:
start = data["response"]["zone"][0]["records"]["nextStart"]
except KeyError:
start = None
for article in data["response"]["zone"][0]["records"]["article"]:
add_article(article["id"])
pbar.update(1)
# Edit/add search values and parameters as required. These are an example only!
trove_params = {
"q": '"inigo jones"', # required -- change to anything you might enter in the Trove search box (including double quotes for phrases and boolean operators like AND)
"zone": "newspaper", # don't change this
"l-illustrated": "true", # edit or remove -- limits to illustrated articles
"l-illtype": "Photo", # edit or remove -- limits to illustrations with photos
"l-word": "1000+ Words", # edit or remove -- limits to article with more than 1000 words
"include": "articleText", # don't change this
"encoding": "json", # don't change this
"key": TROVE_API_KEY, # don't change this
}
upload_trove_search(trove_params)
You can upload any newspaper articles stored in a Trove list to your Omeka site.
To find the list_id
, just go to the list's web page. The list_id
is the string of numbers that appears after 'id=' in the url. Once you have your list_id
, paste it where indicated in the cell below.
Once you've edited the list details, hit Shift+Enter to run the cells below and start your upload.
def upload_trove_list(list_id):
"""
Upload any newspaper articles in the given Trove list to Omeka.
"""
url = "http://api.trove.nla.gov.au/v2/list/{}".format(list_id)
params = {"include": "listItems", "encoding": "json", "key": TROVE_API_KEY}
response = s.get(url, params=params)
data = response.json()
for item in tqdm(data["list"][0]["listItem"]):
for zone, record in item.items():
if zone == "article":
add_article(record["id"])
# Paste the identifier of your list between the quotes
list_id = "[Your list ID]"
upload_trove_list(list_id)
You can upload any Trove newspaper articles stored in a Zotero collection to your Omeka site.
To access Zotero you need four pieces of information:
ZOTERO_KEY
– generate an API for this application by going here once you've logged into the Zotero web site. Your key only need read access.LIBRARY_ID
– a string of numbers that identifies your library. For personal libraries, go to this page once you've logged into the Zotero web site and look for the line that says 'Your userID for use in API calls is...'. For group libraries, just open the groups web page – the LIBRARY_ID
will be the string of numbers after '/groups/' in the url.LIBRARY_TYPE
– either 'user' or 'group', depending on whether it's a personal library of a group library.collection_id
– the id of the collection that contains your Trove newspaper articles. Just open the collection on the Zotero website – the collection_id
is the string of letters and numbers that comes after '/collectionKey/' in the url.For additional information see the Pyzotero documention.
When you have all the values you need, simply paste them where indicated in the cells below.
Once you've added your details, hit Shift+Enter to run the cells below and start your upload.
# ENTER YOUR VALUES BETWEEN THE QUOTES WHERE INDICATED
ZOTERO_KEY = "YOUR ZOTERO KEY" # The Zotero API key you generated
LIBRARY_TYPE = "user" # user or group
LIBRARY_ID = "YOUR ZOTERO ID" # Either a personal user id or a group id
# Or you can store your information in a .env file
if os.getenv("ZOTERO_KEY"):
ZOTERO_KEY = os.getenv("ZOTERO_KEY")
if os.getenv("ZOTERO_ID"):
LIBRARY_ID = os.getenv("ZOTERO_ID")
def upload_zotero_collection(coll_id):
"""
Upload any Trove newspaper articles in the given collection to Omeka.
"""
zot = zotero.Zotero(LIBRARY_ID, LIBRARY_TYPE, ZOTERO_KEY)
items = zot.everything(zot.collection_items(coll_id))
articles = []
for item in items:
# Filter out things that aren't newspaper articles
try:
url = item["data"]["url"]
if (
item["data"]["itemType"] == "newspaperArticle"
and "nla.news-article" in url
):
article_id = re.search(r"(\d+)$", url).group(1)
articles.append(article_id)
except KeyError:
pass
for article_id in tqdm(articles):
add_article(article_id)
# Paste your collection ID between the quotes below.
collection_id = "YOUR COLLECTION ID"
upload_zotero_collection(collection_id)
This is useful for testing – just get some Trove newspaper article identifiers (the number in the url) and upload them to your Omeka site.
Once you've edited the list of article, hit Shift+Enter to run the cells below and start your upload.
# Edit the list of articles as you see fit...
article_ids = [130413505, 65179201]
for article_id in tqdm(article_ids):
add_article(article_id)
I'd originally hoped to use the Omeka-S API to do most of the configuration, including importing vocabularies and resource templates. But I just couldn't get it to work properly. If I figure it out I'll add the details.
It would be good to be able to upload other types of things from Trove such as images, maps and journal articles. This should be possible, however, things are not as standardised as they are with the newspaper articles, so it'll take a bit of work. Let me know if you're interested and it'll give me a bit more motivation.
Also, going back to my Omeka how-to from 8 years back, I'll be creating a RecordSearch to Omeka-S pipeline as well.
Created by Tim Sherratt for the GLAM Workbench.
Support this project by becoming a GitHub sponsor.