Processing and plotting R15 geophysics data held in the DART postgres database

Created on 20140305 15:30

@author: Anthony Beck - more details available from OrcID

Installation pre-requisites:

  • Python3 (or python 2 with future_print)
  • All the described libraries
  • ipython notebook

These setups are described for an ArchLinux environment here and here. There are a number of walkthroughs that describe such installation. Remember google is your friend

This snippet does the following:

  • extracts data from the DART postgres database by a remote query
  • converts it into an array of values
  • transposes these values to represents the spatial structure of the survey
  • visualizes (produces an image) of these values
  • undertakes image enhancement
  • visualizes the enhanced image

Initially we will bring in the different libraries we want to use:

In [1]:
import psycopg2 #the python postgres query library
import numpy as np #Array mathematics
import matplotlib.pyplot as plt #plotting environment
from matplotlib import rc #import the subfunction 'rc' from matplot so that graphics can be  controlled
rc('font', family='serif')
import scipy.ndimage as nd #Supports multidimensional image processing
from __future__ import print_function #For Python2 print compatibility comment out if causing a problem in python 3

Query the DART postgres database

This example will query the public postrgres database exposed on dartportal.leeds.ac.uk and then place it into a numpy array (remember the library we use for array mathemetics described above).

I need to insert the database citation here. I should also consider a way to embed the data citation into the database. At present we have a bibtex citation snippet and associated metadata from the repository.

Bibtex citation: @data{dart_r15_ddcf_20110928_20110928_rawpreserve_b.csv, doi = {not allocated}, url = {http://dartportal.leeds.ac.uk/storage/f/dart_r15_ddcf_20110928_20110928_rawpreserve_b.csv}, author = "{Robert Fry}", publisher = {DART repository, School of Computing, University of Leeds}, title = {dart_r15_ddcf_20110928_20110928_rawpreserve_b.csv}, year = {2013}, note = {DART is a Science and Heritage project funded by AHRC and EPSRC. Further DART data and details can be found at http://dartportal.leeds.ac.uk} }

In [2]:
#Retrieve data from the database
#set up connection string to Postgres
connection_string ="dbname='dart_public' user='dart_public_ro' host='dartportal.leeds.ac.uk' password=''"

#connect to database
try:
    conn = psycopg2.connect(connection_string)
except:
    print ("I am unable to connect to the database")

cur = conn.cursor()



#set up the SQL query
inputSQL = """SELECT 
  geophysics_s_local_survey.eastinglocal, 
  geophysics_s_local_survey.northinglocal, 
  geophysics_s_local_survey.measurementvalue
FROM 
  public.geophysics_s_local_survey
WHERE 
  geophysics_s_local_survey.sensorconfiguration = 'B' AND 
  geophysics_s_local_survey.observationtimestamp = '2012-03-21' AND 
  geophysics_s_local_survey.dartfieldcode = 'HHCC';"""
#2012-03-21 HHCC
#2011-09-28 DDCF

'''
#set up the SQL query
inputSQL = """SELECT 
  geophysics_s_local_survey.eastinglocal, 
  geophysics_s_local_survey.northinglocal, 
  geophysics_s_local_survey.measurementvalue
FROM 
  public.geophysics_s_local_survey
WHERE 
  geophysics_s_local_survey.sensorconfiguration = 'VCPIB' AND 
  geophysics_s_local_survey.observationtimestamp = '2012-09-27' AND 
  geophysics_s_local_survey.dartfieldcode = 'HHCC';"""
'''
#run query on the database
cur.execute(inputSQL)

#place query results into list called rows
rows = cur.fetchall()
print ("\nShow me the data:\n")
#iterate on a row by row basis over the list rows and print out the result
for row in rows:
    print ("   ", row)

#read the data into a list
data = np.array(rows)
print ('the data as a list')
print (data)
Show me the data:

    (0.25, 0.25, 48.3)
    (0.75, 0.25, 47.6)
    (1.25, 0.25, 46.95)
    (1.75, 0.25, 46.4)
    (2.25, 0.25, 46.5)
    (2.75, 0.25, 46.65)
    (3.25, 0.25, 46.45)
    (3.75, 0.25, 47.25)
    (4.25, 0.25, 46.55)
    (4.75, 0.25, 45.75)
    (5.25, 0.25, 44.45)
    (5.75, 0.25, 44.0)
    (6.25, 0.25, 44.9)
    (6.75, 0.25, 46.1)
    (7.25, 0.25, 47.55)
    (7.75, 0.25, 48.8)
    (8.25, 0.25, 49.0)
    (8.75, 0.25, 49.45)
    (9.25, 0.25, 49.5)
    (9.75, 0.25, 49.15)
    (0.25, 0.75, 48.2)
    (0.75, 0.75, 47.8)
    (1.25, 0.75, 47.25)
    (1.75, 0.75, 47.0)
    (2.25, 0.75, 47.65)
    (2.75, 0.75, 47.65)
    (3.25, 0.75, 47.0)
    (3.75, 0.75, 46.3)
    (4.25, 0.75, 45.8)
    (4.75, 0.75, 43.05)
    (5.25, 0.75, 41.45)
    (5.75, 0.75, 42.55)
    (6.25, 0.75, 44.8)
    (6.75, 0.75, 46.65)
    (7.25, 0.75, 47.3)
    (7.75, 0.75, 48.1)
    (8.25, 0.75, 48.8)
    (8.75, 0.75, 48.4)
    (9.25, 0.75, 48.95)
    (9.75, 0.75, 48.15)
    (0.25, 1.25, 47.95)
    (0.75, 1.25, 48.1)
    (1.25, 1.25, 48.15)
    (1.75, 1.25, 48.85)
    (2.25, 1.25, 48.75)
    (2.75, 1.25, 47.95)
    (3.25, 1.25, 46.6)
    (3.75, 1.25, 46.3)
    (4.25, 1.25, 45.55)
    (4.75, 1.25, 42.8)
    (5.25, 1.25, 41.9)
    (5.75, 1.25, 41.6)
    (6.25, 1.25, 44.2)
    (6.75, 1.25, 46.4)
    (7.25, 1.25, 48.2)
    (7.75, 1.25, 47.7)
    (8.25, 1.25, 48.05)
    (8.75, 1.25, 48.3)
    (9.25, 1.25, 48.45)
    (9.75, 1.25, 49.1)
    (0.25, 1.75, 48.45)
    (0.75, 1.75, 48.35)
    (1.25, 1.75, 48.65)
    (1.75, 1.75, 49.4)
    (2.25, 1.75, 49.45)
    (2.75, 1.75, 48.65)
    (3.25, 1.75, 47.95)
    (3.75, 1.75, 46.65)
    (4.25, 1.75, 45.0)
    (4.75, 1.75, 42.8)
    (5.25, 1.75, 41.25)
    (5.75, 1.75, 41.5)
    (6.25, 1.75, 43.55)
    (6.75, 1.75, 46.6)
    (7.25, 1.75, 47.95)
    (7.75, 1.75, 48.65)
    (8.25, 1.75, 47.6)
    (8.75, 1.75, 47.8)
    (9.25, 1.75, 48.5)
    (9.75, 1.75, 48.05)
    (0.25, 2.25, 47.25)
    (0.75, 2.25, 47.35)
    (1.25, 2.25, 47.7)
    (1.75, 2.25, 48.75)
    (2.25, 2.25, 48.05)
    (2.75, 2.25, 47.45)
    (3.25, 2.25, 46.45)
    (3.75, 2.25, 45.25)
    (4.25, 2.25, 43.55)
    (4.75, 2.25, 41.5)
    (5.25, 2.25, 41.0)
    (5.75, 2.25, 42.0)
    (6.25, 2.25, 43.95)
    (6.75, 2.25, 46.2)
    (7.25, 2.25, 47.7)
    (7.75, 2.25, 48.3)
    (8.25, 2.25, 47.75)
    (8.75, 2.25, 48.1)
    (9.25, 2.25, 48.5)
    (9.75, 2.25, 48.85)
    (0.25, 2.75, 46.45)
    (0.75, 2.75, 47.65)
    (1.25, 2.75, 47.55)
    (1.75, 2.75, 47.5)
    (2.25, 2.75, 47.9)
    (2.75, 2.75, 47.2)
    (3.25, 2.75, 46.6)
    (3.75, 2.75, 46.0)
    (4.25, 2.75, 44.35)
    (4.75, 2.75, 42.65)
    (5.25, 2.75, 40.5)
    (5.75, 2.75, 41.45)
    (6.25, 2.75, 43.6)
    (6.75, 2.75, 46.35)
    (7.25, 2.75, 48.2)
    (7.75, 2.75, 48.5)
    (8.25, 2.75, 48.75)
    (8.75, 2.75, 49.3)
    (9.25, 2.75, 49.65)
    (9.75, 2.75, 50.8)
    (0.25, 3.25, 47.3)
    (0.75, 3.25, 47.95)
    (1.25, 3.25, 48.5)
    (1.75, 3.25, 49.05)
    (2.25, 3.25, 48.6)
    (2.75, 3.25, 47.9)
    (3.25, 3.25, 46.8)
    (3.75, 3.25, 45.7)
    (4.25, 3.25, 43.9)
    (4.75, 3.25, 41.75)
    (5.25, 3.25, 39.9)
    (5.75, 3.25, 40.95)
    (6.25, 3.25, 43.5)
    (6.75, 3.25, 46.5)
    (7.25, 3.25, 48.35)
    (7.75, 3.25, 49.05)
    (8.25, 3.25, 49.3)
    (8.75, 3.25, 49.85)
    (9.25, 3.25, 48.4)
    (9.75, 3.25, 51.0)
    (0.25, 3.75, 48.2)
    (0.75, 3.75, 47.9)
    (1.25, 3.75, 47.8)
    (1.75, 3.75, 48.4)
    (2.25, 3.75, 48.7)
    (2.75, 3.75, 47.5)
    (3.25, 3.75, 46.85)
    (3.75, 3.75, 45.15)
    (4.25, 3.75, 44.35)
    (4.75, 3.75, 42.5)
    (5.25, 3.75, 41.3)
    (5.75, 3.75, 40.9)
    (6.25, 3.75, 42.6)
    (6.75, 3.75, 46.45)
    (7.25, 3.75, 47.75)
    (7.75, 3.75, 47.1)
    (8.25, 3.75, 48.7)
    (8.75, 3.75, 48.9)
    (9.25, 3.75, 49.9)
    (9.75, 3.75, 50.85)
    (0.25, 4.25, 49.55)
    (0.75, 4.25, 49.9)
    (1.25, 4.25, 50.05)
    (1.75, 4.25, 50.1)
    (2.25, 4.25, 48.7)
    (2.75, 4.25, 47.2)
    (3.25, 4.25, 46.35)
    (3.75, 4.25, 45.5)
    (4.25, 4.25, 43.2)
    (4.75, 4.25, 40.95)
    (5.25, 4.25, 40.25)
    (5.75, 4.25, 40.55)
    (6.25, 4.25, 43.6)
    (6.75, 4.25, 46.45)
    (7.25, 4.25, 48.1)
    (7.75, 4.25, 49.85)
    (8.25, 4.25, 49.3)
    (8.75, 4.25, 49.35)
    (9.25, 4.25, 50.05)
    (9.75, 4.25, 50.05)
    (0.25, 4.75, 50.25)
    (0.75, 4.75, 49.65)
    (1.25, 4.75, 49.6)
    (1.75, 4.75, 49.4)
    (2.25, 4.75, 49.05)
    (2.75, 4.75, 48.15)
    (3.25, 4.75, 48.3)
    (3.75, 4.75, 47.15)
    (4.25, 4.75, 46.15)
    (4.75, 4.75, 42.9)
    (5.25, 4.75, 41.05)
    (5.75, 4.75, 40.95)
    (6.25, 4.75, 42.75)
    (6.75, 4.75, 46.2)
    (7.25, 4.75, 48.45)
    (7.75, 4.75, 49.75)
    (8.25, 4.75, 50.6)
    (8.75, 4.75, 51.25)
    (9.25, 4.75, 51.35)
    (9.75, 4.75, 51.4)
    (0.25, 5.25, 49.1)
    (0.75, 5.25, 49.6)
    (1.25, 5.25, 50.35)
    (1.75, 5.25, 50.25)
    (2.25, 5.25, 49.6)
    (2.75, 5.25, 48.3)
    (3.25, 5.25, 48.45)
    (3.75, 5.25, 47.35)
    (4.25, 5.25, 45.85)
    (4.75, 5.25, 42.8)
    (5.25, 5.25, 40.85)
    (5.75, 5.25, 40.8)
    (6.25, 5.25, 43.55)
    (6.75, 5.25, 47.8)
    (7.25, 5.25, 48.85)
    (7.75, 5.25, 50.05)
    (8.25, 5.25, 50.55)
    (8.75, 5.25, 50.7)
    (9.25, 5.25, 50.1)
    (9.75, 5.25, 50.5)
    (0.25, 5.75, 50.45)
    (0.75, 5.75, 50.5)
    (1.25, 5.75, 50.2)
    (1.75, 5.75, 50.25)
    (2.25, 5.75, 49.95)
    (2.75, 5.75, 49.95)
    (3.25, 5.75, 49.75)
    (3.75, 5.75, 48.55)
    (4.25, 5.75, 47.95)
    (4.75, 5.75, 44.85)
    (5.25, 5.75, 41.85)
    (5.75, 5.75, 41.25)
    (6.25, 5.75, 43.25)
    (6.75, 5.75, 47.8)
    (7.25, 5.75, 49.4)
    (7.75, 5.75, 50.75)
    (8.25, 5.75, 50.8)
    (8.75, 5.75, 51.8)
    (9.25, 5.75, 52.65)
    (9.75, 5.75, 52.3)
    (0.25, 6.25, 49.25)
    (0.75, 6.25, 49.75)
    (1.25, 6.25, 50.25)
    (1.75, 6.25, 49.35)
    (2.25, 6.25, 49.15)
    (2.75, 6.25, 49.9)
    (3.25, 6.25, 50.25)
    (3.75, 6.25, 49.1)
    (4.25, 6.25, 48.4)
    (4.75, 6.25, 44.65)
    (5.25, 6.25, 41.6)
    (5.75, 6.25, 41.55)
    (6.25, 6.25, 43.4)
    (6.75, 6.25, 46.15)
    (7.25, 6.25, 50.15)
    (7.75, 6.25, 50.65)
    (8.25, 6.25, 53.0)
    (8.75, 6.25, 53.15)
    (9.25, 6.25, 52.7)
    (9.75, 6.25, 53.0)
    (0.25, 6.75, 49.25)
    (0.75, 6.75, 49.35)
    (1.25, 6.75, 49.55)
    (1.75, 6.75, 48.95)
    (2.25, 6.75, 49.6)
    (2.75, 6.75, 50.4)
    (3.25, 6.75, 51.75)
    (3.75, 6.75, 49.4)
    (4.25, 6.75, 48.35)
    (4.75, 6.75, 44.6)
    (5.25, 6.75, 42.45)
    (5.75, 6.75, 41.95)
    (6.25, 6.75, 42.7)
    (6.75, 6.75, 46.3)
    (7.25, 6.75, 49.95)
    (7.75, 6.75, 51.2)
    (8.25, 6.75, 53.0)
    (8.75, 6.75, 53.45)
    (9.25, 6.75, 53.5)
    (9.75, 6.75, 52.7)
    (0.25, 7.25, 49.8)
    (0.75, 7.25, 49.0)
    (1.25, 7.25, 49.45)
    (1.75, 7.25, 50.1)
    (2.25, 7.25, 49.65)
    (2.75, 7.25, 50.35)
    (3.25, 7.25, 49.9)
    (3.75, 7.25, 50.05)
    (4.25, 7.25, 48.4)
    (4.75, 7.25, 46.55)
    (5.25, 7.25, 43.1)
    (5.75, 7.25, 41.4)
    (6.25, 7.25, 41.95)
    (6.75, 7.25, 44.8)
    (7.25, 7.25, 48.75)
    (7.75, 7.25, 50.95)
    (8.25, 7.25, 52.4)
    (8.75, 7.25, 53.35)
    (9.25, 7.25, 54.45)
    (9.75, 7.25, 54.65)
    (0.25, 7.75, 49.05)
    (0.75, 7.75, 49.85)
    (1.25, 7.75, 49.9)
    (1.75, 7.75, 50.05)
    (2.25, 7.75, 50.7)
    (2.75, 7.75, 50.05)
    (3.25, 7.75, 50.65)
    (3.75, 7.75, 50.75)
    (4.25, 7.75, 49.9)
    (4.75, 7.75, 47.1)
    (5.25, 7.75, 43.7)
    (5.75, 7.75, 41.75)
    (6.25, 7.75, 42.4)
    (6.75, 7.75, 44.8)
    (7.25, 7.75, 48.5)
    (7.75, 7.75, 51.35)
    (8.25, 7.75, 52.9)
    (8.75, 7.75, 53.9)
    (9.25, 7.75, 54.7)
    (9.75, 7.75, 53.95)
    (0.25, 8.25, 49.45)
    (0.75, 8.25, 50.0)
    (1.25, 8.25, 50.15)
    (1.75, 8.25, 50.55)
    (2.25, 8.25, 50.75)
    (2.75, 8.25, 51.1)
    (3.25, 8.25, 50.15)
    (3.75, 8.25, 49.9)
    (4.25, 8.25, 49.65)
    (4.75, 8.25, 46.8)
    (5.25, 8.25, 43.15)
    (5.75, 8.25, 41.65)
    (6.25, 8.25, 41.75)
    (6.75, 8.25, 44.4)
    (7.25, 8.25, 48.55)
    (7.75, 8.25, 50.4)
    (8.25, 8.25, 51.45)
    (8.75, 8.25, 53.3)
    (9.25, 8.25, 52.95)
    (9.75, 8.25, 52.15)
    (0.25, 8.75, 49.65)
    (0.75, 8.75, 50.5)
    (1.25, 8.75, 50.75)
    (1.75, 8.75, 50.45)
    (2.25, 8.75, 49.55)
    (2.75, 8.75, 51.1)
    (3.25, 8.75, 50.3)
    (3.75, 8.75, 50.55)
    (4.25, 8.75, 49.9)
    (4.75, 8.75, 47.6)
    (5.25, 8.75, 43.95)
    (5.75, 8.75, 42.45)
    (6.25, 8.75, 42.35)
    (6.75, 8.75, 45.1)
    (7.25, 8.75, 47.8)
    (7.75, 8.75, 50.65)
    (8.25, 8.75, 51.85)
    (8.75, 8.75, 53.55)
    (9.25, 8.75, 54.45)
    (9.75, 8.75, 69.8)
    (0.25, 9.25, 51.9)
    (0.75, 9.25, 51.45)
    (1.25, 9.25, 51.3)
    (1.75, 9.25, 50.65)
    (2.25, 9.25, 50.25)
    (2.75, 9.25, 50.35)
    (3.25, 9.25, 50.75)
    (3.75, 9.25, 50.0)
    (4.25, 9.25, 50.1)
    (4.75, 9.25, 46.55)
    (5.25, 9.25, 42.8)
    (5.75, 9.25, 41.95)
    (6.25, 9.25, 42.15)
    (6.75, 9.25, 44.95)
    (7.25, 9.25, 45.4)
    (7.75, 9.25, 48.65)
    (8.25, 9.25, 50.6)
    (8.75, 9.25, 52.0)
    (9.25, 9.25, 52.3)
    (9.75, 9.25, 53.5)
    (0.25, 9.75, 51.55)
    (0.75, 9.75, 50.8)
    (1.25, 9.75, 50.65)
    (1.75, 9.75, 50.1)
    (2.25, 9.75, 49.25)
    (2.75, 9.75, 50.6)
    (3.25, 9.75, 50.7)
    (3.75, 9.75, 50.6)
    (4.25, 9.75, 49.6)
    (4.75, 9.75, 47.8)
    (5.25, 9.75, 44.45)
    (5.75, 9.75, 41.6)
    (6.25, 9.75, 41.45)
    (6.75, 9.75, 43.25)
    (7.25, 9.75, 45.75)
    (7.75, 9.75, 48.4)
    (8.25, 9.75, 50.8)
    (8.75, 9.75, 51.3)
    (9.25, 9.75, 52.45)
    (9.75, 9.75, 52.55)
the data as a list
[[  0.25   0.25  48.3 ]
 [  0.75   0.25  47.6 ]
 [  1.25   0.25  46.95]
 ..., 
 [  8.75   9.75  51.3 ]
 [  9.25   9.75  52.45]
 [  9.75   9.75  52.55]]

Transposes these values to represents the spatial structure of the survey

The imported array looks like this:

[[ 0.25 0.25 39.15]

[ 0.75 0.25 31.75]

[ 1.25 0.25 28.8 ]

...,

[ 8.75 9.75 42.85]

[ 9.25 9.75 33.6 ]

[ 9.75 9.75 52.4 ]]

This represents an array with 'rows' that contain three values. These three values are 'x-coordinate', 'y-coordinate', and 'measurement value'. The x and y coordinates are represented in relation to the 'local grid'. This means they are not tied into a national or global co-ordinate system. We will cover this later.

You can think of this structure like an Excell spreadsheet with three columns. However, This is not what we want. The data file contains 400 rows which corresponds with a 20 readind x 20 reading square. We need to reshape this image to represent this data. For this we use the .reshape command in numpy.

Rather than reshaping all the data we will reshape solely on the third column. We do this using a filter:

data[:,2]

This means the following:

  • 'data' is the numpy array holding the geophysics data
  • The square brackets [] hold the filter expression
    • the content before the colon ':' contains the row filter (in this case nothing)
    • the content after the colon ':' contains the column filter (in this case ONLY 2 (represented as ',2'))
      • NOTE: Python counts from zero. So a three column array is referenced as 0,1,2

This is demonstrated below:

In [3]:
#Display unly the third column in the geophysics data
data[:,2]
Out[3]:
array([ 48.3 ,  47.6 ,  46.95,  46.4 ,  46.5 ,  46.65,  46.45,  47.25,
        46.55,  45.75,  44.45,  44.  ,  44.9 ,  46.1 ,  47.55,  48.8 ,
        49.  ,  49.45,  49.5 ,  49.15,  48.2 ,  47.8 ,  47.25,  47.  ,
        47.65,  47.65,  47.  ,  46.3 ,  45.8 ,  43.05,  41.45,  42.55,
        44.8 ,  46.65,  47.3 ,  48.1 ,  48.8 ,  48.4 ,  48.95,  48.15,
        47.95,  48.1 ,  48.15,  48.85,  48.75,  47.95,  46.6 ,  46.3 ,
        45.55,  42.8 ,  41.9 ,  41.6 ,  44.2 ,  46.4 ,  48.2 ,  47.7 ,
        48.05,  48.3 ,  48.45,  49.1 ,  48.45,  48.35,  48.65,  49.4 ,
        49.45,  48.65,  47.95,  46.65,  45.  ,  42.8 ,  41.25,  41.5 ,
        43.55,  46.6 ,  47.95,  48.65,  47.6 ,  47.8 ,  48.5 ,  48.05,
        47.25,  47.35,  47.7 ,  48.75,  48.05,  47.45,  46.45,  45.25,
        43.55,  41.5 ,  41.  ,  42.  ,  43.95,  46.2 ,  47.7 ,  48.3 ,
        47.75,  48.1 ,  48.5 ,  48.85,  46.45,  47.65,  47.55,  47.5 ,
        47.9 ,  47.2 ,  46.6 ,  46.  ,  44.35,  42.65,  40.5 ,  41.45,
        43.6 ,  46.35,  48.2 ,  48.5 ,  48.75,  49.3 ,  49.65,  50.8 ,
        47.3 ,  47.95,  48.5 ,  49.05,  48.6 ,  47.9 ,  46.8 ,  45.7 ,
        43.9 ,  41.75,  39.9 ,  40.95,  43.5 ,  46.5 ,  48.35,  49.05,
        49.3 ,  49.85,  48.4 ,  51.  ,  48.2 ,  47.9 ,  47.8 ,  48.4 ,
        48.7 ,  47.5 ,  46.85,  45.15,  44.35,  42.5 ,  41.3 ,  40.9 ,
        42.6 ,  46.45,  47.75,  47.1 ,  48.7 ,  48.9 ,  49.9 ,  50.85,
        49.55,  49.9 ,  50.05,  50.1 ,  48.7 ,  47.2 ,  46.35,  45.5 ,
        43.2 ,  40.95,  40.25,  40.55,  43.6 ,  46.45,  48.1 ,  49.85,
        49.3 ,  49.35,  50.05,  50.05,  50.25,  49.65,  49.6 ,  49.4 ,
        49.05,  48.15,  48.3 ,  47.15,  46.15,  42.9 ,  41.05,  40.95,
        42.75,  46.2 ,  48.45,  49.75,  50.6 ,  51.25,  51.35,  51.4 ,
        49.1 ,  49.6 ,  50.35,  50.25,  49.6 ,  48.3 ,  48.45,  47.35,
        45.85,  42.8 ,  40.85,  40.8 ,  43.55,  47.8 ,  48.85,  50.05,
        50.55,  50.7 ,  50.1 ,  50.5 ,  50.45,  50.5 ,  50.2 ,  50.25,
        49.95,  49.95,  49.75,  48.55,  47.95,  44.85,  41.85,  41.25,
        43.25,  47.8 ,  49.4 ,  50.75,  50.8 ,  51.8 ,  52.65,  52.3 ,
        49.25,  49.75,  50.25,  49.35,  49.15,  49.9 ,  50.25,  49.1 ,
        48.4 ,  44.65,  41.6 ,  41.55,  43.4 ,  46.15,  50.15,  50.65,
        53.  ,  53.15,  52.7 ,  53.  ,  49.25,  49.35,  49.55,  48.95,
        49.6 ,  50.4 ,  51.75,  49.4 ,  48.35,  44.6 ,  42.45,  41.95,
        42.7 ,  46.3 ,  49.95,  51.2 ,  53.  ,  53.45,  53.5 ,  52.7 ,
        49.8 ,  49.  ,  49.45,  50.1 ,  49.65,  50.35,  49.9 ,  50.05,
        48.4 ,  46.55,  43.1 ,  41.4 ,  41.95,  44.8 ,  48.75,  50.95,
        52.4 ,  53.35,  54.45,  54.65,  49.05,  49.85,  49.9 ,  50.05,
        50.7 ,  50.05,  50.65,  50.75,  49.9 ,  47.1 ,  43.7 ,  41.75,
        42.4 ,  44.8 ,  48.5 ,  51.35,  52.9 ,  53.9 ,  54.7 ,  53.95,
        49.45,  50.  ,  50.15,  50.55,  50.75,  51.1 ,  50.15,  49.9 ,
        49.65,  46.8 ,  43.15,  41.65,  41.75,  44.4 ,  48.55,  50.4 ,
        51.45,  53.3 ,  52.95,  52.15,  49.65,  50.5 ,  50.75,  50.45,
        49.55,  51.1 ,  50.3 ,  50.55,  49.9 ,  47.6 ,  43.95,  42.45,
        42.35,  45.1 ,  47.8 ,  50.65,  51.85,  53.55,  54.45,  69.8 ,
        51.9 ,  51.45,  51.3 ,  50.65,  50.25,  50.35,  50.75,  50.  ,
        50.1 ,  46.55,  42.8 ,  41.95,  42.15,  44.95,  45.4 ,  48.65,
        50.6 ,  52.  ,  52.3 ,  53.5 ,  51.55,  50.8 ,  50.65,  50.1 ,
        49.25,  50.6 ,  50.7 ,  50.6 ,  49.6 ,  47.8 ,  44.45,  41.6 ,
        41.45,  43.25,  45.75,  48.4 ,  50.8 ,  51.3 ,  52.45,  52.55])

That's great! We've extracted the third column. But haven't we lost the relative spatial location information? The short answer is formally.... yes we have. The longer answer is.... no we havent. Let me explain....

The data is ordered. Let's look at the data again.

[[ 0.25 0.25 39.15]

[ 0.75 0.25 31.75]

[ 1.25 0.25 28.8 ]

...,

[ 8.75 9.75 42.85]

[ 9.25 9.75 33.6 ]

[ 9.75 9.75 52.4 ]]

The x-coordinate field increments until it reaches the end of it's row then the y-coordinate field increments. It's essentially a nested loop. We can use this property to help us. The order of the data implicitly maintains the underlying spatial structure.

However, is there a problem here? This code relies on all the data we ever get from this instrument being formatted in this way. How could you amend the code to ensure that it is always structued in this way

As we know this is a 20x20 measurement matrix we can re-grid the data as follows:

In [4]:
#restructure the array into a 20 x 20 matrix (appropriate for this data)
#although not sure if the orientation is right
griddata = data[:,2].reshape(20,20)


#print data
print('The data recast as in a 20x20 grid array')
print (griddata)
The data recast as in a 20x20 grid array
[[ 48.3   47.6   46.95  46.4   46.5   46.65  46.45  47.25  46.55  45.75
   44.45  44.    44.9   46.1   47.55  48.8   49.    49.45  49.5   49.15]
 [ 48.2   47.8   47.25  47.    47.65  47.65  47.    46.3   45.8   43.05
   41.45  42.55  44.8   46.65  47.3   48.1   48.8   48.4   48.95  48.15]
 [ 47.95  48.1   48.15  48.85  48.75  47.95  46.6   46.3   45.55  42.8
   41.9   41.6   44.2   46.4   48.2   47.7   48.05  48.3   48.45  49.1 ]
 [ 48.45  48.35  48.65  49.4   49.45  48.65  47.95  46.65  45.    42.8
   41.25  41.5   43.55  46.6   47.95  48.65  47.6   47.8   48.5   48.05]
 [ 47.25  47.35  47.7   48.75  48.05  47.45  46.45  45.25  43.55  41.5   41.
   42.    43.95  46.2   47.7   48.3   47.75  48.1   48.5   48.85]
 [ 46.45  47.65  47.55  47.5   47.9   47.2   46.6   46.    44.35  42.65
   40.5   41.45  43.6   46.35  48.2   48.5   48.75  49.3   49.65  50.8 ]
 [ 47.3   47.95  48.5   49.05  48.6   47.9   46.8   45.7   43.9   41.75
   39.9   40.95  43.5   46.5   48.35  49.05  49.3   49.85  48.4   51.  ]
 [ 48.2   47.9   47.8   48.4   48.7   47.5   46.85  45.15  44.35  42.5
   41.3   40.9   42.6   46.45  47.75  47.1   48.7   48.9   49.9   50.85]
 [ 49.55  49.9   50.05  50.1   48.7   47.2   46.35  45.5   43.2   40.95
   40.25  40.55  43.6   46.45  48.1   49.85  49.3   49.35  50.05  50.05]
 [ 50.25  49.65  49.6   49.4   49.05  48.15  48.3   47.15  46.15  42.9
   41.05  40.95  42.75  46.2   48.45  49.75  50.6   51.25  51.35  51.4 ]
 [ 49.1   49.6   50.35  50.25  49.6   48.3   48.45  47.35  45.85  42.8
   40.85  40.8   43.55  47.8   48.85  50.05  50.55  50.7   50.1   50.5 ]
 [ 50.45  50.5   50.2   50.25  49.95  49.95  49.75  48.55  47.95  44.85
   41.85  41.25  43.25  47.8   49.4   50.75  50.8   51.8   52.65  52.3 ]
 [ 49.25  49.75  50.25  49.35  49.15  49.9   50.25  49.1   48.4   44.65
   41.6   41.55  43.4   46.15  50.15  50.65  53.    53.15  52.7   53.  ]
 [ 49.25  49.35  49.55  48.95  49.6   50.4   51.75  49.4   48.35  44.6
   42.45  41.95  42.7   46.3   49.95  51.2   53.    53.45  53.5   52.7 ]
 [ 49.8   49.    49.45  50.1   49.65  50.35  49.9   50.05  48.4   46.55
   43.1   41.4   41.95  44.8   48.75  50.95  52.4   53.35  54.45  54.65]
 [ 49.05  49.85  49.9   50.05  50.7   50.05  50.65  50.75  49.9   47.1
   43.7   41.75  42.4   44.8   48.5   51.35  52.9   53.9   54.7   53.95]
 [ 49.45  50.    50.15  50.55  50.75  51.1   50.15  49.9   49.65  46.8
   43.15  41.65  41.75  44.4   48.55  50.4   51.45  53.3   52.95  52.15]
 [ 49.65  50.5   50.75  50.45  49.55  51.1   50.3   50.55  49.9   47.6
   43.95  42.45  42.35  45.1   47.8   50.65  51.85  53.55  54.45  69.8 ]
 [ 51.9   51.45  51.3   50.65  50.25  50.35  50.75  50.    50.1   46.55
   42.8   41.95  42.15  44.95  45.4   48.65  50.6   52.    52.3   53.5 ]
 [ 51.55  50.8   50.65  50.1   49.25  50.6   50.7   50.6   49.6   47.8
   44.45  41.6   41.45  43.25  45.75  48.4   50.8   51.3   52.45  52.55]]

Visualize (produce an image) of the data

This means we can now plot the data using matplotlib. The workings of matplotlib are for another lesson

In [5]:
#Plot the data
plt.imshow(griddata,origin='lower')
#plt.imshow(griddata)

plt.gray()
cb=plt.colorbar()
cb.set_label('Value Range')
plt.xlabel('GridEast')
plt.ylabel('GridNorth')
plt.suptitle('Raw data')
plt.show()

Getting fancy - but what if we don't know the collection interval of the measurements?

When we reshaped the data above we needed to tell the system that it was a 20x20 measurement grid (or a measurement every 0.5 metres). What happens if we want to make this more generic and build in the principle that people can use different collection intervals during the survey (say every metre). The ultimate corrolary is that the measurements can be collected in a whole heap of different ways:

  • a cart with an arrays of machines moving in a zig zag pattern
  • single machines attached to a differential GNSS device using an ungridded collection strategy
  • etc

For now we will assume that we have a single machine, but it can have variable collection intervals. We will also assume that the collection interval has the same number of columns as rows (i.e. they are evenly distributed). Note these are assumptions and they may be wrong. Depending on how you configure the instrument you may have no choice on how rapidly it collect measurements along a line but have complete control about how far apart the grid lines are.

We can analyse the array to get some basic information about the matrix:

In [6]:
print('Total number of elements in INPUT data: ', size(data[:,2])) #search on third column
print('Number of dimensions for INPUT data: ',ndim(data[:,2]))
print('SQRT of total number of elements in INPUT data: ', sqrt(size(data[:,2])))
Total number of elements in INPUT data:  400
Number of dimensions for INPUT data:  1
SQRT of total number of elements in INPUT data:  20.0

Based on our assumptions (it being square with the same number of measurements in the rows and columns) we can automatically determine the size of the reshape by taking the square root of the size of the input data.

In [7]:
griddata = data[:,2].reshape(sqrt(size(data[:,2])),sqrt(size(data[:,2])))

print (griddata)
[[ 48.3   47.6   46.95  46.4   46.5   46.65  46.45  47.25  46.55  45.75
   44.45  44.    44.9   46.1   47.55  48.8   49.    49.45  49.5   49.15]
 [ 48.2   47.8   47.25  47.    47.65  47.65  47.    46.3   45.8   43.05
   41.45  42.55  44.8   46.65  47.3   48.1   48.8   48.4   48.95  48.15]
 [ 47.95  48.1   48.15  48.85  48.75  47.95  46.6   46.3   45.55  42.8
   41.9   41.6   44.2   46.4   48.2   47.7   48.05  48.3   48.45  49.1 ]
 [ 48.45  48.35  48.65  49.4   49.45  48.65  47.95  46.65  45.    42.8
   41.25  41.5   43.55  46.6   47.95  48.65  47.6   47.8   48.5   48.05]
 [ 47.25  47.35  47.7   48.75  48.05  47.45  46.45  45.25  43.55  41.5   41.
   42.    43.95  46.2   47.7   48.3   47.75  48.1   48.5   48.85]
 [ 46.45  47.65  47.55  47.5   47.9   47.2   46.6   46.    44.35  42.65
   40.5   41.45  43.6   46.35  48.2   48.5   48.75  49.3   49.65  50.8 ]
 [ 47.3   47.95  48.5   49.05  48.6   47.9   46.8   45.7   43.9   41.75
   39.9   40.95  43.5   46.5   48.35  49.05  49.3   49.85  48.4   51.  ]
 [ 48.2   47.9   47.8   48.4   48.7   47.5   46.85  45.15  44.35  42.5
   41.3   40.9   42.6   46.45  47.75  47.1   48.7   48.9   49.9   50.85]
 [ 49.55  49.9   50.05  50.1   48.7   47.2   46.35  45.5   43.2   40.95
   40.25  40.55  43.6   46.45  48.1   49.85  49.3   49.35  50.05  50.05]
 [ 50.25  49.65  49.6   49.4   49.05  48.15  48.3   47.15  46.15  42.9
   41.05  40.95  42.75  46.2   48.45  49.75  50.6   51.25  51.35  51.4 ]
 [ 49.1   49.6   50.35  50.25  49.6   48.3   48.45  47.35  45.85  42.8
   40.85  40.8   43.55  47.8   48.85  50.05  50.55  50.7   50.1   50.5 ]
 [ 50.45  50.5   50.2   50.25  49.95  49.95  49.75  48.55  47.95  44.85
   41.85  41.25  43.25  47.8   49.4   50.75  50.8   51.8   52.65  52.3 ]
 [ 49.25  49.75  50.25  49.35  49.15  49.9   50.25  49.1   48.4   44.65
   41.6   41.55  43.4   46.15  50.15  50.65  53.    53.15  52.7   53.  ]
 [ 49.25  49.35  49.55  48.95  49.6   50.4   51.75  49.4   48.35  44.6
   42.45  41.95  42.7   46.3   49.95  51.2   53.    53.45  53.5   52.7 ]
 [ 49.8   49.    49.45  50.1   49.65  50.35  49.9   50.05  48.4   46.55
   43.1   41.4   41.95  44.8   48.75  50.95  52.4   53.35  54.45  54.65]
 [ 49.05  49.85  49.9   50.05  50.7   50.05  50.65  50.75  49.9   47.1
   43.7   41.75  42.4   44.8   48.5   51.35  52.9   53.9   54.7   53.95]
 [ 49.45  50.    50.15  50.55  50.75  51.1   50.15  49.9   49.65  46.8
   43.15  41.65  41.75  44.4   48.55  50.4   51.45  53.3   52.95  52.15]
 [ 49.65  50.5   50.75  50.45  49.55  51.1   50.3   50.55  49.9   47.6
   43.95  42.45  42.35  45.1   47.8   50.65  51.85  53.55  54.45  69.8 ]
 [ 51.9   51.45  51.3   50.65  50.25  50.35  50.75  50.    50.1   46.55
   42.8   41.95  42.15  44.95  45.4   48.65  50.6   52.    52.3   53.5 ]
 [ 51.55  50.8   50.65  50.1   49.25  50.6   50.7   50.6   49.6   47.8
   44.45  41.6   41.45  43.25  45.75  48.4   50.8   51.3   52.45  52.55]]

Voila! We can also report on the recast data

In [8]:
print('Total number of elements in OUTPUT data: ', size(griddata))
print('Number of dimensions for OUTPUT data: ',ndim(griddata))
print('SQRT of total number of elements in OUTPUT data: ', sqrt(size(griddata)))
print('The data recast as in a ', sqrt(size(griddata)), ' x ', sqrt(size(griddata)), ' grid array')
Total number of elements in OUTPUT data:  400
Number of dimensions for OUTPUT data:  2
SQRT of total number of elements in OUTPUT data:  20.0
The data recast as in a  20.0  x  20.0  grid array

Undertake image enhancement

With our data in a proper grid (reminiscent of the matrix of a grey-scale photographic image) we can very easily create undertake image enchancement using the algorithms contained in scipy.ndimage (imported into the library as 'nd').

The following code calculates a number of different derivatives from the gephysics data held in 'griddata':

  • Sobel
  • Laplace
  • Prewitt
  • Gaussian
  • Minimum filter
In [9]:
#Calculate derivatives
gridSobel = nd.sobel(griddata)
gridLaplace = nd.laplace(griddata)
gridPrewitt = nd.prewitt(griddata)
gridGaussian = nd.gaussian_filter(griddata,1)
gridMinimum = nd.minimum_filter(griddata,size=(3,3))

Visualize the enhanced image

The enhanced images can be simply plotted in matplotlib

In [10]:
#Plot a derivative
plt.imshow(gridGaussian,origin='lower')
plt.gray()
#show image
cb=plt.colorbar()
cb.set_label('Value Range')
plt.xlabel('GridEast')
plt.ylabel('GridNorth')
plt.suptitle('Raw data')
plt.show()

Fancy image plotting

We can also configure matplotlib to allow multiple plots of the different derivatives so they can be compared.

In [11]:
#The data and derivatives
plt.subplot(1,3,2)
plt.imshow(griddata,origin='lower')
plt.gray()
cb=plt.colorbar()
cb.set_label('Value Range')
plt.xlabel('GridEast')
plt.ylabel('GridNorth')
plt.title('Raw data')


plt.subplot(3,3,1)
plt.imshow(gridSobel,origin='lower')
plt.gray()
plt.title('Sobel filter')

plt.subplot(3,3,3)
plt.imshow(gridLaplace,origin='lower')
plt.gray()
plt.title('Laplace filter')

plt.subplot(3,3,4)
plt.imshow(gridPrewitt,origin='lower')
plt.gray()
plt.title('Prewitt filter')


plt.subplot(3,3,6)
plt.imshow(gridGaussian,origin='lower')
plt.gray()
plt.title('Gaussian filter')

plt.subplot(3,3,7)
plt.imshow(gridMinimum,origin='lower')
plt.gray()
plt.title('3x3 minimum filter')

plt.subplot(3,3,9)
plt.imshow(griddata-gridGaussian,origin='lower')
plt.gray()
plt.title('Data - Gaussian Filter')

plt.suptitle('Raw data and derivatives')
plt.show()

What next

Check out the other training material from DART (when I produce them put a link in here) Change the 'inputURI' file to a different file from the DART repository to see how easy it is to change this for different data files (hint: there's already one there for you to try)

In [11]: