The Nature Conservancy

Detecting Coral Reefs and Scuba Diving Imagery

Background

This proof of concept will be focused on detecting corals, scuba diving equipment, and other reef-related images located in user-defined geospatial locations. We will also explore a potential edge case with skydiving, as colors and gear visually is similar to scuba diving.

First Image: Coral Reef

Our first image is very obviously a photo of a coral reef with wildlife and fish; the keyword reef is included in the JSON response that we obtain from Cognitive Services. To extract similar images from your corpus of files, you would first filter by geospatial location and then search for reef in Cognitive Services JSON responses.

In [7]:
# Define the Subscription key for making API calls.
secret = 'INSERT YOUR API KEY HERE'
In [14]:
# Import required libraries for request headers and json extraction.
import http.client, urllib.request, urllib.parse, urllib.error, base64, json

# Replace the subscription_key string value with your valid subscription key.
subscription_key = secret

# Replace to match your region.

uri_base = 'westcentralus.api.cognitive.microsoft.com'

headers = {
    # Request headers.
    'Content-Type': 'application/json',
    'Ocp-Apim-Subscription-Key': subscription_key,
}

params = urllib.parse.urlencode({
    # Request parameters. All of them are optional.
    'visualFeatures': 'Description,Color,Tags',
    'language': 'en',
})
In [15]:
body = "{'url':'https://timeincsecure-a.akamaihd.net/rtmp_uds/293884104/201703/2681/293884104_5360456295001_5360434467001-vs.jpg?pubId=293884104&videoId=5360434467001'}"

try:
    # Execute the REST API call and get the response.
    conn = http.client.HTTPSConnection('westcentralus.api.cognitive.microsoft.com')
    conn.request("POST", "/vision/v1.0/analyze?%s" % params, body, headers)
    response = conn.getresponse()
    data = response.read()

    # 'data' contains the JSON data. The following formats the JSON data for display.
    parsed = json.loads(data.decode())
    print ("Response:")
    print (json.dumps(parsed, sort_keys=True, indent=2))
    conn.close()

except Exception as e:
    print('Error:')
    print(e)

####################################
Response:
{
  "color": {
    "accentColor": "048AC7",
    "dominantColorBackground": "Brown",
    "dominantColorForeground": "Brown",
    "dominantColors": [],
    "isBWImg": false
  },
  "description": {
    "captions": [
      {
        "confidence": 0.266922154795059,
        "text": "a group of colorful underwater"
      }
    ],
    "tags": [
      "nature",
      "covered",
      "colorful",
      "orange",
      "reef",
      "lot",
      "colored",
      "many",
      "sitting",
      "table",
      "surrounded",
      "painted",
      "colors",
      "fire",
      "old",
      "field",
      "water",
      "large",
      "group",
      "underwater",
      "blue",
      "display",
      "room",
      "hydrant",
      "umbrella",
      "people",
      "standing",
      "street",
      "white"
    ]
  },
  "metadata": {
    "format": "Jpeg",
    "height": 720,
    "width": 1280
  },
  "requestId": "452dd51a-49c7-448b-8c08-1705633a8514",
  "tags": [
    {
      "confidence": 0.8286102414131165,
      "name": "nature"
    },
    {
      "confidence": 0.47426894307136536,
      "name": "colorful"
    },
    {
      "confidence": 0.4733331501483917,
      "name": "orange"
    },
    {
      "confidence": 0.39928770065307617,
      "name": "reef"
    },
    {
      "confidence": 0.28420546650886536,
      "name": "colored"
    },
    {
      "confidence": 0.1980312019586563,
      "name": "surrounded"
    },
    {
      "confidence": 0.17965476214885712,
      "name": "painted"
    }
  ]
}

Second Image: Scuba Diver

The second image is not as straightforward to parse; however, you would be able to deduce that this is a scuba diving photo by searching through geospatially-filtered JSON responses for water sport and underwater, as well as potentially man or woman and swimming.

In [18]:
body = "{'url':'https://www.deeperblue.com/wp-content/uploads/2016/03/AdobeStock_62701813.jpeg'}"

try:
    # Execute the REST API call and get the response.
    conn = http.client.HTTPSConnection('westcentralus.api.cognitive.microsoft.com')
    conn.request("POST", "/vision/v1.0/analyze?%s" % params, body, headers)
    response = conn.getresponse()
    data = response.read()

    # 'data' contains the JSON data. The following formats the JSON data for display.
    parsed = json.loads(data.decode())
    print ("Response:")
    print (json.dumps(parsed, sort_keys=True, indent=2))
    conn.close()

except Exception as e:
    print('Error:')
    print(e)

####################################
Response:
{
  "color": {
    "accentColor": "016BCA",
    "dominantColorBackground": "Blue",
    "dominantColorForeground": "Blue",
    "dominantColors": [
      "Blue"
    ],
    "isBWImg": false
  },
  "description": {
    "captions": [
      {
        "confidence": 0.3619696364682312,
        "text": "a statue of person in a swimming pool"
      }
    ],
    "tags": [
      "sport",
      "swimming",
      "statue",
      "water",
      "small",
      "sitting",
      "air",
      "top",
      "man",
      "holding",
      "flying",
      "yellow",
      "riding",
      "boat",
      "blue"
    ]
  },
  "metadata": {
    "format": "Jpeg",
    "height": 3456,
    "width": 5184
  },
  "requestId": "baf20f4c-7e65-4051-b72c-1db944696017",
  "tags": [
    {
      "confidence": 0.9971852898597717,
      "name": "sky"
    },
    {
      "confidence": 0.9742870926856995,
      "name": "sport"
    },
    {
      "confidence": 0.9645830988883972,
      "hint": "sport",
      "name": "swimming"
    },
    {
      "confidence": 0.9560804963111877,
      "hint": "sport",
      "name": "water sport"
    },
    {
      "confidence": 0.45918408036231995,
      "name": "statue"
    },
    {
      "confidence": 0.013368669897317886,
      "name": "ocean floor"
    }
  ]
}

Image 3: Scuba Diving

Similar to the image above; but ocean floor, underwater, and swimming.

In [19]:
body = "{'url':'http://www2.padi.com/blog/wp-content/uploads/2016/10/scuba-diving-reef-e1476482719834.jpg'}"

try:
    # Execute the REST API call and get the response.
    conn = http.client.HTTPSConnection('westcentralus.api.cognitive.microsoft.com')
    conn.request("POST", "/vision/v1.0/analyze?%s" % params, body, headers)
    response = conn.getresponse()
    data = response.read()

    # 'data' contains the JSON data. The following formats the JSON data for display.
    parsed = json.loads(data.decode())
    print ("Response:")
    print (json.dumps(parsed, sort_keys=True, indent=2))
    conn.close()

except Exception as e:
    print('Error:')
    print(e)

####################################
Response:
{
  "color": {
    "accentColor": "0143C6",
    "dominantColorBackground": "Blue",
    "dominantColorForeground": "Blue",
    "dominantColors": [
      "Blue"
    ],
    "isBWImg": false
  },
  "description": {
    "captions": [
      {
        "confidence": 0.36038400178616214,
        "text": "a group of people in a swimming pool"
      }
    ],
    "tags": [
      "swimming",
      "sitting",
      "table",
      "laptop",
      "man",
      "orange",
      "woman",
      "colorful",
      "group",
      "computer",
      "many",
      "people",
      "holding",
      "blue",
      "water",
      "large",
      "standing",
      "display",
      "underwater"
    ]
  },
  "metadata": {
    "format": "Jpeg",
    "height": 465,
    "width": 995
  },
  "requestId": "dfa2f409-a28b-4b9e-8bdd-8830300c23a8",
  "tags": [
    {
      "confidence": 0.7886989116668701,
      "name": "sport"
    },
    {
      "confidence": 0.5646587610244751,
      "hint": "sport",
      "name": "swimming"
    },
    {
      "confidence": 0.013779879547655582,
      "name": "ocean floor"
    }
  ]
}

Testing 4: Skydiving

Our final test image would be a potential edge case, as skydiving and scuba diving are very visually similar. However, you can see that Cognitive Services detects this is an aerial photo with high confidence. This would not be filtered as a potential scuba photo.

In [17]:
body = "{'url':'http://www.skydiveoz.com.au/wp-content/uploads/2016/12/B-Licence-Pic-1.jpg'}"

try:
    # Execute the REST API call and get the response.
    conn = http.client.HTTPSConnection('westcentralus.api.cognitive.microsoft.com')
    conn.request("POST", "/vision/v1.0/analyze?%s" % params, body, headers)
    response = conn.getresponse()
    data = response.read()

    # 'data' contains the JSON data. The following formats the JSON data for display.
    parsed = json.loads(data.decode())
    print ("Response:")
    print (json.dumps(parsed, sort_keys=True, indent=2))
    conn.close()

except Exception as e:
    print('Error:')
    print(e)

####################################
Response:
{
  "color": {
    "accentColor": "133A7B",
    "dominantColorBackground": "White",
    "dominantColorForeground": "White",
    "dominantColors": [
      "White"
    ],
    "isBWImg": false
  },
  "description": {
    "captions": [
      {
        "confidence": 0.8847015046488436,
        "text": "a man flying through the air on a cloudy day"
      }
    ],
    "tags": [
      "outdoor",
      "air",
      "flying",
      "water",
      "cloudy",
      "surfing",
      "clouds",
      "blue",
      "top",
      "riding",
      "man",
      "high",
      "ocean",
      "wave",
      "jumping",
      "skiing",
      "snow",
      "plane",
      "board",
      "trick",
      "large",
      "colorful",
      "doing",
      "standing",
      "boat",
      "white",
      "airplane",
      "field"
    ]
  },
  "metadata": {
    "format": "Jpeg",
    "height": 1307,
    "width": 2321
  },
  "requestId": "231ebc16-cc50-419f-81a9-3e1e4b1653da",
  "tags": [
    {
      "confidence": 0.9999339580535889,
      "name": "sky"
    },
    {
      "confidence": 0.9850667715072632,
      "name": "outdoor"
    },
    {
      "confidence": 0.5827982425689697,
      "name": "air"
    },
    {
      "confidence": 0.4614807963371277,
      "name": "cloudy"
    },
    {
      "confidence": 0.4217190742492676,
      "hint": "sport",
      "name": "surfing"
    },
    {
      "confidence": 0.404719740152359,
      "name": "clouds"
    },
    {
      "confidence": 0.2684488892555237,
      "name": "high"
    },
    {
      "confidence": 0.20249073207378387,
      "name": "day"
    }
  ]
}