Title: Cameras, Eyes, and Color Author: Thomas M. Breuel Institution: UniKL
from IPython.core.display import Image
from pylab import *
def fig(s): return Image(filename="Figures-intro/"+s+".png")
def figs(*args,**kw):
rs = kw.get("rows",1)
cs = kw.get("cols",len(args))
for i,f in enumerate(args):
subplot(rs,cs,i+1)
axis("off")
imshow(imread("Figures-intro/"+f+".png"))
from scipy.ndimage import filters
## LCD display
figs("rgb-color-display")
Images:
# RGB images
image = zeros((10,10,3))
image[3,4,:] = (0.0,1.0,0.0)
image[4,3,:] = (0.0,0.0,1.0)
image[7,7,:] = (1.0,1.0,0.0)
imshow(image,interpolation='nearest')
<matplotlib.image.AxesImage at 0x39c6b7d0>
People started color photography in the 19th century. For this, they used black-and-white film plus color filters.
# color photograph with filters
# Sergey Prokudin-Gorsky
fig("color-channels-historical")
(color photography with b/w film)
(trichromatic vision)
Questions:
Answers:
# Maxwell-Helmholtz color mixing
figs("maxwell-helmholtz")
(Tristimulus Theory)
The observation that all colors humans perceive can be decomposed into three stimuli is called tristimulus theory or trichromacy.
These experiments predicted the existence of three different visual cells in the human retina (later confirmed).
(Psychophysics)
Experiments like the Maxwell-Helmholtz experiment study human perception quantitatively.
These kinds of experiments are called psychophysics. Psychophysics forms the basis of many compression algorithms.
Common experiments involve:
(Definition of Color)
Definition Farbe ist diejenige Gesichtsempfindung eines dem Auge des Menschen strukturlos erscheinenden Teiles des Gesichtsfeldes, durch die sich dieser Teil bei einäugiger Beobachtung mit unbewegtem Auge von einem gleichzeitig gesehenen, ebenfalls strukturlosen angrenzenden Bezirk allein unterscheiden kann. (DIN 5033)
Note that this definition does not involve "wavelengths" or anything about physics.
Color is defined purely as a human perceptual phenomenon.
# camera model
fig("camera-model")
# camera model
figs("camera-model-2")
(image formation)
Image resulting from an illuminated, reflecting surface:
$$ \hbox{intensity-reaching-camera} = \hbox{intensity-of-light-source} \times \hbox{reflectance} $$That is:
(spatial dependency)
(reflectance values)
Typical reflectance values are:
(camera and human sensors)
Taking an image through three filters gives us RGB information everywhere. That only works for subjects that don't move.
Human eyes and cameras don't work that way. Instead of taking three full pictures, they take a single picture through multiple filters simultaneously.
Cameras usually work the same way. For cameras, the most commonly used pattern is the Bayer pattern.
# Bayer Pattern
fig("bayer-pattern")
# CCD sensor
fig("ccd-sensor")
# CCD lenslet
fig("lenslet")
The human eye has a somewhat similar structure to a camera. It is composed of a lens and a retina ("color sensor").
# the human eye
figs("eye2")
# what you think you see
figs("see1")
What the human eye records is actually very different from a camera. Only the fovea is sharp and in color.
# what your eye records
figs("see2")
# structure of the human retina
figs("retina")
# human retina
figs("human-retina")
# human cones
figsize(10,20)
figs("human-cones")
(summary human eye)
We talked about color as a perceptual phenomenon.
How does color relate to spectra and wavelengths?
## Prism
fig("prism")
Light is part of the electromagnetic spectrum. The visible part of the EM spectrum is between ultraviolet and infrared. The spectrum is indexed by the wavelength or frequency of light.
# Electromagnetic Spectrum
fig("spectrum")
Are all "pure" colors represented in the spectrum?
# Purple
fig("newton-purple")
Some pure colors do not exist in the spectrum, namely purple or magenta. Overlapping the red and the blue ends of the spectrum (using two prisms) generates a new color.
(Spectra and Color Perception)
We know:
Question:
In a spectrum, we plot for each wavelength how much of light of that wavelength is present. In physics, we actually measure how much energy there is contained per unit area and wavelength interval.
# Spectrum
fig("solar-spectrum")
The receptors in the human eye have different sensitivities at different wavelengths. At IR wavelengths, they have no sensitivity = zero sensitivity. At some wavelength, they have peak sensitivity = sensitivity of 1.0.
# Spectral Sensitivities
fig("sensitivity-curves")
(Receptor Response)
To get the response of a receptor to incoming light...
(discretized spectra)
If we approximate the integral with a sum:
$$ R = \hbox{const} \cdot \sum_{\lambda=300}^{700} S(\lambda) \cdot I(\lambda) $$If we treat $S$ and $I$ as 400-dimensional vectors, we can also write this just as a dot product:
$$ R = \hbox{const} \cdot \vec{S} \cdot \vec{I} $$Let's do this as an actual calculation. This is also a good review of simple linear algebra.
Let's represent these spectra by representing the amount of light within each band of 1 nm. This is an approximation to the continuous spectra found in nature.
Here, we're just using Gaussians to approximate the spectral response curves of the photo receptors; of course, this isn't very realistic. Think of it as some "alien eye".
def g(s,c): return exp(-((s-c)/80.0)**2)
wavelengths = arange(300,700)
figsize(6,4)
eR = g(wavelengths,564); plot(wavelengths,eR,color='r')
eG = g(wavelengths,534); plot(wavelengths,eG,color='g')
eB = g(wavelengths,420); plot(wavelengths,eB,color='b')
l = wavelengths
#:h:1.5
Note that a spectrum and a spectral sensitivity curve is now a 400-dimensional vector. You can think of the distribution of light intensities by wavelengths (i.e., spectra) as very high dimensional vectors like this.
Let's generate some (arbitrary) spectrum for a mostly reddish light source.
light = zeros(len(l))
light[150] = 0.2; light[200] = 0.1; light[310] = 0.7
light = filters.gaussian_filter(light,10.0,mode='constant')
plot(l,light)
[<matplotlib.lines.Line2D at 0x4762a150>]
We now obtain the response of each of the RGB sensors by multiplying, at each wavelength, the sensitivity of the sensor at that wavelength with the amount of light actually present at that wavelength, and sum it all up.
This can be written concisely as a dot product.
rgb = (dot(eR,light),dot(eG,light),dot(eB,light))
print rgb
(0.58067310677164286, 0.4375692991745056, 0.21208597770753429)
We can put together these three dot products into a matrix multiplication.
The "eye sensitivity matrix" maps the high-dimensional space of spectra into the low dimensional space of RGB values.
esm = array([eR,eG,eB])
print esm.shape
print dot(esm,light)
(3, 400) [ 0.58067311 0.4375693 0.21208598]
Note that we properly talk about RGB values only when talking about images and monitors.
The actual stimulus values exist in a different space and are often called XYZ values.
In the real world, most light is reflected from surfaces. What happens with spectra is similar to what happens with eye sensitivity curves:
## surface colors and perception
#:h:2.5
figsize(10,8)
figs("reflected-light")
Let's start by constructing an illuminant for a sort-of wide spectrum illuminant (don't worry about the details of this construction, just look at the final curve).
# constructing an illuminant
#:h:1.5
illum = zeros(len(l))
illum[150] = 0.3; illum[200] = 0.3; illum[310] = 0.3
illum = filters.gaussian_filter(illum,50.0,mode='constant')
illum /= amax(illum)
figsize(6,6)
plot(l,illum)
[<matplotlib.lines.Line2D at 0x5af72b90>]
Next, let's construct a surface reflectance for a reddish object (again, these values are just arbitrary).
# constructing a surface reflectance
refl = zeros(len(l))
refl[150] = 0.2; refl[200] = 0.1; refl[310] = 0.7
refl = filters.gaussian_filter(refl,10.0,mode='constant')
refl /= amax(refl)
plot(l,refl)
[<matplotlib.lines.Line2D at 0x19cbffd0>]
# the reflected spectrum and RGB values
plot(l,illum)
plot(l,refl)
# the reflected spectrum
plot(l,illum*refl)
# the photoreceptor activations
print dot(esm,illum*refl)
[ 13.54537488 11.23021505 6.76528753]
Note:
There are many 400-dimensional vectors that project onto the same RGB values.
# metameric spectra
figsize(10,4)
figs("metameric-patch","metamerism")
# a color
#:h:1
figsize(6,4)
color1 = zeros(len(l))
color1[100] = 1.0
color1[200] = 0.3
rgb1 = dot(esm,color1)
print "rgb value",rgb1
plot(l,color1)
rgb value [ 0.17314586 0.31089347 1.0497769 ]
[<matplotlib.lines.Line2D at 0x19cc38d0>]
# constructing a metameric color for color1
color2 = dot(pinv(esm),rgb1)
rgb2 = dot(esm,color2)
# note that we can construct others; pinv is convenient
print "rgb1",rgb1
print "rgb2",rgb2
plot(l,color1/amax(color1),label="rgb1")
plot(l,color2/amax(color2),label="rgb2")
legend()
rgb1 [ 0.17314586 0.31089347 1.0497769 ] rgb2 [ 0.17314586 0.31089347 1.0497769 ]
<matplotlib.legend.Legend at 0x57a85090>
(metameric colors)
The example shows that we can have two completely different spectra that project to the same stimulus values.
Note that rgb1
is composed of narrow-band signals (e.g., spectral lines; example: sodium lamps), while rgb2
is broadband (example: candle, fire).
# spectrally pure colors
#:h:1.5
samples = []; colors = []
for i in range(len(l)):
s = zeros(len(l))
s[i] = 1.0
xyz = dot(esm,s)
samples.append([xyz[0]/sum(xyz),xyz[1]/sum(xyz)])
colors.append(xyz/amax(xyz))
ylim((0,1)); xlim((0,1))
for i,c in enumerate(colors): plot(samples[i][0],samples[i][1]/0.53,'.',c=c)
# the space of possible colors
samples = []; colors=[]
for i in range(10000):
s = 1.0*(rand(len(l))>0.99)
xyz = dot(esm,s)
if sum(xyz)<1e-6: continue
samples.append([xyz[0]/sum(xyz),xyz[1]/sum(xyz)])
colors.append(xyz/amax(xyz))
ylim((0,1)); xlim((0,1))
for i,c in enumerate(colors): plot(samples[i][0],samples[i][1]/0.53,'.',c=c)
# The CIE XYZ Color Space
#:h:2.5
fig("cie-colorspace")
# RGB Pixel
fig("rgb-pixel-lcd")
# LCD Spectra
fig("lcd-spectra")
Let's construct a coarse approximation to this numerically.
# Simple Monitor Model
#:h:1.5
lB = zeros(len(l)); lB[100:120] = 0.5
lG = zeros(len(l)); lG[180:210] = 1.0
lR = zeros(len(l)); lR[350:400] = 0.6
plot(l,lB,color='b')
plot(l,lG,color='g')
plot(l,lR,color='r')
lcd = array([lR,lG,lB]).T
Let's now plot the original color space and overlay the space of colors realizable by the monitor.
# Monitor Gamut vs Color Space
lsamples = []; lcolors = []
for i in range(10000):
p = rand(3)
s = dot(lcd,p)
rgb = dot(esm,s)
lsamples.append([rgb[0]/sum(rgb),rgb[1]/sum(rgb)])
lcolors.append(rgb)
lsamples = array(lsamples)
ylim((0,1)); xlim((0,1))
for i,c in enumerate(colors): plot(samples[i][0],samples[i][1]/0.53,'.',c=(.5,.5,.5))
for i,c in enumerate(lcolors): plot(lsamples[i,0],lsamples[i,1]/0.53,'.',c=c/amax(c))
The monitor gamut is a triangle while the color space is a weird roundish thing.
The reason the gamut is a triangle is because it is a convex linear combination of the spectra of the three primary phosphors of the monitor.
The reason the color space itself is not a triangle is because it is the projection of a very high dimensional convex combination into two dimensional space.
Here is what the color space and monitor gamut look like in the more usual coordinates.
# CIE Color Space and Typical Gamut
#:h:2.5
fig("gamut")
When we display RGB images on a monitor, the following happens:
(Gamut)
Above, we analyzed the relationship between wavelengths, spectra, and color perception.
It turns out that that is far from sufficient for characterizing color perception. Color perception also depends on context.
There are a number of illusions illustrating this. Let's look at two of them.
# Color Constancy
figs("cc1","cc2")
# Color Constancy Revealed
figs("cc1","cc2","cc3","cc4",rows=2,cols=2)
Mechanisms similar to color constancy also play a role in the perception of intensities.
# Adelson Illusion
figs("aillusion")
(Mechanisms for Color Constancy)
How do these mechanisms works? It's not completely understood, but you can simulate color constancy by...
(Summary)
(Summary 2)
(Summary 3)