CSE 559A: Computer Vision


Use Left/Right PgUp/PgDown to navigate slides

Fall 2018: T-R: 11:30-1pm @ Lopata 101

Instructor: Ayan Chakrabarti (ayan@wustl.edu).
Course Staff: Zhihao Xia, Charlie Wu, Han Liu

http://www.cse.wustl.edu/~ayan/courses/cse559a/

Sep 27, 2018

General

  • Almost all PSET 1 submissions in !
    • Try not to use all your late days on the first problem set.
  • Reminder: PSET 2 out.
    • Get started early !
  • Start thinking about final project.
    • Will discuss next week.

Some Clarifications

Intensity, Distance, and sub-tended angles

  • Does an object appear brighter if we're closer to it ?
  • Consider taking an image of an object in a room where everything else is completely black.
  • Take an image, and then move your camera closer, and take another image.
  • The average intensity of pixels, that are on the object, won't increase.
  • But now more pixels in your image will correspond to the object. And so average / total intensity across all pixels in the image will increase.
  • So your camera is receiving more total light, but the light along an individual ray doesn't change.

Some Clarifications

Diffuse Lighting vs Diffuse Surface

  • Diffuse lighting means the incoming light from light sources is nearly the same.
  • Diffuse Surface or Diffuse Reflection or Lambertian Reflection: Light reflected by the surface is the same in all directions.
  • A Diffuse Surface under a Non-diffuse (e.g., point) light source will have sharp shading variations.
  • But the shading will be the same from all viewing directions.
  • A non-diffuse (specular / shiny) surface under relatively diffuse light can still have sharp variations in appearance based on viewing direction.

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

Photometric Stereo++

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

General Shading

Natural Shading

Natural Shading

Natural Shading

Geometry

Geometry

Geometry

\[\text{3D}~~~(x,y,z) \Rightarrow \left(-f\frac{x}{z}, -f \frac{y}{z}\right)~~~\text{2D}\]

  • The division is annoying, makes projection non-linear.
  • Can no longer use matrices / linear operations to relate co-ordinates.
  • But we like matrix operations !

Solution: Homogeneous Co-ordinates

Homogeneous Co-ordinates

Book-keeping trick !

  • 2D Cartesian Co-ordinates: \((x,y)\)
  • 2D Homogeneous Co-ordinates: \((\alpha~x,\alpha~y,\alpha)\)
  • Cartesian to Homogeneous: \((x,y) \rightarrow (\alpha x, \alpha y, \alpha)\)
    • When \(\alpha=1\), this is called "augmented": \((x,y,1)\)
  • Homogeneous to Cartesian: \((x',y',\alpha) \rightarrow \left(\frac{x'}{\alpha},\frac{y'}{\alpha}\right)\)
  • A whole family of homogeneous co-ordinates map to the same cartesian co-ordinate
    • Over-parameterization of a 2D point
    • Denote this equality by \(\sim:~~(\alpha_1 x, \alpha_1 y, \alpha_1) \sim (~\alpha_2 x, \alpha_2 y, \alpha_2)\)
  • Space of 2D Homogeneous co-ordinates denoted as \(\mathbb{P}^2 = \mathbb{R}^3 - (0,0,0)\)
  • Note that \((x,y,0)\) is defined. In cartesian co-ordinates, it is the point at infinity
    along the line joining \((0,0)\) to \((x,y)\).
  • 3D Homogeneous Co-ordinates: \((x,y,z) \Rightarrow (\alpha x, \alpha y, \alpha z, \alpha)\)

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

  • Turned non-linear perspective projection into a linear operation.
  • Here's a different projection matrix:

\[P_{2d} = \left[\begin{array}{cccc}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{array}\right] P_{3d}\]

What does this represent ?

\[(x,y,z) \rightarrow (\color{red}{?}, \color{red}{?})\]

Homogeneous Co-ordinates

  • Turned non-linear perspective projection into a linear operation.
  • Here's a different projection matrix:

\[P_{2d} = \left[\begin{array}{cccc}1 & 0 & 0 & 0\\0 & 1 & 0 & 0\\0 & 0 & 0 & 1\end{array}\right] P_{3d}\]

What does this represent ?

\[(x,y,z) \rightarrow (x,y)\]

Orthographic Projection

Homogeneous Co-ordinates

Homogeneous Co-ordinates

Homogeneous Co-ordinates

  • Also useful to represent translation, rotation, skew in addition to projection
  • Learn to chain together all these operations to:
    • Relate points in 3D to points in image
    • Verify angles, metric lengths from calibration targets, ...
    • Relate points in two images from different cameras

Homogeneous Co-ordinates

  • Useful way to think about 2-D Homogeneous Co-ordinates \(\mathbb{P}^2\)

"Rays" in \(\mathbb{R}^3\)

Homogeneous Co-ordinates

  • Useful way to think about 2-D Homogeneous Co-ordinates \(\mathbb{P}^2\)

"Rays" in \(\mathbb{R}^3\)

  • Cartesian form is "intersection" with plane \(z=1\).
  • \((x,y,0)\) are forms that are parallel to the \(z=1\) plane, intersect at infinity.
  • 3-D Homogeneous Co-ordinates are rays in 4D, intersection with a hyper-plane.

Quick word about notation

  • We assume vectors are column vectors.
  • \(p' = [x,y]\) implies a 2-D row vector (of size \(1\times 2\))
  • \(p = [x,y]^T = \left[\begin{array}{c}x\\y\end{array}\right]\) implies a 2-D column vector (of size \(2\times 1\))

Homogeneous Co-ordinates: 2D

Lines

Equation of a line in 2D:

\[ax+by+c = 0\]

Let \(p=[\alpha~x,\alpha~y,\alpha]^T\) be homogeneous co-ordinates of a point \((x,y)\). Then,

What is the equation ? \[l^Tp = 0,~~~l = [a,b,c]^T\]

Interestingly, \(l\) is also defined "upto scale": \(~l' = [\beta a, \beta b, \beta c]^T\) describes the same line as \(l\).

Homogeneous Co-ordinates: 2D

Lines

Since \(l^Tp = 0\) for all points that lie on a line:

Homogeneous Co-ordinates: 2D

Lines

Given two points \(p_1\) and \(p_2\), what is the homogeneous vector for the line joining them ?

It has to be an \(l\) such that \(l^Tp_1 = 0\) and \(l^Tp_2 = 0\).

Is that sufficient to determine \(l\) ?

Yes. Because, only need \(l\) upto scale.

Solution given by: \(l = p_1 \times p_2\) (Vector Cross-product)

Recap: Writing \(u = [u_1,u_2,u_3]^T = u_1 \hat{i} + u_2 \hat{j} + u_3 \hat{k}\), and \(u = [v_1,v_2,v_3]^T = v_1 \hat{i} + v_2 \hat{j} + v_3 \hat{k}\)

\[u \times v = \text{det} \left|\begin{array}{ccc}\hat{i}& \hat{j}& \hat{k}\\u_1 & u_2 & u_3 \\v_1 & v_2 & v_3\end{array} \right| = (u_2v_3-u_3v_2)\hat{i} + (u_3v_1-u_1v_3)\hat{j} + (u_1v_2-u_2v_1)\hat{k}\]

\[= [(u_2v_3-u_3v_2),(u_3v_1-u_1v_3),(u_1v_2-u_2v_1)]^T~~~~~~~~~~\]

Homogeneous Co-ordinates: 2D

Lines

Given two lines \(l_1\) and \(l_2\), what is the homogeneous co-ordinate vector \(p\) for the point of their intersection ?

Same idea: \(l_1^T p = p^T l_1 = 0\) and \(p^T l_2 = 0\)

\(p = l_1 \times l_2\)

  • Cross product between two points gives us the line between them
  • Cross product between two lines gives us the point common to both
  • What happens if \(l_1\) and \(l_2\) are parallel ?

Answer: Third co-ordinate of \(l_1\times l_2\) is 0. Point at infinity.

Homogeneous Co-ordinates: 2D

Transformations

  • Translation:
    • \(x' = x-c_x\), \(y'=y-c_y\)
  • Express as, \(p' = T~p\) where \(T\) is a \(3\times 3\) matrix.

Homogeneous Co-ordinates: 2D

Transformations

  • Translation:
    • \(x' = x-c_x\), \(y'=y-c_y\)
  • Verify this works for any scaled version of \(T\) above
  • Verify this works for \(p=[\alpha x,\alpha y,\alpha]\), for any \(\alpha \neq 0\)

Homogeneous Co-ordinates: 2D

Transformations

  • Rotation Around the Origin
    • \(x' = x \cos \theta - y \sin \theta\), \(x \sin \theta + y \cos \theta\)

\[p' = \left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\\sin\theta&\cos\theta&0\\0&0&1\end{array}\right] p \]

  • Rotation around a different point \(c_x,c_y\) ?

\[p' = \left[\begin{array}{ccc}1 & 0 & c_x\\0&1&c_y\\0&0&1\end{array}\right] \left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\\sin\theta&\cos\theta&0\\0&0&1\end{array}\right] \left[\begin{array}{ccc}1 & 0 & -c_x\\0&1&-c_y\\0&0&1\end{array}\right] p \]

Homogeneous Co-ordinates: 2D

Transformations

  • Euclidean Transformation \[p' = \left[\begin{array}{cc}R & t\\0^T & 1\end{array}\right]p\]
    • \(R\) is a \(2\times 2\) rotation matrix, \(R^TR=I\)
    • \(t\) is a \(2\times 1\) translation vector
    • \(0^T\) here represents a \(1\times 2\) row of two zeros
    • Preserves orientation, lengths, areas

If \(R^TR = I\), is \(R\) always of the form:

\[R = \left[\begin{array}{cc}\cos \theta & -\sin \theta\\\sin \theta & \cos\theta\end{array}\right]\]

Homogeneous Co-ordinates: 2D

Transformations

  • Euclidean Transformation \[p' = \left[\begin{array}{cc}R & t\\0^T & 1\end{array}\right]p\]
    • \(R\) is a \(2\times 2\) rotation matrix, \(R^TR=I\)
    • \(t\) is a \(2\times 1\) translation vector
    • \(0^T\) here represents a \(1\times 2\) row of two zeros
    • Preserves orientation, lengths, areas

If \(R^TR = I\), is \(R\) always of the form:

\[R = \left[\begin{array}{cc}-\cos \theta & -\sin \theta\\-\sin \theta & \cos\theta\end{array}\right]\]

Homogeneous Co-ordinates: 2D

Transformations

  • Euclidean Transformation \[p' = \left[\begin{array}{cc}R & t\\0^T & 1\end{array}\right]p\]
    • \(R\) is a \(2\times 2\) rotation matrix, \(R^TR=I\)
    • \(t\) is a \(2\times 1\) translation vector
    • \(0^T\) here represents a \(1\times 2\) row of two zeros
    • Preserves orientation, lengths, areas
  • Isometries
    • \(R^TR=I\) can also correspond to reflections \[R = \left[\begin{array}{cc}1&0\\0&-1\end{array}\right]\]
    • If we allow this in \(R\) above, more general than euclidean
    • Preserves lengths, areas, but not orientation.

Homogeneous Co-ordinates: 2D

Transformations

What about scaling ?

Allow uniform scaling \(s\) along both co-ordinates: \[p' = \left[\begin{array}{cc}sR & t\\0^T & 1\end{array}\right]p\]

Called a similarity: preserves ratio of lengths, angles.

Homogeneous Co-ordinates: 2D

Transformations

Affine Transformation

\[p' = \left[\begin{array}{cc}A & t\\0^T & 1\end{array}\right]p\]

where \(A\) is a general invertible \(2\times 2\).

Preserves ratios of areas, parallel lines stay parallel.

Prove that parallel lines stay parallel.

  • Consider the homogeneous vector \(q\) for intersection of two lines that are parallel.
  • The third co-ordinate of \(q\) is 0, because the lines don't intersect.
  • The affine transform doesn't change the third co-ordinate.
  • Hence, the lines still intersect at infinity after the transformation.

Homogeneous Co-ordinates: 2D

Most general form:

\[p' = H p\]

where \(H\) is a general invertible \(3\times 3\) matrix.

  • Called a projective transform or homography.
  • All bets are off! Parallel lines can now intersect. Maps quadrilaterals to quadrilaterals.
  • Defined upto scale. So 8 degrees of freedom.
  • Hierarchy of Transforms
    • Translation (2 dof) < Euclidean (3 dof) < Affine (6 dof) < Homography (8 dof)
  • Defines mapping of co-ordinates of corresponding points in two images taken from different views:
    • If all corresponding points lie on a plane in the world.
    • If only the camera orientation has changed in two views (center is at the same place)