Name: Chpt_7_opticalFlow_2
Uploaded: 2022-04-11
Duration: 4 min 59 s
Description: . . Optical Flow. From images to videos. • A video is a sequence of frames captured over time • Now our image data is a function of space (x, y) and time (t).

. . Optical Flow. From images to videos. • A video is a sequence of frames captured over time • Now our image data is a function of space (x, y) and time (t). 

Scene 1

• A video is a sequence of frames captured over time 
• Now our image data is a function of space (x, y) and time (t)

• Optical flow: Recover image motion at each pixel from spatio‐temporal image brightness variations 
(optical flow)

• Feature‐tracking: Extract visual features (corners, textured areas) and “track” them over multiple 
frames.

Let us begin with a high-level understanding of optical flow. Optical flow is the motion of objects 
between consecutive frames of sequence, caused by the relative movement between the object and 
camera.

• Motion vector (u,v), i.e. image displacement in x and y directions between two consecutive frames. 
• Motion Flow 
• Displacement Vector 
• Disparity (general case)

Scene 2

. . • Optical Flow Applications. – Motion based segmentation – Structure from Motion(3D shape and Motion) – Alignment (Global motion compensation). 

Scene 3

– Motion based segmentation 
– Structure from Motion(3D shape and Motion) 
– Alignment (Global motion compensation)

 For brightness constancy assumption, the optical flow equation is:

. . . The problem of optical flow may be expressed as:. 

Scene 4

The problem of optical flow may be expressed as:

Between any consecutive frames, we can express the image intensity (I) as a function of space (x, y) and time 
(t). In other words, if we take the image I(x, y, t) at time (t) and move image pixels by (dx, dy) over (t) we obtain 
the new image defined by:

This Brightness Constancy Assumption states that the intensities of a pixel in the first and the second 
frame are the same due to a short duration between the frames. It can be written using the following 
mathematical expression. 
Use Taylor Series Approximation for the RHS:

Then, After neglecting the higher order terms and simplifying the above expansion further, we can 
write the new expression as:

 
Divide by (dt) to derive the optical flow equation:

Here, we can say that every pixel with coordinates (x,y) in the first image, moves along the motion 
vector (u,v) where u is the pixel’s velocity in the x direction and v is the pixel’s velocity in 
the y direction, respectively. 
 
Furthermore, since we have sampled a video sequence for our example, we can say that we are 
working at discrete times and not continuous time. Thus, we can set dt=1 and arrive at the equation 
below.

. .  Optical flow interpretation:. The Optical Flow equation has essentially two unknowns.. 

Scene 5

The Optical Flow equation has essentially two unknowns.

1. Velocity of each pixel in the x-direction, represented by u 
2. Velocity of each pixel in the y-direction, represented by v

However, if you notice, we have an unconstrained system with two unknown variables and only one 
equation. If we were to better understand this situation, we should study the (u,v) plot, which is 
nothing but an equation of a line.

Compute three derivatives ?? , ?? ??? ?? for pixels, then use v-equation to compute optical flow (u, v) 
which can be any point on the line.

We don’t know a unique value of optical flow because v-equation is out of constraint; there is one 
equation with two unknown.

Suppose we assume the optical flow is with a vector (u1, v1) as shown in above diagram, then it can be 
analyzed it into two vectors d (normal vector) and p (parallel vector). If the optical flow has another 
value with vector (u2, v2), the two vector are share with same normal vector (d) with different parallel 
vectors (p).

p(parallel flow) — is unknown
d(normal flow) =
fX2

. . Horn & Schunck represent this mathematically through optimization or cost function as following. 

Scene 6

Horn & Schunck represent this mathematically through optimization or cost function as following

In this cost function, the first term on the right hand side shows how well the pixels fit the Brightness 
constancy assumption. To this, we add some factor λ (regularizer) multiplied by the value that 
represents smoothness. In simpler terms, our aim with the above equation is to minimize the value for 
brightness constraint and the value for smoothness constraint.

There are two equations and two unknown which can be solve it.

In view of programming, we use Derivative Masks (Roberts):

To calculate motion for two frames, we use 2x2 derivative masks with 4 differences.

If we use 3x3 masks this make number of differences (6)  for both fx, fy, but 9 for ft. for that must 
normalize the weight by divide on 6 for first two derivatives and by 9 for ft. in addition it is require 
three frames to estimate the motion.

Apply first masks to 1st  image(frame) and second mask to 2nd  image then add them to responses to get 
fx, fy, ft.

n JO ue!selden
0 = (atv))? + y (f + + nil)
smnopo leuogeueA)
Aouusuoo
Jillel)suoo ssalltpootus
+ Y)? + + nXJ)}

. . (fxu + + + — 11m,) = O discrete version. ?ℎ??? ??? ????????? ?ℎ? ??????? ??? ????ℎ???ℎ??? ??????, ℎ?? ??? ?????????? ???, ???. 

Scene 7

(fxu + + + — 11m,) = O
discrete version

?ℎ??? ??? ????????? ?ℎ? ??????? ??? ????ℎ???ℎ??? ??????, ℎ?? ??? ?????????? ???, ???

There two equations with two unknowns, by solving them we get:

 K=0 
 Initialize uk, vk, in the beginning both u and v are zeros. 
 Repeat until some error measure is satisfied (converges)

In each iteration calculate fx, fy, ft, uav and vav then compute

The following two black and white images, the shape in the middle is shifted to right one pixel

. . . The result of optical flow calculation after one iteration and 10 iterations are shown below.. 

Scene 8

The result of optical flow calculation after one iteration and 10 iterations are shown below.

As you notes the optical flow is more clear and strong after 10 iterations.

Original the idea to compute disparity in stereo vision, as we know the two eyes construct two images 
that is one of them is shifted in x-direction, but this method is work also for calculating optical flow.

It is simple method and depend on the least square error in derivatives fx, fy, and  ft.

 As we mention before that above equation is out of constraint, two unknowns in one equation for

that researchers care about how can solved it. Lucas & Kanade proposed the following:

Consider 3 by 3 window around the current pixel in both images (two consequence frams) , then 
compute optical flow for all 9-pixels we get:

. . To calculate (u and v) it must find A-1 which is impossible because A is not square matrix. Then to find solution used Pseudo Inverse.. 

Scene 9

To calculate (u and v) it must find A-1 which is impossible because A is not square matrix.  Then to 
find solution used Pseudo Inverse.

Au = ft
Multiply both sides by transpose of A
ATAu = ATft
Tft
Pseudo Inverse
find (u,v) s.t.
Least Squares Fit

. . . Then. Eftll+Ef Efuf„ -Ef„fn = -ELL -Efyrfä. 

Scene 10

Eftll+Ef
Efuf„
-Ef„fn
= -ELL
-Efyrfä

. . . Lucas-Kanade without pyramids Fails in areas of large motion. 

Scene 11

Lucas-Kanade
without pyramids
Fails in areas of large
motion

. . Comments. • Horn‐Schunck and Lucas‐Kanade optical methods work only for small motion.. 

Scene 12

• Horn‐Schunck and Lucas‐Kanade optical methods work only for small motion.

• If object moves faster, the brightness changes rapidly, 2x2 or 3x3 masks fail to estimate 
spatiotemporal derivatives.

• Pyramids can be used to compute large optical flow vectors.

Chpt_7_opticalFlow_2

Scene 1 (0s)

Scene 2 (32s)

Scene 3 (38s)

Scene 4 (56s)

Scene 5 (1m 46s)

Scene 6 (2m 34s)

Scene 7 (3m 15s)

Scene 8 (3m 38s)

Scene 9 (4m 12s)

Scene 10 (4m 27s)

Scene 11 (4m 35s)

Scene 12 (4m 42s)