|
|
|
|
To obtain the visual information that can be
extracted from the spatial and temporal changes occurring in an image
sequence. |
|
Not so much to detect the changes induced by
motion but to measure and use them to recover the 3-D structures in motion |
|
|
|
|
Ullman’s counter-rotating cylinders
demonstration |
|
|
|
|
Our visual system can recover the shapes of
unknown structures simply from the way their appearances change in the
image. |
|
|
|
|
|
|
Correspondence problem: Which elements of a
frame correspond to which elements of the next frame of the sequence? |
|
Structure from motion or reconstruction problem:
The task is to recover 3D structure from the measurements supplied by the
correspondence task. |
|
|
|
|
Similar and closer items more likely correspond. |
|
Human visual system incorporates a permanent
table of similarities by which the similarities and dissimilarities in the
various parameters are compared. |
|
Affinity measure: based on 2D measurements. |
|
|
|
|
Gestaltist approach: maximize overall
similarity. |
|
Movements of wholes are of critical importance. |
|
The phenomenon cannot possibly be explained in
purely local way. |
|
Intractable in computational perspective |
|
Mathematical ignorance |
|
|
|
|
|
How is the correspondence established? |
|
Largely independent of shape and form? |
|
Much more complex interpretation of wholes are
involved before frames are compared? |
|
Experiments rule out the second alternative |
|
|
|
|
The observers see the central ring rotating one
way and the inner and outer rings rotating the other. |
|
|
|
|
Conclusion: Matching is carried out locally |
|
|
|
|
|
|
In (a) and (b) A goes to B. |
|
Conclusion: Correspondence is governed by the
motions of constituent elements, not by complete forms or shapes. |
|
|
|
|
Conclusion: Matching can take place between
higher-order borders or tokens even when the constituents do not match. |
|
Does not comprise with Ullman’s conclusions. |
|
|
|
|
Conclusion: 3D measures were irrelevant to the
correspondence process. 2D configurations are sufficient. |
|
|
|
|
Smoothness of motion was determined entirely by
perceived 3D distance rather than by objective. (Attneave and Block 1973) |
|
Inconsistent with Ullman’s claims. |
|
Smoothness and correspondence strength are
different phenomena.(Ullman 1978 experiment 5) |
|
|
|
|
|
Ullman’s theory: |
|
Correspondence strength (CS) |
|
The affinity between each pairing is measured |
|
Local interactions take place on these |
|
The interactions weakens the CS when splitting
or fusion occurs |
|
|
|
|
|
|
|
Ullman’s Assumptions |
|
Independent pairing decisions |
|
The set of pairings should cover both sets of
tokens. |
|
The projected velocities are small rather than
large. |
|
Probabilistic approach( Maximum Likelihood
solution). |
|
|
|
|
|
Marr’s critique of Ullman’s theory |
|
Independence assumption |
|
Group correspondence can occur without
constituent correspondence |
|
Linearity for global minimization by local
interactions. |
|
Better constraints are needed instead of
probabilistic approaches in low levels |
|
|
|
|
Rigid body assumption |
|
Local smoothness assumption |
|
Uniqueness assumption |
|
Motion correspondence problem is equivalent to
stereo correspondence problem under the rigid body assumption |
|
Consistency of an object’s identity through time |
|
|
|
|
Marr proposes two separate systems for structure
from motion problem and object identity problem. |
|
Key difference between visual motion and
stereopsis is : the objects can change in the next temporal state in visual
motion. |
|
|
|
|
Ullman’s counter-rotating cylinders experiment |
|
Infinite number of 3D configurations, but we see
the correct one. |
|
We must integrate additional information or
constraints. |
|
Additional information drives us to a unique ,
physically correct and unspecific (without a priori knowledge) solution. |
|
|
|
|
|
Helmholtz’s approach |
|
Recovering structure from motion is analogous to
recovering distance from disparity. |
|
Incorrect conclusions |
|
Common velocities are grouped together |
|
Changes in velocities provide recovery of 3D
structure. |
|
|
|
|
|
|
|
|
Rigidity constraint |
|
It enables us to solve the structure from motion
problem unambiguously. |
|
Ullman: 4-points 3-views after correspondence
process |
|
Structure from motion theorem: if a body is
rigid, we can find its 3D structure from three frames. |
|
The algorithms that use the theorem’s proof
methods fail under noisy data and non-rigidity |
|
|
|
|
J. J. Gibson “ the fundamental visual perception
is that of approach to a surface. This percept always has a subjective
component as well as an objective component, i.e it specifies the
observer’s position, movement, and direction as much as it specifies the
location, slant and shape of the surface”. |
|
Optical flow deals with planar surfaces. |
|
|
|
|
|
|
Instantaneous positional velocity field |
|
Specifying the optical flow is equivalent to
solving correspondence problem in apparent motion. |
|
Focus of expansion |
|
Most probably human visual system does not use
optical flow. But birds, insects. |
|
|
|
|
The peering locust is estimating the range of
the target in terms of the speed of the image on the retina. |
|
|
|
|
|
|
Flying bees infer range from apparent speed |
|
Bees tend to fly through the center of holes or
tunnels. How? |
|
|
|
|
|
The bees can distinguish between objects at
different distances by using cues based on image motion. |
|
|
|
|
|
|
|
|
Distance flown is gauged by integrating the
speed of the images of the walls and floor on the eyes while flying through
the tunnel (visual odometry). |
|
|
|
|
|
Purpose: Estimating the motion field |
|
The image brightness constancy equation |
|
Aperture problem: |
|
The component of the motion field in the
direction orthogonal to the spatial image gradient is not constrained by
the image brightness constancy equation |
|
Optical Flow: a vector field subject to the
image brightness constraint |
|
|
|
|
|
|
|
Differential Techniques |
|
Based on the spatial and temporal variations of
the image brightness at all pixels |
|
Correspondence-based Techniques |
|
Estimate the disparity of special image points
(features) between frames. |
|
|
|
|
|
|
|
Some of them require |
|
The solution of a system of PDE (Partial
Differential Equations) |
|
The computation of second and higher-order
derivatives of the image brightness |
|
Least-square estimates of the parameters
characterizing the optical flow |
|
|
|
|
|
The latter has advantages |
|
Not iterative; therefore, they are local, and
less biased than iterative methods by possible discontinuities of the
motion field |
|
Not involve derivatives of order higher than the
first; therefore less sensitive to noise. |
|
|
|
|
|
|
|
Cross-correlation methods (Fourier methods) |
|
Searches for the same object in two images is
that of using cross-correlation |
|
Function fitting |
|
Attempts to fit a mathematical function of known
properties to the objects of interest and measure the motion of these
objects by tracking the function distinctive features |
|
|
|
|
|
|
|
|
|
Feature identification |
|
Focus on the suitability of different types of
features to characterize objects and track their motion. |
|
Block matching |
|
The images are divided into blocks or areas,
each of which is assumed to undergo approximately the same translation |
|
|
|
|
|
Non-rigid bodies |
|
Cross-correlation techniques complemented with
some description of the shape change. |
|
|
|
|
|
Differential Techniques |
|
are less expensive in terms of computational complexity |
|
The error in boundaries (discontinuities) |
|
Correspondence-based Techniques |
|
Determination of invariant features |
|
Prior image segmentation requires |
|
|
|
|
Differential Technique |
|
Horn-Schunck Algorithm |
|
Integration of Horn-Schunck with discontinuities
is needed for better performance. |
|
|
|
|
begin
for j=: 1 to N do for i=:1 to M do begin:
calculate the values Ex(i, j, t), Ey(i,j,t), and
Et(i, j, t);
initialize the values u(i, j)
and v(v, j) with zero
end {for};
choose a suitable weighting value lambda {e.g.
lambda=0.1)
choose a suitable number n0>=1 of iterations; {n0=8}
n:=1;
while
n<=n0 do begin
for j:= 1 to N do for i:=1 to M do begin
udash=(u(i-1,j) +u(i+1, j) + u(i, j-a) +u(i,
j+1))/4;
vdash=(v(i-1,j) +v(i+1, j) + v(i, j-a) +v(i, j+1))/4;
alpha=Ex(i,j,t)*udash+Ey(i,j,t)*vdash +Et(i, j, t)*lambda/(1+lambda(Ex(i,
j, t)^2 +Ey(i, j, t)^2);
u(i, j)=udash -alpha* Ex(i,j,t);
v(i, j)=vdash-alpha*Ey(i, j, t)
end{for};
n=n+1;
end{while}
end; |
|
|
|
|
|
|
A needle map is used to represent the results. |
|
A grid was formed via traversing a mask to
refine the view of needle map |
|
From testing it has shown that too many needles
in the image only confuse and hide information. |
|
|
|
|
|
|
Constant time difference between capturing of
two pictures of the image sequence. |
|
The change of brightness is entirly caused by
absolute object motions, not by camera or light motion. The whole scene is
stable over time. |
|
Points from the one image appear in the second
results at other positions but with the same brightness value. |
|
|
|
|
The optical flow is uniquely defined (actually
different optical flow models may specify different vector fields). |
|
The high correlation between optical flow and
local displacement is expected. An error between the ideal and the actual
correlation can be used to define an error measurement for evaluating the
calculated optical flow as well as being an additional assumption for the
derivation. |
|
|
|
|
|
|
The image function E(x,y,ti) can locally be
represented by a Taylor expansion for a small step
(delta_x,delta_y,delta_t). |
|
Image brightness E(x,y,t) is a continuous and
differentiable function of three variable x, y and t. |
|
The higher order terms of the Taylor expansion
of E(x, y, t) are zero (e=0). |
|
|
|
|
Fs indicates the smoothness error for a function
pair (u(x,y), v(x,y)). |
|
It constrains that variation of the flow between
neighboring locations is small. |
|
nearby points within the image plane move in a
similar manner so minimize |
|
|
|
|
|
|
|
|
Fh is error functional of Horn-Schunck
constraint |
|
It indicates that the observed intensity is
constant for a constant location in world space. |
|
Since the Horn-Schunck constraint has to be
satisfied at the same time for all image points p=(x,y) of the given image
sequence for a moment t, a solution for (u,v) for this time t also has to
minimize the functional |
|
|
|
|
|
|
|
|
|
|
|
Optical Flow techniques can be used in real time
systems. |
|
The integration of smoothness constraint, image
brightness constraint and discontinuities is an obligation for finding
correct optical flows. |
|
Optical flow can be used in insect’s or bird’s
visual systems but human visual system probably uses correspondence based
approaches as supported by the experiments. |
|
|
|
|
S. Ullman. The Interpretation of Visual Motion,
MIT Press, 1979. |
|
M. V. Srinivasan, J. S. Chahl, K. Weber, S.
Venkatesh, M. G. Nagle, S. W. Zhang. Robot navigation inspired by
principles of insect vision. Robotics and Autonomous Systems, pages
26:203-216, 1999. |
|
B.K.P. Horn and B.G. Schunk. Determining Optical
Flow. Artificial Intelligence, Vol. 17, pages: 185-203, August, 1981. |
|
M. Middendorf and H.H. Nagel. Estimation and
Interpretation of Discontinuities in Optical Flow Fields. Computer Vision,
2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on
, Volume: 1 , 7-14 July 2001 Pages:178 - 183 vol.1. (Not used) |
|
J.F. Vega-Riveros and K. Jabbour. Review of
Motion Analysis Techniques. IEEE Proceedings, Vol. 136, No. 6, December,
1989. |
|
|
|
|
|