Claims:
1. A device for detecting video jitter, comprising:
a framing processing module configured to perform a framing process on a to-be-
detected
video to obtain a frame sequence;
a feature point detecting module configured to perform feature point detection
on the frame
sequence frame by frame, obtain feature points of each frame by employing a
feature point
detecting algorithm including fused FAST (Features from Accelerated Segment
Test) and
SURF (Speeded-Up Robust Features), and generate a frame feature point sequence
matrix;
a vector calculating module configured to perform an operation on the frame
feature point
sequence matrix to obtain a motion vector of each frame based on an optical
flow tracking
algorithm;
a feature value extracting module configured to obtain a feature value of the
to-be-detected
video according to the motion vector of each frame; and
a jitter detecting module configured to take the feature value of the to-be-
detected video as
an input signal of a detection model to perform operation to obtain an output
signal, and
judge whether jitter occurs to the to-be-detected video according to the
output signal.
2. The device of claim 1, wherein the device further comprises a data
preprocessing module
configured to preprocess the frame sequence.
3. The device of any one of claims 1 to 2, wherein the/a data preprocessing
module further
comprises:
a grayscale-processing unit configured to grayscale-process the frame
sequence, and obtain
a grayscale frame sequence; and
a denoising processing unit configured to denoise the grayscale frame
sequence;
wherein the feature point detecting module is configured to perform feature
point detection
on the preprocessed frame sequence frame by frame.
24
4. The device of any one of claims 1 to 3, wherein the feature point
detecting module is further
configured to:
employ the feature point detecting algorithm in which are the fused FAST
(Features from
Accelerated Segment Test) and the SURF (Speeded-Up Robust Features) to perform
feature point detection on the frame sequence frame by frame, and to obtain
the feature
points of each frame.
5. The device of any one of claims 1 to 4, wherein the vector calculating
module further
comprises:
an optical flow tracking unit configured to perform optical flow tracking
calculation on the
frame feature point sequence matrix of each frame, and obtain an initial
motion vector of
each frame;
an accumulation calculating unit configured to obtain a corresponding
accumulative
motion vector according to the initial motion vector;
a smoothening processing unit configured to smoothen the accumulative motion
vector,
and obtain a smoothened motion vector; and
a vector readjusting unit configured to employ the accumulative motion vector
and the
smoothened motion vector to readjust the initial motion vector of each frame,
and obtain
the motion vector of each frame.
6. The device of any one of claims 1 to 5, wherein the feature value
extracting module further
comprises:
a matrix converting unit configured to merge and convert the motion vectors of
all frames
into a matrix;
a standard deviation calculating unit configured to calculate unbiased
standard deviations
of various elements in the matrix; and
a weighting and fusing unit configured to weight and fuse the unbiased
standard deviations
of the various elements, and obtain a weighted value.
7. The device of any one of claims 1 to 6, wherein a to-be-detected video
is obtained.
8. The device of any one of claims 1 to 7, wherein a framing extraction
process is performed on
the to-be-detected video to obtain a frame sequence corresponding to the to-be-
detected video.
9. The device of any one of claims 1 to 8, wherein the frame sequence is
expressed as L, (1=1, 2,
3, ..., n) , where Li represents the ith frame of the video, and n represents
the total number of
frames of the video.
10. The device of any one of claims 1 to 9, wherein a current frame and an
adjacent next frame is
selected from the to-be-detected video.
11. The device of any one of claims 1 to 10, wherein a current frame and a
next frame by an
interval of N frames is selected from the to-be-detected video.
12. The device of any one of claims 1 to 11, wherein corresponding feature
points are obtained
from the current frame and the adjacent next frame.
13. The device of any one of claims 1 to 12, wherein corresponding feature
points are obtained
from the current frame and the next frame by an interval of N frames.
14. The device of any one of claims 1 to 13, wherein corresponding matching is
performed
according to the feature points of the two frames to judge whether offset
(jitter) occurs
between the current frame and the adjacent next frame.
15. The device of any one of claims 1 to 14, wherein corresponding matching is
performed
according to the feature points of the two frames to judge whether offset
(jitter) occurs
between the current frame and the next frame by an interval of N frames.
16. The device of any one of claims 1 to 15, wherein a feature point detecting
algorithm is
employed to perform feature point detection on the processed frame sequence Li
(i=1, 2, 3, ...,
n) frame by frame.
26
17. The device of any one of claims 1 to 16, wherein a feature point detecting
algorithm is
employed to obtain feature points of each frame, that is, feature points of
each frame of image
are extracted.
18. The device of any one of claims 1 to 17, wherein a feature point detecting
algorithm is
employed to generate a frame feature point sequence matrix, which is expressed
as Z (i=1,
2, ..., n).
19. The device of any one of claims 1 to 18, wherein Z (i=1, 2, ..., n) is
expressed as:
<IMG>
Wherein apq
represents a feature point detection result at row p , column q of the ith
frame
matrix, 1 is a feature point, 0 is a non-feature point, p represents the
number of rows of the
matrix, and q represents the number of columns of the matrix.
20. The device of any one of claims 1 to 19, wherein the optical flow tracking
algorithm is
employed to perform optical flow tracking calculation on the frame feature
point sequence
matrix.
21. The device of any one of claims 1 to 20, wherein the optical flow tracking
algorithm is
employed to track the change of feature points in the current frame to the
next frame.
22. The device of any one of claims 1 to 21, wherein the change of a feature
point sequence matrix
in the ith frame is tracked to the next i+1 th frame, and a motion vector ai
is obtained, whose
expression is as follows:
27
<IMG>
where dr represents a Euclidean column offset from the ith frame to the i+lth
frame, cly'
represents a Euclidean row offset from the PI frame to the i+lth frame, and
dr' represents
an angle offset from the ith frame to the i+lth frame.
23. The device of any one of claims 1 to 22, wherein an extracted feature
value at least includes
the feature value of four dimensions.
24. The device of any one of claims 1 to 23, wherein the detection model is
well trained in advance.
25. The device of any one of claims 1 to 24, wherein sample video data in a
set of selected training
data is correspondingly processed to obtain a feature value of the sample
video data.
26. The device of any one of claims 1 to 25, wherein a detection model is
trained according to the
feature value of the/a sample video data and a corresponding annotation result
of the sample
video data to obtain the final detection model.
27. The device of any one of claims 1 to 26, wherein after the motion vectors
have been
dimensionally converted for an 111th video sample, unbiased standard
deviations of various
elements and their weighted and fused value are calculated and obtained, which
are
<IMG>
respectively expressed as
and Km , and the
annotation resultym of the 111th video sample is extracted to obtain the
training sample of the
111th video sample, which is expressed as follows:
<IMG>
28. The device of any one of claims 1 to 27, wherein the annotation result ym
indicates that no
jitter occurs to the video sample if ym=0.
28
29. The device of any one of claims 1 to 28, wherein the annotation result ym
indicates that jitter
occurs to the video sample if ym=1.
30. The device of any one of claims 1 to 29, wherein a video sample makes use
of features of at
least five dimensions.
31. The device of any one of claims 1 to 30, wherein the detection model is
selected from an SVM
(Support Vector Machine) model.
32. The device of any one of claims 1 to 31, wherein the feature value of the
to-be-detected video
is input to a well-trained SVM model to obtain an output result.
33. The device of any one of claims 1 to 32, wherein if the/a SVM model output
result is 0, this
indicates that no jitter occurs to the to-be-detected video.
34. The device of any one of claims 1 to 33, wherein if the/a SVM model output
result is 1, this
indicates that jitter occurs to the to-be-detected video.
35. The device of any one of claims 1 to 34, wherein the use of a trainable
SVM model as a video
jitter decider enables jitter detection of videos captured in different
scenarios.
36. The device of any one of claims 1 to 35, wherein the amount of information
of the image is
greatly reduced after grayscale-processing.
37. The device of any one of claims 1 to 36, wherein the frame sequence L,
(i=1, 2, 3, ..., n) is
further grayscale-processed.
38. The device of any one of claims 1 to 37, wherein a grayscale frame
sequence is obtained to
be expressed as G, (1=1, 2, 3,
n), in which the grayscale conversion expression is as follows:
G=Rx0.299+Gx0.587+Bx0.114
39. The device of any one of claims 1 to 38, wherein a TV denoising method
based on a total
variation model is employed to denoise the grayscale frame sequence G, (1=1,
2, 3, ..., n) to
obtain a denoised frame sequence expressed as T, (1=1, 2, 3, ..., n).
29
40. The device of any one of claims 1 to 39, wherein the/a denoised frame
sequence is a
preprocessed frame sequence to which the to-be-detected video corresponds.
41. The device of any one of claims 1 to 40, wherein the denoi sing method is
randomly selectable.
42. The device of any one of claims 4 to 41, wherein the SURF algorithm is an
improvement
over SIFT (Scale-Invariant Feature Transform) algorithm.
43. The device of claim 42, wherein SIFT is a feature describing method with
excellent robustness
and invariant scale.
44. The device of any one of claims 42 to 43, wherein the SURF algorithm
improves over the
problems of large amount of data to be calculated, high time complexity and
long duration of
calculation inherent in the SIFT algorithm at the same time of maintaining the
advantages of
the SIFT algorithm.
45. The device of any one of claims 4 to 44, wherein SURF is more excellent in
performance in
the aspect of invariance of illumination change and perspective change,
particularly excellent
in processing severe blurs and rotations of images.
46. The device of any one of claims 4 to 45, wherein SURF is excellent in
describing local features
of images.
47. The device of any one of claims 4 to 46, wherein FAST feature detection is
a corner detection
method.
48. The device of any one of claims 4 to 47, wherein FAST feature detection
has the most
prominent advantage of its algorithm in its calculation efficiency, and the
capability to
excellently describe global features of images.
49. The device of any one of claims 4 to 48, wherein use of the feature point
detecting algorithm
in which FAST features and SURF features are fused to perform feature point
extraction gives
consideration to global features of images.
50. The device of any one of claims 4 to 49, wherein use of the feature point
detecting algorithm
in which FAST features and SURF features are fused to perform feature point
extraction fully
retains local features of images.
51. The device of any one of claims 4 to 50, wherein use of the feature point
detecting algorithm
in which FAST features and SURF features are fused to perform feature point
extraction has
small computational expense.
52. The device of any one of claims 4 to 51, wherein use of the feature point
detecting algorithm
in which FAST features and SURF features are fused to perform feature point
extraction has
strong robustness against image blurs and faint illumination.
53. The device of any one of claims 4 to 52, wherein use of the feature point
detecting algorithm
in which FAST features and SURF features are fused to perform feature point
extraction
enhances real-time property and precision of the detection.
54. The device of any one of claims 1 to 53, wherein while the optical flow
tracking calculation
is performed on the frame feature point sequence matrix, a pyramid optical
flow tracking LK
(Lucas-Kanade) algorithm is employed.
55. The device of any one of claims 1 to 54, wherein the change of a feature
point sequence matrix
Z in the ith frame is tracked to the next i+lth frame, and a motion vector ai
is obtained, whose
expression is as follows:
<IMG>
where dr' represents a Euclidean column offset from the ith frame to the i+ th
frame, dy'
represents a Euclidean row offset from the ith frame to the i+ th frame, and
dri represents
an angle offset from the PI frame to the i+ th frame.
31
56. The device of any one of claims 1 to 55, wherein use of the/a pyramid
optical flow tracking
LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative structure
solves the problem
of failed tracking due to unduly large change from feature points of a current
frame to feature
points of a next adjacent frame.
57. The device of any one of claims 1 to 56, wherein use of the/a pyramid
optical flow tracking
LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative structure
solves the problem
of failed tracking due to unduly large change from feature points of a current
frame to feature
points of a next frame by an interval of N frames.
58. The device of any one of claims 1 to 57, wherein accumulative integral
transformation is
performed on the initial motion vector ai of each frame to obtain an
accumulative motion
vector, expressed as ,
of each frame, in which the expression of the accumulative motion
vector f3i is as follows:
<IMG>
59. The device of any one of claims 1 to 58, wherein a sliding average window
is used to
smoothen the motion vector )3i to obtain a smoothened motion vector Y , whose
expression is:
32
<IMG>
where n represents the total number of frames of the video.
60. The device of any one of claims 1 to 59, wherein the radius of the smooth
window is r, and
its expression is:
<IMG>
where tt indicates a parameter ot the sliding window, and la is a positive
number.
61. The device of any one of claims 1 to 60, wherein the specific numerical
value of can be
dynamically adjusted according to practical requirement.
62. The device of any one of claims 1 to 61, wherein the specific numerical
value of jt is set as
=30.
63. The device of any one of claims 1 to 62, wherein
and Y are used to readjust ai to
obtain a readjusted motion vector 4, whose expression is:
33
<IMG>
64. The device of any one of claims 1 to 63, wherein the readjusted motion
vector Ai is taken as
the motion vector of each frame to participate in subsequent calculation, so
that the calculation
result is made more precise.
65. The device of any one of claims 1 to 64, wherein the motion vectors of all
frames obtained
are merged and converted into a matrix.
¨
66. The device of any one of claims 1 to 65, wherein the motion vector Ai is
converted into the
<IMG>
form of a matrix
and unbiased standard deviations of its elements
are calculated by rows, the specific calculation expression is as follows:
<IMG>
67. The device of any one of claims 1 to 66, wherein the unbiased standard
deviations of the
<IMG>
various elements in the matrix are respectively expressed as
and
<IMG>
, in which A represents the average value of the samples.
68. The device of any one of claims 1 to 67, wherein weig)its are assigned to
the unbiased standard
deviations of the various elements according to practical requirements.
34
69. The device of any one of claims 1 to 68, wherein the unbiased standard
deviations of the
various elements are weighted and fused according to the weights.
70. The device of any one of claims 1 to 69, wherein the weights of the
unbiased standard
deviations of the various elements can be dynamically readjusted according to
practical
requirements.
<IMG>
71. The device of any one of claims 1 to 70, wherein the weight of is
set as 3, the
<IMG> <IMG>
weight of is set as 3, and the weight of is
set as 10, then the fusing
expression is as follows:
<IMG>
72. The device of any one of claims 1 to 71, wherein the feature value of the
to-be-detected video
is the unbiased standard deviations of the various elements and their weighted
value, which
are expressed as:
{c4.1.(dx)]5 a[A.(dy)], cr[A(dr)], K.,}(s) .
73. A computer-readable storage medium for detecting video jitter, storing
therein a computer
program, wherein the computer program is configured to:
perform a framing process on a to-be-detected video to obtain a frame
sequence;
perform feature point detection on the frame sequence frame by frame, obtain
feature
points of each frame by employing a feature point detecting algorithm
including fused
FAST (Features from Accelerated Segment Test) and SURF (Speeded-Up Robust
Features), and generate a frame feature point sequence matrix;
perform an operation on the frame feature point sequence matrix to obtain a
motion vector
of each frame based on an optical flow tracking algorithm;
obtain a feature value of the to-be-detected video according to the motion
vector of each
frame; and
take the feature value of the to-be-detected video as an input signal of a
detection model to
perform operation to obtain an output signal, and judge whether jitter occurs
to the to-be-
detected video according to the output signal.
74. The computer-readable storage medium of claim 73, wherein the computer
program is further
configured to:
grayscale-process the frame sequence, and obtain a grayscale frame sequence;
and
denoise the grayscale frame sequence;
wherein the step of feature point detection on the frame sequence frame by
frame is feature
point detection on the preprocessed frame sequence frame by frame.
75. The computer-readable storage medium of any one of claims 73 to 74,
wherein the computer
program is further configured to:
employ the feature point detecting algorithm in which are the fused FAST
(Features from
Accelerated Segment Test) and the SURF (Speeded-Up Robust Features) to perform
feature point detection on the frame sequence frame by frame, and to obtain
the feature
points of each frame.
76. The computer-readable storage medium of any one of claims 73 to 75,
wherein the computer
program is further configured to:
perform optical flow tracking calculation on the frame feature point sequence
matrix of
each frame, and obtain an initial motion vector of each frame;
obtain a corresponding accumulative motion vector according to the initial
motion vector;
smoothen the accumulative motion vector, and obtain a smoothened motion
vector; and
36
employ the accumulative motion vector and the smoothened motion vector to
readjust the
initial motion vector of each frame, and obtain the motion vector of each
frame.
77. The computer-readable storage medium of any one of claims 73 to 76,
wherein the computer
program is further configured to:
merge and convert the motion vectors of all frames into a matrix, and
calculate unbiased
standard deviations of various elements in the matrix;
weight and fuse the unbiased standard deviations of the various elements, and
obtain a
weighted value; and
take the unbiased standard deviations of the various elements and the weighted
value as
the feature value of the to-be-detected video.
78. The computer-readable storage medium of any one of claims 73 to 77,
wherein a to-be-
detected video is obtained.
79. The computer-readable storage medium of any one of claims 73 to 78,
wherein a framing
extraction process is performed on the to-be-detected video to obtain a frame
sequence
corresponding to the to-be-detected video.
80. The computer-readable storage medium of any one of claims 73 to 79,
wherein the frame
sequence is expressed as Li (i=1, 2, 3, ..., n), where Li represents the ith
frame of the video,
and n represents the total number of frames of the video.
81. The computer-readable storage medium of any one of claims 73 to 80,
wherein a current frame
and an adjacent next frame is selected from the to-be-detected video.
82. The computer-readable storage medium of any one of claims 73 to 81,
wherein a current frame
and a next frame by an interval of N frames is selected from the to-be-
detected video.
83. The computer-readable storage medium of any one of claims 73 to 82,
wherein corresponding
feature points are obtained from a current frame and the adjacent next frame.
37
84. The computer-readable storage medium of any one of claims 73 to 83,
wherein corresponding
feature points are obtained from a current frame and the next frame by an
interval of N frames.
85. The computer-readable storage medium of any one of claims 73 to 84,
wherein corresponding
matching is performed according to the feature points of the two frames to
judge whether
offset (jitter) occurs between the current frame and the adjacent next frame.
86. The computer-readable storage medium of any one of claims 73 to 85,
wherein corresponding
matching is performed according to the feature points of the two frames to
judge whether
offset (jitter) occurs between the current frame and the next frame by an
interval of N frames.
87. The computer-readable storage medium of any one of claims 73 to 86,
wherein a feature point
detecting algorithm is employed to perform feature point detection on the
processed frame
sequence L, (i=1, 2, 3, ..., n) frame by frame.
88. The computer-readable storage medium of any one of claims 73 to 87,
wherein a feature point
detecting algorithm is employed to obtain feature points of each frame, that
is, feature points
of each frame of image are extracted.
89. The computer-readable storage medium of any one of claims 73 to 88,
wherein a feature point
detecting algorithm is employed to generate a frame feature point sequence
matrix, which is
expressed as Z (1A, 2, ..., n) .
90. The computer-readable storage medium of any one of claims 73 to 89,
wherein Z, (i=1, 2, ...,
n) is expressed as:
<IMG>
38
l
Wherein a ' q represents a feature point detection result at row p , column q
of the ith frame
matrix, 1 is a feature point, 0 is a non-feature point, p represents the
number of rows of the
matrix, and q represents the number of columns of the matrix.
91. The computer-readable storage medium of any one of claims 73 to 90,
wherein the optical
flow tracking algorithm is employed to perform optical flow tracking
calculation on the frame
feature point sequence matrix.
92. The computer-readable storage medium of any one of claims 73 to 91,
wherein the optical
flow tracking algorithm is employed to track the change of feature points in
the current frame
to the next frame.
93. The computer-readable storage medium of any one of claims 73 to 92,
wherein the change of
a feature point sequence matrix Z in the ith frame is tracked to the next
i+lth frame, and a
motion vector ai is obtained, whose expression is as follows:
<IMG>
where dr represents a Euclidean column offset from the ith frame to the i+lth
frame, dy'
represents a Euclidean row offset from the ith frame to the i+lth frame, and
dr' represents
an angle offset from the ith frame to the i+lth frame.
94. The computer-readable storage medium of any one of claims 73 to 93,
wherein an extracted
feature value at least includes the feature value of four dimensions.
95. The computer-readable storage medium of any one of claims 73 to 94,
wherein the detection
model is well trained in advance.
39
96. The computer-readable storage medium of any one of claims 73 to 95,
wherein sample video
data in a set of selected training data is correspondingly processed to obtain
a feature value of
the sample video data.
97. The computer-readable storage medium of any one of claims 73 to 96,
wherein a detection
model is trained according to the feature value of the sample video data and a
corresponding
annotation result of the sample video data to obtain the final detection
model.
98. The computer-readable storage medium of any one of claims 73 to 97,
wherein after the
motion vectors have been dimensionally converted for an 171th video sample,
unbiased standard
deviations of various elements and their weighted and fused value are
calculated and obtained,
<IMG>
which are respectively expressed as
and Km , and
the annotation result))m of the 171th video sample is extracted to obtain the
training sample of
the 711th video sample, which is expressed as follows:
<IMG>
99. The computer-readable storage medium of any one of claims 73 to 98,
wherein the annotation
result))m indicates that no jitter occurs to the video sample if y.=0.
100. The computer-readable storage medium of any one of claims 73 to 99,
wherein the annotation
result))m indicates that jitter occurs to the video sample if )).=1.
101. The computer-readable storage medium of any one of claims 73 to 100,
wherein a video
sample makes use of features of at least five dimensions.
102. The computer-readable storage medium of any one of claims 73 to 101,
wherein the detection
model is selected from an SVM (Support Vector Machine) model.
103. The computer-readable storage medium of any one of claims 73 to 102,
wherein the feature
value of the to-be-detected video is input to a well-trained SVM model to
obtain an output
result.
104. The computer-readable storage medium of any one of claims 73 to 103,
wherein if the/a SVM
model output result is 0, this indicates that no jitter occurs to the to-be-
detected video.
105. The computer-readable storage medium of any one of claims 73 to 104,
wherein if the/a SVM
model output result is 1, this indicates that jitter occurs to the to-be-
detected video.
106. The computer-readable storage medium of any one of claims 73 to 105,
wherein the use of a
trainable SVM model as a video jitter decider enables jitter detection of
videos captured in
different scenarios.
107. The computer-readable storage medium of any one of claims 73 to 106,
wherein the amount
of information of the image is greatly reduced after grayscale-processing.
108. The computer-readable storage medium of any one of claims 73 to 107,
wherein the frame
sequence L, (1=1, 2, 3, ..., n) is further graysc ale-processed.
109. The computer-readable storage medium of any one of claims 73 to 108,
wherein a grayscale
frame sequence is obtained to be expressed as G, (i=1, 2, 3, ..., n), in which
the grayscale
conversion expression is as follows:
G = Rx 0.299+G x 0.587 B x a 114
110. The computer-readable storage medium of any one of claims 73 to 109,
wherein a TV
denoising method based on a total variation model is employed to denoise the
grayscale frame
sequence G, (i=1, 2, 3, ..., n) to obtain a denoised frame sequence expressed
as T, (1=1, 2,
3, ..., n).
111. The computer-readable storage medium of any one of claims 73 to 110,
wherein the/a
denoised frame sequence is a preprocessed frame sequence to which the to-be-
detected video
corresponds.
112. The computer-readable storage medium of any one of claims 73 to 111,
wherein the/a TV
denoising method is randomly selectable.
41
113. The computer-readable storage medium of any one of claims 75 to 112,
wherein the SURF
algorithm is an improvement over SIFT (Scale-Invariant Feature Transform)
algorithm.
114. The computer-readable storage medium of claim 113, wherein SIFT is a
feature describing
method with excellent robustness and invariant scale.
115. The computer-readable storage medium of any one of claims 113 to 114,
wherein the SURF
algorithm improves over the problems of large amount of data to be calculated,
high time
complexity and long duration of calculation inherent in the SIFT algorithm at
the same time
of maintaining the advantages of the SIFT algorithm.
116. The computer-readable storage medium of any one of claims 75 to 115,
wherein SURF is
more excellent in performance in the aspect of invariance of illumination
change and
perspective change, particularly excellent in processing severe blurs and
rotations of images.
117. The computer-readable storage medium of any one of claims 75 to 116,
wherein SURF is
excellent in describing local features of images.
118. The computer-readable storage medium of any one of claims 75 to 117,
wherein FAST feature
detection is a corner detection method.
119. The computer-readable storage medium of any one of claims 75 to 118,
wherein FAST feature
detection has the most prominent advantage of its algorithm in its calculation
efficiency, and
the capability to excellently describe global features of images.
120. The computer-readable storage medium of any one of claims 75 to 119,
wherein use of the
feature point detecting algorithm in which FAST features and SURF features are
fused to
perform feature point extraction gives consideration to global features of
images.
121. The computer-readable storage medium of any one of claims 75 to 120,
wherein use of the
feature point detecting algorithm in which FAST features and SURF features are
fused to
perform feature point extraction fully retains local features of images.
42
122. The computer-readable storage medium of any one of claims 75 to 121,
wherein use of the
feature point detecting algorithm in which FAST features and SURF features are
fused to
perform feature point extraction has small computational expense.
123. The computer-readable storage medium of any one of claims 75 to 122,
wherein use of the
feature point detecting algorithm in which FAST features and SURF features are
fused to
perform feature point extraction has strong robustness against image blurs and
faint
illuminati on.
124. The computer-readable storage medium of any one of claims 75 to 123,
wherein use of the
feature point detecting algorithm in which FAST features and SURF features are
fused to
perform feature point extraction enhances real-time property and precision of
the detection.
125. The computer-readable storage medium of any one of claims 73 to 124,
wherein while the
optical flow tracking calculation is perfoimed on the frame feature point
sequence matrix, a
pyramid optical flow tracking LK (Lucas-Kanade) algorithm is employed.
126. The computer-readable storage medium of any one of claims 73 to 125,
wherein the change
of a feature point sequence matrix Z in the ith frame is tracked to the next
i+lth frame, and a
motion vector (xi is obtained, whose expression is as follows:
<IMG>
where cbcr represents a Euclidean column offset from the ith frame to the
i+lth frame, dy'
represents a Euclidean row offset from the ith frame to the i+lth frame, and
dr' represents
an angle offset from the th frame to the i+lth frame.
43
127. The computer-readable storage medium of any one of claims 73 to 126,
wherein use of the/a
pyramid optical flow tracking LK (Lucas-Kanade) algorithm that utilizes the
pyramid iterative
structure solves the problem of failed tracking due to unduly large change
from feature points
of a current frame to feature points of an adjacent next frame.
128. The computer-readable storage medium of any one of claims 73 to 127,
wherein use of the/a
pyramid optical flow tracking LK (Lucas-Kanade) algorithm that utilizes the
pyramid iterative
structure solves the problem of failed tracking due to unduly large change
from feature points
of a current frame to feature points of a next frame by an interval of N
frames.
129. The computer-readable storage medium of any one of claims 73 to 128,
wherein accumulative
integral transformation is performed on the initial motion vector ai of each
frame to obtain
an accumulative motion vector, expressed as fii , of each frame, in which the
expression of
the accumulative motion vector fli is as follows:
<IMG>
130. The computer-readable storage medium of any one of claims 73 to 129,
wherein a sliding
average window is used to smoothen the motion vector 13i to obtain a
smoothened motion
vector Y , whose expression is:
44
<IMG>
where n represents the total number of frames of the video.
131. The computer-readable storage medium of any one of claims 73 to 130,
wherein the radius of
the smooth window is r, and its expression is:
<IMG>
where indicates a parameter of the sliding window, and la is a positive
number.
132. The computer-readable storage medium of any one of claims 73 to 131,
wherein the specific
numerical value of can be dynamically adjusted according to practical
requirement.
133. The computer-readable storage medium of any one of claims 73 to 132,
wherein the specific
numerical value of is set as .=30.
134. The computer-readable storage medium of any one of claims 73 to 133,
wherein 13 and
are used to readjust ai to obtain a readjusted motion vector 114, whose
expression is:
<IMG>
135. The computer-readable storage medium of any one of claims 73 to 134,
wherein the readjusted
motion vector Ai is taken as the motion vector of each frame to participate in
subsequent
calculation, so that the calculation result is made more precise.
136. The computer-readable storage medium of any one of claims 73 to 135,
wherein the motion
vectors of all frames obtained are merged and converted into a matrix.
137. The computer-readable storage medium of any one of claims 73 to 136,
wherein the motion
<IMG>
vector is converted into the form of a matrix
and unbiased
standard deviations of its elements are calculated by rows, the specific
calculation expression
is as follows:
<IMG>
46
138. The computer-readable storage medium of any one of claims 73 to 137,
wherein the unbiased
standard deviations of the various elements in the matrix are respectively
expressed as
<IMG>
, in which A represents the average value of the
samples.
139. The computer-readable storage medium of any one of claims 73 to 138,
wherein weights are
assigned to the unbiased standard deviations of the various elements according
to practical
requirements.
140. The computer-readable storage medium of any one of claims 73 to 139,
wherein the unbiased
standard deviations of the various elements are weighted and fused according
to the weights.
141. The computer-readable storage medium of any one of claims 73 to 140,
wherein the weights
of the unbiased standard deviations of the various elements can be dynamically
readjusted
according to practical requirements.
142. The computer-readable storage medium of any one of claims 73 to 141,
wherein the weight
<IMG> <IMG>
of is set as 3, the weight of is
set as 3, and the weight of
<IMG>
is set as 10, then the fusing expression is as follows:
<IMG>
143. The computer-readable storage medium of any one of claims 73 to 142,
wherein the feature
value of the to-be-detected video is the unbiased standard deviations of the
various elements
and their weighted value, which are expressed as:
fo-[Ä(dx)], a[A.(dy)], o[2(dr)], ic,}(s)
144. The computer-readable storage medium of any one of claims 73 to 143,
wherein the computer-
readable storage medium is a read-only memory.
47
145. The computer-readable storage medium of any one of claims 73 to 144,
wherein the computer-
readable storage medium is a magnetic disk.
146. The computer-readable storage medium of any one of claims 73 to 145,
wherein the computer-
readable storage medium is an optical disk.
147.A computer device for detecting video jitter, comprising:
one or more processors, configured to:
perform a framing process on a to-be-detected video to obtain a frame
sequence;
perform feature point detection on the frame sequence frame by frame, obtain
feature points of each frame by employing a feature point detecting algorithm
including fused FAST (Features from Accelerated Segment Test) and SURF
(Speeded-Up Robust Features), and generate a frame feature point sequence
matrix;
perform an operation on the frame feature point sequence matrix to obtain a
motion vector of each frame based on an optical flow tracking algorithm;
obtain a feature value of the to-be-detected video according to the motion
vector of each frame; and
take the feature value of the to-be-detected video as an input signal of a
detection model to perfolln operation to obtain an output signal, and judge
whether jitter occurs to the to-be-detected video according to the output
signal.
148. The computer device of claim 147, wherein the computer device is further
configured to:
grayscale-process the frame sequence, and obtain a grayscale frame sequence;
and
denoise the grayscale frame sequence;
48
wherein the step of feature point detection on the frame sequence frame by
frame is feature
point detection on the preprocessed frame sequence frame by frame.
149. The computer device of any one of claims 147 to 148, wherein the computer
device is further
configured to:
employ the feature point detecting algorithm in which are the fused FAST
(Features from
Accelerated Segment Test) and the SURF (Speeded-Up Robust Features) to perform
feature point detection on the frame sequence frame by frame, and to obtain
the feature
points of each frame.
150. The computer device of any one of claims 147 to 149, wherein the computer
device is further
configured to:
perform optical flow tracking calculation on the frame feature point sequence
matrix of
each frame, and obtain an initial motion vector of each frame;
obtain a corresponding accumulative motion vector according to the initial
motion vector;
smoothen the accumulative motion vector, and obtain a smoothened motion
vector; and
employ the accumulative motion vector and the smoothened motion vector to
readjust the
initial motion vector of each frame, and obtain the motion vector of each
frame.
151. The computer device of any one of claims 147 to 150, wherein the computer
device is further
configured to:
merge and convert the motion vectors of all frames into a matrix, and
calculate unbiased
standard deviations of various elements in the matrix;
weight and fuse the unbiased standard deviations of the various elements, and
obtain a
weighted value; and
take the unbiased standard deviations of the various elements and the weighted
value as
the feature value of the to-be-detected video.
49
152. The computer device of any one of claims 147 to 151, wherein a to-be-
detected video is
obtained.
153. The computer device of any one of claims 147 to 152, wherein a framing
extraction process
is performed on the to-be-detected video to obtain a frame sequence
corresponding to the to-
be-detected video.
154. The computer device of any one of claims 147 to 153, wherein the frame
sequence is
expressed as Li (1=1, 2, 3, ..., n) , where L, represents the ith frame of the
video, and n represents
the total number of frames of the video.
155. The computer device of any one of claims 147 to 154, wherein a current
frame and an adjacent
next frame is selected from the to-be-detected video.
156. The computer device of any one of claims 147 to 155, wherein a current
frame and a next
frame by an interval of N frames is selected from the to-be-detected video.
157. The computer device of any one of claims 147 to 156, wherein
corresponding feature points
are obtained from the current frame and the adjacent next frame.
158. The computer device of any one of claims 147 to 157, wherein
corresponding feature points
are obtained from the current frame and the next frame by an interval of N
frames.
159. The computer device of any one of claims 147 to 158, wherein
corresponding matching is
performed according to the feature points of the two frames to judge whether
offset (jitter)
occurs between the current frame and the adjacent next frame.
160. The computer device of any one of claims 147 to 159, wherein
corresponding matching is
performed according to the feature points of the two frames to judge whether
offset (jitter)
occurs between the current frame and the next frame by an interval of N
frames.
161. The computer device of any one of claims 147 to 160, wherein a feature
point detecting
algorithm is employed to perform feature point detection on the processed
frame sequence L,
(i=1, 2, 3, ..., n) frame by frame.
162. The computer device of any one of claims 147 to 161, wherein a feature
point detecting
algorithm is employed to obtain feature points of each frame, that is, feature
points of each
frame of image are extracted.
163. The computer device of any one of claims 147 to 162, wherein a feature
point detecting
algorithm is employed to generate a frame feature point sequence matrix, which
is expressed
as Z, (i=1, 2, ..., n).
164. The computer device of any one of claims 147 to 163, wherein Z, (1=1, 2,
..., n) is expressed
as:
<IMG>
Wherein p'q represents a feature point detection result at row p, column q of
the ith frame
matrix, 1 is a feature point, 0 is a non-feature point, p represents the
number of rows of the
matrix, and q represents the number of columns of the matrix.
165. The computer device of any one of claims 147 to 164, wherein the optical
flow tracking
algorithm is employed to perform optical flow tracking calculation on the
frame feature point
sequence matrix.
166. The computer device of any one of claims 147 to 165, wherein the optical
flow tracking
algorithm is employed to track the change of feature points in the current
frame to the next
frame.
167. The computer device of any one of claims 147 to 166, wherein the change
of a feature point
sequence matrix Z in the ith frame is tracked to the next i-Elth frame, and a
motion vector ai
is obtained, whose expression is as follows:
51
<IMG>
where dr represents a Euclidean column offset from the ith frame to the i+lth
frame, dy'
represents a Euclidean row offset from the PI frame to the i+lth frame, and
dr' represents
an angle offset from the ith frame to the i+lth frame.
168. The computer device of any one of claims 147 to 167, wherein an extracted
feature value at
least includes the feature value of four dimensions.
169. The computer device of any one of claims 147 to 168, wherein the
detection model is well
trained in advance.
170. The computer device of any one of claims 147 to 169, wherein sample video
data in a set of
selected training data is correspondingly processed to obtain a feature value
of the sample
video data.
171. The computer device of any one of claims 147 to 170, wherein a detection
model is trained
according to the feature value of the sample video data and a corresponding
annotation result
of the sample video data to obtain the final detection model.
172. The computer device of any one of claims 147 to 171, wherein after the
motion vectors have
been dimensionally converted for an 177th video sample, unbiased standard
deviations of
various elements and their weighted and fused value are calculated and
obtained, which are
<IMG>
respectively expressed as
and m , and the
annotation resultym of the 117th video sample is extracted to obtain the
training sample of the
177th video sample, which is expressed as follows:
fa[2.(dx)]?õ, a[A(dy)].õ, ()POOL, Km ym)(m)
52
173. The computer device of any one of claims 147 to 172, wherein the
annotation result yn,
indicates that no jitter occurs to the video sample if
174. The computer device of any one of claims 147 to 173, wherein the
annotation result yn,
indicates that jitter occurs to the video sample if y .
175. The computer device of any one of claims 147 to 174, wherein a video
sample makes use of
features of at least five dimensions.
176. The computer device of any one of claims 147 to 175, wherein the
detection model is selected
from an SVM (Support Vector Machine) model.
177. The computer device of any one of claims 147 to 176, wherein the feature
value of the to-be-
detected video is input to a well-trained SVM model to obtain an output
result.
178. The computer device of any one of claims 147 to 177, wherein if the/a SVM
model output
result is 0, this indicates that no jitter occurs to the to-be-detected video.
179. The computer device of any one of claims 147 to 178, wherein if the/a SVM
model output
result is 1, this indicates that jitter occurs to the to-be-detected video.
180. The computer device of any one of claims 147 to 179, wherein the use of a
trainable SVM
model as a video jitter decider enables jitter detection of videos captured in
different scenarios.
181. The computer device of any one of claims 147 to 180, wherein the amount
of information of
the image is greatly reduced after grayscale-processing.
182. The computer device of any one of claims 147 to 181, wherein the frame
sequence L, (1=1, 2,
3, ..., n) is further grayscale-processed.
183. The computer device of any one of claims 147 to 182, wherein a grayscale
frame sequence is
obtained to be expressed as G, (i=1, 2, 3, ..., n), in which the grayscale
conversion expression
is as follows:
G=Rx0.299+Gx0.587+Bx0.1.1A
53
184. The computer device of any one of claims 147 to 183, wherein a TV
denoising method based
on a total variation model is employed to denoise the grayscale frame sequence
G, (1A, 2,
3, ..., n) to obtain a denoised frame sequence expressed as T, (1=1, 2, 3,
..., n) .
185. The computer device of any one of claims 147 to 184, wherein the/a
denoised frame sequence
is a preprocessed frame sequence to which the to-be-detected video
corresponds.
186. The computer device of any one of claims 147 to 185, wherein the
denoising method is
randomly selectable.
187. The computer device of any one of claims 149 to 186, wherein the SURF
algorithm is an
improvement over SIFT (Scale-Invariant Feature Transform) algorithm.
188. The computer device of claim 187, wherein SIFT is a feature describing
method with excellent
robustness and invariant scale.
189. The computer device of any one of claims 187 to 188, wherein the/a SURF
algorithm
improves over the problems of large amount of data to be calculated, high time
complexity
and long duration of calculation inherent in the SIFT algorithm at the same
time of
maintaining the advantages of the SIFT algorithm.
190. The computer device of any one of claims 149 to 189, wherein SURF is more
excellent in
performance in the aspect of invariance of illumination change and perspective
change,
particularly excellent in processing severe blurs and rotations of images.
191. The computer device of any one of claims 149 to 190, wherein SURF is
excellent in describing
local features of images.
192. The computer device of any one of claims 149 to 191, wherein FAST feature
detection is a
corner detection method.
193. The computer device of any one of claims 149 to 192, wherein FAST feature
detection has
the most prominent advantage of its algorithm in its calculation efficiency,
and the capability
to excellently describe global features of images.
54
194. The computer device of any one of claims 149 to 193, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction gives consideration to global features of images.
195. The computer device of any one of claims 149 to 194, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction fully retains local features of images.
196. The computer device of any one of claims 149 to 195, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction has small computational expense.
197. The computer device of any one of claims 149 to 196, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction has strong robustness against image blurs and faint
illumination.
198. The computer device of any one of claims 149 to 197, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction enhances real-time property and precision of the detection.
199. The computer device of any one of claims 147 to 198, wherein while the
optical flow tracking
calculation is performed on the frame feature point sequence matrix, a pyramid
optical flow
tracking LK (Lucas-Kanade) algorithm is employed.
200. The computer device of any one of claims 147 to 199, wherein the change
of a feature point
sequence matrix Z, in the ith frame is tracked to the next i+1th frame, and a
motion vector cti
is obtained, whose expression is as follows:
<IMG>
where dxt represents a Euclidean column offset from the ith frame to the i+lth
frame, clyi
represents a Euclidean row offset from the ith frame to the i+lth frame, and
dr' represents
an angle offset from the ith frame to the i+lth frame.
201. The computer device of any one of claims 147 to 200, wherein use of the/a
pyramid optical
flow tracking LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative
structure solves
the problem of failed tracking due to unduly large change from feature points
of a current
frame to feature points of an adjacent next frame.
202. The computer device of any one of claims 147 to 201, wherein use of the/a
pyramid optical
flow tracking LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative
structure solves
the problem of failed tracking due to unduly large change from feature points
of a current
frame to feature points of a next frame by an interval of N frames.
203. The computer device of any one of claims 147 to 202, wherein accumulative
integral
transfommtion is perfomied on the initial motion vector ai of each frame to
obtain an
accumulative motion vector, expressed as fli , of each frame, in which the
expression of the
accumulative motion vector /3i is as follows:
<IMG>
204. The computer device of any one of claims 147 to 203, wherein a sliding
average window is
used to smoothen the motion vector to
obtain a smoothened motion vector Y , whose
expression is:
56
<IMG>
where n represents the total number of frames of the video.
205. The computer device of any one of claims 147 to 204, wherein the radius
of the smooth
window is r, and its expression iS:
<IMG>
where ix indicates a parameter of the sliding window, and is a positive
number.
206. The computer device of any one of claims 147 to 205, wherein the specific
numerical value
of can be dynamically adjusted according to practical requirement.
207. The computer device of any one of claims 147 to 206, wherein the specific
numerical value
of is set as =30.
208. The computer device of any one of claims 147 to 207, wherein /3 and
are used to
readjust ai to obtain a readjusted motion vector whose expression is:
57
<IMG>
209. The computer device of any one of claims 147 to 208, wherein the
readjusted motion vector
is taken as the motion vector of each frame to participate in subsequent
calculation, so that
the calculation result is made more precise.
210. The computer device of any one of claims 147 to 209, wherein the motion
vectors of all frames
obtained are merged and converted into a matrix.
211. The computer device of any one of claims 147 to 210, wherein the motion
vector /14 is
<IMG>
converted into the form of a matrix ,
and unbiased standard deviations
of its elements are calculated by rows, the specific calculation expression is
as follows:
<IMG>
58
212. The computer device of any one of claims 147 to 211, wherein the unbiased
standard
<IMG>
deviations of the various elements in the matrix are respectively expressed as
<IMG>
, in which A represents the average value of the samples.
213. The computer device of any one of claims 147 to 212, wherein weights are
assigned to the
unbiased standard deviations of the various elements according to practical
requirements.
214. The computer device of any one of claims 147 to 213, wherein the unbiased
standard
deviations of the various elements are weighted and fused according to the
weights.
215. The computer device of any one of claims 147 to 214, wherein the weights
of the unbiased
standard deviations of the various elements can be dynamically readjusted
according to
practical requirements.
<IMG>
216. The computer device of any one of claims 147 to 215, wherein the weight
of is
<IMG> <IMG>
set as 3, the weight of is set as 3, and the weight of is
set as 10,
then the fusing expression is as follows:
<IMG>
217. The computer device of any one of claims 147 to 216, wherein the feature
value of the to-be-
detected video is the unbiased standard deviations of the various elements and
their weighted
value, which are expressed as:
{0[.1(dx)], cr[A(dy)], cy[2(dr)], KS)(s) .
218.A computer system for detecting video jitter, comprising:
a processor, configured to:
59
perform a framing process on a to-be-detected video to obtain a frame
sequence;
perform feature point detection on the frame sequence frame by frame, obtain
feature points of each frame by employing a feature point detecting algorithm
including fused FAST (Features from Accelerated Segment Test) and SURF
(Speeded-Up Robust Features), and generate a frame feature point sequence
matrix;
perform an operation on the frame feature point sequence man-ix to obtain a
motion vector of each frame based on an optical flow tracking algorithm;
obtain a feature value of the to-be-detected video according to the motion
vector of each frame; and
take the feature value of the to-be-detected video as an input signal of a
detection model to perform operation to obtain an output signal, and judge
whether jitter occurs to the to-be-detected video according to the output
signal;
a computer-readable storage medium.
219. The computer system of claim 218, wherein the computer system is further
configured to:
grayscale-process the frame sequence, and obtain a grayscale frame sequence;
and
denoise the grayscale frame sequence;
wherein the step of feature point detection on the frame sequence frame by
frame is feature
point detection on the preprocessed frame sequence frame by frame.
220. The computer system of any one of claims 218 to 219, wherein the computer
system is further
configured to:
employ the feature point detecting algorithm in which are the fused FAST
(Features from
Accelerated Segment Test) and the SURF (Speeded-Up Robust Features) to perform
feature point detection on the frame sequence frame by frame, and to obtain
the feature
points of each frame.
221. The computer system of any one of claims 218 to 220, wherein the computer
system is further
configured to:
perform optical flow tracking calculation on the frame feature point sequence
matrix of
each frame, and obtain an initial motion vector of each frame;
obtain a corresponding accumulative motion vector according to the initial
motion vector;
smoothen the accumulative motion vector, and obtain a smoothened motion
vector; and
employ the accumulative motion vector and the smoothened motion vector to
readjust the
initial motion vector of each frame, and obtain the motion vector of each
frame.
222. The computer system of any one of claims 218 to 221, wherein the computer
system is further
configured to:
merge and convert the motion vectors of all frames into a matrix, and
calculate unbiased
standard deviations of various elements in the matrix;
weight and fuse the unbiased standard deviations of the various elements, and
obtain a
weighted value; and
take the unbiased standard deviations of the various elements and the weighted
value as
the feature value of the to-be-detected video.
223. The computer system of any one of claims 218 to 222, wherein a to-be-
detected video is
obtained.
224. The computer system of any one of claims 218 to 223, wherein a framing
extraction process
is performed on the to-be-detected video to obtain a frame sequence
corresponding to the to-
be-detected video.
61
225. The computer system of any one of claims 218 to 224, wherein the frame
sequence is
expressed as L, (1A, 2, 3, ..., n) , where L, represents the th frame of the
video, and n represents
the total number of frames of the video.
226. The computer system of any one of claims 218 to 225, wherein a current
frame and an adjacent
next frame is selected from the to-be-detected video.
227. The computer system of any one of claims 218 to 226, wherein a current
frame and a next
frame by an interval of N frames is selected from the to-be-detected video.
228. The computer system of any one of claims 218 to 227, wherein
corresponding feature points
are obtained from the current frame and the adjacent next frame.
229. The computer system of any one of claims 218 to 228, wherein
corresponding feature points
are obtained from the current frame and the next frame by an interval of N
frames.
230. The computer system of any one of claims 218 to 229, wherein
corresponding matching is
performed according to the feature points of the two frames to judge whether
offset (jitter)
occurs between the current frame and the adjacent next frame.
231. The computer system of any one of claims 218 to 230, wherein
corresponding matching is
performed according to the feature points of the two frames to judge whether
offset (jitter)
occurs between the current frame and the next frame by an interval of N
frames.
232. The computer system of any one of claims 218 to 231, wherein a feature
point detecting
algorithm is employed to perform feature point detection on the processed
frame sequence L,
(i=1, 2, 3, ..., n) frame by frame.
233. The computer system of any one of claims 218 to 232, wherein a feature
point detecting
algorithm is employed to obtain feature points of each frame, that is, feature
points of each
frame of image are extracted.
234. The computer system of any one of claims 218 to 233, wherein a feature
point detecting
algorithm is employed to generate a frame feature point sequence matrix, which
is expressed
as Z, (i=1, 2, ..., n).
62
235. The computer system of any one of claims 218 to 234, wherein Z (1=1, 2,
..., n) is expressed
as:
<IMG>
Wherein /3'g represents a feature point detection result at row p, column q of
the ith frame
matrix, 1 is a feature point, 0 is a non-feature point, p represents the
number of rows of the
matrix, and q represents the number of columns of the matrix.
236. The computer system of any one of claims 218 to 235, wherein the optical
flow tracking
algorithm is employed to perform optical flow tracking calculation on the
frame feature point
sequence matrix.
237. The computer system of any one of claims 218 to 236, wherein the optical
flow tracking
algorithm is employed to track the change of feature points in the current
frame to the next
fram e.
238. The computer system of any one of claims 218 to 237, wherein the change
of a feature point
sequence matrix Z in the ith frame is tracked to the next i+ith frame, and a
motion vector ai
is obtained, whose expression is as follows:
<IMG>
63
where dxt represents a Euclidean column offset from the ith frame to the i+lth
frame, clyi
represents a Euclidean row offset from the ith frame to the i+lth frame, and
dr' represents
an angle offset from the th frame to the i+Ph frame.
239. The computer system of any one of claims 218 to 238, wherein an extracted
feature value at
least includes the feature value of four dimensions.
240. The computer system of any one of claims 218 to 239, wherein the
detection model is well
trained in advance.
241. The computer system of any one of claims 218 to 240, wherein sample video
data in a set of
selected training data is correspondingly processed to obtain a feature value
of the sample
video data.
242. The computer system of any one of claims 218 to 241, wherein a detection
model is trained
according to the feature value of the sample video data and a corresponding
annotation result
of the sample video data to obtain the final detection model.
243. The computer system of any one of claims 218 to 242, wherein after the
motion vectors have
been dimensionally converted for an frith video sample, unbiased standard
deviations of
various elements and their weighted and fused value are calculated and
obtained, which are
<IMG>
respectively expressed as
and K m , and the
annotation result ym of the mth video sample is extracted to obtain the
training sample of the
/nth video sample, which is expressed as follows:
[2.(dx)]n cr[A(cly)].rn ak(dr)1m ICm ym)(m) .
244. The computer system of any one of claims 218 to 243, wherein the
annotation result ym
indicates that no jitter occurs to the video sample if ym-0.
245. The computer system of any one of claims 218 to 244, wherein the
annotation result ym
indicates that jitter occurs to the video sample if ym=1.
64
246. The computer system of any one of claims 218 to 245, wherein a video
sample makes use of
features of at least five dimensions.
247. The computer system of any one of claims 218 to 246, wherein the
detection model is selected
from an SVM (Support Vector Machine) model.
248. The computer system of any one of claims 218 to 247, wherein the feature
value of the to-be-
detected video is input to a well-trained SVM model to obtain an output
result.
249. The computer system of any one of claims 218 to 248, wherein if the SVM
model output
result is 0, this indicates that no jitter occurs to the to-be-detected video.
250. The computer system of any one of claims 218 to 249, wherein if the SVM
model output
result is 1, this indicates that jitter occurs to the to-be-detected video.
251. The computer system of any one of claims 218 to 250, wherein the use of a
trainable SVM
model as a video jitter decider enables jitter detection of videos captured in
different scenarios.
252. The computer system of any one of claims 218 to 251, wherein the amount
of information of
the image is greatly reduced after grayscale-processing.
253. The computer system of any one of claims 218 to 252, wherein the frame
sequence L, (i=1, 2,
3, ..., n) is further grayscale-processed.
254. The computer system of any one of claims 218 to 253, wherein a grayscale
frame sequence is
obtained to be expressed as G, (i=1, 2, 3, ..., n) , in which the grayscale
conversion expression
is as follows:
G=Rx0.299+Gx0.587+Bx0.1.IA
255. The computer system of any one of claims 218 to 254, wherein a TV
denoising method based
on a total variation model is employed to denoise the grayscale frame sequence
G, (i=1, 2,
3, ..., n) to obtain a denoised frame sequence expressed as T, (i=1, 2, 3,
..., n) .
256. The computer system of any one of claims 218 to 255, wherein the denoised
frame sequence
is a preprocessed frame sequence to which the to-be-detected video
corresponds.
257. The computer system of any one of claims 218 to 256, wherein the
denoising method is
randomly selectable.
258. The computer system of any one of claims 220 to 257, wherein the SURF
algorithm is an
improvement over SIFT (Scale-Invariant Feature Transform) algorithm.
259. The computer system of claims 258, wherein SIFT is a feature describing
method with
excellent robustness and invariant scale.
260. The computer system of any one of claims 258 to 259, wherein the SURF
algorithm improves
over the problems of large amount of data to be calculated, high time
complexity and long
duration of calculation inherent in the SIFT algorithm at the same time of
maintaining the
advantages of the SIFT algorithm.
261. The computer system of any one of claims 220 to 260, wherein SURF is more
excellent in
performance in the aspect of invariance of illumination change and perspective
change,
particularly excellent in processing severe blurs and rotations of images.
262. The computer system of any one of claims 220 to 261, wherein SURF is
excellent in
describing local features of images.
263. The computer system of any one of claims 220 to 262, wherein FAST feature
detection is a
corner detection method.
264. The computer system of any one of claims 220 to 263, wherein FAST feature
detection has
the most prominent advantage of its algorithm in its calculation efficiency,
and the capability
to excellently describe global features of images.
265. The computer system of any one of claims 220 to 264, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction gives consideration to global features of images.
66
266. The computer system of any one of claims 220 to 265, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction fully retains local features of images.
267. The computer system of any one of claims 220 to 266, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction has small computational expense.
268. The computer system of any one of claims 220 to 267, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction has strong robustness against image blurs and faint
illumination.
269. The computer system of any one of claims 220 to 268, wherein use of the
feature point
detecting algorithm in which FAST features and SURF features are fused to
perform feature
point extraction enhances real-time property and precision of the detection.
270. The computer system of any one of claims 218 to 269, wherein while the
optical flow tracking
calculation is performed on the frame feature point sequence matrix, a pyramid
optical flow
tracking LK (Lucas-Kanade) algorithm is employed.
271. The computer system of any one of claims 218 to 270, wherein the change
of a feature point
sequence matrix Z in the ith frame is tracked to the next i+ith frame, and a
motion vector ai
is obtained, whose expression is as follows:
<IMG>
where dr' represents a Euclidean column offset from the ith frame to the i+lth
frame, dy'
represents a Euclidean row offset from the ith frame to the i+lth frame, and
dri represents
an angle offset from the th frame to the i+lth frame.
67
272. The computer system of any one of claims 218 to 271, wherein use of the/a
pyramid optical
flow tracking LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative
structure solves
the problem of failed tracking due to unduly large change from feature points
of a current
frame to feature points of an adjacent next frame.
273. The computer system of any one of claims 218 to 272, wherein use of the/a
pyramid optical
flow tracking LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative
structure solves
the problem of failed tracking due to unduly large change from feature points
of a current
frame to feature points of a next frame by an interval of N frames.
274. The computer system of any one of claims 218 to 273, wherein accumulative
integral
transformation is performed on the initial motion vector Cti of each frame to
obtain an
accumulative motion vector, expressed as Pi , of each frame, in which the
expression of the
accumulative motion vector Pi is as follows:
<IMG>
275. The computer system of any one of claims 218 to 274, wherein a sliding
average window is
used to smoothen the motion vector 13 to obtain a smoothened motion vector Y ,
whose
expression is:
68
<IMG>
where n represents the total number of frames of the video.
276. The computer system of any one of claims 218 to 275, wherein the radius
of the smooth
window is r, and its expression is:
<IMG>
where ix indicates a parameter of the sliding window, and is a positive
number.
277. The computer system of any one of claims 218 to 276, wherein the specific
numerical value
of can be dynamically adjusted according to practical requirement.
278. The computer system of any one of claims 218 to 277, wherein the specific
numerical value
of is set as =30.
279. The computer system of any one of claims 218 to 278, wherein /6 and
are used to
readjust ai to obtain a readjusted motion vector whose expression is:
69
<IMG>
280. The computer system of any one of claims 218 to 279, wherein the
readjusted motion vector
is taken as the motion vector of each frame to participate in subsequent
calculation, so that
the calculation result is made more precise.
281. The computer system of any one of claims 218 to 280, wherein the motion
vectors of all
frames obtained are merged and converted into a matrix.
282. The computer system of any one of claims 218 to 281. wherein the motion
vector /14 is
<IMG>
converted into the form of a matrix ,
and unbiased standard deviations
of its elements are calculated by rows, the specific calculation expression is
as follows:
<IMG>
283. The computer system of any one of claims 218 to 282, wherein the unbiased
standard
deviations of the various elements in the matrix are respectively expressed as
<IMG>
<IMG>
, in which A represents the average value of the samples.
284. The computer system of any one of claims 218 to 283, wherein weights are
assigned to the
unbiased standard deviations of the various elements according to practical
requirements.
285. The computer system of any one of claims 218 to 284, wherein the unbiased
standard
deviations of the various elements are weighted and fused according to the
weights.
286. The computer system of any one of claims 218 to 285, wherein the weights
of the unbiased
standard deviations of the various elements can be dynamically readjusted
according to
practical requirements.
<IMG>
287. The computer system of any one of claims 218 to 286, wherein the weight
of is
<IMG> <IMG> set as 3, the weight of is
set as 3, and the weight of is set as 10,
then the fusing expression is as follows:
<IMG>
288. The computer system of any one of claims 218 to 287, wherein the feature
value of the to-be-
detected video is the unbiased standard deviations of the various elements and
their weighted
value, which are expressed as:
{cî[A(dx)] o-[A(dy)], a[2.(dr)],
289. The computer system of any one of claims 218 to 288, wherein the computer-
readable storage
medium is a read-only memory.
71
290. The computer system of any one of claims 218 to 289, wherein the computer-
readable storage
medium is a magnetic disk.
291. The computer system of any one of claims 218 to 290, wherein the computer-
readable storage
medium is an optical disk.
292.A method of detecting video jitter, comprising:
performing a framing process on a to-be-detected video to obtain a frame
sequence;
performing feature point detection on the frame sequence frame by frame,
obtaining feature
points of each frame by employing a feature point detecting algorithm
including fused
FAST (Features from Accelerated Segment Test) and SURF (Speeded-Up Robust
Features), and generating a frame feature point sequence matrix;
basing on an optical flow tracking algorithm to perform an operation on the
frame feature
point sequence matrix to obtain a motion vector of each frame;
obtaining a feature value of the to-be-detected video according to the motion
vector of each
frame; and
taking the feature value of the to-be-detected video as an input signal of a
detection model
to perform operation to obtain an output signal, and judging whether jitter
occurs to the to-
be-detected video according to the output signal.
293. The method of claim 292, wherein the method further includes:
grayscale-processing the frame sequence, and obtaining a grayscale frame
sequence; and
denoising the grayscale frame sequence;
the step of performing feature point detection on the frame sequence frame by
frame is
performing feature point detection on the preprocessed frame sequence frame by
frame.
294. The method of any one of claims 292 to 293, wherein the method further
includes:
72
employing the feature point detecting algorithm in which are the fused FAST
(Features
from Accelerated Segment Test) and the SURF (Speeded-Up Robust Features) to
perform
feature point detection on the frame sequence frame by frame, and to obtain
the feature
points of each frame.
295. The method of any one of claims 292 to 294, wherein the method further
includes:
performing optical flow tracking calculation on the frame feature point
sequence matrix of
each frame, and obtaining an initial motion vector of each frame;
obtaining a corresponding accumulative motion vector according to the initial
motion
vector;
smoothening the accumulative motion vector, and obtaining a smoothened motion
vector;
and
employing the accumulative motion vector and the smoothened motion vector to
readjust
the initial motion vector of each frame, and obtaining the motion vector of
each frame.
296. The method of any one of claims 292 to 295, wherein the method further
includes:
merging and converting the motion vectors of all frames into a matrix, and
calculating
unbiased standard deviations of various elements in the matrix;
weighting and fusing the unbiased standard deviations of the various elements,
and
obtaining a weighted value; and
taking the unbiased standard deviations of the various elements and the
weighted value as
the feature value of the to-be-detected video.
297. The method of any one of claims 292 to 296, wherein a to-be-detected
video is obtained.
298. The method of any one of claims 292 to 297, wherein a framing extraction
process is
performed on the to-be-detected video to obtain a frame sequence corresponding
to the to-be-
detected video.
73
299. The method of any one of claims 292 to 298, wherein the frame sequence is
expressed as L,
(i=1, 2, 3, ..., n), where L, represents the th frame of the video, and n
represents the total
number of frames of the video.
300. The method of any one of claims 292 to 299, wherein a current frame and
an adjacent next
frame is selected from the to-be-detected video.
301. The method of any one of claims 292 to 300, wherein a current frame and a
next frame by an
interval of N frames is selected from the to-be-detected video.
302. The method of any one of claims 292 to 301, wherein corresponding feature
points are
obtained from the current frame and the adjacent next frame.
303. The method of any one of claims 292 to 302, wherein corresponding feature
points are
obtained from the current frame and the next frame by an interval of N frames.
304. The method of any one of claims 292 to 303, wherein corresponding
matching is performed
according to the feature points of the two frames to judge whether offset
(jitter) occurs
between the current frame and the adjacent next frame.
305. The method of any one of claims 292 to 304, wherein corresponding
matching is performed
according to the feature points of the two frames to judge whether offset
(jitter) occurs
between the current frame and the next frame by an interval of N frames.
306. The method of any one of claims 292 to 305, wherein a feature point
detecting algorithm is
employed to perform feature point detection on the processed frame sequence L,
(1=1, 2, 3, ...,
n) frame by frame.
307. The method of any one of claims 292 to 306, wherein a feature point
detecting algorithm is
employed to obtain feature points of each frame, that is, feature points of
each frame of image
are extracted.
308. The method of any one of claims 292 to 307, wherein a feature point
detecting algorithm is
employed to generate a frame feature point sequence matrix, which is expressed
as Z, (i=1,
2, ..., n).
74
309. The method of any one of claims 292 to 308, wherein Z (i=1, 2, ..., n) is
expressed as:
<IMG>
Wherein apq
'
represents a feature point detection result at row p, column q of the ith
frame
matrix, 1 is a feature point, 0 is a non-feature point, p represents the
number of rows of the
matrix, and q represents the number of columns of the matrix.
310. The method of any one of claims 292 to 309, wherein the optical flow
tracking algorithm is
employed to perform optical flow tracking calculation on the frame feature
point sequence
matrix.
311. The method of any one of claims 292 to 310, wherein the optical flow
tracking algorithm is
employed to track the change of feature points in the current frame to the
next frame.
312. The method of any one of claims 292 to 311, wherein the change of a
feature point sequence
matrix Z in the ith frame is tracked to the next i+ lth frame, and a motion
vector ai is obtained,
whose expression is as follows:
<IMG>
where dxf represents a Euclidean column offset from the ith frame to the i+lth
frame, dyi
represents a Euclidean row offset from the ith frame to the i+lth frame, and
dri represents
an angle offset from the ith frame to the i+lth frame.
313. The method of any one of claims 292 to 312, wherein an extracted feature
value at least
includes the feature value of four dimensions.
314. The method of any one of claims 292 to 313, wherein the detection model
is well trained in
advance.
315. The method of any one of claims 292 to 314, wherein sample video data in
a set of selected
training data is correspondingly processed to obtain a feature value of the
sample video data.
316. The method of any one of claims 292 to 315, wherein a detection model is
trained according
to the feature value of the sample video data and a corresponding annotation
result of the
sample video data to obtain the final detection model.
317. The method of any one of claims 292 to 316, wherein after the motion
vectors have been
dimensionally converted for an 171th video sample, unbiased standard
deviations of various
elements and their weighted and fused value are calculated and obtained, which
are
<IMG>
respectively expressed as ,
and the
annotation result ym of the 117th video sample is extracted to obtain the
training sample of the
177th video sample, which is expressed as follows:
fak(dx)lm a[A(dy)].õ, a[A.(dr)lm Km ym)(m).
318. The method of any one of claims 292 to 317, wherein the annotation
resultym indicates that
no jitter occurs to the video sample if ym-0.
319. The method of any one of claims 292 to 318, wherein the annotation
resultym indicates that
jitter occurs to the video sample if ym=1.
320. The method of any one of claims 292 to 319, wherein a video sample makes
use of features
of at least five dimensions.
321. The method of any one of claims 292 to 320, wherein the detection model
is selected from an
SVM (Support Vector Machine) model.
76
322. The method of any one of claims 292 to 321, wherein the feature value of
the to-be-detected
video is input to a well-trained SVM model to obtain an output result.
323. The method of any one of claims 292 to 322, wherein if the SVM model
output result is 0,
this indicates that no jitter occurs to the to-be-detected video.
324. The method of any one of claims 292 to 323, wherein if the SVM model
output result is 1,
this indicates that jitter occurs to the to-be-detected video.
325. The method of any one of claims 292 to 324, wherein the use of a
trainable SVM model as a
video jitter decider enables jitter detection of videos captured in different
scenarios.
326. The method of any one of claims 292 to 325, wherein the amount of
information of the image
is greatly reduced after grayscale-processing.
327. The method of any one of claims 292 to 326, wherein the frame sequence L,
(1=1, 2, 3, ..., n)
is further grayscale-processed.
328. The method of any one of claims 292 to 327, wherein a grayscale frame
sequence is obtained
to be expressed as G, (i=1, 2, 3, ..., n), in which the grayscale conversion
expression is as
follows:
G=Rx0.299+Gx0.587+Bx0.114
329. The method of any one of claims 292 to 328, wherein a TV denoising method
based on a total
variation model is employed to denoise the grayscale frame sequence G, (i=1,
2, 3, ..., n) to
obtain a denoised frame sequence expressed as T1(i=1, 2, 3, ..., n) .
330. The method of any one of claims 292 to 329, wherein the denoised frame
sequence is a
preprocessed frame sequence to which the to-be-detected video corresponds.
331. The method of any one of claims 292 to 330, wherein the denoising method
is randomly
selectable.
77
332. The method of any one of claims 294 to 331, wherein the SURF algorithm is
an improvement
over SIFT (Scale-Invariant Feature Transform) algorithm.
333. The method of claim 332, wherein SIFT is a feature describing method with
excellent
robustness and invariant scale.
334. The method of any one of claims 332 to 333, wherein the SURF algorithm
improves over the
problems of large amount of data to be calculated, high time complexity and
long duration of
calculation inherent in the SIFT algorithm at the same time of maintaining the
advantages of
the SIFT algorithm.
335. The method of any one of claims 294 to 334, wherein SURF is more
excellent in performance
in the aspect of invariance of illumination change and perspective change,
particularly
excellent in processing severe blurs and rotations of images.
336. The method of any one of claims 294 to 335, wherein SURF is excellent in
describing local
features of images.
337. The method of any one of claims 294 to 336, wherein FAST feature
detection is a comer
detection method.
338. The method of any one of claims 294 to 337, wherein FAST feature
detection has the most
prominent advantage of its algorithm in its calculation efficiency, and the
capability to
excellently describe global features of images.
339. The method of any one of claims 294 to 338, wherein use of the feature
point detecting
algorithm in which FAST features and SURF features are fused to perform
feature point
extraction gives consideration to global features of images.
340. The method of any one of claims 294 to 339, wherein use of the feature
point detecting
algorithm in which FAST features and SURF features are fused to perform
feature point
extraction fully retains local features of images.
78
341. The method of any one of claims 294 to 340, wherein use of the feature
point detecting
algorithm in which FAST features and SURF features are fused to perform
feature point
extraction has small computational expense.
342. The method of any one of claims 294 to 341, wherein use of the feature
point detecting
algorithm in which FAST features and SURF features are fused to perform
feature point
extraction has strong robustness against image blurs and faint illumination.
343. The method of any one of claims 294 to 342, wherein use of the feature
point detecting
algorithm in which FAST features and SURF features are fused to perform
feature point
extraction enhances real-time property and precision of the detection.
344. The method of any one of claims 292 to 343, wherein while the optical
flow tracking
calculation is performed on the frame feature point sequence matrix, a pyramid
optical flow
tracking LK (Lucas-Kanade) algorithm is employed.
345. The method of any one of claims 292 to 344, wherein the change of a
feature point sequence
matrix Z in the ith frame is tracked to the next i+1 th frame, and a motion
vector ai is obtained,
whose expression is as follows:
<IMG>
where dxi represents a Euclidean column offset from the ith frame to the i+lth
frame, dy'
represents a Euclidean row offset from the ith frame to the i+ 1 th frame, and
dr' represents
an angle offset from the ith frame to the i+lth frame.
346. The method of any one of claims 292 to 345, wherein use of the pyramid
optical flow tracking
LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative structure
solves the problem
of failed tracking due to unduly large change from feature points of a current
frame to feature
points of an adjacent next frame.
79
347. The method of any one of claims 292 to 346, wherein use of the pyramid
optical flow tracking
LK (Lucas-Kanade) algorithm that utilizes the pyramid iterative structure
solves the problem
of failed tracking due to unduly large change from feature points of a current
frame to feature
points of a next frame by an interval of N frames.
348. The method of any one of claims 292 to 347, wherein accumulative integral
transformation is
performed on the initial motion vector ai of each frame to obtain an
accumulative motion
vector, expressed as fli , of each frame, in which the expression of the
accumulative motion
vector is as follows:
<IMG>
349. The method of any one of claims 292 to 348, wherein a sliding average
window is used to
smoothen the motion vector fii to obtain a smoothened motion vector r , whose
expression is:
<IMG>
where n represents the total number of frames of the video.
350. The method of any one of claims 292 to 349, wherein the radius of the
smooth window is r,
and its expression is:
<IMG>
where ix indicates a parameter of the sliding window, and la is a positive
number.
351. The method of any one of claims 292 to 350, wherein the specific
numerical value of ti can
be dynamically adjusted according to practical requirement.
352. The method of any one of claims 292 to 351, wherein the specific
numerical value of 1.1 is set
as j.t=30.
353. The method of any one of claims 292 to 352, wherein /3 and
are used to readjust ai to
obtain a readjusted motion vector 4, whose expression is:
81
<IMG>
354. The method of any one of claims 292 to 353, wherein the readjusted motion
vector /1.4 is taken
as the motion vector of each frame to participate in subsequent calculation,
so that the
calculation result is made more precise.
355. The method of any one of claims 292 to 354, wherein the motion vectors of
all frames obtained
are merged and converted into a matrix.
356. The method of any one of claims 292 to 355, wherein the motion vector Ai
is converted into
<IMG>
the form of a matiix ,
and unbiased standard deviations of its elements
are calculated by rows, the specific calculation expression is as follows:
<IMG>
82
357. The method of any one of claims 292 to 356, wherein the unbiased standard
deviations of the
<IMG>
various elements in the matrix are respectively expressed as
and
<IMG>
in which A represents the average value of the samples.
358. The method of any one of claims 292 to 357, wherein weights are assigned
to the unbiased
standard deviations of the various elements according to practical
requirements.
359. The method of any one of claims 292 to 358, wherein the unbiased standard
deviations of the
various elements are weighted and fused according to the weights.
360. The method of any one of claims 292 to 359, wherein the weights of the
unbiased standard
deviations of the various elements can be dynamically readjusted according to
practical
requirements.
<IMG>
361. The method of any one of claims 292 to 360, wherein the weight of is
set as 3,
<IMG> <IMG>
the weight of is set as 3, and the weight of is
set as 10, then the
fusing expression is as follows:
<IMG>
362. The method of any one of claims 292 to 361, wherein the feature value of
the to-be-detected
video is the unbiased standard deviations of the various elements and their
weighted value,
which are expressed as:
{cî[A(dx)] o-[A(dy)], a[2.(dr)],
83
