Note: Descriptions are shown in the official language in which they were submitted.
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
IMAGE-SYNCHRONIZED MULTICHANNEL
BIOMEDICAL DATA ACQUISITION SYSTEM
FIELD OF THE INVENTION
The present invention is directed to event synchronized data
acquisition systems, and more particularly to video synchronized multi
channel data acquisition for the analysis of biomedical information.
BACKGROUND OF THE INVENTION
Video-synchronized data acquisition is finding more applica-
tions in biomedical engineering research. Some of these applications require
to a video-synchronized data-acquisition system with multi-data channels and
high-frequency bandwidth, and also maintain long acquisition duration.
One example of high-frequency application is frequency-domain muscle-
fatigue analysis, for which a mufti-channel data acquisition system with a
high raw-data signal frequency bandwidth of up to 3 kHz per channel is
s5 recommended by D. G. Gerleman and T.M. Cook, "Instrumentation," in
Selected Topics in Surface Electromyography for Use in the Occupational
Setting:
Expert Perspectives, U.S. Department of Health and Human Services, Public
Health Service, Centers for Disease Control, National Institute for Occupa-
tional Safety and Health, pp. 44-68,1992. A number of schemes have been
2o published to achieve video-synchronized data acquisition, such as, D. M.
Gaskill, "Techniques for synchronizing thermal array chart recorders to
video," 28th International Telemetering Conference (Proceedings)-
ITC/USA/92, Vol. 28, pp. 61-64,1992; M. Vannier et al., "Time and motion
studies of medical imaging workstations," SPIE, Vol.1653, Image Capture,
25 Formatting, and Display, pp. 274-280,1992; T. Y Yen and R. G. Radwin, "A
video-based system for acquiring biomechanical data synchronized with
arbitrary events and activities," IEEE Transactions on Biomedical Engineering,
Vol. 42, pp. 944-948,1995; M. Gacia et al., "Characterization of near-bed
coherent structures in turbulent open channel flow using synchronized high-
-1-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
speed video and hot-film measurements," Experiments in Fluids, Vol. 19, pp.
16-28,1995; T. Engstrom and P. Medbo, "Data collection and analysis of
manual work using video recording and personal computer techniques,"
International Journal of Industrial Ergonomics,19 (1997), pp. 291-298,1997.
Some of the schemes consist of simple event analyses stamped
with video time codes, e.g., M. Vannier et al., "Time and motion studies of
medical imaging workstations," SPIE, Vol.1653, Image Capture, Formatting,
and Display, pp. 274-280,1992; T. Engstrom and P. Medbo, "Data collection
and analysis of manual work using video recording and personal computer
to techniques," International Journal of Industrial Ergonomics,19 (1997), pp.
291-298,1997. Some other sophisticated schemes have achieved multi-
channel data acquisition. Yen and Radwin utilized an audio-track digital
recording method to record up to 32 channels of 8-bit data onto video tape
audio tracks, and to record corresponding video images onto the video track
of the same tape. See, T. Y Yen and R. G. Radwin, "A video-based system
for acquiring biomechanical data synchronized with arbitrary events and
activities," IEEE Transactions on Biomedical Engineering, Vol. 42, pp. 944-
948,
1995. This method naturally synchronizes the video frames with the
acquired data for extensive recording duration. But, because of the
2o frequency-bandwidth limit of the audio tracks (20 Hz - 20 kHz), the
scanning
frequency of the recorded data is limited to 60 Hz per channel. Higher-
scanning-rate multi-channel data acquisition was realized by using a high-
speed video camera (1000 frames/sec) which outputs a synchronization
signal in each video frame to trigger a computer data acquisition sequence.
See, M. Gacia et al., "Characterization of near-bed coherent structures in
turbulent open channel flow using synchronized high-speed video and hot-
film measurements," Experiments in Fluids, Vol.19, pp. 16-28,1995. The
maximum scanning frequency of this method may reach 1000 Hz per
channel for 20 seconds of 1000 frames per second video recording. To
3o acquire high frequency multi-channel data for longer duration, Gaskill
_2_
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
suggested a data acquisition system using a digital audio tape recorder or a
thermal chart recorder controlled by video time codes or vertical-video-sync
pulses output from the video recording system.
This literature reports a new computerized data-acquisition
system which extends Gaskill's suggestion by using a computer synchroni-
zation program to realize high-frequency long-duration video-synchronized
biomedical-data acquisition. It would be desirable therefore to provide a
broad band mufti-channel video-synchronized data acquisition system
which correlates video signal information with numerous data leads
io associated with biomedical signals and the like. Additionally, it would be
advantageous to provide for the acquisition of frequency domain signals
which may be correlated with video signals acquired by the system. The
system records the storage-intensive video images onto a video tape, and
simultaneously acquires biomedical data and video time codes onto a
computer hard disk. A time-code-bridge-file is created by the computer to
synchronize the acquired biomedical data on the hard drive with the
recorded human images on the video tape. Without manipulating video
signals in the system computer, this separated recording method boosts the
data acquisition speed, enables the realtime data synchronization and long
2o recording duration. The video time-code frame-start bits were selected as
the synchronization reference. With this reference, the synchronization error
can be theoretically controlled within one SMPTE time-code bit duration
(0.417 mS).
SUMMARY OF THE INVENTION
A video-synchronized mufti-channel data acquisition system is
provided for frequency domain analysis of biomedical signals acquired with
video signals, and particularly for correlating analysis with muscle fatigue
research. The disclosed system records storage-intensive video images onto
a video tape, and also provides for the data acquisition of information
-3-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
signals with associated video time codes on a personal computer hard disk
to achieve high speed recording over a long duration. A video time code-
bridge file has been provided for use by the computer to synchronize the
biomedical data with the recorded video frame in real time.
Briefly summarized, the present invention relates to a data
acquisition system, in which an apparatus such as a video tape recorder is
provided for recording video frames from an input video signal. A
biomedical instrument generates an input biomedical data stream to a
system computer providing at least one digital input-output port and
to interface circuitry for collecting the input video time code signal and for
collecting the input biomedical data stream. A bridge file is provided with
the system computer for synchronizing the video frames from the input
video signal with the input biomedical data stream.
The appended claims set forth the features of the present
invention with particularity. The invention and its advantages may be best
understood from the following detailed description taken in conjunction
with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG.1 shows a video-synchronized biomedical data acquisition
2o system in accordance with the invention;
FIG. 2 illustrates in block diagram form the program flow of
the data acquisition system of FIG. 1;
FIG. 3 illustrates a longitudinal time code output decoding
procedure in accordance with the invention;
FIG. 4 shows the structure of the program files of the data
acquisition system correlating acquired data file information with elements
per video frame;
FIG. 5 shows the structure of the interface board shown in the
system diagram of FIG.1;
-4-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
FIGS. 6a and 6b illustrate the instrumentation setup for
synchronizing extracted video information with biomedical signal
acquisition in accordance with the invention; and
FIG. 7 illustrates an instrumental setup for data and video
acquisition by one computer;
FIG. 8 illustrates block diagrams for data and video acquisition
by one computer;
FIG. 9 illustrates program file structures for data and video
acquisition by one computer;
so FIG.10 illustrates an instrumental setup for data and video
acquisition by separate computers;
FIG.11 illustrates block diagrams for data and video
acquisition by separate computers; and
FIG. 12 illustrates two time-code-bridge-files for data and
video acquisition by separate computers.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
As shown in FIG.1, a computerized video-synchronized data
acquisition system 10 is shown a video camera 12 to output NTSC color
video signals 14, a digital VTR 16 with a SMPTE longitudinal time code
(LTC) output 22, a TV monitor 18, a mufti-channel biomedical-signal-
measuring instrument 20 with parallel digital output 24, a signal-interface
board 26 to condition the incoming (22,24) and outgoing (28,30) data and
control the direction of data flow, a 266-MHz Pentium II microprocessor-
based system PC computer 32 with a 32-bit digital I/O board, and an
analysis PC computer 34 if the system needs to run the instrument's own
analysis software. LabVIEW graphical programming software (National
Instruments, Austin, TX) was used to program the data acquisition,
processing, storage and replay, and VTR control.
-5-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
The system block diagrams are shown in FIG. 2. During video-
synchronized data acquisition (FIG. 2 (a)), the VTR records NTSC human
activity images and outputs a 1-bit LTC stream to the interface board. The
LTC stream is bi-phase modulated by frame numbers and sync words, as
shown in FIG. 3(a). Such techniques for modulations with frame numbers
and sync word have been demonstrated in D. M. Huber, Audio production
technigues for video. White Plains, NY and London: Knowledge Industry
Publications, Inc., pp. 93-100,1987.
FIG. 2 represents three flow charts, i.e., video synchronized
to data acquisition 36 (2a); data search 38 (2b); and video search 40 (2c).
The
video synchronization data acquisition 36 is illustrated in block diagram
form in 2(a), and further broken down into hardware component 54 and
software component 56. The hardware component 54 illustrates video
recording 42 and data recording 44 which may be provided simultaneously
i5 to provide longitudinal time code (LTC) output to record human activity
with video imaging, and additionally provide a biomedical data stream from
a data recording instrument 44, which is provided as a mufti-channel
electromyography (EMG) data acquisition instrument. An interface board
46 corresponding to the interface board 26 of FIG.1 operates in a data
2o acquisition mode to receive the LTC and data signals from the video
recording and data recording blocks 42 and 44 respectively. Accordingly,
the interface board in its data acquisition mode 46 provides output signals to
display biomedical data on an analysis computer monitor 48, and an output
signal to a data acquisition board 50. The data acquisition at block 50
25 provides write data to the system computer buffer at block 52. Next, a
software loop is provided in the software section 56, which obtains data
from the system computer buffer in a read data buffer block 58. An acquired
data file is written to the hard disk of the system computer at block 60, and
an LTC decode is performed at block 62. Block 64 is provided to correct LTC
3o errors, and interpolation of the LTC signal at block 66 is used with the
-6-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
bridge file elements in a block 68 for forming bridge file elements. The
bridge file elements from block 68 are provided as appended elements to the
bridge file on the hard disk at block 70.
At FIG. 2(c), a data search 38 is illustrated in which a replay of
the video tape is performed at 72, from which selected video frames at 74 are
presented to the interface board 26 in its acquisition mode at block 76. A
decode of the LTC selected frame at 78 along with a search of the bridge file
80 is used to retrieve data from the acquired data file at block 82. The
interface board 26 then provided in a replay mode at block 84 facilitates the
io replay and output of data to an analysis computer at block 86.
At FIG. 2(c) the video search 40 is illustrated, wherein the
interface board 26 in its replay mode at block 88 replays a data segment 90
from which a selected data frame is provided at block 92. At block 94 the
selected data frame is used to determine LTC of the selected frame. An
RS 232 interface is facilitated at block 96 which is used with a program loop
for reading the video tape LTC starting at block 98. Block 100 is used to
compare the LTC difference and block 102 calculates forward or reverse
time. Block 104 converts the calculated time to forward or reverse on the
vide tape recorder 16. A loop decision block at 106 determines whether the
2o video frame selected is correct or forward or reverse video frames must be
used, and then if the video frame is correct at decision block 106, a display
of
the video frame is provided on the TV monitor at block 108.
FIGS. 3(a), 3(b), 3(c), and 3(d) illustrate the LTC decoding
procedure from a data stream 110 obtained from the video recording. In
FIG. 4(a), the acquired data file 112 is illustrated as several elements per
video frame (EPVF) arranged as 32-bit integer data elements provided with
a video frame number and an index number at the beginning of each frame.
FIG. 4(b) shows the interpolated time code bridge file which may be used to
correlate the video frame number with the LTC and data file index columns.
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
Simultaneously, the biomedical instrument measures human
biomedical signals, and outputs up to 31 bits of data/sync signals to the
interface board. The interface board in turn buffers the total of 32-bit
data/ sync/ LTC signals, and outputs them to the 32-bit digital I/ O board
(National Instrument AT-DIO-32F). The interface board also directs the
31-bit data/sync signals to the data analysis computer for realtime data
display. The digital I/ O board scans the 32-bit signal stream at a scanning
frequency of up to 72 kHz, converts the stream into 32-bit integers, and
writes the integers into the system computer buffer. Whenever the buffer is
to half full, the program retrieves all the integers from the buffer, and
appends
them to an acquired-data-file on the hard disk (see FIG. 4(a) for acquired-
data-file structure). In the meantime, the program retrieves the LTC bit
stream from the integers, and sends the LTC stream to the decoding
subroutine for LTC decoding. During decoding, the program determines
~5 the index number of the acquired-data-file at each video frame-start, and
translates each 80-bit LTC into a unique 8-digit frame number integer as
shown in FIGS. 3(b), 3(c) and 3(d). After decoding, the program forms a
time-code-bridge-file with a two-column array. As shown in FIG. 4(b), the
LTC column of the array contains the 8-digit frame-number integers of all
2o recorded video frames, and the data-file-index column contains the
corresponding index numbers of the acquired-data-file at frame-starts. In
this way, each row of the array represents a unique video frame and its
corresponding acquired-data-file index number at the frame-start. With this
time-code-bridge-file, the computer is able to search for any data segment
25 correspondent to a given video frame, or search for any video frame
correspondent to a given data segment.
The interface board has two flow-control modes: data-
acquisition mode and data-replay mode that are shown in FIGS. 5(a) and
5(b), respectively. The main task of the interface board is to control the
data
3o flow direction. FIGS. 5(a) and 5(b) illustrate the interface board 26 in
data
_g_
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
acquisition mode and data replay modes, respectively. As shown in
FIG. 5(a), the data acquisition mode configuration of the interface board 26
facilitates the receipt of LTC from the video tape recorder 16, as well as the
biomedical data stream from the biomedical instrument 20, which signals
are buffered in a tristate digital buffer of the data acquisition board 26,
which provides buffered output signals to the digital I/O board which
provides an input-output port for the system computer 32. Additionally, a
buffered output of data/sync signals are provided to the analysis computer
34 . In FIG. 5(b), the interface board 26 configured in the data replay mode
1o receives control data from the system computer 32, which provides data
signals to the analysis computer 34 to replay the acquired data to the
analysis computer 34.
The flow-control mode is selected by the program which
outputs a control bit from the digital I/O board to the interface board. The
interface board buffers the incoming and outgoing data and converts the
polarity of the LTC stream from ~10 V to 0-5 V. In data-acquisition mode,
the 31-bit data/ sync signals from the biomedical instrument, and the
corresponding 1-bit LTC stream from the VTR are input to the interface
board. The interface board directs the total 32-bit data/sync/LTC signals
2o to the data acquisition board for data acquisition, and directs the 31-bit
data/sync signals to the data analysis computer for realtime display. In
data-replay mode, the program outputs the desired 31-bit data/ sync signal
segment through the I/O board to the interface board. The interface board
directs these 31-bit data only to the data analysis computer for further data
analysis.
After every scan, the computer appends a 32-bit integer, which
contains data/sync/LTC signals, as a file element to the acquired-data-file
on the hard disk. As shown in FIG. 4(a), the acquired-data-file is a one-
column array containing a large number of elements that equals the
3o scanning frequency X seconds of recording. Every video frame of the
-9-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
acquired-data-file contains a number of elements which equals scanning
frequency = 29.97 frames/second.
The SMPTE LTC output from the VTR are bi-phase modulated,
as shown in FIG. 3(a). When the recorded signal pulse shifts up or down
only at the extremes of the period for a signal bit (417 ~S), the pulse is
coded
as a binary "0". A binary "1" is coded for a bit when a pulse shift occurs
halfway through the bit period, as shown by Huber. The first decoding step
was to demodulate the bi-phase modulated LTCs to "0"s and "1"s, as shown
in FIG. 3(b). The second step is to determine the frame-starts. During
decoding, the program continuously compares the demodulated 0-1
sequence with the 16-bit "sync word" pattern shown as the second row in
FIG. 3(b). Whenever the pattern of any 16-bit segment of the 0-1 sequence
matches that of the "sync word," the bit after this segment would be
determined as the frame-start bit. In the last step, the program converts 64
LTC bits from each frame-start bit into a unique eight-digit LTC integer, as
shown in FIGS. 3(c) and 3(d).
During data acquisition, the program retrieves all 32-bit
integers from the buffer whenever the buffer is half full, writes the data to
the hard disk, decodes the LTCs and forms the time-code-bridge-file all in
2o realtime. Even with a 266-MHz Pentium II microprocessor-based personal
computer, the program cannot run fast enough for realtime decoding all
LTCs (29.97 LTCs/second). The program decodes only one set of frame-
start and LTC integer for every 15 video frames, and uniformly interpolates
the remaining frame-start/LTC-integer values between current and last
correctly decoded frame-starts. Since the video tape speed of a digital VTR
is highly stable, the timing of the interpolated video frame-starts can be
very
accurate.
LTC decoding errors can be caused by a VTR coding error,
noise, electromagnetic interference and overflow of the computer buffer.
3o The program has an LTC decoding error correction subroutine to detect
-10-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
decoding errors. The subroutine scrutinizes every decoded LTC to
determine whether the decoded frame-start is within a predicted tolerance
range (1/80 of a video frame, set by this program) and whether the decoded
LTC integer matches the predicted number. If it is not, the program
automatically discards this erroneous LTC.
After each LTC decoding, the program appends the correctly
decoded frame-start and LTC integer to a compact-time-code-bridge-file. In
turn, the program uniformly interpolates the frame-starts and LTC integers
between the current and the last correctly decoded frame-starts, then
io appends the decoded and interpolated values to a two-column interpolated-
time-code-bridge-file that contains the frame-starts and LTC integers of all
recorded video frames (number of rows = 29.97 frames/second X seconds of
recording), as shown in FIG. 4(b). Like the interpolated bridge file, the
compact bridge file is also a two-column array except that it contains only
decoded values. Its size is 1/15 of that of the interpolated bridge file when
no decoding error occurs, and will be smaller if decoding errors occur and
the consequent LTC decodings are discarded. When the program uses the
bridge file for data segment or video frame search, it first searches the
whole
compact bridge file to determine an estimated file index number range, then
2o searches only the portion of the interpolated bridge file within the
estimated
range to determine the exact index number of the acquired-data-file. This
search procedure would reduce the data-searching time in long-duration
data recording when the interpolated bridge file becomes very large.
The flow chart for data search is shown in FIG. 2(b). When an
interesting video frame is determined and frozen on the TV monitor during
a video replay, the program turns the interface board to acquisition mode,
acquires and decodes the frozen LTC. Then, the program searches the
compact/ interpolated time-code files for the corresponding frame-start
index number of the acquired-data-file. Obtaining the expected
3o corresponding index number, the program turns the interface board to the
-11-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
data-replay mode, retrieves the expected data segment from the acquired-
data-file, displays it on the computer monitor and outputs it to the data
analysis computer for further data analysis using the data analysis software.
As shown in the video-search flow chart in FIG. 2(c), the
interface board is switched to data-replay mode during video frame search.
First, the program uses the compact/interpolated-time-code-bridge-files to
retrieve a desired length of data from the acquired-data-file, replays the
data
frame-by-frame on the system and analysis computer monitors, and tracks
the corresponding LTCs. When an interesting frame of replayed data is
1o determined and frozen on the computer monitor, the program retrieves the
corresponding LTC and controls the VTR through an RS232 serial bus to
search for the expected video frame. First, the program reads the LTC of the
current video-tape position through the RS232 interface and determines the
difference between the video-tape LTC and the frozen-data LTC. Second,
s5 the program calculates the "Forward" or "Rewind" running time from the
current video-tape position to the expected frame. The program then
controls the VTR to forward or rewind the video tape to that frame. When
the VTR stops, the program finally examines the LTC from the VTR to
determine whether the current LTC matches the expected frame number. If
2o it is not, the program will repeat the search procedure.
In a synchronization accuracy test, an accuracy testing
program compared the frame numbers, frame-starts and frame lengths of
the acquired biomedical data with those of the recorded video time codes
frame-by-frame, and calculated the mean and standard deviation of the
25 whole acquisition synchronization errors. As shown in FIG. 6, the video
vertical-sync pulses from a video camera (Panasonic AJ-D200P) and the
corresponding LTCs were acquired by the system computer through the
data channels as simulated dual-channel biomedical data. After data
acquisition, the acquired LTC frame numbers of the simulated data would
3o be compared with the corresponding frame numbers of the acquired LTC
-12-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
from the VTR to determine frame-number synchronization error; and the
acquired vertical-sync pulses of the simulated data would be compared with
the corresponding frame-starts of the acquired LTC from the VTR to
determine synchronization errors in frame-start and frame length. This
synchronization test was reliable because the two signal references, video
vertical-sync pulses and the corresponding LTC frame-starts, are
consistently synchronized.
With reference to FIGS. 6(a) and 6(b), the instrumentation
setup is illustrated for synchronization accuracy testing, in which the video
camera 12 provides a video output signal to a time code generator 122, from
which vertical sync pulse extraction is provided at block 124. Additionally,
a time code amplifier 126 and a video tape recorder 128 generate output
signals to the interface board 26, which is configured in its data acquisition
mode at block 130. A digital input-output board provided in an input mode
s5 132 facilitates the synchronization as accuracy test data to the system
computer 32, configured as a control computer. FIG. 6(b) further illustrates
the vertical sync pulse extractor 124, illustrating the use of a monostable
mufti-vibrator 134 for pulse detection and extraction of the vertical sync
pulse signals from the video output of the time coder generator 122.
2o In the test setup (FIG. 6(a)), the LTC was generated by an LTC
generator (Horita TG-50) and synchronized with the video signals from a
video camera (Panasonic AJ-D200P). The odd-field vertical-sync pulses were
extracted from the video signals (FIG. 6(b)), and input to one of the 31
data/sync channels (Bit 15) of the digital I/O board via the interface board
25 as the simulated data for frame-start and frame-length comparisons. The
LTC stream was input both to the data-channel Bit 16 via the time-code
amplifier and interface board, and to the "LTC In" of the I/O board via the
VTR (Panasonic AJ-D750P) and interface board. The signal/ data latencies
through the VTR and interface board would be counted in the
3o synchronization test.
-13-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
After data acquisition, the accuracy testing program searched
the bridge file for the acquired-data-file index at each frame-start, in order
to
retrieve the corresponding simulated data from the acquired-data-file.
When the program obtained the simulated data, it first decoded the frame
number of the acquired LTC from Bit 16, and determined the difference
between the frame number of the LTC from Bit 16 and the frame number in
the bridge file. Second, the program compared the timing difference
between the rising edge of the odd-field vertical-sync pulse from Bit 15 and
the corresponding frame-start in the bridge file. Finally, the program
io compared the duration difference between odd-field vertical-sync pulse
period from Bit 15 and the corresponding frame-start period in the bridge
file. The mean, standard deviation, and maximum and minimum values of
these differences were then calculated.
The use of the time-code-bridge-files enables this video-
s5 synchronized data acquisition system to perform data acquisition and
subsequent video synchronization. The use of LTC decoding interpolation
enables the system to perform the video synchronization with acquired data
in realtime. Currently, the maximum scanning frequency for realtime video-
synchronized 32-bit data acquisition (31-bit data/ sync signals, and 1 bit LTC
2o stream) is 72 kHz without interruption. The maximum summed bit rate is
2.16 Mbits/ second. This 32-bit data acquisition system can be flexibly
configured. As the system is configured as a 30-channel-data plus one sync-
signal channel with 12 bits of resolution, the maximum scanning frequency
for each channel would be 6 kHz. At 72 kHz of the maximum scanning
25 frequency, the maximum recording duration without data overflow is 30
minutes, which records 506.70 Mbytes of information onto the hard disk:
506.25 Mbytes for the acquired-data-file, 421.5 kbytes for the interpolated-
time-code-bridge-file and 28.1 kbytes for the compact-time-code-bridge-file.
The program effectively decoded the LTC stream in realtime with typically
3o two decoding errors in 30 minutes of acquisition at 72 kHz. The decoding
-14-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
errors would be even fewer if a lower-scanning frequency were used in data
acquisition. The program detects and discards all erroneously decoded
LTCs to ensure that the two time-code-bridge-files are always error free.
Table 1 shows a system comparison of this new method (bridge file based)
with other published data.
In the timing accuracy test,1800 seconds of video signals and
corresponding simulated-biomedical signals were acquired at a scanning
frequency of 72 kHz using the testing setup in FIGS. 6(a) and 6(b). The
accuracy testing program examined the acquired-data-file and the
interpolated-time-code-bridge-file, frame by frame. The frame numbers of
the acquired video images and the corresponding simulated-biomedical data
were examined to be totally synchronized without any detected frame-
number errors. In the frame-start timing examination, the average frame-
start of the recorded video images was slightly ahead of those of the
s5 simulated-biomedical data. In the frame-length examination, the average
frame length of the recorded video images was slightly shorter than that of
the simulated biomedical data. The mean total synchronization error
( ~ frame-start error ~ + ~ frame-length error ~ ) was 0.216 mS (0.647% of a
frame) and the maximum error was 0.653 mS (1.957% of a frame). The
2o maximum latency of the interface board (10.0 nS on rising edges and 18.0 nS
on falling edges for biomedical data, and 1.24 ~,S on rising edges and 0.36
~.S
on falling edges for LTCs), was counted in the above synchronization errors.
166.80 mS of the VTR latency and 79.6 ~S of the inherent phase difference
between frame-starts and the odd-filed vertical-sync pulses were already
25 compensated for in the accuracy testing program. The synchronization
errors also counted in the timing errors caused by the VTR latency
fluctuation, and by the vertical-sync pulse/frame-start phase difference
fluctuation. The statistical test data of the new system along with other
available synchronization-error test data are listed in Table 2 for
comparison.
-15-
CA 02387441 2002-04-11
WO 01/27855 PCT/LTS00/28170
Table 1
Comparison of Video-Synchronized Data Acquisition Systems
System Bridge File FSK Video Hi. Speed Camera
Frame Rate 29.97 29.97 500
(Frame/ sec)
ype of Input Parallel Serial Un-specified
Signal
Summed Data Rate2160 15.36 Un-specified
(kbit/ sec)
to umber of Channels30 32 2
Scanning Freq. 6,000 60 500
/ Channel (Hz)
Resolution (bit)12 8 Un-specified
Max. Recording 30 >_ 120 0.33
s5 Duration (min)
Table 2
Comparison of the System Synchronization Tests
System Bridge FSK Video
File
Replications53835 11848
(frames)
2o Sync. ErrorsFrame Frame- Frame Total Total
(mS) Number start Length
Mean 0 -0.216 +0.003 0.219 16.6
Std. Dev. 0 0.124 0.062 0.186 0.1
Maximum 0 +0.653 +0.399 1.052 16.6
25 Minimum 0 0.000 +0.003 0.003 15.1
This time-code-bridge-file method could also be applied to
synchronize the biomedical data acquired on a computer hard disk and
corresponding video frames acquired on the same or separate hard disk.
The bridge file would be a three-column array, with a column of recorded
-16-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
video frame numbers, and two columns of the index numbers of the
acquired video-signal file and data file, at the beginning of the
corresponding video frames.
The weight of the interface board is less than 1 kg. Along with
a laptop computer, a PCMCIA data acquisition card, and a compact VCR
with LTC output, the whole system would be portable for field data
collection.
The system accurately synchronizes the acquired biomedical
data recorded on the computer hard disk with the corresponding human
1o activity images recorded on a video tape in realtime. With 2.16 Mbit/sec
of summed data rate and less than 1.1 mS of the maximum data-
synchronization error, this data acquisition system is adequate for
frequency-domain muscle-fatigue research, and would be a useful tool for
human hazard exposure assessment, rehabilitation monitoring, and athlete
s5 monitoring in sport physiological programs. While present preferred
embodiments of the mufti-channel, video synchronized EMG data
acquisition system has been illustrated, it should be appreciated by those
skilled in the art that modifications of the foregoing preferred embodiments
may be made in various aspects. The present invention is set forth with
2o particularity in the appended claims. It is deemed that the spirit and
scope
of the invention encompasses such modifications and alterations to the
preferred embodiments as would be apparent to one of ordinary skill in the
art and familiar with the data acquisition teachings of the present
application.
25 Synchronization of data acquired by a computer with the video images
acquired by the same computer
Instrumental Setup
As shown in Fig. 7, this computerized video-synchronized data
acquisition system consists of a video camera to output NTSC color video
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
signals, a time-code generator with SMPTE longitudinal time code (LTC)
output synchronized with the video signals from the video camera, a TV
monitor, a mufti-channel biomedical-signal-measuring instrument with
parallel digital output, a signal-interface board to condition the incoming
and outgoing data and control the direction of data flow, a system computer
connected with a 32-bit digital I/O board and a realtime video capture
board, and an analysis computer if the system needs to run the instrument's
own analysis software.
Formation of Acquired-Data-File and Acquired-Video-File
to The system block diagrams are shown in Fig. 8. During video-
synchronized data acquisition (Fig. 8 (a)), the video signals from the video
camera are sent through the time-code generator to synchronize the LTC
stream generated by the time-code generator. Then the video signals are
sent to the TV monitor for display and to the video capture board for
~5 realtime video-frame capture by the system computer. The LTC from the
time-code generator is sent to the interface board (the interface board is in
Data Acquisition Mode), and to the video-capture board. Simultaneously,
the biomedical instrument measures human biomedical signals, and outputs
up to 31 bits of data/ sync signals to the interface board.
2o To form an acquired-data-file, the interface board buffers the
total of 32-bit data/sync/LTC signals, and outputs them to the 32-bit digital
I/O board. (The interface board also directs the 31-bit data/sync signals to
the analysis computer for realtime data display.) The digital I/O board
scans the 32-bit signal stream at a scanning frequency of up to 72 kHz,
25 converts every 32-bit signal sample into a 32-bit integer, and writes these
data/sync/LTC integers into a data buffer. Every time the buffer is half full,
the system computer program retrieves all the integers from the buffer, and
appends them to an acquired-data-file on the hard disk (Fig. 9 (a)).
-ls-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
To form an acquired-video-file, simultaneously, the system
computer controls the video capture board to capture video stream in
realtime along with the LTC stream, converts the video/LTC signals to
mufti-bit integers, and writes these video/LTC integers to a video buffer.
When the buffer is half full, the program retrieves all video/LTC integers
from the buffer, and appends them to an acquired-video-file on the same
hard disk (Fig. 9 (b)).
Decoding of Frame-Starts and Frame-Numbers
In realtime or offline, depending on the computer signal-
processing speed, the program retrieves the LTC bit stream from the
data/LTC integers, and sends the LTC stream to the decoding subroutine for
LTC decoding. During decoding, the program determines the index number
of the acquired-data-file at each video frame-start, and translates the
corresponding 80-bit LTC into a unique 8-digit frame number integer as
shown in Fig. 9.
In the meantime, as with the data/LTC decoding, the program
also retrieves the LTC bit stream from the video/LTC integers, and sends
the LTC stream to the decoding subroutine for LTC decoding. During
decoding, the program determines the index number of the acquired-video-
2o file at each video frame-start, and translates the corresponding 80-bit LTC
into a unique 8-digit frame number integer (Fig. 9).
Formation of Time-Code-Bridge-File
After decoding, the program forms a time-code-bridge-file
with a three-column array. As shown in Fig. 9 (c), the LTC column of the
array contains the 8-digit frame-number integers of all recorded video
frames. The data-file-index column contains the corresponding index
numbers of the acquired-data-file at frame-starts, and the video-file-index
column contains the corresponding index numbers of the acquired-video-file
-19-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
at frame-starts. In this way, each row of the array represents a unique video
frame, and its corresponding acquired-data-file index number and acquired-
video-file index number at the frame-start. With this time-code-bridge-file,
the computer is able to search for any data segment correspondent to a given
video frame, or search for any video frame correspondent to a given data
segment.
Similar Hardware and Software Implementations to those in Manuscript
The description of the interface board, LTC decoding
procedure, interpolated video frame decoding, LTC decoding error
to correction, compact and interpolated time-code-bridge-files are the same as
those under the respective subtitles in Hardware and Software
Implementation section in the manuscript. The interpolation is not very
necessarily needed if the decoding procedure is conducted offline.
Search for Data Segment
r5 The flow chart for data search is shown in Fig. 8 (b). Given the
desired begin and end frame numbers of video to be displayed, the program
uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired video
frames
from the acquired-video-file (Fig. 9 (b)), display the video images on the
system computer monitor, frame by frame, and track the corresponding
2o LTCs. When an interesting video frame is determined and frozen on the
monitor during a video replay, the program tracks the LTC of the frozen
frame. Then, the program searches the time-code-bridge-file for the
corresponding frame-start index number of the acquired-data-file (Fig. 9 (a)).
Obtaining the expected corresponding index number, the program retrieves
25 the corresponding data frame, or multiple data frames related to the
corresponding data frame, from the acquired-data-file, and turns the
interface board to Data Replay Mode. Finally, the program displays the data
-20-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
on the system computer monitor, and/ or outputs it to the analysis computer
for further data analysis using the data analysis software.
Search for Video Frame
As shown in the video-search flow chart in Fig. 8 (c), the
interface board is switched to Data Replay Mode during video frame search.
Given desired begin and end frame numbers of data to be displayed, the
program uses the time-code-bridge-file (Fig. 9 (c)) to retrieve the desired
data frames from the acquired-data-file (Fig. 9 (a)]), replay the data segment-
by-segment (one segment contains one or multiple frames, determined by
io the user) on the system computer and/or analysis computer monitors, and
track the corresponding LTCs. When one or multiple interesting frames of
replayed data are determined and frozen on the computer monitor, the
program tracks the corresponding frozen LTCs, searches the time-code-
bridge-file for the corresponding frame-start index numbers of the acquired-
video-file (Fig. 9 (b)). Obtaining the expected corresponding index numbers,
the program retrieves the expected video frames from the acquired-video-
file, displays the corresponding video images and LTCs, frame by frame, on
the computer monitor.
Synchronization of data acquired by a computer with the video images
2o acquired by a separate computer
Instrumental Setup
As shown in Fig.10, this computerized video-synchronized
data acquisition system consists of a video camera to output NTSC color
video signals, a time-code generator with SMPTE longitudinal time code
(LTC) output synchronized with the video signals from the video camera, a
TV monitor, a multi-channel biomedical-signal-measuring instrument with
parallel digital output, a signal-interface board to condition the incoming
and outgoing data and control the direction of data flow, a system computer
-21-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
connected with a 32-bit digital I/O board, a video computer connected with
a realtime video capture board, and an analysis computer if the system
needs to run the instrument's own analysis software.
Formation of Acquired-Data-File and Acquired-Video-File
The system block diagrams are shown in Fig.11. During
video-synchronized data acquisition (Fig.11 (a)), the video signals from the
video camera are sent through the time-code generator to synchronize the
LTC stream generated by the time-code generator. Then the video signals
are sent to the TV monitor for display and to the video capture board for
to realtime video-frame capture by the video computer. The LTC from the
time-code generator is sent to the interface board (the interface board is in
Data Acquisition Mode), and to the video-capture board connected with the
video computer. Simultaneously, the biomedical instrument measures
human biomedical signals, and outputs up to 31 bits of data/ sync signals to
s5 the interface board.
To form an acquired-data-file, the interface board buffers the
total of 32-bit data/sync/LTC signals, and outputs them to the 32-bit digital
I/ O board connected with the system computer. (The interface board also
directs the 31-bit data/sync signals to the analysis computer for realtime
2o data display.) The digital I/O board scans the 32-bit signal stream at a
scanning frequency of up to 72 kHz, converts every 32-bit signal sample into
a 32-bit integer, and writes these data/sync/LTC integers into a data buffer.
When the buffer is half full, the system computer program retrieves all the
integers from the buffer, and appends them to an acquired-data-file on the
25 system computer hard disk (Fig. 9 (a)).
To form an acquired-video-file, simultaneously, the video
computer controls the video capture board to capture video stream in
realtime along with the LTC stream, converts the video/ LTC signals to
multi-bit integers, and writes these video/LTC integers to a video buffer.
-22-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
When the buffer is half full, the program retrieves all video/LTC integers
from the buffer, and appends them to an acquired-video-file on the video
computer hard disk (Fig. 9 (b)).
Decoding of Frame-Starts and Frame-Numbers
In realtime or offline, depending on the computer signal-
processing speed, the system computer program retrieves the LTC bit
stream from the data/LTC integers, and sends the LTC stream to the
decoding subroutine for L TC decoding. During decoding, the system
computer program determines the index number of the acquired-data-file at
so each video frame-start, and translates the corresponding 80-bit LTC into a
unique 8-digit frame number integer as shown in Fig. 9.
In the meantime, as with the data/ LTC decoding, the video
computer program retrieves the LTC bit stream from the video/LTC
integers, and sends the LTC stream to the decoding subroutine for LTC
s5 decoding. During decoding, the video computer program determines the
index number of the acquired-video-file at each video frame-start, and
translates the corresponding 80-bit LTC into a unique 8-digit frame number
integer (Fig. 9).
Formation of Time-Code-Bridge-File
2o The system may use two different time-code-bridge-file
configurations. After decoding, the system forms one or two time-code-
bridge-files depending on the bridge file configuration.
Configuration 1. The system forms a three-column time-code-
bridge-file which may be on either the system computer hard disk or the
25 video computer hard disk. As shown in Fig. 9 (c), the LTC column of the
array contains the 8-digit frame-number integers of all recorded video
frames. The data-file-index column contains the corresponding index
numbers of the acquired-data-file at frame-starts, and the video-file-index
-23-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
column contains the corresponding index numbers of the acquired-video-file
at frame-starts. In this way, each row of the array represents a unique video
frame, and its corresponding acquired-data-file index number and acquired-
video-file index number at the frame-start. With this time-code-bridge-file,
the computer is able to search for any data segment correspondent to any
given video frame, or search for any video frame correspondent to any given
data segment.
Configuration 2. The system forms a two-column data-time-
code-bridge-file on the system computer hard disk (Fig.12 (a)), and a two-
lo column video-time-code-bridge-file on the video computer hard disk (Fig.12
(b)). As in Configuration 1, each row of the data-time-code-bridge-file and
the corresponding row of the video-time-code-bridge-file represent a unique
video frame, and its corresponding acquired-data-file index number and
acquired-video-file index number at the frame-start. The two time-code-
i5 bridge-files communicate between the two computers to synchronize the
data with video.
Similar Hardware and Software Implementations to those in Manuscript
The description of the interface board, LTC decoding
procedure, interpolated video frame decoding, LTC decoding error
2o correction, compact and interpolated time-code-bridge-files are the same as
those under the respective subtitles in Hardware and Software
Implementation section in the manuscript. The interpolation is not
necessarily needed if the decoding procedure is conducted offline.
Search for Data Segment
25 Configuration 1. The flow chart for data search is shown in
Fig.11 (b). Given the desired begin and end frame numbers of video to be
displayed, the system uses the three-column time-code-bridge-file (Fig. 9 (c))
to retrieve the desired video frames from the acquired-video-file (Fig. 9 (b))
-24-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
in the video computer, display the video images on the video computer
monitor, frame by frame, and track the corresponding LTCs. When an
interesting video frame is determined and frozen on the monitor during a
video replay, the system tracks the LTC of the frozen frame. Then, the
system searches the time-code-bridge-file for the corresponding frame-start
index number of the acquired-data-file (Fig. 9 (a)). Obtaining the expected
corresponding index number, the system retrieves the corresponding data
frame, or multiple data frames related to the corresponding data frame, from
the acquired-data-file in the system computer, and turns the interface board
io to Data Replay Mode. Finally, the system computer program displays the
data on the system computer monitor, and/ or outputs it to the analysis
computer for further data analysis using the data analysis software.
Configuration 2. The flow chart for data search is also shown
in Fig.11 (b). Given the desired begin and end frame numbers of video to be
displayed, the system uses the video-time-code-bridge-file (Fig.12 (b)) to
retrieve the desired video frames from the acquired-video-file (Fig. 9 (b)) in
the video computer, display the video images on the video computer
monitor, frame by frame, and track the corresponding LTCs. When an
interesting video frame is determined and frozen on the video computer
2o monitor during a video replay, the system tracks the LTC of the frozen
frame. Then, the system searches the data-time-code-bridge-file (Fig.12 (a))
for the corresponding frame-start index number of the acquired-data-file
(Fig. 9 (a)) in the system computer. Obtaining the expected corresponding
index number, the system retrieves the corresponding data frame, or
multiple data frames related to the corresponding data frame, from the
acquired-data-file in the system computer, and turns the interface board to
Data Replay Mode. Finally, the system computer program displays the data
on the system computer monitor, and/ or outputs it to the analysis computer
for further data analysis using the data analysis software.
-25-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
Search for Video Frame
Configuration 1. As shown in the video-search flow chart in
Fig. 11 (c), the interface board is switched to Data Replay Mode during video
frame search. Given the desired begin and end frame numbers of data to be
displayed, the system uses the time-code-bridge-file (Fig. 9 (c)) to retrieve
the desired data frames from the acquired-data-file (Fig. 9 (a)) in the system
computer, replays the data segment-by-segment (one segment contains one
or multiple frames, determined by the user) on the system computer and/ or
analysis computer monitors, and track the corresponding LTCs. When one
so or multiple interesting frames of replayed data are determined and frozen
on the system computer monitor, the system tracks the corresponding
frozen LTCs, searches the time-code-bridge-file for the corresponding frame-
start index numbers of the acquired-video-file (Fig. 9 (b)) in the video
computer. Obtaining the expected corresponding index numbers, the
s5 system retrieves the expected video frames from the acquired-video-file in
the video computer, displays the corresponding video images and LTCs,
frame by frame, on the video computer monitor.
Configuration 2. As shown in the video-search flow chart in
Fig.11 (c), the interface board is switched to Data Replay Mode during video
2o frame search. Given the desired begin and end frame numbers of data to be
displayed, the system uses the data-time-code-bridge-file (Fig.12 (a)) to
retrieve the desired data frames from the acquired-data-file (Fig. 9 (a)) in
the
system computer, replay the data segment-by-segment (one segment
contains one or multiple frames, determined by the user) on the system
25 computer and/ or analysis computer monitors, and track the corresponding
LTCs. When one or multiple interesting frames of replayed data are
determined and frozen on the system computer monitor, the system tracks
the corresponding frozen LTCs, searches the video-time-code-bridge-file
(Fig.12 (b)) for the corresponding frame-start index numbers of the
3o acquired-video-file (Fig. 9 (b)) in the video computer. Obtaining the
-26-
CA 02387441 2002-04-11
WO 01/27855 PCT/US00/28170
expected corresponding index numbers, the system retrieves the expected
video frames from the acquired-video-file in the video computer and
displays the corresponding video images and LTCs, frame by frame, on the
video computer monitor.
-27-
