Language selection

Search

Patent 2147164 Summary

Third-party information liability

Some of the information on this Web page has been provided by external sources. The Government of Canada is not responsible for the accuracy, reliability or currency of the information supplied by external sources. Users wishing to rely upon this information should consult directly with the source of the information. Content provided by external sources is not subject to official languages, privacy and accessibility requirements.

Claims and Abstract availability

Any discrepancies in the text and image of the Claims and Abstract are due to differing posting times. Text of the Claims and Abstract are posted:

  • At the time the application is open to public inspection;
  • At the time of issue of the patent (grant).
(12) Patent: (11) CA 2147164
(54) English Title: METHOD AND APPARATUS INCLUDING SYSTEM ARCHITECTURE FOR MULTIMEDIA COMMUNICATIONS
(54) French Title: METHODE ET APPAREIL POUR LES COMMUNICATIONS MULTIMEDIA
Status: Expired
Bibliographic Data
(51) International Patent Classification (IPC):
  • H04N 19/61 (2014.01)
  • G06F 9/46 (2006.01)
  • G06F 9/48 (2006.01)
  • G06T 1/20 (2006.01)
  • G06T 1/60 (2006.01)
  • G06T 9/00 (2006.01)
  • G09G 5/39 (2006.01)
  • H04N 1/32 (2006.01)
  • H04L 69/32 (2022.01)
  • H04N 1/00 (2006.01)
  • H04L 29/06 (2006.01)
  • H04L 29/08 (2006.01)
(72) Inventors :
  • SHAW, VENSON M. (United States of America)
  • SHAW, STEVEN M. (United States of America)
(73) Owners :
  • SHAW, VENSON M. (United States of America)
  • SHAW, STEVEN M. (United States of America)
(71) Applicants :
  • SHAW, VENSON M. (United States of America)
  • SHAW, STEVEN M. (United States of America)
(74) Agent: KIRBY EADES GALE BAKER
(74) Associate agent:
(45) Issued: 2005-12-20
(86) PCT Filing Date: 1992-10-16
(87) Open to Public Inspection: 1994-04-28
Examination requested: 1997-10-16
Availability of licence: N/A
(25) Language of filing: English

Patent Cooperation Treaty (PCT): Yes
(86) PCT Filing Number: PCT/US1992/008018
(87) International Publication Number: WO1994/009595
(85) National Entry: 1995-04-13

(30) Application Priority Data: None

Abstracts

English Abstract





A controller apparatus and method for executing a plurality of control
functions for communication of multimedia articles
including voice, audio, text, still image, motion video and animated graphics
which incorporates a frame memory system which is
compatible with multiple standard or customized coding algorithmic signals
such as H.261, MPEG, JPEG, EDTV or HDTV or in
multiple incompatible video coding equipment employing different video coding
algorithms can now freely communicate with
each other, the controller apparatus further including a band width management
function for automatically scaling the multime-
dia article signal to conform to available bandwidth.


Claims

Note: Claims are shown in the official language in which they were submitted.





-57-

Claims:

1. A controller apparatus for executing a plurality
of control functions for communication of multimedia
articles including voice, high quality audio, text, still
image, motion video and animated graphics between
multimedia equipment and communications networks,
selectively providing receiving functions, retrieving
functions, and transmitting functions, wherein said
controller is compatible with multiple band widths and
algorithmic coded signals, said controller providing a
scalable architecture for communication of multimedia data,
formatting from internal format to external format in
compliance with any selective one of a plurality of
international standardized and customized coded algorithms,
and band width determination of available display band
widths according to frame rate and available transmission
band widths, said controller comprising:
means for receipt of an external network signal, said
means comprising a network domain decoder;
means for receipt of a local origination signal; and
means for image processing of said origination signal for
storage coding simulation, and transmission;
a first control means and format means for selectively
formatting the format of said external network signal and
local origination signal to a compatible first internal
format in compliance with internal processing and memory
capabilities;
means for decoding said external network signal
according to said first internal format, said means
selectively comprising a transform domain decoder alone and
in combination with a pixel domain decoder;




-58-

a second control means and format means for
selectively formatting said external network signal and
said origination signal to a second internal format in~
compliance with a selective plurality of coding algorithms
for processing, storage and transmission, said means
further comprising an encoding means which comprises a
pixel domain encoder alone and in combination with a
transform domain encoder for encoding said external network
signal and said local origination signal;
means for reconfiguring the memory means for
selectively associating with said apparatus encoders and
decoders for directing said first and second format means;
band width management means, in communication with
said second format means and said control means, for
directing available run time transmission band widths and
appropriate compression ratios for selectively formatting
said second internal format to external transmission format
in compliance with available transmission band width range,
said band width management means continuously moderating
and correcting said selective formatting of said second
internal format to said second external transmission format
based on transmission band width availability;
means for transmission of said internally formatted
external network signal and local origination signal
according to said external transmission format, said means
comprising a network domain encoder.

2. A controller apparatus in accordance with claim 1
wherein said network domain decoder for receipt of said
external network signal selectively comprises a front end
transceiver; protocol processor; network communications
processor; variable length decoding processor; run length
decoding processor; and filtering processor for sampling




-59-

said external network signal, said network domain decoder
in communication with said first control means and said
format means to convert horizontal and vertical resolution
of said external network signal to an internally-defined
file format size for reducing said internal real time video
processing and frame memory updating requirements.

3. A controller apparatus in accordance with claim 1
wherein said second controlling means and said second
formatting means, in communication with said network domain
encoder and decoder selectively codes and decodes said
incoming signal based upon the availability of customized
and international standardized algorithms and selectively
performs motion compensation processes and motion
simulation processes to improve quality of said motion
video article, said processes include pattern match, edge
detection, image enhancement, color mapping and pixel
interpolation of said external network and local
origination signal.

4. A controller apparatus in accordance with claim 1
wherein external network signal is selectively transferred
to said transform domain decoder and said pixel domain
decoder for decoding; said transform domain decoder
selectively providing denormalization, dequantization, scan
conversion, matrix operation, inverse Haar transform,
inverse Hadamard transform and inverse discrete cosine
transforms on said external network signal and said pixel
domain decoder selectively performing color mapping, vector
dequantization, pixel extrapolation, color space
conversion, image processing, pixel domain decoding, and
pixel domain preprocessing on said external network signal
for formatting of said external network signal and




-60-

conversion to said first internal format for either storage
processing, display or transmission.

5. A controller apparatus in accordance with claim 1
wherein said means for image preprocessing of said local
origination signal comprises an integrated circuit
subsystem further comprising:
means for performing integrated parallel sensing and
storage wherein a plurality of input frames for said local
origination signal can be sensitized, digitized and stored;
means for performing detection of instantaneous scene
changes wherein only the corresponding frame will be
registered as intraframe, wherein all other frames will be
registered as interframes, said detection means further
comprising means for moderating a threshold level for
proper determination of said scene changes; and
means for performing in analog and digital forms
parallel image preprocessing for said interframes whereby
each interframe image can be retrieved and compared to
develop a plurality of corresponding frame differencing
articles whereby only said frame differencing articles are
required for further processing and encoding including
motion estimation, discrete cosine transform, quantization,
and huffman-like variable length coding.

6. A controller apparatus in accordance with claim 5
wherein said means for image preprocessing for each said
interframe further comprise a means for motion estimation,
motion simulation and motion compensation operations on
motion video articles comprising:
means to perform edge detection operations for each of
said frame differencing articles wherein a selective
plurality of detected edge articles are derived to identify




-61-

said motion video articles during motion estimation and
compensation operations;
means to perform feature extraction operations for
each of said frame differencing articles wherein said
extracted features for said frame differencing articles can
identify from said motion video articles during said motion
estimation process; and
means to perform forward, backward and random
reference of selective interframes employing said selective
features and said selective detected edge articles for said
frame differencing articles.

7. A controller apparatus as set forth in claim 6
wherein said motion video articles include a plurality of
moving foreground pixels overlaying can be referenced using
transmission of a single or plurality of preceding
intraframes wherein said moving foreground pixels for
subsequent plurality of interframes can be referenced using
frame differencing techniques wherein each group of said
moving foreground pixels are characterized employing said
frame differencing of a plurality of said detected edge
articles and a plurality of said extracted features.

8. A controller apparatus in accordance with claim 7
wherein a motion video article can be applied with one or
more pattern searching and recognition algorithms to derive
alternative reference motion vectors wherein said motion
vectors can forward, backward and random reference said
motion video article during said interframe sequencing said
motion vector can be represented by the interframe
displacement of symbolic representation further comprising
a plurality of detected edges, frame differences and




-62-


extracted features for each corresponding motion video
article, said symbolic representation and alternate motion
vector can be applied for transmission simulation, coding,
storage and processing of said motion video articles.
9. A controller apparatus in accordance with claim 5
wherein said means for interframe image preprocessing
further comprises parallel processor means for performing
pixel level operations for an n×n pixel block sub-image,
said means comprising an m×m array of said parallel
processor elements wherein n and m are integers and said
parallel processor means can retrieve, decode and execute
instructions for pixel input data from internal memory; and
interconnection means for selective communication between
each of said parallel processors whereby said pixel data
inputs and incoming control messages can be received from
said selected parallel processors and said pixel data
outputs and outgoing messages can be broadcast to said
selected parallel processing neighbors.
10. A controller apparatus in accordance with claim 5
wherein said means for interframe image preprocessing
further comprises a processor array configured as an n×n
cellular logic array processor where n is typically of
value in a range of between 8 and 32 wherein said cellular
logic array processor can be employed to perform a
plurality of block processing operations for said n×n pixel
block of sub-image, said processor array further
comprising:
a means for frame differencing between interframes
wherein a group of n×n difference blocks are identified,
registered and stored;




-63-


a means for template matching and article recognition
operations for each of said n×n pixel blocks of said sub-
image article extracting from said input image and
comparing with selective n×n reference data stored
previously in memory;
a means for determination of threshold level for
selective matching with sections of said sub-image article
to identify a motion vector article;
a means for motion estimation operation which can be
applied to each of said n×n sub-image blocks for comparison
between a previously-displayed frame article and a current
frame article, said n×n sub-image performing matching
functions at said frame differencing locations for
generating horizontal and vertical distance for
representing said motion vector;
a means for edge detection for each of said n×n blocks
of said sub-image and means for edge enhancement for each
decompressed image blocks prior to display, said edge
detection for deriving said motion vector;
a means for feature extraction and comparison of said
previously-displayed frame with said current frame;
Hadamard transform means for each of said n×n blocks
of said sub-image wherein each of said n×n blocks is
represented by its pixel domain transform coefficient
multiplied by the n×n Hadamard coefficient matrix thereby
reducing transmission band width through pixel domain band
width compression; and
frequency domain transform means including discrete
cosine transform means wherein each of an 8×8 frame
differencing sub-image block can be multiplied by an 8×8
transform coefficient block to derive 8×8 blocks of frame
differences identified by said frequency domain transform
coefficients.




-64-


11. A controller apparatus in accordance with claim 1
wherein said second control means and second format means
for selectively formatting said internal format further
comprises a means for template matching pattern recognition
and motion estimation and compensation, comprising:
means for extracting each of the n×n luminance pixel
blocks from a previous reference interframe;
means for performing template matching operation of
said n×n reference block within a m×m window of surrounding
pixel blocks residing in a current interframe wherein m and
n are integers and m is greater than n, said template
matching generating a motion vector for said pixel block in
the event of a match; and
means for determining and adjusting said m×m window
size in the event of no match;
means for performing pattern recognition on said
adjusted n×n window sub-image.
12. A controller apparatus in accordance with claim 1
wherein said image preprocessing apparatus further
comprises a means for identifying an estimating randomly-
generated signal noise embedded within the input image
article; and means for subtracting said randomly-generated
signal noise from an original input image article by means
of a first sensor and a second sensor, said first sensor
sensitizing image and noise and said second sensor
sensitizing noise for comparison of said sensors and
subtraction of said noise signal.
13. A controller apparatus in accordance with
claim 10 wherein said article means fob template matching
further comprises:




-65-


means for retaining color representation of each
significant object of said frame;
means for storing said color representation;
means for retrieving said color representation and
performing pel domain transform operations; and
means for converting said color representation from
said pel domain to said frequency domain in order to
perform additional frequency domain processing means such
as discrete cosine transform and fast fourier transform.

14. A controller apparatus in accordance with claim 1
wherein said band width management means comprises a means
for generating a network transmission request signal for
the transmission band width requirement for outbound
transmission multimedia articles;
means for receiving an acknowledgement signal from an
extension network controller with respect to transmission
band width requirement;
means for comparing said acknowledgement signal with
said outbound transmission signal to determine sufficiency
of transmission band width;
means for adjusting said outbound transmission signal
to accommodate said band width availability;
means for compressing and decompressing said outbound
transmission signal;
means for transmitting said outbound transmission
signal; and
means for continuous monitoring said outbound
transmission and said band width availability and
selectively adjusting said outbound transmission signal to
conform to said availability of said band width during said
transmission.




-66-

15. A controller apparatus in accordance with claim 1
wherein said means for selectively reconfiguring memory
comprises:
means for definition of a default internal file format
and size based upon coder processing and frame memory
system throughput;
means for receipt of an external algorithmic coded
signal during network inbound stage, said means comprising
a network domain decoder;
means for receipt of a local origination signal, said
means comprising a capture processor;
means for identifying and receiving article types and
file size during said network inbound stage, said means
comprising a host processor and its interface to said
capture processor or said network decoder;
means for providing adequate downsampling ratio,
wherein said receiving article can be conformed and reduced
to a predefined internal file format and size; said means
comprising a reconfiguration unit and a staler circuit;
means for manipulation of said internal file articles,
said means comprising an article-oriented instruction set,
and means for storing, retrieving, decoding, and executing
said instruction set by said host processor and said pixel
processor;
means for readjustment of said internal file format
and size in conformance with band width availability, said
means comprising a band width management unit in
communication with said staler circuit, reconfiguration
unit and said host processor;
means for adjustment of said internal file format and
size during interframe coding modes, wherein a plurality of
motion video foreground articles can be specified to proper




-67-


size, and wherein static background articles can be
pretransmitted during intraframe coding modes, said means
comprising a frame differencing circuit and motion
processor interfaced with said pixel processor; and
means for providing upsampling ratio during
transmission, wherein said transmitting article can be
conformed and expanded to a selectively-desired file format
and size, said means comprising said reconfiguration unit,
and said scaler circuit interfaced with said band width
management unit.

16. A controller apparatus in accordance with claim 1
wherein said first control means and second control means
communicate across the network with external devices
including pagers, remote control means and host computers
for the selective exchange of command, data, status and
control information with said external devices wherein said
external devices perform selective multimedia applications
for said controller, said communication means between said
controlling means and said external devices comprising:
means for providing command layer protocol control
functions in the form of predefined commands selectively
interpreted by said control means;
means for updating and rearranging said predefined
commands;
means for prioritizing said predefined commands;
means for providing transport layer protocol control
functions;
means for data transfer and data tracking in
cooperation with said command layer protocol; and
means for providing physical layer protocol control
functions for implementation of initialization termination
of mechanisms.




-68-


17. An apparatus for the adaptive continuous
management and control of network traffic condition and
band width availability for the communication of multimedia
articles including voice, audio, text, still image, motion
video and animated graphics between multimedia transmitter
and receivers wherein said apparatus is compatible with
multiple analog and digital transmission standards,
selected from the group consisting of analog voice grade
line, PSDN, basic rate ISDN, primary rate ISDN/Tl, LAN, and
FDDI, the apparatus having a band width detection means
determining available display band widths in accordance
with frame rate and run time transmission band width, said
apparatus comprising:
a means for generating a transmission request signal
provided with message retaining transmission band width for
an outbound multimedia article;
a means for receiving a transmit acknowledge signal
from a network controller and destination receiver
indicating said band width availability;
a means for relaying a band width availability signal
to a band width management unit for comparison of band
width availability to transmission signal;
a means for generating an alternate compression ratio
for said transmission signal compatible with said band
width availability;
a means for implementing said alternate compression
ratio through a host or encoding processor for further
compression of said multimedia article to meet said band
width availability;
a means for transmitting said multimedia article on
said available band width; and




-69-


a means for monitoring said multimedia article
transmission in said band width availability during
transmission.
18. An apparatus for the continuous management and
control of transmission band width availability in
accordance with claim 17 wherein said means for generating
said transmission request signal and said means for
generating an alternative compression ratio further
comprise:
a means for sensitizing outbound line condition
changes whereby outbound transmission bit rate is adjusted
to provide sufficient composition of said multimedia
article groups, said means for sensitizing said outbound
line condition comprising a selection of a plurality of
quality levels for digitally-compressed audio articles; a
selection of a plurality of quality levels for digitally-
compressed motion video articles; a selection of a
plurality of quality levels for digitally-compressed still
images; a selection of a plurality of frame updating rates
for digitally-compressed motion video articles; a selection
of plurality of quality levels for digitally-coded animated
bit mapped or vector graphics articles; and a selection of
a plurality of options for composites of said digital,
audio, image, video and animated graphics articles; and
a means for selectively monitoring and adjusting said
rate requirements for said outbound multimedia articles.
19. An apparatus in accordance with claim 17 for the
continuous management and control of said band width
availability, said apparatus further comprising:
a plurality of predefined program sequences for a
plurality of transmission line conditions whereby each of




-70-


said program sequences comprises a specific group of
multimedia articles, predefined for said transmission band
width availability; and
a means to store said predefined program sequence and
reference said predefined program sequence during line
condition changes.

20. A method for the adaptive continuous management
and control of network traffic condition and band width
availability having a band width detection means
determining available display band widths in accordance
with frame rate and run time transmission band width, for
the communication of multimedia articles including voice,
audio, text, still image, motion video and animated
graphics between multimedia transmitter and receivers
whereby said method is compatible with multiple analog and
digital transmission standards selected from the group
consisting of analog voice grade line, PSDN, basic rate
ISDN, primary rate ISDN/T1, LAN, and FDDI, the method
comprising:
(a) generating a transmission request signal provided
with message band width requirement for outbound multimedia
articles;
(b) receiving a transmit acknowledge signal from an
external network controller and destination receiver
indicating said band width availability;
(c) relaying said band width availability to a band
width management unit for comparison with band width of
said outbound multimedia articles;
(d) generating an alternative compression ratio for
said outbound multimedia articles compatible with said
available band width;




-71-


(e) relaying said alternative compression ratio to a
host or encoding processor for further compression of saic
outbound multimedia articles in conformance with said bane
width availability;
(f) transmitting said outbound multimedia articles in
conformance with said transmission band width availability;
(g) monitoring said band width availability and said
transmission of said outbound multimedia articles during
said transmission;
(h) generating alternative compression ratios during
transmission to adapt to changes in said band width
availability:
(i) repeating steps (a) through (h) as required.
21. A scalable, reconfigurable memory apparatus for
executing a plurality of preprocessing and post-processing
functions for the communication and storage of multimedia
articles including voice, audio, text, still image, motion
video, and animated graphics between selective multimedia
transmitters and receivers, said memory apparatus
compatible with multiple standard or customized coding
algorithmic signals including H.261, MPEG, JPEG, EDTV or
HDTV, said apparatus comprising:
means for definition of a default internal file format
and size based upon coder processing and frame memory
system throughput;
means for receipt of an external network algorithmic
coded signal, said means comprising a network domain
decoder;
means for receipt of a locally-originated signal, said
means comprising a capture processor;


-72-


means for identifying and receiving said multimedia
article and file size, said means comprising a host
processor selectively interfaced with said capture
processor and network domain decoder;
means for downsampling wherein said multimedia article
can be conformed and reduced to a predetermined internal
file format and size, said means comprising a
reconfiguration unit and staler circuit
means for manipulation of said internal file format,
said means comprising said host processor and a pixel
processor;
means for adjustment of said internal file format and
size to conform to band width availability, said means
comprising a band width management unit in communication
with said scaler circuit, said reconfiguration unit and
said host processor;
means for adjustment of said internal file format and
size during interframe coding, wherein motion video
foreground articles and still background articles are
differentiated and separately transmitted, said means
comprising a frame differencing circuit, and motion
processor in interface with said pixel processor; and
means for providing upsampling during transmission
wherein said transmission of said multimedia article can be
formatted and sized, said means comprising said
reconfiguration unit, and said staler circuit in interface
with a band width management unit.
22. A scalable, reconfigurable memory apparatus for
executing a plurality of preprocessing and post-processing
functions for video coding in accordance with claim 21,
further comprising:


-73-


an integrated parallel processor and storage array
whereby a plurality of frames of input image can be
sensitized, digitized and stored;
a parallel image preprocessor array for performing
real time image encoding operations, said preprocessor
array including a frame differencing operation whereby
during image encoding operations, said input images can be
sensitized and compared to develop frame differencing
articles whereby said frame differencing articles can be
further processed for motion estimation, discrete cosine
transform, quantization, and huffman variable length coding
whereby said motion estimation and motion compensation
operations comprise an edge detection operation for each of
said frame differencing articles whereby a plurality of
detected edge articles are derived to identify said motion
video objects during said motion estimation and
compensation operations; and
a feature extraction operation whereby extracted
specific features for each of said frame differencing
articles can be identified.
23. A scalable, reconfigurable memory apparatus for
executing a plurality of preprocessing and post-processing
functions in accordance with claim 22 wherein said input
images comprise moving foreground pixels overlaying still
image background pixels whereby said background pixels may
be characterized singly during interframe coding, requiring
single transmission, and said moving foreground pixels are
detected utilizing said interframe differencing technique
in said motion estimation and compensation technique
whereby said parallel processor performs pixel level
operations for an n×n pixel block sub-image, said parallel


-74-


processor comprising an m×m array of parallel processor
elements whereby n and m are integers of equal or unequal
value.
24. A scalable, reconfigurable memory apparatus for
executing a plurality of preprocessing and post-processing
functions in accordance with claim 23 wherein said parallel
processor elements can retrieve, decode and execute
instructions for said pixel input data in its internal
buffer or in neighboring parallel processor elements
buffer, said pixel output data can be stored internally and
in neighboring parallel processors in an interconnection
means programming each of said parallel processors to a
selective group of neighboring parallel processors whereby
pixel data inputs and control messages can be received from
said selected parallel processing neighbors and pixel data
outputs and outgoing messages can be transmitted to said
selected group of parallel processor neighbors.
25. A scalable, reconfigurable memory apparatus for
executing a plurality of preprocessing and post-processing
functions in accordance with claim 24 wherein said parallel
processors can be arrayed and configured as an n×n cellular
logic array processor wherein n is a value of between 8 and
32 and said cellular logic array processor can perform
processing operations for said n×n pixel blocks of sub-
images including said frame differencing operation between
a current frame and a previously-displayed frame whereby a
group of n×n differencing blocks are identified, registered
and stored; template matching operations for each of said
n×n pixel blocks of sub-images, said template matching
operation extracting an input image and comparing with a


-75-

previously-stored input image thereby deriving a motion
vector; pattern matching; motion estimation applied to each
of said n×n sub-image blocks residing within said
previously-displayed frame; edge detection for each of said
n×n blocks; feature extraction; Hadamard transform for each
of said n×n blocks of sub-images whereby each of said n×n
blocks can be represented by a pixel domain transform
coefficient and a Hadamard coefficient matrix; and a
discrete cosine transform operation.
26. A scalable, reconfigurable memory array method
for executing a plurality of preprocessing and post-
processing functions for the communication and storage of
multimedia articles including voice, high quality audio,
text, still image, motion video and animated graphics
between multimedia transmitters and receivers, whereby said
method is compatible with multiple standard customized
coding algorithmic signals including H.261, MPEG, JPEG,
EDTV or HDTV, wherein said method provides a scalable
architecture for communication of multimedia data,
formatting from internal format to external format in
compliance with any selective one of a plurality of
international standardized and customized coded algorithms,
and band width determination of available display band
widths according to frame rate and available transmission
band widths, said method comprises:
defining of a default internal file format and size
based upon coder processing and frame memory system
throughput;
selectively receiving an external network algorithmic
coded signal and local origination signal;
identifying the multimedia article signal and size;


-76-


downsampling the received multimedia article to a
predefined internal file format and size;
manipulating said internal file format to conform to
transmission band width availability;
distinguishing between motion video foreground
articles and still background articles during interframe
coding and intraframe coding modes;
upsampling said internal file format to selectively
desired format and size for transmission in accordance with
transmission band width availability:
distinguishing between motion video foreground
articles and still background articles during interframe
coding and intraframe coding modes;
upsampling said internal file format to selectively
desired format and size for transmission in accordance with
transmission band width availability.
27. A scalable, reconfigurable memory array method in
accordance with claim 26 wherein said method for
manipulating said internal file format to conform to band
width availability and distinguishing between motion video
foreground articles and still background articles during
interframe coding and intraframe coding modes further
comprises:
differentiating between still image background pixels
and moving foreground pixels;
performing frame differentiating techniques on an n×n
pixel block sub-image by means of an m×m array of parallel
processors whereby n and m are integers of equal or unequal
value;
comparing current frame images to previous frame
images and template matching said images;


-77-


deriving a motion vector for each of said frame
differences;
performing pattern matching, motion estimation, edge
detection, feature extraction and Hadamard transform, for
each of said n×n blocks of sub-images;
representing each of said n×n blocks of sub-images by
a pixel domain transform coefficient, Hadamard coefficient
matrix and discrete cosine transform.
28. A method for the application of image
preprocessing techniques for the interframe coding of
multimedia articles including voice, audio, text, still
image, motion video and animated graphics between
multimedia transmitters and receivers, whereby said method
can be integrated with an image capturer circuit and band
width management circuit, said method comprising:
(a) capturing a local origination still image in
accordance with frame update requirements established by a
host processor;
(b) capturing and storing the complete image frame
during the intraframe coding mode;
(c) comparing and differencing the captured image
frame to a previous captured frame during the interframe
coding mode;
(d) storing the frame differences between the captured
image frame and said previous captured frame in blocks and
macro blocks;
(e) registering and coding said frame differencing
blocks or said frame differencing macro blocks into a frame
differencing bit map;
(f) converting color space from analog RGB or NTSC
format into a digital RGB or YUV format;


-78-


(g) retrieving luminance macro blocks according to
said frame differencing bit map;
(h) deriving the appropriate motion vector for each of
said macro blocks through comparison and selection of the
minimum distortion among adjacent macro blocks;
(i) reducing from an external file format to an
internally-defined format in accordance with a downsampling
ratio established by a host processor, a reconfiguration
unit and a sealer;
(j) translating said frame differencing bit map to
conform with said internally-defined format;
(k) performing pixel domain and transform domain
coding operations for said frame differencing macro blocks
for said Y, U and V format;
(l) enlarging from said internal file format to an
external file format according to an appropriate
downsampling ratio as established by said host processor,
reconfiguration unit and sealer;
(m) adding said frame difference to said previous
frame in order to display and update said current frame;
(n) repeating steps (a) through (m) as required;
(o) repeating steps (a) through (n) for live video
sequence as required.

29. A method for the application of frame
differencing techniques, pixel interpolation techniques and
annealing techniques for the compensation of live motion
video articles which may include voice, audio, text, still
image and animated graphics, said motion compensation
method provides a scalable architecture for communication
of multimedia data, formatting from internal format to
external format in compliance with any selective one of a


-79-


plurality of international standardized and customized
coded algorithms, and band width determination of available
display band widths according to frame rate and available
transmission band widths, said method comprising:
(a) capturing the scene of a motion video article
through frame differencing techniques;
(b) representing the motion video profile of said
motion video article in an article-oriented instruction set
format;
(c) defining said motion video profile of said motion
video article by individual object and relative movement
association;
(d) augmenting said motion video profile of said
motion video article by means of rotation, shifting and
shuffling operations;
(e) reformatting said motion video profile of said
motion video article to an internal file format;
(f) enlarging said motion video article by means of
upsampling and reducing said motion video article by means
of downsampling operations;
(g) overlaying said motion video profile of said
motion video article with said voice, audio, text, still
image or said animated graphics to construct a complete
multimedia presentation;
(h) defining said motion video profile of said motion
video article;
(i) interpolating said motion video profile of said
motion video article through regeneration of absent pixels
to further refine said motion video article;
(j) simulating said motion video profile of said
motion video article through capturing and recording
selective, nonconsecutive frame references;


-80-


(k) annealing said motion video profile of said motion
video article through reconstruction of said absent pixels,
absent motion articles or absent frames, employing graphics
animation techniques;
(l) selectively repeating steps (a) through (k) as
required.
30. A communication system apparatus executing a
plurality of control, processing, communication and
interface functions for the communication and storage of
multimedia articles including voice, audio, text, still
image, motion video, and animated graphics between
multimedia transmitters and receivers whereby said
communication system is compatible with multiple standard
or customized coding algorithmic signals selected from the
group consisting of H.261, MPEG, JPEG, EDTV and HDTV
wherein multiple incompatible video coding equipment
employing different video coding algorithms can communicate
with each other, said communication system providing a
scalable architecture for communication of multimedia data,
formatting from internal format to external format in
compliance with any selective one of a plurality of
international standardized and customized coded algorithms,
and band width determination of available display band
widths according to frame rate and available transmission
band widths, said communication system providing:
means for network communication processing, said means
comprising a network domain decoder and network domain
encoder, said means can provide control interface to a
telecommunication network or microwave network, said means
for network communication processing including receiving
functions for serial compressed video bit stream;


-81-


means for transmission processing, said means
comprising a formatter to transform between a CIF/QCIF
format and selective CCITT H.261, MPEG or JPEG formats,
said means providing conversion functions for serial to
parallel and parallel to serial, said means further
comprising means to encode and decode header messages for
H.261, JPEG and MPEG;
means for host processing, said means providing a high
speed interface between said communication system and an
external host, said means further providing a system of
independent communication between said communication system
and said external host for the exchange sequence of command
and data messages, said host means further providing
windowing operations permitting the system memory to be
memory mapped into a frame memory whereby said host
processor can view frame memory status and operations in
real time;
means for system host bus, said bus means permitting
said host processor to control access and communicate with
network communication processors, transmission processors,
pixel processors and frame memory;
means for a communication pipelining subsystem, said
means providing real time frame formatting, protocol
controlling transmission and receiving;
means for pixel processing, said means comprising a
bus controller and frame memory controller wherein said
pixel processing means serves as a bus master for a video
bus wherein said pixel processing means can access frame
video memory for pixel level operations, said means further
comprising bit level manipulation capabilities for variable
length coding and decoding, scan formatting conversion and
quantization;


-82-

means for display processing, said means for display
processing performing digital YUV to digital RGB
conversion, digital RGB to analog RGB conversion, digital
RGB to VGA conversion and analog RGB to NTSC conversion;
means for capture processing, said means for capture
processing permitting decoding of analog video input formats
into CCIR 601, said means for capture processing further
decoding said CCIR 601 into CIF formulated YUV signals;
means for frame storage, said means for frame storage
comprising a twin memory access storage for previous frames
and current frames according to a formulated CIF format in
order to facilitate frame differencing operations;
means for motion processing, said means for motion
processing including a means for loading luminance signal
in frame memory for performing motion estimation and
generating a motion vector for each block of storage in
frame memory;
means for transform processing, said means for
transform processing comparing the differences between the
previous and the present current blocks or macro blocks and
coding said differences in discrete cosine transform
coefficients;
means for video bus, said means for video bus providing
a high speed, bi-directional, parallel interconnect between
said frame memory, said capture processor, said display
processor, said transform processor, said pixel processor
and said motion processor; and
means for video subsystem pipelining, said means
comprising system-wide direct interconnection between said
capture processor, said pixel processor, said motion
processor, said transform processor, said frame memory and
said display processor.

Description

Note: Descriptions are shown in the official language in which they were submitted.





WO 94/09595 PCT/US92/08018
_ ~1~:'~1~~
METHOD AND APPARATUS INCLUDING SYSTBM
ARCHITECTURE FOR MULTIMEDIA COMMUNICATIONS
FIELD OF INVENTION
The present invention relates to a method and appa-
ratus for improving the efficiency of electronic communica-
tion and, in particular, to a method and apparatus which
can communicate with available electronic desk top equipment.
such as persona? computers, workstations, video cameras,
television. VCR's, CD players and telephones and receive,
store, process and send multiple forms of media information,
such as sound. image, graphics, video and data, both digital-
ly and algorithmically based on a plurality of selective
band widths.
HACRGROUND OF T8E INVENTION
Technology allows the individual to communicate with
others not only by the telephone, but also by telefax machines,
personal computers and workstations utilizing modems and
telephone lines and data and video information can also
be stored and disseminated by means of videotapes, compac t
discs and television monitors.
There are methods and apparatus available which allow
for large amounts of data to be reduced and transmitted
SUBSTiTlITE SHEET




WO 94/09595 PCT/US92/0801H
~~ ~ _2_ _
in a very short amount of time, such methods and apparatus
are known as compressing the data. Similarly, there are
methods and apparatus available for enhancing the image
quality tof visual and graphic data that has been compressed
and is now being displayed. For example, see U.S. Patents
4,772,947 to Kono:.U.S. Patent 4,703,350 to Hinman: U.S.
Patent 4,727,589 to Hirose; U.S. Patent 4,777,620 to Shimoni;
U.S. Patent 4,772,946 to Hammer: and U.S. Patent 4,398,256
to Nussmier.
While the aforesaid patents teach various methods
and apparatus for compressing and decompressing data and
enhancing the image quality of the data, none of the afore-
said patents have directed themselves to the concept and
structure of a method and apparatus which would communicate
with and share resources among the telephone, personal computer
or workstation, video screen and VCR to allow the individual
to select and convey multiple forms of media information
such as sound, image, graphics, data and live video in an
efficient and effective architecture which would automatically
adjust to available band widths and which would be capable
of communicating in multiple band widths.
SUBSTITUTE S~EE'i'




W~ 94/09595 ~ ~ ~ PCT/US92/08018
- -3-
OBJECTS OF THE INYENTION
An object of the present invention is to define an
integrated process architecture which can accommodate com-
munications~ both transmission and retrieval, of all di-
gitally-coded or algorithmic multimedia information.
Another object of the invention is to provide for
a novel system architecture which is flexible and allows
control of the variable communications band widths and
allows for flexible combinations of digitally-coded multiple
media information having application to teleconferencing
or educational instruction.
A still further object of the present invention is
to provide for a novel process architecture which not only
allows for digital coding techniques, but also can inter-
face with traditional analog storage or transmission tech-
niques.
A still further object of the present invention is
to provide for a novel. process architecture which allows
the user to control. program and select the appropriate
media combination either before or during the communication
session.
SUBSTITUTE SHEET


CA 02147164 2004-06-18
-4-
SUi~IARY OF THE INVENTION
The present invention provides a controller apparatus
for executing a plurality of control functions for
communication of multimedia articles including voice, high
quality audio, text, still image, motion video and animated
graphics between multimedia equipment and communications
networks, selectively providing receiving functions,
retrieving functions, and transmitting functions, wherein
said controller is compatible with multiple band widths and
algorithmic coded signals, said controller providing a
scalable architecture for communication of multimedia data,
formatting from internal format to external format in
compliance with any selective one of a plurality of
international standardized and customized coded algorithms,
and band width determination of available display band
widths according to frame rate and available transmission
band widths, said controller comprising: means for receipt
of an external network signal, said means comprising a
network domain decoder; means for receipt of a local
origination signal; and means for image processing of said
origination signal for storage coding simulation, and
transmission; a first control means and format means for
selectively formatting the format of said external network
signal and local origination signal to a compatible first
internal format in compliance with internal processing and
memory capabilities; means for decoding said external
network signal according to said first internal format,
said means selectively comprising a transform domain
decoder alone and in combination with a pixel domain
decoder; a second control means and format means for
selectively formatting said external network signal and
said origination signal to a second internal format in


CA 02147164 2004-06-18
-4a-
compliance with a selective plurality of coding algorithms
for processing, storage and transmission, said means
further comprising an encoding means which comprises a
pixel domain encoder alone and in combination with a
transform domain encoder for encoding said external network
signal and said local origination signal; means for
reconfiguring the memory means for selectively associating
with said apparatus encoders and decoders for directing
said first and second format means; band width management
means, in communication with said second format means and
said control means, for directing available run time
transmission band widths and appropriate compression ratias
for selectively formatting said second internal format to
external transmission format in compliance with available
transmission band width range, said band width management
means continuously moderating and correcting said selective
formatting of said second internal format to said second
external transmission format based on transmission band
width availability; means for transmission of said
internally formatted external network signal and local
origination signal according to said external transmission
format, said means comprising a network domain encoder.
The present invention further provides an apparatus
for the adaptive continuous management and control of
network traffic condition and band width availability for
the communication of multimedia articles including voice,
audio, text, still image, motion video and animated
graphics between multimedia transmitter and receivers
wherein said apparatus is compatible with multiple analog
and digital transmission standards, such as analog voice
grade line, PSDN, basic rate ISDN, primary rate ISDN/T1,
LAN, and FDDI, the apparatus having a band width detection,
means determining available display band widths in


CA 02147164 2004-06-18
-4b-
accordance with frame rate and run time transmission band
width, said apparatus comprising: a means for generating a
transmission request signal provided with message retaining
transmission band width for an outbound multimedia article;
a means for receiving a transmit acknowledge signal from a
network controller and destination receiver indicating said
band width availability; a means for relaying a band width
availability signal to a band width management unit for
comparison of band width availability to transmission
signal; a means for generating an alternate compression
ratio for said transmission signal compatible with said
band width availability; a means for implementing said
alternate compression ratio through a host or encoding
processor for further compression of said multimedia
article to meet said band width availability; a means for
transmitting said multimedia article on said available band
width; and a means for monitoring said multimedia article
transmission in said band width availability during
transmission.
The present invention also provides a method for the
adaptive continuous management and control of network
traffic condition and band width availability having a band
width detection means determining available display band
widths in accordance with frame rate and run time
transmission band width, for the communication of
multimedia articles including voice, audio, text, still
image, motion video and animated graphics between
multimedia transmitter and receivers whereby said apparatus
is compatible with multiple analog and digital transmission
standards such as analog voice grade line, PSDN, basic rate
ISDN, primary rate ISDN/T1, LAN, and FDDI, the method
comprising: (a) generating a transmission request signal
provided with message band width requirement for outbound


CA 02147164 2004-06-18
-4c-
multimedia articles: (b) receiving a transmit acknowledge
signal from an external network controller and destination
receiver indicating said band width availability; (c)
relaying said band width availability to a band width
management unit for comparison with band width of said
outbound multimedia articles; (d) generating an alternative
compression ratio for said outbound multimedia articles
compatible with said available band width; (e) relaying
said alternative compression ratio to a host or encoding
processor for further compression of said outbound
multimedia articles in conformance with said band width
availability; (f) transmitting said outbound multimedia
articles in conformance with said transmission band width
availability; (g) monitoring said band width availability
and said transmission of said outbound multimedia articles
during said transmission; (h) generating alternative
compression ratios during transmission to adapt to changes
in said band width availability; (i) repeating steps (a)
through (h) as required.
In another aspect, the present invention provides a
scalable, reconfigurable memory apparatus for executing a
plurality of preprocessing and post-processing functions
for the communication and storage of multimedia articles
including voice, audio, text, still image, motion video,
and animated graphics between selective multimedia
transmitters and receivers, said memory apparatus
compatible with multiple standard or customized coding
algorithmic signals including H.261, MPEG, JPEG, EDTV or
HDTV, said apparatus comprising: means for definition of a
default internal file format and size based upon coder
processing and frame memory system throughput; means for
receipt of an external network algorithmic coded signal,
said means comprising a network domain decoder; means for


CA 02147164 2004-06-18
-4d-
receipt of a locally-originated signal, said means
comprising a capture processor; means for identifying and
receiving said multimedia article and file size, said means
comprising a host processor selectively interfaced with
said capture processor and network domain decoder; means
for downsampling wherein said multimedia article can be
conformed and reduced to a predetermined internal file
format and size, said means comprising a reconfiguration
unit and scaler circuit; means for manipulation of said
internal file format, said means comprising said host
processor and a pixel processor; means for adjustment of
said internal file format and size to conform to band width
availability, said means comprising a band width management
unit in communication with said scaler circuit, said
reconfiguration unit and said host processor; means for
adjustment of said internal file format and size during
interframe coding, wherein motion video foreground articles
and still background articles are differentiated and
separately transmitted, said means comprising a frame
differencing circuit, and motion processor in interface
with said pixel processor; and means for providing
upsampling during transmission wherein said transmission cf
said multimedia article can be formatted and sized, said
means comprising said reconfiguration unit, and said scaler
circuit in interface with a band width management unit.
In yet another aspect, the present invention provides a
scalable, reconfigurable memory array method for executing
a plurality of preprocessing and post-processing functions
for the communication and storage of multimedia articles
including voice, high quality audio, text, still image,
motion video and animated graphics between multimedia
transmitters and receivers, whereby said method is
compatible with multiple standard customized coding


CA 02147164 2004-06-18
-4e-
algorithmic signals including H.261, MPEG, JPEG, EDTV or
HDTV, wherein said method provides a scalable architecture
for communication of multimedia data, formatting from
internal format to external format in compliance with any
selective one of a plurality of international standardized
and customized coded algorithms, and band width
determination of available display band widths according to
frame rate and available transmission band widths, said
method comprises: defining of a default internal file
format and size based upon coder processing and frame
memory system throughput; selectively receiving an external
network algorithmic coded signal and local origination
signal; identifying the multimedia article signal and size;
downsampling the received multimedia article to a
predefined internal file format and size; manipulating said
internal file format to conform to transmission band width
availability; distinguishing between motion video
foreground articles and still background articles during
interframe coding and intraframe coding modes; upsampling
said internal file format to selectively desired format and
size for transmission in accordance with transmission band
width availability; distinguishing between motion video
foreground articles and still background articles during
interframe coding and intraframe coding modes; upsampling
said internal file format to selectively desired format and
size for transmission in accordance with transmission band
width availability.
In a further aspect, the present invention provides a.
method for the application of image preprocessing
techniques for the interframe coding of multimedia articles
including voice, audio, text, still image, motion video and
animated graphics between multimedia transmitters and
receivers, whereby said method can be integrated with an


CA 02147164 2004-06-18
-4f-
image capturer circuit and band width management circuit,
said method comprising: (a) capturing a local origination
still image in accordance with frame update requirements
established by a host processor; (b) capturing and storing
the complete image frame during the intraframe coding mode;
(c) comparing and differencing said present captured frame
to a previous captured frame during the interframe coding
mode; (d) storing the frame differences between said
present frame and said previous frame in blocks and macro
blocks; (e) registering and coding said frame differencing
blocks or said frame differencing macro blocks into a frame
differencing bit map; (f) converting color space from
analog RGB or NTSC format into a digital RGB or YUV format;
(g) retrieving the luminance macro blocks according to said
frame differencing bit map; (h) deriving the appropriate
motion vector for each of said macro blocks through
comparison and selection of the minimum distortion among
adjacent macro blocks; (i) reducing from an external file
format to an internally-defined format in accordance with a
downsampling ratio established by said host processor, the
reconfiguration unit and the staler; (j) translating said
frame differencing bit map to conform with said internally-
defined format; (k) performing pixel domain and transform
domain coding operations for said frame differencing macro
blocks for said Y, U and V format; (1) enlarging from said
internal file format to an external file format according
to an appropriate downsampling ratio as established by said
host processor, reconfiguration unit and staler; (m) adding
said frame difference to said previous frame in order to
display and update said current frame; (n) repeating steps
(a) through (m) as required; (o) repeating steps (a)
through (n) for live video sequence as required.


CA 02147164 2004-06-18
_qg_
The present invention also provides a method for the
application of frame differencing techniques, pixel
interpolation techniques and annealing techniques for the
compensation of live motion video articles which may
include voice, audio, text, still image and animated
graphics, said motion compensation method provides a
scalable architecture for communication of multimedia data,
formatting from internal format to external format in
compliance with any selective one of a plurality of
international standardized and customized coded algorithms.,
and band width determination of available display band
widths according to frame rate and available transmission
band widths, said method comprising: (a) capturing the
scene of a motion video article through frame differencing
techniques; (b) representing the motion video profile of
said motion video article in an article-oriented
instruction set format; (c) defining said motion video
profile of said motion video article by individual object
and relative movement association; (d) augmenting said
motion video profile of said motion video article by means
of rotation, shifting and shuffling operations; (e)
reformatting said motion video profile of said motion video
article to an internal file format; (f) enlarging said
motion video article by means of upsampling and reducing
said motion video article by means of downsampling
operations; (g) overlaying said motion video profile of
said motion video article with said voice, audio, text,
still image or said animated graphics to construct a
complete multimedia presentation; (h) defining said motion
video profile of said motion video article; (i)
interpolating said motion video profile of said motion
video article through regeneration of absent pixels to
further refine said motion videc article; (j) simulating


CA 02147164 2004-06-18
-4h-
said motion video profile of said motion video article
through capturing and recording selective, nonconsecutive
frame references; (k) annealing said motion video profile
of said motion video article through reconstruction of said
absent pixels, absent motion articles or absent frames,
employing graphics animation techniques; (1) selectively
repeating steps (a) through (k) as required.
In a final aspect, the present invention provides a
communication system assembly apparatus executing a
plurality of control, processing, communication and
interface functions for the communication and storage of
multimedia articles including voice, audio, text, still
image, motion video, and animated graphics between
multimedia transmitters and receivers whereby said
communication system is compatible with multiple standard
or customized coding algorithmic signals such as H.261,
MPEG, JPEG, EDTV or HDTV wherein multiple incompatible
video coding equipment employing different video coding
algorithms can communicate with each other, said
communication system providing a scalable architecture for
communication of multimedia data, formatting from internal
format to external format in compliance with any selective
one of a plurality of international standardized and
customized coded algorithms, and band width determination
of available display band widths according to frame rate
and available transmission band widths, said communication
system providing: means for network communication
processing, said means comprising a network domain decoder
and network domain encoder, said means can provide control
interface to a telecommunication network or microwave
network, said means for network communication processing
including receiving functions for serial compressed video
bit stream; means for transmission processing, said means


CA 02147164 2004-06-18
-4i-
comprising a formatter to transform between a CIF/QCIF
format and selective CCITT H.261, MPEG or JPEG formats,
said means providing conversion functions for serial to
parallel and parallel to serial, said means further
comprising means to encode and decode header messages for
H.261, JPEG and MPEG; means for host processing, said means
providing a high speed interface between said communication
assembly and an external host, said means further providing
a system independent communication between said controller
assembly and said external host for the exchange sequence
of command and data messages, said host means further
providing windowing operations permitting the system memory
to be memory mapped into a frame memory whereby said host
processor can view frame memory status and operations in
real time; means for system host bus, said bus means
permitting said host processor to control access and
communicate with network communication processors,
transmission processors, pixel processors and frame memory;
means for a communication pipelining subsystem, said means
providing real time frame formatting, protocol controlling
transmission and receiving; means for pixel processing,
said means comprising a bus controller and frame memory
controller wherein said pixel processing means serves as a
bus master for a video bus wherein said pixel processing
means can access frame video memory for pixel level
operations, said means further comprising bit level
manipulation capabilities for variable length coding and
decoding, scan formatting conversion and quantization;
means for display processing, said means for display
processing performing digital YUV to digital RGB
conversion, digital RGB to analog RGB conversion, digital
RGB to VGA conversion and analog RGB to NTSC conversion;
means for capture processing, said means for capture


CA 02147164 2004-06-18
_47_
processing permitting decoding of analog video input
formats into CCIR 601, said means for capture processing
further decoding said CCIR 601 into CIF formulated YUV
signals; means for frame storage, said means for frame
storage comprising a twin memory access storage for
previous frames and current frames according to a
formulated CIF format in order to facilitate frame
differencing operations; means for motion processing, said
means for motion processing including a means for loading
luminance signal in frame memory for performing motion
estimation and generating a motion vector for each block of
storage in frame memory; means for transform processing,
said means for transform processing comparing the
differences between the previous and the present current
blocks or macro blocks and coding said differences in
discrete cosine transform coefficients; means for video
bus, said means for video bus providing a high speed, bi-
directional, parallel interconnect between said frame
memory, said capture processor, said display processor,
said transform processor, said pixel processor and said
motion processor; and means for video subsystem pipelining,
said means comprising system-wide direct interconnection
between said capture processor, said pixel processor, said
motion processor, said transform processor, said frame
memory and said display processor.




Wp 94/09595 ~ ~ PCT/US92/08018
-- - S-
BRIEF DESCRIPTION OF T8E DRAWINGS
These and other objects of the present invention will
become apparent particul ' ly when. taken in view of the following
illustrations wherein:
Figure 1 is a pictorial illustration of the communication
system;
Figure 2 is a schematic diagram illustrating the overall
system methodology;
Figure 3 is a schematic of the controller's internal
operating mode for illustrating band width management;
Figure 4 is a schematic of the internal circuitry of
the multimedia communications assembly:
Figure 5 is a schematic of the network communications
processor and its design relationship to the transmission
processor;
Figure 6 is a schematic illustrating the communication
between the host processors system memory, pixel processor,
frame memory and display processor:
Figure 7 is a schematic of the video codec and display
subsystem;
Figure 8 is a schematic illustration of the standard
CIF and QCIF memory format;
Figure 9 is a schematic illustration of applicant's
scalable memory array reconfigurable technique;
Figure 10 is a schematic illustrating the pixel processor
flexibility to various video coding algorithms;
SUBSTCrUTE SMEET


CA 02147164 2003-06-25
Figure 11A is a schematic: of tale nuotion processor
subsystems;
Figure 11B is a detail of the cellular logic processor
of Figure 11A;
Figure 12A illustrates a programmable logic device
employing cellular, array logic:: arch.tec°ture;
Figure 12B illustrates the imp~emE~ntation of cellular_
logic processing; anc~
F figure 13 is a ~chem~~t_i ~.~, ~:~f trmV~ multimedia a~asembly.


CA 02147164 2003-06-03
DETAILED DESCRIPTION OF THE DRAWINGS
Referring to Figure 1, there i.s shown a pictorial
illustration deb>ictin.g the electronic devices 132
available presently for the home or office. These
include a VCR 102, CD player 103, telephone :L04,
television 106, personal computer 108 and fa:~ machine
110. Each of these electronic devices 132 has a distinct
function. The telephone 104 can transmit and receive
audio and dat:a; a fax: ~rmchine 1.10 can transmit and
receive text documents, a television 106 can receive
video broadcasts and audio; and a personal computer 108
can be used f_or many data processing applications. It is
Applicant's intention i~o disclose an assembly 112 which.
can physically communicate with these electronic devices
132, via a remote cont:ool device 105, to permit them to
function complimentarvy with each other and to communicate
with other electronic devices regardless of whether the
other communication dev:icE~s were digital or algorithmic
and t.o code and decode automaticallv too the available
band width. The communication is accomplished by a
multimedia communicati.c>n:~ assembly 11:?, being of size a.nd
shape, similar to thar_ of_ a VCR. The aforementioned
electronic devices 132 wowld interconnect 1.20 with the
multimedia communications assembly :11.2 to allow the
user/operator to control, complement and ut.i:lize the
functions of the el.ect:ronic dev ices 1.32 by means of they
mufti-media communications assembly lla.
Figures 2A and 2B illustrate the overall system
operation fo:r the multimedia communications assembly 11.2.


CA 02147164 2003-06-03
Assembly 112 makes it possible to exchange a multitude of
different forms of media objects over a wide range of
communicaticaz networks. Prior art ha:~ shown methods anal
apparatus to improve cornpression and decompression
techniques for individual media types and individual band
width ranges. HowevEr, ;~i.nce video coding algorithms are
intrinsicall~,r incompat.ible~ with each other, t=here is need
for an assembly 112 to provide a common interface whereby
incompatible equipment ca:r~ freely exchange me=dia objects
through interfacing with assembly 112.
Figures 2A .and 2F~ :i=ilustrate the following major
system components: a network communication processor 202;
a transmission processor 204; a pixel processor 206; a
display processor 2.12; a (came memory 214 and a host
processor 218.
The design of the system architecture as described.
in detail hereafter is to gain the ability too interface
with multiple types of media objects, including
audio, still image, mc~t:ion video, text. and graphics .
2 ~ As illustrated in Figi~i-e s 2A and 2b, graphics
input might possi_bl~~ be in the f-_orm


CA 02147164 2003-06-03
_.g_
of an RGH format 229; YGA format 226; XGA format 228; or
SVGA format 230. Text media objects could be either in
the form of a (~rouF: 3 (G;e i r=orrnat= 23'<?; wroup 4 (G9 ) format 234; o:r
ASCI format 235. Motion media objects may conform either
to H.261 format 238; MPEG format 240; or other specialized
formats 242, Still background media objects could be con-
forming either to JPEG format 294 ar other specialized formats
239. Input audio media objects could be conformfing to CD
audio format 296; voice grac9e audio 248 or FM audio format
250.
Each media object within a category, namelyr audior
still image. motion video, text and graphics would be im-
ported to a multiplexes 252 dedicated to each category in
order to identify the input signal and then be directed
to a dedicated overlay 259 for each category of media ob-
ject. The overlay 254 provides the ability for the assembly.
disassembly. deletion, addition end modification of a
selected group of multimedia objects. The input signals,
be they audio, still images motion video, text or graphicsr
are converted into computer object-oriented language format
for encoding into a frame memory 214 as described hereafter.
This conversion before ator;ing into frame memory 219 in
cooperation r~i.th the major components of the system described
hereafter, permit the c:ompi:lation of selected input signals




WO 94/09595 PCT/US92/0801 R
-io- _
which have been stored in the frame memory 214 to be as-
sembled. interpreted and translated to other system formats
with relative ease as a result of the intelligent memory
management capability inherent in this design.
The system architecture provides for an interface
which will enable multiple incompatible video coding equip-
meet employing different video coding algorithms to communi-
cate. This is accomplished through a scalable frame memory
architecture reconfigurable technique (SMART) described
in Figure 9.
In simplistic terms to be described in detail here-
after, the design of assembry 112 allows host processor
218 to identify the types of input articles during the im-
port stage, the host processor will then instruct the recon-
figuration circuit 256 and the scaler circuit 258 to pro-
vide the required down-sampling ratio. The media article
being imported can then conform or be reduced to the internal
file format during the import stage. The reverse is true
during the exporting stage when the media article in the
internal file can be enlarged and made to conform to the
appropriate algorithm for the exporting stage. As a result
of our smaller internal file size, the real time performance
requirement of our pixel processor 206, graphics processor
SUBSTITUTE SHEET



WO 94/09595 ~ ~ ~ PCT/US92/08018
-11-
222, transform processor 210 and motion processor 208 is
reduced. Further, the speed and size of the frame memory
214 is also proportionately reduced. This design allows
' various coding algorithms to be micro-coded at pixel processor
206.
Assembly 112 also optimizes the video coding for
specific compression ratios in order to meet specific band
width requirements. In order to adjust the band width to
meet the various communication network requirements, band
width controller 260 receives the band width requirement
from the network communication processor 202, the band width
controller 260 will then instruct the host processor 218
to develop the appropriate compression ratio in order to
meet the real time performance requirements. Band width
controller 260 will also interface with transmission pro-
cessor 204 in order to import and export the media article
at the appropriate band width. Assembly 112 can program
the network communication processor 202, the transmission
processor 204 and the display processor 212 to provide the
various types of communications interfaces.
The internal operation modes of host processor 218
permit it to adapt to different compression ratio require-
ments and network band width requirements. As an example,
the following are some popular network band width inter-
faces:
SUBS~Z~'~l'TE SHEE'~




WO 94/09595 PCT/US92/08018
(~~ ~~ ~ -12- _
1. Communicating over an analog phone line employing
Y.32 modem, 9,600 bit per second (bps) band width is required,
a quarter common immediate frame (QCIF) format is displayed
at 7.5 frames per second (fps).
2. Communicating over a digital ISDN D channel at
16 kilo bits per second (kbs). The user has two options,
either two quarter common intermediate frame (QCIF) formats
can be displayed at 7.5 frames per second or one quarter
common intermediate frame can be displayed at 15 frames
per second.
3. Communicating over an analog phone line whereby
19,200 bit per second band width is required. The user
has two options, either two QCIF (common intermediate frame)
formats can be displayed at 7.5 frames per second or one
QCIF (quarter common intermediate frame) can be displayed
at 15 frames per second.
4. Communicating over switched 56 kilo bits per
second digital network. Quarter common intermediate frames
with three quality level options will be updated at 15 frames
per second.
5. Communicating over a single ISDN B channel over
an ISDN basic rate interface network, four quarter common
intermediate frames will be concurrently updated at 15 frames
per second.
5~,~8STITt3T'E ~H~~'~'



WC194/09595 ~ ~ ~ ~ ~ PCT/US92/08018
-13-
6. Communicating over a dual ISDN B channel in
a ISDN basic rate interface network, quarter common intermediate
frames will be transmitted at 30 frames per second.
7. Communicating over a 384 kilo bits per second
ISDN H1 network, common intermediate frame will be transmitted
at 15 frames per second.
8. Communicating over a 1.544 kilo bits per second
T1 network, common intermediate frames (CIF) will be trans-
mitted at 30 frames per. second.
As a result of the aforesaid plurality of band widths,
it is necessary for the multimedia assembly to continuously
monitor the processor and network band width availability
and to simultaneously determine the amount of compression
or decompression that is required with respect to the data
in frame memory 314 to be transmitted. Due to the variable
band width or throughput requirement for each transmission
network, only dedicated processor approaches have been shown
in the prior art to meet a specific band width performance.
For example, three video conferencing techniques are required
at the 112 Kbs, 384 Kbs and 1.544 Mbs band width range.
The multimedia assembly disclosed herein, includes different
transceiver pairs for each specific network type. The system
SU8ST1TUTE SI~EE~'


CA 02147164 2003-06-03
-14-
architecture disclosed herein, and in particular, host pro-
cessor 218 in conjunction with band width controller 260
unity sealer 256 and reconfiguration unit 256, can continu-
ously adapt to a variety of network and processor band width
changing situations, for exampler noisy local line condition
and network traffic congestion. 'his is possible as a result
of the scalable memory architecture Which permits the con-
tinuous reprogramming of they internal file format of frame
memory 214 so that it is nuitable for the specific band
width requirement at that me>ment.
During the interframe coding mode 278, after the
incoming media articles are receivedt the appropriate frame
size 262 will be adjusted fi.rstr frame by frame difference
264 will then be calculated. For consecutive frame processing.
an appropriate motion vector 2110 can be derived,. For selective
frame processing, due to the difficulty to identify a suitable
motion vector 270,. interpolation techniques 266 can be employed
to simulate frame difference signal. Decision logic 272
is employed to analyze situation and make a final decision.
_.. ca>e of sce:~e changes a:. ~, ~~ysterr~ vai~1 be reset tc~ intraframe
cccing mode 1228 for furth~~i:v processing. p detailed design of
the motion pr~acessor 208 is further chown in Figure 11.




WO 94/09595 ~ ~ ~ PCT/US92/08018
-15-
Although our invention entitled "multimedia", we
have been mostly focued on "new hardware and software tech-
piques" for the "motion video". In addition, we have also
shown new techniques how to~integrate (overlay) motion video
with other media article in order to create a complete multi-
media presentation. Since there have been plenty of prior
arts showing techniques to handle other media, i.e., CD
audio, fax, telephone, computer graphics, or digital camera.
Also because the performance requirement for these media
types are much less demanding. Therefore, the encoding
and decoding of other media types in our invention can be
easily implemented in general purpose computer hardware
and software, embedded hardware controller, or special pur-
pose digital-signal processors.
Figure 3 is a schematic illustration of the controller's
operating modes for band width management based upon the
international compression standard CCITT H.261. Based upon
this standard, each common intermediate format frame (CIF
frame) 302 consists of twelve (12) groups of blocks and
each group of blocks would consist of thirty-three (33)
macro-blocks with each macro-block consisting of six (6)
blocks (4 Y's and 2 U/~'s). Each block would consist of
BX8 pixels and each pixel would consist of an 8 bit value.
SUBSTfTUZE SH~~~


CA 02147164 2003-06-25
The t~uarter common intermediate format frame (QCIF frame)
304 would consist of three groups of blacks and these would
be identical to those of the CIP""s 302.
In multimedia assembly 112, host processavr 21B has
eight (8) different network interface modes 306. The first
interface mode 310 is for 9.6 Kbs analog modems. The second
interface mode 312 is for 16 Kbs 'ISDN I7 channel. The third
network interface mode 314 is for 19.2 Kbs high speed analog
modems. The fourth network interface mode 316 is for .'S6
Kbs digital network (PSDN). The fifth network interfa~CQ
mode 318 i~ for 64 Kbs ISDN aingl~e S channel. The sixth
network interface mode 320 is for dual 8 channel 128 Rbs
ISDN HRI netWOrk. The seventh network interface mode 322
is for 384 Itba ISDN Hl net~rork and .the eighth network in-
terface mode 324 is for 1.544 Mbs ISSN PRx or T1 network.
Host processor 218 also has programmable frame updating
rate capability 325. Frame updating rate :326 provides bast
processor 218 With five options. They can be either 30
frame per second (fp";) 3=~:; 15 fFo;s 3;~~~; ~0 tps 334; ~a.5 fp;> 3:35
or 1 fps 336.
The standard fc~ame upc~at~ rvGtE~ 326 f~a:~ eacab network
interface mode 305 would be 1 :fps 336 i:o:~ firsts network
interface mode 310; 7.,5 fps .for second network interface
mode 312; 2 fps :fo:c~ t.hirr_nE:t:w~:rt,: mite>ra;~o~~ cncdr~~ 319, 6.5 fps r


CA 02147164 2003-06-25
for fourth network interface mode. 31.6; ~,~ fps 3:35 for fifth
interface mode 3~.$P i.5 fps ~i33 i~:az sixtt: and seventh interface:
mode 32n~ and 322, r_espect~:i ve;~.ly aa~d ~0 f~>.> a:3i:' for eighth
interface mode 324.
Ire Figure 3, w~:~ have estaka:l.ishec9 ~~G a:ps :332 of frame
update rate 326 as t:he default update rat.p? for C:If format 3~J2
transmission and '7.5 :Fps 335 as t:r~e de:rau:L~ up,~ai:<.=, rate
for QCIF format 304 trsna~misa~ian. Tha compression ratios
illustrated in Figure 10 and described hereafter are far
this default update rate.
The CIF format 302 aya~tem throughput requires 4.6
mega by.tQa per second (MBS~. The QCIF formal 304 requires
288 kilo bytes ger second. l~a~ruming We use 8 kilo bytes
per second as the measuring base for. real time video trans-
mission over fifth network interface mode 318. the eIF format
302 system Would reguire a~ compression ratio of 576:1 based
upon the CCITT H.261 compre$~ion standard. The QCIF format
304 voul8 require a 36:1 compression ratio. Similarly,
with respect to the other netWark interface modes 3Q5, the
compression ratios t~onld be has folloWa: The eighth network
interface mode 321 ~oul~d require a CIF format 302 compreasiors
ratio of 24:1 vhsr~ea~s Q'CIF format 3D~1 would require a 1.5:1
compression ratio; seventh networ~t inter:~e~ce mode 322 would


CA 02147164 2003-06-03
-ifi-
require a CIF format 3D1 comgressiorr ratio of 96:1 and a
QCIF format 304 ratio of 6:1; fourth network interface mode
316 would require a CIf fermat 302 compression ratio of
658:1 and a QCxF format 304 ratio of 41:1: third network
interface mode 314 world require a CIF format 30Z compres-
sion ratio of 1,920:1 and a QCxF format 304 ratio of 120:I;
the first network interface mode .sIO would require a CIF
format 302 ratio of 3,H40:1 and a QCIF format 304 ratio
of 20:1.
As a standard operation in Applicant's mu3timedia
assembly single QCIF format 304 will be employed for the
first through fifth network interface modes 310r 312r 314,
336 and 318 respectively. Double oCIF format will be employed
for sixth network interface mode 320 and single CIF format
302 or quadruple QCIF t:ormat 304 4equences will. be utilized
for the seventh and eighth network interface modes 322 send
324.
The advantagew of Applicant's multimQdia communica-
tions assembly 112 and its operation and capabilities will
be discussed r.erea'ter, f:igio°es 4A and 9Es illustrav_e a schematic:
view of the multimedia communications assembly 112. It
consists of the following mayor system components. They
are a network communications processor 202; a transmission



WO 94/09595 PCT/US92/08018
-19-
processor 204; a pixel processor 206; a motion processor 208;
a transform processor 210: a display processor 212:
a capture processor 220: a frame memory 214 and a host
processor 218. These system components can be imple-
mented either using custom integrated circuit devices,
a programmable integrated circuit: microprocessor: micro-
controller: digital signal processor or software. depending
upon the specific~system performance requirement.
The system components are interconnected through
a system host bus 418 and a high speed video bus 422. The
system host bus 418 allows the host processor 218 to control
access and communicate with the system components such as
the network communication processor 202, the transmission
processor 204, the pixel processor 206, and the frame memory
214. The video bus 422 interconnects the frame memory 214
with such components as the capture processor 220, the display
processor 212, the transform processor 210, the pixel proces-
sor 206 and the motion processor 208 to perform high speed
video signal processing functions. Both the system host bus
418 and the video bus 422 are bi-directional parallel buses.
Due to the real time performance requirements for
the high speed video frame processing, two system-wide inter-
connections are implemented. The first is the video pipeline
424 consisting of a direct interconnection between the cap-
ture processor 220, pixel processor 206, motion processor
208 transform processor 210, frame memory 214 and display
St3BSTtTtITE SHEET




WO 94/09595 PCT/US92/08018
-20-
processor 212. The second system interconnect 342 consists
of the direct interconnection between the network communication
processor 202, transmission processor 204, host processor
218 and pixel processor 206. In order to facilitate these
interconnect operations, first in, first out memory devices
428 are inserted where appropriate.
The frame memory 214 can be implemented either in
static random access memory 430 or video random access memory
434. The static random access memory 430 is easier to imple-
ment~ but at a higher cost. The video random access memory
(VRAM) 434 is less expensive, but slower than the static random
access memory 430 and requires a controller 434 to update
the memory array. The video random access memory 434 is provided
with two access ports 436 and 437 providing access to the
random accessible memory array. This is done since many video
coding algorithms employ frequent use of the interframe coding
440 to reduce band widths. Namely, only the frame difference
signal 442 will be transmitted. Therefore, the twin memory
accesses are required to store both the new frame 444 and
the old frame 448 and to facilitate frame differencing
operations 450. In this design, the pixel processor 206
serves as the bus master 420 for the video bus 422 by having
the video random access memory (YRAM) controller 434 function
SU~STIT~E SHEET


CA 02147164 2003-06-03
_21-
positioned within the pixel processor 206 care. This alloys
pixel processor 206 the ability to control video bus 422 and
to access video random access memory pixel storage for pixel
level operations 454. Pixel proces$or 206 also is equipped
with the bit level manipulation functions 456 such as variable
A
length coaer and decadexv (VLC~ 4584 scan format converter
460 and a_uantization converter 4&2. These permit the pixel
processor to utilize international video coding algorithms
for communicating as discussed hereafter.
The capture processor 220 can decade various type$
of analog videc input formats and convert them (e. g., NTSC
464, PAL 466, SCAM 468, or SYHS 469 to CCIR 601 4?O.~Y 471
4;2;2 47~. The ability of the capture processor 220 to de-
code the aforesaid formats provide f.or a convenient inter-
face between the multimedia communications assembly 112 and
the television 106r VCR 102 or video camera 465.
The CIF 302 formulated YuY 471 signals will first
transfer out of the capture processor 220 and store in the
frame memory i19. The luminance (Y) signal 474 will be loaded
into the motion processor 208 to perform motion estimation
475. A motion vector ~~ uill be developed for each macro
block 477 and store in the associated frame memory 214 location.
The difference between the new and old macro blocks will




WO 94/09595 PCT/US92/08018
-22- _
also be coded in discrete cosine transform (DCT) coefficients
478 using the transform processor 210. Pixel processor 206
will perform a raster to zig.~zag scan conversion 460, quanti-
zation 462 and YLC coding 458 of the DCT coefficients 478
for each macro block 4?7 of luminance 474 and chrominance
473. The transmission processor 204 will format the CIF 302
frames into the CCITT H.261 238 format and attach the appropriate
header 481 information. As an example, a CIF frame 302 will
partition into twelve groups of blocks 482 and each group
of blocks 4a2 will consist of thirty-three macro blocks 477
and each macro block 477 will be composed of four luminance
signals 474, and one U & V signal 473. The network communica-
tion processor 202 will provide the control interface to the
telecommunications network 480 or to a microwave link 483.
On the receiving side, the serial compressed video
bit stream 484 will be received from the network communica-
tion processor 202. The bit stream will be converted from
serial to parallel and decode the appropriate header message
481 using the transmission processor 204. The information
will then be sent to the frame memory 214 through pixel pro-
cessor 206. Pixel processor 206 will then perform a variable
length decoder 458, zig-zag-to-raster scan conversion 460
and dequantization 463. The YUV 471 macro block 477 of DCT
S~3'~S~Tt'I'~'~'c ~HEET

I
I
CA 02147164 2004-11-22
-23-
coefficients 478 will be sent to frame memory 214 through
pixel processor 206. Pixel processor 206 will then send
YUV 471 macro blocks 477, one at a time to the transform
processor 210 to perform inverse DCT operation 485. The
YUV 471 difference 450 will then be added to the old signal
452 to conform to a new YW pixel 446 for each macro block
477. The display processor 212 will then perform YUV 471
to RGB 224 conversion and generate analog signal from the
RGB 224 or thence generate an 8 bit VGA 226 color image
through color mapping 486. The display processor 212 will
then provide a convenient interface to various displays
such as television 106, personal computers 108 or monitor.
For ease of interface, host processor 218 also
provides for a high speed small computer system interface
(SCSI) 488 with the external host 487 such as a personal
computer or work station. The advantage of the small
computer system interface 488 is that it provides a system
independent interface between the external host 487 and
the multimedia communications assembly 112. Since only
simplified control messages 489 are required to pass
between the two hosts, modifications to the system to
provide for various operation formats such as DOST"' 491,
UNIXTM 490 or MacintoshTM 492 can easily


CA 02147164 2003-06-03
_Zf,~_
be accomplished. The thigh speed small computer syste m inter-
face .488 will also allcyw the transmission of video sequence,
bQtween the two hosts.
In the case c>f high speed digital network communi-
cationr the communication pipeline is employed to facilitate
real time frame formatting 4113, protocol controlling 412,
transmission and decoda,ng. The host processor 21B is
the bus master 420 for the system bus 418. Consequently,
host processor 2I8 wild. be able to access to the fra me memory
214 and/or system memory 216, and monitor progress through
a rrindos~ing operation X194. The windowing operation 494 essen-
tially allows a portion of 'the system memory 216 to be memory
mapped 495 tc~ the frame memory 214 sc that the host processor
218 can use it as a window to vier~ frame memory 214 status
and operations in real time.
Figure 5 illustrates the network cammunication pro-
ceaaor 202 and its design relationship tc transmission processor
204. Network communication processor 202 is comprised of
an analog front end transceiver 514r digital signal processor
modem 515 and a buffer memory 518. These network communication
processor 202 components are interconnected through a private
NCP bus 520. The trane,misaion processor 204 consis to of a
frame formatter 522, a protocol controller ~~.~ and an error



2I~7~,~4
WO 94/09595 PCT/US92/08018
-25-
processor 526. The transmission processor 204 components
and the buffer memory 518 are interconnected through another
private X bus 528. The bit-serial D bus 530 facilitates the
network communication processor 202 and transmission processor
204 communication through digital signal processor modem 516
and frame formatter 522 sub-systems. The private NCP bus
520, D bus 530 and X bus 528 are designed to facilitate effective
data addressing and transfer in between the sub-system blocks.
Furthermore, the buffer memory 518, digital signal processor
modem 516 and protocol controller 524 are interconnected to
the host processor 218 through system bus 418.
The specific requirement of the bus design, which
may include address 510, data 512 and control 502 sections
is dependent upon the data throughput 504, word size 506 and
bus contention 508 considerations. The network communications
processor 202 implements the DTE 536 function while the host
processor 218, and transmission processor 204, perform the
DCE 532 function. This allows the proper pairing of the DCE
532 and DTE 536 interfaced to a local customer premises equipment
534 so as to perform conference control 538, store and forward
540 or band width management 542.
Within the network communication processor 202 sub-
system, digital signal processor modem 516 is the local host
SUBSTITUTE SHEET




WO 94/09595 ~ ~ PCT/US92/08018
-26-
controller 544. Analog front end 514 consists of an analog
to digital converter (ADC) 546 and a digital to analog conver-
ter (DAC) 548: The analog-to-digital converter 546 samples
and holds the analog input signal 550 and converts it to a
digital bit stream. The digital-to-analog converter 548 buffers
the digital output bit streams and converts them into an analog
output signal. The analog front end is the front end interface
to the telephone network 480 from the system. The output
digital bit stream from the analog-to-digital converter 546
is then transferred to the buffer memory 518 for temporary
storage. The digital signal processor modem 516 will access
this information, through buffer memory 518 to perform line
coding functions. Inside the digital signal processor modem
516 is a programmable digital signal processor 552. Digital
signal processor 552 is programmable allowing for easy imple-
mentation of line coding 554 and control 556 functions for
many of the analog front end 514 functions.
Within the transmission processor 204 sub-system,
the frame formatter 522 first received the incoming information
frame 558 and header message 481 from the digital signal pro-
cessor modem 516 and identifies the proper receiving video
coding algorithm types 560. Protocol controller 524 then
takes over and starts the appropriate protocol decoding 562
SUBSTITUTE SHEET


CA 02147164 2003-06-03
- ~ j ~.
procedures. Once the control frame 564 and information frame
558 header information are fully decoded, the information
frame 558 is sent to the error processor for error checking
and correction. Corrected bit streams are then convertQd
from aerial to parallel farm using serial to parallel con-
verter 5b8 and are stored in the first in and first out buffer
428 for further processing The first in. first out buffer
428 is designed into four 32K bite section. ~ach section
allows for storage of 32FC bit which is the maximum allow-
once of a compressed CIF frame. Therefore, 128K bits in the
first in~ first out buffer allows double buffering and simul-
taneo~xs transmitting and receiving of the incoming and aut-
going video information frames.
In ordet~ to accommodate the variau5 network environ-
menu, the netmork c:ommun::::ations processor 20:: is designed to
operate =:-~ the following specific speeds.
9.5 Kbps (Kiln bits per second), 19.2 Kbps, 56 Kbps,
64 Kbps, 128 Kbps, 384 Kbps, 1.549 Mbkps (mega bits
per second? and 2.048 Mbps. 8P Will offer three
options as the net:faork interface mode 306. ~r
mode 2, single CIF or four QCIF sequences rill be
offered at 38A Kbps and higher. In mode 3r two
QCIf seguence:a will be offered simultaneously at
12E Kbps.




WO 94/09595 PCT/US92/080~ 8
_28_
When line conditions degrade, the analog front end
514 will become aware of the degradation as a result of in-
coming frame synchronous signal 570. Analog front end 514
will then notify the digital signal processor modem 516 and
host processor 218. Host processor 218 will then switch'from
a standard operation to an exception operation mode. Host
processor 218 has three options to lower the bit rate in order
to accommodate and correct the degradation. Option 1 would
be for the host processor 218 to notify the pixel processor
206 and select a coarser quantization level 572. Option 2
would be to drop the frame update rate and increase the in-
terpolation rate 574. Option 3 would be to drop from CIF
to QCIF 576. When the error processor 526 detects more than
two single bit errors, the error processor 526 will notify
the pixel processor 206 and host processor 218. Host proces-
sor 218 again has two options. Either pixel processor 206
can request for an retransmission or host processor 218 can
delete the complete macro block 477 and wait until the next
macro block is sent. Meanwhile host processor 218 will send
the old macro block 308 from the frame memory 214 and use
it to update the display.
Figure 6 illustrates the interactions between the
front end communication systems and the host processor 218,
SUBSTITUTE SHEET



Wn 94/09595 ~ ~ ~ ~~ ~ ~ ~ ~ PCT/US92/08018
-29-
system memory 216, pixel processor 206, frame memory 214 and
display processor 212. These interactions are performed through
system bus 418. The incoming video sequence 602 is first
received by a front end demodulator 515. Network communica-
tions processor 202 and transmission processor 204 will decode
the control message and header information 606. The pixel
processor 206 and transform processor 210 will then transform
these sequences from frequency domain to pixel domain and
store same in the frame memory 214. The display processor
212 performs the appropriate interpolation to display the
output video sequence at the selected frame rate. Similarly,
the outgoing video sequence 603 can be prepared through coding
of the frame difference 442 for each macro block 477 to con-
vert from pixel to frequency domain. to transmit out through
front end modulators 514.
Once the incoming video sequence 602 is received
and stored in the buffer memory 518 the control message and
header 606 information will then be stored in a first in,
first out memory 428 for further decoding by the network com-
munications processor 202 and transmission processor 204.
A self-contained micro controller 608 could provide the frame
formatting 610, error processing 612 and protocol control
functions 524. This would provide service at low bit rate
SUBSTITUTE SHEET




WO 94/09595 PCT/US92/0801 R
_30_
applications up to 64 Kbs range. For higher speed applica-
tions 16 bit or 32 bit high performance embedded micro con-
trolleys could be employed.
Figure 7 illustrates a block diagram of the design
of the video codec and display subsystem 702 and its inter-
action with the transmission~processor 204 and host processor
218. The video codec and display subsystem 702 consists of
pixel processor 206, transform processor 210 frame memory
214 and display processor 212. Pixel processor 206 is the
host controller for the video codec and display sub-system
702. Pixel processor 206 is also the controller for the video
bus 422. Pixel processor 206 communicates with the host proces-
soy 218 through system bus 418 using its internal host inter-
face circuit 704. Pixel processor 206 also interconnects
to transmission processor 204 through a first ins first out
memory buffer 706 using its internal serial interface 708.
Pixel processor 206 interfaces and controls frame memory 214
through video bus 422 using its~internal PRAM controller circuit
434. Pixel processor 206 interfaces with motion processor
208 through video bus 422 and with display processor 212 through
private DP bus using its internal display processor decoder
714. The pixel processor 206 also interfaces with transform
processor 210 through first ins first out memory 707 and input
multiplexes 716.
SUBSTITUTE SHEET



wn 94/09595 PCT/US92/080i8
-31-
Pixel processor 206 is also required to perform
time critical pixel domain video coder and decoder functions
718. These include variable length coder and decoders run
level coder and decoderr quantization and dequantization,
zig-zag to raster or rastar to zig-zag scan conversion.
Since most video coding algorithms employ frame
differencing 450 techniques to reduce band widths only the
frame difference signals 442 will require to be coded and
decoded. Frame memory 214 is designed to store old frames
714 and new frames 712 at two discrete section. Old frame
714 being stored as the reference model while the difference
between the new and old frames are being updated via a dif-
ferencing signal 442 which will be either coded for transmis-
sion or decoded and added back to the old frame 714 for the
reconstruction of new frame 309.
As an encoders pixel processor 206 will retrieve
from the frame memory 214 these frame differencing signals
442 in macro blocks 477. Transform processor 210 will per-
form the DCT (discrete cosine transform) function 716 to trans-
late each of the Y~ U~ and Y block from pixel to frequency
domain. The pixel processor 206 will apply these discrete
cosine transforms to the decoder or encoder function before
SUBSTITUTE SHEET




WO 94/09595 PCT/US92/0801 R
-32-
forwarding the coded bit stream to the transmission processor
204 for transmission.
As a decoder, pixel processor 206 will retrieve
these frame difference bit streams 442 from the transmission
processor 204 first in, first out buffer 706, apply the de-
coding procedures. and then communicate with the transform
processor 210 through its input first in, first out buffer
707. Transform processor 210 will perform the inverse DCT
(discrete cosine transform) operation 485 to derive the pixel
domain values for each Y, U and Y block 471. These pixel
values will be stored in the transform processor output first
in, first out 710 until the pixel processor 206 retrieves
the old pixel block from frame memory 214. The signal dif-
ferential will then be forwarded to the pixel processor to
update the new values of Y, U and Y.
Transform processor 210 also performs matrix trans-
position 736, two-dimensional filter 738, matrix multiplica-
tion 740 and matrix addition 742. These are required since
whenever motion compensation techniques are applied, the old
frame 714 must be filtered before it can be added to the new
frame difference 442. Additionally, the Inverse (Discrete
Cosine Transform) 485 output must be transposed before
SUBSTITUTE SHEET


CA 02147164 2003-06-03
final addition. The coubie buffeted input 707 and output
714 first inr first out memories ant? the input multiplexer
716 are employed to allow the four stage pipeline required
for the discrete cosine transform operation. Additional. speed
may be obtained through the use of additional transform pipeline
processor 744 arranged in parallel.
Referring to Figure 8, as background to Applicant's
scalable memory array reconfigurable technique to be described
hereafterr an understanding of the CIF format 302 and QCIF
format 304 is necessary. These formats are designed for the
transportation of videc:.information over a telecommunication
netwcrk 480. 'Chey are c~:nunolly applied by international coding
algorithms such as CCITT H.261 298 and MPG 240 standards.
The CIF format v~02 consi~t~ of 352 pixels for each
horizontal scan line with 288 scar. lines on the vertical di-
mension. The CIF format 30:: is further partitioned into twelve
groups of blocks 482. Each group of block consists of 33
macro blocks X77 and each macro block consists of 9 Y blocks
474r 1 tJ block 473 and ,1. v block ~4?3 and each block consists
of 64 8-bit pixels.
The QCIF format 304 consists of 176 pixels for each
horizontal scan line with 114 scan lines on the vertical ai-
mansion. The QCIF format 3G9 is further partitionec into
three groups of blacks 482r each group of block =8~ consist
ins of 33 macro blacks 477 with, each macro block consistinc
of 4 Y blocks 474r I U block. 473 one i Y block 473.


CA 02147164 2003-06-03
- 34.-
Each macro block 4?? comprises 384 bytes of YLTY data
since the frame rate for CIF' format 302 is 30 fps (frame per
second) and each CIF format 302 frame consists of 396 macro
blacks. The band width reguired to send uncompressed CIA'
format 149 frames would be 4.6 mega bytes per second which
is the equivalent to a total of 576 channels of 64 Kbs B channels
Each QCIF format 304 has 99 macro blocks 47? and frame
updates at 7.5 fps. The system throughput requires 288 KBe
which is the equivalent of 36 channels of 64 KBs based B channels
802. Thereforer an uncompressed CIF format 302 frame transmitting
at 30 fps requires 24 T:x lease lines 804 and the QCIF format
349 transmitting at ?.5 fps requires i.5 T1 lines 804. As
suchr 75 micro seconds yxould be required to code an incoming
CIF format 304r 1.2 mil,~iseconds would be required for each
macro block at 7.S fps. ,
The CCITT H.251 standard 238 reguires a switch from
inter co intro fr~xme coding rro-~c~e 7_~''~P after every i32 frames of
transmission
in order to avoid accumulative error. This means that in
a ~0 fps transmission, r~very 4.4 seconds intro CIF format
30i frame coding will ba engaged and in QCIF format 304r at
,.5 f~>s, intro frame coding 12<'?8 will be engaged evexy,~ 11.6 seconds.
Ficure 9 is a schematic illustration of the scalable
memory array reconfigurable technique utilized by Applicant's
multimedia assembly 112 in order to optimize the performance
for encoding CIF format 30I. Tc achieve 3C- fps updates, the
time reouired to encode a macro bicck ~_ is 75 microseconds.
A single 6 x 8 DCf operation will consume 6.4 microseconds.
Since it takes 6 DCT operations to complete each 4Y:lU:lY



Wn 94/09595 ~ ~ PCT/US92/08018
-35-
block within a macro block 477, the time required for a single
hardware device to execute DCT transform coding will take
38.4 microseconds which would mean that there would only be
36.6 microseconds left for other time demanding tasks such
as motion estimation, variable length coding and quantization.
Although pipeline 902 and parallel processing 904
techniques can be applied to improve system performance such
as multiple DCT transform pipeline processors 744 can be cas-
cades in parallel as shown in Figure 7, this solution is not
acceptable for the consumer based mass market.
The scalable memory array reconfigurable technique
reduces the standard CIF format 302 to a modified CIF format
906 with slightly coarser resolution and yet retain all of
the integrity of the standard CIF format 302 and QCIF format
304. The scalable memory array has the option to choose between
the CIF format 302 or QCIF format 304.
The modified CIF format 906 provides a 288h X 192v
resolution 908 and the modified QCIF format 907 provides a
144h X 96v resolution 910. This provides close to the original
CIF and QCIF 302 and 304 quality respectively and also maintains
the 4:1:1 integrity of the YUV signal 471. Each CIF format
302 will still retain twelve (12) groups of blocks 482 and
each QCIF format 151 will still maintain three (3) groups
SUE3STiTUTE SHEET




WO 94/09595 PCT/US92/08018
_.
-36-
of block 482. The macro blocks 477 and pixel 912 format will
remain the same. The only difference is that each group of
block 482 will now consist of 18 macro blocks (9h X 2v) while
the original CIF format 302 group of blocks consisted of 33
macro blocks (llh X 3v).
This is accomplished during the input and output
color conversion process in that CCIR 601 image 916 input
which consists of 720h X 480v resolution can be downsampled
(5:2) 918 to the 288h X 192v Y resolution and further down-
sampled 5:1 920 to the 144h X 96v Ur V resolution. At the
output display, the Y, U, Y can perform 2:5 upsampling 922
for the Y and 1:5 upsampling 924 for the U and Y. The signi-
ficance of this modified CIF format 908 design is that the
internal processing performance requirement is reduced by
46% which means'we are now allowed to use slower and more
economical hardware for encoder processing. Meanwhile,
memory subsystems, such as frame memory 214 and first-in,
first-out memory 428, can employ slower memory devices that
reduce costs.
Secondly, scalable memory array 926 permits the
further scaling down of our modified CIF format 908 to meet
either application requirements or cost production require-
ments or to simply drop from a higher resolution format to
SUBSTITUTE St-IEET



Wn 94/09595 PCT/US92/08018
-37-
a coarser resolution format to meet the real time and coding
requirement. As an exampler the CIF frame format could be
implemented at 144h X 96v resolution and a QCIF frame format
in 72h X 48v resolution. Consequently, the multimedia as-
sembly 112 can employ the standard CIF format 302 or QCIF
format 304 when cost and performance are acceptable. In other
instances, the scalable memory array 926 would be adopted
so that the CIF and QCIF formats would be adapted as per the
following frame selection examples.
Mode CIF QCIF TYPE
1 352h x 288v 176h x 144v Standard
2 288h x 192v 144h x 96v Modified
3 144h x 96v 72h x 48v Modified
4 72h x 48v 36h x 24v Modified
36h x 29v 18h x 12v Modified
The scalable memory array also allows the parti-
tion of frame memory 214 into sections of modified frames
to allow multiple processes to run in each frame section.
As an example, a frame memory 214 of 352h X 288v size can
be scaled down to either a single 288h X 192v section: 4
144h X 98v sections; 16 72h X 48v sections; 64 36 x 24v
sections or any of the mixed combinations, all of the
sections being processed in parallel.
The scalable memory array can also provide remote
MPEG 240 video playback. Standard MPEG provides four
times the resolution improvement over the existing CCIR 601
S I.J ~ ~'~' I'~ lJ T E ~ E~ ~ E'f


CA 02147164 2003-06-03
_~J$~
standard. Namely, the standard MPEG 188 can provide 1440h X
960v resolution. The aigni~icance is now we are not only
able to run each memory section as a parallel process, but
we are also able to provide compatibility between the two
standards MPEG 240 ana H.261 238. Now, the MPEG standard
240 designed originally only to provide high resolution motion
video playback locally can now be useo to transmit compressed
MPEC; programs acrcass th~_: net=work 4B(. employing the widely availatle
H.261 video cadet fac~.lities. The scalable memory array also
enables the user to manage and provide the remote transmission
of MPEG 240 video programs employing conference controller
928, store and ~forwarc~ 930 ~snd video distribution 932.
It is therefore possible to either downsample a
compressed MPEG frame 24C into one of the modified CIF for-
mat 908 or simply send multiple compressed MPEG subframes
by partitiaa. For example, a 1440h X 960v MPEG frame 290
can downsample 5:1 into a 288h X 192v modified CIF frame 908
for transmission and decode and upsample at 1:5 to display
it at standard MPEG resalution at the corresponding output.
an exemplar the fol3owing frame formats could
bQ utilized to interchange between F:.261 238 and MPEG 240
standards.


CA 02147164 2003-06-03
_39_
Mode MPBG 0-MPEG TYPE


1 1A40h x g60v ?20h x ABOv Standard MPEG


2 1152h x ?58v 5?6h x 384v Modified MPEG


3 576h 384v 288h x 192v Modified MPEG
x


4 352h 268v I?6h x 144v Standard CIF/MPEG
x


288h 192v 144h x 96v Modified CIF/MPEG
x


6 144h 96v ?2h x 48v Modified CIF/MPEG
x


7 72h 48v 35h x 24v Modified CIF/MPEG
x


6 36h 24v 18h x 12v Modified CIF/MPEG
x


The scalable memory array formats have aignificanae
in that due to their compact size, they became useful in repre-
seating moving objects in the foreground when the background
information is ~till. The background Information Would be
pretransmitted during the intro frame coding mode 1228, whil a
the different moving objects would be transmitted during the
interframe coding mode =.~<<. Depending upon the size of the
moving objectr the appropriate size of the modified format
Will be employed. At the decoder endr the moving objects
will be overlayed witr~ the still background context to provide
motion sequence.
The scalable memory array is particularly suitable
to progressive encoding of images when band width needs to
be conserved. The scalable memory array Will choose the
coarser modified CIF format to transmit the initial frames
and then utilize a larger modified CIF format to send eub-
seguent frames ouch tt:at the complete image sequence will
gradually be upgraded to the original CIF quality.


CA 02147164 2003-06-03
The scalable memory array controller performs as
a result of the cooperation between pixel processor 206 and
host processor 218. ?ixel processor 206 is the local host
controller for the video codes and display subsystem 702 and
the host processor 216 is the global host controller for the
overall system. The pixel processor 206 serves a9 the bus
master for video bas 422 and host processor 218 serves as
the bus master for the system bus 918. Hoth the video bus
422 and the system bus 418 are system-wide parallel inter-
connects. Video bus 422 is speci~fiCally designed to facili-
tats the high speed vide~~ information transfer among subsystem
components.
Figure 10 illustratesa the Pixel processor 206
designing to meet the flexible performarsce for various types
of popular video coding algorithms such as the MPEGr H.261
or JPEG. Meanwhile, pixel processor 206 can also perform
other pixel domain-based proprietary methods. While most
pixel algorithms are eiG,her inter 1~:2'; or intro 1%28 frame codi.ngr
the CCITT and I50 standara algorithms (MPEGr JPEG and H.261)
are transformed domain coding methods employing fast DCT im-
plementation and inter frame differencina techniques. Addi-
tionally. MPEG and H.261 also apply motion compensation te~h-
niques.


CA 02147164 2003-06-03
-41.-
The pixel processor 206 is equipped with a 24 bit
address line 1002 to permit it to access 1B mega bytes of
program memory ~1a . 'The prvagram memory :.=9 can Further be: partitioned
into separate segments with each segment designated far a
specific coding algorithm. Since pixel processor 306 is micra-
programmable, it is relatively easy to update the changes
while MpEG 240, H.261 23fi and SPEC 244 standards are still
evolving.
The pixel processor 206 is alas de~siqned with par-
allel processing in mind.. The micro programmable architecture
allows multiple pixel processors 206 to couple over video
bus 420 to provide concurrent program execution for an
extremely high throughput. This wi~.l allow each pixel pro-
cessor 206 to be dedicated to-a coder 1008 function or a de-
code= 1010 function. If 6 pixel processors 206 axe employed,
this will allow the concurrent execution of an entire macro
block 477. Similarly the multiplicity of pixel processors
depending upon cost and size could permit the process of an
entire group of block 482 simultaneaualy.
The choice of host processor 218 is somewhat critical
in that it must be able to provide an interface ~rith the external
host 4~%.3 it must be able to execute the popular DO& 491


CA 02147164 2003-06-03
-42-
or UNIX program 490 suctv as word processing ar spread sheet
programs and it must be economical. .A suggested choice is
intei 80286 ar 80386 microprocessors. These provide a convenient
bus interface :L001 with t oe a'= bus w~u crv has suffi.ci.ert bus band
Width to be used as the system bus 418 of the system. The
aforesaid micro-processors also provide compatibility vith
a wide variety' of DOS 491 based software application programs.
Additionally, the small computer system interfacQ 488 is readily
available and capable of providing high speed interface between
the internal system bus and the external host ~8~.
Figure llris a schematic illustration of motion
processor 208 subsystems. Conforming to one of the H.261
coding options , motion processor 20E is designed to iden-
tify and specify a motion vec:tar _ , for each macro block
4?? within the existing luminance (Y) frame 474. The motion
vector 2~o for the color difference for (Uf v) frames 473
can then be derived as either 50% or Lhe truncated integer
value of the Y frame. The principle is that for each 16h
X 16v source macro block 110F3, the surrounding 4Bh X 48v area
1100 of updated new frame ?12 trill be needed to be searched
and compared. The new macra~block 477 having the least dis-
tortion Will be identified as the destination siacro block
1104 and the distance between the source an~i destination macro
block will be defined as. the motion vect.cr


CA 02147164 2003-06-03
-4~-
The direct implementation of motion processor 208
requires that fox each of the four blocks 1109 residing c~ithi.n
the old source macro block 1108 of the existing framer the
corresponding destination macro block 1174 centered within
the nerr frame must be identified. Therefore, every correspond-
ing, surrounding 6h X 6v area 1106 of blacks in the nev frame
must be searched and c:ompare<::i Hrlth ?~e olcsource mac-rc blocs 1108
reference
in order to derive the :best match with :least distortion.
This approach will require 589, 824 cycles of search and compare
operations. Provided the search anc3 compare operations can
be fully pipelines, an instruction cycle time of .13 nano
seconds is still requfred which is too time-consuming for
the 75 microsecond per ,macro block real time requirement at
38 fps updates.
In order to meet such real time performance re-
quirementsr t:he motion processor 206 must employ parallel
processing and mufti-processing techniques. The multimedia
assembly 112 incbrporates a dine grain, tightly coupledr paral-
lel pixel processor archa.tecture 1112 which provides faster
speed and better results.. 9'his is accomplished by partition-
ing existing macrp block ~17i' intc 4 8 X 8 blocks 1109. Four
parallel processing array' 1116 consisting of 24h X 29v pro-°
cessar elements are configurec into nine (°; zegaons. ThesE~


CA 02147164 2003-06-03
-44-
nine reg ions of macro processor elements 1114 are tightly
coupled together. ~ach region of the existing f tame can have
direct interconnection and simultaneous access to its eight
(8) nearest neighl~ori»g regions from the corresponding new
frame. Each region of macro processing elements 1114 is des-
ignated to perform various types of pixel domain processing
functions for the 8 X Es block extracted from the old source
saacro block 1108.
Figure 11 illustrates a parallel search method for
8 X 8 blocks residing within the old source macro block 1108.
~ach can conduct simultaneous match and compare operations
with all of their nine neatest neighboring blocks. The out-
puts of the nine matching operations are first locally stored
at the corresponding regional pixel processor arrays 1116.
They are then shifted out and summed at the output accumula-
tor 1118 and adder circuits 1120. The results are then compared
using the comparetor circuit. 1122 to obtain the beat match.
The physical distance between the new macro block which result9
in the best match and the old reference macro block will be
applied as the motion vec::to~ 270 for thF,~ olcx l.umi.naruce nacro bloc:K.
The regional ,pixel procenaor array 1116 can be recon-
figurable and is designed based upon nine banks of processor
element arrays 1126. Each processor element array 882 consists


21~"~~64
WO 94/09595 PCT/US92/08018
-45-
of sixty-four processor elements 1128. The nine banks of
processor element arrays 1126 are interconnected through shift
registers 1130 and switches 1132. In a three-dimensional
implementations a vertically-cascaded processor array 1138
crossbar switch array 1134 and shift register array 1136 can
be implemented. Additional layer such as storage array can
be added to provide the additional functionality. This array
will be extremely powerful when multi-layered packaging becomes
available for the chip level modules and intergrated circuit
technologies.
A two-dimensional pixel processor array 1116 can
also be designed using nine banks of processor element arrays
1126 equipped with peripheral switches 1132 and shift registers
1130. The switches 1132 can be reconfigurable to guide direction
about the date of flow where the shift registers 1130 can
transfer data from any processor element array 1126 or input
to any other~processor element array 1126 or output. Both
switches 1132 and shift registers 1130 are byte wide to facili-
tate parallel data flow. The processor element arrays 1126
were designed based upon an 8 X 8 array of simple processor
elements 1128.
The processor element arrays 1126 are designed for
interconnection among the processor elements so that recon-
figuration can be accomplished to meet different application
SUBSTITUTE SHEET


CA 02147164 2003-06-03
-46-
needs. The processor elements 1128 are designed so that each
can be programmed to execute ~imple instructions. ~ach pro-
cessor element 1128 consists of a simple ALL 1190 which can
execute simple instructions such as add, subtract, load, store,
compare, etc.
Figure 12A illustrates the design example of a program-
enable logic device 1201. which employs a cellular array logic
architecture. This figure i~ used to demonstrate the function-
ality and physical design of the device. The practical size
far an N X N array is dependent upon the application re9uire-
menta and the state of the art of the implementing technology.
Figure 128 illustrates the practical implementation
of a cellular logic processor element 1204 using a charge
couple device 970 technology. The objective is to provide
an intergrated itaage sensor array 1205 with the digital pre-
processing capabilities so that image coding for the macro
bloc k 977 and pixel doma~_~ image coding funrtir_~ns can be performed.
The other objective is to allow the implementation of on-chip
parallel image sensing and parallel image processing 976 utili-
zing the same or compatible technology. The cellular array
logic architecture illustrated in Figure 12H are useful
that they can implement fine grain, tightly-coupled parallel
processing systems. They employ aingle_instruction-multiple-


CA 02147164 2003-06-03
_LF%_
data 1206 or multiple-instruction-multiple-data 1210 techniques
to provide system throughput where traditional xequential
computing fails.
Many cellular array processors have been designed
in the past. lost of them employ a processor array which
consists of a matrix of processor elements 1328 and awiteh
arrays ii3~ which can prcvide prograrnmai~le v=raerconnect '!20 net-
works among the processor elements. ThQSe cellular array
processors are extremely expensive.
The design illustrated in Figure 12a is based upon
a mush simpler architecture, the design being dedicated only
to image processing and coding appliaatione. The major ob-
jective is to meet real time performance requirements for
macro block 97 ; pixel dom<:ain p~:~ocessing .functions or motior_
processing.
Figure 12h is employed to demonstrate how frame
differencing functions can be performed for each of the in-
coming sub-image macro blocks 477. For illustration, 3 X
3 array iq used to represent macro block sub-image 477 uhichr
from the current framer is first shifted into the processor
element; the corresponding saacra block sub-image of the previous
frame 1218 is then loaded into the processor element and the
comparison functions are performed between the two macro blocks 9~%


CA 02147164 2003-06-25
", ~ ~j...
to detect if there is any frame difference. Provided the
difference is larger than the preset threshold value. the
macro blocks will be marked and the macro block marker 1242
and macro block difference 1244 between the two frames will
be stored in frame memory 2I8. If there is no difference,
the current fra~ne'macro blockwalue 1216 will be deleted and
the old source macro block value 1108 will be used for
display updates.
If an exc~raaiue number of macro blocks 47? ere
identified with frame differ~nce, then a. scene or illumina-
tion change 273 has o<::curred an~:: m~rcra b:lr.:~c,k. pxoce:~ssor 1.20 will
notify heat processor and pixel groce,aerar 206 and suitc~h the
operation from interframe coding 1227 to intraframe coding
1228. The significance fe that while incoming images sensed
from the camera 465, the spec;if~c macro i~.ice.ks with the .frame
differencing can be identified and stared. Consequentlyr
in the interframe coding modes 122?, only those macro blocks
4?7 requiring motion estimation anti compensation 1222, trans-
form coding 1229 or quantization 1226 will k~e marked and stored
in the frame memory 214 to regrea~rzat the image sequence of
the current frame. In the case of scene or illumination changes,
enough macro blocks will be detected with frame differencing
that the system will automatically switch to intraframe ceding
mode 1228.



WO 94/09595 PCT/US92/08018
-49-
Figure 12B illustrates additional pixel domain pro-
cessing functions including low pass filtering 1230: high
pass filtering 1232 and quantization 1226. The variable
quantization 1226 can be performed by presetting the threshold
value .1234 and then shifting and quantisizing the corresponding
transform domain coefficients based upon the zig-zag scan
format at each low, medium and high frequency regions. The
threshold value can be reprogrammed to adjust the quantization
level. The advantage is that as soon as the input image is
detected, sampled and thresholded, several pixel domain prepro-.
cessing functions, such as frame differencing and motion estima-
tion, can be performed right away. The differencing macro
blocks will be sent to transform processor 210 to perform
DCT operation 1224, the output of the DCT coefficients can
further be reloaded into the processor element array to perform
quantization. When band width reduction control 260 is required,
initial thresholding is combined with a coarser quantization
1226 level to reduce the image resolution. When the system
demands faster performance, multiple parallel processor element
arrays can be cascaded to perform concurrent macro block operations
such as frame differencing, motion processing and quantization.
SUBSTITUTE S1-~~~T


CA 02147164 2003-06-03
S
J ~°
The advantage of charge couple device technology
1202 is its suitability for image processing, multiplexing,
and storage 431 operatio~is. This can be done both i.ri the analog
and digital domain. Therefore, depending upon the applica-
tion reguirement, both analog processing 1238, digital pro-
cessing 1240 and memory functions using these processor ele-
ment arrays 1126 can be accomplished.
1
Figure 13 is .a schematic illustration of the func-
tional model architecture ire order to simplify the functional
groaesses covered out by the hardware previously discussed.
The principal functional elements comprise a band width manager
1304. a formatter 1302. a pixel-domain-codes encoder 1304
coupled with 3 pixel-domain-codes decoder 1306, a transform-
domain-codes encoder 1308 coupled with a transform-domain-
codes decoder I3I0, a netwar~-domain-codes encoder 1312 coupled
with a network-domain-codes decoder 1319 and a controller
1316.
The band width manager 1300 provides band width
control, capability Wherein a two-dimensional band width-over-
lay-lookup-table (80LUT~ can be constructed to map the specific
band width ranges, i.e.,~ 2.4 Kbs to 100 Mbs, et a1, into selective
options of media combina~t,iona such as overlay in the audior
video, text and graphics with various types of quality and
resolution.


CA 02147164 2003-06-03
-51.-
Additionally, during noisy communication environ-
manta, the band Width manager 1300 function is to constant-
ly monitor tree network ~:13G tc; deteo~ abrupt: network band width
changes caused by local line degradation or netWOrk traffic
congestion. The band viath~ manager 1300 will respond by ad-
jesting the media combinations to accommodate the available
band width.
fluting stable communication environment, band width
manager 2300 operates to reconfigure the different band widths
spec_.fied by i=he network 9~G providing upqradabili:,~Y~ and paral-
lelism for time-sharinr~.
The formatter 130 communicates with the band width
manager 1300 to ascertain the band width availability for
incoming or outgoing signals. The formatter translates this
external information into an interna3ly-operating format.
The scalable memory array reconfigurable technique will re-
configure the internal proceswer and frame memory structure
pursuant to the directions of the formatter. Thin allor~s
the external format to be translated into a suitable internal
format to provide system compatibility. The scalable-memory-
array-reconfigurable-technique (SMART) as discussed is Figure
3 is capable of translating a programmable internal format


CA 02147164 2003-06-03
_5p_
in compliance with ~_. wide vr=.sty of international standard and
custom video coding algori.t~:rns such as MPEG, Ei.261, JFEG and
vector quantization while H.a?61 performs in rnecro blc:ck 4'~7, MPEG
performs in slice 299. Farrnvatt:er 1307;= identifies the transmit-
ting or receiving coding algorithms. derives their specific
format requirements and if these external format requirements
are different from the current internal formats the formatter
reformats the horizontal and vertical resolutian which result:
in a separate internal format which is compatible with the
external format. These internal format operations, such as
the reduction of the horizontal and vertical resolution, are
performed by employing interpolation and dawnsampling tech-
nigues or upsampling techniques. The formatter 1302 also
communicates with the frame memory so that the frame memory
is aware of the internal format to be stared. This allows
the formatter :1302 in cor7junction with the scalable memory
array configurable technique to formulate a scalable pro-
censor and frame memory architecture so that the internal
processor and frame memory can be continually adjusted in
order tc reconfigure or modify a suitable internal format
for any type of external format either being received or
sent by the netwdrk-domain-co~dec 131,
The network-domain-rodeo encoder 1312 and decoder
1314 are used to provide Line coding and decoding functions.


CA 02147164 2003-06-03
-5 3-
NetWOrk domain codes decoder 1314 world receive network trans-
missions via its front end transceiver 1320. It would then
perform protocol procedures 1322, network communication pro-
cedures 1324, variable length coding 1325, run length coding
1328 and filtering 1330. The resultant transform coefficients
and gixel date. will then be forwarded to either pixel-domain,-
codec decoder 3.308 or transform-domain-codes decoder 1310.
The network-domain-codes, encodQr 1312 would receive encoded
pixel data or transform coefficients from the other encoders
and convert them into serial codes for network transmission
performing functions similar to the network domain codes decoder
1314. Simultaneously, band width manager l3aQ will interface
with encoder 1312 and decoder 1314 tc exchange protocol con-
trol and applications information regarding band width avail-
ability.
The pixel-domain-codes encoder 1304 and decoder
130fi are designed for c~ratom coding algorithms such as vector
quantization, pixel domain aperation5 for the DCT transform
based standard coding algorithms such as MPEG, et al. pixel
domain operations for moti::n compensation 122a and image post-
procesaing functions and analysis and preprocessing techniaues
for video coding, thus, the pixel-domain-codes provides for
pixel domain preprocessing, 1331, pixel domain coding 1334,




WO 94/09595 PCT/US92/08018
_.
image processing 1336 color space conversion 1338 pixel
interpolation 1340 vector quantization 1342 and color lookup
mapping 1344.
The transform-domain-codec encoder 1308 and decoder
1310 are specifically designed for forward and inverse trans-
formation operations required by the international standard
coding algorithms such as MPEG,.et al. Transform-domain-codec
encoder and decoder 1308 and 1310 also provide forward and
inverse transform-based operations such as Harr transform
and Hadamard transform. Additionally generic matrix opera-
tions and post-matrix operations such as scan conversion,
quantization and normalization techniques are performed by
the transform-domain-codec.
The controller 1316 comprised of either a single
or plurality of local host processors which manage the in-
struction sequencing and system control functions for data
transfers memory management input/output interfacing and
processor pipelining.
In Figure 4, we demonstrated a host processor used
to manage the communications pipelines the network domain
codec and the system memory. It also performed general ad-
ministrative tasks and controlled the system bus and access
to other subsystem buses while communicating with the band
width manager 1300


CA 02147164 2003-06-03
~5.5-
A second controller is a single or plurality of
pixel processors used to manage the video pipelines the
scalable memory array reconfigurable technique, frame
memories, formatters and display processing. Additional-
ly. the pixel processor is used to perform pixel-domain-codes
encoding and decoding functions and can Ge used in multiples
in order to facilitate macr-o ~~.lock 97"1 Gnd grcup of block 1006 process-
ing. Similarly, a single o:c plurality of transform processors
can be employ$d as coprocessor for the pixel processors.
in performing transforr.~-domain--codes encoding and decoding
functions.
Ali network 980 transmissions or receivin<J functions
would first pass through the network-domain-codes and then
be directed to the pixel-domain-codes or transform-domain-
codes after suitable formatting. The media information could
then be displayed via the pixel-domain-codes decoder 1306.
Origination signa:Ls from either storage 981, camera 465, TV 109 or CD player
103
s~ould be subjected to frame differencing 1364 and frame image
capture 1366 before being encoded by pixel-domain-codes en-
coder 1304. ThQSe origination signals could then be trans-
mitted via network-domain-codes, encoder 1312 dependent upon
the band width manager 1300 and controller 1360 monitoring
of band width availability.




WO 94/09595 PCT/US92/08018
-S6-
While the invention has been described with reference
to its preferred embodiment thereof it will be appreciated
by those of ordinary skill in the art that various changes
can be made in the process and apparatus without departing
from the basic spirit and scope of the invention.
SUE3STITUTE SI"I~~T

Representative Drawing
A single figure which represents the drawing illustrating the invention.
Administrative Status

For a clearer understanding of the status of the application/patent presented on this page, the site Disclaimer , as well as the definitions for Patent , Administrative Status , Maintenance Fee  and Payment History  should be consulted.

Administrative Status

Title Date
Forecasted Issue Date 2005-12-20
(86) PCT Filing Date 1992-10-16
(87) PCT Publication Date 1994-04-28
(85) National Entry 1995-04-13
Examination Requested 1997-10-16
(45) Issued 2005-12-20
Expired 2012-10-16

Abandonment History

There is no abandonment history.

Payment History

Fee Type Anniversary Year Due Date Amount Paid Paid Date
Application Fee $0.00 1995-04-13
Maintenance Fee - Application - New Act 2 1994-10-17 $50.00 1995-04-13
Maintenance Fee - Application - New Act 3 1995-10-16 $50.00 1995-09-28
Maintenance Fee - Application - New Act 4 1996-10-16 $50.00 1996-09-05
Maintenance Fee - Application - New Act 5 1997-10-16 $75.00 1997-09-18
Request for Examination $200.00 1997-10-16
Maintenance Fee - Application - New Act 6 1998-10-16 $75.00 1998-09-29
Maintenance Fee - Application - New Act 7 1999-10-18 $75.00 1999-10-04
Maintenance Fee - Application - New Act 8 2000-10-16 $75.00 2000-09-20
Maintenance Fee - Application - New Act 9 2001-10-16 $75.00 2001-07-19
Maintenance Fee - Application - New Act 10 2002-10-16 $100.00 2002-07-17
Maintenance Fee - Application - New Act 11 2003-10-16 $100.00 2003-07-25
Maintenance Fee - Application - New Act 12 2004-10-18 $125.00 2004-10-15
Final Fee $150.00 2005-09-26
Maintenance Fee - Application - New Act 13 2005-10-17 $125.00 2005-10-07
Maintenance Fee - Patent - New Act 14 2006-10-16 $125.00 2006-10-05
Maintenance Fee - Patent - New Act 15 2007-10-16 $225.00 2007-10-04
Maintenance Fee - Patent - New Act 16 2008-10-16 $225.00 2008-09-26
Maintenance Fee - Patent - New Act 17 2009-10-16 $225.00 2009-08-27
Maintenance Fee - Patent - New Act 18 2010-10-18 $225.00 2010-10-12
Maintenance Fee - Patent - New Act 19 2011-10-17 $225.00 2011-09-20
Owners on Record

Note: Records showing the ownership history in alphabetical order.

Current Owners on Record
SHAW, VENSON M.
SHAW, STEVEN M.
Past Owners on Record
None
Past Owners that do not appear in the "Owners on Record" listing will appear in other documentation within the application.
Documents

To view selected files, please enter reCAPTCHA code :



To view images, click a link in the Document Description column. To download the documents, select one or more checkboxes in the first column and then click the "Download Selected in PDF format (Zip Archive)" or the "Download Selected as Single PDF" button.

List of published and non-published patent-specific documents on the CPD .

If you have any difficulty accessing content, you can call the Client Service Centre at 1-866-997-1936 or send them an e-mail at CIPO Client Service Centre.


Document
Description 
Date
(yyyy-mm-dd) 
Number of pages   Size of Image (KB) 
Claims 2002-06-11 8 411
Drawings 1994-04-28 14 450
Representative Drawing 1998-02-11 1 16
Description 2003-06-03 56 1,805
Drawings 2003-06-03 19 626
Claims 2003-06-03 12 489
Drawings 2003-06-25 19 660
Claims 2003-06-25 12 495
Description 2003-06-25 56 1,825
Description 1994-04-28 56 1,792
Claims 1998-02-16 8 354
Cover Page 1995-08-10 1 15
Abstract 1994-04-28 1 55
Claims 1994-04-28 27 937
Claims 2004-06-18 26 1,042
Description 2004-06-18 66 2,305
Description 2004-11-22 66 2,305
Claims 2004-11-22 26 1,046
Claims 2005-03-04 26 1,027
Representative Drawing 2005-11-22 1 13
Cover Page 2005-11-22 1 46
Prosecution-Amendment 2004-08-31 2 57
Assignment 1995-04-13 6 178
PCT 1995-04-13 16 399
Prosecution-Amendment 1997-10-16 33 1,269
Prosecution-Amendment 2002-02-12 2 42
Prosecution-Amendment 2002-06-11 10 510
Prosecution-Amendment 2002-12-03 4 121
Prosecution-Amendment 2003-06-03 64 2,272
Prosecution-Amendment 2003-06-25 10 433
Prosecution-Amendment 2003-12-22 3 94
Prosecution-Amendment 2004-06-18 39 1,594
Prosecution-Amendment 2004-11-22 17 670
Correspondence 2005-02-09 1 2
Prosecution-Amendment 2005-03-04 14 536
Correspondence 2005-09-26 1 38
Fees 1996-09-05 1 72
Fees 1995-09-28 1 42
Fees 1995-04-13 1 55