Note: Descriptions are shown in the official language in which they were submitted.
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
REMOTE NETWORK MANAGEMENT SYSTEM
1 FIELD OF THE INVENTION
2 The present invention relates generally to a remote network management
system
3 for remotely controlling network and computer equipment from one or more
local user
4 workstations through a remote control device. Specifically, a keyboard,
video monitor,
and cursor control device attached to a user workstation are utilized to
remotely control
6 domain servers, file/print servers, headless servers, network appliances,
serial IT
7 equipment, switches, routers, ~rewalls, security interfaces, application
servers, load
8 balancers, and environmental controls as their associated power supplies are
connected to
9 a remote control device.
11 BACKGROUND OF THE INVENTION
12 In many situations, it is desirable to manage networking equipment,
servers, and
13 computers located at a location remote from the system administrator. If
the distance is
14 great enough, the Internet is commonly utilized to control computers from a
remote
location. For example, a software program such as pcAnywhere may be utilized
to
16 access a remote computer over the Internet or a LAN utilizing the keyboard,
video
17 monitor, and cursor control device attached to a local user workstation.
Remolte
18 computer access programs, such as pcAnywhere, typically require that host
software is
19 installed on the remote computer and client software is installed on the
user workstation.
To access a remote computer, a user of the user workstation selects the
desiredt remote
21 computer from a list and enters the appropriate username and password. Once
access has
22 been granted to the remote computer, the user utilizes the keyboard, video
monitor, and
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 cursor control device attached to the local user workstation to access and
operate the
2 remote computer.
3 Hardware solutions also exist for operating a remote computer from a user
4 workstation over the Internet or via a modem. In contrast to software
solutions, hardware
solutions do not typically require host and/or client software. Instead,
hardware solutions
6 typically utilize a keyboard, video monitor, and mouse ("KVM") switch which
is
7 accessible over the Internet or LAN via a common protocol, such as TCP/IP.
The
8 hardware solutions may also utilize a modem to connect to the Internet.
Generally, a user
9 or system administrator accesses the remote computers attached to the KVM
switch
utilizing an Internet web-browser or client software associated with the KVM
switch.
11 Once the remote computer has been selected, the remote computer's video
signal is
12 routed to the user workstation's video monitor and a user may then utilize
a keyboard
13 and/or mouse to control the remote computer. The KVM switch may
additionally include
14 a connection to the power source of the remote computer for a hard reboot
in case of
system failure.
16 The aforementioned hardware and software solutions generally utilize
17 compression algorithms to reduce the necessary bandwidth required to
transmit the video
18 signals. For example, the remote network management system of the present
invention
19 uses the compression algorithm disclosed in application serial no.
10/233,299, which is
incorporated herein by reference, to reduce and compress the digital data that
must be
21 transmitted to the remote computers and/or video display devices.
Generally, video
22 signals generated by a personal computer have both spatial and interframe
redundancies.
23 For example, in a near idle personal computer, the only change between
successive
-2 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 frames of video might be the blinking of a cursor. Even as a user types a
document, a
2 majority of the screen does not change over a period of time. Hence, the
compression
3 algorithm used by the present invention takes advantage of these
redundancies, both
4 between successive frames of video and within each individual frame, to
reduce the
amount of digital video signal data that is transmitted to the remote
computers and/or
6 video display devices. Reducing the amount of digital data transmitted over
the
7 communication medium decreases communication time and decreases the required
8 bandwidth.
9 Most forms of video compression known in the art require complicated
calculations. For example, Moving Pictures Experts Group ("MPEG") video
11 compression algorithms use the discrete cosine transform as part of its
algorithm. Also,
12 the MPEG standard relies on the recognition of "motion" between frames,
which requires
13 calculation of motion vectors that describe how portions of the video image
have changed
14 over a period of time. Since these algorithms are calculation intensive,
they either require
expensive hardware or extended transmission times that allow sufficient time
for slower
16 hardware to complete the calculations.
17 In addition to complexity, many existing video compression techniques are
lossy
18 (i.e., they do not transmit all of the video signal information in order to
reduce the
19 required bandwidth). Typically, such lossy techniques either reduce the
detail of a video
image or reduce the number of colors utilized. Although reducing the number of
colors
21 could be part of an adequate compression solution for some computer
management
22 systems applications, in many other applications, such a result defeats the
intended
23 purposes of the computer management system.
-3 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 A system known in the art discloses a method and apparatus for coupling a
local
2 user workstation, including a keyboard, mouse, and/or video monitor, to a
remote
3 computer. The claimed invention discloses a system wherein the remote
computer is
4 selected from a menu displayed on a standard size personal computer video
monitor.
Upon selection of a remote computer by the system user, the remote computer's
video
6 signals are transmitted to the local user workstation's video monitor. The
system user
7 may also control the remote computer utilizing the local user workstation's
keyboard and
8 monitor. The system is also capable of bi-directionally transmitting mouse
and keyboard
9 signals between the local user workstation and the remote computer. The
remote
computer and the local user workstation may be connected either via the Public
Switched
11 Telephone System ("PSTN") and modems or via direct cabling.
12 Another system known in the art discloses a specific implementation of a
13 computerized switching system for coupling a local keyboard, mouse and/or
video
14 monitor to one of a plurality of remote computers. In particular, a first
sig~lal
1 S conditioning unit includes an on-screen programming circuit that displays
a list of
16 connected remote computers on the local video monitor. To activate the
menu, a user
17 depresses, for example, the "print screen" key on the local keyboard. The
user selects the
18 desired computer from the list using the local keyboard and/or mouse.
19 According to the system, the on-screen programming circuit requires at
least two
sets of tri-state buffers, a single on-screen processor, an internal
synchronization
21 generator, a synchronization switch, a synchronization polarizes, and
overlay control
22 logic. The first set of tri-state buffers couples the red, green, and blue
components of the
23 video signals received from the remote computer to the video monitor. That
is, when the
-4-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 first set of tri-state buffers are energized, the red, green, and blue video
signals are passed
2 from the remote, computer to the local video monitor through the tri-state
buffers. When
3 the first set of tri-state buffers are not active, the video signals from
the remote computer
4 are blocked. Similarly, the second set of tri-state buffers couples the
outputs of the single
on-screen processor to the video monitor. When the second set of tri-state
buffers is
6 energized, the video output of the on-screen programming circuit is
displayed on the local
7 video monitor. When the second set of tri-state buffers is not active, the
video output
8 from the on-screen programming circuit is blocked. Alternatively, if both
sets of tri-state
9 buffers are energized, the remote computer video signals are combined with
the video
signals generated by the on-screen processor prior to display on the local
video monitor.
11 The on-screen programming circuit disclosed in the invention also produces
its
12 own horizontal and vertical synchronization signals. To dictate which
characters are
13 displayed on the video monitor, the CPU sends instructional data to the on-
screen
14 processor. This causes the on-screen processor to retrieve characters from
an internal
video RAM for display on the local video monitor.
16 The overlaid video image produced by the on-screen processor, namely a
17 Motorola MC 141543 on-screen processor, is limited to the size and quantity
of colors and
18 characters that are available with the single on-screen processor. In other
words, the
19 system is designed to produce an overlaid video that is sized for a
standard size computer
monitor (i.e., not a wall-size or multiple monitor type video display) and is
limited to the
21 quantity of colors and characters provided by the single on-screen
processor.
22 During operation of the system, a remote computer is chosen from the
overlaid
23 video display. Thereafter, the first signal conditioning unit receives
keyboard and mouse
-5 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 signals from the local keyboard and mouse and generates a data packet for
transmission
2 to a central cross point switch. The cross point switch routes the data
packet to the
3 second signal conditioning unit, which is coupled to the selected remote
computer. The
4 second signal conditioning unit then routes the keyboard and mouse command
signals to
the keyboard and mouse connectors of the remote computer. Similarly, video
signals
6 produced by the remote computer are routed from the remote computer through
the
7 second signal conditioning unit, the cross point switch, and the first
signal conditioning
~ unit to the local video monitor. The horizontal and vertical synchronization
video signals
9 received from the remote computer are encoded on one of the red, green or
blue video
signals. This encoding reduces the quantity of cables required to transmit the
video
11 signals from the remote computer to the local video monitor_
12 Yet another system discloses a keyboard, video, mouse ("I~VM") switching
13 system capable of coupling to a standard network (e.g., a Local Area
Network) operating
14 with a standard network protocol (e.g., Ethernet, TCP/IP, etc.). The system
couples a
central switch to a plurality of computers and at least one user station
having a keyboard,
16 video monitor, and mouse. The central switch includes a network interface
card ("NIC")
17 for connecting the central switch to a network, which may include a number
of additional
1 ~ computers or remote terminals. Utilizing the system, a user located at a
remote terminal
19 attached to the network may control any of the computers coupled to the
central switch.
Still another system known in the art discloses a computer system having
21 remotely located Il0 devices. The system of Thornton includes a computer, a
first
22 interface device, and a remotely located second interface de~rice. The
first interface
23 device is coupled to the computer and the second interface device is
coupled to a video
-6 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 monitor and as many as three I/O devices (e.g., keyboard, mouse, printer,
joystick,
2 trackball, etc.) such that a human interface is created. The first and
second interface
3 devices are coupled to each other via a four wire cable. The first interface
device
4 receives video signals from the connected computer and encodes the
horizontal and
vertical synchronization signals of the received video signals onto at least
one of the red,
6 green, and blue components of the video signal. The first interface device
also encodes
7 the I/O signals received from the connected computer into a data packet for
transmission
8 over the fourth wire in the four wire cable. Thereafter, the encoded, red,
green, and blue
9 components of the video signals and the data packet are transmitted to the
second
interface device located at the human interface. The second interface device
decodes the
11 encoded red, green, and blue components of the video signal, separates the
encoded
12 horizontal and vertical synchronization signals, and decodes the I/O signal
data packet.
13 The video signal and the synchronization signals are then output to the
video monitor
14 attached to the second interface and the decoded I/O signals are routed to
the proper I/O
device, also attached to the second interface. The second interface device may
optionally
16 include circuitry to encode I/O signals received from the I/O devices
attached to the
17 second interface for transmission to the first interface device.
18 A still different system discloses a keyboard, video, mouse, and power
switching
19 ("KVMP") apparatus for connecting a plurality of computers to one or more
user stations
having an attached keyboard, video monitor, and mouse. On screen display
("OSD")
21 circuitry embedded within the KVMP switching apparatus allows a user
located at a user
22 station to select and operate any one of the computers utilizing the
keyboard, video
23 monitor, and mouse attached to the user station. Secondary switching
circuitry located
_7_
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 within the KVMP switching apparatus allows a user located at a user station
to
2 additionally control the electrical power supply supplying each computer.
3 In view of the foregoing, a need clearly exists for a self contained remote
network
4 management system capable of operating and controlling networking equipment,
servers,
and computers connected to a remote control switching unit. Furthermore, such
a system
6 should allow a user to control the power supply attached to the remote
networking
7 equipment, servers, and computers. The system should aid in managing remote
network
8 environments, thereby reducing the need to have an on-site system
administrator.
9
SUMMARY OF THE INVENTION
11 The present invention provides a self contained remote network management
12 system for administrating a remote computer networking environment from one
or more
13 local user workstations with attached peripheral devices (i.e., keyboard,
video monitor,
14 cursor control device, etc.). The remote network management system of the
present
invention allows a user located at a user workstation to access, operate, and
control
16 networking equipment, servers, and computers located at a remote location.
The remote ,
17 network management system also allows a user to control the power supply to
each piece
18 of remote equipment. The networking equipment (e.g., hubs, switches,
routers, etc.) is
19 typically controlled via a serial interface. In contrast, servers and
computers are
controlled and operated utilizing a keyboard, video monitor, and mouse.
21 The remote networking equipment, servers, and computers are all connected
to a
22 central remote management unit ("RMU"), and in turn, the RMU is connected
to the
23 Internet or a LAN via an Ethernet or modem connection. The RMU has serial
ports for
_g _
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 connection to the networking equipment as well as keyboard, video, and
cursor control
2 device ports for connection to the servers and computers. The RMU
additionally
3 contains a port for connection to a power supply capable of controlling the
power to the
4 networking equipment, servers, and computers. Standard cabling is utilized
to connect
the networking equipment, servers, and computers to the appropriate ports on
the RMU.
6 The RMU also provides compatibility between various operating systems and/or
7 communication protocols, including but not limited to, those manufactured by
Microsoft
8 Corporation ("Microsoft") (Windows), Apple Computer, Inc. ("Apple")
(Macintosh), Sun
9 Microsystems, Inc. ("Sun") (Solaris), Digital Equipment Corporation ("DEC"),
Compaq
Computer Corporation ("Compaq") (Alpha), International Business Machines
("IBM")
11 (RS/6000), Hewlett-Packard Company ("HP") (HP9000) and SGI (formerly
"Silicon
12 Graphics, Inc.") (IRIX).
13 To utilize the remote network management system of the present invention, a
user
14 first initiates a management session by utilizing client software located
on a user
workstation to connect to the RMU. Alternatively, the user may utilize an
Internet
16 browser to connect to the RMU. The user is then prompted by the RMU to
provide a
17 user name and a password. The RMU is capable of storing multiple profiles
and different
18 levels of access for each profile. Once a user has been authenticated, the
user is provided
19 an option menu on the user workstation's monitor produced by option menu
circuitry
located in the RMU. The option menu consists of a menu listing all the
networking
21 equipment, servers, and computers at the remote location. The option menu
additionally
22 contains a menu allowing a user to control the power to each piece of
remote equipment.
23 The user selects the desired networking equipment, server, or computer by
utilizing the
-9 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 keyboard and/or cursor control device attached to the user workstation. Once
a user
2 makes a selection, the user is provided access to the remote equipment as if
the user is
3 physically located at the remote site.
4 The RMU and the user workstation communicate via TCP/IP. Bef~re
transmission via TCP/IP, the unidirectional video signals (i.e., from the
Rl~;~U to the user
6 workstation) are digitized by a frame grabber. This circuit captures video
output from the
7 initiating computer at a speed of at least 20 frames/second and converts the
captured
8 analog video signals to a digital representation of pixels. Each pixel is
digitally
9 represented with 5 bits for red, 5 bits for green, and 5 bits for blue. The
digital
representation is then stored in a raw frame buffer. The compression algorithm
then
11 processes the digital data contained in the raw frame buffer. The
compression algorithm
12 is actually a combination of four sub-algorithms (i.e., the Noise Reduction
and Difference
13 Test ("NRDT"), Smoothing, Caching, and Bit Splicing/Compression sub-
algorithms) as
14 described in greater detail below.
After the video signals have arrived at the user workstation, decompression
16 occurs. The user workstation operates as a decompression device by
executing a
17 decompression algorithm. Along with any transmitted video or data signals,
the RMU
18 transmits messages to the decompression devices regarding the portions of
the video that
19 yielded "cache" hits (i.e., portions of unchanged video). In response, the
decompression
device constructs the video frame based upon the transmitted video signals and
the blocks
21 of pixels contained in its local cache. Also, the decompression device
updates its local
22 cache with the new blocks of pixels received from the RMU. In this manner,
the
23 decompression device caches remain synchronized with the compression device
cache.
-10-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 Both the compression device and the decompression device update their
respective cache
2 by replacing older video data with newer video data.
3 Furthermore, the video signals transmitted by the RMU have been compressed
4 using a Iossless compression algorithm. Therefore, the decompression device
(e.g.,
software on the user workstation) must reverse this Iossless compression. This
is done by
6 identifying the changed portions of the video image, based upon flags
transmitted by the
7 RMU. From this flag information, the decompression device is able to
reconstruct full
8 frames of video.
9 In addition, the decompression device converts the video frame to its
original
color scheme by reversing a color code table ("CCT") conversion. The
decompression
11 device, like the RMU, locally stores a copy of the same CCT used to
compress the video
12 data. The CCT is then used to convert the video data received from the RMU
to a
13 standard RGB format that may be displayed on the monitor attached to the
user
14 workstation.
1 S The decompression algorithm can be implemented in the remote network
16 management system of the present invention in a variety of embodiments. For
example,
17 in one embodiment, it can be implemented as a software application that is
executed by
I8 the user workstation. In an alternate embodiment, the decompression
algorithm can be
19 implemented to execute within a web browser such as Internet Explorer or
Netscape°
Navigator". Such an embodiment eliminates the need for installation of
application
21~ specific software on the user workstation. Also, this embodiment allows
the RMU to
22 easily transmit the video signals to any user workstation with Internet
capabilities,
23 regardless of the distance at which the computer is located from the
initiating computer.
-11-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 This feature reduces the cabling cost associated with the remote network
management
2 system of the present invention.
3 Since the present invention can be used to display video signals at
locations that
4 may be at a great distance from the RMU, it is important to ensure that the
video signal
transmission is secure. If the transmission is not secure, hackers,
competitors, or a~her
6 unauthorized users could potentially view confidential information contained
within the
7 video signals. Therefore, the remote network management system of the
present
8 invention is designed to easily integrate with digital encryption techniques
known in the
9 art. In one embodiment of the present invention, a 128-bit encryption
technique is used
both to verify the identity of the RMU and to encrypt and decrypt the
transmitted ~deo
11 and data signals. In this embodiment, a 128-bit public key RSA encryption
technique is
12 used to verify the remote participant, and a 128-bit RC4 private key
encryption is used to
13 encrypt and decrypt the transmitted signals. Of course, other encryption
techniques or
14 security measures may be used.
Finally, since the remote network management system of the present inventnon
16 allows for platform independent communications, the compression algorithm
utiliaed
17 does not employ operating system specific hooks, nor does it use platform
specific GDI
18 calls.
19 In the preferred embodiment, the compression algorithm described herein and
in
co-pending application serial no. 10/233,299 is used to transmit the video
signals.
21 However, the video transmission system is not limited to such an
embodiment. Rather,
22 this system may be employed with any compression algorithm without
departing fnam the
23 spirit of the invention.
-12-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 Therefore, it is an object of the present invention to provide an improved,
remote
2 network management system that enables a user to control a remote networking
3 environment from one or more Local user workstations. Such a remote
networking
4 environment may include domain servers, file/print servers, headless
servers, network
appliances, serial IT equipment, switches, routers, firewalls, security
interfaces,
6 application servers, load balancers, and environmental controls.
7 Further, it is an object of the present invention to provide a remote
network
8 management system that allows one or more local user workstations to access
and
9 operate remote networking equipment, servers, and computers connected to a
remote
management unit.
11 It is another object of the present invention to provide a single, platfonn-
12 independent remote network management system offering centralized,
integrated, and
13 secure control.
14 It is an additional object of the present invention to provide a network-
1 S independent remote network management system containing a modem for
emergency
16 access.
I7 It is a further object of the present invention to provide a remote network
18 management system capable of BIOS-level control of KVM equipment and
console-level
I9 control of serial devices.
Additionally, it is an object of the present invention to provide a remote
network
21 management system which provides a single consolidated view of all servers
and other
22 connected devices from one screen via a web browser.
-I3-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 It is another object of the present invention to provide a remote network
2 management system which contains a single sign-on and interface.
3 Additionally, it is an object of the present invention to provide a remote
network
4 management system which is upgradeable.
It is a further object of the present invention to provide a remote network
6 management system which provides high performance over low bandwidth
connections
7 including modem, wireless, cable, DSL, and fractional T1.
8 It is another object of the present invention to provide a remote network
9 management system which utilizes a video compression algorithm and frame-
grabber
technology to ensure the fastest possible transmission of high quality video.
11 Furthermore, it is an object of the present invention to provide a remote
network
12 management system including built-in serial port buffering to provide views
of recent
13 console history.
14 It is still a further object of the present invention to provide a remote
network
management system that is easy to install and operate.
16 In addition, it is an object of the present invention to provide a remote
network
17 management system that is compact and provides readily accessible
communications
18 ports.
19 Further, it is an object of present invention to provide a remote network
management system, which allows error-free communications between peripheral
21 devices of a local user workstation and networking equipment, servers, and
computers
22 located at domain servers, file/print servers, headless servers, network
appliances, serial
_I4-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 IT equipment, switches, routers, firewalls, security interfaces, application
servers, load
2 balancers, and environmental controls.
3 It is also an object ofthe present invention to provide a remote network
4 management system capable of controlling the power supply to remotely
located
networking equipment, servers, and computers.
6 Other objects, features, and characteristics of the present invention, as
well as the
7 methods of operation and functions of the related elements of the structure,
and the
8 combination of parts and economies of manufacture, will become more apparent
upon
9 consideration of the following detailed description with reference to the
accompanying
drawings, all of which form a part of this specification.
11
12 BRIEF DESCRIPTION OF THE DRAWINGS
13 A further understanding of the present invention can be obtained by
reference to a
14 preferred embodiment set forth in the illustrations of the accompanying
drawings.
Although the illustrated embodiment is merely exemplary of systems for
carrying out the
16 present invention, both the organization and method of operation of the
invention, in
17 general, together with further objectives and advantages thereof, may be
more easily
18 understood by reference to the drawings and the following description. The
drawings are
19 not intended to limit the scope of this invention, which is set forth with
particularity in
the claims as appended or as subsequently amended, but merely to clarify and
exemplify
21 the invention.
22 For a more complete understanding of the present invention, reference is
now
23 made to the following drawings in which:
-15-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 FIG. 1 is a schematic representation of a remote network management system
2 according to the preferred embodiment of the invention illustrating the
connection of a
3 user workstation that includes a keyboard, video monitor, and cursor control
device to
4 networking equipment, servers, and computers through a remote management
unit
("RMU").
6 ' FIG. 2 is a screen-shot of an example option menu utilized to control the
7 networking equipment, servers and computers.
8 FIG. 3A is a block diagram of the preferred embodiment of the RMU shown in
9 FIG. 1 according to the preferred embodiment of the present invention
illustrating the
internal structure of the RMU and connectors for serial devices, keyboards,
video
11 monitors, cursor control devices, and a power supply.
12 FIG. 3B is a detailed block diagram of the serial card shown in FIG. 3A.
13 FIG. 3C is a detailed block diagram of the KVM port header shown in FIG.
3A.
14 FIG. 3D is a detailed block diagram of the video processor shown in FIG.
3A.
FIG. 4 depicts a flowchart of the compression algorithm utilized by the
preferred
16 embodiment of the RMU in accordance with the present invention.
17 FIG. 5A depicts a flowchart detailing the Noise Reduction and Difference
Test
18 and smoothing sub-algorithms of the compression algorithm utilized by the
preferred
19 embodiment of the present invention.
FIG. 5B depicts a flowchart that details the caching and bit
splicing/compression
21 sub-algorithms of the compression algorithm utilized by the preferred
embodiment of the
22 present invention.
-16-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 FIG. 6 depicts a flowchart that details the nearest match function and its
2 integration with the CCT of the compression algorithm utilized by the
preferred
3 embodiment of the present invention.
4 FIG. 7 depicts a flowchart that details the Noise Reduction and Difference
Test
sub-algorithm of the compression algorithm utilized by the preferred
embodiment of the
6 present invention.
7 FIG. 8 depicts an example application of the Noise Reduction and Difference
Test
8 sub-algorithm to a sample block of pixels as performed by the compression
algorithm
9 utilized by the preferred embodiment of the present invention.
FIG. 9 depicts a detailed flowchart of the operation of the decompression
11 algorithm used by the preferred embodiment of the present invention.
12
13 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
14 As required, a detailed illustrative embodiment of the present invention is
disclosed herein. However, techniques, systems and operating structures in
accordance
16 with the present invention may be embodied in a wide variety of forms and
modes, some
17 of which may be quite different from those in the disclosed embodiment.
Consequently,
18 the specific structural and functional details disclosed herein are merely
representative,
19 yet in that regard, they are deemed to afford the best embodiment for
purposes of
disclosure and to provide a basis for the claims herein which define the scope
of the
21 present invention. The following presents a detailed description of the
preferred
22 embodiment (as well as some alternative embodiments) of the present
invention.
-17-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 Referring first to FIG. 1, depicted is the architecture of the preferred
embodiment
2 of a remote network management system in accordance with the present
invention.
3 Specifically, a remote network management system is shown comprising user
4 workstation 101 including keyboard 103, video monitor 105, and cursor
control device
107, remote management unit ("RMU") 109, Internet/LAN/WAN 108, public switched
6 telephone network ("PSTN") 106, serial devices 11 la and 11 1b, servers 1
I3a and 113b,
7 remote computers 115a and 115b, and power supply 117. Preferably, user
workstation
8 101 and RMU I09 are connected. to Internet/LAN/WAN 108 via communication
lines
9 119 and 12I, respectively. Although CAT 5 cabling is the preferred cabling
for
communication lines 119 and 121, other cabling may be used, such as coaxial,
fiber optic
11 or multiple CAT 5 cables. CAT 5 cabling is preferred because it reduces
cabling cost
12 while maintaining the strength of signals that are transmitted over an
extended distance.
13 Alternatively, wireless networking equipment may also be utilized to
connect RMU 109
14 to Internet/LAN/WAN 108 and sexial devices l l la and l l 1b, servers 113a
and 1 I3b,
computers 115a and 115b, and power supply 117. Similarly, wixeless networking
16 equipment may also be utilized to connect user workstation 101 to
Tnternet/LAN/WAN
17 108.
18 In an alternate embodiment, user workstation 101 may utilize PSTN 106 to
19 connect to RMU I09. If PSTN 109 is utilized to connect to RMU 109,
communication
lines I20 and 122 would preferably be CAT 3 cables. As an example, this means
of
21 communication may be utilized in emergency situations, such as if
Internet/LAN/WAN
22 108 is not functioning properly.
-18-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 Communication lines 119 and 121 are connected to user workstation 101 and
2 RMU 109 by plugging each end into a RJ-45 socket located on the respective
pieces of
3 equipment to be coupled by the CAT 5 cable. Although RJ-45 sockets and plugs
are
4 preferred, other types of connector may be used, including but not limited
to RJ-11, RG-
58, RG-59, British Naval Connector ("BNC"), and ST connectors.
6 The remote management system includes local user workstation 101, preferably
7 comprising dedicated peripheral devices such as keyboard 103, video monitor
105 and/or
8 cursor control device 107. Other peripheral devices may also be located at
workstation
9 101, such as a printer, scanner, video camera, biometric scanning device,
microphone,
etc. Each peripheral device is directly or indirectly connected to user
workstation 101,
11 which is attached to Internet/LAN/WAN 108 via communication line 119. Of
course,
12 wireless peripheral devices may also be used with this system. In a
preferred mode of
13 operation, all electronic signals (i.e., keyboard signals and cursor
control device signals)
14 received at user workstation 101 from attached peripheral devices are
transmitted to
Internet/LAN/WAN 108 via communication line 119. Thereafter, the signals are
16 transmitted to RMU 109 via communication line 121. RMU transmits the
received
17 signals to the respective remote equipment, which, in this figure, includes
serial devices
18 l l la and 11 1b, servers 113a and 113b, computers 115a and 115b, and power
supply 117.
19 RMU 109 may be compatible with all commonly used, present day computer
operating systems and protocols, including, but not limited to, those
manufactured by
21 Microsoft (Windows), Apple (Macintosh), Sun (Solaris), DEC, Compaq (Alpha),
IBM
22 (RS/6000), HP (HP9000) and SGI (IRIX). Additionally, local devices may
communicate
23 with remote computers via a variety of protocols including Universal Serial
Bus ("USB"),
-19-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 American Standard Code for Information Interchange ("ASCII") and Recommend
2 Standard-232 ("RS-232").
3 Serial devices 1 I3a and 113b are connected to RMU 109 via communication
lines
4 112a and 1 I2b, respectively. Preferably, communication lines 112a and 112b
are CAT 5
cables ternninated with RJ-45 connectors. However, a special adapter may be
required to
6 properly connect communication lines 112a and 112b to serial devices 111 a
and 11 1b
7 since not all serial devices are outfitted with RJ-45 ports. For example, if
serial device
8 11 la only contained a serial port, the adapter would interface the RJ-45
connector of
9 communication line 112a to the serial port located on serial device 11 Ia.
Similarly, power supply 117 is connected to RMU 109 via communication line
11 118. Preferably, communication line 118 is a CAT 5 cable terminated with an
RJ-45
I2 connector on each end.
13 Servers 113a and 113b and computers 115a and 1 ISb are connected to RMU 109
14 via communication lines 114a, 1 I4b, I 16a, and I I6b, respectively.
Preferably,
communication lines I 14a, 1 I4b, I 16a, and I 16b are three-to-one coaxial
cables which
I6 allow the keyboard, video, and cursor control device ports of servers 113a
and 113b and
17 computers I 15a and l I5b to be connected to a single port on RMU I09 as
shown.
i
18 To connect to the remote networking environment for administration and
access, a
19 user initiates a remote management session at user workstation 1 O1. The
user first
accesses client software located using workstation 1 Ol, which prompts the
user for a user
21 name and password. However, the system may utilize any combination of
identification
22 data to identify and/or authenticate a particular user. Utilizing the
attached keyboard 103,
23 cursor control device 107 or other peripheral device, the user enters the
user name and
-20 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 password. Once the user name and password have been entered, user
workstation Z O 1
2 connects to Internet/LAN/WAN 108 via communication line 119. User
workstation 10l
3 may connect to Internet/LAN/WAN 108 in a variety of ways. Fox example, user
4 workstation 101 may be connected to Internet/LAN/WAN 108 through an Ethernet
connection. In this example, communication line 119 would be a CAT 5 cable.
The
6 connection to Internet/LAN/WAN 108 may also be accomplished through a
wireless
7 connection which precludes the need for communication line 119. For example,
RMU
8 109 may utilize standard Wireless Fidelity ("Wi-Fi") networking equipment to
9 communicate with Internet/LAN/WAN 108.
Alternatively, user workstation 101 may connect to RMU 109 via PSTN 106 by
11 utilizing a modern connection. In this alternative example, communication
lines 120 and
12 122 would be CAT 3 cables.
13 The username and password are then routed through Internet/LAN/WAN I08 to
14 RMU 109 via communication line 12I . RMU I09 receives the username and
password
and authenticates the user located at user workstation 101. Once the user has
been
16 authenticated by RMU 109, an option menu circuit located in RMU 109
provides an
I 7 option menu to user 101 via monitor 105 listing all the devices accessible
through RIVIU
18 I09. The user makes selections from this option menu utilizing keyboard
103, cursor
19 control device 1 O5, or some other peripheral device attached to user
workstation 101.
As shown in FIG. 2, option menu 201 consists of device list 203, first desktop
21 window 205, power control window 207, second desktop window 209, and serial
device
22 window 211. Device list 203 lists all active and inactive devices connected
to RMU 109.
23 A user utilizes this menu to select the desired device for control. In this
example, first
-21-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 desktop window 205 displays the desktop of one of the remote computers. By
selecting
2 first desktop window 205, a user may utilize keyboard 103, cursor control
device 107, or
3 some other peripheral device to control the displayed remote computer. In a
similar
4 manner, a user may utilize power control window 207 to access and operate
power
supply 117. Power control window 207 displays a list of all devices connected
to power
6 supply 117 as well as the status of each attached device such as average
power utilized,
7 RMS current, RMS voltage, internal temperature, etc. Power control window
207 is
8 primarily utilized to cycle the power to the devices attached to power
supply 117.
9 However, since power supply 117 is programmable, power control window 207
may be
utilized to perform any functions possible with power supply 117.
11 Second desktop window 209 is utilized to access and operate a second remote
12 computer or server. Serial device window 211 is utilized to operate and
access any
13 remote serial device attached to remote management unit 109. Serial device
window 211
14 displays the current output produced by the serial device as well as the
previous output
produced by the serial device. The previous output of the serial device is
stored in a
16 buffer located in RMU 109.
17 Preferably, option menu 201 consists of a menu in which the attached
devices are
18 arranged by their connection to RMU 109. For example, serial devices 111 a
and 111 b
19 preferably would be listed in a menu different from servers 113a and 113b
and computers
11 Sa and 11 Sb. The option menu also consists of a sub-menu for controlling
power
21 supply 117.
22 RMU 109 may additionally contain an attached keyboard 123, cursor control
23 device 125, and video monitor 127 which allow a user local to RMU 109 to
control the
-22
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 attached serial devices 11 la and 1 I 1b, servers 113a and 113b, and
computers 115a and
2 115b, power supply 117, etc. Keyboard 123, cursor control device 125, and
video
3 monitor 127 may also be utilized to configure RMU 109 locally. Keyboard 123,
cursor
4 control device I25, and video monitor 127 are connected to RMU 109 via
interface cable
129. Alternatively, keyboard 123, cursor control device 125, and video monitor
127 may
6 be connected to RMU 109 via standard keyboard, cursor control device, and
video
7 monitor connectors.
8 Referring next to FIG. 3A, depicted is the preferred embodiment of RMU 109
9 according to the present invention. Keyboard and mouse signals arrive at RJ-
45 port 201
from Internet/LAN/WAN 108 via communication line 121. RMU 109 consists of RJ-
45
11 port 201, RJ-11 port 202, Ethernet connector 205, modem module 204,
communications
12 port connector 206, CPU 207, communications port connector 208, PCI riser
card 209,
13 serial card 211, video processor 212, serial ports 213, frame grabber 2I 5,
KVM port
14 header 217, KVM ports 219, power supply 221, power port 223, reset
circuitry 225, local
KVM port 227, and option menu circuit 229. As shown, the keyboard and/or
cursor
16 control device signals initially arrive at RJ-45 port 201 if RMU 109 is
connected to
17 Internet/LAN/WAN I08 via an Ethernet connection. The signals are then
transmitted to
18 Ethernet connector 205 which depacketizes the signals. Alternatively, the
signals may
19 arnve from PSTN 106 at RJ-11 port 202 if the keyboard and/or cursor control
device
signals were transmitted via a modem. In this case, the signals are
transmitted to modem
21 module 204, which demodulates the received signals, and subsequently to
22 communications port connector 206 which depacketizes the signals.
-23 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 From Ethernet connector 205 or communications port connector 206, the
2 keyboard and/or cursor control device signals are then transmitted to CPU
207 via video
3 processor 212. CPU 207 utilizes routing information contained within the
keyboard
4 and/or cursor control device signals to determine the proper destination for
the keyboard
and cursor control device signals. If the keyboard and cursor control device
signals
6 specify a command to power supply 117, CPU 207 interprets the received
command
7 (e.g., utilizing a look-up table) and sends the proper command to power
supply 117 via
8 communications port connector 208 and power port 210. Preferably, power port
210 is
9 an RJ-45 connector to allow the RMU to interface with a power strip and
control it as if it
were a serial device.
11 If CPU 207 determines that the keyboard and cursor control device signals
12 contain a serial device routing instruction, the keyboard and cursor
control device signals
13 are transmitted to serial card 21 I through PCI riser card 209. As shown in
FIG. 3B, serial
14 port 2I 1 consists of UART/switch 301, serial transceivers 303, and
programmable
memory 305. Serial card 211 is capable of bidirectional signal transmission.
When
16 keyboard and/or cursor control device signals are being transmitted from
PCI riser card
I7 209 to serial port 213, the signals are initially transmitted to
UART/switch 301 which,
18 utilizing data and logic stored in memory 305, determines the proper serial
transceiver
I9 303 to which the keyboard and/or cursor control device signals are to be
sent. In the
preferred embodiment of serial card 211, UART/switch 301 is an EXAR XR17c158.
21 Subsequently, the analog signals are transmitted to the appropriate serial
transceiver 303
22 which converts the signals from a parallel format to a serial format.
Serial transceiver
_24 _
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 303 is preferably a HIN23E serial transceiver from Intersil. The keyboard
and/or cursor
2 control device signals are then transmitted to serial port 213.
3 In contrast, when commands from serial device 11 la or 11 1b are transmitted
to
4 CPU 207 via serial port 213, serial card 211, and PCI riser card 209, the
commands are
initially transmitted to serial transceiver 303 which converts the serial
commands to a
6 parallel format. Subsequently, the commands are transmitted to UART/switch
301 which
7 re-transmits the commands to CPU 207 via PCI riser card 209. CPU 207
interprets the
8 received commands and emulates a virtual terminal for display on video
monitor 105.
9 The present invention may incorporate any number of serial ports 213. In the
example
shown in FIG. 3A, two serial devices, 1 l la and 1 l 1b, are connected to
serial ports 213a
11 and 213b, respectively.
12 If CPU 207 determines that the keyboard and/or cursor control device
signals are
13 Tllcat7l iv1 .,~;!'vel'a I I y p?!i 1 1 ~~1~ nr ~nt>~ ??te,os 1 1 Ju aiiu i
i W , t_ t'L~ ~v i ii ~ttioitUt~ ii if,
14 keyboard and cursor control device signals through PCI riser card 209 and
frame grabber
215 to KVM port header 217 which transmits the signals to the appropriate KVM
port
16 2I9. As shown in FIG. 3C, KVM port header 217 consists of switch 350, video
switch
17 352, and UARTs 354. When keyboard and/or cursor control device signals are
18 transmitted from KVM port 219 to KVM port header 217, the signals are
initially
19 received at UART 354. UART 354 converts the received serial keyboard and/or
cursor
control device signals to a parallel format. The converted keyboard and/or
cursor control
21 device signals are then transmitted to switch 350 which retransmits the
signals to frame
22 grabber 215.
-25 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
I In a similar manner, bi-directional keyboard and/or cursor control device
signals
2 are also transmitted from frame grabber 215 to KVM port 219. Keyboard and/or
cursor
3 control device signals received from frame grabber 215 are transmitted to
switch 350
4 located in KVM port header 217. Utilizing control signals contained within
the keyboard
and/or cursor control device signals, switch 350 transmits the received
keyboard andlor
6 cursor control device signals to the appropriate DART 354. DART 354 then
converts the
7 keyboard and/or cursor control device signals from a parallel format to a
serial format for
8 transmission to KVM port 219.
9 KVM port header 217 also transmits uni-directional video signals received at
KVM port 219 to frame grabber 215. The analog video signals received from KVM
port
11 219 initially are transmitted to video switch 352. Video switch 352 then
retransmits the
12 video signals to frame grabber 215 which converts the received analog video
signals to a
13 digital format. ' .. . ..,..,".,;,: ...~~c ~ ...a ! <: , y.! : . ,
14 Turning to FIG. 3D, after the video signals have been digitized by frame
grabber
215, the digitized video signals are transmitted to video processor 212 via
CPU 207.
16 Video processor 212 consists of video-in port 370, R-out 376a, G-out 376b,
B-out 376c,
17 pixel pusher 378, frame buffers 380, compression device 382, flash memory
384, RAM
18 386, microprocessor 388, and switch 390. Shown at the top of FIG. 3D, video-
in port
19 370 receives the digitized video signals from CPU 207. The outputs of video-
in port 370
are shown as R-out 376a, G-out 376b, and B-out 376c, which represent the red
2I component, green component, and blue component of the digitized video
signal,
22 respectively. Video-in port 370 outputs these digitized video signal
components in the
23 form of pixels, which are transmitted to and stored in pixel pusher 378.
Pixel pusher 378,
-26 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 flash memory 384, and Random Access Memory ("RAM") 386 communicate with
2 microprocessor 388 via communication bus 387. Pixel pusher 378 also
communicates
3 with frame buffers 380 (e.g., raw frame buffer, compare frame buffer,~etc.)
and
4 compression device 382 via communication buses 379 and 381, respectively.
The
compression algorithm is executed by microprocessor 388. Generally, the
compression
6 operates as follows:
7 Noise Reduction and Difference Test:
8 As discussed above, digitization of the analog video signals is necessary to
allow
9 these signals to be transmitted via a digital communication medium (e.g., a
network,
LAN, WAN, Internet, etc.). However, a detrimental side effect of the
digitization process
11 is the introduction of quantization ezrors and noise into the video
signals. Therefore, the
12 Noise Reduction and Difference Test sub-algorithm ("NRDT sub-algorithm") is
designed
13 to reduce the noise introduced during the digitization of the video
signals. In addition,
14 the NRDT sub-algorithm simuiiai~c:;~~'.determines vh a differences between
the recently
captured frame of video (i.e., the "current frame") and the previously
captured frame of
16 video (i.e., the "compare frame").
17 First, the NRDT sub-algorithm divides the current frame, which is contained
in
18 the raw frame buffer, into 64 x 32 blocks of pixels. Alternatively, other
sizes of blocks
19 may be used (e.g., 8x8 pixels, 16x16 pixels, 32x32 pixels, etc.) based upon
criteria such
as the size of the entire video frame, the bandwidth of the communication
medium,
21~ desired compression yield, etc.
-27-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 After the current frame is divided into blocks, a two-level threshold model
is
2 applied to the block of pixels to determine whether it has changed with
respect to the
3 compare frame. These two thresholds are the "pixel threshold" and the "block
threshold."
4 First, a given pixel is examined and the value of each of the three colors
(i.e., red,
green, and blue) of the pixel is calculated with the value of its
corresponding pixel in the
6 compare frame. From this calculation, a distance value is computed. If the
distance
7 value is greater than the pixel threshold (i.e., the first threshold of the
two-level
8 threshold), this distance value is added to a distance sum. This process is
performed for
9 each pixel in the block.
Next, after the distance value of all of the pixels in the block have been
calculated
11 and processed in the aforementioned manner, the resulting value of the
distance sum is
12 compared to the block threshold (i.e., the second threshold of the two-
Ievel threshold). If
13 the distance sum exceeds the block threshold, then this block of pixels is
considered
14 changed in comparison to the corresponding block of pixels in the compare
frame. If a
1 S change is determined, the compare frame, which is stored in the compare
frame buffer,
16 will be updated with the new block of pixels. Furthermore, the new block of
pixels will
17 be further processed and transmitted in a compressed format to the user
workstation.
18 In contrast, if the distance sum is not greater than the block threshold,
the block of
19 pixels is determined to be unchanged. Consequently, the compare frame
buffer is not
updated, and this block of pixels is not transmitted to the user workstation.
Eliminating
21 the transmission of unchanged blocks of pixels reduces the overall quantity
of data to be
22 transmitted, thereby increasing transmission time and decreasing the
required bandwidth.
-28 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 The NRDT sub-algorithm is ideal for locating both a Large change in a small
2 quantity ofpixels and a small change in a Large quantity of pixels.
Consequently, the
3 NRDT sub-algorithm is more efficient and more accurate than known percentage
4 threshold algorithms that simply count the number of changed pixels in a
block of pixels.
With such an algorithm, if a few pixels within the block of pixels have
changed
6 drastically (e.g., from black to white), the algorithm would consider the
block of pixels to
7 be unchanged since the total number of changed pixels would not exceed the
percentage
8 threshold value. This result will often lead to display errors in the
transmission of
9 computer video.
Consider, for example, a user that is editing a document. If the user were to
11 change a single letter, such as changing an "E" to an "F", only a few
pixels of the video
12 image would change. However, based upon this change, the resulting document
is
13 dramatically different than the original document. A percentage threshold
algorithm
14 would not register this change and, therefore, would Lead to a display
error. A percentage
threshold algorithm, by only looking at the number of pixels within a block
that have
16 changed, generally fails to recognize a video image change in which a few
pixels have
I 7 changed substantially. However, the NRDT sub-algorithm used by the present
invention,
I 8 by virtue of its two-level threshold, will recognize that such a block of
pixels has
19 significantly changed between successive frames of video.
Smoothing:
2I When the NRDT sub-algorithm determines that a block of pixels has changed,
the
22 digital data that represents this block is further processed by a smoothing
sub-algorithm.
23 This sub-algorithm reduces the noise introduced during the analog-to-
digital conversion.
-29-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 First, each digital pixel representation is converted to a representation
that uses a
2 lower quantity of bits for each pixel. It is known in the art to compress
color video by
3 using a fewer number of bits to represent each color of each pixel. For
example, a
4 common video standard uses 8 bits to represent each of the red, green, and
blue
components of a video signal. Because 24 total bits are used to represent a
pixel, this
6 representation is commonly referred to as "24 bit RGB representation". If
only the four
7 most significant bits of the red, green, and blue components of the pixel
are used to
8 represent its color in lieu of all eight bits, the size of the data used to
represent the block
9 of pixels, and thus a frame of video, is reduced by fifty percent.
This method of compression is simple and generally degrades the quality of the
11 video. In contradistinction, the smoothing sub-algorithm of the present
invention
12 incorporates a more intelligent method of compression. This method uses a
Color Code
13 Table ("CCT") to map specific RGB representations to more compact RGB
14 representations. Both the compression and decompression algorithms of the
present
invention use the same CCT. However, different color code tables may be chosen
16 depending on the available bandwidth, the capabilities of the local display
device, etc.
17 For each block of pixels, a histogram of pixel values is created arid
sorted by
18 frequency such that the smoothing sub-algoritlun may determine how often
each pixel
19 value occurs. Pixel values that occur less frequently are compared to pixel
values that
occur more frequently. To determine how similar pixel values are, a distance
value is
2I calculated based upon the color values of the red, green, and blue ("RGB")
components
22 of each pixel. During the histogram analysis, a map of RGB values to color
codes (i.e., a
23 CCT) is created. If a less frequently occurring pixel value needs to be
adjusted to a
-30-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 similar, more frequently occurnng pixel value, the CCT is used to map the
less frequently
2 occurring pixel value to the color code of the more frequently occurring
pixel value.
3 Thus, the noise is efficiently removed from each block and the number of
bits ~.ised to
4 represent each pixel is reduced.
For illustrative purposes, suppose that an 8x8 pixel block is being processed.
6 Further suppose that of the 64 pixels in the current block, 59 are blue, 4
are red, and 1 is
7 light blue. Further assume that a low frequency threshold of 5 and a high
frequency
8 threshold of 25 are used. In other words, if a pixel value occurs less than
5 times within a
9 block, it is considered to have a low frequency. Similarly, if a pixel value
occurs more
than 25 times within a block, it is considered to have a high frequency. In
the preferred
11 embodiment of the present invention, the smoothing sub-algorithm ignores
pixel values
12 occurnng between these two thresholds. Therefore, in the present example,
the
13 smoothing sub-algorithm determines that the red and light blue pixels occur
with low
14 frequency, and the blue pixels occur with high frequency.
I S In the next step, the values of the 4 red pixels and the 1 light blue
pixel are
16 compared with the value of the 59 blue pixels. In this step, a pre-
determined distance
17 threshold is used. If the distance between the less frequent pixel value
and the more
I 8 frequent pixel value is within this distance threshold, then the less
frequent pixel value is
I9 converted to the more frequent pixel value. Therefore, in our present
example, it is likely
that the light blue pixel is close enough in value to the blue pixel that its
distance is less
21 than the distance threshold. Consequently, the light blue pixel is mapped
to the blue
22 pixel. In contrast, it is likely that the distance between the red and blue
pixels exceeds
23 the distance threshold and, therefore, the red pixel is not mapped to the
blue pixel. With
-31-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 the smoothing sub-algorithm of the present invention, although the red
pixels occur
2 rarely, the distance between the red pixel value and the blue pixel value is
large enough
3 that the red pixels are not converted to blue pixels. In this manner, the
smoothing sub-
4 algorithm of the present invention increases the redundancy in compared
images by
eliminating changes caused by superfluous noise introduced during the analog-
to-digital
6 conversion while retaining real changes in the video image.
7 Caching:
8
9 After the smoothing sub-algorithm has been applied to the digital video
image
data, an optional caching sub-algorithm may be applied to further minimize the
11 bandwidth required for transmitting the video images. The caching sub-
algorithm uses a
12 cache of previously transmitted blocks of pixels. Similar to the NRDT sub-
algoritlun, the
13 caching sub-algorithm is performed on a bloclc of pixels within the video
frame. Again,
14 any block size may be used (e.g., 8x8, I6xI6, 32x32 or 64x32).
First, the caching sub-algorithm performs a cache check, which compares the
16 current block of pixels with blocks of pixels stored in the cache. The size
of the cache
17 may be arbitrarily large. Large caches generally yield a higher percentage
of "cache
18 hits." However, memory and hardware requirements increase when the size of
the cache
19 is increased. Furthermore, the number of comparisons, and thus the
processing power
requirements, also increases when the size of the cache increases.
2I A "cache hit" occurs when a matching block of pixels is located within the
cache.
22 A "cache miss" occurs if a matching block of pixels is not found in the
cache. When a
23 cache hit occurs, the new block of pixels does not have to be
retransmitted. Instead, a
24 message and a cache entry identification ("ID") are sent to the remote
participant
-32
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 equipment. Generally, this message and cache entry ID will consume less
bandwidth
2 than that required to transmit an entire block of pixels.
3 If a "cache miss" occurs, the new block of pixels is compressed and
transmitted to
4 the user workstation. Also, both the RMU and user workstation update their
respective
cache by storing the new block of pixels in the cache. Since the cache is of
limited size,
6 older data is overwritten. One skilled in the art is aware that various
algorithms can be
7 used to decide which older data should be overwritten. For example, a simple
algorithm
8 can be employed to overwrite the oldest block of~pixels within the cache,
wherein the
9 oldest block is defined as the least recently transmitted block.
In order to search for a cache hit, the new block of pixels must be compared
with
11 all corresponding blocks of pixels located within the cache. There are
several ways in
12 which this may be performed. In one embodiment, a cyclic redundancy check
("CRC"j is
13 computed~for the new block of pixels and all corresponding blocks of
pixels. The CRC is
14 similar to a hash code for the block. A hash code is a smaller, yet unique,
representation
of a larger data source. Thus, if the CRCs are unique, the cache check process
can
16 compare CRCs for a match instead of comparing the whole block of pixels. If
the CRC
17 of the current block of pixels matches the CRC of any of the blocks of
pixels in the
18 cache, a "cache hit" has been found. Because the CRC is a smaller
representation of the
19 block, less processing power is needed to compare CRCs. Furthermore, it is
possible to
construct a cache in which only the CRCs of blocks of pixels are stored at the
remote
21 participant locations. Thus, comparing the CRCs in lieu of comparing a full
block of
22 pixels saves processor time and thus improves performance.
23 Bit Splicing/Compression:
24
-33-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 Once the NRDT, smoothing, and optional caching sub-algorithms are performed,
2 each block of pixels that must be transmitted is compressed. In the
preferred
3 embodiment of the present invention, each block is compressed using the
Joint Bi-level
4 Image Group ("JBIG") lossless compression algorithm.
The JBIG compression algorithm was designed for black and white images, such
6 as those transmitted by facsimile machines. However, the compression
algorithm utilized
7 by the present invention can compress and transmit color video images.
Therefore, when
8 utilizing the JBIG compression algorithm, the color video image must be bit-
sliced, and
9 the resulting bit-planes must be compressed separately.
A bit plane of a color video image is created by extracting a single bit from
each
11 pixel color value in the color video image. For example, if 8 bits axe used
to represent the
12 color of the pixel, then the color video image is divided into 8 bit
planes. The
13 compression algorithm, in conjunction with the CCT discussed above,
transmits the bit
14 plane containing the most significant bits first, the bit plane containing
the second most
significant bits second, etc. The CCT is designed such that the most
significant bits of
16 each pixel color are stored first and the Lesser significant bits are
stored last.
17 Consequently, the bit planes transmitted first will always contain the most
significant
18 data, and the bit planes transmitted last will always contain the least
significant data.
19 Thus, the remote video monitor will receive video from the RMU
progressively,
receiving and displaying the most significant bits of the image before
receiving the
21 remaining bits. Such a method is less sensitive to changes in bandwidth and
will allow a
22 user to see the frame of video as it is transmitted, rather than waiting
for all details of the
23 frame to be sent.
-34-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 After compression of the video signals is complete, the resulting video
signals are
2 transmitted to either Ethernet connector 205 or communications port
connector 206 via
3 switch 390.
4 Referring back to FIG. 3A, RMU 109 also contains a power supply 221 which
provides power to RMU 109. Preferably, power supply 221 is a redundant power
supply
6 which contains backup circuitry in case the main circuitry fails. Power
supply 221
7 receives power through power port 223 from an external power supply. The
power to
8 RMU is controlled by reset circuitry 225 which is interfaced directly to CPU
207. Reset
9 circuitry 225 is utilized to turn the power onloff and reset RMU 109.
RMU 109 also contains local KVM port 227 interfaced to CPU 207. Local I~VM
11 port 227 allows for connection of local keyboard 123, video monitor 127,
and cursor
12 control device 125 to RMU 227 via cable 129 (FIG. 1). Local keyboard I23,
video
I3 monitor 127, and cursor control device 125 may be utilized for onsite
control of the
14 attached serial devices I l la and 1 I Ib, servers 113a and 1 I3b,
computers I 15a and 115b,
and power supply I I7.
16 Option menu circuit 229, under control of CPU 207, provides the option menu
to
I7 a user of the present invention. As previously discussed, the option menu
contains menus
18 for selecting a serial device, a remote server or computer, or options to
control the power
19 to all devices connected to power supply 117.
To utilize the system of the present invention, a user first initiates a
remote
21 management session at user workstation I01 and enters the required username
and
22 password. However, any unique combination of authentication information may
be
23 utilized. User workstation 101 packetizes the entered information and
routes it to
-35-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 Internet/LAN/WAN 108 via communication line 119 and then to RMU 109 via
2 communication Line 121. The entered data is received at CPU 207 via RJ-45
connector
3 201 (or alternatively RJ-11 connector 202). Ethernet connector 205 removes
the network
4 protocol and transmits the received keyboard and/or cursor control device
signals to CPU
207. CPU 207 utilizes a lookup table containing all user profiles stored in
the system to
6 authenticate the user. Different user profiles may be given different levels
of access to
7 the system. For example, certain users may only be able to access and
operate computers
8 115a and 115b and be restricted from operating servers 113a and 113b, serial
devices
9 111 a and 111 b, and power supply 117.
Once a user has been authenticated, option menu circuit 229 produces an option
11 menu containing all the devices attached to RMU 109. In this case, the
attached devices
12 include serial devices l l la and l l 1b, servers 113a and 113b, computers
115a and I 15b,
13 and power supply 117. However, it would be apparent to one skilled in the
art that RMU
14 109 may accommodate any number of serial devices, servers, computers, and
associated
power supplies. The option menu produced by option menu circuit 229 is
compressed by
I6 video processor 2I2 and packetized by Ethernet connector 205 and then
transmitted to
17 user workstation 101 through RJ-45 connector 201, communication line I21,
18 Internet/LAN/WAN 108, and communication Iine 119, in that order. The option
menu is
19 depacketized and decompressed at user workstation 1 O1 for display on video
monitor
105. The user then utilizes keyboard 103 and cursor control device 107 to
select the
21 desired device from the option menu. The user-entered keyboard and cursor
control
22 device signals are then encoded by user workstation 101, transmitted to RMU
109 via
23 Internet/LAN/WAN 108, and subsequently decoded by CPU 207 located in RMU
109.
-36-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 CPU 207 interprets the received keyboard and cursor control device signals
and
2 interfaces the user with the selected device as previously described.
3 If the user selects to be interfaced with servers 1 I3a or 113b or computers
115a
4 and 115b, the video signal of the selected device is displayed on video
monitor 105. The
video signal initially arrives from the selected device at KVM port 219 and is
routed to
6 KVM port header 2I7. The video signal is then routed to frame grabber 2I5
which
7 converts the analog video signal to a digital signal. The resulting
digitized video signal is
8 then routed to CPU 207 through PCI riser card 209. CPU 207 then determines
the correct
9 location to transmit the video signal (i.e., to local KVM port 227 or video
processor 212).
I O If the video signal is routed to local KVM port 227, the video signal is
displayed on local
11 video monitor 127. Alternatively, if the video signal is routed to video
processor 212, it
12 is compressed by video processor 212 and packetized by either Ethernet
connector 205 or
13 communications port connector 206 for transmission via communication line
12I through
14 either RJ-45 port 201 or RJ-11 port 202. Ethernet connector 205 or
communications port
connector 206 also appends any other signals (i.e., keyboard signals, cursor
control
16 device signals, etc.) onto the compressed video signal for transmission to
user
17 workstation 101.
18 To switch to another connected device, the user presses a "hotkey" such as
19 "printscreen" or "F I " on keyboard 103 attached to user workstation 1 O l
(FIG. I ). This
causes option menu 229 to open an option menu allowing the user to select a
new serial
21 device, server, computer, or modify the power supply to one of the
connected devices.
22 Referring now to FIG. 4, depicted is a flowchart illustrating the operation
of the
23 compression algorithm utilized by video processor 212 in the preferred
embodiment of
-37-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 the present invention. The compression algorithm is executed internal to RMU
I09 by
2 video processor 212 (FIG. 3). The digitized video signal is initially stored
in a raw frame
3 buffer (step 402), which is one of the frame buffers 380 (FIG. 3D). At this
point, the
4 compression algorithm is performed to process the captured video data
contained in the
raw frame buffer and prepare it for transmission to user workstation 10l .
6 The first step of the compression algorithm is the NRDT (step 403). The NRDT
7 sub-algorithm is also executed internal to RMU I09 by video processor 212
(FIG. 3).
8 The NRDT sub-algorithm determines which blocks of pixels, if any, have
changed
9 between the current frame and the compare frame, also discussed above.
In the preferred embodiment, the video frame is first divided into 64x32 pixel
1 I blocks. Subsequently, the NRDT sub-algorithm is applied to each block of
pixels
12 independently. Alternative embodiments of the present invention may utilize
smaller or
I3 larger blocks depending on criteria such as desired video resolution,
available bandwidth,
14 etc.
Next, the NRDT sub-algorithm employs a two-threshold model to determine
16 whether differences exist between a block of pixels in the current frame
and the
17 corresponding block of pixels in the compare frame. These two thresholds
are the pixel
18 threshold and the block threshold.
I9 First, each pixel of the pixel block is examined to determine if that pixel
has
changed relative to the corresponding pixel of the corresponding block in the
compare
21 frame. The distance value of each of the three colors (i.e., red, green,
and blue) of each
22 pixel in relation to the corresponding eompare pixel is calculated, as
described in greater
23 detail below with respect to FIG. 7. If the distance value is larger than
the pixel threshold
-38-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 (i.e., the first threshold of the two-threshold model), this distance value
is added to a
2 distance sum value.
3 Then, after all pixels within the pixel block have been examined, if the
resulting
4 distance sum value is greater than the block threshold (i.e., the second
threshold of the
two-threshold model), the block is determined to have changed. Every block of
pixels in
6 the video frame undergoes the same process. Therefore, after this process
has been
7 applied to an entire video frame, the process will have identified all pixel
blocks that the
8 process has determined have changed since the previous video frame. At this
point, the
9 compare frame is updated with the changed pixel blocks. However, the pixel
blocks of
the compare frame that correspond to unchanged pixel blocks of the current
frame will
I 1 remain unchanged. In this manner, the two-threshold model used by the NRDT
sub-
12 algorithm eliminates pixel value changes that are introduced by noise
created during the
13 analog to digital conversion and also captures the real changes in the
video frame.
14 After the video data is processed by the NRDT sub-algorithm, it is next
processed
by the smoothing sub-algorithm (step 419). The smoothing sub-algorithm is
designed to
16 create a smooth, higher-quality video image by reducing the roughness of
the video
17 image caused by noise introduced during the analog to digital conversion.
18 The smoothing sub-algorithm first converts the pixel representation that
resulted
19 from the NRDT sub-algorithm into a pixel representation that uses a lesser
quantity of
bits to represent each pixel. This is performed using a CCT that is specially
organized to
21 minimize the size of the pixel representation. The smoothing sub-algorithm
uses the
22 CCT to choose color codes with the least number of 1-bits for the most
commonly used
23 colors. For example, white and black are assumed to be very common colors.
Thus,
_39 _
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 white is always assigned 0 and black is always assigned 1. That is, white
will be
2 represented by a bit value of 0 on all planes. Black, the next most common
color, will
3 show up as a bit value of 1 on all but one plane. This reduces the quantity
of data to be
4 compressed by the compression algorithm. Then, for each pixel in the block,
a color
code is assigned. Simultaneously, a histogram of color codes is created to
store the
6 number of occurrences of each of the unique colors in the block of pixels.
This
7 histogram of color codes is then sorted to produce a list of color codes
from the least
8 number of occurrences to the dominant number of occurrences.
9 Once the sorted list of color codes is created, the next step is to merge
colors.
I O Working from the beginning of the sorted list, the smoothing sub-algorithm
compares the
11 least frequently occurring colors to the more frequently occurring colors.
If the less
I2 frequently occurring color is very similar to a more frequently occurring
color, then the
13 pixels having the less frequently occurnng color will be changed to the
more frequently
14 occurring color. Determination of whether two colors are similar is
performed by
I 5 calculating the distance between the three-dimensional points of the RGB
space. The
16 formula is:
1~ I7 = ,~(R1 - gz)z -I- (G1 - Gz)z -I- (B~ _ Bz)z
I8 where D is the distance, R1 is the red value of the low frequency pixel, RZ
is the red value
I 9 of the high frequency pixel, G~ is the green value of the low frequency
pixel, GZ is the
20 green value of the high frequency pixel, B I is the blue value of the low
frequency pixel,
21 and B~ is the blue value of the high frequency pixel. If the distance is
within a distance
22 threshold, the two colors are determined to be similar. In the preferred
embodiment of
23 the present invention, system performance is increased by squaring the
distance threshold
-40-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 and comparing this value with the sum of the squares of the RGB differences.
This step
2 eliminates taking the square root of the sum, which requires a greater
amount of
3 processing time.
4 Each block of pixels is filtered for noise and translated from a RGB
representation
to a color code representation. The noise that is introduced by LCD controller
215 (FIG.
6 3) during conversion of the analog signals to digital signals distorts the
values of some
7 pixels. Thus, the smoothing sub-algorithm corrects distorted pixels. The
smoothing sub-
8 algorithm minimizes noise by reducing the number of different colors present
in each
9 video image block. Further, such smoothing creates an image with greater
redundancy,
thus yielding higher compression ratios.
11 After smoothing, caching is performed (step 421). Caching is a sub-
algorithm of
12 the overall compression algorithm executed by video processor 212 of RMU
109 (FIG.
13 3). Caching requires RMU 109 (FIG. 3) to retain a cache of recently
transmitted images.
14 Such a cache can be implemented and stored in RAM 386 (FIG. 3D). The
caching sub-
algorithm compares the most recent block of pixels with the corresponding
block of
16 pixels in the video images stored in the cache (step 405). If the most
recently transmitted
17 block of pixels is the same as one of the corresponding blocks of pixels
stored in the
18 cache, the caching sub-algorithm does not retransmit this portion of the
video image.
19 Instead, a "cache hit" message is sent to user workstation 101, which
indicates that the
most recently transmitted block is already stored in the cache (step 407). The
"cache hit"
21 message contains information regarding which cache contains the
corresponding block of
22 pixels, thereby allowing user workstation 1 O 1 to retrieve the block of
pixels from its
-41-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 cache and use it do create the video image to be displayed on its attached
video display
2 device.
3 The next step in the process, step 409, determines if the NRDT determined
that
4 the block of pixels has changed since the corresponding block of pixels in
the compare
frame. This step can also be implemented before or in parallel with step 405.
Also, steps
6 421, 405, and 407 may be eliminated entirely.
7 The main purpose of step 409 is to determine whether the block has changed
since
8 the last frame. If the block has not changed, there is no need to send an
updated block to
9 user workstation 101. Otherwise, if the block of pixels has changed, it is
prepared for
compression (step 411 ). In the preferred embodiment, step 409 uses a
different technique
I I than step 405. With two ways of checking for redundancy, higher
compression will
12 result. Both steps 409 and 41 I are executed by a caching sub-algorithm
executed by
13 microprocessor 388 of video processor 212 (FIG. 3D).
14 For any areas of the image that have changed, the cache is updated, and the
data is
compressed before being sent to the server stack. In the preferred embodiment,
the image
I6 is compressed using the IBM JBIG compression algorithm. JBIG is designed to
I7 compress black and white images. However, the present invention is designed
to
18 transmit color video images. Therefore, bit planes of the image are
extracted (step 41 I),
19 and each bit plane is compressed separately (step 4I3). Finally, the
compressed image is
ZO transmitted to server stack 417 (step 415), which transmits the data to
switch 390 (FIG.
1 3D).
?2 FIG. 5A and FIG. 5B provide detailed flowcharts of a preferred embodiment
of
?3 the compression process. The digital representation of the captured video
image is
-42 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 transferred and stored in either frame buffer 0 503 or frame buffer 1 505. A
frame buffer
2 is an area of memory that is capable of storing one frame of video. The use
of two frame
3 buffers allows faster capture of image data. The captured frames of video
are stored in
4 frame buffer 0 503 and frame buffer 1 505 in an alternating manner. This
allows the next
frame of video to be captured while compression is being performed on the
previous
6 frame of video. In video processor 212, frame buffer 0 503 and frame buffer
1 505
7 comprise a portion of frame buffers 380 (FIG. 3D).
8 An NRDT test is performed on each block of pixels stored in frame buffer 0
503
9 and frame buffer 1 505 (step S 19), which compares each block of the
captured video
image to the corresponding block of the previously captured video image. Step
S I9
11 compares blocks of pixels from the video image stored in the current raw
frame buffer
12 (i.e., frame buffer 0 503 or frame buffer 1 505) with the corresponding
block of pixels
13 stored in compare frame buffer 521. This step is discussed in greater
detail below with
14 respect to FIGS. 6A and 6B.
If step 5 I9 determines that the current block of pixels has changed, then
nearest
16 color match function processes the video images contained in frame buffer 0
503 and
17 frame buffer I 505 (step 509) in conjunction with the information contained
in the client
18 color code table ("CCT from client") 51 I, which is stored in flash memory
239 (FIG. 3).
I9 The nearest color match function can be executed as software by
microprocessor 388. A
detailed explanation of the nearest color match function is provided below
with respect to
21 FIG. 6.
22 The CCT obtained from CCT 513 by the nearest color match function is used
for
23 color code translation (step 515), which translates the digital RGB
representation of each
-43 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 pixel of the changed block of pixels to reduce the amount of digital data
required to
2 represent the video data. Color code translation (step 515) receives blocks
of pixels that
3 the NRDT sub-algorithm (step 519) has determined have changed relative to
the previous
4 captured video image. Color code translation then translates this digital
data into a more
compact form and stores the result in coded frame buffer 517. Coded frame
buffer 517
6 can be implemented as a portion of RAM 386 (FIG. 3D).
7 Alternatively, steps 509 and 515 may be performed in parallel with step 519.
8 Performing these steps in parallel reduces the processing time required for
each block of
9 pixels that has changed. In this scenario, steps 509 and 515 are performed
in anticipation
of the block of pixels having changed. If this is the case, the processing for
steps 509 and
11 515 may be completed at the same time as the processing for step 519 is
completed.
12 Therefore, the algorithm may move directly to step S23 from step 509
without having to
13 wait for the processing of steps 509 and 515. Otherwise, if step 519
determines that the
14 block of pixels has not changed, and therefore the results of steps 509 and
515 are not
required, these results may simply be discarded.
16 Upon completion of step 515, caching begins by performing a cyclical
17 redundancy check (CRC)(step 523). Cyclic redundancy check (CRC) is a method
known
18 in the art for producing a checksum or hash code of a particular block of
data. The CRCs
19 may be computed for two blocks of data and then compared. If the CRCs
match, the
blocks are the same. Thus, CRCs are commonly used to check for errors. In the
present
21 invention, the CRC is used to compare a block of pixels with blocks of
pixels stored in a
22 cache. Thus, in step 523, the CRC is computed for each block of pixels that
was
-44-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 determined to have changed by the NRDT sub-algorithm. The array of CRCs is
stored in
2 CRC array 525.
3 Turning next to FIG. 5B, depicted is an overview of the caching and bit
4 splicing/compression sub-algorithms. This portion of the algorithm begins
waiting for
information from coded frame buffer 517 and CRC array 525 (step 527). Next, a
6 decision is made as to whether a new video mode has been declared (step
529). A new
7 video mode can be declared if, for example, user workstation 101 has
different bandwidth
8 or color requirements. If a new video mode has been declared, all data is
invalidated
9 (step 53 I ) and the sub-algorithm returns to step 527 to wait for new
information from
I O coded frame buffer 517 and CRC array 525. Downscaler circuit 362 and/or
upscaler
11 circuit 364, located in LCD controller 215, may be utilized to adjust the
outputted
12 digitized video to be compatible with the new video mode. Steps 527, 529,
and 531 are
13 all steps of the overall compression algorithm that is executed by
microprocessor 388
14 (FIG.3D).
If in step 529 it is deemed that a new video mode has not been declared, then
the
16 comparison of the current block of pixel's CRC with the cached CRCs is
performed (step
17 533). This block compares the CRC data of the current video frame contained
in CRC
18 array 525 with the cache of previous CRCs contained in block info array
535. Block info
19 array 535 stores the cache of pixel blocks and the CRCs of the pixel blocks
and can be
implemented as a device in RAM 386 (FIG. 3D). Step 533 is also a part of the
overall
21 compression algorithm executed by microprocessor 388 (FIG. 3D).
22 Next, if the current block of pixels is located within the pixel block
cache
23 contained in block info array 535 (step 537), a cache hit message is sent
to user
-45 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 workstation 101 and the block of pixels is marked as complete, or processed
(step 539).
2 Since user workstation 101 contains the same pixel block cache as RMU I09
(FIG. 3D),
3 the cache hit message simply directs user workstation I01 to use a specific
block of
4 pixels contained in its cache to create the portion of the video image that
corresponds to
the processed block of pixels.
6 Next, a check is performed for unprocessed blocks of pixels (step 539). All
7 blocks of pixels that need to be processed, or updated, are combined to
create a compute
8 next update rectangle. If there is nothing to update (i.e., if the video has
not changed
9 between frames), then the algorithm returns to step 527 (step 543). Thus,
the current
frame will not be sent to the remote participation equipment. By eliminating
the
11 retransmission of a current frame of video, the sub-algorithm reduces the
bandwidth
12 required for transmitting the video.
13 If, however, there are areas of the image that need to be updated, the
update
14 rectangle is first compressed. The update rectangle must first be bit
sliced (step 545). A
bit plane of the update rectangle is constructed by taking the same bit from
each pixel of
16 the update rectangle. Thus, if the update rectangle includes 8-bit pixels,
it can be
17 deconstructed into 8 bit planes. The resulting bit planes are stored in bit
plane buffer 547.
18 Again, steps 541, 543, and 545 are all part of the bit splicing/compression
sub-algorithm
19 executed by microprocessor 388 of RMU 109 (FIG. 3).
Each bit plane is compressed separately by the compression sub-algorithm (step
21 549). In this case, compression is performed on each bit plane and the
resulting data is
22 sent to server stack 417 (step 551). In the preferred embodiment,
compression is
-46-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 performed by video compression device 382 (FIG. 3) (step 549). Thereafter,
the
2 compressed bit planes are sent to switch 390 (FIG. 3D).
3 Since the preferred embodiment captures frames 20 times per second, it is
4 necessary to wait 300 ms between video frame captures. Thus, the algorithm
waits until
300 ms have passed since the previous frame capture before returning the sub-
algorithm
6 to step 527 (step 553).
7 Referring now to FIG. 6, illustrated is the nearest color match function
(step 509
8 of FIG. 5A) that selectively maps less frequently occurring colors to more
frequently
9 occurring colors using a CCT. Nearest color match function 509 processes
each block of
pixels of the video image stored in frame buffer 0 503 or frame buffer 1 505
successively.
11 As shown in FIG. 6, a block of pixels is extracted from the video image
stored in frame
12 buffer 0 503 or frame buffer 1 505 (step 600). In the preferred embodiment,
the extracted
13 block has a size of 64 by 32 pixels, however, any block size may be
utilized.
14 The nearest color match function eliminates noise introduced by the A/D
conversion by converting less frequently occurring pixel values to similar,
more
16 frequently occurring pixel values. The function utilizes histogram analysis
and difference
I7 calculations. First, nearest color match function 509 generates a histogram
of pixel
18 values (step 601 ). The histogram measures the frequency of each pixel
value in the block
19 of pixels extracted during step 600. The histogram is sorted, such that a
list of frequently
?0 occurring colors (popular color List 603) and a list of least frequently
occurring colors
1 (rare color list 605) are generated. The threshold for each list is
adjustable.
?2 Then, nearest color match function 509 analyzes each low frequently
occurring
?3 pixel to determine if the pixel should be mapped to a value that occurs
often. First, a
-47 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 pixel value is chosen from rare color list 605 (step 607). Then, a pixel
value is chosen
2 from popular color list 603 (step 609). These distance between these two
values is then
3 computed (step 611). In this process, distance is a metric computed by
comparing the
4 separate red, green and blue values of the two pixels. The distance value D
may be
computed in a variety of ways. One such example is:
6 D = (R1 - RZ)2 -I- (G1 - Gz)2 -i- (B1 - gz)a
7 In this formula, R1 is the red value of the low frequency pixel, R2 is the
red value of the
8 high frequency pixel, G1 is the green value of the low frequency pixel, G2
is the green
9 value of the high frequency pixel, BI is the blue value of the low frequency
pixel, and B2
is the blue value of the high frequency pixel.
11 This formula yields a distance value, D, which indicates the magnitude of
the
12 similarity or difference of the colors of two pixels, such as a less
frequently occurring
13 pixel versus a more frequently occurring pixel. The goal of the sub-
algorithm is to find a
14 more frequently occurring pixel having a color that yields the lowest
distance value when
compared to the color of a less frequently occurring pixel. Therefore, a
comparison is
16 performed for each computed distance value (step 613). Every time a
distance value is
17 computed that is less than all previous distance values, the distance value
is written to the
18 closest distance variable (step 615).
19 Once it is determined that all more frequently occurring pixels have been
compared to less frequently occurring pixels (step 617), a computation is
performed to
21 determine if the lowest occurring D is within a predefined threshold (step
619). If this D
22 is within the predefined threshold, CCT 513 is updated by mapping the low
frequently
23 occurring pixel to the color code value of the high frequently occurring
pixel that yielded
-48 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 this D value (step 621). This process is repeated for all low frequency
pixels and CCT
2 513 is updated accordingly.
3 Turning to FIG. 7, RGB NRDT step S I 9 (FIG. 5A) is illustrated in further
detail.
4 This process operates on every block of pixels. Current pixel block 700
represents a
block of pixels of the video image contained in the current frame buffer
(i.e., frame buffer
6 0 503 or frame buffer 1 505 (FIG. 5A)). Previous pixel block 701 contains
the
7 corresponding block of pixels of the video image contained in compare frame
buffer 521
8 (FIG. 5A). Step 519 begins by extracting corresponding pixel values for one
pixel from
9 the current pixel block 700 and previous pixel block 701 (step 703). Then,
the pixel color
values are used to calculate a distance value, Which indicates the magnitude
of the
11 similarity or difference between the colors of the two pixels (step 705).
In the preferred
12 embodiment of the present invention, the distance value is computed using
the following
13 formula: ,
14 D = (R1 - Rz)Z -I- (G1 - G2)2 -I- (B1 - Bz)2
As before, R1, G1, and B1 are the red, green and blue values respectively of
the frame
16 buffer pixel. Similarly, R2, G2, and B2 are the red, green and blue values
respectively
17 for the compare frame buffer pixel.
18 Next, the computed distance value D is compared with a pixel threshold
(step
19 707). If D is greater than the pixel threshold, it is added to an
accumulating distance sum
(step 709). If the value of D is less than the pixel threshold, the difference
is considered
21 to be insignificant (i.e., noise) and it is not added to the distance sum.
22 This process of computing distance values and summing distance values that
are
23 greater than a predefined pixel threshold continues until it is determined
that the last pixel
-49 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 of the block of pixels has been processed (step 711). Once the last pixel is
reached, the
2 distance sum is compared with a second threshold, the block threshold (step
7I3). If the
3 distance sum is greater than the block threshold, the current block of
pixels designated as
4 changed as compared to the corresponding block of pixels from the previously
captured
frame. Otherwise, if the distance sum is less than the block threshold, the
block of pixels
6 is designated as unchanged.
7 If the block of pixels is designated as changed, step 715 is executed. Step
715
8 sets a flag that indicates that the particular block of pixels has changed.
Furthermore, the
9 new block of pixels is written to compare frame buffer 521 (FIG. 5A) to
replace the
corresponding previous block of pixels.
I 1 Otherwise, if the distance sum does not exceed the block threshold, the
block is
12 designated unchanged and, a flag is set to indicate that this block of
pixels does not need
13 to be re-transmitted to the remote participation equipment (step 721).
Rather, the remote
14 participation equipment will recreate the portion of the video image
represented by the
block of pixels using the same block of pixels displayed fox the previous
frame of video.
16 At this point the system computers CRCs for changed blocks of pixels (step
523 of FIG.
17 SA) as discussed in greater detail above with respect to FIG. 5A.
18 FIG. 8 further illustrates the two level thresholding used by the NRDT sub-
19 algorithm shown in FIG. 7. For illustrative purposes only, 4x4 blocks of
pixels are
shown. Each pixel is given red, green, and blue color values that range from 0
to 255, as
21 is commonly performed in the art. A pixel having red, green, and blue
values of 0
22 represents a black pixel, whereas a pixel leaving red, green, and blue
values of 255
23 represents a white pixel. Previous pixel block 751 is a block of pixels
grabbed from
-50-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 compare frame buffer 521 (FIG. 5A). Previous pixel 1 752 is the pixel in the
upper, Left
2 corner of previous pixel block 751. Since every pixel of previous pixel
block 751 has a
3 value of 0, previous pixel block 751 represents a 4x4 pixel area that is
completely black.
4 Current pixel block 753 represents the same spatial area of the video frame
as
previous pixel block 751, but it is one frame later. Here current pixel 1 754
is the same
6 pixel 1 as previous pixel 1 752, but is one frame later. For simplicity,
suppose a small
7 white object, such as a white cursor, enters the area of the video image
represented by
8 previous pixel block 75 I . This change occurs in current pixel I 754 of
current pixel
9 block 753. In current pixel block 753, the majority of the pixels remained
black, but
current pixel 1 754 is now white, as represented by the RGB color values of
255, 255,
I 1 and 255.
I2 Further suppose that noise has been introduced by the A/D conversion, such
that
13 previous pixel 755 has changed from black, as represented by its RGB values
of 0, 0, and
14 0, to gray. The new gray color is represented by the RGB values of 2, 2,
and 2 assigned
to current pixel 756.
I 6 Further suppose that the pixel threshold is 100, and the block threshold
is 200.
17 The NRDT sub-algorithm calculates the distance value between each pixel of
current
18 pixel block 753 and previous pixel block 751. The formula used in the
preferred
19 embodiment of the present invention, as discussed above with respect to
FIG. 7, is:
D - (R1 - Rz)2 ~- (G'1 - G2)2 -~-. (g1 _ BZ)a
21 Therefore, the distance value between current pixel 1 754 and previous
pixel 1 752 is:
22 D = (255 - 0)2 + (255 - 0)2 + (255 - 0)2
-51-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 or 195,075. This distance value is added to the distance sum because 195,075
exceeds
2 the pixel threshold of 100. However, the distance value between the black
previous pixel
3 755 and the gray current pixel 756 is not added to the distance sum because
the distance
4 between the pixels, as calculated using the above distance formula, equals
12, which does
not exceed the pixel threshold of I00. Similarly, the distance value is
computed for all of
6 the remaining pixels in the two pixel blocks. Each of these distance values
equals zero,
7 therefore, since these distance values are less than the pixel threshold,
they axe not added
8 to the distance sum.
9 Consequently, after the distance values for all pixels have been processed,
the
I O distance sum equals 195,075. Since this value is greater than the block
threshold of 200,
I I the block is designated. This example illustrates the advantages of the
two-level
12 thresholding feature of the NRDT sub-algorithm. That is, the noise that
occurred in
13 current pixel 756 of current pixel block 753 was ignored, whereas the real
change in
14 video that occurred in current pixel I 754 of current pixel block 753 was
recognized.
Turning finally to FIG. 9, shown is a flowchart of the decompression algorithm
16 executed by user workstation 101 (FIG. I). The decompression algorithm
begins by
17 waiting for a message (step 801). This message is transmitted from server
stack 417 of
18 RMU 109 to user workstation 1 O 1. Thereafter, user workstation 101
receives the
t 9 information and writes the data to client stack 803. Client stack 803 may
be a register or
?0 some other device capable of permanently or temporarily storing digital
data. In one
I embodiment of the present invention, messages are transmitted using the
TCP/IP
'2 communication protocol. In this scenario, client stack 803 is the local
TCP/IP stack.
,3 Other embodiments may use a protocol other than TCP/IP. However,
irrespective of the
-52
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 communication protocol, the present invention uses client stack 803 to store
received
2 messages for processing.
3 Once a message is received in client stack 803, it is processed to determine
4 whether the message is a new video mode message (step 805). A new video mode
message may be sent for a variety of reasons including a bandwidth change, a
change in
6 screen resolution or color depth, a new client, etc. This list is not
intended to limit the
7 reasons for sending a new video mode message, but instead to give examples
of when it
8 may occur. If the message is a new video mode message, application layer 823
is
9 notified of the new video mode (step 807). According to the preferred
embodiment,
application layer 823 is software executed by user workstation 101 that
interfaces with
11 the input and output devices of user workstation 101 (i.e., keyboard 103,
video monitor
12 105, and cursor control device 107). Any video updates must therefore be
sent to
13 application layer 823. Also, the old buffers are freed, including all
memory devoted to
14 storing previously transmitted frames, and new buffers are allocated (step
809). The
decompression algorithm then returns to step 801.
16 If the new message is not a video mode message, the message is further
processed
17 to determine if it is a cache hit message step 8I 1). If yes, the cache hit
message is
18 deciphered to determine which block of pixels, of the blocks of pixels
stored in the three
19 cache frame buffers 815, should be used to reconstruct the respective
portion of the video
image. Although three cache frame bufl~ors 815 are used in the preferred
embodiment of
21 the present invention, any quantity of cache frame buffers may be used
without departing
22 from the spirit of the invention. Cache frame buffers 815 store the same
blocks of pixels
23 that are stored in the cache frame baffer~ located internal to RMU 109
(FIG. 3). Thus,
-53 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 the cache hit message does not include video data, but rather simply directs
the remote
2 participation equipment as to which block of pixels contained in the cache
frame buffer
3 81 S should be sent to merge frame buffer 8 I 7. The block of pixels
contained within the
4 specified cache is then copied from cache frame buffer 815 to merge buffer
817 (step
S 813). Finally, application layer 823 is notified that an area of the video
image has been
6 updated (step 825). Merge buffer 817 contains the current representation of
the entire
7 frame of video in color code pixels. Application layer 823 copies the pixel
data from
8 merge bu~'fer 817 and formats the data to match the pixel format of the
connected video
9 monitor I05 (step 819). Thereafter, the formatted pixel data is written to
update firame
I O buffer 821, which then transmits the data to video monitor 105.
Alternatively, in lieu of a
I 1 video monitor, the formatted pixel data may be written to a video card,
memory, and/or
12 any other hardware or software commonly used with video display devices.
I 3 Further, if the new message is not a new video mode or cache hit message,
it is
14 tested to determine if it is a message containing compressed video data
(step 827). If the
1 S message does not contain compressed video data, the decompression
algorithm re~uams to
16 step 80I and waits for a new message to be transmitted from server stack
417.
17 Otherwise, if the message does contain compressed video data, the data is
decompressed
18 and transferred to bit plane frame buffer 833 (step 829). As described
above, the
19 preferred embodiment incorporates the JBIG lossless compression technique.
Therefore,
20 decompression of the video data must be performed for each individual bit
plane. After
21 each bit plane is decompressed, it is merged with previously decompressed
bit planes,
22 which are stored in bit plane frame buffer 833 (step 829). When a
sufficient number of
23 bit planes have been merged, the merged data contained in bit plane frame
buffer 833 is
-54-
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 transferred to merge frame buffer 817 (step 831). Alternatively, individual
bit planes
2 may be decompressed and stored directly in merge frame buffer 817, thereby
eliminating
3 step 831. When all of the data required to display a full frame of video is
transferred to
4 merge frame buffer 817, application layer 823 copies the data in merge frame
buffer 817
to update frame buffer 821 (step 819). Thereafter, the data is transferred to
video monitor
6 105.
7 In an alternate embodiment, the video displayed on video monitor 105 can be
8 updated after each bit plane is received. In other words, a user does not
have to wait until
9 the whole updated frame of video is received to update portions of the
displayed video.
I O This alternative method is desirable when the bandwidth available for
video transmission
11 varies. Also, this progressive method of updating the video display is one
of the
12 advantages of using the JBIG compression algorithm.
13 Next, the decompression algorithm determines whether all of the color code
data
14 from one field of the current video frame has been received (step 835). If
a full field has
not been received, the decompression algorithm returns to step 801 and waits
for the
16 remainder of the message, which is transmitted from server stack 417 to
client stack 803
17 in the form of a new message. Otherwise, if a full field has been received,
the
18 decompression method notifies application layer 823 (step 837). Similar to
that
19 described above with respect to processing cache hit messages, this
notification directs
application layer 823 to read the data in merge frame buffer 817 and convert
it to the
21 current screen pixel format (step 819). Thereafter, the formatted data is
written to update
22 frame buffer 821, which transmits the data to video monitor 105.
-55 -
CA 02546952 2006-05-24
WO 2005/054980 PCT/US2004/035943
1 After a full field has been received and application layer 823 has been
notified, a
2 second determination is made to determine if the full field is the Iast
field included in the
3 message. If it is, the newly decompressed block of pixels is written to one
of the cache
4 frame buffers 8 I 5 (step 84I ). Otherwise, the decompression algorithm
returns to step
801 and continues to wait for a new message. Preferably, the new block of
pixels written
6 to cache frame buffer 815 overwrites the oldest block of pixels contained
therein. Step
7 841 ensures that the cache is up-to-date and soynchronized with the cache of
RMU 109.
8 After the completion of the cache update, the decompression algorithm
returns to step
9 801.
While the present invention has been described with reference to the preferred
11 embodiments and several alternative embodiments, which embodiments have
been set
12 forth in considerable detail for the purposes of making a complete
disclosure of the
13 invention, such embodiments are merely exemplary and are not intended to be
limiting or
14 represent an exhaustive enumeration of all aspects of the invention. The
scope of the
1 S invention, therefore, shall be defined solely by the following claims.
Further, it will be
16 apparent to those of skill in the art that numerous changes may be made in
such details
17 without departing from the spirit and the principles of the invention. It
should be
18 appreciated that the present invention is capable of being embodied in
other forms
I9 without departing from its essential characteristics.
21
-56-
