Note: Descriptions are shown in the official language in which they were submitted.
CA 02228679 1998-02-04
Surveillance System
Field of the Invention
This invention relates to surveillance systems and more
particularly to such systems employing cameras at a remote
location with the camera output and any alarm conditions
encoded and selectively reported to a monitoring station
over a communications network.
Background
Surveillance systems for fire and burglary detection
are well known. In such systems motion and or heat detectors
are strategically located in and around the premises and
upon the detection of an event which causes an alarm a
signal is forwarded over the telephone line to a monitoring
or supervising station. An attendant at the monitoring
station then contacts the local authorities to report the
occurrence. If the occurrence represents a burglary it is
likely that the intruders will have had time to leave the
scene before the local authorities arrive. Evidence, which
might be useful in identifying the intruders, will likely be
limited to fingerprints.
Video surveillance in banks, convenience stores and the
like is also known. In these surveillance systems cameras
continually record activity at the selected location and
after an incident the entire record needs to be reviewed to
determine whether any information of value has been
captured.
There is, therefore, a need to provide a surveillance
system which has a motion detector and camera to monitor a
particular location and to provide a real time alarm to a
supervising station when unexpected motion is detected.
Preferably the system will be able to provide a video record
of events immediately prior to the alarm as well as a record
of the event itself. It is also desirable for the
1
CA 02228679 1998-02-04
supervising station to have some configuration control of
the cameras at the remote location such that certain cameras
can be turned on and off and that the area scanned by the
selected cameras can be varied. The real time recording of
events at the supervising station provides a record that can
be saved in the event that the cameras and/or records at the
remote location are destroyed.
Summary of the Invention
It is therefore an object of the present invention to
provide a surveillance system having selective remote video
monitoring, with motion detection, of a selected.location
and. means to provide an alarm and record pre and post event
camera images.
Therefore in accordance with a broad aspect of the
present invention there is provided a surveillance system
comprising: a capture station having monitoring means to
detect and dynamically record events occurring at a selected
location;a surveillance station remote from the capture
station and connected thereto via a communications network,
the surveillance station selectively monitoring the capture
station and receiving notification of an event occurring at
the selected location; and a database server connected to
the capture station and the surveillance station to support
operation thereof, the database having storage means to
retain a record of selected events.
Brief Description of the Drawing
The invention will now be described in grater detail
with reference to the attached figure which is a high level
system diagram.
Detailed Description of the Invention
As shown in Figure 1 four separate (but interdependent)
applications have been developed. These are remote or
2
CA 02228679 1998-02-04
capture station, operator and/or management station also
known as surveillance station, and video database.
The remote station or capture station is the software that
runs on the computer to which one or several surveillance
cameras send their output. An operator station or
surveillance station is software which can connect to a
remote station for the purpose of viewing output from
cameras that are associated with the remote station. A
management station, which is a more sophisticated version of
the operator station, can maintain open connections to one
or several remote stations. In addition to viewing output
from cameras associated with the remote stations, the
management station can control and configure the remote
stations (and the associated devices). The video database
archives video that is correlated with events logged either
by operators who are sitting at an operator station or by
the software which is executing on the server at the remote
site.
The remote station listens for connection requests from
clients who wish to monitor output from a camera that is
hosted at that station. When a connection request is
received, the remote station determines whether the
connection attempt is valid or invalid. If the connection
attempt is determined to be valid, a Transmission Control
Protocol (TCP) connection is opened to the connecting client
and initialization data is sent to the client. This data
consists of the list of those cameras hosted at the remote
station that the connected client has access privileges for.
If the connecting client is an ordinary user, the client can
select cameras for viewing. Once selected, output from the
cameras is sent to the operator station. If the connecting
client is a privileged user, additional actions are
available to the client.
The server listens for incoming connections. When a
connection request is detected, the server determines the
3
CA 02228679 1998-02-04
validity of the request. If the request is accepted, the
server opens a TCP channel to the client.
The remote station maintains a database of users. When a
connection request is made, the client that issues the
request must submit a username and password. If the
username-password combination entered by the client is not
in the database, the connection attempt is rejected. If the
username-password combination is in the database, the user
connection request is accepted. Information in the user
record corresponding to the client that issued the
connection request is used to determine the access level of
the connection. For example, users may ~e restricted'to
viewing output from a subset of the cameras that are hosted
at the remote station.
Apart from the security represented by the login
sequence, additional mechanisms for securing the remote
station from hostile connections will be implemented. IP
filtering is a rudimentary form of security that can be
implemented at no cost. Authentication headers will be used
to create an additional security layer.
Once a client connection is accepted, a TCP connection to
the client is created. The TCP connection is added to a
table of connected clients. The connection is encapsulated
in a class of client connections that receive and dispatch
requests from a connected client.
The client connection on the server handles a11
communications with the remote client. The remote client
issues a request that is received by the client connection.
The client connection parses the request and dispatches it
to the appropriate handler. The client connection maintains
a table of cameras that the client is monitoring and manages
the flow of data from the camera to the client.
There are different classes of clients categorized by access
level and other characteristics. In this embodiment, two
kinds of clients are defined - ordinary users and privileged
4
CA 02228679 1998-02-04
users. Ordinary users are restricted to viewing output from
cameras but cannot modify the state of the cameras hosted by
the remote station or the state of the remote station.
Privileged users on the other hand are permitted to
configure the remote station and the devices that are
associated with it. Privileged users can also monitor
activity on the remote station and take corrective action if
it is required. For example, if an ordinary client is
monitoring output from too many cameras and thereby choking
the flow of video data to other clients or to a management
station, the privileged user can remove devices from the
list of devices that the offending client is monitoring.
The command channel is a TCP connection that is used to send
requests from the client to the remote station. The requests
that can be sent over this channel will vary with the type
of client.
Ordinary users can send requests to view the output
from a particular camera, to stop viewing output from a
particular camera and to logout out from the remote station.
2o Ordinary users cannot send commands that would alter the
state of any of the monitoring devices.
The command channel will also be used for transmission
of statistics that are relevant for flow control. Two kinds
of data are relevant. The average time for a block of video
data to travel over the network from sender to receiver and
the amount of data loss (as a ratio of packets sent to
packets received). This information will be used to adjust
the rate at which the sender transmits data to the receiver.
A client connected to the remote station can view output
30 from one or several cameras hosted by the remote station.
The output from a camera is transmitted to the client over a
video channel. For applications of the system to
surveillance and security it is important to incorporate
security and caching into transmission of video.
5
CA 02228679 1998-02-04
The payload is encrypted using an implementation of the
public domain "Blowfish" algorithm. The Blowfish algorithm
is a private key encryption scheme. To enhance the security
of the system, the keys are replaced on a periodic basis. To
accomplish this in a secure manner, a scheme for key
management has been implemented.
For applications to security it will be necessary to
cache output from each camera so that in the event that an
l0 unusual condition or unexpected event is detected prior
footage is available for review. The amount of pre-event
video that is recorded is under the control of the end user.
Likewise, the amount of post-event video that is
recorded is also under the control of the end user. For most
situations, a fifteen second interval prior to the
occurrence of an alarm appears to be adequate. Once an alarm
has occurred recording will continue until there is a
fifteen second interval that is free of alarms. The event
video that is recorded at the remote site is stored at the
20 same frame rate and image quality as the video that is
captured from the cameras tethered to the video server. In
particular, the video that is transmitted to the operator
station for viewing by central station operators can be of a
lower quality and frame rate than the event video stored at
the remote site. The event video that is stored at the
remote site is downloaded to a video database for
correlation with other event-related material.
The video channel will be implemented using the TCP
protocol. The most significant implementation related issue
30 is the reduction of video "fitter", i.e. the uneven temporal
spacing of video frames due to varying network conditions.
This problem has been addressed by adjusting the video
6
CA 02228679 1998-02-04
throughput from the video server to the current load on the
network.
Motion detection is implemented on the remote station.
The parameters that determine the mode of operation of the
mechanism for motion detection can be assigned default
values or can be set remotely by a privileged user who is
connected to the remote station through a management
station.
This motion detection technique consists of six (6)
steps that are performed sequentially. These steps, listed
in order of execution, are as follows.
1. Acquisition of 3 sequential images.
2. Change filtering
3. Edge detection
4. Perimeter filter
5. Proportion filter
6. Temporal filter
Data representative of three images, image I at time t
( I ( t ) [ i, j ] ) , image I at t ime t - 1 ( I ( t - 1 ) [ i, j ] ) , and
image I at time t - 2 (I(t - 2)[i,j)) are digitally stored
in such a way that each image can be accessed independently
of each other.
Data I ( t ) [ i, j ] is compared to I ( t-1 ) [ i, j ] and
I ( t-2 ) [ i, j ] . When pixels I ( t ) [ I, j ] and I ( t-1 ) [ i, j ] , and
I(t)[i,j] and I(t-2)[i,j] differ by a predefined threshold
f, a "1"is recorded in matrix d[i,j]. If the difference is
not greater than f then a "0" is recorded at the
corresponding position in matrix d[i,j]. This statement can
be described by:
IF (ABS (I (t) [i, j ] - I (t-1) [i, j ] ) > f) AND (ABS (I (t) [i, j ) -
I(t-2)[i,j]) > f) THEN d[i,j] - 1 ELSE d[i,j] - 0
7
CA 02228679 1998-02-04
where ABS means the absolute value.
Matrix d is put through an edge-detection algorithm.
This algorithm stores the edge data in matrix e. This
process acts to separate different components in the image
so that they can later be measured.
Each of the components that were separated in Edge
detection is then measured for the length (in pixels) of
their perimeter. If the component is not larger than
pre-defined size p then it is removed from matrix e.
The height-by-width proportions of remaining components are
then compared to a user-defined range r. If the components
height-by-width proportions do not fall within the specified
range, it is removed from matrix e. This statement can be
described by:
IF (r * width) > height THEN remove component
If any components remain the algorithm records this in
the form: Event at time t equals true.
If event at t-2, event at t-1, and event at t are a11
true then an alarm is considered to have occurred.
Prior to transmission over the network, data from the
cameras is encoded in order to reduce the amount of data
that must be sent.
A smoothing filter is applied to the image prior to
compression to eliminate noise introduced by the cameras.
Various techniques for compression are being employed to
transmit intra- and inter-frame information.
3o Still images are compressed by first applying a wavelet
transform to the image intensities and then encoding using
an adaptive arithmetic encoder.
Moving images are compressed using either unconditional
or conditional refreshes.
8
CA 02228679 1998-02-04
Under the unconditional refreshing scheme each frame is
compressed as though it were a still image and not part of a
video. This is the compression scheme that is preferred for
security applications since in those applications image
quality is more important than the transfer rate.
Under the conditional refreshing compression scheme,
most frames are first compared to the preceding frame. The
difference between the current frame and the preceding frame
is calculated, compressed and transmitted. If there is
to little difference between the current frame and the
preceding frame, the amount of information that needs to be
transmitted over the network is minimal thus significantly
reducing bandwidth consumption in the case of scenes that
are largely static. At the same time, the apparent transfer
speed is significantly increased. Sending key frames
accomplishes resynchronization with the scene at periodic
intervals. Key frames are treated as still images and are
compressed without reference to surrounding frames.
The last operation in the processing pipeline prior to
20 transmission is bit packing of the encoded data.
The remote station maintains a record of connections
including host IP and port number, user identity, time and
duration of connection, and time and duration of connection
to each monitoring device to which the client has connected.
In addition, a record is kept of alarm related events. The
time of onset and the duration of alarms are recorded. Other
information that is associated with an alarm is the camera
that triggered the alarm and the zone (an operator defined
sub-field of the entire field of view of the camera) in
30 which the alarm condition was detected.
For applications to security, the principal criteria of
the remote or capture station are reliable operation and
image quality.
The operator station is software used by security
operators and other users to connect to the remote stations
9
CA 02228679 1998-02-04
that host the monitoring devices. The software used by
privileged users to connect to remote stations extends the
operator station. To distinguish it from the software used
by ordinary users, the software used by privileged users
will be called a management station. The operator station
has limited functionality. From a operator station, a user
can login to a remote station, select devices (cameras) to
monitor, receive and view data from the monitoring devices,
record data received and logout from the remote station. The
l0 operator station will receive alarms that are generated at
the remote sites being monitored by the operator. Once
received, the alarms can be handled in different ways. In
particular, a facility is provided to enable the operator to
capture event video and retransmit to another location.
During login, the only actions enabled on the operator
station are login and quit. The user is prompted to enter a
username and password.
Once entered, the login information is sent to the
remote station for validation. If the login attempt is
20 successful, a list of cameras that the operator station can
connect to is sent to the operator station from the remote
station. Once received, a select button is enabled on the
operator station. If no response is received from the login
station within a specified period of time, the login times
out. If the login attempt is rejected, no other actions are
enabled at the operator station except those that are
enabled on start up, quit and login.
On successful login to a remote site, the user can
select from a list of cameras to monitor. The list that is
30 presented to a particular user will depend on the login
information that was sent during connection establishment.
Once a camera is selected, video output from that camera
will be sent to the operator station until the camera is
deselected. A single operator station can open connections
to multiple cameras hosted on the same remote station. The
CA 02228679 1998-02-04
upper limit will be set by the operator station and/or by
the processing load on the remote station. Privileged users
will be given priority over ordinary users in the event that
video channels are closed because of the load on the remote
station.
For different applications it may be necessary to
implement other image operations. Among those that have been
considered are edge enhancement and smoothing.
The maximum number of video windows that can be open
l0 simultaneously and the manner of switching between views if
a11 windows cannot be simultaneously displayed are
implementation issues.
The management station is an operator station with
enhanced capabilities. Like the operator station, the
management station can connect to multiple remote stations.
Once connected, many more actions are enabled on the
management station than are enabled on the operator
stations. Privileged users use the management stations and
the following describes functionality of the management
20 station that is not inherited from the operator station. In
general, the management software can connect to multiple
sites, query the status of devices hosted at remote sites,
monitor the state of remote sites and configure and control
remote sites and the devices associated with them.
Each management station has access rights to a set of
remote stations. From the management station an authorized
user can login to any of the remote stations that are linked
to the management station. The login process is identical to
the sequence of steps required to login to a remote station
30 from an operator station.
The management station permits authorized users to
configure and control remote devices hosted by remote
stations. Some of the management functions that are specific
to cameras are listed below.
11
CA 02228679 1998-02-04
A privileged user will be able to start and stop
devices from the management station. In contrast, ordinary
users can only elect to receive output from devices that are
live. Ordinary users do not have the power to activate or
deactivate devices that are hosted on a remote station.
A privileged user can configure a camera to scan a
rectangular sub-region of the entire field of view. This
facility will be useful in cases where events of interest
are known to occur in one region and that activity in. the
l0 remainder of the scene is of no consequence.
The rate at which frames are captured from the device
(capture rate) and the rate at which frames are delivered to
the network (transmission rate) are both remotely
configurable.. A possible use for this would be to capture
frames at a constant rate but deliver frames at a low rate
unless an unusual condition is detected. In case an unusual
condition is detected the frames can be delivered as fast as
they are being captured. This strategy would conserve
bandwidth for transmission of data during the critical
20 periods when an unusual condition has been detected.
The management station will permit authorized users to set
properties of the cameras that are being monitored such as:
brightness, contrast and white balance. The resolution of
the image that is being sent can also be varied within
certain limits. The upper limit is determined by the
capabilities of the camera. The lower limit is determined by
properties of the human visual system. The reason for
allowing variation of the resolution is the same as the
reason for allowing variation of the frame rate. In order to
30 conserve bandwidth, frames can be transmitted at a low
resolution if an unusual condition has not been detected. As
frames are continuously cached at a high resolution no
information is lost if an unusual condition is detected.
When an unusual condition has been detected frames will be
12
CA 02228679 1998-02-04
transmitted at a high resolution until the unusual condition
disappears.
The management station can implement various types and
levels of security. IP masks can be defined for each remote
station that the management station is linked to. By
defining a mask, connections from hosts with an IP that does
not match the mask are refused. This strategy will not
prevent unauthorized access by IP spoofing.
The management station can obtain information about
clients that are connected to the remote station. This
information will include host IP and port number, user
identity, remote devices that are being monitored and time
and duration of connection. In the event that suspicious
activity is detected, the management station can terminate
the connection to the suspect client. In addition, the
management station can take control of the TCP connection to
the client and thereby gain remote access to the operator
station.
The management station can monitor live connections. In
addition, it can view the logs that are maintained at each
remote station.
The management station can remotely monitor the state
of each device associated with each of the remote stations
to which it is linked. The principal purpose of this
facility will be to perform remote diagnostics in the event
that a device malfunctions.
As part of the monitoring process, the management
station will send a heartbeat to each of the remote devices
that are live. If a device fails to respond, the device is
offline for an unknown reason. This would constitute an
exceptional condition requiring further action.
The management station can control the image-processing
pipeline, i:e. the various operations that are applied to a
frame before it is sent over the network. The management
13
CA 02228679 1998-02-04
station can specify the filter that is to be used for
smoothing an image to remove random noise.
The management station can specify the algorithm that
will be used to remove noise introduced by the scanning
process and to remove variations in the image caused by
regular fluctuations in ambient conditions such as
fluorescent lighting.
The management station can specify acceptable tradeoffs
between compression ratio and speed for compression of still
and moving images from each camera.
The management station allows authorized users to set the
parameters that control the mode of operation of motion
detection on the remote stations. The parameters that can be
set are:
a) sensitivity level;
b) definition of sub-regions of the field of view; and
c) motion estimation.
Each parameter can be assigned a value manually or
configured automatically and dynamically.
Motion detection in the last instance demands that a
decision be made as to whether a change that has been
calculated between one frame and preceding frames is
significant or not. Some way is needed for measuring the
significance of a difference. This is the threshold value.
Differences that exceed the threshold value are deemed to be
significant whereas differences that do not exceed the
threshold value are not deemed to be significant.
The management station permits authorized users to
define zones of interest that are rectangular sub-regions of
the entire field of view of a particular camera. The
threshold for motion detection can be defined independently
for each zone. This permits the management station to focus
attention on zones in which exceptional events are most
likely to occur.
14
CA 02228679 1998-02-04
In addition to motion detection, primitive judgements
concerning the direction of motion and the size of the
projected area that is moving can also be made. However,
computing this information places an additional load on the
processor of the remote station. An authorized user may
decide to enable or disable this facility on the basis of
other factors.
The management station can record images from any of
the cameras to which it has access. The management station
l0 will archive recorded sequences and will have the capacity
to playback previously recorded sequences.
The database system is designed to record significant
events that occur while the application is running, as well
a-s supply a central repository of information needed by the
application. Additionally, the database system serves to
manage video recordings of alarms to allow easy and
consistent access by users of the software to these
recordings.
The structure of the surveillance system consists of
20 stations which have cameras connected to them (servers).
These stations are connected to other computers called
clients which receive information from the servers when
motion is detected. Additionally, video can be sent from a
server to these clients on a real-time basis.
The database system is based on a piece of software
that runs on a computer also connected to the other stations
of the system. This is called the database server. It
provides centralized access to the following pieces of
information:
- User logs in to camera server
- Connection closed
- Alarm detected
- Camera server startup
- Camera server shutdown
CA 02228679 1998-02-04
- Video file deleted
These events are recorded on the camera servers and clients
and stored in local log files. These files are periodically
downloaded from these servers and clients by the database
server and analyzed to derive a coherent record of events
that occurred in the system. These records are maintained on
the database server.
The database system also manages a central repository
to of video files and records what videos are archived and may
be deleted from this storage area. Since these video files
are initially stored on the camera server where the alarm
occurred, the database server downloads videos from the
camera servers on a periodic basis. It derives knowledge of
what video files exist from the log files mentioned earlier.
Because these files are potentially very large the database
server only downloads them when activity on the network is
at a minimal level to ensure it does not interfere with
other operations on the network.
20 The users of this software are responsible for
archiving the videos stored on the database server on a
regular basis using the backup medium of their choice. This
process is needed because disk space is limited and the size
of these files means that it can be used up quite quickly.
Although the database server does not handle the backup
process, it manages the video files to allow the user to
specify which files are false alarms (and may be deleted ,
which ones are to be kept on the database server, and which
ones are to be archived. By moving each of these files to
30 different directories, accidental deletion of files can be
avoided. The database server provides an interface to manage
this.
The database server also maintains a record of what
camera servers are at a given location, and the properties
of each of those cameras are stored. Certain information is
16
CA 02228679 1998-02-04
needed by a client in order to connect to a camera server.
This is also maintained by the database server which
supplies this information to the clients on demand.
When an alarm occurs the client will supply information
about who to contact to inform of the alarm, if necessary.
This information is maintained on the database server and
can be sent on demand to the client if it requests it.
A11 the information stored by the database can be browsed
and altered by users responsible for supervising the
l0 operation of the surveillance system. A software application
allows these users to view a11 these recorded events and the
videos associated with them (assuming these video files are
still on the server and have not been archived or deleted by
the users).
The database will also occasionally need to be changed
to reflect modifications to the system (a new camera is
added to a camera server, for example). The database server
also provides an interface to modify this information by
authorized users.
2o While particular embodiments of the invention have been
described and illustrated it will be apparent to one skilled
in the art that numerous changes can be made without
departing from the basic concept. It is to be understood
that such changes will fall within the scope of the
invention as defined by the appended claims.
17
