Note: Descriptions are shown in the official language in which they were submitted.
CA 02475387 2004-07-21
Embedded System Administration
FIELD OF INVENTION
[0001] The present invention relates to computing systems and more
particularly to a
system and method for system administration.
BACKGROUND OF THE INVENTION
[0002] Information Technology organizations are frustrated with the cost of
server
downtime, the rising frequency of malicious attacks and programs, and the
significant
operational expenditures required for managing their systems. The complexity
and
cost of server management is increasing. The server management tasks include:
fault
1o diagnosis, software upgrade and patching, backup and recovery, monitoring
and
resource reallocation. Many architectural solutions to the server management
problem
have been devised with a variety of hardware and software components; however
complete delegation of management authority to the server has not yet been
achieved.
High levels of human intervention are still required far server management.
15 [0003] IBMTM has established a new paradigm in computer networks called
"autonomic computingTM". In short, autonomic computing is an approach to "self
managed" computing systems with a minimum of human interference. A true,
complete autonomic system requires.that computer systems be reconstructed from
software at the high end to hardware components in every computer. In addition
to
2o IBM, Microsoft~M, HPTM, DelITM and SunT~ have begun their own autonomic
computing initiatives. While "total" autonomic computing may be the ideal its
successful and complete implementation will take years: There are alternative
approaches that will help increase their systems' uptime, while lowering their
systems'
Total Cost of Ownership (TCO).
2S [0004] The quickest way to begin reducing the TCO is to attack the area of
"least
value-add", namely labor. The two most common existing approaches to reducing
labor costs are adding additional hardware or additional software.
[0005] With regard to hardware based approaches so called "watchdog" cards are
inserted into host computers. A separate software pxocess runs on the host,
and
3o communicates heartbeats to the watchdog card across the bus. After not
receiving a
-1-
CA 02475387 2004-07-21
heartbeat for a pre-set amount of time, an on-board timer will expire, causing
the
watchdog card to cycle the power of the host. Remote Management Cards (RMCs)
are
add-in server cards that allow system administrators (SYSADMINs) independent
access to the server. The key problem with RMCs is that a SYSADMIN is required
to
intervene if the card detects a problem, thus mitigating any cost savings.
Furthermore,
RMCs are not autonomic. RMCs are able to scan vast amounts of information, but
they are unable to make decisions on their own. RMCs are generally higher-
power
cards with more powerful CPU (e.g. 200MHz CPU). This allows a remote system
administrator to log into a host machine, and redirect KVM (Keyboard, Video,
1o Mouse) across the remote connection. This facilitates configuration and
troubleshooting as though the admin was sitting directly in front of the
actual host.
Host video and keyboard connectors are actually inserted into the remote
management
card ports. Most remote management cards offer watchdog timers and
functionality.
[0006] Figure 1 is a schematic diagram of a current network administration
system. A
server system 1 includes a server chassis 2, a server 4, a Network Interface
Card (NIC)
6 and a lights out card 10. Communications in the server chassis 2 are
provided for by
communications bus 16. The lights out card I O and network administration
system I2
are elements of a management system 8. The lights out card 10 provides local
control
capability, including power, and intercepts for the keyboard, video and mouse.
The
lights out card 10 further gathers data and invokes simple actions initiated
by a remote
user. Monitoring of the server 4 is performed by a Host Management Agent (HMA)
14. The observations made by the HMA 14 are sent to the Network Administration
System 12. The system 12 receives telemetry data from and provides policy
management for the server 4. It will be apparent to one skilled in the art
that there
may be more than one server and one or more peripheral cards in the sever
chassis 2.
The system 12 further determines appropriate actions based on the collected
information and provides information to a user for evaluation.
[0007] Host Monitoring Agents (HMAs) are software-oriented products that are
relatively inexpensive and are typically based on industry standards. HMAs
reside on
3o a server's primary hardware and operate in conjunction with the host
operating system.
However, the major drawbacks of HMAs are that they impact host performance
(HMAs take away CPU cycles from the server to run monitoring software), are
-2-
CA 02475387 2004-07-21
susceptible to host faults (if the host crashes the HMA crashes), and have
narrow
monitoring visibility (do not monitor internal functions of operating systems
and
applications). This category of products features softwaxe agents that monitor
the host
systern(s). Operational information collected is passed 1:o a separate
management
station across the network, where it passes through several "filters". Upon
successful
match by a filter, a response action is initiated, being either a correction
action on the
host (e.g. restart failed process) or notification (e.g. by email) ~f a human
system
administrator.
[0008] There have been some recent technologies entering the market that allow
a
1o server or desktop to create a protected partition on the host hard drive.
In the event of
hard drive corruption, the user can essentially replace the master partition
with the
copy in the protected partition.
[0009] There is a need for providing new hardware and software that addressed
the
issues left unresolved, thereby increasing the availability and reducing the
TCO of
15 servers.
SUMMARY OF THE INVENTION
[0010] The present invention relates to an embedded administration system.
[0011 ] It is an obj ect of the invention to obviate or mitigate at least one
of the
disadvantages of existing administration systems.
20 [0012] According to an aspect of the invention an administration system for
use
within a server system is provided. The server system having a server that
provides
host management functions and the server system being able to accept computer
cards
inserted therein. The administration system comprises a computing system that
is
inserted in the server system, the computing system having a controller that
assumes
25 control over the communications bus.
[0013] According to another aspect of the invention an administration system
for use
within a server system is provided. The server system has a server that
provides host
management functions, the server system being able to accept computer cards
inserted
therein. The administration system comprises a computing system that is
inserted in
-3-
CA 02475387 2004-07-21
the server system, the computing system having a controller for performing
administration, wherein the controller comprises; a server monitoring unit for
monitoring the server and making observations, an analyzing unit for analyzing
the
observations, a determining unit for determining if a policy violation has
occurred, a
management unit for assuming host management functions from the server, and an
executing unit for executing specific actions.
[0014] According to another aspect of the invention a method of performing
administration of a server system having a server that performs host
management
functions by an administration system located within the server system is
provided.
to The method comprising the steps of; monitoring the sen~er, making
observations
thereof, analyzing the observations, determining if a policy violation has
occurred,
assuming host management functions from the server, amd executing specific
actions.
[0015] According to another aspect of the invention a server system capable of
having
at least one computer card inserted therein is provided. The server system
comprises a
15 server, the server providing host management functions for the server
system, a
communications bus providing communications within the server system, and an
administration system that is inserted in the server system, wherein the
administration
card is capable of assuming host management function for the server system.
[0016] According to another aspect of the invention a server system capable of
having
2o at least one computer card inserted therein is provided. 'The server system
comprises a
server, the server providing host management functions for the server system,
a
communications bus providing communications within l:he server system, and an
administration system that is inserted in the server system, wherein the
administration
system assumes control over the communications bus.
25 [0017] This summary of the invention does not necessarily describe all
features of the
invention.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] These and other features of the invention will become more apparent
from the
following description in which reference is made to the appended drawings
wherein:
-4-
CA 02475387 2004-07-21
[0019] FIGURE 1 is a schematic diagram of a server system with a current
server
management system;
[0020] FIGURE 2 is a system block diagram of a server system with an
administration
system according to an embodiment of the invention;
[0021] FIGURE 3 is a high level schematic diagram of an administration card
according to an embodiment of the invention;
[0022] FIGURE 4 is a block schematic diagram of the communications channels
between a server and an administration card according to an embodiment of the
invention;
to [0023] FIGURE 5 is a block software architecture diagram according to an
embodiment of the invention;
[0024] FIGURE 6A is a block diagram of an autonomic controller according to an
embodiment of the invention;
[0025] FIGURE 6B is a schematic diagram of an autonomic controller according
to an
15 embodiment of the invention;
[0026] FIGURE 6C is a schematic diagram of a server system with an
administration
system according to an embodiment of the invention;
[0027] FIGURE 7 is a high level flow chart of life cycle operation of the
autonomic
controller according to an embodiment of the invention;
20 [0028] FIGURE 8 is a flow chart of initialization of the autonomic
controller system
according to an embodiment of the invention;
[0029] FIGURE 9A is a flowchart of the operation of the autonomic controller
according to an embodiment of the invention;
[0030] FIGURE 9B is a flowchart of the operation of the autonomic controller
25 according to an embodiment of the invention;
-5-
CA 02475387 2004-07-21
[0031 ] FIGURE 10 is a flow chart of the shutdown process of the autonomic
controller according to an embodiment of the invention;
[0032] FIGURE 11 is a schematic diagram of management of the card according to
an
embodiment of the invention;
s [0033] FIGURE 12 is a schematic diagram of the lifecycle of a module
according to
an embodiment of the invention;
[0034] FIGURE 13 is a message interaction diagram illustrating server control
by the
card according to an embodiment of the invention;
[0035] FIGURE 14 is a message interaction diagram according to an embodiment
of
to the invention;
[0036] FIGURE 15 is a message interaction diagram according to an embodiment
of
the invention;
[0037] FIGURE 16 is a message interaction diagram according to an embodiment
of
the invention;
15 [0038] FIGURE 17 is a message interaction diagram according to an
embodiment of
the invention;
[0039] FIGURE 18 is a message interaction diagram according to an embodiment
of
the invention;
[0040] FIGURE 19 is a message interaction diagram according to an embodiment
of
20 the invention;
[0041] FIGURE 20 is a message interaction diagram according to an embodiment
of
the invention;
[0042] FIGURE 21 is a message interaction diagram according to an embodiment
of
the invention;
25 [0043] FIGURE 22 is a message interaction diagram according to an
embodiment of
the invention;
-6-
CA 02475387 2004-07-21
[0044] FIGURE 23 is a message interaction diagram according to an embodiment
of
the invention;
[0045] FIGURE 24 is a message interaction diagram according to an embodiment
of
the invention;
[0046] FIGURE 25 is a message interaction diagram according to an embodiment
of
the invention;
[0047] FIGURE 26 is a message interaction diagram according to an embodiment
of
the invention
DETAILED DESCRIPTION
to [0048] The present invention provides a system and method for embedded
system
administration. Namely, it provides a system and instructions appropriate for
assuming control of the host of a server in response to the occurrence of one
or more
events. Upon assuming control the system assumes host management functions.
[0049] Figure 2 is a block diagram of a server system according to an
embodiment of
the invention. The server system 100 includes a server chassis 102 that has a
plurality
of slots appropriate for accepting peripheral cards therein. These slots,
including a
slot comprising the server 104, are interconnected with communication buses or
channels. These buses are appropriate for providing communications between the
components connected thereto. In the current embodiment the communications bus
116 is a PCI bus. It will be apparent to one skilled in the art that the
communications
bus 116 may be any communications bus appropriate for providing communications
between computing systems including PCI-X, USB, IPMB, Ethernet, TCP/IP
connection and any other means of communication as would be apparent to one
skilled in the art.
[0050] The server chassis 102 incorporates the server 104 and one or more
peripheral
cards that provide for particular functionality. In the current embodiment the
server
chassis 102 incorporates an NIC card 106 and a card 202 that is appropriate
for
providing embedded system administration. The card 202 will be referred to as
the
Intelligent System Administrator on-a-Card (ISAC). The server 104 is
appropriate for
CA 02475387 2004-07-21
performing all hosting functions for the peripheral cards provided in the
server chassis
102. The server 104, NIC 106 and ISAC 202 communicate with one another using
the
PCI bus 116.
[0051] The ISAC 202 is a central element of the system of the present
invention. The
ISAC 202 includes components appropriate for it to act as a single board
computer.
The ISAC 202 is inserted into an available bus slot of the server chassis 102.
The
ISAC 202 is capable of assuming the role of a control plane while subsuming
the
functions of the network administration system 112 and lights out card 110
functionality, as shown in Figure 1. The ISAC 202 communicates with the server
104
1o using the PCI bus 116. The ISAC 202 supports software that provides
processing
functions for observations made on the server 104.
(0052] Figure 3 is a schematic diagram that presents further detail of the
ISAC 202
and of the communications channels used to communicate with the ISAC 202. A
compute module 302 comprises much of the hardware that allows ISAC 202 to
function as a single board computer (SBC). In the current embodiment the
compute
module 302 comprises an XSCALETM PXA255 processor, 64 MB of DRAM, 64 MB
of FLASH and 8 Mb of ROM. The ISAC 202 also has a video module 304, which
includes an ATI RageTM chip set and video RAM. Two PCI bridge chips facilitate
bi-
directional communication between the ISAC 202 and tile server 104. A non-
2o transparent PCI bridge 306 and a transparent PCI bridge 308 form the bridge
between
the compute module 302 and the communications bus 116. The non-transparent PCI
bridge 306 ensures that the server 104 cannot write into the memory space of
the
compute module 302. The transparent PCI bridge 308 allows video memory of the
server 104 to be written to the video memory of video module 304. The ISAC 202
further comprises an NIC interface 310 and a USB interface 308. Standard
direct
memory access (DMA) mechanisms and interrupt handling techniques familiar to
those skilled in the art of video card and network interface card programming
are used
in the current embodiment of the invention in order to transfer data from the
address
space of the server to that of the card and vice versa.
[0053] Figure 4 is a schematic diagram of the communications channels present
between the ISAC 202 and the server 104. As indicated in Figure 4 there are
_g_
CA 02475387 2004-07-21
numerous communications channels that can be used for communication with and
for
control of the server 104, communication with System Administrators
(SYSADMINS) and other applications in the current embodiment. T'he
communications bus 116 provides for communication between the ISAC 202 and a
host service executing on the server 104. Another communications bus 402 is
used as
an alternative communications path to a host service operating on the server
104. Port
forwarding as provided by the operating system running on the ISAC 202 allows
application level protocols such as FTP and HTTP to be redirected to the
server 104
using either a communications bus 402 or the PCI bus 116. Port Forwarding is a
l0 combination of routing by port combined with packet rewriting. A convention
router
examines the packet header and dispatches the packet on one of it's other
interfaces,
depending on the packet's destination address. Port Forwarding examines the
packet
header and forwards it on to another host (after a little header rewriting)
depending on
the destination port. In the current embodiment the bus 402 is a TCPIIP
connection. It
will be apparent to one skilled in the art that a PCI bus may also be used for
the bus
402. An alternate computer network 404 provides an interface used by the
management software (Figure 11) for software distribution and control. In the
current
embodiment of the invention, the network 404 is implemented using a private
LAN/WAN through a network interface chip on the ISAC 202 and through a USB
dial-out modem. An external notification 406 of state is achieved through
light
emitting diodes on the back of the ISAC 202. Power control of the server 104
is
achieved by electronic simulation of pressing the power switch on the server
104
using a fourth communications bus 408. In an alternative embodiment
notifications
such as audible indicators of state are provided using a speaker of the server
104
[0054] Another element of the system of the current invention is software.
Figure 5 is
a block diagram of the basic elements of software residing on the ISAC 202 and
the
server 104. The elements of software, that are pertinent to the current
discussion,
executing on the server 104 implement the functionality of host management
agent 14
as shown in Figure 1. There are three software components that reside and
execute on
3o the server 104 and provide for the provision of hosting functions by the
server 104: a
host service 502, a bus adapter 504 and a PCI driver 506. The host service 502
has
three main functions. The first function is the acquisition of perform;~nce
information
_g_
CA 02475387 2004-07-21
from managed objects that are part of the server 104, the operating system
executing
on the server 104 or applicatians hosted by it. Acquisition of infarmation is
achieved
by polling and by event notification with polling frequency being determined
by
software running on the ISAC 202. The second function is to act as a proxy for
the
ISAC 202, running commands that have been generated by software running on the
ISAC 202. The final function of the host software is to facilitate upgrade of
the ISAC
202 software. The host service 502 communicates with software on the ISAC 202
by
invoking functions within the bus adapter 504. The bus adapter 504 its an
implementation of an abstract communications interface providing bus
independence
1o and presents an OSI layer 7 interface to the host service 502. Read and
write
functions are provided. The bus adapter 504 communicates with the PCI driver
506.
[0055] The ISAC 202 supports five distinct softwaxe components: a data store
508, a
web server 510, a controller 512, a bus adapter 514 and a PCI driver '.i 16.
In the
current embodiment of the invention controller 512 is an autonomic controller.
The
functions of the bus adapter 514 and PCI driver 516 are equivalent to those
provided
by the analogous components on the server 104. The web server 510 provides an
interface to allow authenticated and authorized users control of the autonomic
controller 512. In the current embodiment a Common Gateway Interface (CGI) is
used for interaction. It will be apparent to one skilled in the art that
servlet interfaces
2o could also be used for interaction. The data store 508 provides persistent
storage for
information generated by the autonomic controller S 12 or by user interaction
with the
web server 510. In the current embodiment of the invention the data store is a
file
system maintained in flash memory. The autonomic controller 512 is a
programmable
software platform that executes policies intended to improve the availability
and
security of the server 104, its operating system and hosted applications.
[0056] The bus adapter 514 provides a hardware abstraction layer, presenting
an OSI
layer 7 communications interface to the autonomic controller 512. The
communications interface provides 8 independent channels: zero through seven.
Each
channel represents an independent session. Each session has an associated
heartbeat;
3o failure to respond to two heartbeats results in the autonomic controller
512 being
notified. The communications interface is preemptive, and priority driven
based upon
channel number; the lower the channel number, the higher the priority of the
data sent
-10-
CA 02475387 2004-07-21
across it. Channel zero is a control channel; it is used during initialization
to ensure
protocol compatibility between the ISAC 202 and the server 104. In the current
embodiment of the invention, failure to agree on protocol version results in
negotiation of a software upgrade that is obtained from an external management
console. In the current embodiment of the invention software upgrade is
achieved by
replacement of a dynamic link library (DLL) on the server side followed by a
service
restart. In the current embodiment of the invention software upgrade is
achieved by
replacement of an executable or object class on the ISAC 202 and, if required,
an
application restart. XML is used to encode the payload of data transferred
between
1o the server 104 and the ISAC 202 and vice versa.
[0057] The autonomic controller 512 is a central aspect of the software of the
current
embodiment of the invention. As shown in the schematic diagram of Figure 6A
the
autonomic controller 512 has a Virtual Machine (VM) 602 which provides an
execution environment for the Policy-Processing Engine (PPE) 604 which
interprets
policies. The VM 602 incorporates an embedded scripting framework. The PPE 604
incorporates pre-programmed policies with default embedded "Best Practices"
for
server and application management. The current embodiment incorporates a LUA
scripting engine; however, it will be apparent to those skilled in the art
that other
scripting languages could be supported. In addition to the server 104 the
autonomic
controller 512 interacts with a management console 606. In the current
embodiment
the management console 606 :is a PC based system from which a user can
interact with
the autonomic controller 512. As such management console 606 provides for the
user
to view information provided by the autonomic controller S 12 and allows the
user to
transfer information to the autonomic controller 512.
[0058) Figure 6B is a block diagram showing additional units within the
autonomic
controller 512. The autonomic controller includes a server monitoring unit
607, an
observation analysis unit 608, a determining unit 609, a management unit 610
and an
executing unit 611.
[0059] The autonomic controller 512 is capable of managing the server 104, its
operating system and hosted applications. A flow chart of the operation of the
autonomic controller 512 according to an embodiment of the invention is
presented in
-li-
CA 02475387 2004-07-21
Figure 6C. The autonomic controller 512 monitors the server 104 at step 613.
At step
612 the autonomic controller 512 analyzes the observations. At step 614 it is
determined whether a policy violation has occurred. If no violation has
occurred the
autonomic controller returns to step 613. If a policy violation has occurred
the
autonomic controller plans the workflow. At step 616 it assumes control of
host
management functions and then executes specific actions at step 618,, where
host
management functions include control over the communications bus 116. Thus,
the
autonomic controller 512 controls the communications bus 116. Host management
functions are returned to server 104 at step 620.
[0060] The above capability is provided through the execution of Intelligent
Control
Loops (ICLs). The ICLs are broken down into four distinct parts: monitor,
analyze,
plan, and execute. ICLs are based on modules that describe policies; carrying
out
tasks as efficiently as possible based on high-level, declarative statements
of intent (or
state). Policies are implemented using software that is adaptive and can be
modified
in real time using the web server interface 510. Policies are driven by event
input that
is derived from observations made on the server 104, its operating system and
its
hosted applications.
[0061] With regard to the above the deployed unit of autonomic behaviour is a
module. The module represents a set of one or more policies that together
implement
one or more ICLs. The policies are implemented as independent sets of rules.
They
are the smallest unit of autonomic behavior that is understood by the 'VM 602
and
evaluated by PPE 604.
[0062] Observations are typically measurements of resource consumption; e.g.
CPU
utilization. ICLs are responsible for the collection of information from
server 104 by
making observations on managed objects, processing and filtering the raw
observations and using the processed observations to make decisions concerning
the
state of server 104 being intelligently managed. Processed observations are
aggregated until an event is considered to have been triggered; e.g. the CPU
utilization
has exceeded 90% for 60 seconds and paging space is low. Actions are then
executed;
making adjustments as needed -installing missing or compromised software,
adjusting
server control parameters, restarting failed server elements, adjusting
current
-12-
CA 02475387 2004-07-21
workloads, and taking resources offline to prevent system corruption through
intrusions. The set of actions undertaken constitutes workflow; i.e. it is
intended that
all actions complete successfully. ICLs are designed and implemented offline
using a
development environment and distributed using system management tools and
embedded in system resources. A forward chaining rule based inference engine
that
uses the Rete algorithm is used to determine context based upon events
received.
[0063] The deployed unit of autonomic behavior is a module. A module
represents a
set of one or more policies that together implement one or more ICLs. Users
may
extend modules. Extension takes three forms: customization, configuration and
l0 programming. Modules represent sets of "Best Practices" captured firom
MicrosoftTM
and other vendor product knowledge bases as well from experienced SYSADMINS.
LUA was chosen for policy implementation because of its compact embedded
footprint and its similarity to traditional scripting languages. A rules;-
based embedded
environment has also been constructed that compiles down to Java byte code
thereby,
15 making it possible to make use of any available class supported by the
virtual
machine. An embodiment of the invention uses the J9 JVM implementation using
the
CDC configuration and Foundation Profile for the autonomic controller 512.
[0064] The autonomic controller 512 has several functions, implemented using a
service-oriented framework in the current embodiment of the invention. The
principal
2o services are: host communications, management console communications,
module
and policy management, short term and long term memory management, alarm
management, notification, health monitor management, managed object
management,
properties management, soft'vare management, logging, audit, security,
scheduling,
and task management. All services have an associated lifecycle, meaning that
they
25 can be initialized, started and stopped. Certain services are managed
services,
meaning that they are monitored for operational integrity. The host
communications
service performs three functions: management of the lifecycle of
communications
between the ISAC 202 and the server 104, monitoring its integrity through a
heartbeat
mechanism and reliably transferring data between the two. The management
console
3o communications service provides an abstract interface for the
communications
between the ISAC 202 and the management console 606. In the current embodiment
of the invention the HTTPS protocol is used for transport, with XML being used
for
-I3-
CA 02475387 2004-07-21
payload description. Module and policy management services ensure that modules
and policies can be loaded from persistent storage. In the current embodiment
of the
invention a file system is used with a known, rooted directory structure. Each
directory within the modules directory is considered a module to be loaded
into the
running autonomic controller 512. Loading requires that several files be read,
and
objects created within memory of the compute module 302. In the current
embodiment of the invention, all loaded modules conform to the Modules
interface
and all policies associated with a module conform to the Policy interface.
Module and
policy interfaces refer to software contracts; i.e. behavior that the
implementations
1o must support. Once loaded, modules and policies are initialized, started
and then
registered with the managed object management service in order that the
management
console 606 can interact with them. Short and long term memory services allow
objects to be remembered for the autonomic controller session or indefinitely
respectively. Alarm management provides the facility for alarms to be
externally
communicated. Alarms are generated as a result of policy actions being
executed.
Practitioners skilled in the axt of Network Management will understand that
alarms
have SET and CLEAR states, the former being created when a policy violation is
observed, the latter being generated when the server 104 returns to an
acceptable state.
The expected semantics for alarm usage is that a SET can only be followed by a
2o CLEAR. The notification service provides an abstract interface for sending
notifications to applications off of the ISAC 202. In the current embodiment
of the
invention, e-mail, window pap-ups and instant messaging notifications are
provided.
Notifications are generated as a result of policy actions being executed. The
health
monitor management service provides a facility that allows a user to see an
iconic
2s representation of the health of the server 104, some aspect of the
operating system or
its hosted applications. In the current embodiment of the invention a SYSADMIN
can
view the iconic representation using the web interface 510. Health monitor
indicator
changes are generated as a result of policy actions being executed. The
managed
object management service allows any object within server 104 to be managed by
the
3o management console 606. Interfaces to get and set attributes as well as
public invoke
operations on managed objects are provided. The management console 606 may
also
register for events of interest to be forward when changes to managed objects
occur.
-14-
CA 02475387 2004-07-21
[0065] In an alternative embodiment the managed object management service will
use
a Common Information Model Object Manager (CIMOM) fox its implementation.
The managed obj ect management service interacts with the security and audit
services
in order to ensure that only authorized individuals can access managed
resources and
that any such access can be traced. The properties management service listens
for
changes to properties that apply to objects being managed. Properties are
stored
persistently in the file system. When an object with properties is loaded, the
properties
management service listens for changes to the file. If detected, the object is
notified
that it should reload its properties. The software management service manages
the
l0 process of updating a class in the system; only services may be updated in
the current
embodiment of the invention. The software management service ensures that the
affected service is stopped, its current state retained, a new service object
created, and
the retained state transferred to the new object. Finally, the service is
restarted. The
logging service allows time-stamped information, events and loggable objects
to be
written to persistent storage. The audit service is a special logging service
that is
intended to store actions of security interest. The security service ha.s the
responsibility of answering authorization questions; namely, is a particular
user
allowed to perform a specific action on a resource? The scheduling service
allows an
operation to be scheduled for some time in the future either on a one-time or
recurring
2o basis. Finally, the task management provides a pool of threads for use by
all other
services. This service avoids the need to create threads every time concurrent
activity
is required.
[0066] Figure 7 is a flow chart of the basic states of the operation of the
autonomic
controller 512. The autonomic controller 512 has three main states:
initialization 702,
autonomic control 706 and shutdown 710. The initialization 702 and shutdown
710
states are transient with the autonomic controller spending the maj ority of
its time in
the autonomic control state 706.
[0067] The autonomic controller 512 enters the initialization state when the
ISAC 202
is rebooted. If the initialization process is successful, a transition is made
to the
autonomic control state 706. If the initialization process is unsuccessful,
several
further attempts will be made to initialize the autonomic controller 512.
Policy
activity may result in the need to shutdown and restart the autonomic
controller 512
-15-
CA 02475387 2004-07-21
a .
and possibly reboot the ISAC 202. In this case, the autonomic control state
706 is
exited. If shutdown can proceed, the shutdown state 710 is entered and
shutdown
executes. If not, the autonomic control state 706 is re-entered until such
time as
shutdown may proceed.
[0068] Figure 8 is a schematic diagram of the autonomic controller's
initialization
process. There are five main steps in the initialization process. A start PCI
communications step occurs at step 802. There are generally five sub-processes
in the
start PCI communications step 802. First, an open request is made. 'This API
initializes data structures within the bus adapter 514; the PCI driver 516 is
contacted
and channel zero is used in order to send a message containing version
information of
the PCI driver 310. This information is received by the PCI driver 506. This
is a
blocking call as no activity can occur on the bus 116 until an agreed version
has been
negotiated. If the versions agree, a channel number in the range 1-7 is
returned.
Failure is indicated by a negative integer. If version numbers do not agree,
and the
ISAC 202 is to upgrade, the PCI driver 516 is reset at step 804, the
management
console 606 is contacted and the upgraded driver is downloaded. The old driver
module is then unloaded and the new driver loaded. A new attempt t~o start PCI
communications then occurs i.e. returned to step 802. Failure to start PCI
communications after an upgrade causes a soft restart of the autonomic
controller 512.
Failure to start PCI communications after an upgrade and a restart of the
autonomic
controller 512 causes an alarrra to be sent to the management console 606 and
the
ISAC's 202 operating system to be rebooted. If an upgrade of the PCI driver
506 is
required the management console 606 is contacted and the upgraded driver is
downloaded. The old driver module is then unloaded and the new driver loaded.
A
new attempt to start PCI communications then occurs i.e. returned to step 802.
The
host service 502 is restarted. Failure to start PCI communications after an
upgrade to
the PCI driver 506 causes the server 104 to be rebooted.
[0069] Once PCI communications have been established, the services stored in
the
services.ini properties file in the root directory of the application
installation are read
in at the startup services step 808. This file has the format "name=service-
class". An
instance of each service is created with the name indicated, and its
associated
properties read in. The service instance is initialized with the properties
stored in the
-16-
CA 02475387 2004-07-21
properties directory; the file <.name>.ini is read in. Once successfully
initialized, the
dependent services are checked. If all services on which the loading service
depends
are in the started state, the loading service is started. If not, the loading
service is
scheduled for a later start. Once initialized, a service registers with the
managed
object management service and logs its state. The managed object management
service is a bootstrap service and is not loaded from the services.ini file.
The service
manager is another bootstrap service that provides lookup services for the
autonomic
controller. If any service fails to start a log is written using a best effort
logging
service, which may result in a simple write of a string to stdout. The first
time
services fail to start a service reset takes place at step 808. The second
time services
fail to start, the autonomic controller 512 continues without the affected
service.
Following service startup, modules are loaded from the file system at step
810. The
modules directory from within the root directory of the installation is read.
Each
directory within the modules directory contains a "Best Practices" module.
[0070] Module loading at step 810 consists of three sub-processes: load
policies, load
events and link events to policies. Loading policies consists of reading the
policies
directory within the current module directory. Policies are then loaded by
reading each
.ini file found in the directory. Each .ini file contains several properties,
including
name, description, the policy class and the class of the knowledge base which
implements it, along with one or more names of event sources that it consumes.
Loading a policy consists of creating an instance of the policy class and
associating an
instance of the knowledge base with it. In the current embodiment o:f the
invention,
knowledge bases are implemented using simple forward chaining rules. Loading
events consists of reading the events directory within the current module
directory.
Events are then loaded by reading each .ini file found in the directory. Each
.ini file
contains several properties, including name, description, the event generator
class and
the classes) of one or more observation processors, along with one or more
names of
observation sources that it listens to. Observation processors are objects
that take raw
observations and aggregate them to generate events. For example, are event
generator
3o might listen to CPU utilization observations and an observation processor
might
report the average over the last 10 observations. This average might then be
pipelined
into another processor that looks at whether a threshold has been exceeded.
Loading
-1~ -
CA 02475387 2004-07-21
an event consists of creating an instance of the event class and associating
observation
processor instances with it and starting observation tasks. The mechanism of
observation processing, observation aggregation and event generation will be
further
described during the explanation of the autonomic control state shown in
Figures 9A
and 9B. Starting an observation task involves communicating with the server to
set
up either a polling frequency for observation delivery; e.g. "tell me the
value of CPU
utilization every 30 seconds" or registration for event notification; e.g.
"tell me when
a new user logs in" or "tell me when a log is written to the event log". The
final sub-
process associated with module loading is linking. In this process, the names
of the
1o event sources associated with policies are resolved to the event generator
objects
created when the event sources were loaded. Module loading is considered
successful
if (a) all classes are successfully instantiated (b) all properties files can
be read and (c)
all event names can be resolved. All loading errors are logged. Failure to
load a
module on the first initialization pass causes that module to be unloaded and
the
module reloaded without policies or event sources that generated errors, this
is
considered the step of resetting modules i.e. step 812. Reloading of modules
continues until no further errors are detected. Once modules have been loaded,
the
autonomic controller 512 announces itself to any management console that it
knows
about at step 814. Known management consoles are found during the BOOTP/DHCP
2o protocol or through SYSADMIN card configuration. Failure to configure a
management console URL locally or through DHCP parameters means that the card
does not automatically announce itself. If the card is unable to announce
itself it tries
periodically to obtain information using DHCP. The final step of
initialization is a
sanity check i.e. step 818. During this check all services are checked for the
"in
service" state and that all event sources are functioning correctly.
[0071] Once the autonomic controller 512 is initialized, tasks associated with
monitoring the server 104 are executed. Figure 9A is a flow chart of the
process of
providing a request to the host service according to the current embodiment of
the
invention. The autonomic control process is driven by observations made on the
host
3o service and communicated asynchronously via the PCI bus 116. Observations
may
either be polled or notified with observations being made by observation
tasks. In the
case of a polled observation, at step 902 an observation task is scheduled to
run
-18_
CA 02475387 2004-07-21
periodically and is registered with the scheduler service during module
loading. The
observation task is run at step 904. When observation tasks run they send a
request
message, using the host communications manager, to the host service 502
running on
the server 104 requesting performance information at step 906; e.g. CPU
utilization.
The message is delivered to the host communications service (Host Mlanager),
where
it is queued. It is noted that a separate thread is used to deliver the
message. The
message is then processed by the host service operating on the server 104 at
step 908.
[0072] Figure 9B is a flow chart of the process of receiving a response from
the host
service of the server 104 by the autonomic controller 512. The request is non-
to blocking and returns immediately at step 912, providing a message
identifier for the
transaction. When the perforrrrance information is available, a callback is
invoked
within the host communications manager service. The HostManager creates an
observation and dispatches the message to the observation task object using
the
message identifier provided for the initial request at step 914. The message
is
15 converted to an observation by the observation task; observations are name,
value
pairs. The observation is then validated by the observation task at step 916.
The
event generator with which the observation task is associated is then notified
of a new
observation and the observation is sent to the event generator. Observation
processor
1 is then run at step 918. Each observation processor associated with the
event
2o generator processes the observation in turn at step 920. Using the CPU
Utilization
observation as an example, the first observation processor may adjust the
utilization
value to represent the average over a sliding window of several minutes; the
second
observation may implement a tripwire that indicates whether the value exceeds
a
threshold, only returning the observation in the event that the value is
exceeded and
25 null otherwise i.e. at step 922 it is determined whether an event has
occurred.
Observation processors represent a pipe and filter model of processing that is
familiar
to those skilled in the art of UhTIX systems programming or the design of the
JXTA
peer-to-peer framework. After all observation processors have accessed the
observation, and the observation is non-null, the event generator creates an
event from
3o the observation and notifies all policies that have registered an interest
in it at step
924. Each policy processes the event in turn, and may modify it. In the
current
embodiment of the invention, policies are implemented using a knowledge base
-19-
CA 02475387 2004-07-21
consisting of rules, which are processed using a Rete forward chaining
algorithm.
This algorithm will be familiar to individuals skilled in the art of Expert
Systems.
The event is inserted into the knowledge base and the Rete algorithm runs,
with zero
or one rule being chosen to fire. Code associated with a simple process-
termination
policy is shown below:
package com.symbium.jeops;
import com.symbium.utility.ErrorLogger;
import com.syrnbium.Event;
1o import java.util.Properties;
import com.symbium.services.*;
public ruleBase HostProcessManagementPolicy f
rule TerminationHostProcessRule {
declarations
Event e;
Properties p;
conditions
2o p.containsKey(e.getProperty("process"));
e.getProperty("state").equals("start");
actions
HostManager hm = ServiceManager.lookup("HostManager");
String[] args = ~ e.getProperty("process") };
hm.sendCommand("stopprocess", args);
Logger.log(l, e.getProperty("process")+" "+e.getProperty("state"));
)
[0073] The event being processed is shown in the declarations as "e"; the
processes to
be prevented from running are stored in the properties "p". The conditions for
the rule
state that, "If the process name that has been started is contained in the
list of
processes that are to be prevented from running, the
TerminationHostProcessRule is
true". When proven to be true, the actions associated with the rule are
executed. In
this case, the host management service is looked up. A stop process command is
then
sent to the host using this service, which will terminate the process.
Finally, a log is
written to the logger containing the process name and the fact that it was
started.
[0074] To summarize, the autonomic control state of the autonomic controller
512
implements a sense-act cycle. Sensors are implemented using observation tasks
that
-20-
CA 02475387 2004-07-21
either poll the server or are notified of changes in the state of the server.
Observations
are abstract objects that are server, operating system, and application.
neutral.
Observations are aggregated by event generators, where they are processed and
distributed as events to registered policies. In the current embodiment of the
invention policies process events use knowledge bases. According to
alternative
embodiments knowledge representations such as Fuzzy Logic, Neural Networks or
Finite State Machines are used.
[0075] Figure 10 is a flowchart detailing the shutdown process i.e. step 710.
The first
step of the shutdown process 710 is the step of stopping PCI communications at
step
1004. A control message is sent indicating that the autonomic controller 512
is going
off line. The host service 502, acting as the host, acknowledges this message.
If
negatively acknowledged, the message is repeated at step 1006. If the message
remains unacknowledged or is negatively acknowledged for more than a user-
defined
period of time, the shutdown process proceeds. A stop services step 1008
simply
invokes the stop lifecycle API on each service in turn. If a service fails to
stop, a best
effort log is generated and an attempt is made to stop it again, at step 1010,
after a
user-defined period of time. If the service fails to stop after a user-defined
number of
attempts, the shutdown process proceeds. An unload module step 1012 invokes
the
stop lifecycle API on each module in turn. The module stop API invokes the
stop
lifecycle API on each policy and event in turn. The policy stop lifecycle API
releases
allocated resources, including closing open files and saving their state
persistently at
step 1014. The event stop lifecycle API invokes the stop lifecycle on
observation
processors and tasks thereby allowing these objects the ability to release
resources
allocated by them. Once all modules have been unloaded, the autonomic
controller
512 sends a goodbye message at step 1016 to all known management consoles. It
then
shuts down.
[0076] Figure 12 is a schematic diagram illustrating the role of Module
Development
Environment (MDE) software in the lifecycle of a module. In the current
embodiment
the MDE software executes on the same platform as that on which the management
3o console software executes. Modules may be altered in two distinct ways;
they may be
configured and extended. Configuration involves the specification of new or
modified
data; e.g. an altered e-mail address. Extension involves policy changes such
as the
-21-
CA 02475387 2004-07-21
addition of modification of rules. The MDE has 2 primary functions. It can be
used
to create new modules and modify modules already in service. The MDE provides
a
visual programming environment that significantly reduces the barrier to entry
for
module developers. The MDE can take a Module Product (MP) 1202 delivered via
the web or from CDROM and specializes it. This MP will generally be stored in
a
module repository 1204. The MP is provided to the MDE at 1208 where it is
specialized, forming a Specialized Module Product (SMP) that can also be
stored in
the Module Repository 1204 maintained by the user. MPs inherit behavior from
MPs,
overriding "Best Practices" where appropriate for the user. Inheritance allows
user
Io behavior to be retained when a new version of the module is made available
to the
user. The MDE 1208 can also be used to create MPs. MPs or SMPs are provided to
the management console 1212 where they are edited and configured in a custom
manner; e.g. specify a default e-mail address for notification or a specific
target for a
window pop. These Configured Module Products (CMPs) 1210 are saved and
distributed to one or more ISACs 1214. Once installed on an ISAC, the
Installed
Module Products (IMPs) 1216 are saved in the file system 1218. IMPs contain
information that is gathered automatically during the module installation
process; e.g.
number and type of printers. IMPS are loaded when the card initializes as
shown in
Figure 6.
[0077] Figure 13 is a message interaction diagram. It describes the character
of the
autonomic control protocol operating across the PCI bus I 16 in the current
embodiment of the invention. The control protocol is one that consists of
monitor,
analyze, plan, and execute. The autonomic controller 512 of the embedded
control
system makes an inquiry request at step 1302. The inquiry request occurs
during
2s module loading and also as a result of a policy being triggered. The
request is
transferred across the PCI bus 116 at step 1304 and one or more inquiry
responses are
provided at step 1306; e.g. the CPU utilization being delivered on a user-
defined
frequency. The one or more responses are transferred across the PCI bus to the
autonomic controller 512 at step 1308. Each response is evaluated by one or
more
3o policies at step 1310. Policies may aggregate state; e.g. only do something
if the CPU
utilization has exceeded 75% for more than 10 minutes. The aggregation of
state
defines stored context. Context may also be affected by external monitoring,
such as
-22-
CA 02475387 2004-07-21
is provided by SYSADMINs changing parameter settings from a remote location
like
a Network Operations Centre (NOC). Depending upon context, follow up actions
are
initiated at step 1312; e.g. determine the process that is taking the most CPU
time.
EXAMPLES
[0078] Figures 14-25 present message interaction diagram of situations for
which the
ISAC 202 is designed detect, diagnose and correct.
[0079] Figure 14 describes a scenario in which a host is determined to be non-
responsive. Under normal operation i.e. in the normal case the host provides a
response to a probe from the ISAC 202. In the host non-responsive scenario, an
1o internally generated event on the ISAC 202 allows a follow up action to be
initiated to
reset the server.
[0080] Figure 15 describes the scenario where combinations of internal probes,
probe
other peripheral cards, probes across the control plane or data plane may be
used to
diagnose network problems. For example, an inability to ping the server using
its
1 s network interface (the data plane) via the card network interface may well
indicate a
network problem if the card can talk to the management console, which is on
the
control plane.
[0081) Figure 16 describes the scenario where a data plane probe times out
while a
bus probe succeeds. A data network fault may be present.
2o [0082] Figure 17 describes the scenario where both internal and data plane
probes fail,
suggesting a host fault and the host should be reset.
[0083] Figure 18 describes a scenario where an internal probe fails and a
probe across
the data plane succeeds, indicating that the host is alive. Thus an internal
fault has
occurred and an administrator should be notified. An attempt to remotely stop
the
25 host service or restart the PCI driver may also be attempted.
[0084] Figure 19 presents a general scenario in which one or more data
requests are
made across the internal bus. Based upon the processing of the returned data,
one or
more actions are initiated. The arrow connecting "Initiate Action" to itself
allows
-23-
CA 02475387 2004-07-21
events to be internally generated as a result of policy rule action. The arrow
connecting initiate action to the control plane indicates that event
information can be
passed to the management console for further processing and action.
[0085] Figure 20 presents the scenario where no heartbeat is detected from the
server.
The ISAC initiates a reset on the auxiliary bus, subsequently sending a SET
alarm to
the management console. When the reset succeeds, communication restarts over
the
internal bus, allowing the CLEAR alarm to be sent to the management console
indicating that the problem ha.s been resolved.
[0086] Figure 21 presents the case where the soft reset fails; i.e. a timeout
occurs after
to the soft reset has been attempted. Upon timeout the power cycle reset is
initiated
across the auxiliary bus and the level of the alarm upgraded to critical. When
the
power cycle reset succeeds internal communications on the bus are restored,
allowing
the CLEAR alarm to be sent to the management console indicating that the
problem
has been resolved.
is [0087] Figure 22 demonstrates how the ISAC can be used to automate routine
tasks.
A routine task, such as defragmentation of the hard drive is initiated by
asking the
host service to run a defragmentation task. The request passes across the
internal bus.
Upon completion, results are returned that indicate the success or failure of
the task.
Analysis of the results indicates whether follow up actions are required;
failure
2o generally causing a SET alarm to be sent to the management console.
[0088] Figures 23-25 illustrate how the ISAC can be used to implement high
value
security services. The ISAC is designed to autonomously follow the standard
CERT
(http://www.cert.org/) security institute guidelines, for recovering from an
event (i.e.
attack). System administrators unfortunately often skip these guidelines, due
to the
25 time-intensive requirements. Since it is often impossible to distinguish
between an
intelligently orchestrated attack and a software fault, the ISAC forces the
recommended CERT strategy on any server that cannot recover using traditional
recovery strategies (e.g. warm reboot). This guarantees recovery is
successful, and any
"backdoor(s)" left by an intruder is removed. Figures 23-25 detail information
flows
3o and actions that occur in certain security-related scenarios.
-24-
CA 02475387 2004-07-21
[0089] Figure 23 presents the scenario when process creation events are
monitored. A
policy is defined that terminates processes that are not explicitly allowed.
If a process
is created that is not allowed a command is sent via the internal bus to
terminate it. An
alarm is optionally generated in order to inform the user that an attempt to
start an
illegal process has occurred. The process creation event and the subsequent
termination command are written to an audit log.
[0090] Figure 24 presents the scenario when service state change events are
monitored. A policy is defined that keeps services alive. If a service stops
that should
always be running, a command is sent via the internal bus to restart it. An
alarm is
to optionally generated in order to inform the user that a service stopped.
but was
restarted.
[0091 ] Figure 25 presents the scenario when registry state change events are
monitored. A policy is defined that maintains the values of user-defined
registry keys.
If a registry key value changes that should remain unchanged, a command is
sent via
1s the internal bus to restore it. An alarm is optionally generated in order
to inform the
user that a registry change was detected but was restored. The registry change
event
and the subsequent restoration of its previous value are written to an audit
log.
[0092] The present invention has been described with regard to one or more
embodiments. However, it will be apparent to persons skilled in the art that a
number
20 of variations and modifications can be made without departing from the
scope of the
invention as defined in the clauns.
-25-
