Note: Descriptions are shown in the official language in which they were submitted.
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
USER INTERFACES FOR DATA INTEGRATION SYSTEMS
RELATED APPLICATIONS
This application claims priority to U.S. Provisional Application Number
60/606,372, filed August 31, 2004
and entitled "User Interfaces for Data Integration Systems".
BACKGROUND
1. Field.
This invention relates to the field of information technology, and more
particularly to the field of data
integration systems.
2. Description of the Related Art.
The advent of computer applications made many business processes much faster
and more efficient;
however, the proliferation of different computer applications that use
different data structures, communication
protocols, languages and platforms has led to great complexity in the
information technology infrastructure of the
typical business enterprise. Different business processes within the typical
enterprise may use coinpletely different
computer applications, each computer application being developed and optimized
for the particular business
process, rather than for the enterprise as a whole. For example, a business
may have a particular computer
application for tracking accounts payable and a completely different one for
keeping track of customer contacts. In
fact, even the same business process may use more than one computer
application, such as when an enterprise keeps
a centralized customer contact database, but employees keep their own contact
information, such as in a personal
infoi7nation manager.
While specialized computer applications offer the advantages of custom-
tailored solutions, the
proliferation leads to inefficiencies, such as repetitive entry and handling
of the same data many times throughout
the enterprise, or the failure of the enterprise to capitalize on data that is
associated with one process when the
enterprise executes another process that could benefit from that data. For
example, if the accounts payable process
is separated from the supply chain and ordering process, the enterprise may
accept and fill orders from a customer
whose credit history would have caused the enterprise to decline the order.
Many other examples can be provided
where an enterprise would benefit from consistent access to all of its data
across varied computer applications.
A number of companies have recognized and addressed the need for sharing of
data across different
applications in the business enterprise. Thus, enterprise application
integration, or EAI, has emerged as a message-
based strategy for addressing data from disparate sources. As computer
applications increase in complexity and
number, EAI efforts encounter many challenges, ranging from the need to handle
different protocols, the need to
address ever-increasing volumes of data and numbers of transactions, and an
ever-increasing appetite for faster
integration of data. Various approaches to EAI have been taken, including
least-common-denominator approaches,
atomic approaches, and bridge-type approaches. However, EAI is based upon
communication between individual
applications. As a significant disadvantage, the complexity of EAI solutions
grows geometrically in response to
linear additions of platforms and applications.
While data integration systems provided useful tools for addressing the needs
of an enterprise, such
systems are typically deployed as custom solutions. They have a lengthy
development cycle, and may require
sophisticated technical training to accommodate changes in business structure
and information requirements. There
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
remains a need for user interfaces to data integration systems that permit
use, reuse, and modification of
functionality through an intuitive interface adapted to the data integration
design and workflow environments.
SUMMARY
Provided herein are methods and systems for managing and using metadata in
comiection with data
integration in an enterprise computing environiinent. Through an integrated
user interface, a user may have
enterprise-wide access to data integration services and underlying data. The
interface may be structured around the
workflow of data integration system design and iinplementation, and may
facilitate reuse and redesign of tools and
jobs in the data integration environment. The interface may present platform-
independent access to all of the data
integration resources and data sources across an enterprise.
In one aspect, there is disclosed herein methods and systems for providing a
user interface for managing
workflow in a computer environment, the user interface including a menu bar
that displays a plurality of controls,
each control accessing an interface corresponding to a phase in a workflow,
each interface configured to (1) retrieve
data from the coinputer environment about the corresponding phase in the
workflow, (2) display the retrieved data
in the interface, (3) receive user input relating to the corresponding phase
in the workflow; and (4) transmit data to
the computer environment based upon the user input.
The computer environment may include a data integration system. The workflow
may relate to a data
integration process. The workflow may include at least one of a data
integration function, a data integration service,
or a data integration job. The user interface may provide varying degree's of
functionality for the plurality of
controls according to a skill level of a user of the interface.
The phases may include one or more of investigate, design, develop, deploy and
operate. In an investigate
phase, one or more data sources in the computer environment may be profiled,
metadata is obtained for one or more
data sources, and/or one or more data sources may be audited for data quality.
In a design phase, one or more data
integration processes may be characterized and/or one or more transformations
may be characterized for data in the
data sources. In a develop phase, one or more data integration processes may
be associated with data sources in the
data integration system, one or more data integration processes may be
associated with other data integration
processes deployed in the data integration system, and/or data staging may be
performed 'on one or more data
sources. In a deploy phase, a data integration process may be made available
to the data integration system and/or a
data integration process is deployed as a real-time data integration service.
In an operate phase, at least one data
integration function, service, or job may be used in the data integration
system.
The user interface may provide varying degrees of functionality for the
plurality of controls according to a
skill level of a user of the interface.
In another aspect, disclosed herein are methods for displaying related objects
in a user interface including
displaying a plurality of objects having relationships; and providing visual
cues for relationships between the
plurality of object including shape, size, color, and opacity of objects, and
distance between objects. Also disclosed
herein are systems for displaying related objects in a user interface
including a user interface displaying a plurality
of objects having relationships, the user interface providing visual cues for
relationships between the plurality of
object including shape, size, color, and opacity of objects, and distance
between objects.
The objects may describe a business context. The objects may include metadata
for one or more data
sources. The objects may include data integration objects. The systems and
methods may further include providing
one or more tools for visualizing, navigating, and editing relationships among
the plurality of objects. A hierarchy
3
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
of the plurality of objects may be displayed as a projection onto a three
dimensional surface. The user interface
may animate a transition to a new current object by rotating point of view of
the three-dimensional surface. The
hierarchy may be displayed as a projection onto a hyperbolic surface. The
current object may be displayed in
greater detail than other objects. The current object may be displayed larger
than other objects. The objects may
include one or more of people, business objects, and data sources.
In anotlier aspect, a method for providing information for an object within a
user interface may include
displaying a container in a user interface representing an object; and
providing an active area within the container,
the active area responding to an event by displaying one or more visual
controls related to the object. A system for
providing information may include a user interface displaying a container that
represents an object; and an active
area within the container, the active area responding to an event by
displaying one or more visual controls related to
the object.
The one or more visual controls may relate to functions performed on the
object, attributes of the object,
and or relationships of the object to other objects. The one or more visual
controls may respond to an event by
displaying additional visual controls related to the object, whereby a
hierarchy of controls is progressively displayed
for the object. The event may be a mouse over event. Manipulations of visual
controls may cause persistent
changes that may be reviewed on subsequent responses of the active area to the
event. The object may relate to a
data integration process.
In another aspect, a method for navigating to a page within a user interface
may include: displaying one of
a plurality of sequential pages within a user interface; providing a control
for navigating forward and backward
within the plurality of sequential pages; upon activation of the control,
displaying a drop down menu of more than
one of the plurality of sequential pages; receiving within the control a
selection of one of the sequential pages
displayed in the drop down menu; and navigating to the selected one of the
sequential pages. A system for
navigating to a page within a user interface may include: a user interface
displaying one of a plurality of sequential
pages; and a control for navigating forward and backward within the plurality
of sequential pages, the control
responsive to activation by displaying a drop down menu of more than one of
the plurality of sequential pages,
wherein a user may navigate to one of the sequential pages displayed in the
drop down menu by selecting the one of
the sequential pages displayed in the drop down menu. The pages may be
represented by tabs along an edge of a
page display of the user interface
In another aspect, a method for maintaining a list of reusable operations in a
user interface may include:
storing a plurality of tasks as they are performed in a user interface as a
history; displaying the history within a
control in the user interface; receiving a selection of one of the plurality
of tasks; and repeating the selected one
of the plurality of tasks in the present context of the user interface. A
system for maintaining a list of reusable
operations in a user interface may include: a user interface displaying a
history of a plurality of tasks as they are
performed; one or more controls for selecting one of the plurality of tasks
and repeating the selected one of the
plurality of tasks in the present context of the user interface.
The tasks may be data integration tasks. Tasks may be be stored at variable
levels of detail. Displaying
the history may include displaying the tasks in a hierarchy of levels of
detail of the tasks. The method may further
include undocking the control from the user interface for use with a plurality
of user interfaces. The plurality of
tasks may be sorted within the history according to a frequency of use. The
tasks may include one or more of
functions, operations, designs, macros, text types, stages, and wizards.
4
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
The method or system may further include receiving a selection of more than
one of the plurality of tasks;
and repeating the selected more than one of the plurality of tasks in the
present context of the user interface. The
method or system may further include storing the selected more than one of the
plurality of tasks for reuse.
In another aspect, there is disclosed herein a method for displaying a
hierarchy of objects within a drop
down menu of a user interface including: displaying list of a plurality of
objects at a plurality of levels of a
hierarchy within a hierarchical tree; providing visual cues to the position of
each object within the hierarchy, the
visual cues including identical shading of items at the same level of the
hierarchy and lines around a group of
adjacent items at the same level of the hierarchy; and providing a control at
each transition between different levels
of the hierarchy for collapsing or expanding a branch of the hierarchical
tree. Also disclosed herein is a system for
displaying a hierarchy of objects within a drop down menu of a user interface
including: a user interface displaying
a list of a plurality of objects at a plurality of levels of a hierarchy
within a hierarchical tree, the each one of the
plurality of objects displayed with one or more visual cues to the position of
each object within the hierarchy, the
visual cues including identical shading of items at the same level of the
hierarchy and lines around a group of
adjacent items at the same level of the hierarchy; and a control at each
transition between different levels of the
hierarchy for collapsing or expanding a branch of the hierarchical tree. The
objects may be data integration objects
and/or data integration tasks.
A method for displaying the status of a process within a user interface may
include: displaying a status bar
in a first mode that provides animation to indicate that a process is running;
in response to positioning a cursor over
the status bar, switching the status bar to a second mode by increasing the
size of the status bar and displaying
within the status bar a name of the process and a percentage of completion of
the process, and further displaying a
nuinber indicative of a number of other processes running; and in response to
positioning a cursor over the number,
switching to a third mode by additionally increasing the size of the status
bar and displaying within the status bar a
list of every process running. A system for displaying the status of a process
within a user interface may include a
user interface displaying a status bar, the status bar having a first mode, a
second mode, and a third mode, the first
mode providing animation to indicate that a process is running, the second
mode accessible by positioning a cursor
over the status bar, the second mode increasing the size of the status bar and
displaying within the status bar a name
of the process and a percentage of completion of the process, and further
displaying a number indicative of a
number of other processes running, and the third mode accessible by
positioning the cursor over the number in the
second mode, the third mode additionally increasing the size of the status bar
and displaying within the status bar a
list of every process running.
After a predetennined period of time, the status bar in the second mode may
cycle through status
information for the other processes. The method and system may further include
displaying within the status bar in
the third mode a start time for every process. The method may further include,
in response to positioning a cursor
over a process in the list of the third mode, providing a detailed view of the
process including one or more of a start
time for the process, a user who initiated the process, and a percentage of
resources used by the process. At least
one of the processes may be a data integration process.
A method for linking a data integration task to a business context may
include: providing a graphical user
interface for manipulating business objects and metadata; placing at least one
business object into the graphical user
interface; placing at least one metadata object from a data source into the
graphical user interface; and linking the at
least one metadata object to the at least one business object within the
graphical user interface. A system for linking
a data integration task to a business context may include a graphical user
interface for manipulating business objects
5
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
and metadata, the graphical user interface configured to link one or more
metadata objects from a data source to one
or more business objects within the graphical user interface.
The method and system may further include placing at least one metadata object
from a data target into the
graphical user interface. The method and system may fnrther include linking
the at least one metadata object from
the data target to a business object within the graphical user interface. The
method and system may further include
deriving a data transformation for one or more links between the data source
metadata and the data target metadata.
In another aspect, a method for designing a data integration process may
include: modeling a business
initiative as a first data structure; modeling a business context as a second
data structure; adding one or more
associations between the first data structure and the second data structure to
provide an infonnation model for the
business initiative in the business context; adding one or more associations
between the information model and one
or more enterprise data sources to provide a mapping model; and applying one
or more design tools to construct a
data integration process from the mapping model. A system for designing a data
integration process may include: a
modeling tool for modeling a business initiative as a first data structure,
modeling a business context as a second
data structure, and adding one or more associations between the first data
structure and the second data structure to
provide an infonnation model for the business initiative in the business
context; a linking tool for adding one or
more associations between the information model and one or more enterprise
data sources to provide a mapping
model; and one or more design tools to construct a data integration process
from the mapping model.
The business initiative may include one or more of an initiative, an
objective, a requirement, and a
statement. The business context may include one or more of a subject, a
process, and a rule. The enterprise data
sources may include data sources external to the enterprise. The associations
among the first data structure, the
second data structure, and the data sources may be created using tools within
a graphical user interface. All of the
steps of the method may be perfonned in an integrated user environment. The
design tools may be provided within
a user interface that includes integrated interfaces for one or more of
investigation, development, design,
deployment, and operation of a data integration process. A collaborative
environment may be provided for creating
one or more of the business initiative, the business context, the information
model, and the mapping model.
In another aspect, a method for providing a configurable user interface may
include: providing a plurality
of data integration tasks; providing a plurality of menus corresponding to
phases of a workflow; and assigning one
or more of the plurality of data integration tasks to each on of the menus
with a task matrix. A system for providing
a configurable user interface may include a task matrix that assigns one or
more of a plurality of data integration
tasks to one or more of a plurality of menus, each menu corresponding to a
phase of a workflow.
At least one of the data integration tasks may assigned to more than one menu.
The task matrix is user-
configurable.
In other aspects, a computer program product may include a computer useable
medium including computer
readable program code, wherein the computer readable program code when
executed on one or more computers
causes the one or more computers to perform any one or more of the methods
above.
"lnternational Business Machines" or "IBM" as used herein shall refer to
International Business Machines
Corporation of Armonk, New York.
As used herein, "data source" or "data target" are intended to have the
broadest possible meaning
consistent with these terms, and shall include a database, a plurality of
databases, a repository information manager,
a queue, a message service, a repository, a data facility, a data storage
facility, a data provider, a website, a server, a
computer, a computer storage facility, a CD, a DVD, a mobile storage facility,
a central storage facility, a hard disk,
6
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
a multiple coordinating data storage facilities, RAM, ROM, flash memory, a
memory card, a temporary memory
facility, a pennanent memory facility, magnetic tape, a locally connected
computing facility, a remotely connected
computing facility, a wireless facility, a wired facility, a mobile facility,
a central facility, a web browser, a client, a
laptop, a personal digital assistant ("PDA"), a telephone, a cellular phone, a
mobile phone, an information platform,
an analysis facility, a processing facility, a business enterprise system or
other facility where data is handled or
other facility provided to store data or other information, as well as any
files or file types for maintaining structured
or unstructured data used in any of the above systems, or any streaming,
messaged, event driven, or otherwise
sourced data, and any combinations of the foregoing, unless a specific meaning
is otherwise indicated or the context
of the phrase requires otherwise. A storage mechanism is any physical or
logical device, resource, or facility
capable of serving as a data source or a data target, or otherwise storing
data in a retrievable form.
"Enterprise Java Bean (EJB)" shall include the server-side component
architecture for the J2EE platform.
EJBs support rapid and simplified development of distributed, transactional,
secure and portable Java applications.
EJBs support a container architecture that allows concurrent consumption of
messages and provide support for
distributed transactions, so that database updates, message processing, and
connections to enterprise systems using
the J2EE architecture can participate in the same transaction context.
"JMS" shall mean the Java Message Service, which is an enterprise message
service for the Java-based
J2EE enterprise arcliitecture. "JCA" shall mean the J2EE Connector
Architecture of the J2EE platform described
more particularly below. It should be appreciated that, while EJB, JMS, and
JCA are commonly used software
tools in conteinporary distributed transaction environinents, any platform,
system, or architecture providing similar
functionality may be employed with the data integration systems described
herein.
"Real time" as used herein, shall include periods of time that approximate the
duration of a business
transaction or business and shall include processes or services that occur
during a business operation or business
process, as opposed to occurring off-line, such as in a nightly batch
processing operation. Depending on the
duration of the business process, real time might include seconds, fractions
of seconds, minutes, hours, or even
days.
"Business process," "business logic" and "business transaction" as used
herein, shall include any methods,
service, operations, processes or transactions that can be performed by a
business, including, without limitation,
sales, marketing, fulfillment, inventory management, pricing, product design,
professional services, financial
services, administration, finance, underwriting, analysis, contracting,
information technology services, data storage,
data mining, delivery ofinfonnation, routing of goods, scheduling,
communications, investments, transactions,
offerings, promotions, advertisements, offers, engineering, manufacturing,
supply chain management, human
resources management, data processing, data integration, work flow
administration, software production, hardware
production, development of new products, research, development, strategy
functions, quality control and assurance,
packaging, logistics, customer relationship management, handling rebates and
returns, customer support, product
maintenance, telemarketing, corporate communications, investor relations, and
many others.
"Service oriented architecture (SOA)", as used herein, shall include services
that form part of the
infrastructure of a business enterprise. In the SOA, services can become
building blocks for application
development and deployment, allowing rapid application development and
avoiding redundant code. Each service
may embody a set of business logic or business rules that can be bound to the
surrounding environment, such as the
source of the data inputs for the service or the targets for the data outputs
of the service. Various instances of SOA
are provided in the following description.
7
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
"Metadata," as used herein, shall include data that brings context to the data
being processed, data about
the data, information pertaining to the context of related information,
information pertaining to the origin of data,
information pertaining to the location of data, information pertaining to the
meaning of data, information pertaining
to the age of data, information pertaining to the heading of data, information
pertaining to the units of data,
information pertaining to the field of data, and/or information pertaining to
any other information relating to the
context of the data.
"WSDL" or "Web Services Description Language" as used herein, includes an XML
format for describing
network services (often web services) as a set of endpoints operating on
messages containing either document-
oriented or procedure-oriented information. The operations and messages are
described abstractly, and then bound
to a concrete network protocol and message format to define an endpoint.
Related concrete endpoints are combined
into abstract endpoints (services). WSDL is extensible to allow description of
endpoints and their messages
regardless of what message formats or network protocols are used to
communicate.
BRIEF DESCRIPTION OF THE FIGURES
Fig. I is a schematic diagram of a business enterprise with a plurality of
business processes, each of which
may include a plurality of different computer applications and data sources.
Fig. 2 is a schematic diagram showing data integration across a plurality of
business processes of a
business enterprise.
Fig. 3 is a schematic diagram showing an architecture for providing data
integration for a plurality of data
sources for a business enterprise.
Fig. 4A shows a layout for a user interface.
Fig. 4B shows a user interface adapted for use in a data integration system.
Fig. 4C depicts a task matrix for the design of a user interface
Fig. 5 shows an investigate phase in a workflow user interface.
Fig. 6 shows a design phase in a workflow user interface.
Fig. 7 shows a develop phase in a workflow user interface.
Fig. 8 shows a deploy phase in a workflow user interface.
Fig. 9 shows an operate phase in a workflow user interface.
Fig. 10 shows a navigation tool for use with a user interface.
Fig. 11 shows a tool that provides progressive disclosure of detail in a user
interface.
Fig. 12 shows a page selection tool for use in a user interface.
Fig. 13 shows a tool for recalling a history of previous operations within a
user interface.
Fig. 14 shows a menu tree for use with a user interface.
Fig. 15 shows a status bar for use with a user interface.
Fig. 16 shows a map for a subject in a business layer of a model.
Fig. 17 depicts rules and definitions for a subject in the business layer.
Fig. 18 depicts a business process in the business layer
Fig. 19 depicts an association of elements between the business layer and the
data layer.
Fig. 20 is a map of linked business and data elements.
Fig. 21 depicts the addition of a data object to a workspace of a user
interface.
Fig. 22 depicts an association of a data object to business objects in a
workspace of a user interface.
8
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
Fig. 23 shows a process for designing a data integration process from a
business context.
Fig. 24 shows decomposition of an initiative into objectives.
Fig. 25 shows decomposition of an objective into statements.
Fig. 26 shows the development of an infonnation model
Fig. 27 shows the addition of business requirements to information model
objects.
Fig. 28 shows linking of an infonnation model to an initiative.
Fig. 29 shows linking of an information model to data structures.
DETAILED DESCRIPTION
Throughout the following discussion, like element numerals are intended to
refer to like elements, unless
specifically indicated otherwise.
The invention(s) described herein can take the fonn of an entirely hardware
embodiment, an entirely
software embodiment or an embodiment containing both hardware and software
elements. In a preferred
einbodiment, the invention is implemented in software, which includes but is
not limited to finnware, resident
software, microcode, etc.
Furthermore, the invention(s) can take the form of a computer program product
accessible from a
computer-usable or computer-readable medium providing program code for use by
or in connection with a
computer or any instruction execution system. For the purposes of this
description, a computer-usable or computer
readable medium can be any apparatus that can contain, store, communicate,
propagate, or transport the program for
use by or in connection with the instruction execution system, apparatus, or
device.
The medium can be an electronic, magnetic, optical, electromagnetic, infrared,
or semiconductor system
(or apparatus or device) or a propagation medium. Examples of a computer-
readable mediuin include a
semiconductor or solid state memory, magnetic tape, a removable computer
diskette, a random access memory
(RAM), a read-only memory (ROM), a rigid magnetic disk and an optical disk.
Current examples of optical disks
include compact disk - read only memory (CD-ROM), compact disk - read/write
(CD-R/W) and DVD.
A data processing system suitable for storing and/or executing program code
will include at least one
processor coupled directly or indirectly to memory elements through a system
bus. The memory elements can
include local memory employed during actual execution of the program code,
bulk storage, and cache memories
which provide temporary storage of at least some program code in order to
reduce the number of times code must
be retrieved from bulk storage during execution.
Input/output or I/O devices (including but not limited to keyboards, displays,
pointing devices, etc.) can be
coupled to the system either directly or through intervening I/O controllers.
Network adapters may also be coupled to the system to enable the data
processing system to become
coupled to other data processing systems or remote printers or storage devices
through intervening private or public
networks. Modems, cable modem and Ethernet cards are just a few of the
currently available types of network
adapters.
Fig. 1 represents a platfonn 100 for facilitating integration of various data
of a business enterprise. The
platform includes a plurality of business processes, each of which may include
a plurality of different computer
applications and data sources. The platform may include several data sources
102, which may be data sources such
as those described above. These data sources may include a wide variety of
data types from a wide variety of
physical locations. For example, the data source may include systems from
providers such as such as Sybase,
9
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
Microsoft, Informix, Oracle, Inlomover, EMC, Trillium, First Logic, Siebel,
PeopleSoft, IBM, Apache, or Netscape.
The data sources 102 may include systems using database products or standards
such as IMS, D32, ADABAS,
VSAM, MD Series, UDB, XML, complex flat files, or FTP files. The data sources
102 may include files created or
used by applications such as Microsoft Outlook, Microsoft Word, Microsoft
Excel, Microsoft Access, as well as
files in standard formats such as ASCII, CSV, GIF, TIF, PNG, PDF, and so
forth. The data sources 102 may come
from various locations or they may be centrally located. The data supplied
from the data sources 102 may come in
various forms and have different formats that may or may not be compatible
with one another.
Data targets are discussed later in this description. In general, these data
targets may be any of the data
sources 102 noted above. This difference in nomenclature typically denotes
whether a data system provides data or
receives data in a data integration process. However, it should be appreciated
that this distinction is not intended to
convey any difference in capability between data sources and data targets
(unless specifically stated otherwise),
since in a conventional data integration system, data sources may receive data
and data targets may provide data.
The platform illustrated in Fig. 1 also includes a data integration system
104. The data integration system
may, for example, facilitate the collection of data from the data sources 102
as the result of a query or retrieval
command the data integration system 104 receives. The data integration system
104 may send commands to one or
more of the data sources 102 such that the data source(s) provides data to the
data integration system 104. Since the
data received may be in multiple formats including varying metadata, the data
integration system may reconfigure
the received data such that it can be later combined for integrated
processing. The functions that may be performed
by the data integration system 104 are described in more detail below.
The platform 100 also includes several retrieval systems 108. The retrieval
systems 108 may include
databases or processing platforms used to further manipulate the data
communicated from the data integration
system 104. For example, the data integration system 104 may cleanse, combine,
transform or otherwise
manipulate the data it receives from the data sources 102 such that a
retrieval system 108 can use the processed data
to produce reports 110 useful to the business. The reports 110 may be used to
report data associations, answer
complex queries, answer simple queries, or fonn other reports useful to the
business or user, and may include raw
data, tables, charts, graphs, and any other representations of data from the
retrieval systems 108.
The platform 100 may also include a database or data base management system
112. The database 112
may be used to store information temporally, temporarily, or for permanent or
long-term storage. For example, the
data integration system 104 may collect data from one or more data sources 102
and transform the data into forms
that are compatible with one another or compatible to be combined with one
another. Once the data is transformed,
the data integration system 104 may store the data in the database 112 in a
decomposed form, combined form or
other form for later retrieval.
Fig. 2 is a schematic diagram showing data integration across a plurality of
entities and business processes
of a business enterprise. In the illustrated embodiment, the data integration
system 104 facilitates the information
flowing between user interface systems 202 and data sources 102. The data
integration system 104 may receive
queries from the interface systems 202, where the queries necessitate the
extraction and possibly transformation of
data residing in one or more of the data sources 102. The interface systems
202 may include any device or program
for communicating with the data integration system 104, such as a web browser
operating on a laptop or desktop
computer, a cell phone, a personal digital assistant ("PDA"), a networked
platform and devices attached thereto, or
any other device or system that might interface with the data integration
system 104.
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
For exainple, a user may be operating a PDA and make a request for information
to the data integration
system 104 over a WiFi or Wireless Access Protocol/Wireless Markup Language
("WAP/WML") interface. The
data integration system 104 may receive the request and generate any required
queries to access information from a
website or other data source 102 such as an FTP file site. The data from the
data sources 102 may be extracted and
transformed into a format compatible with the requesting interface system 202
(a PDA in this example) and then
communicated to the interface system 202 for user viewing and manipulation. In
another embodiment, the data
may have previously been extracted from the data sources and stored in a
separate database 112, which may be a
data warehouse or other data facility used by the data integration system 104.
The data may have been stored in the
database 112 in a transformed condition or in its original state. For example,
the data may be stored in a
transformed condition such that the data from a number of data sources 102 can
be combined in another
transformation process. For example, a query from the PDA may be transmitted
to the data integration system 104
and the data integration system 104 may extract the information from the
database 112. Following the extraction,
the data integration system 104 may transform the data into a combined format
compatible with the PDA before
transmission to the PDA.
Fig. 3 is a schematic diagram showing an architecture for providing data
integration for a plurality of data
sources 102 for a business enterprise. An embodiment of a data integration
system 104 may include a discover data
stage 302 to perform, possibly among other processes, extraction of data from
a data source and analysis of column
values and table structures for source data. A discover data stage 302 may
also generate recommendations about
table stracture, relationships, and keys for a data target. More sophisticated
profiling and auditing functions may
include date range validation, accuracy of computations, accuracy of if-then
evaluations, and so forth. The discover
data stage 302 may normalize data, such as by eliminating redundant
dependencies and other anomalies in the
source data. The discover data stage 302 may provide additional functions,
such as drill down to exceptions within
a data source 102 for further analysis, or enabling direct profiling of
mainframe data. A non-limiting example of a
commercial embodiment of a discover data stage 302 may be found in IBM's
WebSphere ProfileStage product.
The data integration system 104 may also include a data preparation stage 304
where the data is prepared,
standardized, matched, or otherwise manipulated to produce quality data to be
later transformed. The data
preparation stage 304 may perform generic data quality functions, such as
reconciling inconsistencies or checking
for correct matches (including one-to-one matches, one-to-many matches, and
deduplication) within data. The data
preparation stage 304 may also provide specific data enhancement functions.
For example, the data preparation
stage 304 may ensure that addresses conform to multinational postal references
for improved international
communication. The data preparation stage 304 may conform location data to
multinational geocoding standards
for spatial information management. The data preparation stage may modify or
add to addresses to ensure that
address information qualifies for U.S. Postal Service mail rate discounts
under Government Certified U.S. Address
Correction. Similar analysis and data revision may be provided for Canadian
and Australian postal systems, which
provide discount rates for properly addressed mail. A non-limiting example of
a commercial einbodiment of a data
preparation stage 304 may be found in IBM's WebSphere QualityStage product.
The data integration system may also include a data transformation stage 308
to transform, enrich and
deliver transformed data. The data transformation stage 308 may perform
transitional services such as
reorganization and reformatting of data, and perform calculations based on
business rules and algorithms of the
system user. The data transformation stage 308 may also organize target data
into subsets known as datamarts or
cubes for more highly tuned processing of data in certain analytical contexts.
The data transformation stage 308
11
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
may employ bridges, translators, or other interfaces (as discussed generally
below) to span various software and
hardware architectures of various data sources and data targets used by the
data integration system 104. The data
transformation stage 308 may include a graphical user interface, a command
line interface, or some combination of
these, to design data integration jobs across the platform 100. A non-limiting
example of a commercial
embodiment of a data transformation stage 308 may be found in IBM's WebSphere
DataStage product.
The stages 302, 304, 308 of the data integration system 104 may be executed
using a parallel execution
system 310 or in a serial or combination manner to optimize the performance of
the system 104.
The data integration system 104 may also include a metadata management system
312 for managing
metadata associated with data sources 102. In general, the metadata management
system 312 may provide for
interchange, integration, management, and analysis of metadata across all of
the tools in a data integration
environment. For example, a metadata management system 312 may provide common,
universally accessible
views of data in disparate sources, such as IBM's WebSphere ODBC MetaBroker,
CA ERwin, IBM WebSphere
ProfileStage, IBM WebSphere DataStage, IBM WebSphere QualityStage, IBM DB2
Cube Views, and Cognos
Impromptu. The metadata management system 312 may also provide analysis tools
for data lineage and impact
analysis for cbanges to data structures. The metadata management system 312
may further be used to prepare a
business data glossary of data definitions, algorithms, and business contexts
for data within the data integration
system 104, which glossary may be published for use throughout an enterprise.
A non-limiting example of a
commercial embodiment of a metadata management system 312 may be found in
IBM's WebSphere MetaStage
product.
Fig. 4A shows a layout for a user interface. The user interface may be
presented as, for example an
interface 5200, which may have a layout including a header 5202, a sidebar
5204, a footer 5206 and a main section
5208, all of which may be displayed within, for example, any of the interface
systems 202 described above, or any
other suitable client device or terminal. The interface system 202 may use,
for example, any conventional web
browser technology, such as Microsoft Internet Explorer or Netscape, with Java
or other client-side applets or the
like providing enhanced functionality within the interface, or the entire
interface, or portions thereof, may be
controlled by other client-server or peer-to-peer communication technologies
and/or rendered locally using
applications or other software running on a client device. The interface may
also, or instead, be deployed as a rich
client, and may be strongly separated from underlying functionality using, for
example, model-view-controller
design concepts.
The various features of the interfaces described below may include any known
media, including animation,
sound, music, graphics, and text. Furthermore, while HTML pages are a useful
starting point for discussing
features of a user interface, it will be appreciated that the user interface
may not employ HTML at all, and/or may
be generated and interpreted using many known programming languages or mark-up
languages, and any associated
plug-ins or add-ons, including HTML, SHTML, XML, DHTML, JavaScript, Java,
Perl, PHP3, CGI, C, and C++, all
of which may be used with the interfaces discussed herein. As used in the
following discussion, the term
"interface" refers to any user interface, however rendered, whether or not it
employs HTML, and whether or not it
is provided over a network.
The header 5202 may include, for example, a title of the page, and one or more
links to other pages or
resources displayed as hyperlinks, labeled tabs, or the like. The sidebar 5204
may include a menu of choices for a
user of the interface 5200. The footer 5206 may include information concerning
the page such as a "help" feature,
status information for processes, and so on. The main section 5208 may display
a user workspace for content such
12
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
as models, design information, and so on, as described more generally in the
following sections. The main section
5208 may also include, for example, tools for adding, deleting, or modifying
information, such as data objects or
jobs within the section 5208, managing and displaying objects, and so forth.
In addition to the various controls
described below, the page 5202 may employ known user interface controls either
within the interface 5200 or in
supplemental dialogue boxes, such as drop-down menus, check boxes, text boxes,
sliders, radio buttons, and the
like.
The interface 5200 may track user activities and user information. Using, for
example, cookies or other
client-side data stores, the system may maintain a degree of persistence
between the system and a particular user or
a particular client device. Content displayed within the interface 5200 may be
determined in whole, or in part, by a
user's previous activities (aside from persistent data, such as models that
have been stored). For example, where an
authorization scheme is employed, previous logged-in sessions may be used to
determine a starting point within the
interface 5200 of the application, such as a specific point within the design
workflow described below. Or, where a
web browser model is employed, a cookie may be employed to recall a previous
state of the browser. A number of
registration/authorization schemes and technologies are known in the art, and
may usefully be employed with the
systems described herein to manage access to data and other system resources
through the interface 5200.
Different interfaces may be provided for different users of the system. For
example, a designer may have
full access to tools and functions provided by the interface. By contrast, a
business analyst may view data using
read-only access to all or some of the data stored in the databases, or may
have a simplified tool set for a limited
range of design activities. A manager may have write access to some or all of
the data, in addition to read access,
so that data can be created, updated, or viewed. Each level of access and use
of data may be realized by providing a
different interface with a different perspective on a common underlying data
set, such as metadata and metadata
models used within the metadata management systems described herein.
It will be appreciated that the description above is generic, and may be
varied according to where a user is
within a user environment related to the interface, as well as according to
any available information about the user
interface system 202 (such as display size, media capabilities, etc.),
available information about the user of the user
interface 202 (such as skill level, job title), or authorization to view or
edit certain information. More generally, any
layout, or combination of layouts suitable for inputting, editing, deleting,
and/or otherwise managing objects and
relationships within a data integration system may be usefully employed with
the interface system 202.
Fig. 4B shows a user interface 5200 adapted for use in a data integration
system. As depicted, the user
interface 5200 may include workflow tools 5212, menu items 5214, tool buttons
5218, a search function 5220, one
or more window scroll bars 5222, and a workspace 5224.
The workflow tools 5212 may be designed around a workflow, such as an
investigate, design, develop,
deploy, operate ("1DDDO") workflow useful in working with data integration
systems such as the data integration
systems 104 described above, or implementing other enterprise initiatives. The
tools may be arranged in a pillar
menu where a group of icons represent each phase of the IDDDO workflow, and
moving a cursor over one of the
icons presents a drop-down tree or "pillar" of useful tools, objects, and
other resources useful at that stage in the
workflow. Aspects of the pillar menu will be described in greater detail in
the following figures. In Fig. 4B, a
"home" icon of the pillar menu is selected, with access to system-wide
functions such as navigating among open
workspaces and global copy, paste, delete, rename, and cut functions.
Other user interface controls may be provided within conventional drop-down
menus 5214 or buttons
5218. The drop-down menus 5214, for example, may provide access to functions
that are grouped according to file,
13
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
edit, view, tools, and help. The buttons 5218 may provide immediate access to
commonly used tools, wliich may
overlap with tools provided in the drop-down menus 5214, the workflow tools
5212, or elsewhere. A search
function 5220 may be provided for searching the contents of one or more
workspaces, data sources, or other areas
within a system. Scroll bars 5222 may be provided to scroll through the
workspace 5224,
The workspace 5224 may display any relevant content, such as content that is
generic to a user
environment, specific to a user, specific to a phase of the workflow, specific
to a project within the workflow
environinent, or any combination of these. The workspace 5224 may be populated
with data or content from
various sources within the system such as message queues, data sources, e-mail
accounts, task monitors, resource
monitors, and so forth. The workspace 5224 may display status information for
processes such as data integration
services, functions, or jobs, running on the system. The workspace 5224 may
also provide controls, such as any of
the controls described above, for taking actions relating to any of the
foregoing, which controls may be global, or
specific to a source for the item being displayed, or specific to a phase
within the workflow. In one aspect, the
workspace 5224 may enable collaboration by providing messaging services
between users, along with the ability to
provide links to various initiatives or sub-parts of initiatives within the
system.
The system may advantageously provide for persistence of data between various
phases of the workflow,
so that a user may transition forward and backward within the workflow, and
between tasks in a particular phase,
without loosing information or data relating to a project. At the same time,
changes within one phase may have
repercussions within other phases, and the results may be readily observed by
navigation among the phases, such as
through use of the icons within the pillar menu.
Several views of a user interface for use with an IDDDO workflow will now be
described in greater detail.
In general, within each phase of the workflow, a user may introduce data or
content, or retrieve data from the
computer environment (such as the data integration system 104 or enterprise
computing system) about the
corresponding phase in the workflow. The data may be displayed within the
workspace 5224 of the user interface
5200, or as appropriate, within windows or frames in the workspace 5224, or in
dialogue boxes or drop-down
menus associated with the workspace 5224 or the user interface 5200. Through
the workspace 5224 and the
associated tools and controls 5212-5222, a user may provide input to the
current workflow phase, which input may
be transmitted back to resources within the computing environment.
Thus, as a significant advantage, a user interface as described herein
provides a work environment where
changes are persistent and consistent across a number of phases in a workflow,
while also permitting a user to
navigate seamlessly among the phases of a staged workflow from a global
interface. Such an inteiface may be
advantageously employed in complex hardware and software environments,
particularly heterogeneous and/or
networked environments, such as might be found in the data integration systems
104 and enterprise computing
systems described above.
The example below will illustrate a design of a user interface to for an IDDDO
data integration design
process. It should be appreciated that, while a specific, detailed interface
is described, the system may be more
generally characterized as a realization of data integration as a navigation
methodology. That is, aspects of an
arbitrary design process may be explicitly expressed within the interface to
facilitate designs according to the
process. From this perspective, a workflow such as an industry accepted design
methodology may serve as the
architecture for the user interface. More generally, any process or result
dictated by external constraints such as
legal statutes, industry standards, protocols, or data structures may suggest
an architecture for a user interface that
14
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
logically and seamlessly leads a designer to a compliant result within the
interface. Similarly, internal constraints
such as a business initiative within a corporation may suggest an architecture
for the user interface.
Further enhancements may be provided in this context. For example, by modeling
the interface around a
known workflow, the interface may generate one or more recommendations for a
next step, which may be
displayed, for exainple as an item or list within a region at the bottom of
the workspace 5224. This technique may
be further refined by adapting to a history of user activities (either by the
current user, by all users, or some
combination of these) from the current location within the workflow to provide
more refined recommendations at
each point in the workflow. The recommendations may be dynamically updated as
the user navigates through the
workflow environment. Similarly, context-sensitive help may be provided for
assistance specific to a users location
within the workflow. More generally, the interface may be context-aware, and
may provide any number of forms of
assistance or feedback to the user according to the user's location within the
workflow.
Fig. 4C depicts a task matrix for the design of a user interface. The user
interface may advantageously be
strongly separated from underlying functionality in order to maintain design
flexibility. In such an architecture, the
tasks dictated by a workflow may be organized within a task matrix 5250 that
is interpreted to provide an interface
to the user. The task matrix 5250 may include one or more contexts 5252, each
including a number of tasks 5254
and a number of menus 5256 (the "pillars" described in the interface below).
The one or more contexts 5252 may relate to, for example, a number of
different presentations of the tasks
within the inenus. Tlirough the use of a number of separately defined
contexts, the interface may be designed with
a variety of optional presentations, such as skill-level sensitive
presentations or security-level sensitive
presentations. More generally, by providing a number of optional contexts, an
additional dimension of design
flexibility is realized within the task matrix 5250. This may be used in a
number of ways. As an example, a health
care provider and an insurer may both be responsible under a regulatory scheme
such as HIPAA for maintaining
internal records and transacting with others in a specific manner. In this
case, the provider and the insurer may have
a number of tasks in common, such as encrypting personal identification
information or using a common format for
a request for payment that is sent by the provider to the insurer. Data
integration processes for the insurer and the
provider may share a substantial number of tasks in common, while also
requiring significant differences. Two
contexts 5252 may be used for the provider and the insurer to define two
different workflows based upon the same
set of common tasks under a more general HIPAA-compliant interface definition.
Each task 5254 may be any number of user-defined tasks or common tasks,
functions, services and the like
provided by the system, or combinations of such tasks, functions, and/or
services combined into a single useful task
relating to a workflow. The tasks 5254 may also include, or be associated
with, dialogs, wizards, help windows,
and the like useful within the interface. The tasks 5254 identified in the
task matrix 5250 may be predefined with
respect to, e.g., their location within the interface, association with
certain control regions of the interface, the
controls, inputs, and outputs associated with the task, and so on. Thus it is
simply necessary to indicate the
presence of the task in the task matrix 5250 for the task to be accessible
through the user interface. Optionally, the
intersection of each task 5254 and menu 5256 may include one or more
specifications relating to the occurrence of
the task 5254. Thus, the task matrix 5250 may specify visual elements,
controls, location, inputs, outputs, skill-
level parameters, and so on.
Each menu item 5256 may correspond to a phase of a workflow. Using the task
matrix 5250, the relevant
tasks for a workflow may be realized at appropriate locations within the user
environment.
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
To further assist in rapid deployment of workflow-based interface designs, the
user interface itself may be
defined as a collection of visual styles, controls, control panels,
workspaces, tools, wizards, dialogs, and so on. At
the same time, the tasks may be defined, for example, as services within a
service-oriented architecture. Thus,
through the use of the task matrix 5250 an arbitrary combination of menu items
5256, each with one or more
supporting tasks 5254, may be conveniently arranged to express any data
integration or other workflow as a
navigation methodology.
The interface may advantageously operate against a common data set, so that
changes are persisted ainong
various phases (i.e., menus) of the workflow, and between tasks within a
phase. Each task 5254 and each menu
5256 may represent a different functional perspective on the same data set.
Thus, the architecture may provide a
significant iinprovement over existing application-based environments in which
a workflow passes data from
application to application for different phases of the workflow.
While a specific construct has been identified for organizing tasks within a
workflow, it should be
appreciated that any number of data structures or other storage mechanisms may
be used to store, retrieve, and
modify definitions of user interfaces, and it should further be appreciated
that the architecture of the user interface
itself may be defined at various levels of specificity, according to the
desired trade-off between flexibility and ease
of design.
Fig. 5 shows an investigate phase in a workflow user interface. By selecting
an investigate icon 5302
within the user interface 5300, which may be any of the user interfaces 5200
described above, a user may access a
number of investigation-related functions and objects, such as through a
pillar menu 5304, in connection with
implementing an enterprise initiative. For example, the pillar menu 5304 may
include selectable items for systems,
users, specification, components, data model, information, information model,
reports, flows, and any of the global
tasks noted above, and all items may be contextual, that is adapted to the
current pillar or workflow phase.
In the investigate phase, available systems may include any systeins within
the computer environment,
including repositories such as message queues, directories, files, and
databases. Systems may include servers such
as DBMS servers, Transaction Coordinator servers, web servers, application
servers, machine configuration servers,
message queue servers, execution configuration servers, or security model
servers. Systems may also include
networks, such as local networks, remote networks, and cluster configurations
within networks. When a system is
selected, information relating to the selected system may be displayed within
the workspace or in a separate
window, where a user may investigate characteristics, configurations, and
resources of the selected system. In
certain configurations, the user with adequate authority on the system may
reconfigure or otherwise alter systems
through the user interface 5300.
Users may include system users having user identities within the computing
environment, such as
authorized employees, as well as external users such as vendors or other
providers, and customers, consumers, or
other external users. When a user is selected, infonnation relating to the
selected user may be displayed within the
workspace or in a separate window. For each user or user type, an operator of
the user interface 5300 may view
(and possibly edit) role data that defines a user's roles and authority within
the system, as well as identification
information such as name, address, e-mail, phone, fax, and the like.
Specification may include any information related to specifying a project
within the workflow
management environment, such as processes and rules describing the project.
Processes may include external
applications or internal business processes. Rules may include business rules,
configurations, or metrics.
16
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
Coinponents may be software objects and tools available to the system that may
be used in various phases
of the workflow, such as scripts (e.g., shell scripts and filters), tools
(e.g., metabrokers, metadata importers, and
modeling tools), containers (e.g., custom components, shared containers, and
existing jobs), services (e.g., web
services and external routines), macros (e.g., OSH scripts, data filters,
shell scripts, and SQL queries), connectors
(e.g., files, EAI connectors, message queues, metabrokers, attached tables,
import utilities, custom adapters, web
service connectors, message switches, databases, applications, and analysis
servers), functions, and routines. A user
may select components to review more detailed information about the
availability, function, and interface of each.
Data models may be models for data sources within the system. Data models may
include definitions of
data type, structure, format, and any other metadata and the like for data
sources, along with an identification of
each specific data source available. Through the data model item, a user may
browse data sources and
corresponding models for data available to be used within phases of the
workflow.
Infonnation may be any information relevant to a project, such as
documentation (e.g., specifications,
manuals, online help, online documentation, and so on), messages (e.g.,
message formats, notifications, and
subscriptions), data (e.g., column analysis, cross table analysis, and table
analysis), metrics (e.g., security,
availability, goals, quality, and performance), and metadata (e.g., for
objects or models). Information provided
through this aspect of the user interface 5300 may be available to a user
throughout the phases of a project.
Similarly, an information definition may be provided for the resulting design.
Reports may be any reports useful for monitoring progress of the design, or
operating data integration
services or other processes. For example, investigation reports may include
word investigation, pattern reports,
character discreet, character concatenate, or delete options. Analysis reports
may include general reports, table
analysis, normalization, column analysis, relationship analysis, and exception
reports. Metric reports may include
various chart or statistical reports. Through this user interface feature, a
user may select and configure various
reports to be used during the workflow.
Flows may include data flows (e.g., JCL flow, batch BTL flow, and RTI flow),
services (e.g., RTI flow or
external flow), and process flow (e.g., BPM flow or job sequence flow).
Through this aspect of the user interface
5300, a user may investigate existing data and process flows within the
system.
More generally, the features noted above may be used to investigate the scope
and content of a system that
will be the subject of the workflow, and to characterize various sources,
resources, services, tools and the like
within the system.
Fig. 6 shows a design phase in a workflow user interface. By selecting a
design icon 5402 within the user
interface 5400, which may be any of the user interfaces 5200 described above,
a user may access a number of
design-related functions and objects, such as through a pillar menu 5404,
associated with the design of an enterprise
initiative. For example, the pillar menu 5404 may include selectable items for
users, reports, flows, systems,
specification, components, and information, as well as the global tasks noted
above. In general these items may be
as described above, with some exceptions such as those described below that
relate more explicitly to the actual use
of various items in the designqphase.
For example, under the information item, data may include table schema, data
schema and domains, and
target schema. Also under information, metadata may be more specifically
characterized in terms of process
definition, operational definitions, rule definitions, system definitions,
data definitions, component definitions,
model/object definitions, message definitions, table definitions, and so on.
Similarly, specifications may explicitly
include mapping specifications.
17
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
More generally, the features noted above may be used to design data
integration processes, jobs, services,
functions, and the like within a data integration system, or still more
generally to design any other enterprise
initiative within a computer environment.
Fig. 7 shows a develop phase in a workflow user interface. By selecting a
develop icon 5502 within the
user interface 5500, which may be any of the user interfaces 5200 described
above, a user may access a number of
development-related functions and objects, such as through a pillar menu 5504,
associated with the development of
an enterprise initiative or project. The pillar menu 5504 may include
selectable items for flows, components,
specifications, systems, and information, along with any of the global tasks,
such as tasks related to data integration
jobs, extraction, transfonnation, loading, performance of real-time
integration services, or the like as described in
this disclosure. In general, these items may be as described throughout this
disclosure, with some exceptions such
as those described below that relate more explicitly to the actual use of
various items in the develop phase.
For example, the specification in the develop phase may include processes and
rules, as described above,
but may also include templates for the project, such as JCL templates,
documentation templates, and job templates
that are to be deployed with the project. As another example, within the
develop phase, data may be characterized
as infonnation that is defined in terms of reference or lookup tables and data
sets.
More generally, the features available in the develop phase may be used to
develop a data integration
process, job, service, function, or the like within a data integration system,
or still more generally to develop any
enterprise initiative within a computer environment.
Fig. 8 shows a deploy phase in a workflow user interface. By selecting a
deploy icon 5602 within the user
interface 5600, a user may access a number of deployment-related functions and
objects, such as through a pillar
menu 5304. The pillar menu 5604 may include selectable items for flows,
systems, components, and information,
as well as any of the global tasks noted above. In general, these items may be
as described above, with some
exceptions such as those described below that relate more explicitly to the
actual use of various items in the develop
phase.
In the deploy phase, the selectable systems item, for example, may relate to
systems as required and/or
defined for deployment. As such, the items may include tools (e.g.,
development tools, installation tools,
administration tools, services and configuration tools), repositories (e.g.,
source control, databases, files, message
queues, and directories), servers (e.g., security model, execution
configuration, and machine configuration), and
environment (e.g., disk resources, roles, CPU resources, projects, groups, and
parameters). It will be noted that the
"systems" as defined for deployment specifically describes an environment and
plattonn onto which processes
might be deployed.
More generally, the features available in the deploy phase may be used to
deploy a data integration
process, job, service, function, or the like within a data integration system,
or still more generally to deploy any
other enterprise initiative within a computer environment.
Fig. 9 shows an operate phase in a workflow user interface. By selecting an
operate icon 5702 within the
user interface 5700, a user may access a number of operation-related
functions, such as through a pillar menu 5704.
The pillar menu 5704 may include selectable items for flows, systems,
specification, components, and infonnation,
as well as any of the global tasks described throughout this disclosure. In
general, these items may be as described
herein, with some exceptions such as those described below that relate more
explicitly to the actual use of various
items in the operate phase.
18
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
For example, in the operate phase the specification for the process is in
tenns of processes and rules. A
full array of reports may also be provided, consistent with actual operation
of the process or project, such s
statistical reports, data lineage reports, metrics, audit reports, charts,
domain analysis, metadata reports, analysis
charts, exception reports, transcripts, logs, impact analysis, and trend
charts.
More generally, the features available in the operate phase may be used to
operate a data integration
process, job, service, function, or the like within a data integration system,
or still more generally to operate any
other enterprise initiative within a computer environment.
Figs. l0A and lOB shows a navigation tool for use with a user interface. The
user interface 5800, which
may include any of the user interfaces and user interface systems 202
described above, may include a text display of
a hierarchy 5802, one or more buttons 5804-5810 or other controls, an object
pallet 5812, a workspace 5814, and
one or more objects 5820 displayed within the workspace 5814.
The text display of hierarchy 5802 may provide an indication, in textual
fonnat, of the relative position of
objects 5820 within a hierarchy, such as by showing a path from a root object
to a current object, or between two
user-selected objects. Each object 5820 may represent a data item, data
structure, data source, function, tool, or
other resource available within the data integration system 104. It should
also be appreciated that the objects 5820
may represent other entities, such as people, services, business units,
business functions, projects, plans, or other
entities within and enterprise, as well as entities outside the enterprise
such as vendors, customers, suppliers,
regulatory bodies, and so on. The objects 5820 may also, or instead,
correspond to physical objects such as
computer tenninals, mainframes, routers, servers, repositories, resources,
services (including web services),
modules, databases, printers, and/or other computer resources. In general,
objects 5820 within the user interface
5800 may represent anything that may have a relationship with other items,
objects, entities, or the like.
The one or more buttons 5804-58 10 or other controls may provide access to
various tools within the user
interface 5800. For example, a control panel button 5804 may provide access to
a control panel for the interface,
that may display descriptive infonnation (e.g., hops or levels within the
hierarchy between selected objects, or
between the current object and a root object), or may be used to control how
many degrees of relationship are
displayed within the workspace (e.g., all objects related to the current
object by two or fewer hops, three or fewer
hops, etc.). The control panel may also provide "skins" or other user-
customizable components for controlling
visualization of objects within the workspace 5814, such as color, shape, and
so on. The control panel may
similarly provide control over a variety of other visual and substantive
features of objects displayed within the
workspace 5814. The center button 5808 may be used to center the display of
objects 5820 within the workspace
5814 around the current object. This feature may be of particular value when
navigating large, complex hierarchies.
The clear button 5810 may be used to clear the workspace 5814.
The object pallet 5812 may contain generic object forms for addition to the
workspace 5814, such as
through conventional drag-and-drop or double-click manipulation with a cursor.
The object pallet 5812 may also
include predefined objects, such as objects already within the workspace 5814
that may be copied, or predefined
object types, such as a "person" or "business unit". Predefined objects may
contain a number of predefined
relationships to other objects. For example, by dragging a "person" object
type into the workspace 5814, the user
interface 5800 may automatically generate a number of related objects such as
a name, an address, a social security
number, an employer, a supervisor, and so on, and relate these new objects to
the person object in the workspace
5814. Optionally, by dragging a predefined person (e.g., John Doe) onto the
workspace 5814, the user interface
5800 may automatically generate known relationships to other objects 5820
within the workspace 5814.
19
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
The workspace 5814 can provide a working area for adding, removing, and
editing objects 5820. Within
the workspace 5814, a user may, for example, navigate through a hierarchy of
objects 5820 by selecting new current
objects from among the objects 5820 visible within the workspace 5814. A
search tool may also be provided for
searching for objects. Within the workspace 5814, a current object may be
edited, and details for an object may be
reviewed, such as by clicking on the object. Relationships between objects may
be added, deleted, or modified.
More generally, the user interface 5800 may provide any tools or controls that
a user might usefully employ to
navigate, create, or modify hierarchical relationships among objects.
In the workspace 5814, visual cues may be provided for the relationships among
the objects 5820. For
example, the current object may appear in a foreground, while other objects
may be projected at various distances in
the background, for example according to their relative distance to the
current object in the hierarchy. By adjusting
the size, shape, and color, a visual representation of a representation of a
three-dimensional distance to the current
object may be created within the workspace 5814, Other visual cues to
relationship, such as opacity and two- or
three-dimensional distance may be further employed to reveal associations and
relative distance within a hierarchy
between objects. In this manner, the hierarchy among objects 5820 may be
displayed as a series of projections of a
three dimensional surface, such as a spherical surface or a hyperbolic surface
where the current object is displayed
larger and in greater detail, while progressively more remote objects are
displayed as progressively smaller with less
detail. Other hyper-dimensional representations may be employed to visually
emphasize distance among objects.
Thus, for example, Fig. I OB depicts a view of another current object 5822
within the same hierarchy
depicted in Fig. 10A. It will be noted that the current object 5822 has more
uumerous close relationships to other
objects both below and above the current object 5822 within the hierarchy. The
workspace 5814 may be animated
so that transitions between different current objects are illustrated by
moving objects 5820 within the workspace
5814, both in terms of X-Y position and in terms of a simulated Z position as
described above. The animation may,
for example, rotate a user point-of-view about the three-dimensional or hyper-
dimensional object, siuch as to bring
the current object 5822 into the center of the workspace 5814. In embodiments,
as an object is brought to the center
of the workspace 5814, other attributes of the object may be modified to
enhance the view of the object; for
example, the object may be increased in size, provided in a darker or more
prominent font, placed on bold, provided
with a more intense, saturated and/or vivid color, provided with greater
opacity, provided with a pattern, or
enhanced with other visual cues, such as legends or markings, to provide the
user with a sense that the object is the
current focus of attention. Similarly, as the hyper-dimensional projection
rotates to place the particular object in the
center of the workspace, other objects 5820 can be rotated out of the center
of the workspace 5814. As the other
objects 5820 are rotated away, they can be made smaller, more transparent,
less colorful, less saturated, less intense,
or the like, and markings or legends can be removed or changed to reflect the
relative distance of the object from
the user's focus. Thus, the user can see objects that are related to the
object of current focus, but the object of focus
is provided with visual cues to assist the user in focusing on that object. If
the user wishes to turn focus to another
object, the user can click on the other object (or move the cursor over it, or
the like), so that the other object moves
"forward" (enlarging) and toward the center, and the previous object recedes
away and shrinks, but remains in view,
preserving the opportunity to shift focus back and forth between objects
without losing the connection between the
objects in the overall hierarchy of objects that is represented within the
multi-dimensional space of the workspace
5814. In embodiments, multiple objects, such as those at the same level of a
hierarchy may be presented in
enhanced format, rather than just a single object.
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
Other functions may be provided that are not specifically illustrated in Fig.
10A. For example the user
interface 5800 may provide drop-down menus or other controls to save, open,
delete, and otherwise manage
hierarchies as files. The user interface 5800 may provide access to help
functions, tools, formatting and display
functions, printing functions, and any other tools or functions useful for
exploring and editing hierarchies, or
investigating resources that may be associated with hierarchies. Other tools
may be provided within the user
interface 5800 for a user to associate objects within the current hierarchy
with existing data items, data structures,
metadata, and the like available within the system.
As an example of another enhancement to the user interface 5800, the user
interface 5800 may include a
skill-level sensitive option. For example, a sophisticated user may view all
available functions within any phase of
the process, while a beginning user may view only a subset of available
functions or a subset of available phases,
with additional functions optionally available upon user request.
As yet another exainple, the user interface 5800 may be adapted for specific
tasks, such as an ETL process
for data conforming to regulations (e.g., the Health Insurance Portability and
Accountability Act ("H1PAA") or
Graham-Leach-Bliley ("GLB")) or industry standards (e.g., Electronic Data
lnterchange ("EDI") or Society for
Worldwide Interbank Financial Telecommunication ("SWIFT"). While EDI and SWIFT
designate specific
protocols for data exchange, HIPAA more generally provides minimum standards
for storing and transferring data
ainong covered entities. The environment may be complex and require
communication among inultiple actors. For
example, in a HIPAA environment, a number of communications take place between
health care providers and
insurers, including eligibility inquiries, descriptions of services provided,
claims for payment, and payment. In
these environments, the user interface 5800 may provide access to pre-built
metadata models, connectors, objects,
hierarchies and the like for rapid prototyping and deployinent of jobs and
processes in a data integration
environment that conform to the external standards.
It will be appreciated that the user interface 5800 described above
advantageously provides a set of tools
and controls specifically adapted to managing objects and relationships, so
that a user may more easily create,
investigate, and modify objects and relationships within large, complex
hierarchies such as might be found in a data
integration system 104 or an enterprise computing system, or more generally
across and enterprise.
This modeling approach has broad utility. While a more detailed description of
the application of these
tools to modeling of business issues will be provided below, a simple use
scenario for the interface 5800 in a
business context should quickly illustrate its capabilities. In this example,
a business statement is provided from a
source within the company. The business statement may be an objective, a
philosophy, a current problem or
shortcoming, or any other business matter than can be clearly articulated. The
business statement may be placed as
an object 5820 within the workspace 5814. The object 5820 may have properties
such as an owner, a creation date,
completion or due date, a textual description of the business statement, and
so on. The workspace 5814 may be
live, so that it is immediately accessible to all or some group of users
within the company. The original creator, or
some other user, may add business requirements as new objects 5820 in the
workspace 5814, and associate the
business requirements with the business statement. Each business requirement
may articulate a specific
requirement for addressing the business statement. Collaboration may be
enhanced through use of messaging so
that, for example, if one of the business requirements is assigned to a new
owner, the new owner may receive an e-
mail or other message alerting her to the new business requirement. She may
review the entire context of the
business requirement within the workspace 5814. The new owner may respond to
notification of the requirement
by revising the textual statement, and creating new child objects for the
business requirement that set out one or
21
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
more business processes for meeting the business requirements. Each business
process may be further refined with
child objects to provide subjects such as definitions, rules, and data
mappings relevant to the process. Finally,
relevant objects may be added, such as contacts, vendors, consultants,
databases, and so on, to accumulate all of the
relevant resources within the company and associate them with appropriate
objects within the workspace 5814.
It will be noted that the collaborative process for defining and refining a
business statement occurred above
with a clear hierarchy: business statement 4 requirements -> processes 4
subjects (rules, definitions, mappings).
A number of positive results may be immediately noted. The visual modeling
provides organizational structure to
the business issue. Further, the use of a shared workspace 5814 and messaging
permits seamless collaboration
ainong users of a systein. As an additional benefit, the textual portions of
the resulting structure may be
automatically compiled into full documentation of the business initiative. A
hierarchy may be strictly enforced by
the tools of the user interface 5800, or loosely enforced, or not enforced at
all, to permit more creative flexibility in
defining business matters.
Fig. 11 shows a tool that provides progressive disclosure of detail in a user
interface. The user interface
may be, for example, any of the user interfaces 5200 described above. In
general a visual container 5900 may be
provided within the user interface, and may include a variety of levels of
detail, such as a top leve15903, a middle
leve15904, and a low leve15910, each selectively exposed under user control,
and each providing progressively
greater detail concerning objects within the container.
For example, the container may include a top leve15903 view, in which only a
title is provided, along with
a contro15902 or other active area that permits a user to expand the container
5900 to view greater detail.
In a middle leve15904, accessed by operating the contro15902 in the top level
5903, a user may review
infonnation relating to the top level, such as a list of objects, properties
of those objects, and tools that relate to
those objects. For each object, the middle leve15904 may provide a button or
other control 5908 or active area to
access one or more additional layers of detail, such as the bottom leve15910,
or to perform a task on the selected
information.
By selecting the control 5908, a user may initiate a task that expands the
container to display a bottom
leve15910, which may include, for example, a list of all tasks associated with
an object, or any other form of greater
detail concerning the objects. Within the list, additional controls 5914 or
active areas may be provided to
selectively display atomic properties for an object, or sub-object, task, or
subtask thereof. For example, the bottom
leve15910 may include drop-down lists 5914, data fields for text entry, or any
other controls for selecting
associations, defining or redefining properties or attributes, or otherwise
changing an object or characteristics
thereof.
Thus control over objects and display of related data may be progressively
revealed in increasing levels of
detail, all under control of a user. An interface 5900 arranged in this manner
may advantageously permit full access
to all of the data and controls relating to an object within the current
interface, while permitting a user to conserve
space on a computer desktop when the controls are not needed. This approach
may also advantageously reduce
stacking of windows or dialog boxes that may use up workspace or otherwise
confuse or clutter a user interface
5200.
Fig. 12 shows a page selection tool for use in a user interface. The user
interface 6000 may be, for
example, any of the user interfaces 5200 described above. Within the user
interface 6000, a number of worksheets
or pages in a workspace may be identified by tabs 6002, which may be similar
to, for example, the tabs used to
navigate among worksheets in Microsoft Excel. The user interface 6000 may also
provide forward and back
22
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
controls 6004 for navigating sequentially forward and backward through a
number of sequentially arranged
worksheets or pages, each represented by a tab. A pages indicator 6008 may
indicate the total number of
worksheets that are open, and may also, or instead, provide the current page
or worksheet number in a format such
as "x of y".
When a user holds a cursor over the pages indicator 6008, or provides another
input, such as hovering the
cursor over the controls 6004, the user interface 6000 may provide a drop down
menu of all, or some subset, of the
worksheets so that a user may select a worksheet for navigation to. In this
manner, a user may conveniently
navigate among a large number of worksheets that are open in a workspace
displayed within the user interface
6000.
Fig. 13 shows a tool for recalling a history of previous operations or tasks
within a user interface. The user
interface 6100 may be, for exarnple, any of the user interfaces 5200 described
above. Within the user interface
6100 a history palette 6102 may be provided, which includes a record of
previous tasks. The tasks may be at any
level within the interface 6100. For example, the task may be to navigate to
one of the phases of the workflow
process, or, to simply delete or save a current object. The task may specify
both navigation and function, for
example, by going to the Batch ETL Flow function of the deploy phase and
publishing a batch ETL flow. The user
may return to this task from any location within the user interface 6100 at
any time. Any function within the user
interface 6100 may be stored in the history palette, either associated with
the current object, or with a particular
phase of the workflow, or with a particular item within a particular phase of
the workflow. Thus, tasks may be
recalled at variable levels of granularity within a user environment. Also, a
useful series of operations may be
conveniently ported between projects or other enterprise initiatives for
improved efficiency.
The user may select from the operations or tasks simply by activating them
with a cursor or other user
input. The history palette maybe undocked so that it floats above a workspace
of the user interface 6100. Items
within the history may be deleted, and the history palette 6102 may sort task
according to frequency of use, current
location, or other user preferences.
Fig. 14 shows a menu tree for use with a user interface. The user interface
may be, for example, any of the
user interfaces 5200 described above. In particular, the menu tree 6200 within
the user interface, as depicted, is the
design pillar menu described above, although it should be appreciated that any
menu of items accessible in a user
interface may be used with the control described herein.
The menu tree 6200 may include a number of objects or items, each having
associated with it a control for
selectively viewing additional layers of hierarchically arranged items. For
example, the control may be in a closed
position 6202, identified here by a horizontally directed arrow, or in an open
position 6204, identified here by a
vertically directed arrow. By activating the control, such as with a cursor or
hot key, a user may toggle the control
between the open position 6204 and the closed position 6202. In the open
position 6204, another layer of hierarchy
may be displayed immediately below the object or item, and in the closed
position 6202, no further layers of detail
may be displayed. An object or item with no additional layers of hierarchy
beneath it may be represented by a
different symbol, such as a bullet point 6206. The controls 6202, 6204 may be
persistent, so that a user may control
the degree of detail displayed to a desired level, such as with specific
objects of interest visible, and the menu will
retain those settings upon subsequent uses of the drop-down menu.
In addition to the control itself, other visual cues may be provided for the
hierarchy of objects or items
within the menu. For example, horizontal lines 6208 may explicitly separate
transitions between depths of
hierarchy. As another example, different colors or degrees of shading may be
used for different depths. Thus, for
23
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
example, areas between horizontal lines 6208 may include progressively
brighter shades of a color to indicate
progressively higher (or lower) levels of hierarchy. Similarly, different
fonts can be used to indicate the progressive
levels of the hierarchy.
Arranging access to objects and items in this manner may provide significant
advantages in a complex user
environment. For example, a user may configure each drop-down menu to provide
ready access to commonly used
functions, while concealing other objects or items to maintain useable
workspace within the user interface.
Fig. 15 shows a status bar for use with a user interface. The user interface
may be, for example, any of the
user interfaces 5200 described throughout this disclosure.
In a first view 6302 of the status bar within the user interface, the status
bar may display a simple progress
indicator to indicate that a process is running, such as an animation of a
status indicator moving horizontally back
and forth within the status bar. The status bar may also display alerts,
messages, warnings, progress, and status
through appropriate text and icons. The process may be ajob, background task,
or any other process that might be
running in the computer environment associated with the user interface. The
status bar may be located at the
bottoin of a window displaying the user interface.
A second view 6304 of the status bar within the user interface may be
activated, such as by hovering a
cursor over the status bar. In the second view 6304, the status bar may expand
and display more detailed status
information. For example, in addition to the animated status indicator
described above, second view 6304 may
include a textual description of an active process, a progress bar and/or text
indicated a percentage or degree of
completion, and a textual description of the number of processes running. The
second view 6304 of the status bar
may also include a control for scrolling through additional processes. The
status bar, although expanded, may be
located at the bottom of the window displaying the user interface.
A third view 6308 of the status bar within the user interface inay be
activated, such as by hovering the
cursor over the second view 6304, or by clicking within the status bar of the
first view 6302 or the second view
6304. In the third view 6308, the status bar may expand further and display,
for example, a list of all processes
along with detailed information such as the start time, the user who started
the process, resources used by the
process, and any other appropriate information. The third view 6308 of the
status bar may be located along a
bottom portion of the window displaying the user interface.
Thus, there is disclosed herein a status bar that progressively displays
increasing levels of detail under user
control. The status bar may advantageously conserve space within the user
interface, while always being available
for a user.
In embodiments the methods and systems described herein may be used to provide
a tool that assists a user
in bringing a specific business context to a data integration initiative of an
enterprise. Thus, the workflow-based
user interfaces described herein may include objects that are relevant to a
business context and that are
conventionally maintained in separate computer systems and managed through
disparate word processing tools,
email exchanges and the like. For example, a business analyst may develop a
process that requires one or more
reports. The analyst conventionally specifies the content of the report to a
programmer, who creates the report.
Rather than use ad hoc reports, the workflow methods and systems described
herein allow an analyst to link objects
related to data integration tasks to other processes and hierarchies of an
enterprise, such as hierarchies related to
plans, processes, decisions, and projects of the analyst. For example, the
analyst can link a particular report, such as
a sales report, to the data objects required to populate the report (such as
data from various sales databases for
different regions of the enterprise) as well as to a hierarchy from a business
process that relates to sales, such as a
24
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
process for determining what commissions should be paid in a given calendar
quarter or what commissions should
be proposed for the next calendar quarter, or the like. By connecting the data
integration task to the business
workflow, the analyst can more readily see other opportunities to improve
either the business workflow, the data
integration workflow or both. For example, a business analyst may have
mistakenly believed a particular rule was
necessary to enforce to ensure that a process was performed correctly and may
have embodied that rule in a
proposed workflow. If the data integration designer can see the relationships
among various business workflows,
linked in the user interface as workflows as described herein, the data
integration designer can build a data
integration workflow that avoids the mistaken constraint, rather than build a
workflow that is mistaken because of
the lack of understanding of the business context.
One recurring problem with using data in a business context is the lack of
coordination between the
structure of data and the structure of a business problem. In a data model, a
tightly coupled hierarchy is maintained
among data, and specific sources are identified. In a business context,
relationships are typically defined by loosely
coupled associations of information, which may or may not form a definite
structure or hierarchy. The following
Figures 16-20 set out guidelines by which business issues may be more clearly
defined and related to data
structures.
Fig. 16 shows a map for a subject in a business model. The subject map 6400
illustrates a mapping of a
subject, in this case an address, to other subjects within the business layer.
The address may have a definition and
some set of business rules. The address may also participate in several other
hierarchies, such as a customer
hierarchy, a vendor hierarchy, and an employee hierarchy. An address may have
two defined forks, such as a
billing address and a shipping address, although the address itself may be
neither of these. Each fork includes its
own definitions and rules. The address subject itself may have a number of
related subjects, such as a location, an
area, and a country. These subjects may in turn have a definition and rules,
and have participating subjects, such as
an area having a city, state, and/or postal code. The business layer may also
have rules, as depicted below, which
further complication the identification of information in the business layer.
Fig. 17 depicts rules and definitions for a subject in the business layer. In
the example of Fig. 17, a billing
address 6502 has a definition 6504 and a plurality of business rules 6506. The
definition 6504 may, for example,
include a textual description of the billing address 6502, and may indicate
its distinction from, and relationship to,
its higher level subject, an address. The business rules 6506 may, for
example, indicate conditions based on the
country and other criteria for an address to be valid. These rules may be
exclusive or inclusive of one another. The
rules may also be inherited by other subjects, particularly subjects dependent
on the subject having the rule. For
example, the rules of the business address may have a child subject for zip
code, which may inherit a business rule
of the business address. By contrast, a related subject such as the shipping
address may or may not inherit a
particular rule.
Fig. 18 depicts a business process in the business layer. In this example, an
address verification process
6600 involves the steps of receiving an address 6602, matching the address to
a reference 6604, and certifying the
address 6606. Each of these steps may have a specific definition of one or
more actions to be taken, and each step
may be expanded into smaller substeps in a hierarchical or other fashion to
fully characterize the process 6600. The
steps may include conditional rules, decision points, or other flow control
mechanisms. Another example of a
business process is householding, the process by which a number of individual
are identified as belonging to a
single household. This may be accomplished by comparing addresses for a number
of people and linking people
having an identical address.
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
There are other aspects of the business layer and business modeling not
depicted explicitly in Figs. 16-18.
One example is a business metric. The business metric, or measure, typically
characterizes a degree to which a
business rule or process achieves a desired result, but may more generally
provide a quantitative measure of any
aspect of the business, such as conformance to business rules, results of
business processes, and so on. Another
example is roles. Roles describe the rules by which individuals interact with
other components of the business
layer, and may relate to an individual's rights or responsibilities with
respect to various components, such as
creation, review, approval, editing, and so on. The particular aspects of a
role may be defined according to the type
of individual and their relationship to the business issue. For example roles
may include a business process owner,
business analyst, subject matter expert, manager, and so on.
Fig. 19 depicts an association of elements between the business layer and the
data layer. The universe of
information may be represented as a business layer 6702 and a data layer 6704
interrelated by tags 6706. The
business layer 6702 may include the business subjects, business processes,
business rules, business metrics,
business roles, and any other infonnation and corresponding relationships
gathered by modeling a business matter
as described above. The data layer 6704 is much more concretely structured.
Data is always contained within a
data source, the source has a defined format, whether relational, hierarchic,
flat/sequential, or complex, Each
format has low-level components that can be termed fields or columns. Some
information can be redefined into
larger multi-field components (e.g. Address Line 1, Address Line 2, and
Address Line 3 fonn an Address). Fields
and columns may be combined into larger structures that may be single
transactions or files or a database table.
These larger structures may be contained within a repository such as a
database or a file directory. All data requires
specific access methods and this information is associated with the data
source. Relationships between components
of the business layer 6702 and components of the data layer 6704 may be
specified using tags 6706. Tags 6706
represent links between any element in the data layer 6704 and any element in
the business layer 6702 at any
hierarchical level. Tags may be directional, and may include one-to-one, one-
to-many, and many-to-one
relationships between the layers 6702, 6704.
Fig. 20 is a map of linked business and data elements. An information map 6800
shows the association of
components of the business layer to components of the data layer using tags.
In the example of Fig. 20, a central
subject 6802, such as an address, may have relationships to other subjects
6804, such as a billing address and a
shipping address that inherit some or all of the elements of the address, as
well as sub-elements 6804 that more
specifically describe and define various aspects of the central subject 6804,
such as a location, and area, and a
country. These, of course, may similarly have sub-elements more specifically
refining them. Each subject 6802,
6804 may be mapped to a specific database component 6806 through a tag. This
may include, for example, an
identification of database, a table within a database, a column within a
table, a reference database, or any other
information specifying a relationship to a database entity or component.
This conceptual framework may be used to associate a business context to a
data environment. More
specifically, the user interface tools described above, such as with reference
to Figs. 4A-10B, may be employed to
model a business initiative and map the business initiative to data sources in
a user environment that seamlessly
transitions between the business layer and the data layer. The following
example shows how a data object may be
associated with subjects in a business layer model.
Fig. 21 depicts the addition of a data object to a workspace of a user
interface. The user interface 6900,
which may include any of the user interfaces 5800 or related components
described above, may particularly include
a menu 6902 of data objects and a workspace 6904, which may be the workspace
5814 described above. The menu
26
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
6902 of data objects may include any data items, data fields, tables,
metadata, data sources and the like known to
the interface 6900 froin any data sources, such as the data sources 102
described above, within an enterprise or
other business or computing entity, and may provide tools for searching and
navigating among the data objects.
Through the user interface 6900, a user may place a specific data object 6906
in the workspace 6902.
Fig. 22 depicts an association of a data object to business objects in a
workspace of a user interface. The
user interface 7000, which may be any of the user interfaces 6900 described
above, may include a workspace 7804
containing a data object 7806 that was placed in the workspace using tools of
the user interface 7000. A user of the
interface 7000 may also retrieve a mode17808 of a business initiative. Using
tools provided by the interface 7000,
a user may then associate the data object 7806 with the inodel 7808 of the
business initiative using any relationships
that the user chooses to define. As an example, the business initiative may be
a marketing company's wish to
identify household groups from lists of individual customers. The business
initiative may define a business process
through which groups of customers with identical addresses are characterized
as a family, but only if they have an
identical last name. A user may place specific address objects from the
company's databases into the workspace
7804 for association with statements, processes or other components of the
mode17808 for the business initiative.
Once the model 7808 has been more fully associated with a company's
database(s), data integration processes may
be quickly prototyped and deployed from the resulting infonnation. A more
detailed example of developing data
integration processes from a business initiative is provided below.
Fig. 23 shows a process for designing a data integration process from a
business context. It will be
appreciated that the processes described below may be realized in hardware,
software, or some combination of
these. The processes may be realized in one or more microprocessors,
microcontrollers, embedded
microcontrollers, programmable digital signal processors ot other
progranunable devices, along with internal and/or
external memory for storing program instructions, program data, and program
output or other intermediate or final
results. It will further be appreciated that these processes may be realized
as computer executable code created
using a structured programming language such as C, an object oriented
programming language such as C#, C++ or
Java, or any other high-level or low-level programming language that may be
compiled or interpreted to run on any
uniform or heterogeneous group of hardware and software platforms, including a
computer or computers, networks
of computers, and combinations thereof. The processes may also employ a wide
variety of tools, platforms and
architectures to achieve a scaleable enterprise metadata management system.
While specific examples of software
tools and platforms have been provided above, other platforms and teclmologies
exist and may be usefully
einployed with the processes described below. Finally, it should be
appreciated that, while a specific example of a
business initiative is provided below for purposes of explanation, this
example in no way limits the scope of the
systems and methods described herein.
The process 7100 may begin by setting up an initiative, as shown in step 7102.
As an example, a business
analyst may set up an initiative or project in the system by defining a
project title, such as "Actuarial UR Project".
The analyst may add description, definition, manager, business sponsor, and
any other descriptive information for
the project. Descriptive data may be captured, for example, through keyboard
input, or may be imported from any
existing documents or other materials available in electronic or other form.
Once the initiative has been set up, the
process 7100 may proceed to step 7104.
As shown in step 7104, the initiative may be broken down into objectives.
Continuing with the actuarial
project exaniple provided above, an analyst may decompose the business
initiative into specific business objectives
27
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
using, for example, the hierarchy of business subjects described with
reference to the business layer above. The
resulting decomposition may yield the following business objectives:
a. Capture 7 years of hospital data
b. Capture 3 years of physician data
c. Capture 2 years of pharmacy data
For each objective, the business analysts may define a natne (title),
description (summary), definition
(detailed), business owner and any relevant notes. The format of definitions
may be controlled by user-defined
formats such as templates, which may be employed in the description process to
establish a useful and consistent
hierarchy for the modeling process. Once the business initiative has been
decomposed into objectives, the process
7100 may proceed to step 7016.
As shown in step 7106, objectives may be further decomposed into statements.
The business analyst may
further de-compose each objective into a set of business statements that are
the general requirements needed to
satisfy that objective. Continuing with the actuarial example provided above,
a breakdown of the first objective
may result in the following more specific statement into requirements:
a. Capture 7 years of hospital data
1. Identify all hospital data sources
2. Transform hospital data to a common format
3. Validate standardized hospital data.
This business initiative model may be stored as a data structure for
subsequent steps of the process 7100,
as well as any other use, reuse and examination by users of the system. Once
the objectives have been decomposed
into specific requirements for meeting the objectives, the process 7100 may
proceed to step 7108.
As shown in step 7108, the model of initiative, objectives, and requirements,
may be published as a
functional requirements document for the initiative. A reporting feature maybe
provided by the process 7100,
through which the functional requirements document may be automatically
generated from data collected directly
from the model. This may be a collaborative process through which numerous
users contribute to and edit both the
structure of the model and the underlying descriptions. A review process and
an approval process may be
conducted through which numerous participants are expressly requested to
comment on and approve the document.
The process 7100 may automate approvals by collecting approvals from
appropriate personnel and ensuring that all
needed approvals are obtained for a final version of the requirements
document. The resulting requirements
document may present the requirements as, for example, a hierarchical tree, an
outline, a full textual description, or
any combination of these. The functional requirements document may be
published, for example, in paper or
electronic form. It will be appreciated that publication in electronic form,
in particular, may permit rapid
dissemination through a business entity, and may provide the functional
requirements document in an interactive
form for convenient navigation among, and investigation of, the initiative,
the objectives, and the requirements, at
any desired level of detail. Once the functional requirements have been
published, the process 7100 may proceed to
step 7110.
As shown in step 7110, an information model maybe developed. A business
analyst may apply
knowledge of an enterprise to develop an information model, a data structure
that is an abstract representation of the
real world in which the enterprise operates. The model may be built by
defining subjects as building blocks that can
be related to one another through graphical relationships that primarily
consist of "is part of' or "is type of'. The
subjects themselves can be of various types in a range from a very high
abstract conceptual level (e.g., "hospital") to
28
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
a very atomic level (e.g., "type of service") equivalent to individual data
elements. For each subject, a user may
define a subject name, description (summary) and definition (detailed),
business owner and any other relevant or
potentially relevant infonnation. The format of definitions may be controlled,
for example, by user-defined fonnats
such as templates to facilitate consistency, and in particular, collaboration
in the development effort. It will be
readily appreciated that user interface tools described generally above may be
conveniently employed in this
modeling step. Once the infonnation model has been developed, the process 7100
may proceed to step 7112.
As shown in step 7112, requirements may be added to the information model to
provide a data structure
that describes a business context. A user may attach specific requirements to
any subject in the information model
to more specifically detail aspects of each. A modeling tool, such as the user
interface 5800 described above in
reference to Figs. 4A-10, may provide specific object types to facilitate this
modeling step. For example, the
following types may be defined for the hospital actuarial initiative:
Business Requirement (general)
(Type of Service valid code values are "01" -"17" )
Business Rule
(Place of Service must be appropriate for Type of Service)
Business Process
(Notice of Admission must precede Inpatient claim)
Business Process Step
(Step 1 - Approve Notice of Admission)
Business Metric
(Type of Service code must contain 100% valid data)
For each object type, a user such as a business analyst may capture an object
name, description (summary),
and definition (detailed), business owner and any other relevant or
potentially relevant infonnation. The fonnat of
definitions may be controlled by user-defined formats such as templates to
provide consistency, and more
particularly to facilitate collaboration among a number of contributors. Once
the information model has been
completed, the process 7100 may proceed to step 7114 below.
As shown in step 7114, the information model may be published. A reporting
feature may be provided by
the process 7100, through which the infonnation model may be automatically
documented in textual or other form
directly from the information model. A review process and an approval process
may be conducted through which
numerous participants are expressly requested to comment on and approve the
document. The process 7100 may
automate approvals by collecting approvals from appropriate personnel and
ensuring that all needed approvals are
obtained for a final version of the infonnation model. It should be
appreciated that, while publication may occur at
a specific point in a development process, the information model, or subsets
of the model and/or related
documentation and annotations, may more generally be published at any time.
The model may be presented in a
number of forms, such as a hierarchical tree, an outline, a full textual
description, or any combination of these. The
information model may be published, for example, in paper or electronic form.
It will be appreciated that
publication in electronic form, in particular, may permit rapid dissemination
through a business entity, and may
provide the model in an interactive form for convenient navigation among, and
investigation of, the subjects, rules,
processes, and so on at any desired level of detail. Once the information
model has been published, the process
7100 may proceed to step 7116.
29
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
As shown in step 7116, the information model may be linked to the business
initiative, or requirements and
other subcomponents thereof, At this point in the development cycle, the
information model should contain a
representation of all of the relevant real-world data and any other relevant
requirements for subjects in the
information model. The applicable data subjects may then be linked to the
specific business statements in the
initiative to provide detailed specifications for each business requirement of
the initiative. Once the applicable
areas of the information model have been linked to the initiative, the
combined information model forms a formal
description of relevant aspects of the business layer discussed above, and may
be provided at any level of
granularity desired by the architects or appropriate to a particular business
initiative. This combined information
model may be submitted for review so that any requirement modifications or
other appropriate additions, deletions
or changes to the initiative, or more generally, the combined information
model, can be made. Once the combined
information model is complete, the process 7100 may proceed to step 7118.
As shown in step 1118, the combined information model may be published as a
functional specifications
document. The combined information model of the functional specifications
document may include the infonnation
model described above as well as the business initiative model that is
integrated therewith. A reporting feature may
be provided by the process 7100, through which the functional specifications
document may be automatically
generated in textual or other form directly from the combined infonnation
model. A review process and an
approval process may be conducted through which numerous participants are
expressly requested to comment on
and approve the document. The process 7100 may automate approvals by
collecting approvals from appropriate
personnel and ensuring that all needed approvals are obtained for a final
version of the functional specifications
document, It should be appreciated that, while publication may occur at a
specific point in a development process,
the infonnation contained in the functional specifications document, or
subsets thereof, along with any related
documentation and annotations, may more generally be published at any time.
The functional specifications
docuinent may be presented in a number of forms, such as a hierarchical tree,
an outline, a full textual description,
or any combination of these. The functional specifications document may be
published, for example, in paper or
electronic fonn. It will be appreciated that publication in electronic fonn,
in particular, may permit rapid
dissemination through a business entity, and may provide the document and
associated models in an interactive
fonn for convenient navigation among, and investigation of, the subjects,
rules, processes, and so on at any desired
level of detail. Once the infonnation model has been published, the process
7100 may proceed to step 7120.
As shown in step 7120, the combined information model of the functional
specifications document may be
linked to real data structures of an enterprise. It will be appreciated that
the information model and portions thereof
are data structures themselves. However in the following description, the term
"data structure" will refer to data
structures within an enterprise other than the model(s), such as data
structures that might be used in implementing a
business initiative or portions thereof as one or more data integration
processes. This phrasing is intended to more
clearly distinguish between data structures created during the modeling
process to describe a business context or
initiative, and data structures already existing within the design process
7100 to which the model might apply. The
use of these tenns in the following description is made without any loss of
generality to the associated tenns
("model", "information", "data", "data structure", etc.) throughout this
description.
Given an approved functional specifications document, a user such as a
business analyst can begin the task
of linking subjects in the (combined) information model to actual data
structures (e.g., tables and columns), which
may, for example, be data structures from any of the data sources 102
described above. A user may collaborate
with other users, for example where a business analyst collaborates with a
data analyst, to identify correspondence
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
between aspects of the infonnation model and aspects of the data structures of
the enterprise, or where appropriate,
data structures outside the enterprise. The links to data structures may
include an identification of a data source,
and any data or metadata related thereto, and a defined target data structure
that may be used for any data
integration process derived from the process 7100.
It should be appreciated that the linking of data structures to models, which
corresponds generally to the
tagging described above with reference to Fig. 19, may occur at a number of
different levels of abstraction and
detail according to the relative complexity of the data structures and/or the
information model. Thus, for example, a
subject may correspond to a database, a table of a database, a column within a
database, or a data instance within
one or more columns, or any combination of any or all of these. In general,
the more accurate and detailed the
mapping, the easier subsequent development of data integration processes
becomes. Once the mapping is complete,
the process may proceed to step 7122 below.
As shown in step 7122, the mapping of the (combined) information model to
other data structures may be
published. Using the links between the infonnation model and the data
structures, a mapping model may be
produced. This may include documentation of the relationship between the
various aspects of the information
model and the surrounding environment of enterprise data sources. This may
also include more functional
documentation such as a mapping of source data to target data. The mapping
document may also be, or include, a
logical mapping that, for each data source and target, maps a source column to
a target column (n to n) to fully
characterize a source-target mapping associated with the modeling process. A
reporting feature may be provided by
the process 7100, through which the mapping model (either the database mapping
model, a more general
information-to-data mapping model, or any other subset or variation of these)
may be automatically generated in
textual, graphical, or other form. A review process and an approval process
may be conducted through which
numerous participants are expressly requested to comment on and approve the
document. The process 7100 may
automate approvals by collecting approvals from appropriate personnel and
ensuring that all needed approvals are
obtained for a final version of the mapping model. It should be appreciated
that, while publication may occur at a
specific point in a development process, the information contained in the
mapping model, or subsets thereof, along
with any related documentation and annotations, may more generally be
published at any time. The mapping model
may be presented in a number of forms, such as a hierarchical tree, an
outline, a full textual description, or any
combination of these. The mapping model may be published, for example, in
paper or electronic form. It will be
appreciated that publication in electronic form, in particular, may permit
rapid dissemination through a business
entity, and may provide the document and associated models in an interactive
form for convenient navigation
among, and investigation of, the subjects, rules, processes, data mappings,
and so on at any desired level of detail.
Once the mapping model has been published, the process 7100 may proceed to
step 7124.
As shown in step 7124, data integration tools may be applied to the mapping
model or, stated differently,
the model may be applied to data integration tools. The specifications and
mappings developed during the process
may be transferred to a variety of data integration tools as design processes
such as, for example, the processes
outlined with respect to Figs. 4A-9. This may include, for example,
comprehensive analysis and cleansing of
source data, the development of technical specifications for data integration
and transformation, and any other tasks
relating to the investigation, development, design, deployment, and operation
of data integration processes.
Similarly, any metadata generated by or identified during the process 7100 may
be reused in other business
initiatives, particularly initiatives involving similar subject areas and
requirements.
31
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
More generally, modeling techniques and processes 7100 described above may
flow directly into the
workflow management tools described in Figs. 4A-9 to yield a complete, end-to-
end solution for generating data
integration jobs, processes, functions, and the like from business
initiatives. At the same time, it should also be
appreciated that many of the tools and techniques described herein may have
separate utility as stand-alone products
or processes. As an example, the workflow-based user interfaces described
herein may be provided with a subset of
the pillars, or a subset of tasks within one or more of the pillars, described
above in Figs. 4A-9. Thus, for example,
a user interface may provide a business context workflow design workspace for
use by non-technical personnel that
might include investigate and design pillars, while omitting pillars related
to development, deployment or operation.
All such combinations, decompositions, and variations to the processes, user
interfaces, and systems described
above form a part of the inventive disclosure provided herein.
While a specific example of a process 7100 is provided, it will be appreciated
that the order of the steps
described in the process 7100 may be changed, or otlier steps may be added or
removed from the process, without
changing the general objectives or advantages thereof. For example, the
decomposition of objectives into specific
statements may be revised after the information model is linked to a specific
business initiative, or an additional
iteration of adding business requirements may be required if the linking of
the information model to actual data
structures reveals potential refinements to the business initiative, or
conversely if the linking of the information
model to actual data structures reveals issues that cannot be addressed with
currently available data. All such
variations and modifications are intended to fall within the scope of the
systems described herein.
Having described a process 7100 for modeling business initiatives, certain
data structures and interface
features associated with the process 7100 are now described in greater detail.
Fig. 24 shows decomposition of an initiative into objectives, as described for
example, with reference to
step 7104 of the process 7100 above. A decomposition 7200 may include a
business initiative 7202 with
characteristics such as a title, description, definition, manager, business
sponsor, and any other descriptive
information for the initiative. The initiative 7202 may be broken down into
any number of objectives 7204 that
more specifically characterize the initiative. The stracture of objectives
7204 may be fixed by an architect of the
initiative 7202, or may permit changes by other users involved in the modeling
process. It should be appreciated
that, while a simple hierarchy is depicted in Fig. 24, any degree of detail
and complexity may be used to decompose
a business initiative 7202 into related objectives 7204.
Fig. 25 shows decomposition of an objective into statements, as described for
example, with reference to _
step 7106 of the process 7100 above. A decoinposition 7300 may include a
business initiative 7302, such as the
business initiative 7202 described above, at least one objective 7304, such as
the objectives 7204 described above,
and any number of statements 7306 more specifically setting out the
requirements for meeting each objective 7304.
It should be appreciated that, while a simple hierarchy is depicted in Fig.
25, any degree of detail and complexity
may be used to decompose a business initiative 7302 into related objectives
7304 and statements 7306,
Fig. 26 shows the development of an information model, as described for
example with reference to step
7110 of the process 7100 above. The subjects of the infonnation model may
correspond, for example, to subjects of
the business layer described above, and may be associated in a hierarchical or
other structure. The relationship of
business subjects may be modeled using, for example, any of the user
interfaces 5800 or components thereof,
described with reference to Figs. 4A-IOB.
Fig. 27 shows the addition of business requirements to information model
objects, as described for
example with reference to step 7112 of the process 7100 above. The subjects
7502 of the information mode17500
32
CA 02579803 2007-03-15
WO 2006/026686 PCT/US2005/031020
may be arranged in a hierarchy as generally described above. Other elements of
the business layer, such as business
requirements 7504, business processes 7506 or substeps 7508 tliereof, business
metrics 7510, and business rules
7512, inay be created and associated with other elements of the model. The
addition of business requirements and
other coinponents may be performed using, for example, any of the user
interfaces 5800 or components thereof,
described with reference to Figs. 4A-10B.
Fig. 28 shows linking of an information model to an initiative. In general the
workspace 7600 depicted in
Fig. 28 may include any number of elements from the information model (e.g.,
subjects, requirements, rules,
processes) and the business initiative model (e.g., initiative, objectives,
statements), along with any other relevant
information contained therein. The components of these models, or subsets
thereof, may be presented within the
workspace 7600 and associated using tools of a graphical user interface, such
as any of the user interfaces 5800
described above.
Fig. 29 shows linking of an information model to data structures. As noted
above, this linking or tagging,
referred to here as a mapping 7700, may be used to specify how aspects of the
information model correspond to
structures such as tables or columns within a source database 7702, which may
be any of the data sources 102
described above. The mapping 7700 may further specify a target database, which
may also be any of the data
sources 102 described above, and inore specifically indicate how structures of
the source database 7702 relate to
structures of the target database 7704 to permit convenient realization of
data integration processes.
While the invention has been described in connection with certain preferred
embodiments, it should be
understood that other embodiments would be recognized by one of ordinary skill
in the art. All such variations and
modifications are intended to fall within the scope of this disclosure.
33
