Note: Descriptions are shown in the official language in which they were submitted.
3 ~
~ACKGRQU~a O/~b~ QY
1. Field of the lnvention.
The present invention relates to the f~eld of
artiflcial intelliqence based solutions to problem solving;
more specifically, the present invention relates to
selection of visually aesthetlc color combinations using an
associati~e memory system, specifically a neural network.
2. Description of Related Art.
It is known in the art to employ various methods for
to accomplish "artificial intelligence" or "AI" with
computer systems. While a strict definition of AI is not
available, generally, AI may be described as a system in
which ~ com~uter is able to emulate deductive inferential
reasoning, e~g.~ the system may reach conclusions based on
certain inputs after it has been "trained" or instructed in
a certain set of rules or experiences.
Such a definition encompasses so-called "heuristic
search" models as well as "associative memory" systems and
"connectionist" models. Each of these terms will be
defined in more detail below.
. , ., "
. , ~ " . . :. :.
', ~
"~ ~
2 ~ 3 2
The Heuri~ aL~h_~Q~91
A heuristic search model may be thought of as a system
in which a problem ls solved by following the branches of a
tree. A set of decisions are made; each decision
corresponding to a branching point on the tree. The search
is controlled by a set of rules or "heuristics"; heuristics
allow the system to focus on the branches cf the tre~ most
likely to provide a solution. Heurlstic search models have
been commercialized in a large number of applications such
as game ~laying, natural-language processing, and so~called
'lexpert system" and l'knowledge-~ased" applications.
An example of the shortcomings of such a heuri~tic
approach is described in David L. Waltz, ~LS~E~5~ ~
uildin~ a Truly In~elli~ent Machin~i, DAEDALUS Journal of
the American Academy of Arts and Sciences, ~olume 117,
Number 1, pp. 191-212.
Waltz describes an application for a heuristic mGdel
for determining whether a given object "x" is a bird. In
this aPplication~ one heuristic rule may be "if x is a
bird, then x can fly". However, Waltz continues to
describe the "qualification problem"; that i~, given a
general rule, one can always devise a fact pattern which
requires further qualification of the rule. For example,
in ~iew of a world in which penguins and ostriches are
.
.~
r
. ,' .. ' :
. ~ ~.r ~
3 ~
classified as ~birds" ~nd in which both of these species
cannot fly, the rule must be m~dified t~ "if x ~s a b~rd,
then x ca~ fly, unless x ~s one of a set of flightless
birds". Further exceptions to the rule are necessary to
reach correct results when operating in a world in which a
bird may have had its wings clipped, may be dead, may have
its wings soiled with oil after an oil spill in its
environment, etc. The possible exceptions to the ~general
rule" are limltless.
Heuristic search models further suffer from
shortcomings in applications where "rules" are difficult to
define. The rules may evade definition because the
cognitive process utilized by humans in solving a
particular problem may be difficult to define; because
so-called "experts" in the field differ on what the l'rules"
are or should be; or some combination of these problems.
As will be seen, it is primarily this latter shortcoming
which leads to the present invention.
Asso~ ive ~emQry Systems
Associat~ve memory systems have been proposed in
which a memory database is provided having some historical
set of solutions to a problem or set of problems. The
associative memory system solves a current problem by
examlning symptoms or characterlstics of the current
' .,.
- ~ ~
..
,
.. ' ' ' ; '
. . . . .
problem and comparing those symptoms to previous solutions
to the problem. A solution is chosen from the historical
set of solutions where the chosen solution had ~ set of
symptoms close the present problem.
This type of syste~ may be differentiated from
heuristic search models in its employing a "lookup" process
rather than a search through a set of rules. The items or
solutions "looked up" are representations of specific
solutions rather than rules.
~n example of such an associative memory system is an
application for diagnosis of medical problems. Gi~en a
database of symptoms and patient characteristics and a set
of symptoms and characteristics for a particular patient,
an associative memory system has been described by Waltz ~n
which the system finds the most similar pre~ious patients
and hypothesizes that the same diagnosis should be applied
to the new patient.
An associative memory system may be designed which can
"learn" by applying its database to a new set of s~mptoms,
determining whether the proposed solution yields a
desirable result, and, if a desirable result is achieved,
storing the new set of symptoms and solution in the
database.
, . .. .
~,1 l~etworks
A "neural network" may ~e described as a particular
implementation of an associative memory. In a neural
network, nodes or processlng units are interconnected by
links, each link having a weight or strength.
Neural networks are inspired by current thought on
processing and storage of informatiQn in the human brain.
(The brain consists of a large number of highly
interconnected elements which apparently send very simple
excitatory and inhibitory messages to each other and update
their excitations based on these messages.)
In a neural network, each individual neuron may act as
an input, as an output or may be "hiddenn. ~A hidden
neuron is connected to other neurons but cannot be accessed
directly from input or output channels. This will be
di~cussed in more detail below.)
The connectlon between any two particular neurons is
strengthened each time information flows between the two
particular neurons; that is, gi~en a problem represented by
input neuron A, if a solution is found represented by
output neuron B, the connection between A and B is
strengthened.
A further description of neural networks may be found
in Jack D. Cowan and David H. Sharp, Neural ~ an~
; , ,
~ ; -
.
:
2~ ~7~
Artificial Intelli~nce, ~AEDALUS Journal of the American
Academy of Arts and Sciences~ Volume 117, Nurnber 1, pp.
85-121 .
S A network is typically subjected to a "training"
process during which it is trained with a set of inputs~
It is desirable that the system, durin~ this training
proces~J eventually reaches a stabili~ed configuration. A
stabilized configuration may be thought of as a state of
the network in which applying a set of ~nputs to the
network will yield a desired set of outputs.
A~to-~ssoci~tive Netwo~ks
Auto-associative networks are a class of associatlve
memory systems which are trained through use of a set of
examples; these examples are often termed "exemplarsn. The
network is trained by using the exemplars as inputs to the
network and also using the exemplars as target outputs.
Weights of links between input and output nodes are
adjusted until applying the set of exemplars at the input
nodes leads to responses at ~he output nodes which are
suffi~iently close to the target responses.
Typlcal known auto-associati~e networks are only
capable of working with binary values (O's and l's). Use
',:
,: . .
, . . ;
. ~ , . ~ .
:
.
" 2~73~
of simple binary inputs allows for simpler stabillzation
and training of the network.
Hov~ield Networ~s
Cowan et al. describes a particular auto-assoclative
memory system known as the Hopfield net. The Hopfield net
is a network of neuron-like elements having symmetric
connections. (Symmetric connections are formally deflned
as follows: If x and y are two neurons, their connections
are symmetric if the weight of the x to y connection is
equal to the weight of the y to x connection.)
Cowan et al~ discusses and points out that Hopfield
nets are not "true~ neural nets because each eleme~t, or
neuron, must both excite and inhibit its neighbors. More
specifically, each neuron will typically excite one
neighbor and inhibit another.
One problem with a Hopfield net is that they can
easily get trapped in a metastable configuration. In order
to avoid metastable configurations, modified Hopfield nets
have bPen proposed using a "Monte Carlo" procedure to allow
the net to escape from the~metastable configuratlon. Using
a Monte Carlo prGcedure, the networ~ makes essentially
random configuratlon changes. If the change results in a
.
~ ; more stable state, the change is retained; if the change
"
xesults in a less stable c~nfiguration the change is ~
,
.
, , ... . , . ~ . ,
':- ' ~ , , . , , ' , . ' ':,:' : ~-
' ' ~ ' ~ , ' ` - ' ~ . :
'~" ' ' , ", ' : i
' '
.
cancelled. In this way, a stable configuration is
eventually found, however, learn~ng can be very slow.
Sinale an~ Multllayer ~et~ork~
;Çowan ~_al~ further describes a type of network known
as a multilayer network. Such a network may be described,
in a rather simplified fashion, with reference to Figure
l(A) and Figure l(B) and with comparison to singl~ layer
networks. The Hopfield net, discussed above, is
computationally equi~ilent to a single layer network.
Figure l(A) lllustrates a single-layer network having
3 input nodes 101, 10~ and 103 and 3 output nodes 104, lD~
and 106. Each of the input nodes 101, 102 and 103 are
directly connected to each of the output nodes 104, 105 and
106. A simple network, such as illustrated by Figure l(A),
may be useful for certain pattern recognition and
classification problems. In general, this type of network
may recognize patterns which can be classified and
separated by a hyperplane. This shortcoming i explained
in greater detail with reference to Richard P. Lippmann,
Introd~ction to ComD~tina with Neural Ne~woIk~, IE~E ASSP
Magazine, April, 1987, pp. 4-22.
A class of networks which may recognize more
complicated patterns and which overcomes many of the
problems associated with the type of network shown in
"
~.:
.. . .
'`' ' ':
. ~ .
2 ~ 3 ~
..
Figure l(A) is termed a "multilayer" network. Such
networks typically have an $nput and output layer
lnterconnected by "hidden" layers.
ippm~nn describes a single-layered network as being
able to distinguish regions bounded by a hyperplane.
Multi-layer networks are arbitxary regions, the complexity
of which is limi~ed by the number of nodes in the network.
Another way of viewing the shortcomings of a single
layer network is that, in general, single layer networks
may be fairly proficient at recognizing patterns which are
reasonably close to their training patterns; however,
single layer networks perform relatively poorly at
recognition of patterns which are dissimiliar to their
training patterns.
Figure l(B) illustrates a multilayered network having
3 input nodes 110, 111 and 112 and 3 output nodes 116, 117
and 118. In addition the network comprises 3 "hidden"
nodes 113, 114 and 115. The ~erm "hidden" is descriptive
of the fact that the node is not directly accessible from
outside the network for either input or output; rather, a
hidden node is connected between input and output nodes in
the networkO
!
, ' `.'. ' ' .
~ 2~73~
Back Propaga~iQn
Back propagation is one technique used for training of
multilayer networks. During triining involving back
propagation, the network ~s first subjected to a forward
stage in which inputs to th~ ~etwork are simulated and
responses recorded. During the backward stage, the weights
of the connections between hidden units and the output
units are adjusted and, after adjusting the weights of
these connections, the weight of the connections between
the input units and the hidden units are adjusted.
Figure l(C) is illustrative of such network which may
be trained with a back propayation technlque. As discussed
above, first t~e network is subjected to an input at input
node 120. The responses at output nodes 122, 123 and 124
are recorded. The weights of connection 126, 127 ~nd 128
are then ad~usted and, after adjusting the weights of these
connections~ the weight of connection 125 is
adjusted--thus, the name back propagation. The example of
Figure ltc) ~llustrates a network with a single input node.
~ther networks may have multiple input nodes; further,
other networks may hav~ a lesser or greater number of
output nodes~
.~ .
, . ..
. , ~ : : . . ~:; . .
,, :.- : :
, ::,. .. .. : . : -: . -
: : : :
.: : ~ . :
" . . j : .;: .: ~
2 ~ 3 ~
Color S~lection Systems
The above~discussion has centered on the history and
development of known AI systems. The present invention is
centered in the use of AI systems for color selectlon;
specifically, as will be descr1bed below, the preferred
embodiment of the present invention is useful in ~he
selection of colors for a computer interface and has
specific application in the selection of colors fox
elements on Apple Macintosh computer interface.
10It is known in the art to utilize AI sys~ems for color
selection applications. Penny Bauersfeld, Col~r Palet~e: ~ -
~nowl~d~-~a~d System ~o ~elect Co]ors fox a Pal~
Masters Thesis, Rochester Institute of Technology, School
of Computer Science and Technology, Au~ust 16, 1988,
describes a system for color selection using a
knowledge-based system. ~auersfeld describes a
knowledge-based system which is used to advise users in
selecting colors for a palette used for compu~er screen
design applications. The knowle~ge-base proposed by
Bauersfeld may reason about the relation of individual
colors in a set with others colors in ~he set based on a
set of rules derived from research on interface design and
color theory. Bauersfeld presents argument stating "The
skills required to perform color selection are cognitive,
~1
"
~'
.. ~ . .
. .
,: . . ,, I.`, . . :
, ~ ... .:
. ~, . -
., .
. . ~
^~' 2 ~ 3 2
and are readily articulated by experts". ~auersfeld, at
page 10.
A second system for color select.ion in computer
systems is described by Barbara J. Meier, A~E-. ~ Color
Expert System fQL User Interfa~ç D~ n, Proceedings of the
ACM, SIGGRAPH Symposium on User Interface Software, Banff,
Alberta, Canada, October 17-19, 198~, pp. 117-128.
M~l~X describes an expert system which employs a set
of rules in the selection of colors for computer interfaces
1~ and the like.
Although each of ~auersf~l~ and ~i~L propose systems
for color selection, both of these references rely on a set
of rules or heuristics to implement their proposed color
selection process. As can be appreciated, although rules
are available to aid in the selection of colors, no one set
of rules is available which satisfies the criteria of all
of the so-called ~experts" or which coYers all possible
color selection combinations.
Therefore, the present invention recogni~es that any
implementation based on a set of rules or heuristics will
suffer from shortcomings similar to the general
shortcomings of heuristic systems discussed above.
It is therefore an object of the present invention to
develop an associative memory system for color selection.
12
, ;- . ., : ;; , ~ , ;
- ., ~
, ~ . : . ~, . . . -:
It ~s a further object of the present inventlon to
develop an assoclative memory system with particular
application in the diesign of color i~terfaces for computer
systems .
It is another object of the present invention to
develop an associative memory system for implementation of
a color selection system.
These and other objects of the present invention will
be seen in more detail with reference to the detailed
description of the preferred embodiment and the
accompanying figures.
.~ . . , : , ~ .,; ~ .. ,
' ' ~ ' , . i '. : !,
~9~a~ THE INVENTION
An apparatus and method for assistlng ln the selection
of colox combinations is disclosed. The present invention
discloses an associative memory system having particular
application in the design of color inter~aces for computer
systems.
Utilizing an associative memory system, a color
interface may be designed based on the preferences supplied
by a user of the computer system. It has been found that
use of an associative memory system for the color selection
process yields advantages over known nrule based" color
se~ection systems.
The system of the present invention is implemented in
a computer system having a plurality of elements on each
interface screen. In paxticular, the preferred compu~er
system utilizes ten separate elements~ The disclosed color
selection process allows a user to designate colors for any
of the elements in the interface from a s lection of
available colors. After selecting preferred colors for the
desired elements, the user invokes the associative memory
system of the present invention to assist ln selection of a
color scheme for all ten elements. The associative mem~ry
14
", ..
- - . , . ~
: ~ : .,.. : ~
2 ~
system provides a color scheme having the user's designated
colors for the selected elements.
The present invention further discloses other methods
useful for the selection of colors for a computer interface
including a method for saving color combinations: a method
for reverting to a saved color set; a method for reviewing
and user "designed" color sets; and a method for setting
the elements of the color interface to a monochrome mode.
The present invention further discloses a neural
network implementation for the color selection system. The
neural network system may be generally described as a
hybrid of an auto-associative memory system~ and a
multi-layer neural net using back propagation for learning.
The network of the present invention is comprises a
plurality of input units; each input unit corresponding to
an element of the computer interface. The network further
comprises a plurality of output units: again, each output
unit corresponds to an element of the computer interface.
Each input unit is coupled w~th all output units, except
for the output unit corresponding to its own element,
through a sub-network. Each output unit is coupled to its
correspondin~ input unit through a recurrent link. Thus,
this main network may be thought of as the auto-associative
''
... .. . . .
~,. . . . . . .
: .
2 ~ 7 3 ?~
network portion of the hybrid network of the present
invention.
Each sub-network comprises a plurarity of input nodes
corresponding to each of the elements of a color
S representatlon system and a plurality of output nodes also
corresponding to the elements of a color representation
system. The input nodes are coupled to the output ~odes
through hidden nodes. In the preferred embodiment, RGB
color space i5 utilized; therefore, there are three input
nodes corresponding to red, green and blue, respecti~ely,
and three output nodes also corresponding to red, green and
blue. Each of the three input nodes is completely
connected to three hid~en nodes. Each of the three hidden
nodes is completely connected to the three output nodes.
Thus, each of the subnets may ~e thought of as a
multi-layer network portion of the hybrid network of the
;present invention.
;~The present invention discloses a new technique for
/relaxation of the network. Relaxation of the network of
:~,
~20 the present invention utilizes a technique termed back
,
activity relaxation. The back activity relaxati~n
technique of the present invention is a method for reducing
;
energy in the network by relaxing the activity (as ~pposed
to known methods of relaxing weights) of the network.
;,~ .
,.. .
- ,~
~ .:
:....
. .
, .
~..,.':
. ,~
BRI~F DESCRIPTION QF ~HE DR~IN~
Figure l(A) is an illustration of a prior art sin~le
layer network.
Figure 1(~) is an illustration of a prior art
multilayer network.
Figure l(C) is an illustration of a prior art network
showing a ~ack propagation technique.
Figure 2 is a illustration of a display screen of the
preferr~d embodiment of the present invention.
Figure 3 is a flow diagram illustrating a method for
accessing of the color selection process as may be utilized
by the present invention.
Figure 4 is a flow diagram illustrating a method for
color selection as may be utilized by the present
invention.
17
, ' .
,
,. .
,.
,
,':''
', ' . ! '
,. '. ' '''.
.' ' . , : ' , ~
2 ~ 7 3 ~
Figure S is a flow d~agram illustrating a method for
setting the colors of elements of a display to monochrome
values as may be utilized by the present invention.
Figure 6 is a flow diagram illustrating a method of
the present invention for reviewing designer color sets.
Figure 7 is a flow diagram illustrating a method of
the present ~nvention for reverting to a previously saved
color set.
Figure 8 is an illustration of a network as may be
utilized by the present invention.
Figure 9 is an illustration of a sub-net~ork as may be
utilized by the present in~ention~
'~'
.:~
, . .
: ,..
~ ` ~
7 ~ ~
A system for selection of colors is described. In the
following description, numerous specific details are set
forth such as specific computer int~r~aces, specific
colors, etc., in order to pro~ide a thorough understanding
of the present invention. It will be obvious, however, to
one skilled ln the art that the present invention may be
practiced without these specific details. In other
10 instances, well-known circuits, structures and techniques
have not been shown in detail in order not to unnecessarily
obscure the present invention.
~n~
The present in~ention descri~es a system for selection
of colors having particular application in the selection of
colors for a computer interface. The preferred embodiment
of the present invention is implemented on the Apple
Macintosh II computer, although it will be obvious to one
of ordinary skill in the art that the present invention
will have application in other computer systems, as well as
outside of the computer arena.
The present invention further describes a preferred
,,
,^- implementation of a color selection system compris~ng a
~ neural network which is described as a multilayered
^,'. 19
;, ~
.- .
,--'
.,
'
'` ' ,
.. . .
s~ 7 ~ ~
auto-associative network with recurrent links from output
back to input. Although the network is described with
respect to the specif.ic application of the color selection
problem, it will be obvious to one of ordinary skill ln the
art that the network may be useful in other applications.
Further, it will be obvious to one of ordinary skill in the
art that alternati~e embodiments may utilize other
associative memory systems to accomplish color selection.
i~CKGR~N~ ~LOR I~T~RE~ YEIÇE~
In modern day computer systems, the role of color is
playing an ever increasing part~ This is ~rue in the use
of computer systems for storing and presenting user
information, such as solor screens and slides for
presentations: in the use of high-resolution color graphics
for engineering and other uses; and in the design of
computer system interfaces.
j In particular, it has been found that use of color in
.j the design of computer interfaces can lead to aesthetically
~ pleasing interfaces as well as impro~ements in the ability
,~ 20 to convey information with the interfaces.
However, it is desirable to allow an end-user of a
computer system to alter the colors of the interface,
either because o~ the end-userls personal preferences or to
~ better convey information in a way useful to the particular
,~ 20
~,'
,
,
' ' ' . . ~ ' : ',, . ~.i , . . . .
,`., : : . "`
2 ~
user. one problem with allowing users to "customize" the
colors in thelr interface is many users may ha~e difficulty
in obtaining aesthetically pleasing results.
Therefore, a system is desired which allows users to
customize their interface while obtaining assistance with
color selection in order to obtain deslrable results. An
analogy may be drawn to use of an interior decorator who
offers assistance in the selection of a color scheme for a
home or office. Typically, the interior decorator obtains
information on the paxticular preferences of the client.
Using this information, the interior decorator attempts to
design a color combination which will be pleasing to the
client.
One observation of the present ~nvention is that the
color expert, such as the interior decorator, works within
a set of general principlPs of color design, however the
skills of the colox expert are more ar~istic than
technical. In other words, any set of rules is incomplete
and the artistic skills of the color expert are required to
.,
design a well-coordinated atmosphere. These artistic
skills are difficult, if not impossible, to reduce to a set
-~ of rules necessary, such as would ~e necess~ry to develop a
; color selection system based ~n a heuristic model.
~;,
~ 21
.
:
,. . . , ~ ,
.
.
.,
` ' : ' . ' :' .'
2 ~ 7 3 ~
~N AS~CL~r~ E~ORY ~ STEM FOR COLOR SEL~C~I~N
Therefore, the present invention proposes a color
selection system using an associative memory network to aid
in the color selection process. An associative memory
network does not require the establishment of a set of
rules; rather, the network is trained with a set of "good"
solutions and acquires the ability to act in a manner as if
it knew a set of rules.
This is similar to the manner a human color expert
typically acquires information and acts to develop new
color combinations, i.eO, the human color expert observes
and is able to construct color combinations which are
~ pleasing, but often is unable to state specific rules or
J reasons for selection or approval sf color combinations.
AS one alternative to the preferred embodiment of the
present in~ention, it may be desirable to implement a color
selection system which uses features of a rule-bas~d expert
j system in combination with the features of an associative
s memory system. For example, the system may first 3pply a
siet of rules and, after applying the rules utilize a neural
network for selection of color combinations conforming to
the set of rules. Alternatively, the system may first
.~ deri~e a color combination and, then, apply a set of rules
to ensure compliance of the proposed color combination with
22
:,.
.
.,
, "
~
: '~
,.,~
:`
.~, ., ", i ~ ,, ," " ,.
\
2 ~ 7 3 2
the established rules. By way of ex~mple, ~ rule may be
enforced whic~ requires a certain level of contrast between
text elements and background elements. Importantly, use of
an associative memory system, either alone or in
combination with a rule-ba~ed system, in the color
selection process offers slgnificant advantages over the .
prior art system using only rule-based color selection
systems.
The system of the present invention and the specific
application of this system to a computer interface will be
described in more detail ~elow.
APPLI~a~ION OF ~ L~R ~L~cTIoN SYSTEM TQ
TH~ MACI~TOS~ ~QMPUT~R INT~REa~E
The preferred embodiment of the present invention is
implemented on the Macintosh II computer for purposes of
aiding users in the selection of colors for elements in the
Macintosh computer interface (the Macintosh computer
interface is commonly referred to as the "desktopn~.
It is, of course, obvious that the present inventi~n
has application in interfaces of alternative computer
systems, as well as other alternative applications in
computer systems generally, such as selection of colors for
graphics designs developed on a color computer system. The
present invention has further use in other color selection
23
... .
, ., - :.
, ;
r~
applications. Interior decorating, make-up and wardrobe
color selectlon are examples of such alternatiYe uses.
~he Colox s~lQ~tion PrQ~ss of the Preferred ~mb~dim~n~
~he interface of the preferred embodiment is shown
S with reference to Figure 2. As will be familiar to many
users of the Macintosh computer, the Macintosh interface
comprises a number of elements such as the desktop 201,
menu bar background 2~2, menu ~ar text 203, menu text 204,
menu background 205, window text 206, window hilite 207,
~0 window bar 2D8, thumb 209 and scroll bar 210. In a color
interface each of these elements may be assigned a
different color. (Of course, a particular ~nterface may
also assign one or more of the elements the same color.
It should be noted that the Macintosh interface
. 15 includes other elements: the preferred embodiment of the
present invention allows assignment of colors to the
above-specified elements, alternative embodiments may allow
. assignments to the full set of elements or alternative
subsets.
In the syste~ of the preferred embodim~nt, a
particular screen may have more than one menu option in the
.~ menu bar text 203 and, consequently, more than one set of
menu text 204 and menu background area 205. In the
~mplementation of the preferred embodiment each of the
2~
',
....
~., .- - ;, . ....
2 ~
options on the menu bar text 203 would appear ln the same
color; each of sets of menu text 204 would appear in the
same color; and each of the menu background areas 205 would
a~pea~ in the same color~ Likewise, there may be multiple
windows having window text areas 206, window hilites 207,
window bars 208, scroll bars 210 and thumbs 209. These
areas of each window would similarly appear in the same
color from window to window in the preferred embodiment.
Figures 3 through 8 are flow diagrams illustrating the
use of the color selection process ~f the preferred
embodiment.
Figure 3 illustrates the method of the preferred
embodiment for accessing the color selection system.
First, a user determines it is desirable to change the
color interface of the computer, block 301. The user
"launches" the color selectlon application, block 302.
.~ Launching of an application is generally accomplished be
selecting an icon on the desktop corresponding to the
application to be launched, or started, and
~double-clicking~' OD the selected ic~n. ("Clicking" is a
term familiar to Macintosh users and is typically
associated with the use of a mouse as a pointing device.
To click ~n an area of a display, the mouse is used to move
the cursor to the desired area of the display and the
,
.:
,
.
'~ .
:. , . ~. . ,, , . ;, .
- , . ..
-;, , , ~ . . :
,, ' ' ": ' .
3 ~
button on the mouse is depressed and released.
Double~clicking invol~es depressing and releasing the
button on the mouse two times ln rapid succession.)
As will be familiar to a user of the Macintosh
computer, a menu bar is available along the top of the
Maclntosh screen. One alternative ~f to the preferred
embodiment for launching the color selection application
would allow the user to select a function from the menu bar
by pointing to ~he menu bar item with a pointing device,
such as a mouse. The user then selects the color selection
function from the selected menu by pulling down the
pointing deYice until the color selection function is
highlighted and then releases the pointing device (e.g. the
button on the mouse).
Responsive to the selection of the color selection
system, a screen such as the screen shown in Fig~re 2
appears on the display, block 303. The screen shDwq the
`~ standard Macintosh interface with label boxes poin~ing to
, each of the elements which may have heir colors changedO
2D The screen also shows a color palette 221 and a set of
function buttons 220 which may be selected by the user.
Each of the function buttons 221 will be descri~ed in more
detail belowO The color palette 221 comprises a 16x16
square of colors (a total of 256 colors).
26
-.
'~':' ; :" ;' ; ' .
2 ~ 3 2
After the color selection screen appears, the user may
select any of the functions available in the color
selection process, block 304.
Figure 4 illustrates the process of the preferred
5 embodiment for selection of colors. The user first
determines it is desired to change the colors of elements
on the interface, ~lock 401.
As disc~ssed previously~ the color selection process
of the preferred embodime~t is designed to assist user in
the color selection process by considering the user's
preferences and suggesting color combinations based on
those preferences. Therefore, the user first provides
information to the system on the user's preferences.
To accomplish providing this input, the user selects
an element, block 402. The selected element may be any one
of the ten elements on the Macintosh interface, i.e., the
desktop 201, menu bar background 20~, menu ~ar text 203,
menu text 204, menu background 205, window text 20S, window
hilite 207, window bar 208, thumb 209 and scroll bar 210.
The user selects the desired element by pointing to the
label asisociated with the element with the pointing device
and clicking on ~he label.
Aftex an element is selected, the user selects a color
fox the element from the color palette 221, block 403. In
27
. . ' ': . `:
.
; . -
2 ~ 7 3, ~,
~he preferred en~odiment, the user directs the pointing
device to the color palette 221 and uses the pointing
de~ice to select a color by point~ng to the desired color
and clicking on it. A~ternati~ely, embodiments of the
present in~ention may be developed ~hich allow for the
cursor to be automatical~y placed in the color palette area
221 a~ter the user selects an element.
After a color is selected, the element on the display
screen appears in that color. In this way, the user can
receive immediate feedback on how ~he selected colors
interact and make changes to the selected colors until the
desired combination is achieved.
After selecting the color for a first element, the
user may select colors for additional el~ments, branch 404.
In this way, the user may pro~ide as much, or as little,
information on the preferred color scheme as desired.
After the user has selected colors ~or desired
elements, branch 405, the user may select the "suggest
colors" function by pointing to and clicking on the
"Suggest Colors" button. The system, using the preferred
embodiments implementation of a neural network, then
suggests colors for each element which the user had not
previously selected, block 406.
28
.~
;~ "
. ~ ... .
, ~ ' ' , , .
, .- , ~ . .
.
~ ` 2~5 ~3
The "suggest colors" funct~on presents the user' 5
selected color scheme to the network as inputs. ~he
networX assume~ the selected elements are not to have their
colors changed. The nctwork produces, as output, ten
colors corresponding to each of the ten elements and
presents the color scheme to the user on the display
screen. The ten colors include the colors selected by the
user, as well as colors determined by the network to be
good l'matches" for the selected colors. The determination
of a good match is based on the networks previous training.
The tr~ining of the network of the preferred embodiment
will be described in more detail below.
After being presented with the networks suggested
colors, the user may select elements ~nd change the color
scheme, block 407. The user accomplishes thi~ by selecting
elements and colors as described in connection wlth blocks
402 and 403.
The user may lock and unlock color choices by
selecting the "lock colors" function, block 4D8. To select
- 20 the "lock colors" function, the user points to and clicks
on the "Lock Color " button. Responsive to selecting the
lock colors function, the user i5 presented with a set of
checkboxes corresponding to each of the element in the
interface. All elements which were previously ~xpli~itly
29
:
'
'`'
: ~ , . . . ; . , : . ~
7~
selected (by the process of blocks 402 and 403) will appear
as being checXed off. All elements whose colors were
selected by the neural network will not be checked off.
The user can check any element which is not chec~ed off and
can uncheck any element which is checked off by cllcking on
the appropriate box. (The checkboxes act as toggle
switches, toggling between being checked and being
unchec~ed.)
There are, of course, a number of alternative methods
of allowing the user to selectively lock and unlock color
choices. One alternative would utilize visual indicators
located either near or within each ~f the labels on the
desktop. An example of a visual indicator would be a
padlock icon. The visual indicator would then indica~e the
the user whether the color for an element is locked (for
example, a the padlock may be highlighted when the color is
locked). The user may toggle the element between being
locked and unlocked by clicking on the ~isual indicator.
The user may then again select the "suggest colors"
function and the system will ~uggest colors for all
unlocked elements, ~lock 409.
- After a des1rable color combination is achieved
through the process of selecting and locking colors and
; utilizing the ~suggest colors" function, the color
,~
~. .
. ,:
.
! ~ -
,' , ' , ' '" ~, , ' '
20 ~ ,r~732
combination may be saved by pointing to and c~lc~ing on the
"Save" button. Tha current color combination will then be
saved and use~ as the color combination for the Mac~tosh
inter~ace after exiting the color selection process.
The preferred embodiment of the present invention
includes several other function in the color selection
process which the user of the process may find useful.
Figure ~ illustrates a method of the preferred
embodiment utilized when the user wishes to set colors of
the interface to standard monochrome values, block 501.
The user first selects the "monochrome" function by
pointing to and clicking on the "Monochrome" button, block
502. Responsive to selecting the "monochrome" function,
the Macintosh interface is reset to standard monochrome
values, block 503.
Figure 6 illustrates the preferred embodiments method
for allowing a user who wishes to review the designer color
sets used for the training the network to examine each of
the exemplar color sets, blork 601. The user may select
the "designer color ~et" function, again, by pointing to
: and clicking on the "Designer Color Set" button. ~rhe
system then allows the user to cycle through each of the
color sets used for training the network, block 6G3. The
31
'~'
,~,
development and selection of color sets for train~ng the
netw~rk will be discussed in more detail below.
It is worth noting that the "save" function may be
utili~ed after reviewing any of the designer color sets to
save a particular designer color se~ ~o be used as the new
color set for the Macintosh interface. The "save" function
may also be used to save the monochrome color set as the
color set for the interface. Further, any of the designer
color sets or the monochrome color set may be modlfied by
lock~ng and unlocking elements and selecting new colors for
elements as described in connection with Figure 4. The
"suggest colors" function may then be used to develop a new
color set.
Referring now to Figure 7, after changing and
examining new color sets, the user may wish to revert to
the most recently saved color set, block 701. To
accomplish this, the user points to and clicks on the
"Revert" button, block 702. Responsi~e to clicking on the
"Revert" button, the system reverts image of interface to
the most recently saved color set.
It will be obvious to one of ordinary skill in the art
that the above-recited set of features is not meant to be
limiting on the scope of the present ~n~ention. Likewise,
other features may be added to an embodiment of the present
32
,~.
, . . :.: . .;
., . ,.. -
- . ~ : . . ,
~,~ ;' :' - ' '
2 ~
lnvention. Far example, ~he currently preferred embodiment
of the present invention provides the user with capabillty
of saving one color set. A feature may be added wh~ch
allows the user to save multiple color sets and to later
select from the saved color sets for use and modification.
A training functlon may be added which allows the user to
design new color sets to be used to further train the
network. Custom designer sets may be made available: for
example, designer sets may be designed and commerclally
sold by ~name" designers. The purchaser of such custom
designer sets would be able to design color sets ~hieh
would be influenc~d by the "nam~" designex's preferences.
The system of the preferred embodiment is designed to read,
from an external file, the values for strengths of
connections. This aspect o~ the preferred embodiment will
act to support the use of "name" designer's color sets.
THE ASS~CIATI~ ~MORY SY~I~M_OF TH~ PR~F~RRED ~M~O~I~E~
The design of the associative memory system f~r the
present invention involved evaluation of a number of
associative memory models. After a study of known
associative memory systems, it was determined that known
systems would not offer a satisfactory solution to the
problems posed by the desired color selection system.
, ' . . ~ !
, - ' , .
'
" .'~
2 ~ 2
The system of the present invention i~volves a large
number of possible combinations. First, there exists a
large number of possible colors from which to choose
particular oolors. The computer system of the preferred
embodiment computationally supports 248 different colors~
Second, any one of these colors may be applied to any one
of a number of elements. In the preferred e~bodiment,
t~ere are 10 elements wh~ch may be assigned colors.
Therefore, there are a total of 24B different combinations
from which the user could choose. Even in the preferred
embodiment, in which the user is limi~ed to 256 (28)
possible color choices, there are 280 possible
combinations.
A successful implementation of a color selection
system should be able to assist ~th color combinations
which the system has not been trained on. That is to say,
a color selecti~n system which merely recalled color
combinations from a memory would not be practical given the
large number of possible combinations. The trait of being
able to generalize from a set of exemplars in order to
provide a solution to a specific situation is termed
"generalization".
Generali~ation is known iQ auto-associative networks.
ln general, auto-associative networks, like many networks,
34
. i
. ~; , ', ;
,;
2~ ~5~6~J
are trained with a set of training examples (nex~mplarsn).
Auto-associative networks store memory patterns (based on
exemplars) in multiple point attxactors and are able to
general~ze by clustering similar patterns. (Instead of
creating different patterns for each pattern of a class,
the network creates a single attractor for the "average" vr
"prototype" of the patterns.)
It is desired to develop a system which allows a user
to select a new, unknown color ~nd for the network to
generalize its knowledge of the numerical relationship
between "matching" colors to determ~ne proper usage of the
new color. An implementation of a n twork with this
ca~abllity requires the networ~ be trained not only to
create multiple point attractors, but als~ t~ trai~ the
"landscape" around the attractor basins. This type of
generalization is known in systems which may be described
as feed-f~rward networks using a single basin. Howe~er,
this type of generalization is not Xnown in
auto-associative networks.
~nother difficulty arises from the fact that the
colors used in the present invention are represented by a
set of continuous values; in the preferred embodiment,
colors are represented by RGB (red-green-blue) values.
Each RGB value is represented by three 16-bit integers, one
.; ,
.
. : , , ,~ - .
, . . . ,. . .: . - .
:. . : ,: . .: . ,
. .: . : . :: .
, :. : . ." . ,. . ::
- - -
.. : :
.. . . .
2 ~ 3 2
for red, one for green and one for blue. Typical
auto-associative models store vnly binary patterns. An
implementation which used binary encoded representation of
t~e continuous RGB values would be prohibiti~e because of
the numher of input and output units required.
~Scription of the NetwQrk of the Prefer~ed Embodiment
Therefore, a network is disclosed which may be
described as a two-layer back propagation network with
recurrent links added to model an asymmetric continuous
auto-associati~e memory.
Figure 8 illustrates the basic configuration of the
color selection network of the present invention. A set of
ten input nodes and ten output nodes are used to represent
colors of the ten interface elements of the Macintosh
display.
Figure ~ illustrates four of the input nodes, desktop
801, me~nu text 802, scroll bar 803 and thumb ao4, as well
as four of the output nodes, desktop 805, menu text 806,
scroll bar 807 and thumb 808. It will be obvious to one of
ordinary skill in the art how the illustration of Figure 8
showing four nodes extends to the implementation of the
preferred embodiment haYing ten nodes.
Each of the input nodes, (e.g., nodes ~01, 802, 803
and 804) are connected to ~ach of the output nodes (e.g.,
36
- , . . . .
- ': 'i ,-, : , ,
, .
,: i , , , . . , . . ~ :
. . ~,
2 ~ 7 3 ~
nodes 805, 806, 807 and 808); except each input node is not
connected to the output node corresponding to itself. By
way of example, input node desktop 801 is connec~ed with
output nodes menu text B06, scroll bar 807 and thumb B08.
lnput node desktop B01 is not connected with output node
desktop 805. Therefore, in the preferred embodiment, each
input node is connected to nine output nodes. The
resulting network can be thought of as a completed
connected ten-node Hopfield network, except the network
includes asymmetric interconnections. tIt is worth noting
the lack of connection from an input unit to its
corresponding output unit ensures the network does not
simply learn to copy an input onto an output directly;
rather, the network mu~t learn to indirectly encode the
mapping through the other interface elements.)
The network also includes recurrent connectians from
each output node to its corresponding input node. For
example, output node desktop 805 is connected with input
node desktop 801. This connection allows the state of the
output node to be copied back to the input node during the
relaxation process (the relaxation process will be
described in more detail below).
Each of the input-output node sets shown in Figure 8
actually corresponds to a sub-network. Figure 9 is
37
'; . . : :," ': " ' ' ' . :,
. . . . . .. .
2 ~ 3 2
illustrative of the sub-network connection between input
node menu text 802 of Figure 8 (shown as input node 910 of
Figure 9) and output node scroll bar 807 (shown as output
~ode ~12 of F~gure 9). This connection comprises three
input units 910 ~i.e., red 901~ green 902 and blue 903~,
three output units 912 (i.e., red 904, green 905 and blue
906), three hidden units 911 and eighteen weighted links,
nine completely connecting the three input units 910 to the
hidden units 911 and nine completely connecting the hidden
units 911 to the output units 912.
The network of the preferred e~bodiment, therefore,
comprises a total of 30 input units and 30 output units.
As has been described, each of the 30 input units and 30
output units are grouped in sets of three units.
Therefore, there are 10 groups of input units corresponding
to the 10 screen elements. Each of the 10 groups of input
elements are indirectly connected to 9 of the groups of
output units, for a total of 90 connection of groups of
input units to groups of output units. ~he groups of input
units are connected t~ output units through groups of three
hidden units. Therefore, the network o the preferred
embodiment further comprises 270 (90*3~ hidden units.
Specifically, each of the three input units in each input
group is connected with each of the three hidden un~ts
38
: - , - ;, , ~.
2 ~ 2
assoc~ated with that group. Each of the three hidden units
is connected with each of the three output unlts associated
with the h~dden units. In total, the system compr~ses lh20
forward links or connections (90 connections from lnput
groups to output groups times; each of these connections
comprising 9 connections from input group to hidden group
plus 9 connections from hidden group to output group~ i.e,
90*~9+9) = 90*1~ = 1620).
As discussed previously, the system further compri~es -
10 10 recurrent connections from output color nodes to their
correspondlng lnput color nodes. Each of the recurrent
co~nections actually connects the red node of the output to
the red node of the corresponding input, the green node of
the output to the green node of the corresponding input and
the blue node of the output to the blue node of the
corresponding input, i.e., a total of 3 times lO or 30
c~nnections.
It is important to note that, although the preferred
embodiment of the present invention is implemented using an
RGB representation of color, alternative embodiment may be
implemented using other representation systems for colors.
For example, common systems include ~SV (hue, saturation
and value) and CMY ~cyan, magen~a and yellow). The RGB
coding ~ystem was chosen for the preferred embodiment
39
., . , ..... . ~
, . ,. .. . ... . ~ " ,
c~ ~
~ecause ~GB encodlng ls the stan~ard for the Macintosh II
hardware and software. lt ~s also worth noting that the
network o the present invention has application outside of
the color selection field; a network of this type may be
useful in many applications where an "expert" in the fiel~
has difficulty formulating precise rules but readily
renders opinions on what i5 ~good".
T~ainina of the N~twork
The network of the present invention i5i trained using
color sets designed by "color experts". In particular/ the
system of the preferred embodiment was trained with 36
color sets designed by 5 different color experts. Back ,
propagation was used to train the network with the color
sets provided by the "experts". The training patterns were
used as input patterns and as target patterns at the output
layer.
Of course, alternati~e color sets may be used to train
the network. An example of alternative color sets was
described above in connection with the discussion of custom
designer sets designed by "name" designers.
Bela~ation of the Netwo~k
The network of the preferred embodiment utilizes a
method termed "hack activity relaxation" for relaxation of
the network. (Relaxation may he thought of as the process
:i.;
'
.. , . .: .
~ . .
::, . . . .
, ~ . . , :
~ '
:,,. ~ .
2 ~ 3 ~
of allowing the network to search for a set of well-matched
colors, l.e, a solutio~ with the lowest energy level. The
term energy is often used in neural network systems to
describe the level of error between the desired or target
5 output and the actual output . )
The back activity relaxation algorithm adjusts the
weight of linkages, searching for a set of weights which
generates the minimum mean squared-error over the training
set where the error is defined as the difference between
the output value and target value. The underlying theory
is that the least squared-error analysis produces a state
where the color of each element best matches the colors of
the other elements. Essentially, the network may be viewed
as ten separate sub-networks, one for each element; each of
the sub-networks belng trained to learn the relationship
between the color of its element and the color of the nine
other elements. Through learning the numerical
relationship of these colors, the network can res~ond to
colors which did not appear in the train~ng sets.
The prior art includes a back acti~ity propagation
technique which is characteri~ed as a learning algor~thm
for reducing the energy of a network by relaxing the
weights of the connections between nodes. Typically, a
back propagation algorithm takes the partial derivative of
~1
., "~ :
-:, .
. ,. ,
': :
.. ..
3 ~
the glob~l energy of the network against the local weight
of a connection.
The present lnvention proposes a back activity
relaxation technique for taking the partial derivativ~ of
energy with respect to activity of a connection, instead of
weight.
The back activity propagation algor.ithm is described
in more detail with reference to Appendix I.
Thus, a network is described which is capable of
storin~ color patterns which are coded with continuous
values and which is capable of learning relationships
between colors of interface elements. The network
comprises a recurrent network with back propagation for
learning and back activity rel xation for deriving
solutions. The network of the preferred embodiment may be
use~ to model an asymmetric auto-associative memory capable
of storing continuous valueq. Capability for modeling such
an auto-associative memory is one novel aspect of the
present invention.
In the present invention, one important aspect is the
fact that any given input unit does not connect directly,
or through hidden units, to its corresponding output unit.
It is theorized that allowing connection of an input uni~
~ ,
, .. . ~., ,, :
: . . ~ .. .
: . , . :. . ' :
: ~ ,
7 3 ~
to its corresponding output unit would lead to the
networking learning to copy the input on~o the output.
~hus, an apparatus and method for color selection is
described. Although the present invention has been
described with specific reference to a number of details of
the preferred embodiment, it will be obvious that a number
of modifications and variations may be employed without
departure from the scope and spirit of the present
invention. Accordingly, all such variations and
modifications are included within the intended scope of the
inventlon as defined by the followlng clalms.
;, :
~ ~3
,~
.
.:
