Note: Descriptions are shown in the official language in which they were submitted.
WO 2022/183138
PCT/US2022/020994
AUTOMATED CLASSIFICATION OF EMOTIO-COGNITON
CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of priority to non-provisional Application
No.
17/589,512, filed January 31, 2022, claims the benefit of priority to
provisional Application No.
63/163,618 filed March 19, 2021, claims the benefit of priority to provisional
Application No.
63/163,621 filed March 19, 2021, claims the benefit of priority to provisional
Application No.
63/162,987 filed March 18, 2021, and claims the benefit of priority to
provisional Application
no. 63/143,730 filed January 29, 2021, the entire contents of which are
incorporated herein by
reference.
BACKGROUND
FIELD OF THE INVENTION
[1] The present disclosure relates generally to monitoring affect, and in
particular classifying
and tracking intensity of emotio-cognition in natural language content.
DESCRIPTION OF THE RELATED ART
[2] Affective computing is the study and development of systems and devices
that can
recognize, interpret, process, and simulate human affects. Affective computing
is a
multidisciplinary field that brings together linguistics, statistics, human
factors and computer
science. One aspect of affective computing is to enable bots and other
computer applications to
respond intelligently to natural human emotional feedback. In the case of
text, affective
computing includes emotion detection from text. More often, a form of emotion
detection known
1
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
as sentiment analysis is used to classify text as positive, negative, or
neutral. Just about every
major company that develops computer software and hardware, as well as
university research,
and several startups have development projects that include some form of tools
for sentiment
analysis. The popularity of sentiment analysis stems from the need to better
understand sentiment
in reactions to news media and various customer comments that are widespread
in social media,
customer product comments, and interactions with chat bots.
[3] In practice, sentiment analysis is the use of natural language
processing, text analysis,
computational linguistics, and biometrics to systematically identify, extract,
quantify, and study
affective emotio-cognition states and subjective information. Sentiment
analysis has been
accelerated in part due to the availability of large data sets of human
dialog, which are obtained
from such sources as various social media platforms, recorded conversations,
and other outlets
for textual expression However, sentiment analysis must deal with the evolving
nature of natural
language. For example, sentiment analysis must deal with subtle distinctions
or variations in
meaning between words, or even whole phrases. Some phrases may appear to
indicate the same
thought, but may indicate a difference in sentiment. Sentiment analysis must
deal with words or
phrases that may have different meanings depending on the context.
[4] Despite significant recent advances in technologies used for natural
language processing,
sentiment analysis suffers from being tied to training sets, which typically
have been manually
classified, and is thereby subjective. In particular, annotation of big
training datasets of text is
performed manually. Training sets from manual classification methods tend to
be slow, poor-
performing, and expensive. Additionally, financially competitive methods of
obtaining raters,
primarily Mechanical Turk, suffer from non-native speakers incentivized to
rush through
classification tasks, resulting in low-quality, conflicting results, with
attempts at prevention of
2
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
these effects limited to cumbersome ID-scanning and unreliable [P-address
filtering. Subtle
emotion detection is difficult and error prone.
[5] Moreover, training sets used for machine learning do not afford for
creativity in
language. Human beings have an apparent infinite ability to generate novel
sentences through
speech and writings, never before written or spoken. Indeed, a necessary
property of language is
precisely this allowance for such creativity, which current state of the art
systems are unable to
effectively accommodate
[6] State-of-the-art technology machine learning models for natural
language processing
(BERT (Bidirectional Encoder Representations from Transformers), GP*172
(Generative Pre-
trained Transformer 2), GPT3), which are seemingly robust tools that can hold
so much in
memory and effectively look bi-directionally, are no match for an evolving
natural language.
Statistical models rely on strong statistical, often probabilistic components.
Supervised machine
learning models predict things they have seen before or relationships they've
seen reoccur. There
are countless phrases and sentences to be made in the future that cannot be
encapsulated by
sentences that have come before, including elements like portmanteaus, slang,
jargon, metaphor
or invented words.
[7] In addition, there is still a semantic deficit in sentiment analysis;
this is in part due to a
lack of effective methods for measuring the intensity of emotions, the use of
culled and
unrepresentative datasets for training purposes. Additionally, overreliance on
lexicon-centric or
token-based solutions prevents such solutions from ultimately obtaining
staying power, as
linguists and engineers contend with the dynamic nature of language, including
semantic shifts,
obsolescence, and various other linguistic changes over time.
3
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[8] As such, sentiment analysis suffers from several major issues; namely,
lack of gold-
standard datasets with objectively rated or labeled data for training,
limitations of n-gram based
solutions, lack of the ability to determine bonafide intensity, difficulty in
parsing hashtags, and
lack of rigorous validation for results. Additionally, while like-scale
affective ratings are
subjectively applied, no sound means exists within the industry or academia to
validate
classification results.
[9] While the occasional attempt to quantify pragmatic factors
(predominantly via
brainstormed enumeration, and occasionally using n-gram selection with high
token:type ratios)
surfaces in the sociolinguistic literature, it is the study of
sociolinguistics that largely focuses on
fINARI studies, social or real-world experiments, intuitive analyses, and
close examination and
comparison of intuitive, pragmatic examples. Specific, but limited,
syntactically-driven theories
exist within the field, largely in individual constructions such as Andrews
construction
(adjectival and adverbial phrases with the construction "X knows," e.g., "God
[only] knows,"
"Who knows") or somewhat more comprehensively, the semantically rich, complex
but verb-
centered seminal analysis of construction grammar, focused on specific
grammatical
constructions for object relationships, known as argument structure, dealing
by nature with
arguments (objects) of verbs.
[10] Preferably, to be a construction, a piece of language has to be
permutable and modular.
By nature, this creates a focus not just on grammatical rules, but on common
usage.
Sociolinguistics have focused on pragmatics, the science of usage in the real
world, analyzing
specific word usage in a magnified view. However, analysis of constructions
has been focused
on argument structure, and verb-focused analysis of form-meaning pairings.
There is a need for
4
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
more efficient, robust and accurate detection of cognition, emotion, and
emotio-cognition, which
bridges cognitive science, social-linguistics and semantics.
[11] The foregoing "Background" description is for the purpose of generally
presenting the
context of the disclosure. Work of the inventors, to the extent it is
described in this background
section, as well as aspects of the description which may not otherwise qualify
as prior art at the
time of filing, are neither expressly or impliedly admitted as prior art
against the present
invention.
SUMMARY
[00121 According to an embodiment of the present disclosure, an aspect is a
method for
automatically augmenting natural language content with emotio-cognition by
processing
circuitry, the method can include receiving, via an input device, the natural
language content as a
textual input; searching, by the processing circuitry, for matches between a
plurality of linguistic
rules for a given emotio-cognition and components of the textual input,
wherein instances of the
linguistic rules have at least one human dimension; activating, by the
processing circuitry, the
matched linguistic rules, and evaluating the at least one human dimension of
the activated
matched linguistic rules; scoring, by the processing circuitiy, each human
dimension to obtain a
prototypical profile of dimension scores for the given emotio-cognition;
aggregating, by the
processing circuitry, the dimensions in the obtained profile of dimension
scores to obtain an
intensity indication for the given emotio-cognition; and displaying, by a
display, the natural
language content augmented in a manner that relates the matched linguistic
rules to the given
emotio-cognition and signals the respective intensity indication of the given
emotio-cognition.
[0013] Further, according to an embodiment of the present disclosure, a
further aspect is an
electronic reader, that can include a touchscreen display; processing
circuitry; and a memory,
wherein the touchscreen display is configured to display text of an electronic
book; the
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
processing circuitry is configured to scan and tag the text using rules that
upon being triggered,
detect emotio-cognitive states, and determines intensity with which the emotio-
cognitive states
have occurred; the processing circuitry is configured to generate and display
one or more
sidebars for listing dynamics and emotio-cognition-intensity information based
on detected
components of the displayed text; the touchscreen, when touched at a position
in the display, is
configured to select a dynamic or emotio-cognition-intensity; and the
processing circuitry is
further configured to generate and display color-coded highlighting that
designates an occurrence
of the selected dynamic or emotio-cognition-intensity.
[0014] Further, according to an embodiment of the present disclosure, a
further aspect is a
system for mitigating a psychological disorder, that can include a mobile
device having
processing circuitry and memory; and a peripheral device having a
communications device and
one or more actuators, wherein the memory of the mobile device stores program
instructions,
which when executed by the processing circuitry of the mobile device, cause
the mobile device
to perform a method including: receiving, via an input device, natural
language content as a
textual input; searching, by the processing circuitry, for matches between a
plurality of linguistic
rules for a given emotio-cognition and components of the textual input,
wherein instances of the
linguistic rules have at least one human dimension; detecting, by the
processing circuitry, the
matched linguistic rules to obtain an intensity indication for the given
emotio- cognition; and
when the intensity indication for the given emotio-cognition reaches a
negative emotio-
cognitional intensity that exceeds a first threshold, transmitting a first
activation signal that
identifies the emotio-cognitional intensity; and the peripheral device is
configured to receive, via
the communications device, the transmitted first activation signal; and
activate the one or more
actuators to create a sensory distraction to mitigate the psychological
disorder.
6
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
The foregoing general description of the illustrative implementations and the
following detailed
description thereof are merely exemplary aspects of the teachings of this
disclosure, and are not
restrictive.
BRIEF DESCRIPTION OF THE DRAWINGS
[15] The accompanying drawings, which are incorporated in and constitute a
part of the
specification, illustrate one or more embodiments and, together with the
description, explain
these embodiments. The accompanying drawings have not necessarily been drawn
to scale. Any
values or dimensions illustrated in the accompanying graphs and figures are
for illustration
purposes only and may or may not represent actual or preferred values or
dimensions. Where
applicable, some or all features may not be illustrated to assist in the
description of underlying
features.
[16] The characteristics and advantages of exemplary embodiments are set out
in more detail
in the following description, made with reference to the accompanying
drawings. In the
drawings:
[17] FIG. 1 is a block diagram of a system for automated classification of
belief, opinion,
sentiment, and emotion in accordance with exemplary aspects of the disclosure;
[18] FIG. 2 is a block diagram of a computer system in accordance with an
exemplary aspect
of the disclosure,
[19] FIG. 3 is a system block diagram for automated classification of emotion
in accordance
with an exemplary aspect of the disclosure;
[20] FIG. 4 is a diagram of a training architecture in accordance with an
exemplary aspect of
the disclosure;
7
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[21] FIG. 5 is a diagram of adaptive operation of the emotion classification
system in
accordance with an exemplary aspect of the disclosure;
[22] FIG. 6 is a diagram showing types of linguistic rules in accordance with
exemplary
aspects of the disclosure;
[23] FIG. 7 is a bottom-up stack for layers of the rules engine in accordance
with exemplary
aspects of the disclosure;
[24] FIG. 8 is a flowchart of a method of operation of a computer system in
accordance with
an exemplary aspect of the disclosure;
[25] FIG. 9 is a flowchart for steps for evaluating using linguistic rules in
accordance with an
exemplary aspect of the disclosure;
[26] FIG. 10 is a flowchart for detecting rules in accordance with an
exemplary aspect of the
disclosure;
[27] FIG. 11 is a flowchart for scoring in accordance with an exemplary aspect
of the
disclosure;
[28] FIG. 12 is a flowchart for detecting rules in accordance with an
exemplary aspect of the
disclosure;
[29] FIG. 13 is a flowchart for determining numeric value for a token
proximity rule in
accordance with an exemplary aspect of the disclosure;
[30] FIG. 14 is a flowchart for classifying in accordance with an exemplary
aspect of the
disclosure;
[31] FIG. 15 is a flowchart for hybrid multi-model learning in accordance with
an exemplary
aspect of the disclosure;
8
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[32] FIG. 16 illustrates an electronic reader in accordance with an exemplary
aspect of the
disclosure;
[33] FIG. 17 is a flowchart for operation of an electronic reader in
accordance with an
exemplary aspect of the disclosure.
[34] FIG. 18 is a flow diagram of the system for a Multimedia Audio :Book or
Visio-Spatial
Data Sentiment Classifier in accordance with an exemplary aspect of the
disclosure.
[35] FIG. 19 is a block diagram of a multi-media rules engine in accordance
with an
exemplary aspect of the disclosure;
[36] FIG. 20 is a flowchart for a rules discovery engine based on HUNCHes in
accordance
with an exemplary aspect of the disclosure;
[37] FIGs. 21A, 21B is a flowchart for rule discovery in audio media in
accordance with an
exemplary aspect of the disclosure;
[38] FIG. 22 is a graph of a speech signal pattern;
[39] FIG. 23 is a flowchart for a method of real time emotion classification
in a stream of
video/audio in accordance with an exemplary aspect of the disclosure;
[40] FIG. 24 illustrates a display device in accordance with an exemplary
aspect of the
disclosure;
[41] FIG. 25 is a system diagram of adaptive operation of the emotion
classification system in
accordance with an exemplary aspect of the disclosure;
[42] F:1:Gs. MA, 26B, 26C is a schematic diagram of an electronic bracelet in
accordance with
an exemplary aspect of the disclosure;
[43] FIG. 27 is a circuit diagram for an electronic bracelet in accordance
with an exemplary
aspect of the disclosure;
9
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[44] FIG. 28 is a system diagram of user interface features for
sociolinguistic data in
accordance with an exemplary aspect of the disclosure; and
[45] FIG. 29 is a flow diagram of a sociolinguistic engine pipeline in
accordance with an
exemplary aspect of the disclosure.
D:ETAILED DESCRIPTION
[46] The description set forth below in connection with the appended drawings
is intended as
a description of various embodiments of the disclosed subject matter and is
not necessarily
intended to represent the only embodiment(s). In certain instances, the
description includes
specific details for the purpose of providing an understanding of the
disclosed embodiment(s).
However, it will be apparent to those skilled in the art that the disclosed
embodiment(s) may be
practiced without those specific details. In some instances, well-known
structures and
components may be shown in block diagram form in order to avoid obscuring the
concepts of the
disclosed subject matter.
[47] As used herein any reference to "one embodiment" or "some embodiments" or
"an
embodiment" means that a particular element, feature, structure, or
characteristic described in
connection with the embodiment is included in at least one embodiment. The
appearances of the
phrase "in one embodiment" in various places in the specification are not
necessarily all referring
to the same embodiment. Conditional language used herein, such as, among
others, "can,"
"could," "might," "may," "e.g.," and the like, unless specifically stated
otherwise, or otherwise
understood within the context as used, is generally intended to convey that
certain embodiments
include, while other embodiments do not include, certain features, elements
and/or steps. In
addition, the articles "a" and "an" as used in this application and the
appended claims are to be
construed to mean "one or more" or "at least one" unless specified otherwise.
1.0
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[48] Referring now to the drawings, wherein like reference numerals designate
identical or
corresponding parts throughout several views, the following description
relates to a system and
method for automated classification of belief, opinion, sentiment, and
emotion. The method
optimally includes a linguistic rules engine as an input layer to a
probability layer, and a layer for
determining intensity.
[49] As mentioned above, language is perpetually novel, and even robust tools
that can hold
so much in memory and effectively look hi-directionally (i .e , BERT) are no
match for an
evolving natural language. There are countless sentences to be made in the
future that cannot be
encapsulated by sentences that have come before, including elements like
portmanteaus, slang,
jargon, metaphor or invented words. Focusing on permutable constructions and
mining them for
extractable belief, opinion, sentiment, and emotional dimensions allows for
more efficient,
robust and accurate detection
[50] Also, in natural language processing, probability is a poor indicator of
verbal intensity,
because frequency does not necessarily equal intensity. As such, in disclosed
embodiments,
probability is used as an indication that an emotion is present in a specified
portion of a natural
language input. Intensity of the emotion is separately determined.
[51] Various techniques may be used for analyzing text in an effort to
understand the
sentiment, emotion, opinion, or belief that may be expressed or implied by the
text. Sentiment
may be defined as an attitude, thought, or judgment prompted by feeling. Kin
to sentiment is
emotion, which may be defined as a strong feeling, such as love, anger, joy,
hate, or fear.
Emotion may include a conscious mental reaction subjectively experienced as
strong feeling and
accompanied by physiological and behavioral changes in the body. Opinion may
be defined as a
belief, judgment, or way of thinking about something. Belief may be something
that is accepted,
11
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
considered to be true, or held as an opinion. For purposes of this disclosure,
the term "emotio-
cognition" will be used as a term to portray each of sentiment, opinion and
belief, emotion,
according to its ordinary meaning, judgment, as well as feelings (affect /
desires, emotional or
sexual bonding), interpersonal / social forces (affinity, community, bonds,
influence), cognitive
elements (thoughts, opinions, beliefs, stances) and the sentimental space in
between (aspirations,
values, motivations, regrets).
[52] Disclosed embodiments utilize lexical rules having semantic-syntactic
constructions, pre-
scored across numerous dimensions, and containing building blocks for semantic
sentimental
logical operations. Rule-searching measures make tagging easier, faster, and
more empirical, and
thus reduce the need for GPLT-fine-tuning (i.e., fine tuning a pre-trained
transformer, such as
BERT, GPT2, GPT3), agreement calculations, or high RAM operations. Rule
suggestion via
dimensional pattern spotting, sentimental occurrence-tracking, and sequence-
spotting can also
save a tremendous amount of resources tagging, retraining, or using resources
to add epochs for
accuracy increase or multiple models to solve ambiguity issues. Logical
derivation of other
sentimental phenomena via aggregation / processing of dimensions allows for
simple search-and-
count processing per-line instead of heavy calculation, making way for novel
calculations
(detection of mental health symptoms) without need for a new or specialized
training set, new
task or added layer.
[53] FIG. 1 is a block diagram of a system for automated classification of
emotio-cognition in
accordance with exemplary aspects of the disclosure. The system 100 includes a
text input 102
which receives text from various sources, including a continuous data stream,
social media
dialog, documents, and whole books. The text received from the text input 102
undergoes a data
cleaning and data normalization process 104. Various tools are available for
cleaning and text
12
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
normalization, and generally involve stripping text data of unwanted
characters and
standardizing words. The choices of which characters are unwanted are user
dependent. For
example, in some cases punctuation marks may be unwanted. in some embodiments,
specific
punctuation marks are not removed, but instead may be used in later
processing. Specific
punctuation marks may include commas, quotation marks, and exclamation marks.
Punctuation
marks that may be removed may include the at (@) symbol, hashtag (a), dollar
($), percent (%),
carrot (A), ampersand (&), and asterisk (*). In some embodiments, markups such
as HTML tags
may be removed. In some embodiments, emoticons may be left intact. In
addition, text may be
converted to lower case. The cleaned and normalized data is then pre-processed
106, including
tokenization, part-of-speech tagging, stemming, and lemmatization.
Tokenization splits text into
individual elements (e.g., split cleaned text at their whitespaces). Part-of-
speech tagging attaches
labels to word tokens to identify their part of speech. Stemming is a process
of transforming a
word into its root form. Lemmatization is a process of obtaining the canonical
forms of
individual words. Another task may be to remove stop words. The pre-processed
data may be
formatted for an unsupervised learning process 108, a rule-based system 110,
and a supervised
learning process 112. A typical method for formatting pre-processed words is
to use Google's
word2vec algorithm.
[54] The unsupervised learning process 108 may classify the data, including
classification of
emotio-cogntition. The unsupervised learning process 108 does not require
labeled data, but
instead may cluster pre-processed data into classes. The pre-processed data
input and resulting
classification may be used for feature/rule suggestion 114. Suggested rules
and feature selection
120 may be performed to generate future linguistic rules.
13
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[55] The rule-based system 110 includes predetermined linguistic rules. The
predetermined
linguistic rules may be organized by categories of emotio-cognition.
[56] The supervised learning process 112 requires labeled data. The labeling
of emotio-
cognition may be performed manually. The supervised learning process 112 may
be used for
machine classification of emotio-cognition 118. Errors in classification may
be
adjusted/corrected 122 in order to improve later classifications. The
supervised learning process
112 generates neural models to perform their own classification, which assigns
a probability. The
neural models also are trained on the rules themselves. They locate similar
cooccurrence vectors,
similar POS-patterns, and similar n-grams, and suggest these as potential
Rules / Suggested
Features.
[57] Tabulation and scoring 116 may be applied to the results of the
unsupervised learning
process 108 and supervised learning process 112. Both the unsupervised
learning process 108
and the supervised learning process 112 may output a probability for each
class (e.g., using the
softmax function).
[58] In one implementation, the functions and processes of the system 100 may
be
implemented by a computer 226. Next, a hardware description of the computer
226 according to
exemplary embodiments is described with reference to FIG. 2. In FIG. 2, the
computer 226
includes a CPU 200 which performs the processes described herein. The process
data and
instructions may be stored in memory 202. These processes and instructions may
also be stored
on a storage medium disk 204 such as a hard drive (HUD) or portable storage
medium or may be
stored remotely. Further, the claimed advancements are not limited by the form
of the computer-
readable media on which the instructions of the inventive process are stored.
For example, the
instructions may be stored on CDs, DVDs, in FLASH memory, RAM, ROM, PROM,
EPROM,
14
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
EEPROM:, hard disk or any other information processing device with which the
computer 226
communicates, such as a server or computer.
[59] Further, the claimed advancements may be provided as a utility
application, background
daemon, or component of an operating system, or combination thereof, executing
in conjunction
with CPU 200 and an operating system such as Microsoft Windows , UNIX ,
Oracle
Solaris, LINUX , Apple macOSO and other systems known to those skilled in the
art.
[60] In order to achieve the computer 226, the hardware elements may be
realized by various
circuitry elements, known to those skilled in the art. For example, CPU 200
may be a Xenon
or Core processor from Intel Corporation of America or an Opteron processor
from AMD of
America, or may be other processor types that would be recognized by one of
ordinary skill in
the art. Alternatively, the CPU 200 may be implemented on an FPGA, ASIC, PLD
or using
discrete logic circuits, as one of ordinary skill in the art would recognize.
Further, CPU 200 may
be implemented as multiple processors cooperatively working in parallel to
perform the
instructions of the inventive processes described above.
[61] The computer 226 in FIG. 2 also includes a network controller 206, such
as an Intel
Ethernet PRO network interface card from Intel Corporation of America, for
interfacing with
network 224. As can be appreciated, the network 224 can be a public network,
such as the
Internet, or a private network such as LAN or WAN network, or any combination
thereof and
can also include PSTN or ISDN sub-networks. The network 224 can also be wired,
such as an
Ethernet network, or can be wireless such as a cellular network including
E:DGE, 3G and 4G
wireless cellular systems. The wireless network can also be WiFie, Bluetooth ,
or any other
wireless form of communication that is known.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[62] The computer 226 further includes a display controller 208, such as a
NVIDIA
GeForce GTX or Quadro graphics adaptor from NVIDIA Corporation of America
for
interfacing with display 210, such as a Hewlett Packard HPL2445w LCD monitor.
A general
purpose IJO interface 212 interfaces with a keyboard and/or mouse 214 as well
as an optional
touch screen panel 216, or haptic device on or separate from display 210.
General purpose I/0
interface also connects to a variety of peripherals 218 including printers and
scanners, such as an
OfficeJet or DeskJet from Hewlett Packard . The I/O Interface 212 may also
connect to a
microphone for voice input and speakers and/or headphones for sound output.
The microphone
and/or headphones may be connected to the I/0 Interface 212 by way of an input
port, including
USB, HDMI, or other peripheral input connection.
[63] The general purpose storage controller 220 connects the storage medium
disk 204 with
communication bus 222, which may be an ISA, EISA, VESA, PCI, or similar, for
interconnecting all of the components of the computer 226. A description of
the general features
and functionality of the display 210, keyboard and/or mouse 214, as well as
the display
controller 208, storage controller 220, network controller 206, and general
purpose I/0 interface
212 is omitted herein for brevity as these features are known.
[64] FIG. 3 is a system block diagram for automated classification of emotion
in accordance
with an exemplary aspect of the disclosure. The system 300 includes a multi-
media classification
engine 312. Multi-media can include video, audio, and text. The multi-media
can include
scripted or subtitled multimedia, such as audio books, visio-spatial
multimedia like movies, TV
shows, augmented reality, or virtual reality, which are transcribed and
scanned into the system
300. Textual and transcribed media is received as input text 302. The input
text 302 is processed
in the pre-processing engine 304 in order to transform the text data into a
form necessary for
16
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
matching with rules 306, 308, 310, or for input to machine learning models
320. In some
embodiments, the pre-processed text data is run through the Natural Language
Rules (306).
When rules match textual or transcribed media, the rule is said to be fired.
Visio-spatial rules 310
can include rules for visual cues. Audio-speech rules 308 can include rules
for speech signals.
[65] The Emotio-Cognitive Sensor (314) processes input on a sentence-,
paragraph-, passage-,
scene-, or chapter-level, and classifies each with a given emotion, cognition,
sentiment, state- or
dynamic- or trait-based tag. When a rule is triggered, an emotion is detected.
Also, linguistic
rules have ratings based on Dimensions. In some embodiments, dimension values
for a rule are
stored as a vector. Emotions can be deduced from the "shape" of the
dimensions. Features of
emotions include the shape of a vector of the dimensions, the values of the
dimensions, and the
difference from or similarity to derived calculations from other vectors
associated with the rule.
The output of the Natural Language Rules 306 is fed to the Emotio-Cognitive
Sensor 314 and
Intensity Rating Sensor 316, allowing otherwise qualitative data to be
transformed into
quantitative data for population analysis, as in a scientific study or
political poll.
[66] The intensity rating sensor (316) determines intensity ratings for each
detected emotion
based on the Dimensions. In some embodiments, the Intensity Rating Sensor
(316) assigns
objective intensity ratings based on subcomponents of each cognitive,
emotional, social,
interpersonal or state-based element, as the Dimensions.
[67] The Emotio-Cognitive Tagging Engine (318) tags the textual data with the
assigned
emotion class. Emotion and intensity of emotio-cognition can be associated
with metadata, such
as demographic information, online profile features, time stamps, sources, geo-
locations. A result
for aggregate emotio-cognitive states for a sample of the population and
labeling of emotio-
cognitive states are returned in the Emotio-Cognitive Tagging Engine 318.
17
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[68] A rules discovery engine 326 can generate new rules, also referred to as
rules suggestion.
The machine learning models 320 can be trained on the rules themselves. The
trained machine
learning models 320 can locate similar co-occurrence vectors, similar POS-
patterns, and similar
n-grams, and suggest these as potential new Rules. In addition patterns of
emotions as they are
detected in the Emotio-Cognitive Sensor (314), as well as dimension patterns
may be used to
generate new rules.
[69] The machine learning models engine 320 can include any machine learning
model or
models from among transformer models, such as BERT, RoBERTa, support vector
machine,
word2vec, KNN model, Long Short-Term Memory model, Convolution Neural Network
model.
[70] A statistical models engine 322 can include one or more statistical
models. Statistical
models can include any statistical model or models from among k-means model,
Bayes model,
document search models, logistic regression model, linear regression model,
polynomial
regression model, recommendation matrix, random forest model, and n-gram
language model.
Each statistical model is used as a classifier.
[71] A lexical engine 324 provides lexicons that can be used in the system
300. Sources of
lexicons include NRCLex, Harvard Inquirer, MPQA, sentiwordnet, textblob,
VADER, and other
lexicons not specifically listed.
[72] In some embodiments, the aggregate emotio-cognition ensemble classifier
328 can output
a final answer, such as an emotion. Random Forest can be used as an ensembling
classifier, using
one-hot coding. In another embodiment, Logistic Regression can be used for
ensembling. In a
further embodiment, a neural layer can be used as an output of the ensembling.
An output can
include association of sentiment and intensity of emotio-cognitive sentiment
with metadata, such
as demographic information, online profile features, time stamps, sources, geo-
locations. An
18
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
output can be a pictorial or graphical representation of a user's, brand's,
business's, celebrity's,
organization's, topic's, word's or phrase's emotio-cognitive summary over a
period of or point in
time, using a color-based geometrical representation. Optionally, the output
can include a report,
at any given time or over time, of dimensions, emotions, dynamics, or societal
currents for a
demographically or otherwise segmented or aggregate sample. Optionally, the
output can include
generation and display of the distribution of emotio-cognitive state for a
given time period, for a
single or aggregate users' emotio-cognitive states over time or at a point in
time.
[73] FIG. 4 is a diagram of a training architecture in accordance with an
exemplary aspect of
the disclosure. In order to accommodate an evolving natural language, the
training architecture
400 uses a machine learning models engine 320 to train machine learning models
to generate
rules. The machine learning models also are trained on the rules themselves.
They locate similar
co-occurrence vectors, similar POS-patterns, and similar n-grams, and suggest
these as potential
new Rules.
[74] The linguistic rules model 310 begins with a set of preprogrammed
linguistic rules that
may be applied to natural language phrases and sentences. Detection can
involve potential
comparison of these rules, recognition of similarity, and creation of rules
that use concatenation
of the rule potential elements (rule types). Creation of rules may occur
through identification of
strings with sufficient similarity via a threshold, then creation of tuples of
each word with word,
part of speech, dependency, stem, and lemma, and matching among similar items
based on index
position to find the coordinating part of each tuple. The results can then be
concatenated, with
optional parts, or wildcards, or nearness calculations, as necessary into a
concatenated formulaic
rule.
19
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
[75] The training architecture 400 includes a text input 302 for accessing,
via processing
circuitry and memory, multiple natural language data items. The multiple
natural language data
items may be read from a data repository, or may be directly input as a stream
of text. The text
may include captured, transcribed, or translated text that has originated from
human speech
input, text databases, documents, or other text data sources.
[76] The training architecture 400 includes a pre-processor 304 for performing
various
preliminary processes that are typically performed on text for natural
language processing. The
pre-processor 304 may utilize any of known software libraries for data
normalization,
tokenization, part-of-speech tagging, dependencies, stemming, and
lemmatization to generate
multiple preprocessed natural language data items. An example of software
libraries is the
Natural Language Toolkit (NLTK). The NLTK includes text processing libraries
for
classification, tokenization, stemming, tagging, and parsing. The NLTK
includes a WordNet
Lemmatizer having a lemmatize function, as well as a variety of other
lemmatizers.
Normalization may be performed as canonical normalization.
[77] In some embodiments, the training architecture 300 may involve labeling
the multiple
preprocessed natural language data items as emotion and an intensity of the
expressed emotion.
The labels and associated natural language data items may be used to train a
supervised learning
model.
[78] The training architecture 400 provides the multiple preprocessed natural
language data
items in parallel to machine learning models engine 320 and a linguistic rules
model 306. The
training architecture 400 performs training, in parallel, of the machine
learning models engine
320 and the linguistic rules model 306 in multiple training epochs to
identify, in the natural
language data, emotion and to determine an intensity of the emotion. Each
training epoch of the
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
machine learning models engine 320 may, based on a decision 412, generate
feature or rule
suggestions 326 for subsequent training epochs of the linguistic rules engine
306. After each
training epoch of the linguistic rules engine 306, subsequent epochs of the
machine learning
models engine 320 are tabulated and scored (provided as probabilities of each
class). In 414, an
output representing the trained machine learning models engine 320 and the
trained linguistic
rules 306 is stored in a non-volatile memory.
[79] FIG. 5 is a system diagram of adaptive operation of the emotion
classification system.
The rule discovery engine 326 can generate new rules while presenting emotion
and intensity
information as an output.
[80] In order to assist in operation, the operation of the emotion
classification system will be
described in terms of an example. The example is simplified for ease of
understanding. The
present disclosure is by no means limited to this example. In the example, a
hashtagged text
"Why am I the one who 'needs' to take out the trash? #NOTANEED" is input 302
by reading the
text from a file, or as a continuous stream that is entered into the computer
system. The text input
is processed by Pre-processing Engine 304, which performs functions including
tagging the
Input text with index positions, tokenizing the Input text, and separating out
the Input hashtag. In
some embodiments, the hashtag is a component that has an associated emotion.
The index
position can be an integer number that indicates a position relative to an
overall text, for example
a sequential number that is generated by a counter or within a tokenized
input's array.
Alternatively, the index position can be an index such as a vector position,
line number, or some
input number that identifies where in a sequence of hashtagged text, the
current text input occurs.
The tokenizing function may selectively separate out punctuation marks, such
as "?" as tokens,
and may delete others. In the example, the quote marks around "needs" may be
preserved in
21
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
order to indicate an emphasis. The hashtag may be separated out from the other
text so that it can
be treated as an identifier, rather than as an emotional expression. In some
embodiments, the
hashtag may be used as an auxiliary indication of the emotion. In this
example, the hashtag may
indicate that the input text is a sarcasm. 1-lashtags can also be used to
indicate emphasis,
commentary / asides, subtweets, organization, continued expression, humor,
context, emotions,
marketing, protest.
[81] The natural language engine has a function of searching among natural
language rules
306 for a rule, or rules, that matches the input text. A rule may indicate a
text pattern, with some
required words or phrases, mixed with syntax. In the example, a Rule, such as
"[why]"[...] +
being verb + "the one who" + verb is pattern matched and triggered.
[82] Rules are grouped by emotions and cognitive terms, and emorio-cognitive
blended terms.
The Emotio-Cognitive Sensor 314 has a function of applying a Emotio-Cognitive
Label
(ANGER) based on the triggered rule.
[83] The Intensity Rating Sensor 316 has a function of activating Dimensions
for that
triggered rule, which can include 3 dimensions with scores - positive,
negative, neutral, or null,
the dimensions being for example, respect, ego, blame. The values of dimension
scores are not
limited as such and can include numeric values, in a predetermined range. The
Intensity Rating
Sensor 316 has a function of aggregating Dimension scores to obtain an
intensity score. The
Intensity Rating Sensor 316 assigns the intensity score to the Emotio-
Cognitive Label and
compares the intensity score against a predetermined threshold. The
predetermined threshold
may be a fraction value, such as .6, an integer in a range, such as 6 in a
range of 0 to 10, a
percentage, such as 60%, that is common across all emotions, or may be set for
each Emotio-
Cognitive Label. In this example, according to the threshold, the emotional
intensity level is
22
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
labeled (ANGER -- MEDIUM). In some embodiments, the intensity score may also
be relative to
the speaker, if the speaker's baseline is known.
[84] The Meta-Emotio-Cognitive Aggregator 328 has a function of assessing an
Emotio-
Cognitive Label and combining the label with other surrounding labels to form
a pattern of
Emotio-Cognitive Labels, e.g., ANGER PEACE.
[85] In a next level, a Dynamics 610 has a function of pattern matching the
pattern of Emotio-
Cognitive Labels with a Dynamic pattern. The Dynamics 610 assigns a label
based on the
matched Dynamic pattern, e.g., FORGIVENESS.
[86] FIG. 6 is a diagram showing types of linguistic rules in accordance with
exemplary
aspects of the disclosure. The types of linguistic rules for sentiment,
emotion, opinion, and
belief, that are applied to the specified portion of the natural language
input can include
[87] a rule that uses part of speech tagging, syntax, or dependency parsing
(502), including:
e.g., modal and optative verbs, tense notations, declension, conjugation,
accents, as well as direct
objects, and proper nouns.
[88] a rule that uses string matching, including exact, inexact, masked or
wildcarded (504),
[89] a rule that uses distance between tokens (506),
[90] a rule that uses punctuation (508),
[91] a rule that uses lemmatization (510),
[92] a rule that uses stemming (512),
[93] a rule that uses lexicon (514), and
[94] a rule that uses word lookup or dictionary (516).
[95] In some embodiments, rule components may include graphical content,
including Emojis,
Glyphs, Emoticons, characters, scripts, and any other graphemes. Applying the
linguistic rules to
23
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
the specified portion of the natural language input results in obtaining the
numeric or Boolean
value for each of the one or more linguistic rules.
[96] In some embodiments, additional linguistic rule types accommodate other
world
languages, including Japanese, Korean, Chinese, Vietnamese, and other Asian
languages The
additional linguistic rule types can include
[97] a rule that tags dependencies,
[98] a rule, or rules, that detects particles,
[99] a rule, or rules, that detects markers,
[100] a rule, or rules, that detect structural-narrative forces, including
topic, subject and
predicate,
[101] a rule, or rules, that signifies classes of nouns grouped by
commonalities,
[102] a rule, or rules, that detects cases, including nominative, ablative,
and others,
[103] a rule, or rules, that detects verbal categories, for verbs,
[104] a rule that detects click transcriptions, for capturing semantic
information from clicks, on
a mouse, touchpad, touchscreen,
[105] In some embodiments, linguistic rules may incorporate an OR operator,
for optional
conditions, and a concatenation operator, for combined conditions. In some
embodiments,
linguistic rules may include intra-rule referents, which refer to other parts
of a rule. An intra-rule
referent may include doubling or replication: ADJ + NOUN + NOUN, with a
condition that the
NOUN is an immediately preceding NOUN. An intra-rule referent may include an
index for
positions in a rule: a placeholder or position Element[0] in a rule.
24
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
[106] In some embodiments, intra-rule referents for linguistic rules may be in
the form of
patterns. A pattern may include: NOUN -> NOUN, for duplicate, NOUN l+NOUN, for
not
duplicate.
[107] The linguistic rules are grouped by types of emotion or cognition. In
some embodiments,
rules may have an order of importance, and the order may be changed, to
indicate precedence
over rules below it, and under rules above it. Each linguistic rule has one or
more dimensions
and a value for each dimension. In one embodiment, dimensions may include
sentiment,
emotion, Emotio-Cognitive attitudes, values, social mores, mindsets, outlooks,
aspects,
responses, traits, beliefs, opinions, perspectives, motivations, biases,
states, manners, approaches,
dynamics, personality trait, emotional approach, emotional choice, reaction,
disposition,
temporary state, change of state, cognitive aspect, behavioral aspect,
internal condition, external
condition, feeling, emotion, proposition, attitude, propositional attitude,
directed attitude,
undirected attitude, self-directed attitude, conscious emotio-cognition,
unconscious emotio-
cognition. In one embodiment, dimensions can include emotional affects of:
anger, anticipation,
disgust, fear, joy, sadness, surprise, and trust (from NRCLex). In another
embodiment,
dimensions can include, but are not limited to, facets, components, and
aspects of emotio-
cognition, such as: ego, blame, conformity, sacredness, kindness, respect,
time (future), (self)
favor, approval, sincerity, vulnerability, judgment, separateness, purpose,
formality,
minimization, specificity, force, action (activeness), agency, curiosity,
clarity, intention,
emphasis, energy, certainty, interest, engagement, shock / surprise, tension,
speed, nuance, logic,
paranoia, trust, distance, identification vd, esteem (self), esteem (other),
objectification,
attachment, empathy, and patience.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[108] In some embodiments, dimensions may be grouped into categories, such as
personality
trait, belief, opinion, perspective, aspect, motivation, bias, state,
emotional approach / choice,
manner, reaction, interpersonal dynamic.
[109] The inventors have found that even entirely accurate classification can
still yield very
limited semantic information. For example, "accuracy rates" do not always
reflect useful
semantic information when it comes to affect detection. Labels for affects can
be too vague
(trying to force more than 25 emotions into six labels). Classification may
still lack useful
subtextual information and may lack orientation toward the cognitive state of
the speaker.
Classification does not provide hints as to why a feeling is present.
Sentences, phrases, or
constructions remain poorly classified for specific affect and deep semantic
value, parsed on the
sentence-level, oftentimes forced into a single class where multiple labels
should apply, or
unable to be accurately understood by machine algorithms, due to the
complexity and depth of
human emotion. Additionally, while conversation can be mined for further
information,
discursive text value has been limited to contextual clues approaches such as
Named Entity
Recognition and speaker-specific information, yielding little about affective
states.
[110] Despite the existence of state-of-the-art technology such as
transformers, these have only
historically excelled at prediction and translation tasks, rather than
semantic interpretation (in
part, due to poorer general performance on semantic tasks). Transformers are
ultimately limited
in their semantic capacity, due to the gap between human and machine
interpretative ability.
Among the semantic limitations, particularly with interpreting extemporaneous
speech and text,
are the ability to identify mixed emotions, complex emotions, figurative
expression and insincere
emotional expressions (such as sarcasm, irony, politeness, and passive
aggression). Disclosed
embodiments are an approach to handling range and depth of human emotion and
cognition.
26
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[111] FIG. 7 is a bottom-up stack for layers of the rules engine. "Dimensions"
(702) are
elements of emotio-cognitive expression and the feeling and thoughts behind
them. In some
embodiments, the system is implemented using object-oriented programming and
emotions are a
class having a set of dimensions as attributes of the class. Intensity is a
method of the emotion
class and is used to determine an intensity value based on the presence of
dimensions. Intensity
values can be stored in a structured or unstructured database. The emotion
class has prototypical
values for each of the attributes/dimensions. The values of dimensions are
tiny valences with
trinary values (+1 positive force, -1 negative force, 0 neutral force, and 0
not present / not
applicable; the latter 2 of which are not equivalent). In some embodiments,
the values for
dimensions include layers of Boolean, such as (1) TRUE-FALSE (Neutral vs.
Not), (2) TRUE-
FALSE (True vs. False), (3) Potentially similarly with null. The values may be
n the form of
floating point numbers or integers.
[112] For each data line, these dimensions receive ratings every time a rule
is triggered. So, for
instance, for the example
("[why / how come] BEING-V + PRON + [TIME-HYPERBOLE] + "the one/s who" +
VERB" ), which denotes a construction with:
an optional Word List item (of 2 items: "why" or "how come") +
a being verb plus a pronoun ("are you" / "am 1" / "are they" / "are we" / "is
he" / "is she," etc.) +
an optional Word List item for Time Hyperbole ("always", "never",
"constantly", etc.) +
fuzzy (inexact string) ("the ones who", "the one who")
virtually any verb ("sits up front", "pays for everything", "brings up
problems", "gets the easy
job", etc.) The rule uses those parts that are modular, and covers some degree
of permutation.
27
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
[ 113] The system detects a given emotion's core dimensional profile.
Different rules find
distinct and partial but strongly matching configurations, which represent
various expressions of
emotional states. In other words, emotions are mental states. Primal emotions
are pure and
unreasoning (e.g., fear, anger, happiness, sadness). Complex emotions are more
social and
cognitive (e.g., grief, depression, shame, insecurity, admiration). Complex
emotions occur when
cognitive states and emotional states co-occur, or when multiple emotions are
co-occurring.
When rules are activated, they provide an indication of when such emotional
states are occurring.
[114] The more dimensions that match a particular emotion, the higher the
intensity of that
emotion. Subsequently, the measure of intensity is an objective measure and
represents a degree
of intensity. In some embodiments, if the same construction happens twice in
an input text, the
matching construction is counted both times. In measuring intensity. Dimension
scores are
normalized over the number of words. Thus intensity values are density-based.
[115] In an example embodiment, there are 50 dimensions. In some embodiments,
there are a
greater number of dimensions. Each rule has ratings across the entire set of
dimensions.
[116] Each rule 704 includes one or more Dimensions. For the above rule, one
example
dimension is Ego. Ego can be absent from a given construction (have nothing to
do with it ¨
null value), be neutral (there is an ego component, but it is very even and
fair), be positive
(egotistical, condescending), or negative (admiring, supplicant, self-
loathing, etc.). Ego is a
dimension that is rated for this rule. Another dimension is curiosity. This
construction detected
by this rule is proactively curious, so it receives a 1 rating.
[117] Rules 704, once activated, indicate detected Emotions. Once the Emotions
have been
detected (in the portion of the sentence they exist; tagging is performed via
index position
instead of simply sentence- or passage- level, which allows for greater
distinction and
28
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
clarity), complex emotions 706 can be identified, and these usually are
comprised of
simultaneous emotions. For instance, contempt is a combination of pride
/condescension and
anger. In addition, emotio-cognitive states can overlap.
[118] Once complex and primal Emotions have been activated in place, patterns
of emotions
708 can be assessed, as emotions naturally shift to other emotions, in
particular situations. In
addition, emotions can progress, escalate, or resolve. Patterns of emotions
708 are formed by
concatenating emotions into an ordered array, or a string, of emotions (e.g.,
HOPE +
EXPECTATION + SURPRISE + SADNESS). These patterns can be assessed somewhat
like a
skip-gram model with limited possibilities. Patterns for shifting emotions,
progressing emotions,
escalating emotions, and resolving emotions are stored as recognizable
emotional patterns.
[119] The patterns of emotions can be used to predict flows of Emotion:
excitement turning to
acceptance to calm; anger shifting to confusion, realization, and back to love
as understanding
arises and empathy or forgiveness develop, etc.
[120] The Emotional Patterns build to Dynamics 710. Dynamics 710 are
descriptive labels for
major emotional events that summarize patterns of emotion. For example, a
Dynamic of
"DISAPPOINTMENT" is a label for an emotional pattern of HOPE + EXPECTATION +
SURPRISE + SADNESS. Dynamic can happen within the Self- this is usually the
complex
emotion. Next, the Dynamic can happen with the Other / interpersonally - this
might be a
phenomenon like Forgiveness. In such case, FORGIVENESS is a label for an
emotional pattern
of ANGER + REALIZATION + LOVE.
[121] A layer above Dynamics 710 may include a Societal currents 712, which
are meta-results
of strong emotional dynamics en masse, often brought to a head by the pressure
of major events
or mass movements ¨ such as unrest or innovation. Major events can include
impending real-
29
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
world events (anger explosion, coup, uprising, pop-culture or societal trend,
stock market event
such as the GameStop stock buy-up, the "canceling" of a public figure,
breaking point into
violence, etc.) These societal currents can also be present for smaller
societal groups, such as
with domestic violence breaking points, falling in love, an anger explosion,
etc. Detection of
Societal Currents enables reaction / prevention or capitalization, or at least
passively monitoring /
identification. Such events can be correlated with specific, prior such real-
world events with
these Societal Currents, enabling speedy reaction, prevention of, or
capitalization upon if similar
events are able to be quickly detected or predicted in the future. Computation
of emotio-
cognitive states, Dimensions, Dynamics, and Societal Currents, from natural
language from one
or more user, via the Natural Language Rules 306.
[122] In some embodiments, each level of Emotional /Cognitive flow can be
predicted for these
layers based on historical data, stored as the classification system is being
used.
[123] FIG. 8 is a flowchart of a method of operation of a computer system in
accordance with
an exemplary aspect of the disclosure. One embodiment relates to a process of
tracking nuanced
psychological affect in natural language content. The method first applies
linguistic rules for
emotion to natural language constructions, phrases, or sentences, each rule
having one or more
dimensions. The results from applying the linguistic rules are used to detect
emotion and
determine intensity of emotion. A machine learning model is used to suggest
new linguistic
rules, and thus serves to augment the application of the linguistic rules. The
process may be
performed in a device having a textual and/or voice input and a textual,
speech, and/or graphical
output. A computer program includes instructions stored on a computer-readable
storage
medium, which when executed by a computer system, as in FIG. 2, performs steps
as shown in
the flowchart.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[124] In S802, the computer 226 is configured to receive the natural language
content as text,
which may be derived from the textual and/or voice input. The natural language
content may be
read from a file stored in the disk storage 204, or may be read from a stream
of data received at
network controller 206, or from text input at a keyboard 214, or from text
from voice input via a
microphone. In addition, input can include aggregation of data, from an online
or offline
database, query, taken from a historical archive or corpus, or scraped from a
website, such as a
social media or user review website. Input can be unprompted or prompted and
can concern a
topic, person, brand, organization, concept, word, or group of words. Input
can include interview
data, transcribed or otherwise obtained from participants in surveys, market
research, or
academic studies. In some embodiments, the input may include time stamps,
which may be read
in conjunction with the natural language content.
[125] For large sections of text that may exceed the size for an input text
that is handled by the
natural language rules engine 306 or machine learning models engine 320, one
approach is to
read the input using a sliding window of fixed size. In optional step S804,
the computer 226 is
configured to apply a scanning window of fixed length to the natural language
content. The
length may be a number of characters. The scanning window may overlap by a
certain number of
characters between successive movements of the scanning window. In step S806,
the computer
226 is configured to evaluate the natural language content using the
linguistic rules for each
emotion in order to obtain linguistic features for human dimensions of
emotion. As described
above, linguistic rules are patterned matched with the natural language
content.
[126] In S808, the computer 226 is configured to score each human dimension
for presence,
neutrality, level or absence as dimension scores for each matched rule.
31
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[127] In S810, the computer 226 is configured to aggregate the dimension
scores for each
emotion to obtain an intensity score for the respective emotion. The
aggregated scores represent
intensity of an emotion. In some embodiments, the starting index and ending
index in the natural
language content for the emotion is determined for each particular dimension.
In some
embodiments, the indexed natural language content and corresponding dimension
may be
forwarded to the machine learning models engine 320. In some embodiments, the
top dimension
for emotion may be forwarded to the machine learning models engine 320
together with a
respective numerical (dimension score) or mapped descriptor. Patterns of
emotion including the
starting index and ending index may be stored for an entire passage having
several constructions,
phrases and sentences.
[128] In S812, the computer 226 is configured to classify the natural language
content as an
emotion class based on the dimension scores. In some embodiments, the
classifying may
generate a probability for each emotion class.
[129] In S814, the computer 226 is configured to label the aggregated value as
an emotion and
determine a context of the label, such as a pattern of emotion labels, for the
natural language
content.
[130] In S816, the computer 226 is configured to output the classification and
pattern of
emotion labels as the textual, speech, and/or graphical output.
[131] In S818, the computer 226 is configured to track the pattern of emotion
labels and
associated components in temporal sequence over the natural language content
in order to track
the emotion labels over time. Each emotion label can be assigned an ordered
index number to
identify their order in sequence. In the case of natural language content that
includes time
stamps, each emotion label may be assigned a time stamp of the associated
components.
32
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[132] FIG. 9 is a flowchart for steps for evaluating using linguistic rules in
accordance with an
exemplary aspect of the disclosure. The evaluating using the plurality of
linguistic rules of 5806,
includes the following steps.
[133] In S902, the computer 226 is configured to detect rules using rule
pattern matching; and
[134] S904, the computer 226 is configured to evaluate the human dimensions
of each
detected rule.
[135] FIG. 10 is a flowchart for detecting rules in accordance with an
exemplary aspect of the
disclosure. The detecting of rules of S902, includes
[136] S1002, detecting presence or absence of constructions in the natural
language content
having components related to an emotion.
[137] FIG. 11 is a flowchart for scoring in accordance with an exemplary
aspect of the
disclosure. The scoring of S808, includes
[138] S1102, evaluating each dimension to determine the dimension scores.
[139] FIG. 12 is a flowchart for detecting rules in accordance with an
exemplary aspect of the
disclosure. The detecting rules, among the plurality of linguistic rules, of
S702, includes
[140] S1202, determining numeric values for
[141] a part of speech tagging or syntax rule,
[142] a string matching rule, exact, inexact, masked or wildcarded,
[143] a token proximity rule,
[144] a punctuation rule,
[145] a lemmatization rule,
[146] a stemming rule,
[147] a lexicon rule, and
33
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[148] a word lookup or dictionary-based rule.
[149] FIG. 13 is a flowchart for determining numeric value for a token
proximity rule in
accordance with an exemplary aspect of the disclosure. The determining the
numeric value for a
token proximity rule includes accessing all tokens having a distance of fewer
than n tokens from
a specified point in the natural language content, wherein n is an integer.
[150] FIG. 14 is a flowchart for classifying in accordance with an exemplary
aspect of the
disclosure. The computer 225 is configured to perform the classifying of S812
using a machine
learning method, including any of supervised learning, unsupervised learning,
and a rule-based
system.
[151] FIG. 15 is a flowchart for machine learning in accordance with an
exemplary aspect of
the disclosure. The machine learning of S1402, including
[152] S1502, receiving a plurality of natural language data items from a data
repository;
[153] S1504, normalizing and tokenizing the plurality of natural language data
items using a
preprocessing engine to generate a plurality of preprocessed natural language
data items, which
may include pre-sorting data lines into Positive, Negative and Neutral, in
order to save
computing power and time in classification;
[154] S1506, labeling the plurality of preprocessed natural language data
items with an
expressed sentiment, emotion, opinion, or belief and an intensity of the
expressed sentiment,
emotion, opinion, or belief;
[155] S1508, providing, in parallel, the plurality of preprocessed natural
language data items to
an unsupervised learning engine, a rule-based engine, and a supervised
learning engine;
34
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[156] S1510, training, in parallel, the unsupervised learning engine, the rule-
based engine, and
the supervised learning engine in multiple training epochs to identify, in the
natural language
data, an expressed emotion, and to determine the scalar measure as an
intensity of the emotion.
[157] Each training epoch of the unsupervised learning engine provides feature
or rule
suggestions to subsequent training epochs of the rule-based engine, and each
training epoch of
the rule-based engine provides tabulation and scoring data to subsequent
epochs of the
unsupervised learning engine and the supervised learning engine.
[158] In SI512, an output is generated representing the trained unsupervised
learning engine,
the trained rule-based engine, and the trained supervised learning engine.
[159] In S1514, human dimensions present within the natural language data are
matched by
matching the human dimensions to existing dimensional arrays, with and without
wildcards or
pattern skips, in order to suggest new rules for the rule-based engine.
[160] The system allows for deduction or recognition of the points and levels
(intensities) at
which feeling turns to action (as described within, and detected by, the
text). The system allows
for the recognition of dynamics within the self alone, on a bipersonal (one-on-
one), interpersonal
/ multi-personal or family, or society (community, country, region, or world).
The system allows
the tracking of intensity of emotion as it fluctuates throughout the sentence,
item / paragraph and
passage. The system allows for the identification of large, important shifts
in perception, such as
a sharp turn in aggregate self-image (via ego dimension), which may indicate a
problem or
significant change has arisen. The system can reveal the relationship between
important factors,
via correlation, such as the relationship between self-esteem / self-image and
general optimism.
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
[161] Numerous modifications and variations are possible in light of the above
teachings. It is
therefore to be understood that within the scope of the appended claims, the
invention may be
practiced otherwise than as specifically described herein.
[162] Thus, the foregoing discussion discloses and describes merely exemplary
embodiments
of the present invention. As will be understood by those skilled in the art,
the present invention
may be embodied in other specific forms without departing from the spirit or
essential
characteristics thereof. Accordingly, the disclosure of the present invention
is intended to be
illustrative, but not limiting of the scope of the invention, as well as other
claims.
[163] Example Implementations
[164] ELECTRONIC READING DEVICE
[165] Embodiments of the present invention include an electronic reader. An
electronic reader
can be a dedicated device that incorporates specialized firmware and a display
that is configured
to optimally display text with high clarity (commonly referred to as an Ebook
Reader), or can be
a general purpose computing device, such as a tablet computer or smartphone,
that is configured
with software for text reading, typically in the form of a mobile application
(App). An electronic
reader has a display screen that is generally 10 inches diagonal or less, and
has limited computer
processing capability and memory. In most eases, the electronic reader can
communicate with a
web service via an Internet connection, typically by way of a WiFi connection.
Some electronic
readers include a communications module for communication by way of cellular
transmission.
[166] 'I'he system 300 having a multi-media classification engine 312 can be
performed in a
device with limited processing power and memory, as most of the processing is
based on
execution of the natural language rules engine 306. The machine learning
models engine 320
may be performed off-line in a separate computer, or in a cloud service.
36
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[167] FIG. 16 illustrates an electronic reader in accordance with an exemplary
aspect of the
disclosure. An electronic reader 1600 includes a display screen, or
touchscreen display 1602.
When the display 1602 is displaying text of a book 1604, the display may
include scrolling
functions displayed as scrollbars (not shown), and page turning functions,
displayed as buttons
1606.
[168] FIG. 17 is a flowchart for operation of an electronic reader in
accordance with an
exemplary aspect of the disclosure. In S1702, the system 300 may assess a
written fiction or
nonfiction work for emotional, cognitive, interpersonal or social dynamic,
motivation, belief,
opinion, or psychological elements, by multi-media classification engine 312.
In S1704, text in
an electronic book is scanned and tagged with rules that trigger, emotional,
cognitive or
otherwise states that are identified, and the intensity with which they have
occurred.
[169] In an embodiment, in S1706, the system 300 may generate and display
color-coded
highlighting that designates occurrence of certain emotional or cognitive or
sociological or
interpersonal dynamics and/or states. In an embodiment, in S1708, the system
300 may generate
and display one or more sidebars for dynamics 1610 and emotion-intensity 1620.
The sidebars
may summarize the emotional, psychological, cognitive, sociological, or
interpersonal dynamics
or states that occur within the text 1604, with added context where available.
In S1710, each
dynamic 1612 or state 1622 can be interacted with by selecting 1630 (via
touch, mouse,
keyboard, etc.) allowing the electronic reader 1600 to be presented with
examples 1632 within
the text of that given dynamic or state.
[170] MULTIMEDIA AUDIO BOOK or VISIO-SPATIAL DATA SENTIMENT
CLASSIFIER
37
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[171] The system 300 having a multi-media classification engine 312 can be
performed in a
device with limited processing power and memory, as most of the processing is
based on
execution of the natural language rules engine 306. The system 300 performs
emotion
classification for sentences, phrases, and constructions, and can perform the
emotion
classification in real time as text is being received. The machine learning
models engine 320 may
be performed off-line in a separate computer, or in a cloud service.
[172] FIG. 18 is a flow diagram for a Multimedia Audio Book or Visio-Spatial
Data Sentiment
Classifier in accordance with an exemplary aspect of the disclosure.
[173] Scripted or subtitled multimedia, such as audio books, or visio-spatial
multimedia like
movies or TV shows, can be scanned into the system and transcribed into text.
[174] In 1802, textual and transcribed media is run through a Natural Language
Rules Engine
(306), matching to rules
[175] The Emotio-Cognitive Sensor (314) processes input on a sentence-,
paragraph-, passage-,
scene-, or chapter-level, and classifies each with a given emotion, cognition,
sentiment, state- or
dynamic- or societal-based tag. Stray, short or partial strings and select
individual words, known
as High Use Non-Construction Hooks (HUNCHes) with partial dimensions are
detected within
the text and matched.
[176] The Intensity Rating Sensor (316), analyzes text and assigns objective
intensity ratings
based on subcomponents of each cognitive, emotional, social, interpersonal or
state-based
element, known as Dimensions.
[177] The Emotio-Cognitive Tagging Engine (318) tags the textual data with the
assigned class.
[178] The Temporal-Positional Coordination (1804) makes timing-based
associations between
the textual information's tagged classes and the coordinated sections of
auditory or visio-spatial
38
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
data and signals, which are then classified and sent to the :Emotio-Cognitive
Tagging Engine
(318) to be tagged.
[179] The systems 300 scans for further instances of matching audio or visual
patterns, both
absolute and relative to the Speaker's audio or visual baseline, and adjusted
for gender, age,
class, race, accent, locale and other demographic information, and situational
information,
automatically tagging them with Emotio-Cognitive classes associated with the
pattern in
question
[180] Text coordinated with auto-tagged emotional classes in the prior step
become
automatically flagged in the text as matches to, or flagged as high-likelihood
for, the Emotio-
Cognitive classes now borne by the audio or visual data.
[181] FIG. 19 is a block diagram of the multi-media rules engine. Textual,
visio-spatial and
audio data are compared and re-processed as Multi-Media Rule Suggestions and
sent back to the
Natural Language Rules Engine (310), Visio-spatial Rules Module (306), and
Audio-Speech
Rules Module (308).
[182] Subsequent audio or visual input exhibiting similar, opposite, partial-
matching or
exhibiting an otherwise mathematically significant ratio or relationship in
audio or visual signals
and patterns will receive tags indicating likelihood of the Emotio-Cognitive
class.
[183] Comparison of likeness, differences, opposites, and other measurements
of each Multi-
Media Rule Engine's 312 Rules and suggested rules will improve the rule-
suggesting ability, and
highlight modular components of each rule.
[184] In one embodiment, HUNCHes are used as hooks, allowing the system to
analyze
surrounding text, suggesting potential new rules back to the Multi-Media Rules
Engine (312).
39
CA 03207044 2023- 7-31
WO 2022/183138
PCT/US2022/020994
Dimensional patterns must match Positive and Negative Dimensions or be
noncontradictory to
the rule pattern.
[185] In particular, the HUNCHes are non-construction based partial pieces
(such as 1-2 words,
or partial phrases), which carry their own dimensions. Feeding these into the
system 300 allows
the neural models 320 to "match" these partial-dimensional patterns and use
them as hooks for
finding indicators of emotion. In an example, an ANGER rule lies undiscovered,
but has Ego +1,
Impatience +1, Force +I, etc. A hook (like "I won't") may only have Ego +1 and
Force +I, but if
it looks enough like it could fit with anger, the surrounding text will be
examined. To do this, it
must have 0 contradictory dimensions, and it must have its existing dimensions
matched. This
allows permutations of rules to be detected (where both are true), or
potentially new rules (where
not contradictory, but where the dimensional profile looks enough like anger).
Through this
technique, a new ANGER construction is found such as "I [won't / am not going
to / am not
gonna] take it ["one more time", "again", "anymore", "at all"], etc., which
might distill down to a
rule like "[IP PRON] + ([AUX VERB] + [NEGATION]+ (("GOING") + [INFINITIVE-
VERB]
or [FUTURE-VERB]) + ((D-OBJECT-PRON))* + [WordList: Time-Phrase] + "!", with
asterisked portions optional, yielding, for example:
I'm not going to take it anymore!
I'm not going to serve you again!
We will not bow anymore!
For some emotions and some dimensions with respect to a given emotion two or
more values
may be acceptable. For example, anger can be blunt or nuanced so could receive
either score.
[186] In some embodiments, depending on the emotion and the dimension, more
than one value
may be acceptable. For example, Anger can (and must be either) ego-neutral or
ego-positive. In
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
other cases, it must be one valence for a dimension. For example, Anger must
be forceful; anger
must be specific. Consecutive HUNCHes together might match an emotion
dimensional array.
They also might not, if emotion shifts.
[187] FIG. 20 is a flowchart for a rules discovery engine based on HUNCHes in
accordance
with an exemplary aspect of the disclosure. The process is performed in the
rules discovery
engine 326.
[188] In S2002, dimensions associated with HUNCHes may be scored.
[189] In S2004, HUNCHes are matched against profiles for emotio-cognitive
states. To be
matched with a possible emotio-cognitive state, they cannot have contradictory
elements or
dimensions to that state. Empty dimensions that do not contradict can
potentially be matched.
[190] In S2006, Negative or Positive dimensions must likely match unless
flagged as otherwise.
[191] In S2008, Emotio-Cognitive states are suggested.
[192] In S2010, new string matches and surrounding text are suggested back to
the Natural
Language Rules Module 306.
[193] In S2012, audio or visio-spatial data corresponding with new strings is
fed into the
corresponding Multi-M:edia Rules Engine 312.
[194] FIGs. 21A, 21B is a flowchart for rule discovery in audio media in
accordance with an
exemplary aspect of the disclosure.
[195] In S2102, audio media (or sound from visual media) is transcribed to
text and inputted.
Optionally, Closed-Captioning transcripts or scripts may become input. in some
embodiments,
gasps, noises, gestures, and other non-textual audio may be transcribed as
annotations in order to
capture more features of the audio media.
41
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[196] In S2104, rules are applied to incoming text to detect emotion (as well
as cognition and
other elements).
[197] In S2106, index positions in the input where rules have been
successfully applied are
noted
[198] S2108, rule dimensions are calculated to determine intensity on a
phrase-, sentence-,
passage-, and chapter- level.
[199] In S2110, passages are annotated with the corresponding Emotion (and/or
other elements)
and respective intensities and attributes.
[200] In S2112, corresponding soundwaves or visio-spatial data are
coordinated, by time, to the
index position of the words in the passage.
[201] In S2114, actual audio characteristics and relative relationships
between the speaker's
baseline audio or visio-spatial profile and a given specific Emotio-Cognitive
profile of theirs are
calculated and stored.
[202] In S2116, when similar, derivative, opposite or otherwise related audio
fragments appear,
they are matched to the Emotio-Cognitive label in question. Subpassages or
strings that trigger
the same rule for different emotions are fed into learning algorithms. Similar
sound or visio-
spatial fragments detected in data are pre-tagged with suggested Emotio-
Cognitive states, based
on similar audio's or visio-spatial data's existing tags. Similar fragments
detected in audio or
visual-spatial data are fed back into the Natural Language Rules Module 306.
[203] FIG. 22 is a graph of a speech signal pattern. one embodiment, the
speech signal
pattern 2210 for a text input 2202 is indexed 2204 with the text input 2202
and is used to mark
the beginning time and ending time (2208) for an emotion 2206.
EMOTIO-COGNITIVE DYNAMIC DISPLAY
42
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[204] Historically, certain subsets of the population have experienced various
barriers to
understanding emotional, cognitive, social, belief-based, interpersonal, or
metacognitive, meta-
emotional, or decision-making cues or dynamics in spoken or written text In
particular, people
who are non-neurotypical (e.g., neurodivergent), with conditions such as
autism, encounter
problematically high rates of misunderstanding or missing social cues,
emotions, and other
dynamic elements present in the language. Non-native speakers of a given
language may also
encounter these difficulties. Media consumed by these populations can be
confounding,
confusing, misleading, and sometimes lead to disconnection, interpersonal
conflict or alienation,
shunning, and isolation when social norms are not understood and/or
transgressed. Enjoyment of
media may also be reduced, or less able to be spoken about for social bonding
reasons.
Entertainment companies can also lose or experience reduced viewership because
audiences do
not pick up on the emotional, interpersonal, belief-based, or cognitive
dynamics at hand.
[205] Analyzed and annotated subtitles can be fed into machine learning
algorithms, for greater
efficacy as new situations, dynamics, emotions, cognitive states and social
aspects, currents, and
dynamics evolve or are introduced. Theory of mind modules can be updated with
additional
iterations, to adjust to a given society, gender, religious, belief, age,
topic, or culture or
subculture, for accuracy.
[206] Training data can be partially automated due to the robust information
retrieved and
matched to incoming media.
[207] M:edia can be created using the training data, and written, collected,
spliced,
compromised or mimicked using the annotations on prior data.
[208] Training data can be auditory, visual, textual, multisensory, or a
combination of all types.
43
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[209] One type of training data includes the GoEmotions dataset, with the
following
modifications: Elimination of the Desire class; Elimination of the Neutral
label, and instead, the
use of the neutral label in deriving a Surety Score of: 1 - Neutral
probability score = Surety
Score.
[210] Training data is sourced from locations where sentiment such as but not
limited to
emotions, cognitions and other dimensions are necessarily or with above-
threshold necessity
logically required to be the case, and/or where the author self-proclaims
their own emotion,
effectively providing tags, and allowing for empirical labels.
[211] An embodiment of the system 300 for automated classification of emotion
is an Emotio-
Cognitive Dynamic Display. The dynamic display can insert subtextual data of
emotion in real
time which allows neurodivergent users to learn and refine their social,
emotional, interpersonal
and theory of mind skills as they go.
[212] The natural language rules engine 306 is configured with rules
consisting of construction-
based, high-use, and/or permutable phrases allowing for quick determination of
linguistic
patterns in textual information made available by subtitles. Hashtags are
broken up by
contextual, and corpus, unigrams and other present n-grams derived from the
input, instead of
merely made available via lexicon(s), providing for more accurate, more
relevant natural
language processing and sentiment analysis. Similar, partial or opposite
dimensional patterns are
used for detection of sentiment. Rules are created to encode dimensions that
can be used to
detect mental health symptoms in each rule-triggering construction.
[213] The emotio-cognitive engine 314 is configured to score subcomponents
("dimensions-)
for each rule allowing for fast and increasingly-accurate recognition of
motivation, influence,
emotion and cognition and similar subcomponents which aggregate in distinct
patterns and
44
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
combinations to identify transient and enduring emotive, cognitive, belief-
based, opinion-
centered, social and personality states. Emotions are deduced from the "shape"
of the
dimensions Features of emotions can include the vector of the dimensions, the
values of the
dimensions, and the difference from or similarity to derived calculations from
these parts.
[214] The emotio-cognitive tagging engine 318 is configured for tagging and
tracking of
development of subtextual information over the course of subtitles and speech
using the
emotional, cognitive, belief, motivational and opinion state subcomponents
Once emotions are
tagged, Aggregate Etnotio-Cognition Ensemble Classifier 328 tags meta-
emotional states, shifts,
and combinations based on emotional patterns.
[215] The rules discovery engine 326 is configured to identify, utilize,
deduce from and infer
gapped emotional states, trends, dimensional transitions, and other
sentimental states and shifts
to suggest potential new rules, which are then fed back into the system 300.
Emotional patterns
and Meta-emotional shifts, states, and combinations are deduced from gaps
within patterns of
emotion, cognition, or other sentimental components. The rules discovery
engine 326 is
configured with affective logic that is used to deduce missing sentimental
states and shifts in the
data, or to solve semantic, cognitive, emotio-cognitive or otherwise
sentimental ambiguities. The
rules discovery engine 326 is configured to control training of the machine
learning models
engine 320 on the "edge" of the classes, such as between near-emotional
states, like Grief and
Sadness, or Sadness and Anger, for more adept, finer, and faster
differentiation.
[216] The display 210 is configured to display juxtaposed subtextual cues
against visual and
auditory (auditory tone and speech) data to enable richer information, and
situational awareness.
Subtextual data is displayed to inform users and viewers and augment and/or
clarify social
situations and complex emotional and cognitive states depicted or described in
the media.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[217] The rule discovery engine 326 is configured to work with the machine
learning models
engine 320. Analyzed and annotated subtitles can be fed into the machine
learning models
engine 320 as new situations, dynamics, emotions, cognitive states and social
aspects, currents,
and dynamics evolve or are introduced. Training data can be obtained from
information retrieved
and matched to incoming media. Training data can be auditory, visual, textual,
multisensory, or a
combination of all modes.
[218] In one embodiment, training data includes the GoEmotions data set. In
the embodiment,
the GoEmotions dataset is modified by deleting the Desire class. The Neutral
label is replaced
with a Surety Score that is derived from the neutral label, as: 1-Neutral
probability score =
Surety Score.
[219] In some embodiments, training data is obtained from sources where
sentiment such as but
not limited to emotions, cognitions and other dimensions are necessarily or
with above-threshold
necessity logically required to be the case, and/or where the author self-
proclaims their own
emotion, effectively providing tags, and allowing for empirical labels.
[220] FIG. 23 is a flowchart for a method of real time emotion classification
in a stream of
video/audio in accordance with an exemplary aspect of the disclosure. The
method is performed
with a display device, including a tablet computer, smartphone, smart TV, that
receives
streaming audio and/or video, and that includes its own built-in computer with
memory.
[221] In S2302, a scene from a movie or television show or streaming show or
captured theatre
play or animated video source is received together with coordinated textual
transcription.
[222] In S2304, rule matching of the textual data is performed by the rule-
based engine 306
emotio-cognitive engine 314, intensity rating sensor 316, and emotio-cognitive
tagging engine
46
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
318, which tags emotional, cognitive, and other such states, with intensity
ratings, in the textual
transcription.
[223] In S2306, the aggregate emotio-cognition ensemble classifier 328
determines contextual
clues based on word co-occurrence, discursive elements, and topic elements.
[224] In S2308, the emotio-cognitive sensor 316 optionally marks individual
strings or n-grams
with trinary dimensional scores.
[225] In S2310, the visio-spatial rules engine 310 and audio-speech rules
engine 308 detect and
enter augmented information (AugD and situational elements (SE) apparent in
visual data or tone
elements apparent in auditory data into a separate, but time-coordinated,
source for the media.
[226] In S2312, the emotio-cognitive sensor 314 performs juxtaposition
(coordination and
divergence, and degree of each) from the context-oriented semantic information
(contextual
clues) and Aug! and SE data, creating context scores for each scene
[227] In S2314, bracketed emotional data is returned inline and inserted into
the textual
transcript for display in the display device 210 so that the viewer may have
an easier time
correctly identifying emotional, cognitive or social elements of the media.
[228] FIG. 24 illustrates a display device in accordance with an exemplary
aspect of the
disclosure. An example display device 2400 includes a display screen 2402 for
displaying a
scene from a movie or television show or streaming show or captured theatre
play or animated
video source together with coordinated textual transcription 2412. In this
example screen,
bracketed emotional data 2414 (e.g., Emotion-Intensity pair, "ANGER-MEDIUM")
is returned
inline and inserted into the textual transcript 2410 and displayed in the
display device 2400. It is
noted that although the Emotion-Intensity pair is shown as a pair of terms,
the Emotion-Intensity
pair 2414 may be displayed in other formats, including Intensity as a
numerical value, or a
47
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
graphical symbol that depicts a relative quantity (Low, Medium High), such as
a bar of a certain
relative size and/or color, or a set of colored dots that vary in length.
Also, an Emotion-Intensity
pair 2414 may be displayed in a graphical container, such as inside a circle
or rectangular shape,
or in a comment balloon-like shape. In addition, although a single Emotion-
Intensity pair 2414 is
illustrated, the number and/or sequence of Emotion-Intensity pairs is
dependent on the contents
of the textual transcription 2412. For example, the Emotion-Intensity pairs
2414 may be
displayed as a pattern
[229] BORDERLINE PERSONALITY DISORDER SOOTHING DEVICE
[230] FIG. 25 is a diagram for the emotion classification system with sensory
distraction in
accordance with an exemplary aspect of the disclosure.
[231] Borderline Personality Disorder (BPD) is a mental health disorder that
impacts the way
one thinks and feels about oneself and others, causing problems functioning in
everyday life. It
includes self-image issues, difficulty managing emotions and behavior, and a
pattern of unstable
relationships. Treatment for BPD includes learning to manage emotions that
feel intensely
overwhelming and can lead to self-harm. Disclosed embodiments include a
peripheral device for
giving feedback based on certain emotions and pattern of emotions, especially
intensity of
emotion.
[232] A patient diagnosed with Borderline Personality Disorder (BPD) can be
provided with a
microphone equipped device or an input device that can accept input from a
keyboard, other text
input device, such as a voice recorder with transcription, or device with a
text input function,
such as a touchscreen equipped device having a text input screen. Speech
signals may be
converted/transcribed into text. Text may be directly inputted (302) to a
device. The device is
preferably a portable/mobile computing device that the patient may carry. In
some embodiments,
48
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
the portable/mobile computing device may be a device that facilitates digital
communication
with a cloud service.
[233] The textual input is processed by the Natural Language Rules Module
(306), which then
provides Emotion labels, and passes the input to the Emotio-Cognitive Sensor
(314).
[234] The Intensity Rating Sensor (316) computes dimensions for each input and
assigns an
objective intensity rating.
[235] Running averages and aggregate scores are computed in the Aggregate
Intensity Module
(2504).
[236] Instantaneous intensity scores are computed in the Instant Intensity
Module (2502).
[237] When the running Negative Emotional Intensity in the Aggregate Intensity
Module
(2504) reaches a high enough threshold, the system may optionally proactively
dispense a
sensory-soothing aid (2506) via a peripheral device, configured for
Dialectical Behavioral
Therapy (DBT), in order to create a sensory soothing distraction for the
patient, including, but
not limited to: Dispensation of spicy candies, Vibration of Bluetooth
bracelet, Heating of
Bluetooth bracelet.
[238] When any given data point in the Instant Intensity Module (2502) reaches
a Negative
Emotional Intensity over a Danger threshold, the system proactively activates
periphery device
2506 to create a sensory distraction.
[239] Intensity monitoring of language or speech resumes via both the
Intensity Rating Sensor
(316) and the Emotio-Cognitive Sensor 314. When aggregate or data-point
intensity reaches a
first threshold, a different randomized sensory soothing distraction is
activated.
[240] Once the Aggregate Intensity Module (510)'s running average and the
Instant Intensity
Module's (2504) ratings have not risen above the threshold for 30 or more
minutes, a diary card
49
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
is dispensed (2508) by the central unit for the patient to fill out and record
the experience, for
personal or therapeutic use.
[241] In other embodiments, the emotion classification system with sensory
distraction has
application beyond BPD. Optionally, a user overcoming their addiction to
alcohol, drugs or
compulsive behavior is monitored in an aftercare rehabilitation program or
sober living home,
for Emotio-Cognitive signs preceding, or of, relapse, and provide textual or
transcribed input
(302) to a device Optionally, a worker's or manager's professional
communications are
monitored for signs of particular emotio-cognitive states, such as stress or
anxiety, and inputted
(302) to a device. Optionally, when the running Negative Emotional :Intensity
in the Aggregate
Intensity Module (2504) reaches a high enough threshold, the system alerts the
facility,
rehabilitation program, or sober living home, or the employer that the user is
at risk of relapse, or
extreme stress or anxiety.
[242] FIGs. 26A, 26B, 26C is a schematic diagram of an electronic bracelet in
accordance with
an exemplary aspect of the disclosure. The electronic bracelet 2600 may be in
the form of a ring
2612 having embedded electronic components.
[243] Wires:
The wires 2610 are a bundle of multiple insulated wires. Individual wires are
either positive
` ' or negative `-`. The wires 2610 are preferably nichrome (nickel chromium),
but may be made
of other wire materials.
[244] 'I'he bracelet 2612 may include an embedded communications chip
(connected to a
microprocessor chip 2620), configured for wireless communication. The wireless
communication is preferably a short range communication for transmitting and
receiving signals
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
from the portable/mobile device. In one embodiment, the communication chip
performs wireless
communications using Bluetooth, or Bluetooth low energy (BLE).
[245] A sliding window feature 2614 is glass or plastic. The sliding window
2614 exposes a
dispenser 2616, which is single aperture. The sliding window 2614 is motorized
with a micro
solenoid step motor 2602. (< 5mm).
[246] The device 2600 is powered by an embedded microprocessor 2620. The
microprocessor
includes seatings for the wires and components using assembly board-type
technology
[247] A micro electromagnetic eccentric motor 2622 is an actuator that
generates vibration via
an imbalanced load.
[248] A copper plate 2608, seated in silicone, provides heat.
[249] The device is powered by a Lithium-ion rechargeable battery 2604. The
rechargeable
battery 2604 has an associated recharging interface 2606.
[250] Indicator lights 2630 on the band 2612 exist for pairing with the
portable/mobile device.
[251] The band 2612 is translucent and can change color via LED lights
depending on the
emotion detected. This may be useful for people with BPD experiencing aversive
tension, in a
heavily activated state, to communicate their emotions with caregivers or
loved ones.
[252] An adaptation of the device 2600 may also be useful for grounding people
with PTSD
amid panic attacks, and transmitting emotional information to loved ones and
caregivers when
triggered.
[253] In the above description, any processes, descriptions or blocks in
flowcharts should be
understood as representing modules, segments or portions of code which include
one or more
executable instructions for implementing specific logical functions or steps
in the process, and
alternate implementations are included within the scope of the exemplary
embodiments of the
51
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
present advancements in which functions can be executed out of order from that
shown or
discussed, including substantially concurrently or in reverse order, depending
upon the
functionality involved, as would be understood by those skilled in the art.
[254] EMOTIO-COGNITIVE PROFILER
[255] :In an example embodiment, emotio-cognitive classification may be
implemented as a
profiler.
[256] Text input is spoken or otherwise entered into the Natural Language
Rules 306 for
analysis. Input may be about the user, another person, or a topic. Optionally,
a user may video-
capture themselves answering randomized job prompts. Optionally, OCR or audio
transcription
transcribes written and video or audio encapsulated (respectively) textual
information to plain
text.
[257] Text is analyzed by the Emotio-Cognitive Sensor 314 for sentimental
content.
[2581 Emotio-cognitive sentiment is labeled by the Emotio-Cognitive
Tagging Engine 318
after classification.
[259] The intensity of sentiment is calculated via dimensions within the
Intensity Rating
Sensor 316.
[260] A profile is generated with Personality Traits, Expression Traits,
Cognitive Traits,
and Emotional Traits as well as Values and Biases, within the Emotio-Cognitive
Profiler.
[261] Demographic information is associated with profile information in the
Emotio-
Cognitive Profiler.
52
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[262] The Aggregate Emotio-Cognition Ensemble Classifier 328 makes
predictions about
the type of candidate who will make a strong fit for the date, job, or jury.
Optionally, the user
inputs, such as by survey or questionnaire, information about the type of
personality traits,
values, cognitive traits, emotional traits, values and decision points their
ideal candidate would
exhibit.
[263] Recommendations are made to dating site users, jury consultants, or
hiring managers
to recommend a given person as a match, rating desirability and fit.
[264] The profiles compiled by the Emotio-Cognitive Profiler are retained
and stored for
future use by the M:achine Learning Models Engine 320.
[265] EMOTIO-COGNITIVE INFORMED TEXTNG/COMMUNICATON
[266] In an example embodiment, emotio-cognitive classification may be
implemented as a
pre-text application, including, among other things, phone banking, or text
banking.
[267] The user submits textual or transcribed input into the system for emotio-
cognitive
guidance to inform the user about healthy, empathetic, positive, and socially-
and /
(sub)culturally-aligned communication techniques and psychology.
[268] Optionally, the relationship in question may be: sales relationship;
customer service
relationship; workplace relationship; academic relationship; counseling /
coaching relationship;
romantic relationship; friendship; family relationship; former or proposed
relationship; estranged
relationship; relationship in crisis; acquaintance relationship; or other
human relationship.
[269] When unhealthy, inappropriate, insensitive, confiising, or perspective-
limited responses
are detected in the user or the user's text, the user is prompted to develop
their communication
skills to learn about stronger communication approaches, via games, modules,
tutorials or other
53
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
teaching devices. Whether or not the user opts for a learning game or module,
the user is asked if
they would like to input more context about the situation.
[270] The user is asked if they would like to try the communication again.
[271] If the user chooses to add context, they can answer a series of
questions that help assess
emotio-cognitive states and the emotional or social situation. If the user
declines to provide extra
context, the user can opt to be asked further questions to assess his or her
own emotio-cognitive
state.
[272] Optionally, users can select from a list of worries, issues, problems
and hopes for the
communication or relationship.
[273] The user is given the opportunity to change their communication
technique and input a
new communication that incorporates the user's new knowledge and awareness, or
reflects their
changed emotio-cognitive state
[274] Users are prompted to start fresh.
[275] DYNAMIC/ADAPTIVE EMBODIMENT
[276] Spontaneous user textual or transcribed input is entered into a
computing device, app,
gaming console, phone, or tablet.
[277] Text goes to the :Pre-Processing Engine 304 and analyzed by Natural
Language Rules 306
and then analyzed by the MultiMedia Classification Engine 312.
[278] Overtly stated (explicit) and subtextually detected (implicit) emotio-
cognitive states are
assessed in the :Emotio-Cognitive Sensor 314 and degree of mental states are
scored in the
Intensity Rating Sensor 316.
[279] User inputs receive labels in the Emotio-Cognitive Tagging Engine 318.
54
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[280] The system provides a crafted, selected, generated, edited, or otherwise
transformed
response and / or adapted to the user's emotio-cognitive state, providing a
customized,
interactive, discursive experience for the user. Optionally, the discourse may
involve more than
one computer-generated and / or guided bot or character. Optionally, the
discourse may occur
during the course of a traditional, virtual reality, or augmented reality
video game. Optionally,
the discourse may involve one or more additional human agents. Optionally, the
discourse may
involve a therapeutic bot providing psychotherapeutic-, counseling-, crisis-
or sobriety help to
the user or groups. Optionally, the discourse may involve a virtual friend or
assistant providing
companionship or help to the user. Optionally, the discourse may involve one
or more automated
sales or customer support bots.
[281] Optionally, environmental controls such as complexity, difficulty,
volume, speed, color
scheme, interventions, plotlines, mini-games and/or side quests, options,
questions, characters,
alerts, commentary, dialogue, capacities or offered choices, and other such
customizations and
custom activities adapt with user response to the most recent and aggregate
bot-driven language
replies.
[282] Past responses from the user/s are stored in a database, paired with
respective iterated
prompts, for future reference during current and future user sessions.
[283] SOCIOLINGUISTICS
[284] PROBLEM:
[285] Historically, sociolinguistics studies up-close attitudes, beliefs,
opinions, and sentiments
of individuals, classes of individuals, people groups, races, genders,
socioeconomic groups and
other people groups via observation and/or analysis of their spoken, written
or gestured words.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[286] Sociolinguistics is a subdiscipline of semantics that assesses text in
relationship to
societal factors, including subcultures, differences among races, genders,
sexes, subclasses of
people, regions, and more. Sociolinguistics has long since studied the context
of language,
including cultural norms, cooperations, expectations, issues of identity,
interpersonal
interactions, and social context.
[287] Researchers in the humanities and social sciences, as well as natural
language processing
(NI,P) engineers in the computer science field, perform analysis of
spontaneous and crafted text
and speech. Historically, these researchers and professionals have had to
deploy computer code
in order to parse, clean, tag, annotate and analyze words to gamer insights.
[288] These shortcomings have alienated Researchers from the Humanities and
Social Sciences
(such as Psychology, Sociology, Applied Behavioral Science, Anthropology,
Communications,
Rhetoric, Women's Studies, Ethnic Studies and Political Science), depriving
society of rich
commentary on everyday language use and its significance. Furthermore, the
general lack of
integrated computational solutions for sociolinguistic and cognitive language
assessment have
prevented acceleration of social progress in an era where textual and speech
communications are
increasingly rapid, vital and centered for societal advancement, equity and
comprehension.
[289] Furthermore, natural language, cognitive and sociolinguistic pipelines
themselves have
generally required difficult installations, dependence on pre-made data flows
and their
constraints, or incompatible solutions that create difficulty and delay to
coordinate.
[290] PRIOR APPROACHES:
[291] Previous attempts to automate sociolinguistic research and analysis of
digital, scanned,
transcribed or translated text has required a succession of multiple tools,
each requiring technical
know-how to classify and integrate. Available corpora often suffer from
limitations of scope,
56
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
accessibility and size, and additional corpora are difficult and expensive to
curate. Few user-
friendly scraping engines exist, offer only basic parsing, such as and do not
integrate with fine-
grained linguistic and natural language tools, allow for observer or tagging
insights in the native
environment, classification of human sentiment, belief, opinion or emotion
have centered on
supervised learning (often with Bayesian probability), unsupervised learning
(such as neural
networks).
[292] No existing solution integrates state-of-the-art social research tools
that cover the entire
study pipeline from top-level study design; survey skip-pattern design;
volunteer recruitment;
consent form integration; polling, surveying, interviewing, or data
contribution; data intake and
ingestion, study supervision, and query creation. Similarly, no existing
solution provides a suite
of modern, custom statistical analysis tools that span linguistic,
computational, aggregation,
cloud integration and machine learning training, testing, and use.
[293] Existing solutions also do not provide a seamless way to download,
disseminate, collect,
comment upon research and academic datasets, statistical sets, relevant code,
and results papers.
Finally, existing solutions do not allow users to contribute datasets back to
the general public or
academic peers for use in further research.
[294] METHOD
[295] This sociolinguistic pipeline is needed and wanting as technology
advances, expanding
reach of sociolinguistic studies, as well as amplifying humanities and
sciences research outside
of the linguistics field to incorporate textual, auditory, and visual
historical, subculture,
population, geographical and meta-data for the advancement of research, as
well as society.
[296] Currently, sociolinguistics is performed on curated, narrower datasets.
Computational
tools for the professional tools for the professional or academic
sociolinguist are wanting, with
57
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
existing solutions as overly fundamental, less fine-grained, and with a high
technical barrier to
entry. The very few user-friendly solutions for sociolinguists tend to be
incomplete pipelines,
allowing for basic curation, simple search and top frequency phrase
calculations, export, and
little else.
[297] This pipeline is novel and valuable for continuous, wide computational
linguistic analysis
on a range of topics, for commercial use in reputation management,
advertisement and marketing
assessment, viral topic tracking, as well as propaganda and "fake news"
detection and associated
ratings.
[298] In general, datasets' performance often varies by type and task.
Existing solutions show
POS-tagging, n-grams, collocations, and frequency and relative frequency, and
allow for
scraping and basic language, term or linguistic analysis. However, existing
solutions do not
provide robust and dedicated, discipline-driven sociolinguistic tools,
centralized academic,
research, and nonprofit-driven tools for sociolinguistic research in the
humanities and sciences,
as well as study design and deployment tools throughout the life cycle of
research studies.
[299] This method also includes optional geolocation tools, word and term
comparison tools,
discourse analysis and social interactions tools, as well as sociolinguistic
sentiment analysis.
Juxtaposed over time, language shifts indicating population value detection
and attitudinal
change are also novel, and can be leveraged for social problem and societal
value tracking.
[300] Advocacy groups, nonprofits, and academic researchers currently have no
comprehensive
sociolinguistic pipeline to handle these needs fully, as well as digital means
to design and create
robust sociolinguistic, short-term, near-term and longitudinal studies end-to-
end. Public policy
could benefit from the digital application of the sociolinguistic pipeline to
other disciplines, and
commercial brands could also benefit from messaging tracking and analysis,
market research,
58
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
product development, focus group testing, and sociolinguistic, automated
sentiment analysis over
time.
[301] Evaluative uses could also be applied to this pipeline, including
national security
analysis, law enforcement or police brutality monitoring, psychological and
social behavioral
studies, social program evaluation, legal analysis, fair and effective hiring
practices, and risk
management.
[302] Finally, mental health and physical health care as well as vaccine,
pharmaceutical and
therapist and physician care greatly needs a comprehensive, integrated-
research analytic
sociolinguistic pipeline for mass assessment of health care issues, adverse
drug reactions, and
disease monitoring. Financial markets are also currently vulnerable to
movement by mass social
media expression, and would benefit from a means to assess, track and monitor
social media
expression through a technical lens.
[303] FEATURES OF METHOD
[304] The Sociolinguistic Ermine provides the following features:
= Integration of automated semantic tools available for sentiment analysis,
including positive-negative-neutral (PNN) polarity-based measurement, specific
affect detection with multiple emotional labels.
= Seamless data processing flows able to be set up hierarchically by the
user, to
completely automate a custom pipeline for textual data processing.
= Integration of connotative tools that establish, detect and allow for
notation and
prediction of latent and subtextual inferential information, word embeddings,
sentence embeddings, cognitive linguistic features, structure-semantic
mappings
and markers for deductive and inductive reasoning.
= User-defined measurable dimensions, including but not limited to
linguistic
features and morpheme, phoneme, lexical, sentence-level and paragraph-level
59
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
structural components, with user-distributed weights for algorithm creation.
= Optional integrated threshold determination and weighting for
categorization of
specific, predefined or layered semantic information.
= Selectable and tuneable Machine Learning integration, including seamless
deployment of 0 to n models of supervised, semi-supervised and unsupervised
learning onto the dataset.
= GUI graphical user interface)-guided wizards that allow nontechnical
users to
make study design selections to set parameters for Research Project observers,
annotators, subjects and raters for subjective labeling of textual or
transcribed
data.
= Calculation of dataset statistical averages, moving averages, and other
methods
of central tendency, historical emotion and sentiment scores linked to user-
defined themes, corpora or search queries, connotations scores for each user-
defined dimension for user-defined themes, corpora or search query, and
statistics about other user-defined syntactic, phonological, morphological,
cognitive linguistic, sociolinguistic and semantic linguistic dimensions of
text.
= Intensity: Our classification model is the only one we are aware of that
classifies
emotion, connotation or lexical frame intensity instead of probability (of a
classification) offered as a proxy for intensity.
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
[305] MODEL.
[306] This invention describes a sociolinguistic pipeline that enables
researchers from social
science, computer science, and humanities disciplines and subfields to analyze
spoken and
written language, garnered online and offline, and create analyzable datasets
analyzed by
machine learning and procedural code to yield insights to the user, without
the need for computer
programming knowledge required.
[307] Researchers, nonprofits, and other users can use the Sociolinguistics
Engine to deeply
analyze speech and text corpora. Text can be time-sliced to be tracked over
time, or constrained
for time-bounded analysis.
[308] Preprocessing includes part of speech tagging (as well as an option to
create POS-grams),
n-gram generation, frequency and relative frequency calculations, Named Entity
Recognition,
col locations, skip-grams, bootstrapped n-grams, bootstrapped and other
lexicons, and other MI'
data preparation features.
[309] TECHNICAL DETAILS
[310] The Core Sociolinguistics Engine System flow incorporates the following
essential and
discretionary customized steps:
I. The system begins by saving, ingesting, scraping, and otherwise
cataloguing text posted,
generated by and within, and replied to, digitized, transcribed, or resident
on the Internet
(including but not limited to audio or video spoken words ["speech"], plain
text, marked
up text and emojis, annotated, hyperlinked or augmented text, transcripted
text or
translated text). Selections are scraped, cleaned and placed into structured
and
unstructured (respectively) databases for storage.
Meanwhile, the Sociolinguistics Engine performs continuous corpus collection
and
aggregation, including but not limited to social media posts, forum posts, Q&A
site
replies, internet comments, advertisement copy, instructional copy, works of
written or
spoken art, and legal text. If applicable, automated tagging commences, with
continuous
Data formatting, indexing, and processing / enriching, including but not
limited to,
storage and tracking of beliefs, opinions, emotions, and stances on and
triggered by a
61
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
range of themes and topics for a given individual or group, in continuous
sentiment
analysis deployment. Data then undergoes indexing for future searches.
DI. Next, the administrator receives the options to deploy sentiment
analysis on the dataset,
allowing the data to be processed for emotion, sentiment, belief, and opinion
by
employing semantic analytical tools and methods, including, but not limited
to: neural
network, random forest algorithm, clustering algorithm, principal component
algorithm,
matrix factorization algorithm, Bayes classification algorithm, rule-based
engine, search
engine.
IV. Once automatic tagging is completed, users can access a graphical user
interface on
personal computers, smartphone, tablet, or mobile handheld device, to create
and
administer surveys, create new data, or tag stored data through surveys.
V. Users may access a graphical user interface to create new cognitive-
linguistic component
tagging rules that can be applied to existing data and future data.
VI. Users can access a graphical user interface to create query-driven
Custom Views on the
data, to answer research questions, including but not limited to academic and
scientific
research questions, business intelligence, marketing intelligence, stock
market analysis,
insights for political campaign.
VII. Finally, surveys, tagged data, and new user-generated corpora and sub-
corpora, can be
incorporated in the data store, to allow new queries to be performed on it.
VIII. Similarly, new, user-created automated tagging rules, query-driven
Custom Views, visual
representations of data, and machine learning models, and their tuneable
thresholds,
parameters and hyperpararneters can be incorporated into the Automated Tagging
Engine, to tag and index new incoming data for future access, querying.
IX. The two previous steps represent an inclusion of data-feedback loops
for improvement of
data analysis statistical and machine learning performance.
X. A graphical user interface allows exporting of selected datasets, with
select features,
components and parameters for analysis.
XI. Contribution of data to an open-source data storehouse upon publication
or project
completion.
62
CA 03207044 2023-7-31
WO 2022/183138
PCT/US2022/020994
[311] EXEMPLARY EMBODIMENTS
[312] A sociology professor wants to conduct a study of words for disease
used over time in
online discourse since 2002.
[313] Previous methods would primarily allow results of top word, phrase or
topics to be
calculated, marked in the data and reported. FIG. 28 is a system diagram of
user interface
features for sociolinguistic data. FIG. 29 is a flow diagram of a
sociolinguistic engine pipeline.
The sociolinguistics pipeline is novel and non-obvious for its ability to take
free-form
spontaneous expression and turn it into quantifiable, calculable data.
[314] Previously, qualitative researchers were primarily able to focus on
impressions that
were both subjective and aggregate. The datasets could only be parsed by
metadata, such as age,
cancer stage, location, or marital status.
[315] The sociolinguistics pipeline is also novel for its ability to
facilitate correlations,
operations, machine learning statistical, data- analytical, class-based or
predictive classifications
from partially or purely qualitative data.
[316] The professor inputs URLs into the engine's Corpus builder, which
aggregates,
normalizes, and merges scraped data into a single textual dataset.
[317] Next, the user imports audio files collected by the sociology
department the last 5
years by students with tape recorders.
[318] The audio files are transcribed into text and added to the meta-
corpus. Each
subcorpus includes a field that distinguishes its source.
[319] The professor enters queries with the purpose of analyzing historical
trends in words
for disease.
63
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[320] Audio files are automatically tagged for responses to open-ended
questions that
previously could not have been quantified, such as:
[321] "Could you tell me a bit more about what happened then?"
[322] "How do cope with that?"
[323] "Did you get support from anyone at the time?"
[324] "What do you think you learned from this experience?"
[325] "How have you applied what you've learned to another life situation?"
[326] For each of these questions, the sociolinguistic pipeline can now
quantify dimensions
(as well as their respective subcomponents), such as but not limited to:
[327] Emotions
[328] Issue Stances
[329] Personality
[330] Beliefs
[331] Perspectives
[332] Sentiment
[333] Perception
[334] Opinions
[335] The professor and her team conducts 8 90-minute audio-recorded
interviews over the
course of 2 years, with patients and their primary caregivers, including 4
interviews throughout
the chemo process, with a follow-up once chemo is completed.
64
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[336] Audio from remaining interviews is transcribed and ingested into the
sociolinguistic
pipeline, where the PI can simply upload the audio interviews using the data-
preprocessing tools.
[337] The professor receives an email notification once the files are
processed and
transcribed into text.
[338] The pipeline's Data Import Wizard allows researchers to declare names
of relevant
fields, input metadata, or merge datasets.
[339] The professor can then run the data through linguistic sentiment
NT,P, statistical or
semantic tools, such as, but not limited to: tokenization (receiving top terms
words and phrases
used by cancer patients in the interviews), collocations to discover important
terms surrounding
terms queried from the interviews, correlations, emotional scores to track
feeling, opinion, or
belief or stance at each interview checkpoint and/or follow-up.
[340] The professor can opt to run comparison metrics to compare language
and/or
sentiment used by patients with respect to time, geographical location or
other metadata, and/or
receive measures of centrality for the dataset and further statistical
interventions, measures and
points of correlation (which otherwise would be unavailable for qualitative
interviews).
[341] Additionally, the professor can make conclusionary comparisons with
similar studies
of cancer patients undergoing different or similar treatments, unify
qualitative interview datasets
with post-interview follow-up surveys in written form, export datasets for
further analysis in
statistical programs, perform machine learning (groupings, clustering users,
predict well-being,
assessing which features matter for well-being.)
[342] If the study generates new data, the professor can train machine
learning algorithms
of their choice on the new data, and/or contribute it back to the Data Store
for use by others.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[343] If the professor wishes, they can study or discover patterns in the
data, for further
conclusions.
[344] Additionally, patients can be clustered, cases can undergo a T-test
or other measures
for anomaly detection.
[345] The professor can opt to use further tools to highlight significant
features, such as but
not limited to Random Forest.
[346] Using machine learning, the professor can detect cancer patients at
lowest well-being
levels, their associated features, and/or those at the highest risk for
suicide.
[347] Once the research paper has been published, the professor can
contribute her datasets
to a general knowledge pool via the Data Store, approve use of them in select
or entire
conditions, and point to her research papers for categorization and use by
other researchers.
[348] While certain embodiments have been described, these embodiments have
been presented
by way of example only, and are not intended to limit the scope of the present
disclosures.
Indeed, the novel methods, apparatuses and systems described herein can be
embodied in a
variety of other forms; furthermore, various omissions, substitutions and
changes in the form of
the methods, apparatuses and systems described herein can be made without
departing from the
spirit of the present disclosures. The accompanying claims and their
equivalents are intended to
cover such forms or modifications as would fall within the scope and spirit of
the present
disclosure. For example, this technology may be structured for cloud computing
whereby a
single function is shared and processed in collaboration among a plurality of
apparatuses via a
network.
[349] SEMIOTIC LINGUISTIC POLITICAL ANALYSIS
[350] PROBLEM
66
CA 03207094 2023- 7-31
WO 2022/183138
PCT/US2022/020994
[351] No sentiment analysis system exists that can enumerate political stance
with reliable
accuracy, including political party affiliation and its psycholinguistic
subcomponents.
Additionally, no semantic system exists that can capture subtextual fine-
grained signals of
political belief in transcribed, translated or authored text or (audio) speech
with inclusionary
semantic framing.
[352] No machine learning solution has captured these implicative and
connotative nuances of
political speech in granular, valence-based quantified format. These are
increasingly important
functions as difficult-to-iterate speech and text communications have marked
effect on the public
and interest in their prevention (such as the January 6, 2021 insurrection on
the US Capitol).
[353] Machine learning has not been applied to these cognitive linguistic
elements with
precision measurements of subcomponents of belief systems, valence of self and
others, or intra-
messaging discursive turns in ways that create robust personal profiles,
clusters, and
relationships of political stance and its sub-parts within communities,
parties, and people groups.
[354] Affect (emotion) detection both in the delivery of political
communications ¨ as well as
the induced and incited emotional reactions within the collective and
individual audience ¨ are
necessary for the protection of society, understanding of and preservation of
political discourse
and free speech, as well as the warding off of future potential political
violence.
[355] Furthermore, a normalized rating and classification set of systems are
needed in order to
further classify and detect other aspects of sentiment implied and connoted in
textual and spoken
political speech, such as but not limited to opinion, beliefs and stances.
[356] Attempts to measure and improve shifts in societal belief systems,
reduction of hate
crimes, reduction of political violence between parties, and other pro-social,
peacekeeping
67
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
outcomes require a set of systems for measuring online, cellular, digital, and
analog
communications, their implications, nuances and effects.
[357] Finally, measurement of positions along the political spectrum is
necessary to detect
political extremism or domestic or foreign terrorism.
[358] PRIOR APPROACHES
[359] Automated classification of political stance has lacked fine-grained,
enriched micro-
information about belief, opinion and feeling (sentiment). No single system
has been able to
reliably predict political stance shiftings, swing-voter type and
inclinations, and likelihoods to
change opinions and parties, all at once.
[360] Furthermore, theoretical cognitive linguistic elements which can capture
nuances of
political beliefs have not been reliably quantified in a normalized way that
allows for comparison
points across sectors of societies.
[361] In the wake of political interferences by foreign entities, such a
system is needed.
Additionally, the use of political language, video and speech in ways that
have marked emotional
and cognitive effects on citizens have risen to public attention in recent
months and years. A
system for detection of nuanced messages and effects upon audiences during and
after political
communication is made is sorely needed.
[362] Furthermore, the rise of subcultures such as white supremacists and
other extremist
groups on respective sides of the political spectrum, as well as their
societal influence and rising
violence, requires that a set of systems for quantifying, identifying and
predicting emotional
escalation and rises towards political violence is necessary.
[363] Prior solutions to this problem have been able to predict violence
itself, but not tie it to
implicature and connotation in specific elements of communication.
68
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[364] Previous solutions to this problem have not addressed sub-elements of
cognition,
linguistics, emotion, intellectual processes and assumptions, as well as micro-
placements across
the political spectrum beyond the major parties.
[365] Classification of opinion, belief, sentiment and emotions in this area
is vital for a peaceful
society, threat detection, public discourse, incitement prevention, and
insight into political
audiences as applied to both human and computing arenas.
[366] METHODS
[367] Ingestion of websites or otherwise digitized text, transcribed audio,
and their linguistic
preprocessing and subsequent linguistic analytical measurements to aggregate
multi-dimensional
scores, valence, polarity and deixis.
[368] Conversion of multi-dimensional author, speaker, party, and news media
or content
source to a single score Political Lean Score (PLS), with sub-dimensions for
attitudinal readings
with respects to governmental, constitutional, ideological and societal issue
tracking
[369] Continuous updates of content and media scores as new datalines are
ingested.
[370] Detection and classification of datalines in regards to Swing Voter
Likelihood Scores
(SVLS), measuring party loyalty thresholds, rates of variance in sequential
stances over time or
political event communications, as well as the degree to which they change
with respect to time
or event, and severity and intensity of the event.
[371] Classes of swing voter that can be detected, according to central cause
and major factors,
for their political ambivalence, openness, flexibility, impressionability,
malleability and motion
along political spectrum and related sub-axes.
69
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[372] Ability to be leveraged in national security efforts, human-robot
interactions, educational
applications and experiments, emotional modeling for social robots, and other
affect detection
security interactions
[373] APS3 score measures loci such as power, control, reform, directionality,
nature, action
and status relations as represented in linguistically computed constructions
and strings.
[374] Ability to integrate and perform translation to robot-readable
measurements of social
psychology, including group dynamics, intergroup relations, prejudice or
stereotyping, and
organizational psychology.
[375] Computed and transmitted integrations of cognitive emotive linguistic
elements such as,
but not limited to, Power valences, Enacted Virtue Score (EVS), Comparative
Value Score
(CVS), Auto-Positioning Social Subcomponent Score (APS3) and their respective
subdimensions
[376] Measurements of feature components in multidimensional space,
resistance, religion,
locality score, ironic construction, rhetorical device type, speech acts,
dynamic psycholinguistic
state change.
[377] User attribute scorings and valence derivations on subcomponents such as
bravery,
curiosity, authenticity, vulnerability, emotional quotient score (EQS),
compassion, judgment, and
leamability as they relate to political locations and positions.
[378] FEATURES OF THE MODEL
[379] Profiles of prototypical users of a given cluster, collectively and
individually, to identify
micro-divisions of political stance and position along associated sub-axes.
[380] Inclusion of micro-labels and sub-axes for political stance, breaking
down political and
party affiliation into fine-grained sentiment-based sub-parts, with opinions
on but not limited to
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
financial, moral, social assistance, economic, governmental reach, civil
liberties, constitutional
position, and other political sectors of sentiment able to be parsed,
collected, and arranged for
mathematical, statistical and vector-based classification.
[381] Theoretical components of cognitive linguistics, including but not
limited to framing,
conceptual metaphor, literary device, metaphor, conceptual blending, mental
spaces, lexical
semantics, frame semantics, force dynamics, and other cognitive linguistic
subcomponents have
been measured and incorporated, where they previously have not in a unified
set of systems that
measures subcomponents for quantification and computation of these emotive
effects through
language.
[382] Incorporation of theoretical cognitive linguistic elements centered on
meaning, applied to
the inputted data in quantifiable form, via conversion to statistical and
machine learning
elements, labels, matrices and inputs, to capture rich semantic political
information about stance,
political lean and belief.
[383] Heuristics for flagging social media comments in response to audio (such
as podcasts),
visual (such as videos and imagery), and textual or speech (including but not
limited to speeches,
commentaries, interviews, news stories, comments, posts, e-mails, and other
inputs) that do not
agree with the observed or witnessed posted content.
[384] Our model has the singular ability to combine machine learning
(supervised and
unsupervised learning) with a rule-based set of systems that provides more
empirical
classification of sentiment, emotion, belief, and opinion, stronger profiling
of the parties using
language, more effective classification, and more flexibility around datasets.
71
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[385] Use of meta-information, synchronized streams, model deployment,
partitioning, baseline
comparisons against perceptual results, and valence computations to detect
political sector and
emotive reasoning.
[386] TECHNICAL DETAILS
[387] Gathering ability for labelled data is done by using heuristics,
filtering, location-based
and rule-based proxies for political affiliation, and stance on a predefined
list of sub-axes,
themes, topics and lexical frames.
[388] Gathering integration for signature n-grams associated to a person,
website, location, or
period of time, by statistical and machine learning approaches, including but
not limited to: n-
gram count and measure of specificity to said person, website, location or
period of time in
comparison to other such entities.
[389] The systems start by identifying characteristic sub-axes, themes, topics
and lexical fields
and frames specific to a particular politicians, public figures, through text-
mining of top n-grams,
and topic modelling of user-generated data in various forms, including but not
limited to: tweets,
transcribed speech.
[390] These systems also can identify characteristic sub-axes, themes, topics
and lexical fields
and frames specific to particular social media user, through text-mining of
top n-grams, and topic
modelling of user-generated data in various forms, including but not limited
to: tweets, reddit
posts, transcribed speech from social media interactions, photos, liked
content, and audio if
available.
[391] These systems also can identify characteristic sub-axes, themes, topics
and lexical fields
and frames specific to particular social media outlet, through text-mining of
top n-grams, and
topic modelling of user-generated data in various forms, including but not
limited to: articles,
72
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
posts, transcribed speech form videos, social media interactions for the
official outlet account,
photos, liked content, and audio if available.
[392] These systems can then define a final list of issues on which to measure
users' stance,
based sub-axes on pre-defined and discovered through statistics and machine
learning.
[393] These systems can then identify user stance on each issue and sub-axis
with a certainty
score, using machine learning including but not limited to the following
approaches: rule-based
indicators such as viral signature n-gram adoption, dominant theme
identification through word,
n-gram, skipgram and part-of-speech (POS) gram co-occurrence and specialized
collocations,
Bayesian classifiers, collaborative filtering, matrix factorization,
clustering algorithm, LSTM
neural network, fine-tuned Transformer neural network, identified through text-
mining and topic
modelling. These systems then associate this data to mined user-metadata,
which can indicate
age, ethnic group, socioeconomic status.
[394] These systems can then predict user-similarity, based on stance-
similarity on the elicited
list of sub-axes, using machine learning approaches, including but not limited
to: neural network,
Bayes classifier, k-nearest neighbor algorithm, clustering algorithm, logistic
regression, SVM,
random forest.
[395] These systems can then predict political affiliation, using user-stances
as input features,
with machine learning approaches, including but not limited to rule-based
indicators, Bayesian
classifiers, LSTM neural networks, fine-tuned Transformer neural networks
trained on political
data and public response.
[396] Gathering of user information can be integrated regarding discovered and
pre-defined
dimensions, including but not limited to: compassion, stubbornness, likelihood
of mutability of
political stance, tendency towards violence, etc.
73
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[397] These steps can then be repeated for any specific point in time, and
analysis of changes in
the data allows to track of political movement and identify trends in
political stances over time,
for a single user, a specific group of users, or on the dataset of all users
as a whole.
[398] By cross-referencing user-generated data with exposure to propaganda,
advertisement,
and other identified sources of influence, these systems can quantify the
degree of change in
stance on any identified sub-axis.
[399] Results of analyses for specific user and time period can be visualized
in a GUI,
including but not limited to: website, browser plugin, smartphone app.
[400] These systems can also be applied to assess the stance of a user on a
single or restricted
data point, such as a single post, video, or any form of incoming text, audio,
video.
[401] PREJUDICE, DISCRIMINATION AND BIAS DETECTION
[402] These methods are unique for their recognition of covert and overt bias,
with granular
linguistic features specifically extracted from and applied to prejudice and
discrimination-based
expressions in natural language, which can be applied and fine-tuned to
instances of racism,
sexism, ageism, homophobia, transphobia, xenophobia, sexual harassment,
classism and ableism
in spontaneous or crafted speech or text, by private and public figures and
entities, nonprofits,
publications, vvebsites and other media sources and their mediums of
communication.
[403] PROBLEM
[404] Detection of discrimination against, prejudices held against, and
otherwise negative or
biased sentiments held in regards towards a given people group, protected
class, minority,
disempowered or disadvantaged population have not easily been unifiedly
quantified in
computational linguistics. No formalized system has been put forth that can be
applied across
time, texts and biases, reliably, as detection as of this writing remains
overly dataset specific.
74
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[405] Additionally, as sociolinguists, sociologists, psychologists, and ethic,
women's, and
queer studies (and their equivalent humanities and science studies) have
examined language up-
close, few reliable sets of rules and cues have been developed as a standard
for detecting racial
bias.
[406] As a result, there has been no unified measure against which to assess
natural language or
spontaneous speech. Because of this, a consistent, relative degree of bias and
prejudice has not
been established, making it difficult to exert accountability and promote more
rapid social
change.
[407] This has created a society where minority suffering is "invisible," with
special
populations effectively gaslit by lack of definitive, unified proof in the
language. This effect has
also made it difficult for organizations and corporations to measure their
performance and
responsiveness as collective entities when it comes to implicit bias.
[408] Finally, real-world outcomes need set of systems of measure to be
correlated with from a
lens of implicit bias and discrimination, so that public and private
individuals, entities and
systems can be held accountable, assessed and improved (as well as
interventions designed for
improvement of such systems, individual communications, and entities'
performance in equitable
treatment). Society requires a solution for bias prediction and degree, as
handled by these
systems, to identify, prevent and detect probability of individual or mass
violence from or against
a people group, in general, as well as at any given time or location.
[409] Subtle forms of bias, such as "othering" (treating minorities and
oppressed classes as if
they are outside the norm of whiteness, heteronormativity, cisgenderedness, or
other majority
status), "white knighting" (treating minorities or oppressed classes as if
they need to be rescued,
with the majority status members centered as saviors), or "microaggressions"
(small, hidden, and
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
easy-to-miss slights frequently unconscious but collectively bringing
psychological harm) are
difficult to enumerate, anticipate, and quantify.
[410] As such, there have been no standardized methods for quantification of
bias and
discrimination until now.
[411] FEATURES
[412] Computational efficiency of discrimination detection due to
implementations of indices
in all integrated algorithms herein.
[413] 1figh clarity and precision due to time- and text-slicing techniques to
pinpoint emotional
and bias reactions during interviews, observations and communications inputted
into the
systems.
1414] Further computational efficiency of accessing results (query speed), due
to
implementation of indices in the algorithms included.
[415] Integration of multi-sensory inputs into a single bias-detection systems
that outputs a set
of normalized scores and a Bias Master Score indicating the presence and
degree of prejudice,
bias, discrimination or uneven favorability in perception and action of a
given input.
[416] Ability to tie discriminatory and bias ratings and inputs to real-world
outcomes, including
but not limited to judicial decisions, fair housing assessment, police
brutality allegations,
employment discrimination, media bias, persuasion and marketing outcomes,
commercial ROI
(results on impression, both literally on web, tablet and smart phone
entities, as well as other
conversions from prospects to sales on other media), and social norm
assessments.
[417] Dexterity in assessing the implicit bias of observers in witnessing and
internally
processing multi-sensory data, as well as the implicit bias resident in the
media consumed itself,
76
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
and the actors therein, allowing for correlations and statistical connections
to be made between
the two.
[418] Fine-tunable applications that can be fitted to parameters for
discriminatory orations and
linguistic expressions, and applied beyond traditional classifications of
bias, to fine-timed
features for intersexism bias, national origin discrimination, pregnancy
discrimination, religious
discrimination, parental or partner status and socioenconomic class.
[419] Use of meta-data and computational linguistic assessments together with
other sensory
inputs, to create profiles of geographical, socioeconomic, political and
social stance, age-related,
and other features for prediction of entities and individuals likely to carry
bias, for effective
interventions.
[420] Ability to monitor interventions against a timeline and location track,
for adjustment,
improvement and grant purposes.
[421] Base State Detection (BSD) markers, allowing for automated data line
identifiers copied
to the data store and sorted according to markers instantiated for gender,
race, socioeconomic
class, sexual orientation, physical degree of ability, and incorporative
dimensions for poverty or
socioeconomic index conversion.
[422] Computation of comparative signals between data with BSD marker
attachment and
incoming data flow, allowing the Al to improve efficiency in processing time
in subtextual
channels, garnering reduction in both training rounds and systems' resources.
[423] METHOD
[424] These systems of bias and prejudice detection quantifies discriminatory
stances and
negative attitudes towards protected, minority, disadvantaged, or particular
classes of people.
These systems elucidate the detection of discriminatory language via detection
and measurement
77
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
of subtextual linguistic cues by feature presence and placement within the
sentence and by
measuring instance(s) (of range [0, id); intrasentence relative position
(IRP); a plurality of
syntactic sequence, fuzzy or exact string matching or a combination thereof;
microdimensional
comparison (MDCs) of any type the following categories of linguistic
constructions and
distinguished sociolinguistic features and their associated points of
measurements:
1. Minimization
2. Scapegoating
3. Issue Reframing as Non-prejudicial
4. Dismissals
5. Otherizing
6. Fear-based Descriptors
7. Threat-based language
8. Accusations of lawlessness
9. Discriminatory denials
10. Animal language
11. Deflective language
12. Strawmanning and hyperbolic mockery
13. Justifications of bias
14. Invalidation of suffering (
15. Extremist framing
16. Insubordination language
17. Victim-blaming
18. Silencing
19. Overt threats
20. Dominance techniques
21. H:onorific insistence
22. Shoulding
23. Defensive techniques
24. Patronizing Explanation Formats (mansplaining, whitesplaining, etc.)
25. Disqualification of victims
26. Stereotyping
27. Exclusive language
28. Class-based language
29. Virtue-questioning
30. Class-based Character-judgments
31. Smear campaigns
32. Double-standards
33. Linguistic mimicry
34. Erasure and posturing
78
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
35. Exoticizing
36. Pre-equality nostalgia
37. Micro-insults
38. Majority-trait perception praise
39. Loaded questions
40. Model minority praise
41. Paternali zing
42. Appropriate language
43. Criminalization Language
44. Moralizing
45. Dehumanizing language
46. Purity speech
47. Intrusion/Invasion/Takeover language
48. Fraud/Social Service Abuse allegations
49. Allegations of Societal Spoiling
50. Assimilation complaints
51. Intelligence language
52. Claims of prejudice as honesty / realism
53 Conservation Terminators
54. Awareness as Special Treatment fallacy
55. Prejudiced prejudice-denial preambles / Qualifiers before a racist
comments
56 Prejudice cloaked as humor
57. Using targets for social validation
58. Invalidation of caused trauma
59. (Faulty / flattening) Generalization ("one of those," "someone like that")
60. Indefinite pronoun of the demonstrative pronouns
61. Gaslighting
62. Dehumanization
63. Normalization of Bias
64. Bandwagon Fallacy
65. Instructions to Ignore Bigotry
66. Allegations of minority status as leverage / privilege
67. Bootstraps mythologies
68. Freudian Slip justifications
69. Trivialization
70. Racialization
71. Restorative Justice as Double-Standard Assertions
72. Backhanded compliments
73. Menacing Minority language
74. Single data point fallacy
75. Dominating language / Assertions of Power
76. Reminders of Lower Societal Status
77. Victims as agitators (claim of deserved harm/force)
79
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
78. Claims of uncivilization
79. Infantilizing language
80. Diminutives
81. Objectifying
82. Sexualization
83. Fetishization
84. Majority-Oriented language
85. Accusations of Dishonesty
86. Accusations of Lack of Virtue
87. Justification by Token Presence or Success
88. Assumption of Majority-race, Majority-ethnicity, Gender in Power, Majority-
Sexual-
Orientation, Able / non-Disabled, or otherwise societally privileged
advantages
89. Semantic Frame-implied slights or accusations
90. Implicit Oppressor Education Demands
[425] EIVEBODEvIENT
[426] These elements are distinguished amid the language by ensembling one or
more of the
following linguistic features:
1. Discover discriminative and prejudicial linguistic markers, of 0 toil,
from the feature list,
where each linguistic marker represents a second linguistic embedding of
cognitive
component measurements (CCMs) that contribute to the overall bias score and
discriminatory label.
2. Available enumeration of skip grams of phrases in the corpus that may
highlight
differences between treatment of people groups, as well as computed cognitive
linguistic
features associated with prejudice, bias, or discriminatory viewpoints or
practices towards
a given people group, minority, or protected class, according to their
presence or absence,
intensity and degree.
3. Optional assembly of a corpus of sound, images and videos, as well as
social media data
and metadata, to be observed by human beings as well as analyzed and rated by
machine
learning algorithms, to assess implicit bias, discrimination, and prejudice
through
microexpression analysis, sound wave analysis and computation in all of its
aspects, as
well as speech transcription, from authorized access, uploads, live video,
audio or camera
capture.
4. Additional optional capture of input from observers themselves, through
images, video,
audio, written or transcribed or translated text, survey data, interviews, or
comments from
live, website or social media interactions as well as any corresponding
metadata.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
5. Perform natural language package deployment where indicated on the text
corpus,
generating part of speech (POS) tags for each word in context in the corpus,
and creation
of mappings of each word to its corresponding part of speech.
6. If appropriate, deploy further linguistic enumeration, notation and
tagging, including but
not limited to structural dependencies, and collocations.
7. Identification and preprocessing of any facial images, including but not
limited to:
semantic segmentation, reframing and normalization.
8. Identification and preprocessing of relevant, timed, user-produced audio
snippets which
can indicate user emotion, intent, opinion.
9. If applicable to the input, train selected models on curated corpora or
prototypical texts
that display bias, discrimination, prejudice, favoritism against, a given
people group,
protected class or minority, as well as on transcribed, translated, inputted
or scraped
natural language texts, gesture inputs, facial expressions or spontaneous
speech.
10. If applicable to the input type, train computer vision emotion detection
models on
gathered, and preprocessed facial data.
11. If applicable to the input, Train audio emotion-recognition models on
gathered, identified
and preprocessed sound, speech produced by the user.
12. Optionally use facial emotion prediction to predict an emotion label for
the image(s), and
associate that label to existing text for the user. This enables additional
elicited data to tag
the images and text data with the level of believability, trust, guilt-
assumption, truth-
assumption and favorability of their responses.
13. Optionally use sound emotion prediction to predict an emotion label for
the image(s), and
associate that label to existing text for the user. This enables additional
elicited data to tag
the sound and text data with the level of believability, trust, guilt-
assumption, truth-
assumption and favorability of their responses.
14. Enumerate part-of-speech grams (POS-grams) that potentially highlight
differences
between treatment of people groups.
15. List and examine frequency for POS-grams and skip-grams resident in the
data.
16. Using one or more statistical approaches, discover skip grams and part-of-
speech grams
that potentially highlight or emphasize differences between treatment of
people groups.
17. As necessary, augment the data using human- or machine-generated word
lists including
an array of sociolinguistic indicators of bias in a plurality of contexts
towards a given
81
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
people group, control group or the general population.
18. Apply data augmentation methods as appropriate to generate similar
synograms
(semantically similar n-grams) to use as features.
19. Augmentation of frequent n-grams as required by replacing each word with
an array
consisting of respective words or sequence of words that have similar semantic
meaning,
valence, sentiment label, sentiment cluster, semantic association or polarity
to the
identified word.
20. Expand any augmentation of the list of frequent n-grams with k-nearest
neighbors in the
Word2Vec embedding space, additionally sorting antonyms and synonyms from the
list,
respectively.
21. For indicated cases, compare skip-grams differentiated by two or more
terms or words
that respectively have a significant difference, distance and directionality
in polarity,
sentiment, valence, cluster, or semantic association or meaning.
22. Detect the presence of enumerated and augmented features in the inputted,
corpus-based,
scraped, translated or transcribed text or speech, distinguishing features
that are
indicators of bias.
23. Compute scores for data line for each applicable label based on the
features detected,
including but not limited to text facial image features.
24. Train machine learning methods on any augmented data and selected features
from inputs
including but not limited to 1.,STM, transformers, CNN, KNN, clustering,
random forest,
linear regression, and Bayesian probability to quantify bias.
25. Use image and text data to output and grade Implicit Bias, Discrimination
Score,
Prejudice Score, Believability score, Truth score, Guilt-Assumption Score, and
Favorability score.
26. Use the ensuing trained model to predict bias, discrimination, or
prejudice in any video,
image or sound wave file associated with given speech or text input.
27. Assessment of audio, text, computer vision reactions to similar or same
communication
by different and distinct people groups by analyzing ratios, points in time or
passage
demarcations, or number of words, of favorable, unfavorable and neutral
facial, textual,
or auditory expressions and reactions (as well as their subdimensions and
specific
emotions) with regards to other people group communication, gesture, images,
voices, or
profile data.
[427] The above disclosure also encompasses the embodiments listed below.
82
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[428] (1) A method for automatically augmenting natural language content with
emotio-
cognition by processing circuitry. The method including receiving, via an
input device, the
natural language content as a textual input; searching, by the processing
circuitry, for matches
between a plurality of linguistic rules for a given emotio-cognition and
components of the textual
input, wherein instances of the linguistic rules have at least one human
dimension; activating, by
the processing circuitry, the matched linguistic rules, and evaluating the at
least one human
dimension of the activated matched linguistic rules; scoring, by the
processing circuitry, each
human dimension to obtain a prototypical profile of dimension scores for the
given emotio-
cognition; aggregating, by the processing circuitry, the dimensions in the
obtained profile of
dimension scores to obtain an intensity indication for the given emotio-
cognition; and displaying,
by a display, the natural language content augmented in a manner that relates
the matched
linguistic rules to the given emotio-cognition and signals the respective
intensity indication of
the given emotio-cognition.
[429] (2) The method of feature (1), in which the human dimensions include one
or more of
emotional affects of sentiment, emotion, Emotio-Cognitive attitudes, values,
social mores,
mindsets, outlooks, aspects, responses, traits, beliefs, opinions,
perspectives, motivations, biases,
states, manners, approaches, dynamics, personality trait, emotional approach,
emotional choice,
reaction, disposition, temporary state, change of state, cognitive aspect,
behavioral aspect,
internal condition, external condition, feeling, emotion, proposition,
attitude, propositional
attitude, directed attitude, undirected attitude, self-directed attitude,
conscious emotio-cognition,
unconscious emotio-cognition, anger, anticipation, disgust, fear, joy,
sadness, surprise, trust, ego,
blame, conformity, sacredness, kindness, respect, time, favor, approval,
sincerity, vulnerability,
judgment, separateness, purpose, formality, minimization, specificity, force,
action , agency,
83
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
curiosity, clarity, intention, emphasis, energy, certainty, interest,
engagement, shock or surprise,
tension, speed, nuance, logic, paranoia, distance, identification, esteem,
objectification,
attachment, empathy, and patience, in which each dimension has a value being
one of +1 for
positive force, -1 for negative force, 0 for neutral force, and 0 for not
present or not applicable,
and in which the scoring, by the processing circuitry, each human dimension
includes scoring
human dimensions for all matching rules.
[430] (3) The method of features (1) or (2), in which the step of searching
using the plurality of
linguistic rules further includes detecting constructions based on the
linguistic rules; and
evaluating the human dimensions of each detected construction.
[431] (4) The method of any of features (1) to (3), in which the step of
scoring including
comparing the intensity indication to thresholds for the given emotio-
cognition to obtain an
emotional intensity level.
[432] (5) The method of feature (3), in which the step of detecting
constructions further
includes detecting a presence or an absence of constructions in the natural
language content
having components related to the given emotio-cognition.
[433] (6) The method of any of features (1) to (5), further including
determining, by the
processing circuitry, a pattern of emotio-cognitions that includes the given
emotio-cognition by
concatenating with other emotio-cognitions detected by other linguistic rules
and identifying the
pattern of emotio-cognitions as a dynamic emotio-cognition; and tracking the
given emotio-
cognition and the other emotio-cognitions together with associated components
in a temporal
sequence over the natural language content.
[434] (7) The method of feature (3), in which the step of detecting the
constructions further
comprises determining a numeric value for one or more of a part of speech
tagging or syntax
84
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
rule, a string matching rule that is exact, inexact, masked, or wildcarded, a
token proximity rule,
a punctuation rule, a lemmatization rule, a stemming rule, a lexicon rule, and
a word lookup or
dictionary-based rule.
[435] (8) The method of feature (7), in which the step of determining the
numeric value for the
token proximity rule comprises accessing all tokens having a distance of fewer
than n tokens
from a specified point in the natural language content, wherein n is a
positive integer.
[436] (9) The method of any of features (1 )to (8), further including
generating new linguistic
rules by a machine learning engine that performs at least one of supervised
learning and
unsupervised learning.
[437] (10) The method of feature (9), further including receiving a plurality
of natural language
data items from a repository; normalizing and tokenizing the received
plurality of natural
language data items using the preprocessing to generate a plurality of
preprocessed natural
language data items; labeling the plurality of preprocessed natural language
data items with an
expressed emotio-cognition and an intensity of the expressed emotio-cognition;
providing, in
parallel, the plurality of preprocessed natural language data items to an
unsupervised learning
engine and a supervised learning engine; training, in parallel, the
unsupervised learning engine
and the supervised learning engine in multiple training epochs to identify, in
the natural language
data, a particular emotio-cognition, and to determine an intensity of the
particular emotio-
cognition, wherein each training epoch of the unsupervised learning engine
provides rule
suggestions to subsequent training epochs of the rule-based engine, and each
training epoch, the
rule-based engine provides tabulation and scoring data to subsequent epochs of
the unsupervised
learning engine and the supervised learning engine; and providing an output
representing at least
one of the trained unsupervised learning engine and the trained supervised
learning engine.
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[438] (11) The method of any of features (1) to (10), further including
generating new linguistic
rules by the processing circuitry that performs matching human dimensions
present within the
natural language content by matching the human dimensions to existing
dimensional arrays,
having wildcards or pattern skips, to identify new rules for the rule-based
engine.
[439] (12) The method of any of features (1) to (11), in which the receiving
step further
comprises continuous reading of a streaming live video or animated video
source together with
coordinated textual transcription, and the method further including
determining contextual clues
based on word co-occurrence, discursive elements, and topic elements; marking
individual
strings or n-grams with trinary dimensional scores; detecting and entering
further information
apparent in visual data or tone elements apparent in auditory data into a
separate, but time-
coordinated, source for the video; and performing juxtaposition from the
contextual clues and
further information, to create context scores for each scene in the video
[440] (13) The method of feature (12), in which the displaying step includes
displaying the
textual transcription in a manner that the given emotio-cognition and
respective intensity
indication is bracketed and inserted inline adjacent to the components.
[441] (14) The method of any of features (1) to (13), further including
generating new linguistic
rules by a rules- discovery engine, and in which the method further includes
detecting, by the
processing circuitry, hook words or pairs of words in the natural language
content; evaluating
one or more human dimensions associated with the detected hook word or pairs
of words to
determine if the hook words or pairs of words indicate a possible emotio-
cognition; when a
possible emotio-cognition exists, extracting a predetermined window of words
surrounding the
hook words or pairs of words; scoring, by the processing circuitry, the one or
more human
dimensions to obtain a profile of dimension score for the hook word or pairs
of words; and when
86
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
the profile of dimension score is above a majority, constructing a new rule
for the possible
emotio-cognition based on the hook word or pairs of words and extracted
surrounding words.
[442] (15) The method of any of features (I) to (14), further including
identifying index
positions in the textual input at positions where linguistic rules are
matched.
[443] (16) The method of feature (15), further including annotating the
textual input with an
emotio-cognition and a respective intensity indication at the index positons.
[444] (17) The method of any of features (1) to (16), in which the receiving
step further
comprises receiving, via the input device, the natural language content as a
audio input and
transcribing the audio input into text input, the method further including
matching a fragment of
the audio input with a stored rule for a similar sound fragment, and assigning
the audio fragment
with an emotio-cognitive label of the stored rule.
[445] (18) An electronic reader, including a touchscreen display; processing
circuitry; and a
memory, in which the touchscreen display is configured to display text of an
electronic book; the
processing circuitry is configured to scan and tag the text using rules that,
upon being triggered,
detect emotio-cognitive states, and determine intensity with which the emotio-
cognitive states
have occurred; the processing circuitry is configured to generate and display
one or more
sidebars for listing dynamics and emotio-cognition-intensity information based
on detected
components of the displayed text; the touchscreen, when touched at a position
in the display, is
configured to select a dynamic or emotio-cognition-intensity; and the
processing circuitry is
further configured to generate and display color-coded highlighting that
designates an occurrence
of the selected dynamic or emotio-cognition-intensity.
[446] (19) A system for mitigating a psychological disorder, including a
mobile device having
processing circuitry and memory, and a peripheral device having a
communications device and
87
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
one or more actuators, in which the memory of the mobile device stores program
instructions,
which when executed by the processing circuitry of the mobile device, cause
the mobile device
to perform a method including receiving, via an input device, natural language
content as a
textual input; searching, by the processing circuitry, for matches between a
plurality of linguistic
rules for a given emotio-cognition and components of the textual input,
wherein instances of the
linguistic rules have at least one human dimension; detecting, by the
processing circuitry, the
matched linguistic rules to obtain an intensity indication for the given
emotio-cognition; and
when the intensity indication for the given emotio-cognition reaches an emotio-
cognitional
intensity that exceeds a first threshold, transmitting a first activation
signal that identifies the
emotio-cognitional intensity; and the peripheral device is configured to
receive, via the
communications device, the transmitted first activation signal; and activate
the one or more
actuators to create a sensory distraction to mitigate the psychological
disorder.
[447] (20) The system of feature (19), in which the program instructions,
which when executed
by the processing circuitry of the mobile device, further cause the mobile
device to perform the
method, including continuing to receive, via an input device, the natural
language content as a
further textual input; and when the intensity indication for the given emotio-
cognition reaches an
emotional intensity for a negative emotion that exceeds a second threshold,
transmitting a second
activation signal that identifies the emotional intensity for the negative
emotion; and the
peripheral device is further configured to receive the transmitted second
activation signal, and
activate the one or more actuators in order to create a different randomized
sensory distraction to
mitigate the personality disorder.
[448] (21) The method of feature (1), further including highlighting words in
the natural
language content based on the intensity indication; transmitting, the natural
language content
88
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
with the highlighted words to the display; and displaying the natural language
content with the
highlighted words as an augmented reality display on the display during a
course of a video
streaming session.
[449] (22) The electronic reader of feature (18), further including detecting
a presence or an
absence of constructions in the text having components related to the emotio-
cognitive states;
and displaying, when a user touches text displayed on a touchscreen,
representations of the
emotions and cognitions of the text, wherein the representations are color
heat maps.
[450] (23) The electronic reader of feature (18), further including annotating
the text with the
emotio-cognitive states and a respective intensity at index positons where
shown when the
electronic reader is touched during reading.
[451] (24) The system of feature (19), further including in response to the
detecting of the
matched linguistic rules, the mobile device is configured to transmit
electrical signals or short
radio waves, in order to trigger, based on the linguistic rules, a color-coded
lighting of the
peripheral device.
[452] (25) The system of feature (19), in which the peripheral device further
comprises a
colored, geometric display configured to activate an 11:7,I) according to the
intensity indication
for the given emotio-cognition.
[453] (26) The system of feature (19), in which the method performed by the
mobile device
further includes comparing the intensity indication to thresholds for the
given emotio-cognition
to obtain an emotional intensity level for a cognition-emotional state; and
the peripheral device
comprising color-emitting diodes and a vibrator, and is configured to
broadcast the cognition-
emotional state, via the color-emitting diodes, and vibrate, via the vibrator,
when the intensity
indication is over a threshold.
89
CA 03207044 2023- 7- 31
WO 2022/183138
PCT/US2022/020994
[454] (27) The system of feature (19), in which the method performed by the
mobile device
further includes determining a pattern of emotio-cognitions that includes the
given emotio-
cognition by concatenating with other emotio-cognitions detected by other
linguistic rules and
identifying the pattern of emotio-cognitions as a dynamic emotio-cognition;
and the peripheral
device comprising LED lights and a vibrating device that vibrates in
coordination with pulsing of
the LED lights, to shift as the emotio-cognitions shift.
[455] (28) The system of feature (19), in which the method performed by the
mobile device
further comprises identifying index positions in the textual input at
positions where linguistic
rules are matched during audio conversations received and transcribed when
spoken by a wearer
of the peripheral device.
CA 03207044 2023- 7- 31
