Note: Descriptions are shown in the official language in which they were submitted.
INSPECTION RISK ESTIMATION USING HISTORICAL INSPECTION DATA
BACKGROUND
[0001] Embodiments of the present disclosure relate to inspection risk
estimation, and more
specifically, to inspection risk estimation using historical inspection data.
BRIEF SUMMARY
[0002] According to embodiments of the present disclosure, methods of and
computer
program products for inspection risk estimation are provided. In various
embodiments,
attributes of a future inspection of a factory and historical data related to
the future inspection
are received. A plurality of features are extracted from the attributes of the
future inspection
and the historical data. The plurality of features are provided to a trained
classifier. A risk
score indicative of a probability of failure of the future inspection is
obtained from the trained
classifier.
[0003] In various embodiments, the historical data are preprocessed. In
various
embodiments, preprocessing the data comprises aggregating the historical data.
In various
embodiments, pre-processing the data further comprises filtering the data.
[0004] In various embodiments, the data further comprise performance history
of the factory.
In various embodiments, the data further comprise geographic information of
the factory. In
various embodiments, the data further comprise ground truth risk scores. In
various
embodiments, the data further comprise product data of the factory. In various
embodiments,
the data span a predetermined time window.
Page 1 of 40
CA 3050952 2019-07-31
[0006] In various embodiments, providing the plurality of features to the
trained classifier
comprises sending the plurality of features to a remote risk prediction
server, and
obtaining from the trained classifier a risk score comprises receiving a risk
score from the risk
prediction server.
[0007] In various embodiments, extracting the plurality of features comprises
removing
features with a low correlation to a target variable. In various embodiments,
extracting the
plurality of features comprises applying a dimensionality reduction algorithm.
In various
embodiments, extracting a plurality of features from the data comprises
applying an artificial
neural network. In various embodiments, applying the artificial neural network
comprises
receiving a first feature vector as input, and outputting a second feature
vector, the second
feature vector having a lower dimensionality than the first feature vector.
[0008] In various embodiments, the risk score is provided to a user. In
various embodiments,
providing the risk score to the user comprises sending the risk score to a
mobile or web
application. In various embodiments, said sending is performed via a wide area
network.
[0009] In various embodiments, the trained classifier comprises an artificial
neural network.
In various embodiments, the trained classifier comprises a support vector
machine. In various
embodiments, obtaining from the trained classifier a risk score comprises
applying a gradient
boosting algorithm.
[0010] In various embodiments, the risk score is related to the probability by
a linear
mapping.
[0011] In various embodiments, the performance of the trained classifier is
measured by
comparing the risk score to a ground truth risk score, and parameters of the
trained classifier
are optimized according to the performance. In various embodiments, optimizing
the
Page 2 of 40
CA 3050952 2019-07-31
parameters of the trained classifier comprises modifying hyperparameters of a
trained
machine learning model. In various embodiments, optimizing the parameters of
the trained
classifier comprises replacing a first machine learning algorithm with a
second machine
learning algorithm, the second machine learning algorithm comprising
hyperparameters
configured to improve the performance of the trained classifier.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
[0012] Fig. 1 is a schematic view of an exemplary system for inspection risk
estimation
according to embodiments of the present disclosure.
[0013] Fig. 2 illustrates a process for inspection risk estimation according
to embodiments of
the present disclosure.
[0014] Fig. 3 illustrates a process for training an inspection risk estimation
system according
to embodiments of the present disclosure.
[0015] Fig. 4 illustrates a process for updating an inspection risk estimation
system according
to embodiments of the present disclosure.
[0016] Fig. 5 illustrates a process for training an inspection risk estimation
system according
to embodiments of the present disclosure.
[0017] Fig. 6 illustrates a process for training an inspection risk estimation
system according
to embodiments of the present disclosure.
[0018] Fig. 7 illustrates a process for training an inspection risk estimation
system according
to embodiments of the present disclosure.
[0019] Fig. 8 depicts a computing node according to embodiments of the present
disclosure.
Page 3 of 40
CA 3050952 2019-07-31
DETAILED DESCRIPTION
[0020] Inspections commonly occur in factories in order to ensure quality
control and
adherence to protocol. Estimating the risk of failing a particular inspection
in advance of the
inspection date allows factories and their business partners the ability to
implement a dynamic
quality control program based on the estimated risk.
[0021] The present disclosure provides a framework for estimating the risk of
failure of an
inspection, prior to the inspection date, using historical inspection data and
machine learning
methods.
[0022] In embodiments of the present disclosure, inspection risk estimation is
performed by
obtaining data related to an inspection, extracting a plurality of features
from the data,
providing the features to a trained classifier, and obtaining from the trained
classifier a risk
score indicative of the probability that the inspection is likely to pass or
fail. In some
embodiments, a feature vector is generated and inputted into the trained
classifier, which in
some embodiments comprises a machine learning model.
[0023] In embodiments of the present disclosure, data may be obtained in a
variety of
formats. Data may be structured or unstructured, and may comprise information
stored in a
plurality of media. Data may be inputted manually into a computer, or may be
obtained
automatically from a file via a computer. It will be appreciated that a
variety of methods are
known for obtaining data via a computer, including, but not limited to,
parsing written
documents or text files using optical character recognition, text parsing
techniques (e.g.,
finding key/value pairs using regular expressions), and/or natural language
processing,
scraping web pages, and/or obtaining values for various measurements from a
database (e.g.,
a relational database), XML file, CSV file, or JSON object.
Page 4 of 40
CA 3050952 2019-07-31
[0024] In some embodiments, factory or inspection data may be obtained
directly from an
inspection management system, or other system comprising a database. In some
embodiments, the inspection management system is configured to store
information related to
factories and/or inspections. The inspection management system may collect and
store
various types of information related to factories and inspections, such as
information
pertaining to purchase orders, inspection bookings, assignments, reports,
corrective and
preventive action (CAPA), inspection results, and other data obtained during
inspections. It
will be appreciated that a large set of data may be available, and in some
embodiments, only a
subset of the available data is used for input into a prediction model. The
subset of data may
contain a sufficient number of attributes to successfully predict inspection
results.
[0024] As used herein, an inspection booking refers to a request for a future
inspection to take
place at a proposed date. The inspection booking may be initiated by a vendor,
brand, or
retailer, and may contain information of a purchase order corresponding to the
future
inspection. As used herein, an assignment refers to a confirmed inspection
booking. The
assignment may contain a confirmation of the proposed date of the inspection
booking, as
well as an identification of an assigned inspector and information related to
the booking.
[0025] Data may be obtained via a data pipeline that collects data from
various sources of
factory and inspection data. A data pipeline may be implemented via an
Application
Programming Interface (API) with permission to access and obtain desired data
and calculate
various features of the data. The API may be internally facing, e.g., it may
provide access to
internal databases containing factory or inspection data, or externally
facing, e.g., it may
provide access to factory or inspection data from external brands, retailers,
or factories. In
some embodiments, data are provided by entities wishing to obtain a prediction
result from a
Page 5 of 40
CA 3050952 2019-07-31
prediction model. The data provided may be input into the model in order to
obtain a
prediction result, and may also be stored to train and test various prediction
models.
[0026] The factory and inspection data may also be aggregated and statistical
analysis may be
performed on the data. According to embodiments of the present disclosure,
data may be
aggregated and analyzed in a variety of ways, including, but not limited to,
adding the values
for a given measurement over a given time window (e.g., 7 days, 14 days, 30
days, 60 days,
90 days, 180 days, or a year), obtaining the maximum and minimum values, mean,
median,
and mode for a distribution of values for a given measurement over a given
time window, and
obtaining measures of the prevalence of certain values or value ranges among
the data. For
any feature or measurement of the data, one can also measure the variance,
standard
deviation, skewness, kurtosis, hyperskewness, hypertailedness, and various
percentile values
(e.g., 5%, 10%, 25%, 50%, 75%, 90%, 95%, 99%) of the distribution of the
feature or
measurement over a given time window.
[0027] The data may also be filtered prior to aggregating or performing
statistical or
aggregated analyses. Data may be aggregated by certain characteristics, and
statistical
analysis may be performed on the subset of data bearing the characteristics.
For example, the
above metrics can be calculated for data related only to inspections that
passed or failed,
related to during product (DUPRO) inspections, or to inspections of above a
minimum sample
size.
[0028] Aggregation and statistical analysis may also be performed on data
resulting from
prior aggregation or statistical analysis. For example, the statistical values
of a given
measurement over a given time period may be measured over a number of
consecutive time
windows, and the resulting values may be analyzed to obtain values regarding
their variation
Page 6 of 40
CA 3050952 2019-07-31
over time. For example, the average inspection fail rate of a factory may be
calculated for
various consecutive 7-day windows, and the change in the average fail rate may
be measured
over the 7-day windows.
[0029] In embodiments of the present disclosure, inspection data include
information
correlated with the results of the inspection (e.g., whether the inspection
was passed or not).
Examples of suitable data for predicting the outcome of an inspection include:
data obtained
from previous inspections at the same factory at which the future inspection
is to take place,
data obtained from inspections at other factories, data obtained from
inspections at other
factories with similar products or product lines to the subjects of the future
inspections, data
obtained from the factory across multiple inspections, attributes of future
inspection bookings
(e.g., the geographic location, time, entity performing the inspection, and/or
the type of
inspection), data related to the business operations of the factory, data
related to product
quality of the factory, general information regarding the factory, data
related to the
sustainability of the factory or other similar factories, and/or data related
to the performance
of the factory or other similar factories. The data may comprise the results
of past inspections
(e.g., whether the inspection was passed or not). The data may comprise
information obtained
from customer reviews on products or product lines similar to those produced
by the factory,
and/or customer reviews on products or product lines originating at the
factory. It will be
appreciated that for some metrics, a factory may be divided into various
divisions within the
factory, with different metrics obtained for each division.
[0030] Examples of data related to future inspection include: the number of
orders placed at
the factory, the quantity of the orders, the quality of the orders, the
monetary value of the
orders, general information regarding the orders, the description of each
product at the
Page 7 of 40
CA 3050952 2019-07-31
factory, (e.g., the product's stock keeping unit (SKU), size, style, color,
quantity, and
packaging method), the financial performance of the factory, the number of
inspected items at
the factory, the number of inspected items at the factory during inspections
of procedures such
as workmanship, packaging, and measurement, information regarding the
acceptable quality
limit (AQL) of processes at the factory (e.g., the sampling number used to
test quality), the
inspection results of past inspections at the factory, the inspection results
of past inspections
for the product/product line, the inspection results at other factories with
similar products, the
inspection results of past inspections at business partners of the factory,
the values for various
metrics collected over the course of inspections, the geographic location of
the factory, the
factory's size, the factory's working conditions and hours of operation, the
time and date of
the inspection, the inspection agency, the individual agents performing the
inspection, and
aggregations and statistical metrics of the aforementioned data.
[0031] As used herein, a product or product line's style refers to a
distinctive appearance of
an item based a corresponding design. A style may have a unique identification
(ID) within a
particular brand, retailer, or factory. Style IDs may be used as an
identifying feature by which
other measurements may be aggregated in order to extract meaningful features
related to
inspection results and risk calculation.
[0032] It will be appreciated that a large number of features may be extracted
by a variety of
methods, such as manual feature extraction, whereby features with a
significant correlation to
the target variable (e.g., the results of the future inspection) are
calculated or extracted from
the obtained data. A feature may be extracted directly from the data, or may
require
processing and/or further calculation to be formatted in such a way that the
desired metric
may be extracted. For example, given the results of various inspections at a
factory over the
Page 8 of 40
CA 3050952 2019-07-31
last year, one may wish to calculate the percentage of failed inspections over
the time period.
In some embodiments, extracting features results in a feature vector, which
may be
preprocessed by applying dimensionality reduction algorithms (such as
principal component
analysis and linear discriminant analysis) or inputting the feature vector
into a neural network,
thereby reducing the vector's size and improving the performance of the
overall system.
[0033] In some embodiments, the trained classifier is a random decision
forest. However, it
will be appreciated that a variety of other classifiers are suitable for use
according to the
present disclosure, including linear classifiers, support vector machines
(SVM), gradient
boosting classifiers, or neural networks such as convolutional neural networks
(CNN) or
recurrent neural networks (RNN).
[0034] Suitable artificial neural networks include but are not limited to a
feedforward neural
network, a radial basis function network, a self-organizing map, learning
vector quantization,
a recurrent neural network, a Hopfield network, a Boltzmann machine, an echo
state network,
long short term memory, a bi-directional recurrent neural network, a
hierarchical recurrent
neural network, a stochastic neural network, a modular neural network, an
associative neural
network, a deep neural network, a deep belief network, a convolutional neural
networks, a
convolutional deep belief network, a large memory storage and retrieval neural
network, a
deep Boltzmann machine, a deep stacking network, a tensor deep stacking
network, a spike
and slab restricted Boltzmann machine, a compound hierarchical-deep model, a
deep coding
network, a multilayer kernel machine, or a deep Q-network.
[0035] In some embodiments, an estimated risk score comprises a value in a
specified range,
e.g., a value in the range [0,100]. For example, a future inspection at a
factory with perfect
performance that has never failed an inspection may achieve a score of 0,
indicating that it is
Page 9 of 40
CA 3050952 2019-07-31
almost certain to pass, while a future inspection at a factory with poor
performance that has
failed every inspection may achieve a score of 100, indicating that it will
almost certainly fail.
In some embodiments, the estimated risk score may be compared against a
threshold value,
and a binary value may be generated, indicating whether the inspection is
likely to pass or not
(e.g., 0 if the score is below the threshold, and 1 otherwise). The threshold
may be chosen
heuristically, or may be adaptively calculated during the training of the
machine learning
model. In some embodiments, determining the risk score is transformed into a
binary
classification problem.
[0036] The performance of machine learning models according to embodiments of
the present
disclosure may be tested against new data, and the machine learning model may
be updated in
order to improve its performance. In some embodiments, updating the machine
learning
model comprises modifying hyperparameters of the model. In some embodiments,
updating
the machine learning model comprises using a different machine learning method
than the one
currently used in the model, and modifying the hyperparameters of the
different machine
learning method in order to achieve a desired performance.
[0037] In embodiments of the present disclosure, historical inspection data
from a number of
inspections during a given time window are used in estimating the risk of
failing a particular
inspection. It will be appreciated that a variety of time windows may be used,
e.g., three
months, six months, nine months, or a year. In some embodiments, the
estimation may be
updated at a regular frequency, e.g., every week, every two weeks, or every
month. Obtaining
updated risk estimations of inspections will assist retailers and
manufacturers in reducing their
potential risk when anticipating an inspection.
Page 10 of 40
CA 3050952 2019-07-31
[0038] In some embodiments, the predicted risk results are converted to a
binary output
indicating whether the inspection is likely to pass or fail.
[0039] In embodiments of the present disclosure, a machine learning model
comprising a
classifier is trained by assembling a training dataset comprising historical
data of inspections
during a variety of time windows, and corresponding performance results for
these
inspections over their respective time windows. In some embodiments, the
inspection data
further comprise data related to the factories in which the inspection took
place, such as data
related to previous inspections at the factory, the performance of the
factory, or general
information related to the factory, as discussed above. In some embodiments,
inspections are
assigned a label indicating whether they are likely to pass or to fail. An
initial training dataset
is generated from the collected data, to which machine learning techniques may
be applied to
generate an optimal model for predicting inspection risk. It will be
appreciated that inspection
risk prediction may be transformed into a binary classification problem, where
a given
inspection is classified as being likely to either pass or fail.
[0040] In some embodiments, training the machine learning model comprises
extracting
features from the initial training dataset. In some embodiments, the selected
features to be
extracted have a high correlation to a target variable. In some embodiments,
the number of
features is reduced in order to reduce the calculation cost in training and
deploying the risk
estimation model. In some embodiments, a number of machine learning methods
and
classification approaches are tested on the training dataset, and a model with
the most desired
performance is chosen for deployment in the risk estimation model. It will be
appreciated that
a variety of machine learning algorithms may be used for risk assessment,
including logistic
regression models, random forest, support vector machines (SVM), deep neural
networks, or
Page 11 of 40
CA 3050952 2019-07-31
boosting methods, (e.g., gradient boosting, Catboost). The hyperparameters of
each model
may be learned to achieve a desired performance. For example, in some
embodiments, the
Institute of Data Science of Technologies (iDST) framework may be used for
hyperparameter
tuning. It will be appreciated that the performance of a machine learning
model may be
measured by different metrics. In some embodiments, the metrics used to
measure the
machine learning model's performance comprise accuracy, precision, recall,
AUC, and/or Fl
score.
[0041] In embodiments of the present disclosure, the hyperparameters for
various machine
learning risk estimation models are learned, and the performance of each model
is measured.
In some embodiments, the metrics used to measure the machine learning model's
performance comprise accuracy, precision, recall, AUC, and/or Fl score. In
some
embodiments, the initial dataset is divided into three subsets: a training
dataset, a validation
dataset, and a testing dataset.
[0042] In some embodiments, 60% of the initial dataset is used for the
training dataset, 20%
is used for the validation dataset, and the remaining 20% is used for the
testing dataset. In
some embodiments, cross validation techniques are used to estimate the
performance of each
risk estimation model. Performance results may be validated by subjecting the
selected risk
prediction model to new inspection data.
[0043] It will be appreciated that predicting the risk of failing an
inspection is useful in
achieving dynamic, risk-based quality control. For example, given the risk of
a particular
inspection, a specific inspection workflow or template may be automatically
generated based
on the requirements of either the factory or a business partner of the
factory. The calculated
risk may be applied to the critical path or time and action plan of a style or
purchase order in
Page 12 of 40
CA 3050952 2019-07-31
order to modify the number of inspections required. Based on the calculated
level of risk of a
particular inspection, an inspection team may assess whether they should waive
or confirm an
inspection booking. Estimated risk may also be leveraged to make
determinations as to the
nature of inspections. For example, for an inspection with a high risk of
failure, the
inspection might be performed via an internal, independent team, while a low
risk inspection
might have the personnel responsible for the performance of the factory
performing the
inspections themselves.
[0044] Referring now to Fig. 1, a schematic view of an exemplary system for
inspection risk
estimation according to embodiments of the present disclosure is shown.
Inspection booking
ID 102 is provided, and relevant features 104 are extracted from inspection
database 112
comprising historical inspection data. The extracted features may be
represented by a feature
vector. The feature vector may be pre-processed prior to being input into
inspection risk
prediction server 106. An estimated prediction result 108 is obtained. In some
embodiments,
pre-processing the feature vector comprises applying a dimensionality
reduction technique to
the vector, such as principal component analysis or linear discriminant
analysis. The
estimated prediction result may comprise a binary value indicating whether the
inspection is
likely to pass or fail. In some embodiments, the estimated prediction result
comprises a value
in a specified range, e.g., a value in the range [0,100]. Relevant features
104 may be obtained
from a factory, from inspection database 112, or from any combination of
sources. The
relevant features may comprise data related to inspections at a factory in
which the future
inspection is to take place, data related to the performance of the factory,
data related to the
factory in general, data relating to a product being inspected, or data
related to the inspection
booking, as discussed above. The relevant features may also be specific to the
type of product
Page 13 of 40
CA 3050952 2019-07-31
the inspection will be conducted for, or the specific product line of the
product. In some
embodiments, estimated prediction result 108 is sent to mobile or web
application 110, where
it may be used for further analysis or decision making. The mobile application
may be
implemented on a smartphone, tablet, or other mobile device, and may run on a
variety of
operating systems, e.g., i0S, Android, or Windows. In various embodiments,
estimated
prediction result 108 is sent to mobile or web application 110 via a wide area
network.
[0045] Referring now to Fig. 2, a process for inspection risk estimation
according to
embodiments of the present disclosure is shown. Inspection booking 201 is
input into
inspection risk prediction system 202 to obtain predicted inspection result
206. In some
embodiments, inspection risk prediction system 202 employs a machine learning
model to
estimate the risk of failure associated with an inspection. In some
embodiments, inspection
risk prediction system 202 is deployed on a server. In some embodiments, the
server is a
remote server. In some embodiments, inspection risk estimation process 200
comprises
performing data processing step 203 to collect and process data related to
inspection booking
201. Data processing may comprise various forms of aggregating the data,
obtaining
statistical metrics of the data, and formatting the data in such a way that
features can be
extracted from them. In some embodiments, the data are obtained from a variety
of sources.
In some embodiments, process 200 comprises performing feature extraction step
204 on the
collected data to extract various features. In some embodiments, feature
extraction step 204 is
performed on data that has been processed at step 203. In some embodiments, a
feature
vector is output. In some embodiments, the features extracted at 204 are input
into a trained
classifier at 205. In some embodiments, the classifier comprises a trained
machine learning
model. In some embodiments, the classifier outputs prediction results 206. In
some
Page 14 of 40
CA 3050952 2019-07-31
embodiments, steps 203, 204, and 205 are performed by inspection risk
prediction system
202. The steps of process 200 may be performed locally to the inspection site,
may be
performed by a remote server, e.g., a cloud server, or may be shared among a
local
computation device and a remote server. In some embodiments, prediction
results 206
comprise a binary value indicating whether or not the inspection is likely to
be failed.
[0046] Referring now to Fig. 3, a process for training an inspection risk
estimation system
according to embodiments of the present disclosure is shown. The steps of
process 300 may
be performed to train an inspection risk estimation model. In some
embodiments, the model
is deployed on a prediction server. The steps of process 300 may be performed
locally to the
factory site, may be performed by a remote server, e.g., a cloud server, or
may be shared
among a local computation device and a remote server. At 302, an initial
training dataset is
created. In some embodiments, the training dataset may comprise data of a
large number of
past inspections from a number of factories, as well as the results of the
inspections (e.g., pass
or fail). The dataset may comprise data related to the factory at which the
inspection took
place and/or the product or product line for which the inspection took place,
and may
comprise various values corresponding to various measurements made over the
course of
previous inspections. In some embodiments, inspection data and corresponding
inspection
results are timestamped. In some embodiments, the data obtained may be
aggregated over a
given length of time or number of inspections. In some embodiments, the data
obtained is
collected only from inspections during a given time window. In some
embodiments, a list of
factories and inspection results may be obtained, with inspection results as
labels for the
inspection data.
Page 15 of 40
CA 3050952 2019-07-31
[0047] At 304, the inspection risk prediction is formulated as a binary
classification problem
wherein a given inspection is classified as either predicted to pass or
predicted to fail. In
some embodiments, a label of 1 is assigned to an inspection if it is predicted
to pass, and a
label of 0 is assigned if the inspection is predicted to fail.
[0048] Useful features are then extracted from the initial training dataset.
The extracted
features may correspond to different time windows, e.g., three months, six
months, nine
months, or a year. The importance of each feature in estimating a final risk
result for an
inspection is calculated. In some embodiments, the importance of each feature
is calculated
by measuring the feature's correlation with the target label (e.g., the
inspection result). At
306, a number of machine learning models are trained on the training dataset,
and the
performance of each model is evaluated. It will be appreciated that acceptable
machine
learning models include a Catboost classifier, a neural network (e.g., a
neural network with 4
fully-connected hidden layers and a ReLU activation function), a decision
tree, extreme
boosting machines, random forest classifier, SVM, and logistic regression, in
addition to those
described above. The hyperparameters of each model may be tuned so as to
optimize the
performance of the model. In some embodiments, the metrics used to measure the
machine
learning model's performance comprise accuracy, precision, recall, AUC, or Fl
score. The
most useful features for performing the desired estimation are selected. At
308, the
performance of the machine learning models are compared. The model with the
most desired
performance is chosen at 310. In some embodiments, a final list of features
used in the
prediction calculation is outputted. At 312, the chosen model is deployed onto
a prediction
server.
Page 16 of 40
CA 3050952 2019-07-31
[0049] Referring now to Fig. 4, a process for updating an inspection risk
estimation system
according to embodiments of the present disclosure is shown. In some
embodiments of
process 400, an existing inspection risk prediction model is updated. In some
embodiments,
updating the prediction model comprises inputting new data and modifying the
parameters of
the learning system accordingly to improve the performance of the system. In
some
embodiments, a new machine learning model may be chosen to perform the
estimation. The
inspection risk prediction model may be updated at regular intervals, e.g.,
monthly,
bimonthly, or quarterly, or may be updated when a certain amount of new data
are
accumulated. It will be appreciated that an updated risk estimation system
provides for more
accurate risk estimation compared to existing methods.
[0050] In some embodiments, new data and inspection results 420 for a number
of
inspections are collected from inspection management platform 410 and used to
generate a
new dataset with labels corresponding to the data for each inspection.
Inspection
management platform 410 may comprise a database containing inspection data and
inspection
results for a number of past inspections. New data and inspection results 420
may comprise
customer feedback regarding prior predictions, and may include ground truth
risk scores
comprising indications of the accuracy of prior predictions, such as which
predictions made
by the prediction model were incorrect, as well as corrected results for the
predictions. It will
be appreciated that the new dataset may be structured in a similar way to the
initial dataset
described above. In some embodiments, the new dataset is combined with an
existing
training dataset 430 to create a new training dataset 440. In some
embodiments, the
performance of the latest version of the trained risk prediction model 499,
comprising
inspection risk predictor 450, is measured on the new training dataset. In
some embodiments,
Page 17 of 40
CA 3050952 2019-07-31
if the performance of the latest version of the trained risk prediction model
499 and predictor
450 is under a certain threshold, feature re-engineering step 460 and/or
applying new machine
learning models 480 may be performed at 470 to retrain the prediction model.
The threshold
may be chosen heuristically, or may be adaptively calculated during training.
[0051] It will be appreciated that the methods of re-training the prediction
model at 470 may
be similar to those used in training the inspection risk estimation system, as
described above.
The process of re-training the prediction model may be repeated a number of
times until the
performance of the model on the new training dataset reaches an acceptable
threshold. In
some embodiments, the latest version of the trained risk prediction model 499
is updated at
490 with the new model trained at 470. The updated risk prediction model may
then be
deployed on prediction server 495. Existing training dataset 430 may also be
updated to
reflect the newly obtained data.
[0052] Referring now to Figs. 5-7, various processes for training inspection
risk estimation
systems according to embodiments of the present disclosure are shown. In
various
embodiments of the present disclosure, generating a trained risk estimation
system comprises
four primary steps: data collection, feature extraction, model training, and
risk prediction. In
some embodiments, data collection comprises creating an initial training
dataset using the
methods described above. In some embodiments, feature extraction comprises
extracting a
number of useful features from the initial training dataset. The features
extracted may be a
subset of a larger number of features that may be extracted from the initial
training dataset. In
some embodiments, the importance of each feature to the risk prediction
calculation is
measured. In some embodiments, the features with the least relevance to the
prediction
calculation are not used in the risk prediction model. In some embodiments, a
fixed number
Page 18 of 40
CA 3050952 2019-07-31
of features are extracted. In some embodiments, determining the relevance of a
feature to the
prediction calculation comprises measuring the correlation of the feature with
the risk
prediction results. In some embodiments, a dimensionality reduction technique
(e.g.,
principal component analysis or linear discriminant analysis) may be applied
to the extracted
features. In some embodiments, the feature extraction step comprises manual
feature
extraction. Model training comprises measuring the performance of a number of
machine
learning models on the extracted features. The model with the most desired
performance may
be selected to perform risk prediction.
[0053] Referring now to Fig. 5, a process for training an inspection risk
estimation system
according to embodiments of the present disclosure is shown. In some
embodiments, manual
feature extraction 502 is performed on an initial training dataset 501
comprising data related
to an inspection booking. Features may be extracted based on inspection data
during a
specific time window (e.g., one year). In some embodiments, a feature vector
corresponding
to each inspection's data are generated from the feature extraction step. In
some
embodiments, a label is assigned to each feature vector. In some embodiments,
the labels are
obtained from the initial training dataset 501. In some embodiments, the label
is a binary
value indicating whether the inspection passed or failed. In some embodiments,
the risk
estimation of an inspection is transformed into a binary classification
problem, wherein an
inspection can be classified as passing or failing. Various machine learning
models (e.g.,
support vector machine, decision tree, random forest, or neural networks) and
boosting
methods (e.g., Catboost or XGBoost) may be tested at 503 on the initial
training dataset.
[0054] In training the various machine learning models and boosting methods,
the initial
training dataset may be divided into a training dataset and a testing dataset.
For example,
Page 19 of 40
CA 3050952 2019-07-31
80% of the initial training dataset may be used to create a training dataset,
and the remaining
20% may be used to form a testing dataset. In some embodiments, the initial
training dataset
may be divided into a training dataset, a testing dataset, and a validation
dataset. In some
embodiments, the hyper-parameters of the machine learning models and boosting
methods are
tuned to achieve the most desired performance. The model with the most desired
performance may then be selected to provide risk estimation on input
inspection data. In
some embodiments, the selected model is deployed onto a prediction server to
providing for
future risk predictions.
[0055] In some embodiments of the present disclosure, a feature vector is
calculated from
inspection data. The feature vector is input into a risk prediction model and
a predicted
failure probability is outputted. The probability may be compared with a given
threshold to
determine whether the inspection should be classified as passing or not. In
some
embodiments, an inspection is considered likely to pass if the predicted
probability is greater
than or equal to the threshold. In some embodiments, a risk score is obtained
based on the
calculated probability. In some embodiments, the risk score comprises a value
in a
predetermined range, e.g., [0, 100]. In some embodiments, testing the risk
prediction model
comprises comparing the predicted inspection results with known data.
[0056] In some embodiments, a risk score R is obtained based on the calculated
probability p
using the following procedure:
[0057] A range [A, B] defining the upper and lower bounds of the risk score is
chosen. For
example, one may consider the risk score R to be within the range [0, 100],
where R = 0
represents a lowest possible risk of an inspection (e.g., the inspection is
almost certain to
pass), and R = 100 represents a highest possible risk of an inspection (e.g.,
the inspection is
Page 20 of 40
CA 3050952 2019-07-31
almost certain to fail). Given that the predicted probability p is within the
unit interval [1, 0],
one can determine a mapping F to assign a predicted probability to a
corresponding risk score
R:
F: [0, 1] ¨> [A, B]
Equation 1
[0058] For a given p,
F(p) = p ¨> R
Equation 2
[0059] F is chosen such that F(0) = A and F(1) = B. For example, a linear
mapping may be
used:
F (p) = Ax p + (1 ¨ p) x B
Equation 3
[0060] Referring now to Fig. 6, a process for training an inspection risk
estimation system
according to embodiments of the present disclosure is shown. In some
embodiments, features
are obtained from inspection data 601 using manual feature extraction 602. It
will be
appreciated that feature extraction may result in a large number of extracted
features for each
inspection, and thus, large feature vectors. The number of features extracted
may number in
the hundreds. Reducing the dimensionality of the feature vectors may result in
more efficient
training, deployment, and operation of the prediction model. In some
embodiments, the
dimensionality of a feature vector is reduced at 603 by calculating the
correlation of each
feature to the target variable, and only keeping those features with high
correlation to the
target variable. In some embodiments, the dimensionality of a feature vector
is reduced at
603 by applying a dimensionality reduction algorithm to the vector, such as
principal
Page 21 of 40
CA 3050952 2019-07-31
component analysis (PCA) or linear discriminant analysis (LDA). In some
embodiments, the
features computed in the resulting smaller-dimension vectors for a number of
inspections are
input into various machine learning and/or gradient boosting models at 604,
and the model
with the most desired performance is selected, as described above.
[0061] Referring now to Fig. 7, a process for training an inspection risk
estimation system
according to embodiments of the present disclosure is shown. In some
embodiments, features
are obtained from inspection data 701 using manual feature extraction 702. In
some
embodiments, the feature extraction step results in a feature vector. In some
embodiments,
the feature vector is input into a neural network at 703. In some embodiments,
the neural
network comprises a deep neural network. In some embodiments, the neural
network
comprises an input layer, a number of fully-connected hidden layers, and an
output later with
a predetermined activation function. In some embodiments, the activation
function comprises
a ReLU or sigmoid activation function, although it will be appreciated that a
variety of
activation functions may be suitable. The output of the neural network may be
considered as
a new feature vector, and may be input into various machine learning models at
704 using
similar steps to those described above. In some embodiments, the new feature
vector is of
smaller dimensionality than the input feature vector.
[0062] Table 1 lists a number of features that may be extracted from
inspection data using the
methods described above. In various exemplary embodiments, gradient boosting
on decision
trees is applied, for example using catboost. These features may have high
correlation with
the target variable. Note that features marked with an asterisk (*) may be
computed right
after an inspection booking is confirmed and becomes an assignment.
Page 22 of 40
CA 3050952 2019-07-31
Total available product quantity in the inspection booking
The standard deviation of available quantities among production items
The weekday of the expected inspection date (e.g., Monday, Tuesday...)
The number of unique measurement sizes among all production items of the
inspection (*)
Factory identification
The number of unique measurement units in the inspection (*)
The ratio between total available quantity and total order quantity in the
inspection
Brand Office Identification
Total order quantity in the inspection
The average fail rate of the factory under consideration during the last three
months
Total available quantity for solid packages
Style Difficulty Coverage: the number of styles present in the current
inspection booking which
can be frequently found in failed inspections (e.g., for all similar
inspections completed in the
last 6 months, the number of styles that are found in at least 50% of the
failed inspections that
are also present in the current inspection booking are counted)
The standard deviation of the number of defects in packaging procedure among
all inspections
managed by the brand office under consideration during the last 3 months (*)
The standard deviation of the total defects among all inspections managed by
the brand office
under consideration during the last 3 months
The difference, in number of days, between the date that an inspection booking
was initiated and
the date that it was confirmed to become an assignment by a brand or retailer.
In some
embodiments, it is the difference, in number of days, between the date that an
inspection
Page 23 of 40
CA 3050952 2019-07-31
booking was initiated and the date that a brand or retailer contacted the
factory to confirm the
final inspection date.(*)
The average number of defects in packaging procedure among all inspections
managed by the
brand office under consideration during the last 3 months
Table 1
[0063] It will be appreciated that a variety of additional features and
statistical measures may
be used in accordance with the present disclosure.
[0064] Referring now to Fig. 8, a schematic of an example of a computing node
is shown.
Computing node 10 is only one example of a suitable computing node and is not
intended to
suggest any limitation as to the scope of use or functionality of embodiments
described
herein. Regardless, computing node 10 is capable of being implemented and/or
performing
any of the functionality set forth hereinabove.
[0065] In computing node 10 there is a computer system/server 12, which is
operational with
numerous other general purpose or special purpose computing system
environments or
configurations. Examples of well-known computing systems, environments, and/or
configurations that may be suitable for use with computer system/server 12
include, but are
not limited to, personal computer systems, server computer systems, thin
clients, thick clients,
handheld or laptop devices, multiprocessor systems, microprocessor-based
systems, set top
boxes, programmable consumer electronics, network PCs, minicomputer systems,
mainframe
computer systems, and distributed cloud computing environments that include
any of the
above systems or devices, and the like.
Page 24 of 40
CA 3050952 2019-07-31
[0066] Computer system/server 12 may be described in the general context of
computer
system-executable instructions, such as program modules, being executed by a
computer
system. Generally, program modules may include routines, programs, objects,
components,
logic, data structures, and so on that perform particular tasks or implement
particular abstract
data types. Computer system/server 12 may be practiced in distributed cloud
computing
environments where tasks are performed by remote processing devices that are
linked through
a communications network. In a distributed cloud computing environment,
program modules
may be located in both local and remote computer system storage media
including memory
storage devices.
[0067] As shown in Fig. 8, computer system/server 12 in computing node 10 is
shown in the
form of a general-purpose computing device. The components of computer
system/server 12
may include, but are not limited to, one or more processors or processing
units 16, a system
memory 28, and a bus 18 that couples various system components including
system memory
28 to processor 16.
[0068] Bus 18 represents one or more of any of several types of bus
structures, including a
memory bus or memory controller, a peripheral bus, an accelerated graphics
port, and a
processor or local bus using any of a variety of bus architectures. By way of
example, and not
limitation, such architectures include Industry Standard Architecture (ISA)
bus, Micro
Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics
Standards
Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus,
Peripheral
Component Interconnect Express (PCIe), and Advanced Microcontroller Bus
Architecture
(AMBA).
Page 25 of 40
CA 3050952 2019-07-31
[0069] Computer system/server 12 typically includes a variety of computer
system readable
media. Such media may be any available media that is accessible by computer
system/server
12, and it includes both volatile and non-volatile media, removable and non-
removable media.
[0070] System memory 28 can include computer system readable media in the form
of
volatile memory, such as random access memory (RAM) 30 and/or cache memory 32.
Computer system/server 12 may further include other removable/non-removable,
volatile/non-volatile computer system storage media. By way of example only,
storage
system 34 can be provided for reading from and writing to a non-removable, non-
volatile
magnetic media (not shown and typically called a "hard drive"). Although not
shown, a
magnetic disk drive for reading from and writing to a removable, non-volatile
magnetic disk
(e.g., a "floppy disk"), and an optical disk drive for reading from or writing
to a removable,
non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can
be
provided. In such instances, each can be connected to bus 18 by one or more
data media
interfaces. As will be further depicted and described below, memory 28 may
include at least
one program product having a set (e.g., at least one) of program modules that
are configured
to carry out the functions of embodiments of the disclosure.
[0071] Program/utility 40, having a set (at least one) of program modules 42,
may be stored in
memory 28 by way of example, and not limitation, as well as an operating
system, one or
more application programs, other program modules, and program data. Each of
the operating
system, one or more application programs, other program modules, and program
data or some
combination thereof, may include an implementation of a networking
environment. Program
modules 42 generally carry out the functions and/or methodologies of
embodiments as
described herein.
Page 26 of 40
CA 3050952 2019-07-31
[0072] Computer system/server 12 may also communicate with one or more
external devices
14 such as a keyboard, a pointing device, a display 24, etc.; one or more
devices that enable a
user to interact with computer system/server 12; and/or any devices (e.g.,
network card,
modem, etc.) that enable computer system/server 12 to communicate with one or
more other
computing devices. Such communication can occur via Input/Output (I/O)
interfaces 22. Still
yet, computer system/server 12 can communicate with one or more networks such
as a local
area network (LAN), a general wide area network (WAN), and/or a public network
(e.g., the
Internet) via network adapter 20. As depicted, network adapter 20 communicates
with the
other components of computer system/server 12 via bus 18. It should be
understood that
although not shown, other hardware and/or software components could be used in
conjunction
with computer system/server 12. Examples, include, but are not limited to:
microcode, device
drivers, redundant processing units, external disk drive arrays, RAID systems,
tape drives, and
data archival storage systems, etc.
[0073] The present disclosure may be embodied as a system, a method, and/or a
computer
program product. The computer program product may include a computer readable
storage
medium (or media) having computer readable program instructions thereon for
causing a
processor to carry out aspects of the present disclosure.
[0074] The computer readable storage medium can be a tangible device that can
retain and
store instructions for use by an instruction execution device. The computer
readable storage
medium may be, for example, but is not limited to, an electronic storage
device, a magnetic
storage device, an optical storage device, an electromagnetic storage device,
a semiconductor
storage device, or any suitable combination of the foregoing. A non-exhaustive
list of more
specific examples of the computer readable storage medium includes the
following: a portable
Page 27 of 40
CA 3050952 2019-07-31
computer diskette, a hard disk, a random access memory (RAM), a read-only
memory
(ROM), an erasable programmable read-only memory (EPROM or Flash memory), a
static
random access memory (SRAM), a portable compact disc read-only memory (CD-
ROM), a
digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically
encoded device
such as punch-cards or raised structures in a groove having instructions
recorded thereon, and
any suitable combination of the foregoing. A computer readable storage medium,
as used
herein, is not to be construed as being transitory signals per se, such as
radio waves or other
freely propagating electromagnetic waves, electromagnetic waves propagating
through a
waveguide or other transmission media (e.g., light pulses passing through a
fiber-optic cable),
or electrical signals transmitted through a wire.
[0075] Computer readable program instructions described herein can be
downloaded to
respective computing/processing devices from a computer readable storage
medium or to an
external computer or external storage device via a network, for example, the
Internet, a local
area network, a wide area network and/or a wireless network. The network may
comprise
copper transmission cables, optical transmission fibers, wireless
transmission, routers,
firewalls, switches, gateway computers and/or edge servers. A network adapter
card or
network interface in each computing/processing device receives computer
readable program
instructions from the network and forwards the computer readable program
instructions for
storage in a computer readable storage medium within the respective
computing/processing
device.
[0076] Computer readable program instructions for carrying out operations of
the present
disclosure may be assembler instructions, instruction-set-architecture (ISA)
instructions,
machine instructions, machine dependent instructions, microcode, firmware
instructions,
Page 28 of 40
CA 3050952 2019-07-31
state-setting data, or either source code or object code written in any
combination of one or
more programming languages, including an object oriented programming language
such as
Smalltalk, C++ or the like, and conventional procedural programming languages,
such as the
"C" programming language or similar programming languages. The computer
readable
program instructions may execute entirely on the user's computer, partly on
the user's
computer, as a stand-alone software package, partly on the user's computer and
partly on a
remote computer or entirely on the remote computer or server. In the latter
scenario, the
remote computer may be connected to the user's computer through any type of
network,
including a local area network (LAN) or a wide area network (WAN), or the
connection may
be made to an external computer (for example, through the Internet using an
Internet Service
Provider). In some embodiments, electronic circuitry including, for example,
programmable
logic circuitry, field-programmable gate arrays (FPGA), or programmable logic
arrays (PLA)
may execute the computer readable program instructions by utilizing state
information of the
computer readable program instructions to personalize the electronic
circuitry, in order to
perform aspects of the present disclosure.
[0077] Aspects of the present disclosure are described herein with reference
to flowchart
illustrations and/or block diagrams of methods, apparatus (systems), and
computer program
products according to embodiments of the disclosure. It will be understood
that each block of
the flowchart illustrations and/or block diagrams, and combinations of blocks
in the flowchart
illustrations and/or block diagrams, can be implemented by computer readable
program
instructions.
[0078] These computer readable program instructions may be provided to a
processor of a
general purpose computer, special purpose computer, or other programmable data
processing
Page 29 of 40
CA 3050952 2019-07-31
apparatus to produce a machine, such that the instructions, which execute via
the processor of
the computer or other programmable data processing apparatus, create means for
implementing the functions/acts specified in the flowchart and/or block
diagram block or
blocks. These computer readable program instructions may also be stored in a
computer
readable storage medium that can direct a computer, a programmable data
processing
apparatus, and/or other devices to function in a particular manner, such that
the computer
readable storage medium having instructions stored therein comprises an
article of
manufacture including instructions which implement aspects of the function/act
specified in
the flowchart and/or block diagram block or blocks.
[0079] The computer readable program instructions may also be loaded onto a
computer,
other programmable data processing apparatus, or other device to cause a
series of operational
steps to be performed on the computer, other programmable apparatus or other
device to
produce a computer implemented process, such that the instructions which
execute on the
computer, other programmable apparatus, or other device implement the
functions/acts
specified in the flowchart and/or block diagram block or blocks.
[0080] The flowchart and block diagrams in the Figures illustrate the
architecture,
functionality, and operation of possible implementations of systems, methods,
and computer
program products according to various embodiments of the present disclosure.
In this regard,
each block in the flowchart or block diagrams may represent a module, segment,
or portion of
instructions, which comprises one or more executable instructions for
implementing the
specified logical function(s). In some alternative implementations, the
functions noted in the
block may occur out of the order noted in the figures. For example, two blocks
shown in
succession may, in fact, be executed substantially concurrently, or the blocks
may sometimes
Page 30 of 40
CA 3050952 2019-07-31
be executed in the reverse order, depending upon the functionality involved.
It will also be
noted that each block of the block diagrams and/or flowchart illustration, and
combinations of
blocks in the block diagrams and/or flowchart illustration, can be implemented
by special
purpose hardware-based systems that perform the specified functions or acts or
carry out
combinations of special purpose hardware and computer instructions.
[0081] The descriptions of the various embodiments of the present disclosure
have been
presented for purposes of illustration, but are not intended to be exhaustive
or limited to the
embodiments disclosed. Many modifications and variations will be apparent to
those of
ordinary skill in the art without departing from the scope and spirit of the
described
embodiments. The terminology used herein was chosen to best explain the
principles of the
embodiments, the practical application or technical improvement over
technologies found in
the marketplace, or to enable others of ordinary skill in the art to
understand the embodiments
disclosed herein.
Page 31 of 40
CA 3050952 2019-07-31
