Iterative Process of Modelling + Decision Making from the 'Software Engineering Manual of Style'

Abstract

Formal definition of the subjective machine learning process vs. the classical objective randomised, controlled experiment approach.

This is an old one that draws on the work of Dr. Andreas Wiegand, Dr. Andrew Ng and Dr. Sebastien Haneuse which forms the basis of an updated process for modelling and decision making.

The ML process of modelling & decision making (see Figure 11) was first developed by Dr. Andeas Wiegand whilst Chief Scientist at amazon.com in the late 90s. The process has since been evolved by Dr. Andrew Ng (Director Emeritus, Stanford AI Lab) and Dr. Sebastien Haneuse, University of Washington.

In my book, I have added two new and additional steps to the process and re-arranged things a little so I can map additional processes by UoW and Stanford onto the same continuum.

If you follow all the steps correctly and you are trained in machine learning, the evaluation step should give you metrics which represent a 'floor' or minimum best performance of your model in production.

The textbook's homepage is here:
https://lnkd.in/gQHTnBBq

Follow me for more updates on InfoSec, SE, CS & AI...

Full text

Capture engagement
data & actions such as
sales, microconversions,
signals, telemetry &
model performance
Adjust objective
function, targets, risk
preferences & exploration
vs. exploitation posture
by assessing VOI
No
Iterative development (Agile Methodologies)
Measure
Collect, store &
manage the data
Describe †
Exploratory data
analysis, held out set(s)
and framing by ML expert
(Re)Define
Objectives &
benchmarks
Feature Selection
Choose & engineer
model features
Predict & Test
Test probabilistic
model designs
Evaluation †
Formal assesment &
model evalulation
by ML expert
launch approval
Execute
Decide & Act
I. Formulate the scientific question
II. Identify a correspond-
ing parameterization
III. Specify prior distribu-
tions for unknown parameters
IV. Characterize linkage between the
parameters and the observed data
V. Apply Bayes Theorem
VI. Interpret the results and
refine the scientific question
1. Select & describe problem
2. Get data
3. Design model
4. Train model
5. Test model
6. Deploy model
7. Monitor outcomes
Dr. Sebastien Haneuse, UoW Dr. Andreas Wiegand, Stanford Dr. Andrew Ng, Stanford
Figure 11: The iterative process of modelling and decision making. †denotes security touchpoints.
(C) COPYRIGHT 2023, Andrew (AP) Prendergast. All rights reserved - @CompSciFutures / blog.andrewprendergast.com Page 60

Machine Learning & AI Best Practices
The iterative process of modelling & deci-
sion making
The ML process of modelling & decision making
(see Figure 11) was first developed by Dr. Andeas
Wiegand whilst Chief Scientist at amazon.com in the
late 90s. The process has since been evolved by Dr.
Andrew Ng (Director Emeritus, Stanford AI Lab) and
Dr. Sebastien Haneuse, University of Washington.
1. Measure: This is where one captures engage-
ment data and actions such as sales, microconver-
sions, signals, telemetry & model performance.
2. Describe: This is where one (I) formulates the
scientific question
•understanding the association between two
variables
•prediction of some future event
•output is a loose framing of the problem do-
main and held out dev & test sets
It is critically important that worknotes such as
Excel files or Tableau workbooks which include
aggregate totals and charts which relate to de-
tailed insights be prepared by an experienced ma-
chine learning specialist in conjunction with a
data scientist in such a way that the analyses can
be re-produced to ensure that the source data has
not been tampered with (this is a security mea-
sure).
3. (Re)Define: This is where one (II) identifies a
corresponding parameterization
•’translation’ of the scientific question into
statistical terms
•Conduct a Bayesian group-think session for
translation of substantive knowledge and
quantities
•output is a better defined framing of the
problem domain
You should also consider at this stage of the pro-
cess adjusting objective functions, targets, risk
preferences & exploration vs. exploitation pos-
ture by assessing value of information (VOI).
4. Feature Selection: This is where one (III)
specifies prior distributions for the unknown pa-
rameters
•can come from data or knowledge
•establish several options towards a sensitiv-
ity analyses
•identification of ’non-informative’ priors
5. Predict & Test: This is where one
(IV ) characterizes the linkage between the pa-
rameters and the observed data:
•design features amenable to Bayesian analy-
sis
•interpret the problem domain as a causal
graph of random variables
•specification of the likelihood, loss & cost
functions
and (V) applies Bayes Theorem:
•posterior ∝prior ×likelihood
•turn the Bayesian ’handle’ with bayesian up-
dates
Andrew Ng specifically enumerates a process for
this stage, as follows:
•Get data
•Design model
•Train model
•Test model (using held out dev set)
6. Evaluation: This is where one (V I) interprets
the results and refines the scientific question
•examine features of the posterior
•review learned random variable parameters
and Bayesian structure
•Assess training set performance against held
out dev & test sets
It is critically important that this step be carried
out by a trained machine learning specialist, and
that numerous performance metrics be calculated
and the scripts required to re-run the experiments
be stored such that they can be re-run to check
that the model has not been tampered with (as a
security measure).
7. Launch Approval: Should only proceed if a hu-
man panel has been out-performed and a neces-
sary, sufficient & complete turing test has been
passed (if required).
8. Execute: If a model passes launch approval, it
may be used to make a single coarse grained deci-
sion worth many millions of dollars, or many mil-
lions of small high frequency decisions each worth
a few cents.
(C) COPYRIGHT 2023, Andrew (AP) Prendergast. All rights reserved - @CompSciFutures / blog.andrewprendergast.com Page 59