This book review was written by Eugene Kernes
“The math-powered applications powering the data economy were based on choices made by fallible human beings. Some of these choices were no doubt made with the best intentions. Nevertheless, many of these models encoded human prejudice, misunderstanding, and bias into the software systems that increasingly managed our lives. Like gods, these mathematical models were opaque, their workings invisible to all but the highest priests in their domain: mathematicians and computer scientists. Their verdicts, even when wrong and harmful, were beyond dispute or appeal. And they tend to punish the poor and the oppressed in our society, while making the rich richer.” – Cathy O'Neill, Introduction, Pages 10-11
“Our own values and desires influence our choices, from the
data we choose to collect to the questions we ask. Models are opinions embedded in
mathematics. | Whether or not a model works is also a matter
of opinion. After all, a key component
of every model, whether formal or informal, is its definition of success. This is an important point that we’ll return
to as we explore the dark world of WMDs.
In each case, we must ask not only who designed the model but also what
that person or company is trying to accomplish.” – Cathy O'Neill, Chapter 1:
Bomb Parts: What Is a Model?, Page 26-27
“That’s a problem, because scientists need their error feedback – in this case the presence of false negatives – to delve into forensic analysis and figure out what went wrong, what was misread, what data was ignored. It’s how systems learn and get smarter. Yet as we’ve seen, loads of WMDs, from recidivism models to teacher scores, blithely generate their own reality. Managers assume that the scores are true enough to be useful, and the algorithm makes tough decisions easy. They can fire employees and cut costs and blame their decisions on an objective number, whether it’s accurate or not.” – Cathy O'Neill, Chapter 7: Sweating Bullets: On the Job, Page 126
Is This An Overview?
Models are abstract representations
of a process, used to make predictions.
Everyone has and uses models every day to form expectations for events,
and then make a decision. Models are
simplifications of reality, with a variety of information left out. Models are based on the choices of the people
that make them. Choices about what
variables and data to include or exclude.
Choices that can have good intentions, but have negative
consequences. Choices that can code
prejudice, bias, and misunderstanding into a model, into a software
system. Models are opinions embedded in
mathematics. Models can be used to
improve society, but models can also be harmful. Models can become weapons of math
destruction.
Models that
function well are those that use enough and as much data as possible, without
trying to exclude data which does not conform to expectations. Data that is relevant to what is trying to be
understood. Data that becomes used to
update the model. Models require
feedback to correct for mistakes.
Without feedback, there is no method of learning from mistakes. Models that seek to understand reality need
to constantly change based on feedback received. Models are based on past experiences, used to
form expectations of the future. Models
need to change as the future is not necessarily like the past.
What turns
functional models into weapons of math destruction, is when the model is
opaque, has scale, and does damage. When
models are opaque, such as when they are hidden from the public, and only
accessible to a select few, the model losses access to feedback. By hiding the details of what goes into the
model, the model is difficult to question and disagree with. The model camouflages human bias with
technology. What is coded becomes
dependent on information that is accessible to being measured, rather than what
is wanted and effective.
Weapons of math
destruction are models that embody their own reality, rather than search to
understand reality. The model defines
reality, with the results being justified by code. Success is based on who sponsored the model,
for business or political reasons. The
models are not necessarily beneficial to the people the model is used on, even
if the model is claimed to be beneficial to those the model is used on. When the model does harm on a massive scale,
the human victims are held to a higher standard of evidence than the model
is. Harm is deemed more acceptable when
the decision was validated by a mathematical model, even though the model was
coded by humans with an intention.
Caveats?
Even as this book is based on math,
there is no need for a mathematical background to understand the content. There is a lack of a technical explanation
for models, nor an explanation for how the weapons of math destruction were formed. The book is based on examples of when the
models do harm on a massive scale. There
are few references to when models can be useful. As models are coded with a human bias, the
author is a human who wrote the book with a bias, revealed by political
decisions and examples.
