Big data (mal)algorithms

I have just finished reading Cathy O’Neil’s Weapons of Math Destruction and it provides some interesting thoughts about how to avoid creating automated inequalities. The book discusses whole areas of human interaction that are driven by algorithms to make punitive decisions; for example, screening job applicants based on glorified magazine personality tests. The three principles are the scale, transparency and a corrective feedback loop. If the algorithm does not scale then its impact is limited; therefore, I will discuss the other two in more detail.

Transparency means cracking opening the black box to see what assumptions are baked into the system. Companies will argue that the algorithms are commercially sensitive or their competitive advantage; however regulations can and must be devised to make it clear that discrimination is unfair and illegal. For example, using markers for gender like occupation to get around regulations prohibiting discriminating on the basis of gender. Regulations and regulators, like Financial Conduct Authority, currently exist so this is not a pipe-dream, even if the quality might be variable.

More importantly, without a corrective feedback loop there is no way of knowing if the algorithm is working as intended. The reason algorithms work well in fields like sport is because the result gives the predictive algorithm more data and a known outcome. To reference the earlier example, if someone is not employed because they were screened out then only if that person is tracked in subsequent employment with another company would there be anyway of knowing if they could have been successful in the role for which they original applied. The algorithm does not have any results to correct any errors and probably only one result, the employed person, for review purposes but any number of applicants may well have performed as well, if not better, as the person who was given the job.

For me, the important point to remember from this book is: how is any algorithm going to be able to learn, and if they cannot learn then the process is just automating guessing.