Collateral Damage from Weapons of Math Destruction, Cathy O’Neil

 Area 1:

We all know that the credit score is used to make decisions about people's reliability. But it's not cut and dried; the credit score is just a proxy. It may be unreliable as a proxy, and it may simply contain incorrect or outdated information. Even if they may be unreliable, people with money have the resources to keep from getting a bad credit score. And if a person knows their credit info is wrong, they can move to have it corrected, although that sometimes takes a lot of time and sometimes legal action.

The e-score is similarly used to make financial decisions about people, and it is even less 'scientific,' but it is also unregulated. It can use race or ZIP code to decide what interest rate someone gets, and they have no recourse to fix it, because they often won't even know it was used against them. Everyone likely has an e-score that is used to make decisions about them, and they have no idea what the score is or that it even exists at all. Internet tracking makes it easy to establish a comprehensive model of someone by their online behavior, and that can be rolled into the e-score. Part of the problem with this is that the 'customer' is not even the customer; they are the product. They are simply part of a large dataset that is sold to someone else, so there is little motive for the e-score provider to correct the information. The result: by a feedback loop, the rich get richer and the poor get poorer.

Area 2:

Less-biased algorithms and more transparency is important, but sometimes the complexity involved precludes easy tweaking, especially even after baked-in bias has been established in the results. The murkier the decision-making process gets, the harder it is to correct.

I think analysis of big data needs some built in scoring. Just like loan applications with fraudulent information (stated income used to be acceptable; now it must be backed up with tax returns), any single bit of information in the large datasets that both the credit score and e-score use must be considered suspect. So along with weighing individual data points, like stated income, a reliability or trust score should be assigned, like fuzzy logic uses. Fuzzy logic deals in 'degrees of truth,' rather than just a true or false. Something can be 0 or 1, false or true, but also any real number in the range between them. I think this would help eliminate bias somewhat. For example, the text discusses ZIP code as a factor in e-score, but we know that this is just a partly reliable predictor, so it should be weighted as less important, and/or should be given a score between 0 and 1 since it may not be trustworthy. 

I had a long discussion with a family member who works in banking last night. We talked about how the banking industry has changed from simple forms and quick loans to stacks of paperwork, each line of which is trying to prevent the reoccurrence of previous fraud or abuse. And while there was clearly a cost of that previous fraud or abuse, there is a threshold where policy becomes necessary, but there is also a cost to that policy, which we hope is much less than the damage it is intended to prevent. Every paragraph in that stack of loan paperwork impacts customers somewhere, and it impacts the way someone somewhere does their job. We talked about how policy spreads the costs of implementation over everyone it impacts. The benefits ostensibly prevent abuse of financial relationships, like fraud or predatory lending, but there are monetary costs to implementing policy, and some folks may be harmed because they can't comply and subsequently have their loan turned down, when it may have been previously approved. Some people who would have previously been an acceptable risk now fall between the cracks and are harmed, in this case by the abuses of the housing loan industry. And it doesn't keep rich bankers, who my family member said are the most risk tolerant and just want to get deals approved, from getting richer in the end. And most who are directly responsible for things like market crashes aren't held responsible. So that cost, besides getting absorbed into the national debt, gets spread amongst all Americans. 

I wonder sometimes if a more granular risk score would even be desired by many banks, especially if it meant denying loans. I'll wager many bankers who make their bread and butter this way would only want it if it gave them more income.