Bias in data-enabled decisions

Many operational decisions are now heavily reliant on automation - data, models, rules etc.

We’re all trying our best to eliminate bias in the decisions we make as humans.

But we still have a lot of work to do in removing bias from our automated decision support tools.

For example, New York city has legislated that employee related decision support tools must be tested for bias.

This is because these tools can exhibit bias - using personal characteristics in models and rules.

One such set of models and rules relates to hiring.

Because of the sheer volume of applicants that agencies and organisations receive for a given job, automated decision tools are used to screen CVs. In many cases, those tools use some form of AI - for example, a machine learning process/algorithm.

Those are often trained on historical data. So if there were, historically, more successful males than females in job applications, the algorithm may favour male candidates.

This is unfair and perpetuates discrimination.

One solution to this problem is to remove any gender data that feeds into the model. We don’t tell the automated tool what the applicant’s gender is.

But this can get a bit complicated. The gender field alone is not enough.

There are other “proxy indicators” of gender.

For example, a person’s name could suggest their gender. Words like “she” in a CV that’s written in the third person would also be indicative of gender, and ML algorithms can be smart/sophisticated enough to notice this. We need to find all such data and remove it from the model input/features.

There are several other examples of potential bias in data-enabled decisions.

Fortunately, we can eliminate them. More on this to come in future articles.