Ethics in AI

Written by

Data Science Lab

Published on

21 June 2021

AI systems are now so widespread and familiar that we often overlook the interaction needed between people and algorithms. Algorithms shape your social media feed, tell you what to buy next on Amazon and show you the best way to get from A to B. We place a lot of trust in AI, but is that trust justified? AI is also used to make potentially life-changing decisions about the risk of reoffending, a medical diagnosis or how your self-driving car responds to a dangerous situation. Algorithms often make the right decision, but when they get it wrong, the consequences can be irreversible. Consider these examples:

At the Belastingdienst, an automated risk selection system determined which applications for benefits required additional checks. The question about dual nationality affected the algorithm’s outcome and contributed to the childcare benefits scandal.
Based on her purchases at a supermarket chain, an American teenager was sent vouchers for baby clothes by post. Her father discovered at home that his daughter was pregnant before she had told him.
Amazon’s system for assessing job applicants’ CVs gave women a lower chance of being hired. The model had been trained on historical data that mainly covered white male employees.

AI has considerable power, and with that comes great responsibility. Ethical questions about AI are no longer just philosophical: they now have practical implications that governments and businesses need to examine. To address this, the EU has set out seven pillars of Responsible AI[1]:

Human agency and oversight
Technical robustness and safety
Privacy and data governance
Diversity, non-discrimination and fairness
Transparency
Accountability
Environmental and societal well-being

Several toolkits, assessments and checklists have been developed to promote transparency around these choices. One example is the Assessment List for Trustworthy Artificial Intelligence (ALTAI). This is a self-assessment for organisations that assesses whether a system meets the seven principles of responsible AI listed above. Developed by the EU, it is intended for use by multidisciplinary teams of developers, specialists, end users, legal/compliance officers and management. At present, AI has not developed to the point where an algorithm can be held accountable for its decisions – there is no e-personhood, or legal personhood for an algorithm. The main responsibility for upholding these principles therefore still lies with companies and governments.

This blog focuses on the steps companies can take to develop and deploy responsible AI. There is much more to say about ethics and AI beyond what we cover here. Rather than looking at systems such as self-driving cars or drones, which face ethical dilemmas, we will look at everyday algorithms that people encounter regularly.

Garbage in, garbage out

First, let’s discuss bias. In this context, it means something like a distortion of reality. Poor data leads to poor models and outcomes. If an algorithm is used to recognise dogs in photos, but the model repeatedly misses photos that clearly show a dog or classifies a blueberry muffin as a chihuahua [2], that gives us reason to think the model is not working as well as it should. Perhaps the input data was poorly labelled or unrepresentative, there was something wrong with the model’s configuration, or there were problems with overfitting or underfittin

The consequences of misclassification in these examples are not far-reaching. But what if a facial recognition model is used to identify criminals? If the model wrongly identifies someone as a criminal, the consequences could unjustly change that person’s life. The problem may lie in how the algorithm is optimised, but a more insidious cause may be bias embedded in the training data. Machines learn only from what you show them – they do not account for nuanced contextual or cultural factors. An algorithm that assesses whether a female applicant is suitable for a role may give her a lower score if the model was trained on historical data in which men are overrepresented. Conversely, datasets in which minorities are overrepresented can lead an algorithm to label these people as ‘higher risk’. This can create a situation in which a group is more likely to go to prison or receive a longer sentence (causing these groups to be overrepresented in new data, increasing the risk that they will go to prison more often, which…).

Sources of potential bias in machine learning (ML) systems include [3]:

Skewed sample: bias in an initial dataset can grow stronger over time. For example, a police department is more likely to send officers to a neighbourhood with a high recorded crime rate. This makes it more likely that a crime will be detected. Even if more crime occurs in another neighbourhood, a lack of police presence can lead to a lower recorded crime rate there. This creates a positive bias towards neighbourhoods with less police presence.
Tainted examples: any machine learning system retains bias in historical data that stems from human bias. One example comes from Google Translate, which often produces stereotypical translations from languages with gender-neutral pronouns: she is a nurse, he is an engineer.
Limited features: if labels for a minority group are much less reliable than labels for the majority group, predictions are usually less accurate for the minority group.
Sample size disparity: if much less training data is available for the minority group than for majority groups, the model performs less well for the minority group.
Proxies: you may choose not to use sensitive information when training a model. Data such as postcodes can serve as a proxy for sensitive information such as ethnicity. Even so, including information such as postcodes during training can be useful. To illustrate: students from cities with a higher average income may have more access to homework support than children from disadvantaged neighbourhoods. If postcodes are not included in the training data, students from affluent cities are more likely to be admitted to a university because the model does not account for this inequality.

Organisations need to think carefully about the data used to train a model and the potential sources of bias in the dataset. The data must be representative. Preventing bias is, of course, ideal. Continued monitoring for bias in the algorithm can help minimise its influence on decisions. Data scientists develop the algorithms, but the responsibility cannot rest entirely on their shoulders. Everyone involved in an AI project has a part to play in actively preventing bias so that decisions are sound and ethical.

Transparency

A previously published blog on explainability and explainable AI (XAI) and why they matter describes how XAI provides insight into how decisions are made and which features play a role. This ‘explainability’ can help prevent unfair outcomes and identify errors in the system early. Explainability is related to transparency. While explainability focuses mainly on a model’s output, transparency is more about the processes that precede training an algorithm and putting it into production: the input data, analytical choices and how the algorithm works.

As an end user, you usually have little visibility into the training data: where it came from, how it was cleaned and exactly which features the model was trained on. You ‘just have to trust’ that the algorithm works in your interests. Being transparent with the public and allowing end users to provide feedback may help build trust in algorithms. The municipality of Amsterdam provides a good example with its algorithm register. The idea is that automated public services should respect the same principles as other municipal services – including being open and accountable, treating people equally, and not restricting their freedom or say in decisions.

Full transparency is not always possible, for example when algorithms are, or have to remain, secret. Take fraud detection at the Dutch Tax and Customs Administration. If it disclosed the algorithm, fraudsters could complete their tax returns in a way that would let them evade detection. For commercial companies, too, releasing an algorithm is not always desirable. Google has proposed using ‘model cards’ to give both experts and non-experts insight into models. Model cards can include information about a model’s performance, the data used, its limitations, ethical risks and the strategies used to address them. A Model Card Toolkit is available that works with scikit-learn to create reports like these. This approach could allow companies that do not want to reveal every detail of their algorithm to be more transparent about the algorithms they use. Even so, transparency is like ‘looking through a window’: you only see what the window lets you see.

Human oversight

Under Article 22 of the GDPR, the final decision generally cannot be left to an algorithm alone when it could have a significant impact on a person’s life:

“The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.”

In practice, people often find it difficult to go against an algorithm. After all, they may assume the computer knows better. So when a person has to review an algorithm’s decision, it is important that automatic approval is not enabled.

Conclusion

We live in a society where algorithms continually influence what we do. Algorithms are also increasingly making decisions that affect our lives, for example when we apply for a loan or mortgage. People, not the algorithm itself, bear the greatest responsibility for the ethical implications. It is therefore important to consider which data you use to train a model and to prevent potential bias from leading to unfair decisions. Within an organisation, this responsibility should not rest entirely with data scientists; everyone involved in the project should be part of the discussion. Being transparent about the choices made when developing an AI system creates opportunities for feedback and gives end users more confidence in the system. The organisation must document all the considerations behind those choices. People must also always be able to reverse a decision prepared by an AI system. There is a great deal of trust in AI, and we need to work to ensure that trust is justified.

Would you like to know more about this subject or see examples of how we approach it in our projects? Get in touch for more information, with no obligation!