Algorithms determine your social media feed, tell you what your next purchase should be on Amazon and how best to get from A to B.
There is a lot of trust in AI, but is that trust justified?
In addition, AI is also being used to make potentially life-changing decisions about the risk of relapse, making a medical diagnosis or how your self-driving car reacts in the event of a dangerous situation.Algorithms often make the right decision, but when it goes wrong, it can have irreversible consequences in certain situations.
Consider the following examples:
- At the Inland Revenue, an automatic risk selection system determined which benefit applications should be checked extra.
The question about dual nationality affected the outcome of the algorithm, which contributed to the child care benefits affair. - An American teenager, based on her purchases at a supermarket chain, received discount coupons for baby clothes in the mail.
Her father found out at home that his daughter was pregnant even before she told him. - Amazon’s system that reviews resumes of potential employees gave women a lower chance of being hired.
That’s because the model was trained on historical data, covering mostly white and male workers.
AI has great power and therefore comes with great responsibility.
Ethical questions surrounding AI are no longer just philosophical questions, but nowadays also have practical significance that should be examined by governments and companies.
To this end, the EU has established seven pillars of responsible AI compiled[1]:
- Human choice and oversight
- Technical robustness and safety
- Privacy and data management
- Diversity, no discrimination and fairness
- Transparency
- Accountability
- Environmental and social welfare
For example, there is the Assessment List for Trustworthy Artificial Intelligence (ALTAI).
This is a self-assessment for organizations that assesses whether a system meets the above seven principles of responsible AI.
This assessment was developed by the EU and is intended to be applied by multidisciplinary teams of developers, specialists, end users, legal/compliance officers and management.At the moment, the landscape of AI is not yet developed to the point where an algorithm can be held accountable for decisions made – there is no e-personhood yet, an algorithm as a legal person.
So the biggest responsibility at this point still lies with companies and governments to keep the principles up.
This blog focuses on the actions companies can take to develop and deploy responsible AI.
In addition to this depth, there is much more to write about ethics and AI.
In this case, we will not focus on systems such as self-driving cars or drones, which face ethical dilemmas, but will look at “home, garden and kitchen algorithms” that our society faces every day.
Garbage in, garbage out
First, we discuss the topic of bias. In this context, that means something like a distortion of reality.
Bad data leads to bad models and outcomes.
If an algorithm is deployed to recognize dogs in pictures, and the model repeatedly misses pictures that clearly show a dog or classifies a blueberry muffin as a chihuahua [2], that gives a reason to think that a model is not working optimally.
Perhaps the input data is poorly labeled, is not representative, there was something wrong with the configuration of the model, or there were problems with over- or underfittin The consequences of misclassification from these examples are not far-reaching.
But what if a facial recognition model is used to identify criminals?
If a person were wrongly mistaken for a criminal by this model, the consequences could be unfairly life-altering.
The cause of this problem may lie in optimizing the algorithm, but a more vicious cause may lie in bias embedded in the training data.
Machines learn only from what you show them – they do not incorporate nuanced contextual or cultural factors.
An algorithm that determines whether a female applicant might be suitable for a job might give her a lower score if the model was trained on historical data in which men are overrepresented.
Conversely, datasets in which minorities are overrepresented may lead to situations in which an algorithm labels these individuals as “higher risk.
As a result, a situation may arise in which a group is more likely to be imprisoned or given a longer sentence (thus overrepresenting these groups in new data, increasing the risk that these groups will be imprisoned more often, thus…). Sources of possible bias in machine learning (ML) systems are [3]:
- Skewed sample: bias in an initial dataset can be amplified by time.
For example, a police department is more likely to send officers to a neighborhood where the amount of crime is high.
This makes it more likely that some form of crime will actually be detected.
Even if more crime is present in another neighborhood, the lack of police presence may lead to a lower recorded crime rate.
This creates a positive bias toward neighborhoods with less police presence. - Tainted examples: any ML system retains the bias that exists in the historical data due to bias by humans.
One example comes from Google Translate, which, for languages with gender-neutral pronouns, often stereotypes translations: she is a nurse, he is an engineer. - Limited features: if the reliability of the label for a minority group is much lower than the reliability of labels for the majority group, the accuracy of predictions is usually lower in the minority group.
- Sample size disparity: if there is much less training data for the minority group than is available for majority groups, the model does not work as well on the minority group.
- Proxies: a choice can be made to deliberately not use sensitive information when training a model.
Data on zip codes, for example, can serve as a proxy for sensitive information such as ethnicity.
Nevertheless, it can be interesting to include information such as zip codes during training.
To illustrate, schoolchildren from cities with higher average incomes would have more opportunities for homework help than children from disadvantaged neighborhoods.
When zip codes are not part of the training data, schoolchildren from affluent cities are more likely to be accepted into college because the model does not take this inequality into account.
It is important that organizations consciously think about the data on which a model is going to be trained and what possible sources of bias are present in the dataset.
The data used should be representative.
Avoiding bias is, of course, ideal.
Continued monitoring of bias within the algorithm can ensure that decisions are influenced as little as possible.
Data scientists develop the algorithms, but the responsibility in this cannot rest entirely on their shoulders.
It is up to everyone involved in an AI project to actively avoid bias in order to make correct and ethical decisions.
Transparency
A previously published blog on explainability and explainable AI (XAI) and its importance describes how XAI provides insight about the formation of decisions and what features play a role in this.
This “explainability” can ensure that unfair situations can be avoided and errors in the system can be tracked in advance.
Explainability is related to transparency.
Whereas explainability is mainly about the output of a model, transparency is more about the processes that preceded training and putting an algorithm into production: the input data, analysis choices and the operation of the algorithm.
As an end user, you usually have little visibility into the training data, where it came from, how it was cleaned and exactly what features the model was trained on.
You “just have to trust” that the algorithm is working in your best interest.
Transparency to the outside world and the ability for end users to provide feedback may promote trust in algorithms.
The City of Amsterdam provides a good example with their algorithm registry.
The idea behind this is that automated services should respect the same principles as other services provided by the municipality – including being open and verifiable, treating people equally and not restricting freedom and control.
Full transparency is not always possible, for example, when algorithms (must) be secret.
Take again the fraud detection at the Tax Office.
If the Inland Revenue were to allow access to the algorithm, fraudsters could fill out their returns precisely so they could bypass detection.
For commercial companies, releasing the algorithm is not always desirable either.
Google has proposed using “model cards” to provide transparency about models to both experts and non-experts.
Model cards can include information about a model’s performance, the data used, limitations of the model, ethical risks and strategies used to overcome them, among other things.
A Model Card Toolkit is available that works with scikit-learn to create such reports yourself.
Such an approach would allow companies that do not want to reveal their entire algorithm to still be more transparent about the algorithms used.
Still, transparency is similar to “looking through a window”: you only see what the window’s window shows.
Human supervision
According to Article 22 of the AVG, as a general rule, the final judgment should not come solely from an algorithm when it comes to decisions that may have far-reaching effects on a person’s life: ” The data subject has the right not to be subject to a decision based solely on automated action, including profiling, which produces legal effects concerning him or her or significantly affects him or her in any other way.” In reality, people often find it difficult to deviate from the algorithm.
After all, the computer will know better.
So it is important that automatic consent is not set in place when a human has to evaluate an algorithm’s decision.
Conclusion
We live in a society where our actions are constantly influenced by algorithms.
Algorithms also increasingly make decisions about our lives, for example, when applying for a loan or mortgage.
People, and not the algorithm itself, have the greatest influence on moral responsibility.
It is therefore important to consider the use of certain data to train a model and avoid the possible presence of bias that leads to unfair decisions.
Within an organization, the responsibility here should not lie entirely with data scientists, but there should be a dialogue with all those involved in the project.
Transparency about choices made in developing an AI system allows for feedback and gives the end user more confidence in the system.
It is up to the organization to document all considerations.
Furthermore, it should always be possible for humans to reverse a decision prepared by an AI system.
There is a lot of trust in AI and we need to make sure that trust is justified.
If you want to learn more about this topic or be inspired about examples of how we approach this in our projects, feel free to contact us for more information!
