Reflecting on the Use of AI: Ethics & Bias

Ethics in the context of artificial intelligence means developing and deploying AI systems in such a way that they respect human rights, transparency, accountability, and social values. It is not just a question of what is technically possible, but also of the impact that the use of AI can have on individuals and groups. Ethical AI requires clear areas of responsibility, transparent decision-making, data protection, and a critical assessment of potential risks. The aim is for AI to be used to support people, rather than putting them at a disadvantage or handing over control and responsibility to non-transparent systems.

Typical guiding questions include:

• What is AI being used for, and is this purpose useful to society?
• Who is responsible when an AI system makes incorrect or harmful decisions?
• Which data is being used, and has it been collected lawfully, securely and sparingly?
• Can users tell that AI was used?
• Are certain individuals or groups being disadvantaged, and are inequalities or injustices being exacerbated?
• Do humans remain capable of acting and taking responsibility when it comes to important decisions?
• Are the benefits of AI proportionate to the costs, resource consumption (water, electricity, etc.), and potential side effects?

AI bias

AI systems can be useful for research, teaching, academic studies, and administration. How they work depends on many factors. Several of these can lead to errors or injustices. We call this ‘bias’ – i.e. distortions or prejudices. That is why AI systems should be examined critically, and content should not be adopted without review.

What does ‘bias’ mean?

Bias refers to systematic distortions in data, which may also reflect prejudices or social structures. When an AI programme delivers incorrect or unfair output because it disadvantages certain people or groups, it may be due to bias. This happens, for example, when AI systems have learned from data that is not balanced. For example, there may be a disproportionately high number of men in a database, or certain language patterns may be over-represented.

Examples: 

An AI programme for literature research that prioritises literature by Anglo-American authors.

The creation of AI-generated personas for marketing purposes produces images of people who conform to Western beauty standards, because the AI has learned that people are supposed to conform to this one beauty standard.

Important:

AI is not automatically ‘neutral’ or ‘objective’. It reflects what it has learned – and that may be wrong.

Responsible use of AI

AI-generated content must not simply be ‘copied’. Academic work requires content to be understood and verified, and someone has to take responsibility for it. For example: A student uses AI to write a summary – but doesn't check it and hands it in. That’s basically equivalent to plagiarism. Content and how it was developed should be verifiable, and the use of AI should remain transparent. Simply reproducing AI-generated content or text increases the risk of failing exams.

Recommendations:

• Use diverse, representative, or justified data
• Check where the data comes from
• Strategically research literature that does not conform to mainstream concepts
• Prompt in a targeted manner to reveal bias
• Critically reflect the use of AI and its output, rather than simply copying it
• Document the use of AI and the decisions made regarding data selection (if applicable, see the Institute’s guidelines and documentation templates)

Bias in AI-generated teaching materials (text and images)

AI can generate texts, tasks, or examples – but in doing so, it may use stereotypical representations, for example, using only men to depict engineers. One reason for this may lie in the AI prompt.

The prompt used to generate content may unconsciously contain biases. For example: “Write a summary about women in science”. AI could focus here on ‘women in science’ as a ‘minority’, rather than portraying them as equal members of the scientific community. Such distortions are also reflected in materials: an AI tool generates an assignment on ‘energy generation’ featuring images of men in laboratories – with no representation of females or other kinds of gender identity.

Equal opportunities and inclusion

AI systems can support learning processes. Nevertheless, they require teachers and learners to possess a range of technical, linguistic and digital skills. In order to ensure equal opportunities for all learners, teachers must take students’ individual circumstances into account. Tailoring the approach to teaching and learning processes can help to promote equal opportunities and foster AI skills.

Protect sensitive data

The use of AI requires careful handling of personal data. Information about students, such as names, academic records, student ID numbers, or other personal details, must not be entered into AI systems. This poses the risk of breaching informational self-determination and violating privacy. The data entered could be stored unintentionally and processed by the system for other purposes. This is not always apparent to the users. Users should therefore exercise particular caution. 

Recommendations:

• Bias-sensitive prompting
• Critically review AI-generated texts
• Use a variety of examples and inclusive language
• Consider the different circumstances of the students 
• Take data protection and confidentiality into account
• Avoid stereotypical depictions when generating images using AI. We particularly recommend this guide to using AI for visual depictions:

Distorted data and prompts

AI learns from data. If the data being used is not sufficiently diverse – for example, if it comes from only one country, one age group, or one gender – the AI may draw incorrect conclusions. An AI model for medical diagnosis that has been mainly fed with data concerning white men tends to perform less accurately for women or people of other ethnicities.

AI responds to research questions. If researchers ask a question that is shaped by their own biases, this can influence the AI – and lead to distorted results. For example: A study that targets only students with an ‘interest in technology’ ignores other perspectives.

Copyright 

AI models are often trained using millions of texts, images or videos – often without the consent of the authors. This may pose legal problems and ethical issues. For example, an AI programme that writes academic texts may be based on works that have not been released for use by the general public.

Mainstreaming in literature research

When conducting literature research with AI, English-language and Western literature tends to dominate. Moreover, dominant studies, models, theories, and researchers are often given preferential treatment. When conducting a literature search, you might therefore only be presented with open-access publications from the Anglo-American world.

Recommendations:

• Use diverse, representative or justified data
• Check where the data comes from
• Document decisions made during data selection
• Strategically research literature that does not conform to mainstream concepts
• Prompt in a targeted manner to reveal bias

Sensitive and personal data

AI systems can assist the administration with analysing data, drafting texts or organising information. Sensitive and personal data, concerning e.g. students or (prospective) staff, must be treated with a special degree of caution. Particular attention should be given to how they are processed. Information must therefore be provided regarding the processing in specific systems. The administration must remain responsible for decision-making in administrative procedures. 

Bias in communication

One problem with generating text using AI is that AI-generated emails or chatbots may not be able to communicate effectively with certain groups – for example, due to the use of formal language or technical terms.

AI-generated images are frequently used in marketing communications and website design. These visualisations are aimed at specific groups and thus subtly reinforce stereotypes.

Example: An institute is using AI-generated images to promote itself on its website. All the people depicted conform to Western beauty standards.

Recommendations:

• If AI is used, inform all parties involved
• Ensure data protection
• Do not leave decisions to AI, responsibility remains with the administration
• Avoid stereotypical depictions when generating images using AI. We highly recommend the guide on the right for using AI to produce images.