In 2021, Toi Āria was commissioned by the Digital Council of Aotearoa to undertake a co-design engagement, to help answer the question: what is needed to ensure people in Aotearoa New Zealand have the right levels of trust required to harness the full societal benefits of digital and data-driven technologies?

Digital Council of Aotearoa’s role is to advise the government of New Zealand on how it can bring about an Aotearoa in which; digital and data-driven technologies are designed and used with ambition and care in ways that grow trust, embrace innovation, increase equity and give effect to Te Tiriti o Waitangi; individuals, whānau and communities have the resources and opportunities to achieve their aspirations; innovators build tools that improve the lives of people and the planet; and the government ensures technology is used to build a more equitable and inclusive Aotearoa New Zealand.

The Council chose automated decision-making (ADM) as a case study to help focus its research. Broadly speaking, ADM refers to decision-making systems where parts of the process are carried out by computer algorithms. ADM is everywhere and has big impacts on our lives. It informs what we see on our social media feeds and what price we are offered when booking a hotel online. It also informs big decisions, like whether people are approved for parole.

Most research on ADM was produced in an academic context and does not usually include hearing directly from people who are most affected by ADM. The Council wanted to hear from a diverse range of people, most of whom do not usually get a seat at the table when it comes to designing and implementing ADM systems.

The team at Toi Āria ran a series of in-person and online workshops with 186 people during 2020. Workshop attendees included: blind and vision impaired; ethnic community leaders and youth; Māori and Pacific youth; women with migrant and refugee backgrounds; Whānau Ora navigators (Māori health advocates); young people with care experience; and members of the general public.

Workshop participants not only shared their insights about how automated decision-making should be used now, but also their vision for building a better digital future that centres on the needs and aspirations of people. The process used Toi Āria’s Comfort Board methodology, and is based on the principle that the people destined to use or be affected by a system should play a critical role in designing it.

With the Council, Toi Āria developed six scenarios based on real-world situations where some aspects of the decision-making processes are carried out or informed by computer algorithms including: media consumption; recruitment; youth support; immigration, parole decisions; and surgical waiting lists.

The findings identified several areas of concern, and what was needed to provide comfort to New Zealanders. A key concern across the workshops was around bias and discrimination, and for these to be allayed people needed more transparency and better communication, as well as better involvement and representation. People appreciated that automated decision-making is useful for processing data at speed and at scale, and as an ‘assistant’ to people, but people worry that systems, programmers and decision-makers can introduce bias into these.

As a result of the research, the Digital Council for Aotearoa has provided seven recommendations to the Government.

For more information about the findings go to: https://digitalcouncil.govt.nz/advice/reports/towards-trustworthy-and-trusted-automated-decision-making-in-aotearoa/