Medizinische Universität Graz Austria/Österreich - Forschungsportal - Medical University of Graz
Gewählte Publikation:
SHR Neuro Krebs Kardio Lipid Stoffw Microb
Huelser, M; Mueller, H; Díaz-Rodríguez, N; Holzinger, A.
On the disagreement problem in Human-in-the-Loop federated machine learning
J IND INF INTEGR. 2025; 45: 100827
Doi: 10.1016/j.jii.2025.100827
Web of Science
FullText
FullText_MUG
- Führende Autor*innen der Med Uni Graz
- Holzinger Andreas
- Co-Autor*innen der Med Uni Graz
- Müller Heimo
- Altmetrics:
- Dimensions Citations:
- Plum Analytics:
- Scite (citation analytics):
- Abstract:
- The popularity of Artificial Intelligence (AI) has risen sharply in recent years, revolutionizing applications in most sectors with unprecedented functionalities. Milestones and achievements like ChatGPT demonstrate not only the impressive capabilities of AI, but also how accessible such technologies have become in recent times. However, the success of AI applications depends heavily on the underlying information integration processes. Among the most important processes are the training of the AI model at the core of the application and the collection and pre-processing of training data. In particular, the task of collecting high-quality training data can be very costly and resource-intensive, as in many cases large amounts of data have to be annotated manually. Human annotators must have extensive expertise for certain tasks in order to provide high-quality training data. In this paper, we present a framework to maximize the efficiency of human experts in a Machine Learning (ML) scenario, with the aim of optimizing the use of human expertise in active learning. This is done by constantly measuring the quality of human experts' input, as well as by involving human annotators only when needed. We showcase the benefits of our proposed framework by applying it to a problem in image classification, proving its usefulness to reduce the cost of annotating training data. The source code of the framework is publicly available at https://github.com/human-centered-ai-lab/app-HITL-annotator.
- Find related publications in this database (Keywords)
- Human-Centric AI
- Human-in-the-Loop
- Human-machine interaction
- Counterfactuals
- Active Learning