“Take Nothing on Its Look”: Revealing Users’ Expectations and Experiences in Social Human–Robot Interaction Jessica Lindblom, Julia Rosén, Maurice Lamb, Erik Billing ACM Transactions on Human Robot Interaction, 2026 The use of social robots in many sectors of society is predicted to progressively increase. Therefore, exploring how expectations play a role in and change users’ experiences when interacting with these robots over time is necessary. From an interpretative and insight-driven approach, our aim was to explore how humans experience in-person interactions with the social robot Pepper, which was equipped with the OpenAI GPT-3 language model. Qualitative data from 62 video recordings of the interactions with Pepper and post-test interviews were collected from 31 participants. An experiential reflexive thematic analysis was applied. The main findings include various levels of interaction quality, different interaction strategies, and elements influencing the users’ expectations and experiences, which were synthesized into a coherent framework. It appears that the participants adapted their interaction strategies based on their expectations and the perceived capability of the robot, which influenced their experiences. This reveals that positive user experience is not solely determined by interaction quality, showing the interplay among these aspects when interacting with a social robot. To conclude, our findings underscore the intricate nature of the role of user expectations and experiences in social human–robot interaction. The work adds complementary qualitative approaches to the Human–Robot Interaction community to provide additional insights on interacting with social robots.
Efficacy and effectiveness of robot-assisted therapy for autism spectrum disorder: From lab to reality Daniel David, Paul Baxter, Tony Belpaeme, Erik Billing, Haibin Cai, Hoang-Long Cao, Anamaria Ciocan, Cristina Costescu, Daniel Hernandez Garcia, Pablo Gómez Esteban, James Kennedy, Honghai Liu, Silviu Matu, Alexandre Mazel, Mihaela Selescu, Emmanuel Senft, Serge Thill, Bram Vanderborght, David Vernon, Tom Ziemke Science Robotics, 2025 The use of social robots in therapy for children with autism has been explored for more than 20 years, but there still is limited clinical evidence. The work presented here provides a systematic approach to evaluating both efficacy and effectiveness, bridging the gap between theory and practice by targeting joint attention, imitation, and turn-taking as core developmental mechanisms that can make a difference in autism interventions. We present two randomized clinical trials with different robot-assisted therapy implementations aimed at young children. The first is an efficacy trial ( n = 69; mean age = 4.4 years) showing that 12 biweekly sessions of in-clinic robot-assisted therapy achieve equivalent outcomes to conventional treatment but with a significant increase in the patients’ engagement. The second trial ( n = 63; mean age = 5.9 years) evaluates the effectiveness in real-world settings by substituting the clinical setup with a simpler one for use in schools or homes. Over the course of a modest dosage of five sessions, we show equivalent outcomes to standard treatment. Both efficacy and effectiveness trials lend further credibility to the beneficial role that social robots can play in autism therapy while also highlighting the potential advantages of portable and cost-effective setups.
Editorial: Human factors and cognitive ergonomics in advanced industrial human-robot interaction Erik Billing, Federico Fraboni, Luca Gualtieri, Patricia Helen Rosen, Peter Thorvald Frontiers in Robotics and AI, 2025 Collaborative robotics is a very promising technology for many industrial processes, including e.g., manufacturing, logistics, or construction. This new technology are also changing the environment for workers in industry. Research on human-robot interaction (HRI) will be crucial for enhancing the operator's work conditions and well-being, as well as production performance. In that regard, human factors, with a special emphasis on cognitive ergonomics are fundamental to implementing safe, fluent, and efficient collaborative applications. This Research Topic gathers a range of contributions on the study of Human Factors and Cognitive ergonomics in user-centered and collaborative applications in industrial settings. Here, we summarize these studies from the perspective of three pivotal areas impacted by collaborative robotics: workers' safety, performance, and well-being. The Reseach Topic provides a timely analysis of the changing landscape of industrial HRI as we stand on the cusp of a new era in industrial automation, defined by the fusion of human ingenuity and robotic efficiency. The contributions within offer practical insights and forwardthinking perspectives on how collaborative robotics can transform industrial workspaces in the future, in addition to reflecting state-of-the-art research in the field. A different aspect of this intricate relationship is covered by each article in this issue, from the social and psychological effects of incorporating robots into human-centered work environments to the complexities of design and implementation. Developing solutions that are both technologically sophisticated and human-centered requires a holistic approach, which is crucial for comprehending the complex nature of HRI.Before delving into the particulars of each contribution, we invite the reader to this brief summary, briefly presenting each contribution to the research topic through the lenses of safety, performance, and well-being.We hope that this will support reflections on the wider societal implications of HRC development, in addition to their technical and ergonomic aspects. A harmonious balance between human needs and machine capabilities will be key to the future of industry.In the field of Human Factors and Cognitive Ergonomics, introducing advanced collaborative robotic systems in production environments necessitates reevaluating safety from different perspectives, namely safety perceptions of workers, safety behaviours and mechanical safety. Integrating this technology in various industrial environments, such as manufacturing and logistics, prompts a critical examination of the interplay of the different elements interacting in the socio-technical system. As with any human-system interaction in the work context, a more ergonomic and anthropocentric system (characteristics that can be measured through optimisation of associated cognitive factors) implies greater safety in terms of prevention and mitigation of potential mechanical risk (understood as collisions, crushing, entrapment, etc.) and psychosocial risk as defined by Occupational Safety and Health Administration (OSHA) such as excessive workload, lack of control, job insecurity or insufficient communication .The present special issue includes diverse studies, each exploring different aspects of safety in human-robot collaboration.The contribution by Mirnig et al. (2023) constitutes an excellent opening to the special issue. While focusing on automated material handling vehicles, Mirnig et al. discuss many design aspects that are applicable also to HRI more broadly, including contextual factors such as purpose and context of use, and many aspects of the interaction itself. The study by Onnasch et al. (2023) investigates how directing a worker's attention to specific targets with gaze communication can improve safety in humanrobot interaction by, first of all, suggesting how robotic eye design could affect operator attention and perceived cognitive workload. Furthermore, the paper indirectly suggests how robotic eyes could potentially prevent mechanical risks like collisions and entrapments. According to research, an operator's situational awareness and capacity to anticipate and respond to possible hazards are enhanced when they focus on anthropomorphic robot eyes. This study highlights anthropomorphism's contribution to improving operator safety and attention, leading to safer and more conscious HRIs in industrial settings. On the effect of anthropomorphic features in collaborative robots, the paper by Roesler (2023) examines the impact of anthropomorphic versus technical framing of robots on operators' trust, particularly in the context of robot failures. The study concludes that although the general levels of trust between technically framed and anthropomorphically framed robots did not significantly differ, people perceived the anthropomorphically framed robots as being more transparent, particularly after understandable failures. Because it improves operators' awareness and skill in anticipating and responding to potential mechanical risks like collisions or entrapments, this increased perceived transparency and positive perception in the event of understandable failures by potentially contributing to increased safety in HRIs. In a complementary way, Freire et al. (2024) also addresses the importance of safety in human-robot collaboration, but through a different mechanism.Their proposed cognitive architecture incorporates a "Socially Adaptive Safety Engine," which dynamically adjusts safety parameters like distance and robot speed based on the worker's trust level and preferences. While Roesler's study emphasizes how transparency in robot behavior following failures can enhance safety, Freire et al. go further by actively modifying robot behavior in real-time to adapt to each worker's trust and comfort, creating a more personalized and context-sensitive safety environment. Together, these articles suggest that fostering both transparency and adaptability in robots-through anthropomorphic design and context-aware systems-can significantly enhance operator safety and well-being in industrial environments.In a comprehensive perspective, Heinold et al. (2023) discusses various occupational safety and health (OSH) risks and benefits associated with the integration of robotic systems in industrial settings. These include both physical risks, such as collisions and mechanical failures, and psychosocial risks, including mental stress and job insecurity, which can arise from the use of advanced robotics in workplaces. The study also explores opportunities, such as the potential for reducing physical strain and improving longterm physical health by automating physically demanding tasks. The peculiarity of this manuscript lies in its comprehensive analysis of both physical and psychosocial OSH risks and opportunities, uniquely incorporating workers' expectations alongside evidence from the literature, offering a dual perspective on the safety implications of HRI. On a similar note, also addressing logistics and agricultural domains in addition to the manufacturing one, Pietrantoni et al. (2024) investigated experts' opinions regarding collaborative robotics safety considerations. Their study emphasized the critical role of tailored safety protocols, highlighting the need for advanced collision avoidance systems, failsafe mechanisms, and emergency stop protocols. Key aspects in agriculture include stability control and navigation on uneven ground for the safety and efficiency of workers. This sectoral approach completes the dual perspective taken by Heinold et al. in that it details how diverse industrial working contexts require tailor-made safety solutions to address both physical risks and ergonomic challenges and further promote the safe integration of robotics into complex work environments.The impact of human autonomy and robot work pace on job quality in collaborative settings is examined by Van Dijk et al. (2023). They find that higher human autonomy levels correlate with lower perceived workloads. The present article generally addresses some of the main working conditions leading to psychosocial risks according to OSHA, namely excessive workloads, lack of involvement in making decisions that affect the worker, and lack of influence over the way the job is done. This study shows that increasing human autonomy and modifying robot work pace can effectively reduce cognitive and temporal demands on workers. It compares scenarios of human-led work, fast-paced robot-led work, and slow-paced robot-led work. According to these results, reducing workload is linked to a lower mechanical risk because there is a lower probability of mistakes in HRI. This suggests that such measures optimise perceived workload and improve safety in collaborative scenarios.In the context of an industrial defect inspection task, the article of Cymek et al. (2023) examines the phenomenon of decreased individual effort and attention in human-robot collaborative tasks. The study finds that individuals searching for defects with a robot partner may have been less focused and exerted more mental energy than those searching alone, who on average, found more defects. Because less alert workers may be more likely to overlook safety hazards in their environment. This lower level of attentiveness and operational performance in human-robot teams affects productivity and may increase exposure to mechanical risks.workload. The article's relevance is critical, considering that collaborative robotics is one of the most promising technology for retaining the ageing workforce and maintaining an appropriate quality of work. It finds that senior workers have a strong acceptance of technology and positive experiences during increased cognitive demand. As a result of increased mental demand during dual-task collaboration, the study found that task errors and duration increased despite these favourable perceptions. This might have detrimental effects on safety behaviours. While senior workers are generally open to working with robots, this increased cognitive workload-as indicated by eye tracking and cardiac activity-indicates that overburdening from collaboration may result in overwork and increase the mechanical risks in the workplace.For human-robot interaction to be considered successful, assessing and supporting the performance of the system as a whole is of utmost importance. In fact, one might even say that successful performance of the system is a necessary requisite when arguing for its existence. Successful performance can be defined in many different ways but in essence it is the combination of two things; doing things accurately (effective), and being efficient while doing it. In the context of collaborative human-robot settings, this research topic investigates relations between human-factors and performance in terms of temporal performance and cognitive load (Van Dijk et al., 2023;Pluchino et al., 2023), collaborative setting and error rate (Cymek et al., 2023), as well as collaborative setting and perceived workload (Van Dijk et al., 2023). While all these papers are mentioned above in relation to safety, they also bring relevant results in relation to performance. Van Dijk et al. (2023) show a positive correlation between temporal performance and cognitive load, comparing two conditions with a fast vs slow scheduling for the HRC setup. Pluchino et al. ( 2023) analyze the performance in terms of errors and time on task of senior workers engaged in a sequential collaborative manufacturing task together with a cobot. A dual task condition where the subjects were challenged with a secondary mathematical assignment is compared to a single task (control) condition. Results show that the dual task condition lead to increases in both errors and time spent on task, which corresponded with higher levels of perceived mental effort. However, no differences in perceived performance, as assessed by the NASA-TLX questionnaire, were found between the conditions. Cymek et al. (2023) compares two versions of an inspection task, one collaborative where a human operator is working together with a robot, and one individual where the operator is working alone. Results show lower performance for the collaborative setting in terms of fewer identified defects during inspection, indicating an reduction in cognitive load compared to the individual condition.As previously discussed, the effects on performance of different types of collaborative queues are investigated by Onnasch et al. (2023). An indirect argument is made for faster reallocation of attention as a result of naturalistic attentional queues leading to increased performance. This paper also provides a brief argumentation that some queues used to improve collaboration, e.g., legible motion, may directly impact performance in a negative way, while robot eyes does not.Finally, in their study of technical expert's opinions of HRC also mentioned earlier, Pietrantoni et al. (2024) found that the introduction of collaborative robots is expected to bring improved efficiency and better worker conditions, e.g. as a result of automation of physically demanding operations. While the participants in the study generally held a positive attitude towards collaborative robots, the increased efficiency was also linked to concerns of job displacement and the need for reskilling.A key concern of cognitive ergonomics is to reduce negative effects of work. This also specifically refers to deployed technologies at the workplace, like advanced robotic systems. However, a truly humancentered approach to workplace and technology design aims at developing a person's personality and fostering individual and organizational health in its broadest sense. A holistic understanding of health goes beyond the physical safety of humans, but includes mental and social well-being of humans. In the everevolving landscape of human-robot interaction, the integration of advanced robotics to different workplaces, raises critical questions about how the well-being of individuals might be affected. This research topic includes different publications, each shedding light on different facets of human-robot-interaction and its implications for the human experience thus potentially leading to well-being in the long-term.As mentioned earlier, Heinold et al. (2023) address the question which psycho-social consequences are associated with a close interaction between humans and robots. By combining scientific perspectives through a literature review and insights from workers' expectations, the study provides a holistic view of the implications of task automation via robotic systems. The findings highlight the psycho-social impacts advanced robotics may have on workers. It becomes clear, that the aspects of task design and function allocation as well as the specific interactions design of systems as well as operation and supervision design are relevant sources potentially affecting the specific user experience and the well-being of workers in the long run.When further considering potential psychological effects, assessing traditional workplace factors can be beneficial. From human factors research it is well understood, that the level of job control or autonomy within a given task is a strong determinant for job quality and well-being (Van Der Doef and Maes, 1999). This also applies to industrial tasks (Rosen and Wischniewski, 2019). As working tasks are newly allocated between humans and robots, human autonomy levels can change. The investigation of human autonomy and robotic work pace by Van Dijk et al. (2023) discussed earlier is also relevant from a well-being perspective.The research underscores the significance of autonomy and work pace in shaping job quality, emphasizing the importance of designing collaborative scenarios that prioritize human autonomy and adjustments to the robot's work pace to optimize workload and enhance overall well-being.Exploring psycho-social effects more on a team level in this research topic is done by Cymek and colleagues. Their contribution focuses on the well-studied phenomenon of social loafing (Cymek et al., 2023). Using a visual-search task, the presented study investigates whether reduced individual effort, the phenomenon in question, which is commonly observed in human teams, also occurs in human-robot teams.The findings suggest that working with a robot team partner may lead to less attentive task execution, highlighting the need to address mental effort and attention allocation in human-robot collaboration to ensure optimal performance and, consequently, well-being.A human-centred technology design can contribute to a positive human-robot interaction and thus ensure a seamless workflow. One very relevant aspect of robot design which is touched upon in research is the application of anthropomorphic design features (Roesler et al., 2021). Two papers of this research topic explore the unique effects of anthropomorphic features in human-robot-interaction on different aspect of the distinct interaction quality and user experience. As mentioned earlier, Onnasch et al. (2023) examine how the design of predictive robot eyes influences human attention. The results indicate that anthropomorphic features contribute to a smooth interaction experience. Anthropomorphic robotic eyes trigger reflexive attention reallocation, hinting at a social and automatic processing of artificial stimuli, emphasizing the emotional and cognitive impact of such interactions on well-being. Through their analysis of anthropomorphic framing discussed earlier, Roesler (2023) show that an adequate level of trust within human-robot-interaction is also an important element contributing to a smooth interaction and a humancentered design. In this paper the perceived transparency of anthropomorphic robots emerges as a key factor, underscoring its role in shaping individuals' well-being.A novel design approach in order to facilitate socially adaptive robot behaviour in industrial settings is presented by Freire and colleaguesFreire et al. (2024). The authors present a theoretical cognitive architecture for robotic actions control, highlighting modules that among others take into account human preferences and situational awareness and by thus can adapt to human needs. The presented cognitive architecture is integrated into a recycling plant use case for disassembly tasks showcasing the basic functionalities of the systems. In the piloted use cases, the architecture demonstrated key Frontiers
Previous Experience Matters: An in-Person Investigation of Expectations in Human–Robot Interaction Julia Rosén, Jessica Lindblom, Maurice Lamb, Erik Billing International Journal of Social Robotics, 2024 The human–robot interaction (HRI) field goes beyond the mere technical aspects of developing robots, often investigating how humans perceive robots. Human perceptions and behavior are determined, in part, by expectations. Given the impact of expectations on behavior, it is important to understand what expectations individuals bring into HRI settings and how those expectations may affect their interactions with the robot over time. For many people, social robots are not a common part of their experiences, thus any expectations they have of social robots are likely shaped by other sources. As a result, individual expectations coming into HRI settings may be highly variable. Although there has been some recent interest in expectations within the field, there is an overall lack of empirical investigation into its impacts on HRI, especially in-person robot interactions. To this end, a within-subject in-person study ($$N=31$$ N = 31 ) was performed where participants were instructed to engage in open conversation with the social robot Pepper during two 2.5 min sessions. The robot was equipped with a custom dialogue system based on the GPT-3 large language model, allowing autonomous responses to verbal input. Participants’ affective changes towards the robot were assessed using three questionnaires, NARS, RAS, commonly used in HRI studies, and Closeness, based on the IOS scale. In addition to the three standard questionnaires, a custom question was administered to capture participants’ views on robot capabilities. All measures were collected three times, before the interaction with the robot, after the first interaction with the robot, and after the second interaction with the robot. Results revealed that participants to large degrees stayed with the expectations they had coming into the study, and in contrast to our hypothesis, none of the measured scales moved towards a common mean. Moreover, previous experience with robots was revealed to be a major factor of how participants experienced the robot in the study. These results could be interpreted as implying that expectations of robots are to large degrees decided before interactions with the robot, and that these expectations do not necessarily change as a result of the interaction. Results reveal a strong connection to how expectations are studied in social psychology and human-human interaction, underpinning its relevance for HRI research.
Kinematic Primitives in Action Similarity Judgments: A Human-Centered Computational Model Vipul Nair, Paul Hemeren, Alessia Vignolo, Nicoletta Noceti, Elena Nicora, Alessandra Sciutti, Francesco Rea, Erik Billing, Mehul Bhatt, Francesca Odone, Giulio Sandini IEEE Transactions on Cognitive and Developmental Systems, 2023 This article investigates the role that kinematic features play in human action similarity judgments. The results of three experiments with human participants are compared with the computational model that solves the same task. The chosen model has its roots in developmental robotics and performs action classification based on learned kinematic primitives. The comparative experimental results show that both model and human participants can reliably identify whether two actions are the same or not. Specifically, most of the given actions could be similarity judged based on very limited information from a single feature domain (velocity or spatial). Both velocity and spatial features were however necessary to reach a level of human performance on evaluated actions. The experimental results also show that human performance on an action identification task indicated that they clearly relied on kinematic information rather than on action semantics. The results show that both the model and human performance are highly accurate in an action similarity task based on kinematic-level features, which can provide an essential basis for classifying human actions.
Language Models for Human-Robot Interaction Erik Billing, Julia Rosén, Maurice Lamb ACM IEEE International Conference on Human Robot Interaction, 2023 Recent advances in large scale language models have significantly changed the landscape of automatic dialogue systems and chatbots. We believe that these models also have a great potential for changing the way we interact with robots. Here, we present the first integration of the OpenAI GPT-3 language model for the Aldebaran Pepper and Nao robots. The present work transforms the text-based API of GPT-3 into an open verbal dialogue with the robots. The system will be presented live during the HRI2023 conference and the source code of this integration is shared with the hope that it will serve the community in designing and evaluating new dialogue systems for robots.
How to train a self-driving vehicle: On the added value (or lack thereof) of curriculum learning and replay buffers Sara Mahmoud, Erik Billing, Henrik Svensson, Serge Thill Frontiers in Artificial Intelligence, 2023 Learning from only real-world collected data can be unrealistic and time consuming in many scenario. One alternative is to use synthetic data as learning environments to learn rare situations and replay buffers to speed up the learning. In this work, we examine the hypothesis of how the creation of the environment affects the training of reinforcement learning agent through auto-generated environment mechanisms. We take the autonomous vehicle as an application. We compare the effect of two approaches to generate training data for artificial cognitive agents. We consider the added value of curriculum learning—just as in human learning—as a way to structure novel training data that the agent has not seen before as well as that of using a replay buffer to train further on data the agent has seen before. In other words, the focus of this paper is on characteristics of the training data rather than on learning algorithms. We therefore use two tasks that are commonly trained early on in autonomous vehicle research: lane keeping and pedestrian avoidance. Our main results show that curriculum learning indeed offers an additional benefit over a vanilla reinforcement learning approach (using Deep-Q Learning), but the replay buffer actually has a detrimental effect in most (but not all) combinations of data generation approaches we considered here. The benefit of curriculum learning does depend on the existence of a well-defined difficulty metric with which various training scenarios can be ordered. In the lane-keeping task, we can define it as a function of the curvature of the road, in which the steeper and more occurring curves on the road, the more difficult it gets. Defining such a difficulty metric in other scenarios is not always trivial. In general, the results of this paper emphasize both the importance of considering data characterization, such as curriculum learning, and the importance of defining an appropriate metric for the task.
Understanding Eye-Tracking in Virtual Reality Ceur Workshop Proceedings, 2022
The Social Robot Expectation Gap Evaluation Framework Julia Rosén, Jessica Lindblom, Erik Billing Lecture Notes in Computer Science Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics, 2022
Action similarity judgment based on kinematic primitives Vipul Nair, Paul Hemeren, Alessia Vignolo, Nicoletta Noceti, Elena Nicora, Alessandra Sciutti, Francesco Rea, Erik Billing, Francesca Odone, Giulio Sandini ICDL Epirob 2020 10th IEEE International Conference on Development and Learning and Epigenetic Robotics, 2020
The DREAM Dataset: Supporting a data-driven study of autism spectrum disorder and robot enhanced therapy Erik Billing, Tony Belpaeme, Haibin Cai, Hoang-Long Cao, Anamaria Ciocan, Cristina Costescu, Daniel David, Robert Homewood, Daniel Hernandez Garcia, Pablo Gómez Esteban, Honghai Liu, Vipul Nair, Silviu Matu, Alexandre Mazel, Mihaela Selescu, Emmanuel Senft, Serge Thill, Bram Vanderborght, David Vernon, Tom Ziemke Plos One, 2020
Robot-Enhanced Therapy: Development and Validation of Supervised Autonomous Robotic System for Autism Spectrum Disorders Therapy Hoang-Long Cao, Pablo G. Esteban, Madeleine Bartlett, Paul Baxter, Tony Belpaeme, Erik Billing, Haibin Cai, Mark Coeckelbergh, Cristina Costescu, Daniel David, Albert De Beir, Daniel Hernandez, James Kennedy, Honghai Liu, Silviu Matu, Alexandre Mazel, Amit Pandey, Kathleen Richardson, Emmanuel Senft, Serge Thill, Greet Van de Perre, Bram Vanderborght, David Vernon, Kutoma Wakanuma, Hui Yu, Xiaolong Zhou, Tom Ziemke IEEE Robotics and Automation Magazine, 2019
Social Robots in Therapy and Care Daniel Hernandez Garcia, Pablo G. Esteban, Hee Rin Lee, Marta Romeo, Emmanuel Senft, Erik Billing ACM IEEE International Conference on Human Robot Interaction, 2019
User experience of conveying emotions by touch B. Alenljung, R. Andreasson, E. A. Billing, J. Lindblom, R. Lowe Ro Man 2017 26th IEEE International Symposium on Robot and Human Interactive Communication, 2017
How to build a supervised autonomous system for robot-enhanced therapy for children with autism spectrum disorder Pablo G. Esteban, Paul Baxter, Tony Belpaeme, Erik Billing, Haibin Cai, Hoang-Long Cao, Mark Coeckelbergh, Cristina Costescu, Daniel David, Albert De Beir, Yinfeng Fang, Zhaojie Ju, James Kennedy, Honghai Liu, Alexandre Mazel, Amit Pandey, Kathleen Richardson, Emmanuel Senft, Serge Thill, Greet Van de Perre, Bram Vanderborght, David Vernon, Hui Yu, Tom Ziemke Paladyn, 2017
