Projects
An adaptive agent dialogue framework for driving sustainable dietary behaviour change
Context
The food system contributes 34% of greenhouse gas emissions, the majority of which coming from animal agriculture [1] also disproportionately contributing to deforestation, water scarcity, biodiversity loss, and ecosystem pollution [2]. Despite this, most consumers are resistant to substantially reduce their meat consumption, even when considering the accompanying health benefits. Efforts to improve eating habits are traditionally approached through behaviour change counselling sessions with dieticians. Such approaches are time and resource consuming, but digital intervention alternatives lack the essential component of human interaction and social support that drives the effectiveness of behaviour change counselling [3]. Virtual agents hold the potential to fill that gap; however past approaches have typically only been loosely coupled to existing social science in behaviour change [4].
Objectives and Novelty
The project will focus on designing a virtual agent dialogue framework for longitudinal behaviour change interactions rooted in an established psychological theory. The adaptive dialogue agent will be able to guide users through their journey towards dietary change, interspersing activities from behaviour change programmes with social dialogue aimed at reinforcing the user-agent relationship while simultaneously probing users’ preferences and attitudes. These preference-infering exchanges will help maintain and update user models including idiosyncratic sensitivities to key variables identified to be key drivers for transitioning to more plant-based foods [5]: Taste expectations (i.e. meat-based foods are expected to be tastier), Availability (i.e. plant-based foods are less widely available in many settings), Skills (many consumers don’t know how to prepare meat-free meals), Identity (Vegetarian/vegan social identities are not seen as positive by many consumers and contribute to the polarization of perspectives on sustainable eating) and Social Norms (consuming meat is seen as normative, and these norms are communicated through features of the food environment and others’ behaviour). These user models will further impact task-related and relationship-building tasks, altering dialogue such as agents’ food presentation strategies. A key research challenge will consist in designing dialogue policies reconciling concurrent but inter-linked dialogue goals, in this case preference-infering, relationship-building, and delivering task-related dialogue.
Methods and Timeline
After a literature review, the student will extend an existing socially-aware recipe recommender agent framework developed at the University of Glasgow [6] with a baseline rule-based dialogue model for inferring user preferences and attitudes and integrating these variables to alter subsequent dialogue. The model will be used to collect initial data and train further model iterations, considering supervised/reinforcement learning approaches. The resulting dialogue models will be deployed in a series of user experiments to evaluate their effectiveness at promoting user engagement and motivation, infering accurate user models, and driving effective and long-lasting behaviour change.
Outputs and Impact
The project is expected to contribute novel dialogue models and policies for human-agent interactions as well as methodological and experimental insights on technologically-mediated behaviour change frameworks. The project’s findings may further feedback into theory formation on habit change and maintenance. The project will have societal impact, both locally through deployments of the resulting behaviour change framework, and further through dissemination with academic and institutional partners.
References
[1] Xu, X., Sharma, P., Shu, S., Lin, T.-S., Ciais, P., Tubiello, F. N., Smith, P., Campbell, N., & Jain, A. K. (2021). Global greenhouse gas emissions from animal-based foods are twice those of plant-based foods. Nature Food, 1–9 (link here).
[2] Poore, J., & Nemecek, T. (2018). Reducing food’s environmental impacts through producers and consumers. Science, 360(6392), 987–992 (link here) / science.aaq0216[4] Graça, J., Godinho, C. A., & Truninger, M. (2019). Reducing meat consumption and following plant-based diets: Current evidence and future directions to inform integrated transitions. Trends in Food Science & Technology, 91, 380–390 (link here).
[3] Schippers, M., et al. “A meta‐analysis of overall effects of weight loss interventions delivered via mobile phones and effect size differences according to delivery mode, personal contact, and intervention intensity and duration.” Obesity reviews 18.4 (2017): 450-459.
[4] Bickmore, Timothy W., et al. “A randomized controlled trial of an automated exercise coach for older adults.” Journal of the American Geriatrics Society 61.10 (2013): 1676-1683.
[5] Papies, E. K., Johannes, N., Daneva, T., Semyte, G., & Kauhanen, L.-L. (2020). Using consumption and reward simulations to increase the appeal of plant-based foods. Appetite, 155, 104812 (link here).
[6] Florian Pecune, Lucile Callebert, and Stacy Marsella. 2020. A Socially-Aware Conversational Recommender System for Personalized Recipe Recommendations. In Proceedings of the 8th International Conference on Human-Agent Interaction (HAI ’20). Association for Computing Machinery, New York, NY, USA, 78–86 (link here).
Augmented reality for cyclists
Main Aims and Objectives
Increasing the number of people cycling will improve population health, and reduce congestion and pollution. However, cycling in cities can be complex and risky with many different things in the environment that must be attended to. Busy roads and junctions are problematic, especially for novices, as it may not be clear which vehicle/object is a potentially dangerous target and which a harmless distractor that could be safely ignored. Safety is the main factor putting people of using bikes as an everyday form of transport. The aim of this project is to use augmented reality to help people cycle more safely by reducing visual complexity and allowing riders to focus their attention appropriately.
Our own previous research (Al-taie et al., 2022, 2023) investigated cyclist/driver interactions and showed where riders look in different road scenarios. The overarching goal of this project is thus to simplify busy road scenarios for the cyclist so that dangerous targets (say a small grey car) are amplified, whereas harmless distractors (say a loud noise) are reduced (Ahrens et al., 2019). Ultimately the cyclist will experience a simplified version of a cluttered busy road environment, reducing the cognitive load required to successfully navigate the cycling path through it. To do this, we will combine state-of-the-art augmented reality based around detailed perceptual psychology to create an application for everyday cycling.
Proposed Methods
This research is at the intersection of eye-tracking, psychology and human-computer interaction. It will involve both empirical and technical work. In the first year, the student will augment experienced cyclists with eye-tracking glasses while they cycle around the city to understand where they look and how they focus their attention in different road scenarios. We will compare this to a group of novice cyclists so that we can see the differences in attention and find where novices become overloaded and make mistakes. The resulting analyses will identify correct and incorrect ‘gaze’ behaviour toward targets and distractors.
The next stage will be to design and test augmented reality solutions to help novice riders focus their attention appropriately. We will do this using attention and perception research from Psychology to create solutions that we can evaluate in our cycling simulator. We anticipate masking distracting elements of the scene that are not relevant, for example desaturating objects that the experienced cyclists did not attend to, while enhancing those that were most important for the experts. Testing our designs in the simulator means that we can re-create different road scenarios with different complexities to test the effectiveness of our designs in safety. In these scenarios, participants/cyclists should have fewer collisions as irrelevant distractors will be masked.
The final part of the work will be to test the best solutions from the simulator in the real world, under carefully controlled settings. These studies will show whether the solutions work in realistic settings. One potential way to do this will be for participants to stand by road junctions and to experience the perceptual and attentional manipulations while we measure their attention. We may also be able to test riders using our solutions in specific environments. From these studies, we will be able to find the optimal solutions as the final output for the project.
References
Ahrens, M., Veniero, D., Freund, I.M., Harvey, M. and Thut, G. (2019). Both dorsal and ventral attention network nodes are implicated in exogenously driven visuospatial anticipation. Cortex, 117, 168-181.
Al-taie, A., Pollick, F. Brewster, S. (2022). Tour de Interaction: Understanding Cyclist-Driver Interaction with Self-Reported Cyclist Behaviour. ACM AutoUI 2022, Seoul, South Korea.
Al-taie, A., Macdonald, S., Adbradou, Y., Pollick, F., Brewster, S. (2023). Keep it Real: Investigating Driver-Cyclist Interaction in Real-World Traffic. ACM CHI 2023, Hamburg, Germany
Better Look Away: Understanding Gaze Aversion in Real and Mixed Reality Settings (exploring the Tell-Tale Task)
Main Aims and Objectives
The eyes are said to be a window to the brain [1]. The way we move our eyes reflects our cognitive processes and visual interests, and we use our eyes to coordinate social interactions (e.g., take turns in conversations) [2]. While there is a lot of research on attentive user interfaces that respond to user’s gaze [3], and directing user’s gaze towards targets [4], there is relatively less work on understanding and eliciting gaze aversion. This is unfortunate as the ability to not look is a classic psychological and neural measure of how much people are in voluntary control over their environment [5]. In fact, people often avert their eyes to alleviate a negative social experience (such as avoiding a fight) and in some cultures, looking someone in the eyes directly can be seen as disrespectful.
Efficient gaze aversion is thus an essential adaptive response and its brain correlates have been mapped extensively [6]. The main aim of this project is to investigate and enhance/train gaze aversion using virtual environments. Two potential examples will be considered in the 1st instance: Cultural gaze aversion training to accustom users to cultural norms, before encountering such a situation. Secondly, gaze elicitation and aversion will be integrated into augmented reality glasses to nudge the user to avert (or instead direct, as appropriate) their gaze while encountering for example an aggressive or socially desirable scenario. Another example could be the use of gaze aversion in mixed reality applications. In particular, guiding the user’s gaze and nudging them to look at targets and away from others, can help guide them in virtual environments, or ensure they see important elements of 360° videos.
Proposed Methods
This research is at the intersection of eye tracking, psychology and human-computer interaction. It will involve both empirical and technical work, exploring the opportunities and challenges of detecting and eliciting intentional and unintentional gaze aversion. Using an eye-tracker as well as a virtual reality headset we will a) investigate and evaluate methods for eliciting explicit and implicit gaze aversion guided by previous research on gaze direction [4,6]; b) study the impact of intentional and unintentional gaze aversion on the brain by measuring its impact on saccadic reaction times, error rates, and other metrics; and c) utilize the findings and developed methods in one or more application areas. Programming skills are required for this project and previous experience in conducting controlled empirical studies also a plus.
Likely Outputs and Impact
The results will inform knowledge and generate state of the art tools on how to best design virtual environments that optimize and measure eye-movement control. The topic spans Psychology, Neuro-and Computing Science and we thus envisage publications in journals and conferences that reach a wide academic audience, spanning a range of expertise (e.g. Psychological Science, PNAS, ACM CHI, PACM IMWUT, ACM TOCHI).
Alignment with Industrial Interests
Khamis is currently collaborating with Facebook on relevant topics, including social interactions in virtual reality. Facebook is one of the global leaders in mixed, augmented and virtual reality. This project has the potential to have direct impact on in their user interfaces. Khamis also has contacts in eye tracking and VR companies such as Blickshift GmbH, Emteq Ltd, Eyeware Tech SA and Pupil Labs GmbH. We will aim to connect with them during the project to collaborate where possible.
Supervision
You will work with Professor Monika Harvey and Dr Mohamed Khamis and be fully integrated into their research teams. See Dr Khamis’ SIRIUS Lab here.
References
[1] Ellis, S., Candrea, R., Misner, J., Craig, C. S., Lankford, C. P., & Hutchinson, T. E. (1998, June). Windows to the soul? What eye movements tell us about software usability. In Proceedings of the usability professionals’ association conference (pp. 151-178).
[2] Majaranta, P., & Bulling, A. (2014). Eye tracking and eye-based human–computer interaction. In Advances in physiological computing (pp. 39-65). Springer, London.
[3] Khamis, M., Alt, F., & Bulling, A. (2018, September). The past, present, and future of gaze-enabled handheld mobile devices: Survey and lessons learned. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (pp. 1-17).
[4] Rothe, S., Althammer, F., & Khamis, M. (2018, November). GazeRecall: Using gaze direction to increase recall of details in cinematic virtual reality. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (pp. 115-119).
[5] Butler, S.H., Rossit, R., Gilchrist, I.D., Ludwig, C.J., Olk, B., Muir, R., Reeves, I. and Harvey, M. (2009) Non-lateralised deficits in anti-saccade performance in patients with hemispatial neglect. Neuropsychologia, 47, 2488-2495.
[6] Salvia, E., Harvey M., Nazarian, B. and Grosbras, M-H. (2020). Social perception drives eye-movement related brain activity: evidence from pro- and anti-saccades to faces. Neuropsychologia, 139, 107360.
Bridging the Uncanny Valley with Decoded Neurofeedback
A problem with artificial characters that appear nearly human in appearance is that they can sometimes lead users to report that they feel uncomfortable, and that the character is creepy. An explanation for this phenomenon comes from the Uncanny Valley Effect (UVE), which holds that characters approaching human likeness elicit a strong negative response (Mori, et al., 2012; Pollick, 2009). Empirical research into the UVE has grown over the past 15 years and the conditions needed to produce a UVE, and reliably measure its effect have been extensively examined (Diel & MacDorman, 2021). These empirical studies inform design standards of artificial characters (Lay et al., 2016), but deep theoretical questions of why the UVE exists and its underlying mechanisms remain elusive. One technique that has shown promise to answer these questions is that of neuroimaging, where brain measurements are obtained while the UVE is experienced (Saygin, et al., 2012). In this research we propose to use the technique of realtime fMRI neurofeedback, which allows fMRI experiments to go past correlational evidence by enabling the manipulation of brain processing to study the effect of brain state on behaviour.
In particular, one possible way forward is to use the technique of decoded neurofeedback (DecNef), which employs methods of machine learning to build a decoder of brain activity. Previous experiments have used DecNef to alter facial preferences (Shibata, et al., 2016) and this study by Shibata and colleagues can guide our efforts to develop a decoder that can be used during fMRI scanning to influence how the UVE is experienced. It is hoped that these experiments will reveal the brain circuits involved in experiencing the UVE, and lead to a deeper theoretical understanding of the basis of the UVE, which can be exploited in the design of successful artificial characters.
The project will develop skills in 1) the use of animation tools to create virtual characters, 2) the ability to design and perform psychological assessment of people’s attitudes and behaviours towards these characters, 3) the use of machine learning in the design of decoded neurofeedback algorithms, and finally 4) how to perform realtime fMRI neurofeedback experiments.
References
Diel, A., & MacDorman, K. F. (2021). Creepy cats and strange high houses: Support for configural processing in testing predictions of nine uncanny valley theories. Journal of Vision.
Lay, S., Brace, N., Pike, G., & Pollick, F. (2016). Circling around the uncanny valley: Design principles for research into the relation between human likeness and eeriness. i-Perception, 7(6), 2041669516681309.
Mori, M., MacDorman, K. F., & Kageki, N. (2012). The uncanny valley [from the field]. IEEE Robotics & Automation Magazine, 19(2), 98-100. (Original work published in 1970).
Pollick, F. E. (2009). In search of the uncanny valley. In International Conference on User Centric Media (pp. 69-78). Springer, Berlin, Heidelberg.
Saygin, A. P., Chaminade, T., Ishiguro, H., Driver, J., & Frith, C. (2012). The thing that should not be: predictive coding and the uncanny valley in perceiving human and humanoid robot actions. Social cognitive and affective neuroscience, 7(4), 413-422.
Shibata, K., Watanabe, T., Kawato, M., & Sasaki, Y. (2016). Differential activation patterns in the same brain region led to opposite emotional states. PLoS biology, 14(9), e1002546.
Causal generative world models for the perception of physical and social motion in human brains and AI
Context and Objectives
Biological and artificial systems need to make sense of the world around them by interpreting sensory evidence in the context of internal models of the physical and social world. In humans, these world models are well-described by causal generative models, encoding the statistical and causal dependencies in the world and enabling mental simulation and predictions of social (Heider & Simmel, 1944) and physical events (Battaglia et al., 2013). Artificial intelligence (AI) is starting to construct causal generative models of the world, as these enable robust inference, powerful and flexible generalization, and learning in low-sample regimes (Schölkopf & von Klügelgen, 2022).
One of the key questions for biological and artificial systems is how sensory evidence should be contextualized within a causal generative model for the purpose of perceptual inference. The predictive processing framework proposes that the human brain contextualizes by combining top-down predictions from the generative model with bottom-up sensory evidence (Petro & Muckli 2020).
So far, little is known whether evidence-contextualization for physical and social entities employs shared or different mechanisms. This project will study how the integration of predictions with sensory evidence occurs for physical and social events and interactions.
Proposed Methods and Expected Results
We will use a novel task in which participants track physical objects and/or agents that move and interact with physical or agent-like dynamics. Concurrent Ultra High Field FMRI measurements and cortical layer-resolved decoding will enable us to dissociate top-down predictions, bottom-up sensory evidence, and error representations.
We will further disentangle the contribution of predictions and sensory evidence by introducing visual occlusions (requiring predictions and mental simulation in the absence of visual input) and violations of expectations (i.e., unexpected switch of the task-generative motion and interaction model from physical to agent-like or vice versa).
We hypothesize that (1) early visual cortices are the locus of error computation for both physical and social perceptual signals. E.g., we expect to find predictions related to both visual objects and agents in the deep layers of V1 in occluded areas and prediction errors in superficial layers. (2) We expect predictions to originate in different higher-level cortices. Namely, we expect object-driven cortex (Yildirim et al., 2019) to be involved in predicting physical motion and interactions, whereas we expect frontal (Sliwa & Freiwald, 2017) and temporal-parietal regions (Frith & Frith, 2006) as the source for predictions of social motion and interaction.
We will, furthermore, construct generative brain-inspired recurrent artificial neural network models for physical and social motion and interactions. We will compare three classes of models: (a) joint models for physical and social motion and interactions, (b) separate models for physical and social motion and interaction, and (c) models with a shared early perceptual stage and disjoint later stages. (3) We expect that the latter class best captures human brain-pattern activations and to be the most sample-efficient AI model in terms of learning.
Impact for Artificial Social Intelligence
Key to human-centric social AI is the alignment of causal generative world models of humans and AI, that underly the perception of our world. Our proposal aims to shed light on these generative models of humans by interrogating the predictions made by their generative world model. The expected results and constructed models are, therefore, important stepping stones toward building social AI that is aligned with humans in their perception of physical and social events.
References
Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013). Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45), 18327–18332.
Frith, C. D., & Frith, U. (2006). The Neural Basis of Mentalizing. Neuron, 50(4), 531–534.
Heider, F., & Simmel, M. (1944). An Experimental Study of Apparent Behavior. The American Journal of Psychology, 57(2), 243–259.
Petro LS, Muckli L. (2020) Neuronal codes for predictive processing in cortical layers. Behav Brain Sci. Jun 19;43:e142.
Schölkopf, B., & von Kügelgen, J. (2022). From Statistical to Causal Learning. arXiv:2204.00607.
Sliwa, J., & Freiwald, W. A. (2017). A dedicated network for social interaction processing in the primate brain. Science, 356(6339), 745–749.
Yildirim, I., Wu, J., Kanwisher, N., & Tenenbaum, J. (2019). An integrative computational architecture for object-driven cortex. Current Opinion in Neurobiology, 55, 73–81.
Developing a brain-controlled Robot hand for future clinical applications
Robotic hands can be of valuable assistance to individuals with upper-limb motor disabilities such as stroke survivors, tetraplegic individuals and amputees. Brain-computer interfaces (BCI) offer an intuitive means to control such assistive robotic manipulators. Further, they can potentially promote rehabilitation and recovery in patients with brain or spinal cord lesions. For instance, in neurofeedback (NF) studies, participants visualise a graphical (i.e. thermometer) representation of their own brain activity and attempt to increase this activation, which can result in motor improvements (Mottaz et al., 2018; Sanders et al., 2022). An alternative is to use these neural signals to control a robotic hand to perform realistic movements (Batzianoulis et al; 2021). This approach has three main aims: to activate the target motor network and promote recovery via neuroplasticity, to provide realistic feedback instead of abstract, and to control the robotic arm to perform tasks using the brain’s motor representation network.
EEG has been the preferred method for NF and BCI, due to its portability, cheaper set up and high temporal resolution. However, fMRI superior spatial resolution allows for measuring individual finger representation, which will be an advantage in these earlier stages of development of brain robot interaction. In this PhD project we aim to answer the following question: can we use the brain’s fingers representation system to effortlessly control the robot’s individual fingers? Therefore, the goal of this PhD project is to develop analysis tools to control the Shadow Robotic hand’s individual fingers using fMRI data.
The student will collect fMRI data from healthy participants while they perform real finger movements inside a 7T scanner. The fMRI data will be analyzed offline, and reinforcement learning models will be trained to obtain the robot path planning strategy for controlling the corresponding robot fingers in real-time using the traditional PID control method. In the second stage of the project, participants will control the robot fingers in real-time inside the MRI scanner using their own finger movements. Finally, naive participants will be trained to control the robot fingers without performing real movements. Future work aims to test this protocol in patients, such as stroke survivors, to promote motor recovery or in immobilized patients to use the robot as an assistive device. The student must have strong coding skills, such as C++ and Python, and experience in signal processing, such as noise removal and filtering techniques. They will program and control the Shadow robot fingers virtually in the Isaac Gym simulator and in the RoS system to control the Shadow robot hand in real-time.
The project’s impact will be broad, affecting health, neuroscience, and the robotic society. The output of the project can be published in various journals and conferences, such as IEEE International Conference on Robotics and Automation (ICRA), IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Science Robotics, Journal of Neural Engineering, and RTFIN conference.
References
Batzianoulis I., Iwane F., Wei S. et al. Customizing skills for assistive robotic manipulators, an inverse reinforcement learning approach with error-related potentials. Commun Biol 4, 1406 (2021) ) (link here).
Mottaz A, Corbet T, Doganci N, Magnin C, Nicolo P, Schnider A, Guggisberg AG. Modulating functional connectivity after stroke with neurofeedback: Effect on motor deficits in a controlled cross over study Neuro Image: Clinical 20:336-346 (2018).
Sanders Z. B., Fleming M. K., Smejka T., Marzolla M. C., Zich C., Rieger S. W., Lührs M., Goebel R., Sampaio-Baptista C., Johansen-Berg H, Self-modulation of motor cortex activity after stroke: a randomized controlled trial, Brain, Volume 145, Issue 10, October 2022, Pages 3391–3404 (link here).
Yu R., Kizilkaya, B. Meng Z., Li L., et. al, “Robot Mimicry Attack on Keystroke-Dynamics User Identification and Authentication System”, in Proc. 2023 IEEE International Conference on Robotics and Automation (ICRA), London, UK, 29 May – 02 Jun 2023, (Accepted for Publication).
Huang L., Meng Z., Deng Z., Wang C., Li L., et al, “Extracting human behavioral biometrics from robot motions,” in Proc. 27th Annual International Conference on Mobile Computing and Networking (MobiCom 2021), Oct. 2021.
Yang G., Riener R., and Dario P., “To integrate and to empower: Robots for rehabilitation and assistance,” Science robotics, vol. 2, no. 6, May 2017.
Enhancing sense of social presence in virtual environments with social elements: Manipulating virtual self awareness to increase interpersonal motor resonance with virtual characters using VR and electro / neurophysiological technologies
Main Aims and Objectives
Interpersonal motor resonance (IMR) is a phenomenon in which an individual’s motor system activates when observing another person’s motor behaviour. This phenomenon is thought to underlie in part interpersonal understanding for the facilitation of social cognition (see [1] for theories on the role of bodily representation in understanding others). This research project seeks to explore the extent to which self body awareness in virtual reality situations is facilitating interpersonal understanding of an autonomous virtual actor, and by extension the experience of social presence.
Proposed Methods and Envisaged Outcomes
We will manipulate virtual self body awareness and its sub-components (sense of self-location, sense of agency, sense of body ownership [2]) in human participants using techniques for illusory embodiment/disembodiment (see e.g.[3]), examining what technological factors of the virtual reality simulation (realism/rendering of the avatar and agent, environment, embodiment through visual, haptic or proprioceptive/ suit-based multisensory signals, see e.g.[2,4-5]) affect immersive (embodied) experience of the virtual environment [4,5]. We will leverage neurophysiological markers of interpersonal motor resonance using transcranial magnetic stimulation (TMS) (e.g.[6]) and electroencephalographic (EEG) approaches (e.g.[7]) to measure IMR/interpersonal understanding as a function of virtual body self awareness. We hypothesize that interpersonal understanding of an avatar is enhanced by heightened immersive experience through embodiment. We envision this project to provide a better understanding of IMR, as well as the role of embodiment and presence in human-agent interaction for social virtual reality settings. This will also provide evidence towards the use of virtual reality and virtual agents as experimental tools for studying IMR and related neural processes [4].
Proposed Experiments
You will be conducting a series of experiments to measure the sense of virtual embodiment and immersive experience across a number of technical manipulations (part 1). You will then measure the participants’ IMR in response to the virtual character’s actions at heightened and low levels of embodiments (part 2). In each experiment, the virtual character will perform simple motor actions. This may involve a gesture of the arms or hands, to which we will measure the participant’s reaction by analysing the neural correlates of motor resonance to the gesture (central mu suppression in EEG, TMS-probed corticospinal excitability changes in MEPs [6,7]), and other relevant objective, behavioural, and subjective measures. Follow-on studies (part 3) may manipulate various factors, including the simulation settings (decontextualised or contextualised, e.g. an empty environment vs a realistic social situation), or participant group (with social anxiety versus no anxiety).
Skills
The project will require the development of a suitable experimental framework, including a virtual environment containing a gesture-enabled virtual agent, as well as integrating tools for tracking and embodying the subject in an avatar in real-time, using a full body tracking suit. This will leverage existing tools and systems available to the supervisors. Additionally, precise temporal synchronisation will need to be implemented between events in the virtual environment (i.e. agent gestures) and the EEG/TMS measures. This will enable precise temporal analysis to measure the subject’s response to the agent’s gesture and analyse the size and temporal dynamics of IMR.
Possible Impact
The project will provide insights into interpersonal motor resonance, and on interpersonal understanding in virtual embodied settings. It could inform the development of more realistically perceived virtual agents, leading to more effective human-agent interaction in a variety of applications.
References
[1] Gallese V, Sinigaglia C (2011). What is so special about embodied simulation? Trends Cogn Sci, 15(11):512-9.
[2] Kilteni K, Groten R, Slater M (2012). The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments, 21(4), 373-387.
[3] Petkova VI, Ehrsson HH (2018). If I were you: perceptual illusion of body swapping. PLoS One, 3(12):e3832.
[4] Pan X, Hamilton AFC (2018). Why and how to use virtual reality to study human social interaction: The challenges of exploring a new research landscape. British Journal of Psychology, 109(3), 395-417.
[5] Fribourg R, Argelaguet F, Hoyet L, Lécuyer A (2018). Studying the sense of embodiment in VR shared experiences. In IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 273-280). IEEE.
[6] Prinsen J, Alaerts K (2019). Eye contact enhances interpersonal motor resonance: comparing video stimuli to a live two-person action context. Soc Cogn Affect Neurosci, 14(9):967-976.
[7] Järveläinen J, Schürmann M, Avikainen S, Hari R (2001). Stronger reactivity of the human primary motor cortex during observation of live rather than video motor acts. Neuroreport, 12(16):3493-5.
Enhancing Social Interactions via Physiologically-Informed AI
Over the past few years major developments in machine learning (ML) have enabled important advancements in artificial intelligence (AI). Firstly, the field of deep learning (DL) – which has enabled models to learn complex input-output functions (e.g. pixels in an image mapped onto object categories), has emerged as a major player in this area. DL builds upon neural network theory and design architectures, expanding these in ways that enable more complex function approximations. The second major advance in ML has combined advances in DL with reinforcement learning (RL) to enable new AI systems for learning state-action policies – in what is often referred to as deep reinforcement learning (DRL) – to enhance human performance in complex tasks. Despite these advancements, however, critical challenges still exist in incorporating AI into a team with human(s). One of the most important challenges is the need to understand how humans value intermediate decisions (i.e. before they generate a behaviour) through internal models of their confidence, expected reward, risk etc. Critically, such information about human decision-making is not only expressed through overt behaviour, such as speech or action, but more subtlety through physiological changes, small changes in facial expression and posture etc. Socially and emotionally intelligent people are excellent at picking up on this information to infer the current disposition of one another and to guide their decisions and social interactions. In this project, we propose to develop a physiologically-informed AI platform, utilizing neural and systemic physiological information (e.g. arousal, stress) ([Fou15][Pis17][Ghe18]) together with affective cues from facial features ([Vin09][Bal16]) to infer latent cognitive and emotional states from humans interacting in a series of social decision-making tasks (e.g. trust game, prisoner’s dilemma etc). Specifically, we will use these latent states to generate rich reinforcement signals to train AI agents (specifically DRL) and allow them to develop a “theory of mind” ([Pre78][Fri05]) in order to make predictions about upcoming human behaviour. The ultimate goal of this project is to deliver advancements towards “closing-the-loop”, whereby the AI agent feeds-back its own predictions to the human players in order to optimise behaviour and social interactions.
References
[Ghe18] S Gherman, MG Philiastides, “Human VMPFC encodes early signatures of confidence in perceptual decisions”, eLife, 7: e38293, 2018.
[Pis17] MA Pisauro, E Fouragnan, C Retzler, MG Philiastides, “Neural correlates of evidence accumulation during value-based decisions revealed via simultaneous EEG-fMRI”, Nature Communications, 8: 15808, 2017.
[Fou15] E Fouragnan, C Retzler, KJ Mullinger, MG Philiastides, “Two spatiotemporally distinct value systems shape reward-based learning in the human brain”, Nature Communications, 6: 8107, 2015. [Vin09] A.Vinciarelli, M.Pantic, and H.Bourlard, “Social Signal Processing: Survey of an Emerging Domain“, Image and Vision Computing Journal, Vol. 27, no. 12, pp. 1743-1759, 2009.
[Bal16] T.Baltrušaitis, P.Robinson, and L.-P. Morency. “Openface: an open source facial behavior analysis toolkit.” Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2016.
[Pre78] D. Premack, G. Woodruff, “Does the chimpanzee have a theory of mind?”, Behavioral and brain sciences Vol. 1, no. 4, pp. 515-526, 1978. [Fri05] C. Frith, U. Frith, “Theory of Mind”, Current Biology Vol. 15, no. 17, R644-646, 2005.
Facial Expression-guided Gaze Response for Autism Screening
Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by significantly impaired social interaction and communication abilities. Such impairments include deficits in perceiving, using and responding to various non-verbal cues of communication, such as emotion related facial expressions. This is often attributed to the atypical eye gaze in deriving these non-verbal cues from. Research has shown that individuals with ASD, fixate less to the eye and face region and more to the human body as compared to their typical counterparts. There is clear evidence that individuals with ASD have atypical gaze pattern which is prominently observed in the context of processing affective expressions from a communicator’s face.
We will leverage glasses with IMUs and front-facing cameras to simultaneously record eye-gaze, facial expressions of a subject undergoing Autism screening and the facial expressions of a virtual agent (or human) they will be interacting with. We will analyse the gaze behaviour of the subject and evaluate the synchrony between their gaze behaviour and facial expressions (also with the agent’s facial expressions) to develop a screening strategy for Autism. We anticipate that our findings will inform the development of new assistive technologies for Autism screening (and possibly intervention) that will have lower cost and higher accessibility (currently Autism diagnosis and intervention are expensive and not always accessible due to unavailability of trained professionals).
References
Ghosh and Guha, “Towards Autism Screening through Emotion-guided Eye Gaze Response”, IEEE Annual Conference on Engineering in Medicine and Biology (EMBC), 2021.
Robot Cuteness: How does morphology affect human social perception of robots?
Have you ever been told you have a “trustworthy face”? Despite little to no validity in such judgements, people quickly and involuntarily form impressions of others based on their facial appearance, which can then influence important social outcomes. Do people form similar judgements about non-human appearance? This project will extend work on the social perception of human morphology to robots and digital avatars.
In the first year, while primarily focussing on the integrated MSc programme, you will also conduct some background literature research, which will involve reviewing existing frameworks of and methodologies for human social perception and the influence of morphology on attitudes towards robots. In the second year you will focus on methods, creating and validating methods for studying robot social perception that can be directly compared to human perception. The third and fourth year will use these validated measures to investigate the role of systematic changes in morphology on social perceptions of robots and digital avatars in various contexts. Such contexts can include those where social perceptions such as trust may be important to foster, and where such perceptions may lead to exploitation.
The PhD work will result in a better understanding of human-robot relationships, better ways of evaluating social perception of technology, and how to better design robots and digital avatars, all of which would be of interest to academia, industry and government.
References
Lacey, C., & Caudwell, C. (2019, March). Cuteness as a ‘dark pattern’ in home robots. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (pp. 374-381). IEEE (link here).
Jones, B. C., DeBruine, L. M., Flake, J. K., + Multiple Researchers, . & Chartier, C. R. (2021). Social perception of faces around the world: How well does the valence-dominance model generalize across world regions? Nature Human Behaviour, 5: 159-169 (link here).
Colleen M. Carpinella, Alisa B. Wyman, Michael A. Perez, and Steven J. Stroessner. 2017. The Robotic Social Attributes Scale (RoSAS): Development and Validation. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’17). Association for Computing Machinery, New York, NY, USA, 254–262 (link here).
Sharing the road: Cyclists and automated vehicles
Automated vehicles must share the road with pedestrians and cyclists, and drive safely around them. Autonomous cars, therefore, must have some form of social intelligence if they are to function correctly around other road users. There has been work looking at how pedestrians may interact with future autonomous vehicles [ROT15] and potential solutions have been proposed (e.g. displays on the outside of cars to indicate that the car has seen the pedestrian). However, there has been little work on automated cars and cyclists. When there is no driver in the car, social cues such as eye contact, waving, etc., are lost [ROT15]. This changes the social interaction between the car and the cyclist, and may cause accidents if it is no longer clear, for example, who should proceed. Automated cars also behave differently to cars driven by humans, e.g. they may appear more cautious in their driving, which the cyclist may misinterpret. The aim of this project is to study the social cues used by drivers and cyclists, and create multimodal solutions that can enable safe cycling around autonomous vehicles. The first stage of the work will be observation of the communication between human drivers and cyclists through literature review and fieldwork. The second stage will be to build a bike into our driving simulator [MAT19] so that we can test interactions between cyclists and drivers safely in a simulation. We will then start to look at how we can facilitate the social interaction between autonomous cars and cyclists. This will potentially involve visual displays on cars or audio feedback from them, to indicate state information to cyclists nearby (eg whether they have been detected, whether the car is letting the cyclist go ahead). We will also investigate interactions and displays for cyclists, for example multimodal displays in cycling helmets [MAT19] to give them information about car state (which could be collected by V2X software on the cyclist’s phone, for example). Or directly communicating with the car by input made on the handlebars or via gestures. These will be experimentally tested in the simulator and, if we have time, in highly controlled real driving scenarios. The output of this work will be a set new techniques to support the social interaction between autonomous vehicles and cyclists. We currently work with companies such as Jaguar Land Rover and Bosch and our results will have direct application in their products.
References
[ROT15] Rothenbucher, D., Li, J., Sirkin, D. and Ju, W., Ghost driver: a platform for investigating interactions between pedestrians and driverless vehicles, Adjunct Proceedings of the International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 44–49, 2015.
[MAT19] Matviienko, A. Brewster, S., Heuten, W. and Boll, S. Comparing unimodal lane keeping cues for child cyclists, Proceedings of the 18th International Conference on Mobile and Ubiquitous MultimediaNovember, pp. 1-11, 2019 (link here).
Social and Behavioural Markers of Hydration States
Aims and Objectives
This project will explore whether data derived from a person’s smartphone can be used to establish that person’s hydration status so that, in a well–guided and responsive way, a system can prompt the person to drink water. Many people are frequently underhydrated, which has negative physical and mental health consequences. Low hydration states can manifest in impaired cognitive and physical performance, experiences of fatigue or lethargy, and negative affect (e.g, Muñoz et al., 2015; Perrier et al., 2020). Here, we will establish whether such social and behavioural markers of dehydration can be inferred from a user’s smartphone, and which of these markers, or their combination, are the best predictors of hydration state (Aim 1). Sophisticated user models of hydration states could also be adapted over time, and help to predict possible instances of dehydration in advance (Aim 2). This would be useful because many individuals find it difficult to identify when they need to drink, and could benefit from clear, personalized indicators of dehydration. In addition, smart phones could then be used to prompt users to drink water, once a state of dehydration has been detected, or when dehydration is likely to occur. Thus, we will also test how hydration information should be communicated to users to prompt attitude and behaviour change and ultimately, improve hydration behaviour (Aim 3). Throughout, we will implement data collection, modelling, and feedback on smartphones in a secure way that respects and protects a user’s privacy.
Background and Novelty
The data that can be derived from smart phones (and related digital services) ranges from low level data on sensors (e.g. accelerometers) to patterns of app usage and social interaction. As such, ‘digital phenotyping’ is a rich source of information on an individual’s social and physical behaviours, and affective states. Some recent survey papers this burgeoning field include Thieme et al. on machine learning in mental health (2020), Chancellor and de Choudhury on using social media data to predict mental health status (2020), Melcher et al. on digital phenotyping of college students (2020), and Kumar et al. on toolkits and frameworks for data collection (2020). Here, we propose that these types of data may also reflect a person’s hydration state. Part of the project’s novelty is in its exploration of a wider range of phone-derived data as a resource for system agency than prior work in this general area, as well as pioneering work specifically on hydration. We will relate cognitive and physical performance, fatigue, lethargy and affect to patterns in phone-derived data. We will test whether such data can be harnessed to provide people with personalized, external, actionable indicators of their physiological state, i.e. to facilitate useful behaviour change. This would have clear advantages over existing indicators of dehydration, such as thirst cues or urine colour, which are easy to ignore or override, and/or difficult for individuals to interpret (Rodger et al, 2020).
Methods
We will build on an existing mobile computing framework (e.g. AWARE-Light) to collect reports of a participant’s fluid intake, and to integrate them with phone-derived data. We will attempt to model users’ hydration states, and validate this against self-reported thirst and urine frequency, and self-reported and photographed urine colour (Paper 1). We will then examine in prospective studies if these models can be used to predict future dehydration states (Paper 2). Finally, we will examine effective ways to provide feedback and prompt water drinking, based on individual user models (Paper 3).
Outputs
This project will lead to presentations and papers at both Computer Science and Psychology conferences outlining the principles of using sensing data to understand physiological states, and to facilitate health behaviour change.
Impact
Results from this work will have implications for the use of a broad range of data in health behaviour interventions across domains, as well as for our understanding of the processes underlying behaviour change. This project would also outline new research directions for studying the effects of hydration in daily life.
References
Chancellor, S., & De Choudhury, M. (2020). Methods in predictive techniques for mental health status on social media: a critical review. Npj Digital Medicine, 3(1), 1–11 (link here).
Melcher, J., Hays, R., & Torous, J. (2020). Digital phenotyping for mental health of college students: a clinical review. Evidence Based Mental Health, 4, ebmental–2020–300180–6 (link here).
Muñoz, C. X., Johnson, E. C., McKenzie, A. L., Guelinckx, I., Graverholt, G., Casa, D. J., … Armstrong, L. E. (2015). Habitual total water intake and dimensions of mood in healthy young women. Appetite, 92, 81–86 (link here).
Rodger, A., Wehbe, L., & Papies, E. K. (2020). “I know it’s just pouring it from the tap, but it’s not easy”: Motivational processes that underlie water drinking. Under Review (link here).
Perrier, E. T., Armstrong, L. E., Bottin, J. H., Clark, W. F., Dolci, A., Guelinckx, I., Iroz, A., Kavouras, S. A., Lang, F., Lieberman, H. R., Melander, O., Morin, C., Seksek, I., Stookey, J. D., Tack, I., Vanhaecke, T., Vecchio, M., & Péronnet, F. (2020). Hydration for health hypothesis: A narrative review of supporting evidence. European Journal of Nutrition (link here).
Thieme, A., Belgrave, D., & Doherty, G. (2020). Machine Learning in Mental Health. ACM Transactions on Computer-Human Interaction (TOCHI), 27(5), 1–53 (link here).
The Virtual Choreographer: Exploring a Dance-Inspired Interactive Direction Scheme to Control Artificial Agents Bodily Behaviour Generation
Main Aims and Objectives
Laban Movement Analysis (LMA) is a popular method for analyzing, interpreting, and communicating movement qualities introduced by Rudolf Laban [1]. Originating from the exploration of dance movement, it has been used extensively to characterize movement not just in dance but in sports, theatre, and film. More recently, LMA has inspired AI methods to automatically characterise human movement [2], and generate animations for virtual agents [3] and humanoid robots [4].
The generation of expressive behaviour for virtual agents and robots has been dominated by inscrutable deep learning end-to-end models [5], with some approaches enabling some degree of control over this generation with high-level variables [6]. This research project aims to explore LMA as a general and intuitive representation for perceiving, generating and controlling artificial agents’ movements and expressive qualities, and assessing its applicability not just in the context of dance, but for any context featuring expressive movement. The project will focus on interactive settings, such as interactive performance with artificial agents, the generation of LMA-based feedback to performers, LMA instruction, etc.
Proposed Methods
We will adapt LMA as a tool for modeling movement computationally, which may include the definition and training of machine learning models from LMA-annotated datasets for learning a representation of movement primitives corresponding to the dimensions of LMA, i.e. body, shape, effort, space, and their sub-dimensions. Modern methods, such as Graph Convolutional Networks – well adapted to representing the skeleton structure of human animations – may be used [2]. These learned representations and models will be deployed in interactive tasks for experimental studies with the goal of assessing LMA as an intuitive scheme for controlling artificial agents’ movements or providing feedback on human movement qualities. Comparisons will be realised between populations of subjects trained in LMA (such as Laban-trained professional dancers) with naive participants to assess its generalisability and intuitiveness.
Likely Outputs
This project will lead to the development of a general and intuitive representation for controlling artificial agents’ movements and expressive qualities based on Laban Movement Analysis (LMA). The project will also generate insights into the controllability of computational methods leveraging artificial intelligence to interact with expressive artificial agents.
Possible Impact
This project has the potential to provide a radically new and intuitive way to control and interact with artificial agents. This could have applications in varied contexts, such as human-agent or human-robot interaction in social or performance settings, for dance and movement training, or for movement assessment, e.g. rehabilitation.
Alignment with Industrial Interests
The project will bridge the gap between the performing arts and the technology industry, providing a glimpse into the future of the performing arts.
Brief Timetable
First, modern methods for characterising and generating expressive movement will be reviewed, with a focus on the use of LMA as a computational representation. This will include the identification of relevant expressive movement datasets. The review will also consider how LMA was adapted in computational methods to discriminate or generate movements.
The project will then move on to designing a LMA-inspired computational representation model, possibly following GCNs or alternative neural network architectures, which will be used as the basis for expressive animation generation. The project will then proceed to deploy this model in a first interactive setting, such as virtual choreographic control of artificial agents, evaluating its use by naive or trained LMA subjects in terms of intuitiveness, expressiveness, and generative control. Other settings will then be explored, such as performance or conversational settings.
References
[1] Laban, R., & Ullmann, L. (1971). The mastery of movement.
[2] M. Li, Z. Miao, X. -P. Zhang, W. Xu, C. Ma and N. Xie, “Rhythm-Aware Sequence-to-Sequence Learning for Labanotation Generation With Gesture-Sensitive Graph Convolutional Encoding,” in IEEE Transactions on Multimedia, vol. 24, pp. 1488-1502, 2022 (link here).
[3] Burton, S.J., Samadani, AA., Gorbet, R., Kulić, D. (2016). Laban Movement Analysis and Affective Movement Generation for Robots and Other Near-Living Creatures. In: Laumond, JP., Abe, N. (eds) Dance Notations and Robot Motion. Springer Tracts in Advanced Robotics, vol 111. Springer, Cham (link here).
[4] Abe, N., Laumond, J.-P., Salaris, P., & Levillain, F. (2017). On the use of dance notation systems to generate movements in humanoid robots: The utility of Laban notation in robotics. Social Science Information, 56(2), 328–344 (link here).
[5] Kucherenko, T., Jonell, P., Van Waveren, S., Henter, G. E., Alexandersson, S., Leite, I., & Kjellström, H. (2020, October). Gesticulator: A framework for semantically-aware speech-driven gesture generation. In Proceedings of the 2020 International Conference on Multimodal Interaction (pp. 242-250).
[6] Alexanderson, S., Henter, G.E., Kucherenko, T. and Beskow, J. (2020), Style-Controllable Speech-Driven Gesture Synthesis Using Normalising Flows. Computer Graphics Forum, 39: 487-496 (link here).
Trustworthy human-agent collaboration in interactive drilling advice
Agents now regularly advise operators in high-risk drilling situations and can automate actions. However, while these agents are highly optimised, they are not very transparent and trustworthy: they lack explanations for their actions and plans and they do not account for human cognitive biases. In addition, currently agents cannot be guided by operators to learn about changing circumstances: operators cannot revise plans easily, cannot tell the agent about changed objectives or constraints, and cannot change the problem structure. This PhD will investigate improved interactions between agents and operators to make the advice more trustworthy and allow the agent to learn from user feedback. The PhD programme will contribute technical approaches to build trustworthy artificial agents and a better understanding of principles underlying trustworthy interactions between agents and humans. In year 1, you will focus on the Masters but start exploring the relevant literature and technologies. In Year 2, you will conduct a systematic literature review and build a prototype to conduct an exploratory study. In year 3-4, you will enhance the prototype and conduct evaluation experiments to investigate how to improve the human-agent interaction.
References
Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ’15), 126–137 (link here).
Ronald Metoyer, Simone Stumpf, Christoph Neumann, Jonathan Dodge, Jill Cao, and Aaron Schnabel. 2010. Explaining how to play real-time strategy games. Knowledge-Based Systems 23, 4: 295–301 (link here).
Jonathan Dodge, Sean Penney, Claudia Hilderbrand, Andrew Anderson, and Margaret Burnett. 2018. How the Experts Do It: Assessing and Explaining Agent Behaviors in Real-Time Strategy Games. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18), 562:1-562:12 (link here).
Salvatore Corrente, Simon French, Salvatore Greco, Milosz Kadzinski, Joshua Knowles, Vincent Mousseau, Johannes Siebert, and Roman Słowiński. 2012. Drafting a Manifesto for DM-DSS Interaction. In Learning in Multiobjective Optimization (2012) (link here).
Manuel López-Ibáñez, and Joshua D. Knowles. 2015. Machine Decision Makers as a Laboratory for Interactive EMO. In EMO (2015): 295-309. Springer-Verlag (link here).
S Mahdi Shavarani, Manuel López-Ibáñez, Richard Allmendinger, Joshua Knowles. 2023. An Interactive Decision Tree-Based Evolutionary Multi-Objective Algorithm. In Proceedings of the International Conference on Evolutionary Multi-criterion Optimization (EMO 2023), In Press (link here).
Vision-based AI for automatic detection of individual and social behaviour in Rodents
Rodents are the most extensively used models to understand the cellular and molecular underpinnings of behaviour, neurodegenerative and psychiatric disorders, as well as, for the development of interventions and pharmacological treatments. Screening behavioural phenotypes in rodents is very time consuming, as a large battery of cognitive and motor behavioural tests is necessary. Further, standard behavioural testing usually requires the removal of the animal from their home-cage environment and individual testing, therefore excluding the assessment of spontaneous social interactions. Monitoring of home-cage spontaneous behaviours, such as eating, grooming, sleeping and social interactions, has already proven to be sensitive to different models of neurodevelopmental and neurodegenerative disorders. For instance, home-cage monitoring can distinguish different mouse strains and models of autistic-like behaviour (Jhuang et al., 2010) and detect early alterations in sleep patterns before behavioural alterations in a rodent model of amyotrophic lateral sclerosis (ALS) (Golini et al., 2020).
Most of the traditional home-cage monitoring systems use sensors and therefore are restricted on the type of activities that it can detect, requiring the animals to interact with the sensors (Goulding et al., 2008; Kiryk et al., 2020; Voikar and Gaburro, 2020). The recent development of vision-based computing and machine learning opens up the possibility to monitor and potentially label all home-cage behaviours automatically (Jhuang et al., 2010; Mathis et al., 2018). Still, most automatic detection machine learning-based work has focused on movements, mainly joints and movements trajectory (Mathis et al., 2018) rather than social or group behaviour.
Aims and Novelty
The aim of this PhD is to leverage the advancements of computer vision for animal behaviour understanding (Pessanha et. al. 2020) and build machine learning models that can automatically interpret and classify different individual and social behaviours by analysing videos collected using continuous monitoring.
Objectives
1. Define a set of behavioural and social cues that are relevant to understanding their interactions and group behaviour. This will include building a dataset of their spontaneous social behaviour.
2. Developing computer vision and machine learning models to automatically detect and classify these behaviours.
2. Validate and evaluate the developed tools on disorder models (e.g., learning deficits, stroke, etc.).
Expected Outcome and Impact
The models developed in this project will have wide applications, both in academic research as well as industry, not only by providing tools for automatic behavioural phenotyping but also as means to measure animal welfare during these experiments and procedures.
References
Golini, E., Rigamonti, M., Iannello, F., De Rosa, C., Scavizzi, F., Raspa, M., Mandillo, S., 2020. A Non-invasive Digital Biomarker for the Detection of Rest Disturbances in the SOD1G93A Mouse Model of ALS. Front Neurosci 14, 896.
Goulding, E.H., Schenk, A.K., Juneja, P., MacKay, A.W., Wade, J.M., Tecott, L.H., 2008. A robust automated system elucidates mouse home cage behavioral structure. Proc Natl Acad Sci U S A 105, 20575-20582.
Jhuang, H., Garrote, E., Mutch, J., Yu, X., Khilnani, V., Poggio, T., Steele, A.D., Serre, T., 2010. Automated home-cage behavioural phenotyping of mice. Nat Commun 1, 68.
Kiryk, A., Janusz, A., Zglinicki, B., Turkes, E., Knapska, E., Konopka, W., Lipp, H.P., Kaczmarek, L., 2020. IntelliCage as a tool for measuring mouse behavior – 20 years perspective. Behav Brain Res 388, 112620.
Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., Bethge, M., 2018. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci 21, 1281-1289.
Voikar, V., Gaburro, S., 2020. Three Pillars of Automated Home-Cage Phenotyping of Mice: Novel Findings, Refinement, and Reproducibility Based on Literature and Experience. Front Behav Neurosci 14, 575434.
Pessanha F., McLennan K., Mahmoud M. Towards automatic monitoring of disease progression in sheep: A hierarchical model for sheep facial expressions analysis from video in IEEE International Conference on Automatic Face and Gesture Recognition, Buenos Aires, May 2020.
Who you gonna call? Developing rat-to-rat communication interfaces
Main Aims and Objectives
Many rats live in laboratory conditions residing in cages in small groups or alone (such as after surgery, dominance issues, safety, or research purposes). Yet, rats are highly social animals with complex social skills needing the company of others. Rats become attached and form solid bonds and large communities in the wild. Thus, while the research done on laboratory rats is vital to human and animal health, their social living conditions are not always ideal. The main aim of this project is to increase rats’ sociality by exploring how rats can use computers that have audio, visual and olfactory output to communicate with other rats. We will then use this output to develop an artificial rat agent to support lonely rats autonomously. While researchers have investigated how rats react to screen systems [1], dog-to-dog interfaces [2] and dog-to-human video interfaces [3], there is no research undertaken around rat-to-rat and rat-to-computer interaction.
Proposed Methods and Outputs
This research is at the intersection of animal-computer interaction and neuroscience, exploring rats’ behaviour, brain, vocal analysis, and computer usage. Using novel devices to enable rats to virtually calling, we will look at a) how connecting to other rats virtually can improve a rats life, b) how different modalities (audio, visual and olfactory) support rats’ social communication, and c) how rats interact virtually with known and unknown rats. To enable rat-to-rat communication, novel remote calling devices will be developed that facilitate rats to trigger and answer calls. The rats’ behaviour and vocal analysis (such as [4]) and brain neuroimaging [5] will be studied to assess the impact of these remote interactions. The results of these studies will inform on how to support rat communication virtually and with virtual agents, and on the impact of rat-computer interfaces on behaviour, social interactions, and brain function and structure. The student will need to apply for a Personal Home Office Licence (PIL) as part of their first year of study and will be working directly with laboratory rats, programming, building devices and encoding the rats’ behaviours. The project is of great industrial and academic interest in social behaviour, laboratory and domesticated rodent welfare as it will provide key insights and tools for supporting their social needs.
References
[1] Yakura T, Yokota H, Ohmichi Y, Ohmichi M, Nakano T, et al. (2018) Visual recognition of mirror, video-recorded, and still images in rats. PLOS ONE 13(3): e0194215 (link here).
[2] Hirskyj-Douglas, I and Lucero, A.(2019). On the Internet, Nobody Knows You’re a Dog… Unless You’re Another Dog. In 2019 CHI Conference on Human Factors in Computing Systems Proceedings (CHI 2019), May 4–9, 2019, Glasgow, Scotland, UK. ACM, New York, NY, USA (link here).
[3] Hirskyj-Douglas, I., Piituanen, R., Lucero, A. (2021). Forming the Dog Internet: Prototyping a Dog-to-Human Video Call Device. Proc. ACM Hum.-Comput. Interact. 5, ISS, Article 494 (November 2021), 20 pages (link here).
[4] Coffey, K.R., Marx, R.G. & Neumaier, J.F. DeepSqueak: a deep learning-based system for detection and analysis of ultrasonic vocalizations. Neuropsychopharmacol. 44, 859–868 (2019) (link here).
[5] Sallet, J., Mars, R.B., Noonan, M.P., Andersson, J.L., O’Reilly, J.X., Jbabdi, S., Croxson, P.L., Jenkinson, M., Miller, K.L., Rushworth, M.F., 2011. Social network size affects neural circuits in macaques. Science 334, 697-700.