Workshop @ HRI’17

ReHRI’17 – International Workshop on reproducible HRI experiments: scientific endeavors, benchmarking and standardization

An international workshop held in conjunction with the 12th Annual Conference on Human-Robot Interaction (HRI2017).

Official website of the conference

LATEST NEWS: The papers and the presentations are now available (under the “program” tab)!

This workshop aims at advancing the topic of the relationships between robotics science and experiments in HRI. The full-day workshop follows on previous events in interdisciplinary venues (ICRA’13, HRI’14, IROS’14-’15, ERF‘15-’16) focusing on efforts to establish reproducible, standardized experiments to adequately benchmark and evaluate human-robot interaction, scientific achievements and results in the areas of industrial, medical, social, service and personal-care robots. The words “standard” and “standardized” sometimes are misunderstood as fixing and blocking the freedom of the researcher to create new experiments and therefore to advance the scientific knowledge. As per this workshop, the words must be intended as a “way of experimenting temporarily agreed by the community, that is evolving and tracking the scientific progress through continuous revision by the community itself”.

The current metrics, scenario settings, corpora and databases applied in HRI experiments are tailored to individual cases and, thus, are not suitable to compare and benchmark results across laboratories and systems. The robotics scientific community should agree on processes, procedures and metrics to establish a standardized benchmarking platform that is widely applicable across all areas of robotics. This workshop will bring together research scientists, representatives of robotics companies and members of standardization working groups. We expect that a fruitful discussion in the definition of experimental scenarios and protocols in HRI will assist to highlight the latest advances in the field and lead to the development of agreed and shared evaluation platforms for objective benchmarking of the broad and ever-growing range of robot applications.

Target Audience

The primary audience of the proposed workshop is intended to be robot researchers/developers and practitioners from academia and industry working in all areas of HRI including industrial, personal care, social, medical and entertainment robotics. However, the workshop has a clear focus on how research and standardization should work together effectively in the field of HRI benchmarking, such that useful and thoughtful working groups for standards are created and a roadmap for the coming years is generated.

Topics of interest include, but are not limited to:

A non-exhaustive list of topics to be covered in the workshop is as follows:

  • Benchmarking HRI in robotic applications
  • Performance indicators for HRI
  • Standardization activities for HRI experiments
  • User studies in Human – robot collaboration
  • Human – robot interfaces and evaluation of user-experience
  • Innovative design of commercial HRI solutions
  • Safety, physical contacts, technical aspects, limitations and regulations influencing HRI design
  • Evaluation of HRI group dynamics
  • HRI metrics for teams
  • Ethics in HRI experiments
  • Corpora for HRI
  • Creation of ISO working group on HRI
  • Repositories of datasets for benchmarking
  • Industrial, medical, social and personal care robots
  • Robotics competitions and their standardized experimental settings
  • Presentation of an editorial proposal, a book on the workshop topics
We invite extended abstracts and short papers of 2-4 pages in the standard ACM HRI conference format. Submissions should be in accordance with the topics of interest, and they will be available on the workshop’s website.

Authors of papers select one of following presentation types:

– lightning talk (5 minutes) – poster presentation – oral presentation(15-20 minutes)

(note: we will also include a few selected papers or invited speakers)

Submissions will be reviewed and accepted based on relevance to the workshop topics, technical quality, novelty, and presentation type.

Templates: http://www.acm.org/sigs/publications/proceedings-templates

The maximum file size is 2 MB. Submissions should be in PDF format through Easy Chair: https://easychair.org/conferences/?conf=rehri17

Important Dates:

Submission deadline (extended): January 31, 2017 February 07, 2017

Acceptance notification: February 14, 2017 

Workshop event: March 6, 2017

Active discussion will be promoted by restricting the time for formal presentations to one-third of the morning workshop time. The workshop is structured to include three parts: an initial introduction to the topic by position presentations, a second part aimed at obtaining as many diverse contributions as possible and a third part which would include the audience in two brainstorming sessions.

The introduction will aim at harmonising the setting, pointing out the main points of discussion and agreeing on how the workshop activities will be conducted. The initial part of the workshop will also include presentations from multiple experts in the primary relevant fields of interest. Once a common ground has been established, it will be easier to convey participants with diverse experiences and backgrounds towards the discussion topic.

Early stage researchers will be encouraged to contribute to the midday workshop session through the presentation of poster work and very short lightning talks. Contrary to formal presentations, posters do not require a long-term research experience, which allows early stage researchers to submit their contributions. The workshop would be an ideal time for young researchers to promote their work and share their ideas with experienced researchers in one-to-one discussions in the informal environment of the workshop.

The workshop will continue with two split sessions. Initially parallel brainstorming discussions about benchmarking of HRI experiments will take place. Afterwards, the participants will focus on a brainstorming session about establishing holistic HRI scenarios. These sessions will provide the opportunity for smaller group discussions and greater involvement from the participants. The results of the split sessions will be the contribution to the final discussion (wrap up session), which aims at establishing common ground on benchmarking and evaluation of reproducible HRI experiments and at providing input to standardisation, benchmarking and robotics competition working groups.

Dimitrios Chrysostomou, Aalborg University, Denmark | e-mail: dimi@m-tech.aau.dk

Paolo Barattini, Kontor46, Torino, Italy | paolo.barattini@yahoo.it

Johan Kildal, IK4-Tekniker, Spain | johan.kildal@tekniker.es

Yue Wang, Clemson University, USA | yue6@clemson.edu

Jacopo Fo, L.U.A., Gubbio, Italy | jacopofo@gmail.com

Kerstin Dautenhahn, University of Hertfordshire, UK | k.dautenhahn@herts.ac.uk

Francois Ferland, ENSTA ParisTech, Palaiseau, France | francois.ferland@ensta-paristech.fr

Adriana Tapus, ENSTA ParisTech, Palaiseau, France | adriana.tapus@ensta-paristech.fr

Gurvinder Singh Virk, InnotecUK Ltd, Cambridge, UK | gurvinder.virk@innotecUK.com

08:00 – 09:00 Registration 
09:00 – 09:10 Welcome – Opening
09:10 – 09:30 Oral presentation #1 – “Challenges in running long-term, multi-site studies with the elderly in the ENRICHME project” – François Ferland – [presentation in pdf]
09:30 – 09:50 Oral presentation #2 – “Challenges for benchmarking when working with children and robots” – Kerstin Dautenhahn – [presentation in pdf]
09:50 – 10:10 Oral presentation #3 – “A project for Artificial stupidity: technology, humorous dialogues and user engagement” – Jacopo Fo  – [presentation in pdf]
10:10 – 10:20 Lighting Talk #1 – “Studying Human-Robot Collaboration in an Artistic Creative Process” – Johan Kildal  – [presentation in pdf][paper]
10:20 – 10:30 Lighting Talk #2 – “Contact Pressure Distribution as an Evaluation Metric for Human-Robot Hand Interactions” – Espen Knoop  – [paper]
10:30 – 11:00 Coffee Break
11:00 – 11:10 Lighting Talk #3 – “Towards a Study to Assess Conversation-based Interaction between People with Dementia and Robots” – Dagoberto Cruz-Sandoval  – [presentation in pdf][paper][poster]
11:10 – 11:30 Oral presentation #4 – “Trust in and dependence on imperfect automation” – Xi Jessie Yang – [presentation in pdf]
11:30 – 11:50 Oral presentation #5 – “Building personalised instructional robots” – Ginevra Castellano
11:50 – 12:00 Lighting Talk #4 – “Lessons learned from EMSHRI workshops” – Céline Jost – [presentation in pdf]
12:00 – 13:00 Lunch Break
13:00 – 15:00 Split brainstorming sessions on benchmarking HRI experiments
15:00 – 15:30 Coffee Break
15:30 – 17:00 Brainstorming session on holistic HRI scenarios
17:00 – 17:30 Wrap up and conclusion