ReHRI’17 – International Workshop on reproducible HRI experiments: scientific endeavors, benchmarking and standardization
An international workshop held in conjunction with the 12th Annual Conference on Human-Robot Interaction (HRI2017).
LATEST NEWS: The papers and the presentations are now available (under the “program” tab)!
The current metrics, scenario settings, corpora and databases applied in HRI experiments are tailored to individual cases and, thus, are not suitable to compare and benchmark results across laboratories and systems. The robotics scientific community should agree on processes, procedures and metrics to establish a standardized benchmarking platform that is widely applicable across all areas of robotics. This workshop will bring together research scientists, representatives of robotics companies and members of standardization working groups. We expect that a fruitful discussion in the definition of experimental scenarios and protocols in HRI will assist to highlight the latest advances in the field and lead to the development of agreed and shared evaluation platforms for objective benchmarking of the broad and ever-growing range of robot applications.
The primary audience of the proposed workshop is intended to be robot researchers/developers and practitioners from academia and industry working in all areas of HRI including industrial, personal care, social, medical and entertainment robotics. However, the workshop has a clear focus on how research and standardization should work together effectively in the field of HRI benchmarking, such that useful and thoughtful working groups for standards are created and a roadmap for the coming years is generated.
Topics of interest include, but are not limited to:
A non-exhaustive list of topics to be covered in the workshop is as follows:
- Benchmarking HRI in robotic applications
- Performance indicators for HRI
- Standardization activities for HRI experiments
- User studies in Human – robot collaboration
- Human – robot interfaces and evaluation of user-experience
- Innovative design of commercial HRI solutions
- Safety, physical contacts, technical aspects, limitations and regulations influencing HRI design
- Evaluation of HRI group dynamics
- HRI metrics for teams
- Ethics in HRI experiments
- Corpora for HRI
- Creation of ISO working group on HRI
- Repositories of datasets for benchmarking
- Industrial, medical, social and personal care robots
- Robotics competitions and their standardized experimental settings
- Presentation of an editorial proposal, a book on the workshop topics
Authors of papers select one of following presentation types:
– lightning talk (5 minutes)
– poster presentation
– oral presentation(15-20 minutes)
(note: we will also include a few selected papers or invited speakers)
Submissions will be reviewed and accepted based on relevance to the workshop topics, technical quality, novelty, and presentation type.
The maximum file size is 2 MB. Submissions should be in PDF format through Easy Chair:
Submission deadline (extended):
January 31, 2017 February 07, 2017
February 14, 2017
Workshop event: March 6, 2017
The introduction will aim at harmonising the setting, pointing out the main points of discussion and agreeing on how the workshop activities will be conducted. The initial part of the workshop will also include presentations from multiple experts in the primary relevant fields of interest. Once a common ground has been established, it will be easier to convey participants with diverse experiences and backgrounds towards the discussion topic.
Early stage researchers will be encouraged to contribute to the midday workshop session through the presentation of poster work and very short lightning talks. Contrary to formal presentations, posters do not require a long-term research experience, which allows early stage researchers to submit their contributions. The workshop would be an ideal time for young researchers to promote their work and share their ideas with experienced researchers in one-to-one discussions in the informal environment of the workshop.
The workshop will continue with two split sessions. Initially parallel brainstorming discussions about benchmarking of HRI experiments will take place. Afterwards, the participants will focus on a brainstorming session about establishing holistic HRI scenarios. These sessions will provide the opportunity for smaller group discussions and greater involvement from the participants. The results of the split sessions will be the contribution to the final discussion (wrap up session), which aims at establishing common ground on benchmarking and evaluation of reproducible HRI experiments and at providing input to standardisation, benchmarking and robotics competition working groups.
|08:00 – 09:00||Registration|
|09:00 – 09:10||Welcome – Opening|
|09:10 – 09:30||Oral presentation #1 – “Challenges in running long-term, multi-site studies with the elderly in the ENRICHME project” – François Ferland – [presentation in pdf]|
|09:30 – 09:50||Oral presentation #2 – “Challenges for benchmarking when working with children and robots” – Kerstin Dautenhahn – [presentation in pdf]|
|09:50 – 10:10||Oral presentation #3 – “A project for Artificial stupidity: technology, humorous dialogues and user engagement” – Jacopo Fo – [presentation in pdf]|
|10:10 – 10:20||Lighting Talk #1 – “Studying Human-Robot Collaboration in an Artistic Creative Process” – Johan Kildal – [presentation in pdf][paper]|
|10:20 – 10:30||Lighting Talk #2 – “Contact Pressure Distribution as an Evaluation Metric for Human-Robot Hand Interactions” – Espen Knoop – [paper]|
|10:30 – 11:00||Coffee Break|
|11:00 – 11:10||Lighting Talk #3 – “Towards a Study to Assess Conversation-based Interaction between People with Dementia and Robots” – Dagoberto Cruz-Sandoval – [presentation in pdf][paper][poster]|
|11:10 – 11:30||Oral presentation #4 – “Trust in and dependence on imperfect automation” – Xi Jessie Yang – [presentation in pdf]|
|11:30 – 11:50||Oral presentation #5 – “Building personalised instructional robots” – Ginevra Castellano|
|11:50 – 12:00||Lighting Talk #4 – “Lessons learned from EMSHRI workshops” – Céline Jost – [presentation in pdf]|
|12:00 – 13:00||Lunch Break|
|13:00 – 15:00||Split brainstorming sessions on benchmarking HRI experiments|
|15:00 – 15:30||Coffee Break|
|15:30 – 17:00||Brainstorming session on holistic HRI scenarios|
|17:00 – 17:30||Wrap up and conclusion|