2019
|
P P Filntisis, N Efthymiou, P Koutras, G Potamianos, P Maragos Fusing Body Posture With Facial Expressions for Joint Recognition of Affect in Child–Robot Interaction Journal Article IEEE Robotics and Automation Letters (with IROS option), 4 (4), pp. 4011-4018, 2019. Abstract | BibTeX | Links: [PDF] @article{8769871,
title = {Fusing Body Posture With Facial Expressions for Joint Recognition of Affect in Child–Robot Interaction},
author = {P P Filntisis and N Efthymiou and P Koutras and G Potamianos and P Maragos},
url = {http://robotics.ntua.gr/wp-content/uploads/sites/2/RAL_2019-5.pdf},
doi = {10.1109/LRA.2019.2930434},
year = {2019},
date = {2019-10-01},
journal = {IEEE Robotics and Automation Letters (with IROS option)},
volume = {4},
number = {4},
pages = {4011-4018},
abstract = {In this letter, we address the problem of multi-cue affect recognition in challenging scenarios such as child–robot interaction. Toward this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child–robot interaction database of emotional expressions collected by us, as well as on the GEneva multimodal emotion portrayal public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In this letter, we address the problem of multi-cue affect recognition in challenging scenarios such as child–robot interaction. Toward this goal we propose a method for automatic recognition of affect that leverages body expressions alongside facial ones, as opposed to traditional methods that typically focus only on the latter. Our deep-learning based method uses hierarchical multi-label annotations and multi-stage losses, can be trained both jointly and separately, and offers us computational models for both individual modalities, as well as for the whole body emotion. We evaluate our method on a challenging child–robot interaction database of emotional expressions collected by us, as well as on the GEneva multimodal emotion portrayal public database of acted emotions by adults, and show that the proposed method achieves significantly better results than facial-only expression baselines. |
2018
|
N. Efthymiou, P. Koutras, P. ~P. Filntisis, G. Potamianos, P. Maragos MULTI-VIEW FUSION FOR ACTION RECOGNITION IN CHILD-ROBOT INTERACTION Conference Proc. IEEE Int'l Conf. on Image Processing, Athens, Greece, 2018. Abstract | BibTeX | Links: [PDF] @conference{efthymiou18action,
title = {MULTI-VIEW FUSION FOR ACTION RECOGNITION IN CHILD-ROBOT INTERACTION},
author = { N. Efthymiou and P. Koutras and P. ~P. Filntisis and G. Potamianos and P. Maragos},
url = {http://robotics.ntua.gr/wp-content/uploads/sites/2/EfthymiouKoutrasFilntisis_MultiViewFusActRecognChildRobotInteract_ICIP18.pdf},
year = {2018},
date = {2018-10-01},
booktitle = {Proc. IEEE Int'l Conf. on Image Processing},
address = {Athens, Greece},
abstract = {Answering the challenge of leveraging computer vision methods in order to enhance Human Robot Interaction (HRI) experience, this work explores methods that can expand the capabilities of an action recognition system in such tasks. A multi-view action recognition system is proposed for integration in HRI scenarios with special users, such as children, in which there is limited data for training and many state-of-the-art techniques face difficulties. Different feature extraction approaches, encoding methods and fusion techniques are combined and tested in order to create an efficient system that recognizes children pantomime actions. This effort culminates in the integration of a robotic platform and is evaluated under an alluring Children Robot Interaction scenario.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Answering the challenge of leveraging computer vision methods in order to enhance Human Robot Interaction (HRI) experience, this work explores methods that can expand the capabilities of an action recognition system in such tasks. A multi-view action recognition system is proposed for integration in HRI scenarios with special users, such as children, in which there is limited data for training and many state-of-the-art techniques face difficulties. Different feature extraction approaches, encoding methods and fusion techniques are combined and tested in order to create an efficient system that recognizes children pantomime actions. This effort culminates in the integration of a robotic platform and is evaluated under an alluring Children Robot Interaction scenario. |
A Tsiami, P Koutras, Niki Efthymiou, P P Filntisis, G Potamianos, P Maragos Multi3: Multi-sensory Perception System for Multi-modal Child Interaction with Multiple Robots Conference IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 2018. Abstract | BibTeX | Links: [PDF] @conference{multi3,
title = {Multi3: Multi-sensory Perception System for Multi-modal Child Interaction with Multiple Robots},
author = {A Tsiami and P Koutras and Niki Efthymiou and P P Filntisis and G Potamianos and P Maragos},
url = {http://robotics.ntua.gr/wp-content/publications/2018_TsiamiEtAl_Multi3-MultisensorMultimodalChildInteractMultRobots_ICRA.pdf},
year = {2018},
date = {2018-05-01},
booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
address = {Brisbane, Australia},
abstract = {Child-robot interaction is an interdisciplinary research area that has been attracting growing interest, primarily focusing on edutainment applications. A crucial factor to the successful deployment and wide adoption of such applications remains the robust perception of the child's multimodal actions, when interacting with the robot in a natural and untethered fashion. Since robotic sensory and perception capabilities are platform-dependent and most often rather limited, we propose a multiple Kinect-based system to perceive the child-robot interaction scene that is robot-independent and suitable for indoors interaction scenarios. The audio-visual input from the Kinect sensors is fed into speech, gesture, and action recognition modules, appropriately developed in this paper to address the challenging nature of child-robot interaction. For this purpose, data from multiple children are collected and used for module training or adaptation. Further, information from the multiple sensors is fused to enhance module performance.
The perception system is integrated in a modular multi-robot architecture demonstrating its flexibility and scalability with different robotic platforms.
The whole system, called Multi3, is evaluated, both objectively at the module level and subjectively in its entirety, under appropriate child-robot interaction scenarios containing several carefully designed games between children and robots.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Child-robot interaction is an interdisciplinary research area that has been attracting growing interest, primarily focusing on edutainment applications. A crucial factor to the successful deployment and wide adoption of such applications remains the robust perception of the child's multimodal actions, when interacting with the robot in a natural and untethered fashion. Since robotic sensory and perception capabilities are platform-dependent and most often rather limited, we propose a multiple Kinect-based system to perceive the child-robot interaction scene that is robot-independent and suitable for indoors interaction scenarios. The audio-visual input from the Kinect sensors is fed into speech, gesture, and action recognition modules, appropriately developed in this paper to address the challenging nature of child-robot interaction. For this purpose, data from multiple children are collected and used for module training or adaptation. Further, information from the multiple sensors is fused to enhance module performance.
The perception system is integrated in a modular multi-robot architecture demonstrating its flexibility and scalability with different robotic platforms.
The whole system, called Multi3, is evaluated, both objectively at the module level and subjectively in its entirety, under appropriate child-robot interaction scenarios containing several carefully designed games between children and robots. |
A Tsiami, P P Filntisis, N Efthymiou, P Koutras, G Potamianos, P Maragos FAR-FIELD AUDIO-VISUAL SCENE PERCEPTION OF MULTI-PARTY HUMAN-ROBOT INTERACTION FOR CHILDREN AND ADULTS Conference Proc. IEEE Int'l Conf. Acous., Speech, and Signal Processing (ICASSP), Calgary, Canada, 2018. Abstract | BibTeX | Links: [PDF] @conference{tsiamifar,
title = {FAR-FIELD AUDIO-VISUAL SCENE PERCEPTION OF MULTI-PARTY HUMAN-ROBOT INTERACTION FOR CHILDREN AND ADULTS},
author = {A Tsiami and P P Filntisis and N Efthymiou and P Koutras and G Potamianos and P Maragos},
url = {http://robotics.ntua.gr/wp-content/publications/2018_TsiamiEtAl_FarfieldAVperceptionHRI-ChildrenAdults_ICASSP.pdf},
year = {2018},
date = {2018-04-01},
booktitle = {Proc. IEEE Int'l Conf. Acous., Speech, and Signal Processing (ICASSP)},
address = {Calgary, Canada},
abstract = {Human-robot interaction (HRI) is a research area of growing interest
with a multitude of applications for both children and adult user groups,
as, for example, in edutainment and social robotics.
Crucial, however, to its wider adoption
remains the robust perception of HRI scenes in natural, untethered,
and multi-party interaction scenarios, across user groups.
Towards this goal, we investigate three focal HRI perception modules
operating on data from multiple audio-visual sensors
that observe the HRI scene from the far-field,
thus bypassing limitations and platform-dependency
of contemporary robotic sensing.
In particular, the developed modules fuse
intra- and/or inter-modality data streams to perform:
(i) audio-visual speaker localization;
(ii) distant speech recognition; and
(iii) visual recognition of hand-gestures.
Emphasis is also placed on ensuring high speech and gesture recognition rates
for both children and adults.
Development and objective evaluation of the three modules
is conducted on a corpus of both user groups,
collected by our far-field multi-sensory setup,
for an interaction scenario of a question-answering
``guess-the-object'' collaborative HRI game with a ``Furhat'' robot.
In addition, evaluation of the game
incorporating the three developed modules is reported.
Our results demonstrate robust far-field audio-visual perception
of the multi-party HRI scene.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
Human-robot interaction (HRI) is a research area of growing interest
with a multitude of applications for both children and adult user groups,
as, for example, in edutainment and social robotics.
Crucial, however, to its wider adoption
remains the robust perception of HRI scenes in natural, untethered,
and multi-party interaction scenarios, across user groups.
Towards this goal, we investigate three focal HRI perception modules
operating on data from multiple audio-visual sensors
that observe the HRI scene from the far-field,
thus bypassing limitations and platform-dependency
of contemporary robotic sensing.
In particular, the developed modules fuse
intra- and/or inter-modality data streams to perform:
(i) audio-visual speaker localization;
(ii) distant speech recognition; and
(iii) visual recognition of hand-gestures.
Emphasis is also placed on ensuring high speech and gesture recognition rates
for both children and adults.
Development and objective evaluation of the three modules
is conducted on a corpus of both user groups,
collected by our far-field multi-sensory setup,
for an interaction scenario of a question-answering
``guess-the-object'' collaborative HRI game with a ``Furhat'' robot.
In addition, evaluation of the game
incorporating the three developed modules is reported.
Our results demonstrate robust far-field audio-visual perception
of the multi-party HRI scene. |
Jack Hadfield, Petros Koutras, Niki Efthymiou, Gerasimos Potamianos, Costas S Tzafestas, Petros Maragos Object assembly guidance in child-robot interaction using RGB-D based 3d tracking Conference 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE 2018. BibTeX | Links: [PDF] @conference{hadfield2018object,
title = {Object assembly guidance in child-robot interaction using RGB-D based 3d tracking},
author = {Jack Hadfield and Petros Koutras and Niki Efthymiou and Gerasimos Potamianos and Costas S Tzafestas and Petros Maragos},
url = {http://robotics.ntua.gr/wp-content/uploads/sites/2/2018_HadfieldEtAl_ObjectAssemblyGuidance-ChildRobotInteraction_IROS.pdf},
year = {2018},
date = {2018-01-01},
booktitle = {2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
pages = {347--354},
organization = {IEEE},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
|
2017
|
A Zlatintsi, P Koutras, G Evangelopoulos, N Malandrakis, N Efthymiou, K Pastra, A Potamianos, P Maragos COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization Journal Article EURASIP Journal on Image and Video Processing, 54 , pp. 1–24, 2017. Abstract | BibTeX | Links: [PDF] @article{ZKE+17,
title = {COGNIMUSE: a multimodal video database annotated with saliency, events, semantics and emotion with application to summarization},
author = {A Zlatintsi and P Koutras and G Evangelopoulos and N Malandrakis and N Efthymiou and K Pastra and A Potamianos and P Maragos},
url = {http://robotics.ntua.gr/wp-content/publications/Zlatintsi+_COGNIMUSEdb_EURASIP_JIVP-2017.pdf},
doi = {doi 10.1186/s13640-017-0194},
year = {2017},
date = {2017-01-01},
journal = {EURASIP Journal on Image and Video Processing},
volume = {54},
pages = {1--24},
abstract = {Research related to computational modeling for machine-based understanding requires ground truth data for training, content analysis, and evaluation. In this paper, we present a multimodal video database, namely COGNIMUSE, annotated with sensory and semantic saliency, events, cross-media semantics, and emotion. The purpose of this database is manifold; it can be used for training and evaluation of event detection and summarization algorithms, for classification and recognition of audio-visual and cross-media events, as well as for emotion tracking. In order to enable comparisons with other computational models, we propose state-of-the-art algorithms, specifically a unified energy-based audio-visual framework and a method for text saliency computation, for the detection of perceptually salient events from videos. Additionally, a movie summarization system for the automatic production of summaries is presented. Two kinds of evaluation were performed, an objective based on the saliency annotation of the database and an extensive qualitative human evaluation of the automatically produced summaries, where we investigated what composes high-quality movie summaries, where both methods verified the appropriateness of the proposed methods. The annotation of the database and the code for the summarization system can be found at http://cognimuse.cs.ntua.gr/database.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Research related to computational modeling for machine-based understanding requires ground truth data for training, content analysis, and evaluation. In this paper, we present a multimodal video database, namely COGNIMUSE, annotated with sensory and semantic saliency, events, cross-media semantics, and emotion. The purpose of this database is manifold; it can be used for training and evaluation of event detection and summarization algorithms, for classification and recognition of audio-visual and cross-media events, as well as for emotion tracking. In order to enable comparisons with other computational models, we propose state-of-the-art algorithms, specifically a unified energy-based audio-visual framework and a method for text saliency computation, for the detection of perceptually salient events from videos. Additionally, a movie summarization system for the automatic production of summaries is presented. Two kinds of evaluation were performed, an objective based on the saliency annotation of the database and an extensive qualitative human evaluation of the automatically produced summaries, where we investigated what composes high-quality movie summaries, where both methods verified the appropriateness of the proposed methods. The annotation of the database and the code for the summarization system can be found at http://cognimuse.cs.ntua.gr/database. |
2015
|
A Zlatintsi, P Koutras, N Efthymiou, P Maragos, A Potamianos, K Pastra Quality Evaluation of Computational Models for Movie Summarization Conference Costa Navarino, Messinia, Greece, 2015. Abstract | BibTeX | Links: [PDF] @conference{ZKE+15,
title = {Quality Evaluation of Computational Models for Movie Summarization},
author = {A Zlatintsi and P Koutras and N Efthymiou and P Maragos and A Potamianos and K Pastra},
url = {http://robotics.ntua.gr/wp-content/publications/ZlatintsiEtAl_MovieSumEval-QoMEX2015.pdf},
year = {2015},
date = {2015-05-01},
address = {Costa Navarino, Messinia, Greece},
abstract = {In this paper we present a movie summarization system and we investigate what composes high quality movie summaries in terms of user experience evaluation. We propose state-of-the-art audio, visual and text techniques for the detection of perceptually salient events from movies. The evaluation of such computational models is usually based on the comparison of the similarity between the system-detected events and some ground-truth data. For this reason, we have developed the MovSum movie database, which includes sensory and semantic saliency annotation as well as cross-media relations, for objective evaluations. The automatically produced movie summaries were qualitatively evaluated, in an extensive human evaluation, in terms of informativeness and enjoyability accomplishing very high ratings up to 80% and 90%, respectively, which verifies the appropriateness of the proposed methods.},
keywords = {},
pubstate = {published},
tppubtype = {conference}
}
In this paper we present a movie summarization system and we investigate what composes high quality movie summaries in terms of user experience evaluation. We propose state-of-the-art audio, visual and text techniques for the detection of perceptually salient events from movies. The evaluation of such computational models is usually based on the comparison of the similarity between the system-detected events and some ground-truth data. For this reason, we have developed the MovSum movie database, which includes sensory and semantic saliency annotation as well as cross-media relations, for objective evaluations. The automatically produced movie summaries were qualitatively evaluated, in an extensive human evaluation, in terms of informativeness and enjoyability accomplishing very high ratings up to 80% and 90%, respectively, which verifies the appropriateness of the proposed methods. |
