From Perception to Action: Data-Efficient and Open-Set Learning for Adaptive Human-Robot Interaction