Human–robot collaboration (HRC) is a branch of human–computer interaction (HCI); HRC investigates how humans and robots can interact in different situations. Collaboration can involve service and industrial robots: collaborative robots are usually called cobots. In particular, the interaction between technicians and cobots in an industrial context is deeply related to Industry 4.0, which is quickly changing the nature of factories in order to foster more flexible and efficient manufacturing.

Different levels of “collaborative work” can be identified [1]: synchronized (human and robot work in the same space but at different times), coexistence (human and robot are in the same environment but do not interact), cooperation (human and robot work in the same space at the same time), and collaboration (human and robot execute a task together). This paper focuses on the collaborative level and considers a human–robot collaborative assembly task (as it represents one of the more profitable HRC forms).

The design of collaborative environments is one of the main goals of HRC. The collaboration between humans and machines involves two dimensions [2]: know-how and know-how to cooperate. The first dimension is related to the capability of humans and machines to control a process, whereas the second one concerns the capability to interact with other “agents” (the term agent can denote both humans and machines/cobots). In a collaborative environment, humans and cobots exchange information by means of interfaces [3] that should also allow agents to recognize each other’s intentions to effectively and safely address the common final goal [4].

Augmented reality (AR) superimposes digital assets (e.g., 3D models and animations) into the real world. Manufacturing and repair have been immediately identified as potential application fields for AR [5]; moreover, AR has been selected as one of the nine pillars of Industry 4.0. With these premises, it is not surprising that AR has often been used to convey cobot intentions and status to human operators. Interested readers can find in [6] a systematic literature review of AR interfaces for HRC.

On the other hand, several different technologies have been considered to convey inputs from humans to cobots. Gesture, speech, and haptic are just some of the possible interfaces [7]. Each of these technologies shows some limitations; for instance, noisy environments (such as industrial ones) may adversely affect the performance of speech interfaces, whereas gestures can be limited when human operators execute assembly/maintenance tasks that require both hands. Further, haptic interfaces can be challenging to adapt to human operators’ intentions.

Brain–computer interfaces (BCIs) may overcome these issues; the following five stages implement a BCI: signal acquisition, preprocessing, feature extraction, classification, and output/control [8]. A BCI enables humans to interact with devices/machines without the involvement of peripheral nerves and muscles, thus “only” using control signals generated from electroencephalographic activity. A survey of BCIs can be found in [9].

This paper presents a novel system that uses an AR interface to convey information about the cobot’s intentions to the user and a BCI to provide the cobot (in this case, a robotic arm) commands to pick the parts of a vise and place them close to the user, thus allowing them to be assembled as the final object. The Microsoft HoloLens 2 [10], a wearable see-through head-mounted display (HMD) that enables users to display and interact with computer-generated objects, was used to implement the so-called mixed-reality paradigm. On the other hand, the NextMind [11] device was used to implement the BCI. The NextMind sensor uses EEG technology to detect neural activity from the visual cortex. Machine learning algorithms translate signals into digital commands in real-time. Since the NextMind differs from the traditional BCI in focusing on signals coming from the visual cortex, there is room for evaluating its effectiveness for human/robot collaborative tasks.

User tests were also performed to assess the system’s usability; ten people were involved in tests and an average score of 75.75 was obtained by considering the System Usability Scale (SUS) questionnaire [12]. Further, the NASA-TLX questionnaire [13] was used to assess the global user workload and, as can be expected, the highest scores were related to mental demand.

The paper is organized as follows: Section 2 reviews both augmented reality and brain interfaces to support human–robot collaboration, whereas Section 3 presents the design and implementation details of the proposed solution as well as the tasks considered for tests. Section 4 shows preliminary usability scores; results are analyzed and discussed in Section 5.

The design of AR interfaces has been investigated since the first years of this century [14]. Several design approaches have been used to support users in different AR contexts. Industrial AR apps should be intuitive, robust, evocative, and usable, as outlined in [15].

A recent and detailed analysis of the state-of-the-art related to AR-enhanced interaction between humans and robots can be found in [16]. The authors categorized augmentation into two different categories: on robots and surroundings. The first approach adds information on top of the robot, whereas augmentation of surroundings uses walls, floors, and other physical objects around the robot to anchor computer-generated information. The authors outline how visual augmentation may provide the following benefits:

Facilitated robot programming;

Moreover, twelve main application domains have been identified: domestic and everyday use, industrial applications, entertainment, education and training, social interaction, design and creative tasks, medical and health, telepresence and remote collaboration, mobility and transportation, search and rescue, robots for workspaces, and data physicalization. The proposed work ranks in the industry application domain.

AR interfaces have been deeply investigated to support HRC effectively, as well as the use of BCIs to control robots. An EEG-based BCI has been used to command a semi-autonomous robotic arm by means of motor imagery (MI); in particular, a way to optimize the on-line classification accuracy of an MI-based BCI system by choosing an optimal time window for calculating common spatial patterns (CSP) was proposed in [17]. Functional magnetic resonance imaging (FMRI) has been used as an input device to identify the user’s intentions and convert them into actions performed by a humanoid robot [18]. Another work based on the classification of the P300 waveform is proposed in [19]. The P300 waveform is affected by visual stimuli, and a six-LED-based stimulator was developed in [19]; couples of LEDs were placed at the end of transparent sticks, which represent directions in the Cartesian space. The user can focus on the visual stimuli and control a Mico Kinova robotic arm according to correspondences among LEDs and commands; for instance, LED 1 is used to convey a stimulus to move the arm forward. A survey of BCIs for humanoid robot control can be found in [20].

Hybrid interfaces, which use both AR and BCI, are also known in the literature. The first attempts to propose AR–BCI-based interfaces can be dated back to 2010 [21,22]. In [21], the BCI is based on P300 waveform analysis, and visual stimuli are six cubes on which are placed unique ARToolkit markers [23]; the AR interface recognizes these markers and augments each cube with a number placed on it. Each number codes well-defined positions that a Kuka robot arm uses to pick and place objects. Further, tests involving four users were performed to evaluate system usability. In [22], integration between BCIs based on the steady-state visual evoked potential (SSVEP) and 3D environments is evaluated. A thorough study about BCI in augmented and virtual spaces was later presented in [24].

Commands of interest are displayed on an HMD also in [25]; commands are overlaid in 3D with respect to the perspective view of the robot, which has to perform both a pick and place and a gaming task. The pick and place task was also supported in [26]. Eye tracking allows the user to select the object to pick, and the BCI is used to confirm object selection, trigger switching of an action sequence, and control the aperture and height of the gripper. The AR interface provides users feedback during the grasping and lifting processes, thus efficiently avoiding obstacles. The feasibility of combining AR and BCI technologies has been assessed in [27]; in particular, the authors focused on BCI based on SSVEP and optical see-through augmented reality. The influence of external and internal noise on BCI accuracy was evaluated, and an extended design space for target integration in AR–SSVEP-based applications was also proposed.

Further, mobile robots such as automated/automatic guided vehicles AGVs, drones, and wheelchairs can be controlled by hybrid AR–BCI interfaces; for instance, a drone is guided in [28], and different kinds of BCIs are investigated in [29] to train users to control wheelchairs. In this case, the authors show how the BCI can be effectively combined both with AR and virtual reality (VR) interfaces. AR and VR interfaces provided the same results in terms of usability and user experience.

All the above-mentioned works detect brain activity by means of “invasive” (and sometimes expensive) devices such as neuroheadsets, and they can be considered laboratory experiments. On the other hand, the proposed solution is based on the NextMind [11]. This affordable and ergonomic device can be easily and comfortably used with HMD AR devices such as the Microsoft HoloLens. The proposed solution can be profitably employed in real work scenarios, and the usability tests (see Section 4) clearly outline users’ appreciation.

The NextMind is a wireless device that can be paired with other devices supporting the Bluetooth protocol. It can be attached by a headband, and it should be placed at the back of the head, where the visual cortex is located, with its lower part over the inion. Nine electrodes, which should always be in contact with the scalp, detect brain activity. The NextMind core is mainly built around two components: NeuroTags and NeuroManager. NeuroTags are identified by blinking/flickering textures (different NeuroTags are related to textures of blinking at different frequencies) that make any object of the scene “mind-controllable”; a NeuroTag represents a unique visual signature that generates a response in the user’s brain. On the other hand, NeuroManager manages the communication between the NeuroTags and the NextMind engine, which is in charge of processing signals received from the electrodes. No more than 10 NeuroTags should be inserted in the same scene; moreover, the NextMind automatically disables the NeuroTags outside the user’s field of view. Figure 1 shows an example of a NeuroTag associated with a flickering texture. The three green segments move to compose a triangle when the visual signature has been detected; this is the standard feedback provided by the NextMind. The NextMind has been used to compare selection performance between gestures and BCI in AR interfaces [30]; results clearly outline the potential of this device.

Contrary to the presented state-of-the-art, the NextMind focuses only on visual cortex signals, providing an interesting and intuitive selection mechanism that is assessed in this paper to verify its effectiveness for human–robot collaborative tasks.

This section presents HW and SW system architecture, the user interface (UI), and the selected tasks for tests.

To the best of the authors’ knowledge, this is the only work that uses the NextMind to design and develop a BCI–AR interface to foster human–robot collaboration. The NextMind is an affordable and ergonomic device designed to be easily and comfortably used together with AR headsets, such as the Microsoft HoloLens. Moreover, the NextMind does not require any kind of waveform analysis since it provides an intuitive selection mechanism based on visual cortex signals that are automatically detected by the device APIs. Therefore, other BCI interfaces were not considered for comparison. Instead, we focused on evaluating the usability of the proposed system in order to assess limitations and users’ appreciation for an industrial assembly task.

This section presents the results analysis along with the current limitations and possible future works.

This paper presents BARI, a novel hybrid brain and AR interface to foster human–robot collaboration in assembly tasks. AR is used to convey to the user both information related to the robot status and graphical representations of the parts to be assembled; on the other hand, a BCI is used to give the robot commands to pick selected objects. Since the NextMind differs from traditional BCI interfaces by focusing only on signals coming from the visual cortex, it provides an interesting and intuitive selection mechanism that has been assessed to verify its effectiveness for human–robot collaborative tasks. An evaluation involving ten users confirmed the potential of these hybrid interfaces to support HRC.

Although obtained results can only be considered as preliminary, and more tests are necessary to validate the system, good levels of usability are the essential prerequisite for further investigation. Overall, the users evaluated the experience positively, and the system could be effectively deployed in industrial assembly scenarios.

The usability evaluation proved that poor calibration or distracting elements in the real environment might interfere with the proposed system. Further tests should clarify how much these factors impact system usability and may suggest best practices for the user interface design. Alternative selection methods such as gestures and gaze tracking will also be compared with inputs by brain activity. Finally, it is possible to apply the proposed interface to other robotic domains, such as service robotics.