Processing Vision
Toby Collett >

-

Developer-oriented Visualisation of a Robot Program:
An Augmented Reality Approach
Research Paper by Toby Collett

Introduction

Currently when researchers wish to program robot applications they tend to use an ad hoc combination of tools and languages selected from both traditional application development tools and proprietary robotic tools. However, robotic systems have a number of unique challenges that these tools are not designed to target, particularly:

• The nature of the robot environment
• Dynamic, realtime
• Unexpected variations cause non-repeatable behaviour and unexpected conditions
• The nature of robot being debugged
• Mobility (i.e. the debugging target hardware moves away from the programmer)
• A large number of devices for input, output and storage, which far exceed human programmers’ familiar senses and effectors, compared to the few devices in a desktop or server
• Wide variations in hardware and interfaces, as opposed to the highly commoditized desktop
• The nature of mobile robot tasks
• Emphasis on geometry and 3D space
• Complex data types and environments
• Potentially uninterruptible
• Simultaneous and unrelated activity on many inputs and outputs

Some of these challenges are also exhibited by some traditional, non-robotic applications, but they have generally not been targeted by mainstream development tools. In addition, it is worth noting that concurrency is still not handled well in many debugging environments, although it is often present in traditional applications. This weakness carries over to the robot development domain.

The result is that each robotics lab develops its own tools to aid in debugging the complexities of robot programming, often with different tools for each robot in the lab. Some tools, in particular Player/Stage1 are emerging as de facto standards due to the large variety of supported hardware and growing user community. We believe that standardisation in this form is a good thing and aim to promote it by player based tools that support developers in the task of programming robots.

A common thread of these challenges is that the environment and the robot’s interaction with the environment are what makes robot programming different and challenging. In other words it is the programmer’s lack of understanding of the robot’s world view that makes robot programs difficult to code and debug.

Augmented reality (AR) provides an ideal presentation of the robot’s world view; it displays robot data layered within the real environment. By viewing the data in context with the real world the developer is able to compare the robot’s world view against the ground truth of the read world image without needing to know the exact state of the world at all times. This makes clear not only the robot’s world view but also the discrepancies with the real world. The main challenge in AR is to accurately track the human user so that the overlaid data is accurately positioned as the human viewer changes position and orientation. However this is generally a simpler problem than the alternative, which is to track the entirety of a potentially large and dynamic real world environment and explicitly calculate a comparison between the robot view and the actual world view.

The aim of this work is to enhance the developer’s interface to the robot by providing Augmented Reality visualisation of Player’s interfaces.

Requirements for Effective Interaction

Breazeal2 identifies three key requirements for effective human–robot interaction; overlapping perceptual space, appropriate interaction distance, and safety. In addition if the interaction is to be natural, it must use as many interface media as possible.

Both the human user and the robot have a perceptual space where communication is meaningful, and it is the overlap of these two spaces that is the interaction space of the humanrobot coupling. The interaction space for input may be different from the output space. For example a robot that knows how to represent an emotion using facial expressions but cannot recognise the facial expressions of its interaction partner will have differing input and output spaces. Any mismatch between the perceptual input and output space may put additional cognitive load on the user, who will have to perform additional translations before communicating with the robot.

This paper focuses on providing a shared perceptual space for the robot-developer interaction, which is where the human and the robot have overlapping perceptual abilities. By allowing the developer to understand the robots world view we are effectively enhancing the shared space as we are allowing the developer to view the world in the same way as the robot.

Related Work

Player1 has a built in tool for visualising its interfaces, playerv, however this tool provides the visualisation in isolation. By providing an aumented reality interface for the visualisation we show the data in context with the measured objects making understanding the data more intuitive. Figure 1 shows and example visualisation of laser data from the playerv tool.

Figure 1. playerv laser visualisation

Some work has already been undertaken using augmented reality with robot systems. Shikata et al3 present an algorithm evaluation system that uses virtual robots in a virtual world to test the performance of avoidance algorithms when human users are involved. The main advantage is that a real human is used rather than a scripted one, thus giving more accurate behaviour while at the same time the virtual robots mean there are no safety issues. In particular the system is used to test avoidance algorithms with no danger to the human user from collisions occurring when the algorithm fails. Milgram et al used AR in creating a virtual measuring tape and a virtual tether to assist inserting a peg in a hole4, 5. Freund et al6 use AR to more simply display complex 3D state information about autonomous systems. Raghavan et al7 describe an interactive tool for augmenting the real scene during mechanical assembly. Pettersen et al8 present a method to improve the teaching of way points to a painting robot, using AR to show the simulated painting process implied by the taught way points. Brujic-Okretic et al9 describe an AR system that integrates graphical and sensory information for remotely controlling a vehicle. Daily et al10 use AR to present information from a swarm robotics network designed for search and rescue. Recently KUKA have also begun to investigate AR to help visualise data during training11.

Amstutz and Fagg12 describe an implementation for representing the combined data of a large number of sensors on multiple mobile platforms (potentially robots), using AR and VR to display the information.

An Augmented Reality Toolkit for Player Interface Visualisation

The remainder of the paper focuses on our implementation of of an augmented reality toolkit for Player interface visualisation. Given the rapidly changing nature of available AR hardware and software techniques any systems making use of the technology must be flexible and independent of any specific AR implementation.

The full source for the library is available for download from http://www.ece.auckland.ac.nz/.

The developed augmented reality toolkit has the following features:

• Modular and flexible in order to make use of core technology advances
• Supports a core set of Player interfaces
• Robust, able to be run as a permanent installation in a robots lab
• Detects player server connections and disconnections
• Underlying augmented reality system must have sufficient accuracy for developer interaction

The software architecture used for this research has been designed in a highly modular fashion allowing for individual components to be replaced as required. The prototype also provides reference implementations of each type of object, particularly focusing on using the Player/Stage project (2005), and Video for Linux capture devices; a full list of implemented objects is given in Table I.

Table 1. Key objects implemented in the prototype system

The architecture is centred around an output device. Each output device contains a capture device for grabbing the real world frame (this could be a null object for optical see through AR or for a purely virtual environment) and a camera device for returning the camera parameters (including pose). Also, the output device maintains three component lists: secondary outputs (which monitor the interface, i.e. movie capture); preprocessing objects (used for image based position tracking); and finally a list of render item pairs. Each Render Pair consists of a render object that performs the actual rendering of a virtual element and a position object that defines where it should be rendered.

Fig. 2 shows the software structure.

Figure 2. ARDev software architecture

In order to accomplish the goal of a permanent lab setup the ability to detect the presence of robots was needed. This presence is in two forms, first the system must cope with network or software presence, if a robot is switched on or off the system must detect this and if required start (or stop) displaying appropriate data. Secondly if a robot is physically removed from the system and unable to be tracked then the data for that robot must not be displayed, this is particularly relevant for optically tracked robots.

In order to achieve these goals a presence variable was added to the PositionObject indicating wether it is sucessfully being tracked, and the player render objects now check their current ocnnection state during their update cycle and attempt to connect to the player server if currently disconnected. In order for the player connections to function correctly a small patch is needed for the player 1.x series to add timeouts to some socket methods. Player 2.x will support this directly.

Currently Implemented Player Interface Visualisations

As can been seen in the screen shots the AR system provides an effective means of visualising the geometric data sets, for example in figure 3(e) it is immediately obvious that the sonar sensor (blue) is unable to correctly measure the distance to the box, whereas the IR sensor (purple) correctly measures this value.

The system has been tested with three different Player supported robots, the B21r from iRobot, the Pioneer 3dx from ActivMedia and our own in house design the Shuriken. These robots represent a rich set of capabilities, and a range of scales.

Discussion & System Performance

The HMD setup has definite advantages in the contextual relevance of data and immersion. However, users experimenting with the system found it difficult to position themselves to obtain the desired view of the robot data, partly due to the cabling restrictions and partly due to the proximity of the robot. In addition, HMD based systems are expensive and technically difficult to implement.

User convenience is also an issue as the user must actively choose when to put on the HMD and when to remove it, returning to the desktop based programming environment. This means that the HMD system will only be used when there is a serious problem the user needs to spend significant time on, reducing its effectiveness.

These drawbacks will largely be solved by developments in tracking and registration within the AR community, and the development of cheaper, lighter and more capable displays in addition to the advancements in wearable computing and wireless technology. User convenience will still be an important issue as will the user’s work flow between the desktop and AR systems.

The overhead setup can be permanently installed in a robotic development facility. This creates an intelligent space for robot development where multiple developers can work with many robots. In order to obtain the additional information about the robot’s state and intentions, the developer can glance up at the display.

The overhead AR system utilises lower cost components than the head mounted system, as such it can be used in a wider variety of applications. The setup used for this research will remain available in the robotics lab for use in undergraduate research projects and courses allowing a rich set of experiments that will help improve the system. If greater coverage is needed with the overhead AR then additional cameras and displays can be added.

A combined setup utilising the low cost overhead camera system for general purpose debugging and a HMD for difficult problems requiring greater immersion would give a good balance of the advantages of the two basic systems. The speed and accuracy of the toolkit are heavily dependant on the particular hardware setup and algorithms used. In the case of the overhead camera, registration accuracy is dominated by the camera calibration accuracy and the accuracy of the robot model. System tests showed an error of less than 1 pixel, or around 3mm at a distance of 3m. The overhead system currently runs at 6 frames per second; this is limited mainly by the capture rate of the networked based implementation, and the processing time to track the robot. The performance was acceptable for the target application. System robustness has been tested in an installation in place in our lab utilising an overhead camera and large screen display. The system has been run for more than 7 days continously without issues before being taken down for upgrades. With a patched version of player (too add timeouts on connects and reads) the system is also robust to robot connections and disconnections.

Conclusions

Robot programmers are faced with the challenging problem of understanding the robot’s view of its world, both when creating and when debugging robot software. Requirements for effective interaction under these conditions include the need to maximize the overlapping perceptual space of robots and humans. An AR system can increase this overlap by overlaying information about the robot’s sensory data on a real view of the robot and its environment. Input, output, state and algorithmic information may be visualised.

Our AR toolkit provides intuitive representations of geometric data from player interfaces for the robot developer. The use of AR for the visualisations gives the developer an immediate understanding of the robot data in context with the real world baseline allowing any limitations in the robots world view to be understood leading to much faster solutions to related software issues. Our initial implementation of these concepts has shown useful and promising results for robot software development.

1. Player/Stage. (2005, January) The player/stage project. Available:
   http:// playerstage.sourceforge.net/
2. C. Breazeal, A. Edsinger, P. Fitzpatrick, and B. Scassellati, Active vision for sociable robots” IEEE Trans. Syst., Man, Cybern. A, vol. 31, no. 5, pp. 443–453, 2001.
3. R. Shikata, T. Goto, H. Noborio, and H. Ishiguro, Wearable-based evaluation of human-robot interactions in robot path-planning, in Proc. IEEE International Conference on Robotics and Automation (ICRA 03), vol. 2, 2003, pp. 1946–1953.
   
4. P. Milgram, S. Zhai, D. Drascic, and J. J. Grodski, Applications of augmented reality for human-robot communication, in Proc. IEEE/RSJ International Conference on Intelligent Robots and System (IROS 93), vol. 3, 1993, pp. 1467–1472.
5. P. Milgram, A. Rastogi, and J. Grodski, Telerobotic control using augmented reality, in Proceedings., 4th IEEE International Workshop on Robot and Human Communication. RO-MAN’95, Tokyo, 5–7 July 1995, pp. 21–9.
6. E. Freund, M. Schluse, and J. Rossmann, State oriented modeling as enabling technology for projective virtual reality, in Proc. IEEE/RSJ International Conference on Intelligent Robots and System (IROS 01), vol. 4, 2001, pp. 1842–1847.
7. V. Raghavan, J. Molineros, and R. Sharma, Interactive evaluation of assembly sequences using augmented reality, Robotics and Automation, IEEE Transactions on, vol. 15, no. 3, pp. 435–449, 1999.
8. T. Pettersen, J. Pretlove, C. Skourup, T. Engedal, and T. Lokstad, Augmented reality for programming industrial robots, in Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 7–10 Oct 2003, pp. 319–20.
9. V. Brujic-Okretic, J.-Y. Guillemaut, L. Hitchin, M. Michielen, and G. Parker, Remote vehicle manoeuvring using augmented reality, in International Conference on Visual Information Engineering. VIE 2003., 7–9 July 2003, pp. 186–9.
10. M. Daily, Y. Cho, K. Martin, and D. Payton, World embedded interfaces for human-robot interaction, in Proc. 36th Annual Hawaii International Conference on System Sciences, 2003, pp. 125–130.
11. R. Bischoff and A. Kazi, Perspectives on augmented reality based human-robot interaction with industrial robots, in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 04), 2004, pp. 3226–3231.
12. P. Amstutz and A. Fagg, Real time visualization of robot state with mobile virtual reality, in Proc. IEEE International Conference on Robotics and Automation (ICRA 02), vol. 1, 2002, pp. 241–24