The computer domain has largely been dominated by systems with a relatively large display, capable of showing (fast moving) high-resolution images in full color and spatial sound output of high-quality. On the input side a wide range of game controllers and pointing devices are available. Most of them are very similar in function and handling. The keyboard is still the standard way to enter text and the mouse plays a major role in interacting with different parts displayed on the monitor.

More recently, camera based approaches have been introduced as generic controllers for games. The Sony EyeToy, e.g., allows interacting in the physical space with games on a Playstation 2 [ Son05 ].

Input Devices Since the invention of the computer mouse, a great varity of input devices has been developed. Most input and interaction devices are not as general as the mouse and hence, they are of great value in specific domains.

Recently, research and industry focus on interactions by directly observing movements or gestures of users and translating them into interaction events. However, this kind of processing needs an augmentation of the users′ environment or the users themselves. Examples are presented in [ VRC99 ],[ MAS04 ] and [ Rek01 ].

Our approach differs from this in that we augment existing objects or appliances that people have already got used to and know their affordances. We then aim for an easily understandable way of translating interaction with these devices into actions and events interpretable by applications. By matching the affordance of the object with the interaction to perform, we can be sure that users will quickly be able to start using the application in the intended way.

For enhancing existing appliances, we identified several different types of sensors that can be cheaply bought and easily integrated:

This is by no means an exhaustive list as there exist many more types of sensors (temperature sensors, microphone, or more general, sonic sensors, etc.). However, the sensors listed above prove to be interesting in our experiments for creating engaging and animating input devices. In many ways users can interact with devices like a cube ([ KSHS05 ]) or a chair ([ Coh03 ]). This can easily be detected and interpreted.

We will show in Section 3.1 how we use a subset of the mentioned sensors to capture movements of interest of a user on a augmented IKEA balance cushion ([ SHK04 ]).

Output Devices On the other end of the application, we observe the trend to use more and different types of screens and displays for output, depending on the type of data that is to be visualized.

We also do not narrow the term output devices to mere visual screens, but also include various means of output devices, e.g., for tactile or haptic output. Simple occurrences of showing a one bit state can include, e.g., LEDs or switching on and off any appropriate appliance.

Particularly interesting seems to be to use a combination of distributed large public displays placed at various points of interest in the environment and small private displays visible only to a specific user or group of users. Such forms of multi-display game environments have been suggested in [ MMES04 ].

From an architectural point of view, we treat sensors and actuators similar as will be shown in Section 2.2 .

Figure 1. Visualization of the basic architecture.

As is shown in Figure 1 , the application is strictly separated from any issues regarding sensor or actuator hardware. There is a clear interface to access the different devices. A realization for that (indicated by the cloud in Figure 1 ) will be presented later in Section 3 . An application needs to send information to output devices and receive information from input devices. Communication in these two directions is aided by two helpers, namely the Sensor Server and the Actuator Server. They are similar in structure in that they have one or several registered devices with which they communicate. The Sensor Server for example collects data from all sensors known to the component. This information can then be queried and used by the application. The Actuator Server on the other hand has a list of actuators, i.e., displays, lights, etc. The application is then granted access to those via the communication layer according to the capabilities of the respective device.

Of course, it is neither needed nor sensible that every sensor sends its data to the Sensor Server by itself. In most hardware platforms, those will be collected and sent by a central component. On the other hand, this approach allows an arbitrary number of sensors / actuators in the environment to be used without needing any sophisticated knowledge in hardware or communication protocols.

To accomplish the design goal described in the last section, we must provide an abstraction from the available hardware. Especially to enable application developers and device builders to easily integrate their products into thearchitecture. In particular, we want to ease the job for people building and integrating hardware components and for game application developers. The field of virtual reality has brought with it some platforms and architectures that use device abstraction. Open Tracker ([ RS01 ]), e.g., bases its tracking framework on XML and defines interfaces to easily support different devices. The Virtual-Reality Peripheral Network (VRPN) [ THS01 ] includes a set of servers that are designed to implement a network-transparent interface between application programs and a set of physical devices. In the following we describe how our concept can simplify the development process of a real world computer application.

To be able to get this kind of architecture, a middle layer is needed that is responsible for providing easily accessible interfaces to the application and sensor sides as well as managing the communication between them.

We draw heavily on available standards to ensure that the largest possible number of applications can be used and that the learning effort for developers is minimized. As is described in more detail in Section 3.2 where our implementation of the layer is shown, we currently favor XML as the data wrapping format as it can be easily validated against, is human readable and parsers exist for most applications and programming languages.

As protocol to access the data, one of the most widely supported formats is HTTP. Nearly all applications or languages that allow some kind of external access are capable of reading web pages. The infrastructure needed can be found nearly everywhere and it is particularly easy to create small viewers for prototyping and testing devices.

We have implemented several sensing devices which share the same basic hardware architecture. The sensors are connected to a micro-controller. The micro-controller is responsible for basic data acquisition and processing. Via RF, the data is then sent to a base unit that is connected to a computer in the network.

In one implementation we used the Smart-Its platform [ GKBS04 ] and attached a custom sensor board. The Smart-Its provide a programmable micro-controller (PIC 18F452) and several analog as well as digital inputs and outputs. Sensor data can be sampled at frequencies of several hundred Hertz (if supported by the sensor). The RF sender can send data at a maximum rate of 14400 bps (including overhead for control, etc.) enabling even those applications that rely on quick updates. This data is then transferred wirelessly using a transceiver of the type Radiometrix SPM 2-433-28. At the PC side, a similar construct is used: Another SPM module receives the data and communicates it to the PC via serial line input. From there, it can be processed by software as is described in further detail in Section 3.2 .

In a new implementation we used Particles, devices similar to Smart-Its. Particles are developed at TecO, Karlsruhe [ DKBZ05 ]. They are smaller and have more sophisticated ways of transmitting data (including acknowledgment etc.). This change has shown one of the strengths of our architecture: since input devices are separated from processing and the final application, it has been very easy to switch to the new platform. Only the receiving part of the communication server had to be adjusted. The data is no longer sent over serial input but in UDP packets.

Apart from the loose coupling to the used sensor and actuator hardware, the framework also abstracts from the actual object that is used as input device and into which we embed the sensors. We deliberately searched for objects that are known to most people, but are not yet used as input devices. In this section, two out of the many possibilities we found are presented. First, we used an IKEA balance cushion named Virrig shown in Figure 3 [ SHK04 ]. It is a flat cushion mounted on a robust hemisphere. Thus, it can be rotated and tilted in all directions. It is very flexible in use as the user can sit, stand, kneel or lie on it and it is very robust, too, as it is designed for use by children. It can be seen as a regular cushion or as a toy to practice balance.

Figure 3. The Virrig input device shown from the side.

The digital device we attached inside the hemisphere does not change the affordance or the physicality of the cushion. The user still can sit or stand on it as before, and since we use radio technology for data transmission there are no cables leaving it, so it can still be tilted and rotated like before. We show how to use the cushion in an edutainment application in Section 4.1 .

Figure 4. Opened Virrig with the integrated Smart-Its.

Inside the cushion, attached to a wooden plate, there is a Smart-Its (see Figure 4 ) with the following components:

The overall hardware architecture is depicted in Figure 5 .

Figure 5. Hardware architecture of the input device.

As another input device, we chose a small appliance that is mounted at the doors of many offices, lecture halls or assembly rooms. It shows the current state of the room or its occupant. A magnetic pin is placed on certain spots of a ferromagnetic plate to indicate whether the person working in that room is in, busy, out for lunch, etc. We enhanced such boards with magnetic switches that recognize where the button / pin is placed and put a Smart-Its in each of them that communicate this state to a central receiver. The application is briefly described in Section 4.2 .

After having described the parts connected to the input device, this section goes into details about the PC side of the system. A Smart-Its equipped with a radio receiver is attached to the PC. The Smart-Its receives sensor data from the cushion and forwards it to a program called Serial Server over RS232 serial line. The Serial Server interprets all received signals and transforms them into XML format. This data is then stored on a web server making it available for any application capable of using the HTTP protocol.

The architecture of the receiver is outlined in Figure 6 . As has already been stated, we believe that providing information via the HTTP protocol is one of the best methods to allow a very high number of different applications as well as programming languages easy access to this data. The choice of using XML as storage and wrapping format has been made in the same sense. A large number of applications and programming languages inherently support reading and writing data coded in XML structures. The fact that the data is human readable and conveys some semantics in its structure and named tags helps to implement applications using devices for which there is no internal knowledge. It also enables the specification of content structure and easy validation of incoming data. The part that generates sensor values as well as the application can therefore rely on a specific DTD or schema being followed by transmitted sensor data. This dramatically reduces the complexity of implementing both sides. In our prototypes we did not experience any significant slow-down using XML instead of, for example, using a file with comma separated values (CSV). Since, however, some system resources are needed for parsing and processing of XML data, this is not the optimal solution for devices with very low processing power. The first step toward tackling this problem is to send only that part of the data that really changed (event based mechanism). Additionally, we are planning on making the protocol XML exchangeable with any other protocol like CSV or binary formats.

To explain how to use the Flash component and how to build applications on top of it we provide a small sample application that outlines the basic concepts in more detail. After that, a more sophisticated program is described for which we plan to undertake user studies to evaluate some assumptions on the impact of physical interaction in learning applications.

Figure 6. Hardware architecture of the receiver.

The component itself requires two input parameters: the locations (URIs) of the XML configuration file and the XML variables file. The XML configuration file specifies the variables for faster parsing in the Flash application. The component reads the file so it can name and initialize variables and set an interval for the reload rate.

Tag Tag Explanation

minimal and maximal sensor refresh rate in seconds

block with available variables

names, start values and types of the variables delivered by the sensor input device

XML Variables File: This file delivers the sensor data to the application. Only the variables that have different values from the last update appear so that the component does not need to read all the variables each time. This renders the Flash application notably faster and also reduces traffic.

Tag Tag Explanation

variables whose value changed

names and values of the variables delivered by the sensor device

When the application starts, the first thing for the component is to analyze the configuration file, to set up and initialize the variables and to set the minimum and maximum interval for reading the XML variables file. Then the application starts reading the variables for the first time. From now on it reads the variables file as often as defined by the minimum interval in the configuration file. It sets the received variables onto the ′root-time-line′. That process is repeated as long as the application is running. This is also shown in the activity diagram in Figure 7 . For the rare case of sensors dynamically changing their update interval, the configuration file could be read from time to time to update the minimum and maximum rate.

Figure 7. Activity diagram of the Flash component.

After programming the component, we implemented the first test application. This application visualizes movements made by the Virrig cushion. The application shows the cushion in the center of the window and indicates tilt and rotation. All possible movements of the input device are shown on the screen. A screen dump of the application is shown in Figure 8 . The green ball in the middle represents the Virrig cushion. The arrow (black triangle in the top right in Figure 8 ) indicates the rotation sent by the sensor device. The tiny black lines around the ball show, if painted red, the activated ball sensors. The text window on the right is an output window for testing and tracing the correct functionality of the component.

Figure 8. Screen dump of the test application.

The Virrig Race Game is a game application with the sensor cushion as physical controller. It is a mixture of car race and learning program, i.e. an edutainment application. The car is controlled by the cushion. Since the expected audience will be primary school children, the whole design is aimed to be interesting for smaller kids. Depending on the complexity and type of the questions, the game is also fun and a nice learning experience for people of all ages.

The first step in integrating the cushion into the framework is to augment the Sensor Server such that it converts the sensor values it receives into an XML tree. Second, the flash component used as described in Section 3.3 . The flash application can now use the given sensor values to define its behavior. As one implementation, we designed it the following way: The user controls the car by tilting the cushion forward and backward to accelerate or brake and left or right to go in that direction, respectively (see Figure 10 ). There are stop signs on the crossings where a question is displayed (see Figure 11 ). In this case, the rotational sensor (compass) is used to implement a simple selection mechanism by turning the Virrig. The user then commits the choice by clicking, i.e. tilting forward. One could also think of realizing the this by means of hopping on the cushion as a load sensor is placed directly below the user in the inner part of the cushion.

Figure 9.  Start screen of the game application. The user can choose to control the game with a standard keyboard or with the Virrig.

Figure 10. Screen dump of the game application.

The user can continue driving after giving a (potentially wrong) answer. If the answer was wrong, the sign gets transparent until he or she has answered another question. So the memory effect is not that serious but imposes at least some pressure on the user who can now try to answer it again. If the question was answered correctly this time, the sign disappears. After all signs have disappeared (i.e. all correct answers have been given) the race has been successfully completed. It could be imagined to display a score board showing the 10 fastest users or those with the least number of wrong answers.

The framework makes it particularly easy for developers to quickly change the interpretation of the sensor values. The compass could for example be used as sensor controlling instead of tilting. It also enables exchanging the particular device that is producing the sensor values or is consuming the software input with another device, possibly with another set of sensors / actuators.

Figure 11. Screen dump of the quiz window showing a question on top and three possible answers below. The one in the center has been selected by the user.

This application visualizes the occupation of rooms in a building, e.g. if the room is used for a meeting, a lesson, or is empty (see Figure 13 ). The natural choice for controlling such an application is an input device placed at the entrance to rooms. This device is very intuitive to use since a simple version of it can already be found at many office doors. It is a board with switches that are activated by magnets (see Figure 12 ). The state of a room is set by putting a magnet on the sensitive area of the Hall-effect switch. This data is wirelessly transmitted per room to the Serial Server. The server collects the information of each room and the application get this information from a specific IP and port from the HTTP server.

As can be seen in Figure 12 , the device is also equipped with four LEDs that serve as feedback when setting the magnets but can also be controlled remotely. Thus, one could replace the magnets with simple buttons and the LEDs could show the current state of the office. This state can then be changed by, e.g., the same Flash application that shows the state of the room. It simply needs to write a specific value coupled with those LEDs and the Actuator Sensor takes over to pass the commands to the hardware component.

Figure 12. Screen dump of the Room Occupation System.

Figure 13.  Picture of the Room Occupation System user interface.