GI VR/AR 2013
Comparison of 2D and 3D GUI Widgets for Stereoscopic Multitouch Setups
urn:nbn:de:0009-6-39934
Abstract
Recent developments in the area of interactive entertainment have suggested to combine stereoscopic visualization with multi-touch displays, which has the potential to open up new vistas for natural interaction with interactive three-dimensional (3D) applications. However, the question arises how the user interfaces for system control in such 3D setups should be designed in order to provide an effective user experience. In this article we introduce 3D GUI widgets for interaction with stereoscopic touch displays. The design of the widgets was inspired to skeuomorphism and affordances in such a way that the user should be able to operate the virtual objects in the same way as their real-world equivalents. We evaluate the developed widgets and compared them with their 2D counterparts in the scope of an example application in order to analyze the usability of and user behavior with the widgets. The results reveal differences in user behavior with and without stereoscopic display during touch interaction, and show that the developed 2D as well as 3D GUI widgets can be used effectively in different applications.
Keywords: 3D Interaction , , Stereoscopic Display , Multitouch Interaction , GUI Widgets
Keywords: 3D Interaction, Stereoscopic Display, Multitouch Interaction, GUI Widgets
:
Recent advances in research and development have laid the groundwork for the combination of two engaging technologies: stereoscopic display and multitouch interaction [ VSB10, VSBH11, Fit54, Uni10 ]. While touch interaction has been found to be well-suited and intuitive for interaction with monoscopically displayed content on responsive tabletops and handhelds, introducing stereoscopic display to such surfaces raises challenges for natural interaction [ HBCR11, BSS13b, HHC09 ]. Stereoscopic display provides the affordances to display virtual objects either with negative parallax in front of the display surface, with zero parallax centered around the display, or with positive parallax behind the display [ Bou99 ]. While direct on-surface touch interaction with objects displayed at a large distance in front of or behind the surface is not possible without significant limitations [ VSB10 ], objects displayed stereoscopically near zero parallax can elicit the illusion of a registered perceptual space and motor feedback [ BSS13a ]. Thus, graphical elements (e. g., buttons or sliders) displayed close to zero parallax may afford a more natural interaction than their monoscopically displayed counterparts. However, it is not yet fully understood how users interact with such simple objects on a stereoscopic touch display. In particular, while the affordances of such widgets may be known from the real world, e. g., that a slider may be moved by pushing it with a finger, many of these mental models for interactions with widgets have been abstracted for use in traditional monoscopically displayed Desktop environments and for use with touch-enabled handhelds. This multitude of realizations of simple mental models results in the question how users behave when graphical widgets are displayed stereoscopically in 3D close to a touch-enabled surface [ ACJH13 ]. Moreover, the question arises how 3D widgets should be designed to provide intuitive interaction when only 2D touches can be detected. In previous work, we introduced 3D widgets in a graphical user interface (GUI) with well-known mental models that can be used on touch-enabled stereoscopic displays [ ZBSL13 ].
Figure 1. This table lists considered widgets with well-known mental models. Available features are marked with (+), whereas unavailable features for respective widgets are marked with (-). The symbol (∅) indicates that features of that category are not supported.
In this article we compare the proposed 3D widgets with their 2D counterparts for the interaction in stereoscopic multi-touch setups.
The remainder of this article is structured as follows. Section 2 summarizes the design process of the 3D GUI widgets. Section 3 describes the application and hardware setup in which we integrated the 3D GUI widgets. Section 4 explains the user study and discusses the results. Section 5 concludes the paper and gives an overview about future work.
In this section, we summarize the design process of the 3D GUI widgets. For the design of 3D GUI widgets for stereoscopic displays we first analyzed which 2D widgets are typically used in current operating systems in Desktop environments and on handhelds from the vendors Apple, Google, Microsoft and also the Linux surface Gnome [ CSH92 ]. For each widget, we identified whether the widget is skeuomorph, i. e., if physical ornaments or designs on the widget resemble another material, technique or object in the real world. Moreover, we analyzed the design of the widgets by comparing them to their counterparts in different operating systems. All considered widgets have a similar look and feel due to the need for external consistency [ MBH09 ]. Finally, we categorized the widgets according to their primary purpose. We identified four different types of widgets (see Figure 1):
-
Action Widgets trigger an immediate action, when the user clicks on them, e. g., by touching with a finger. Usually, a label or an icon symbolizes the behavior that the user can expect.
-
Choice Widgets allow either single or multiple choices. In most of the cases the options must be pre-defined. The only widget that allows users to add new options is the combo box. The appearances of choice widgets vary, in particular, on mobile platforms.
-
Status Widgets display their current status inherently in their design. They can be used to change the status of a software, e. g., enable/disable 24-hour time. Mobile platform mainly use toggle buttons to perform status changes. Traditional operating systems prefer check boxes.
-
Data Widgets allow manipulation of any kind of value. The slider, the control knob and the stepper all belong to this kind of category. These three widgets can further be divided depending on whether the value is changed continuously or discretely and whether the value is limited or infinite. The control knob, for example, is the most generalizable of all widgets; it supports all types of value changing.
Based on the classification, we realized at least one representative of each of the categories for use on touch-enabled stereoscopic displays. Therefore, we created a 3D model from a corresponding real-world object, e. g. the slider of an audio mixer console. Figure 2 shows the 3D widgets that we have designed. The sliders and the control knobs (upper right corner) are examples for data widgets . The two switches (lower left corner) represent the status widgets . Examples of the choice widgets (displayed next of the switches) allow single and multiple choice and are here shown with their two possible states. Finally, the two action widgets allow to initiate immediate actions.
As illustrated in Figure 2 the skeuomorph nature of the corresponding real-world objects was maintained for their 3D counterparts. In our previous work [ ZBSL13 ], we hypothesized that users will interact differently with the 3D GUI widgets on stereoscopic touch displays than with similar objects in the real world, and may be influenced by known interactions in Desktop- or touch environments. We found that stereoscopic display and support of head tracking resulted in different user behavior, and changed how users interpret interaction affordances.
In order to evaluate interactions with the 3D GUI widgets in a real-world application, we integrated the widgets in a visualization environment for vehicle configurations in cooperation with T-Systems Multimedia Solutions GmbH. As illustrated in Figure 3 the prototype runs on a responsive touch-enabled stereoscopic display (cf. [ Fit54 ]).
Figure 4. Screenshot of the prototype with 2D and 3D GUI widgets respectively displayed on the right viewport of the screen: (a) 2D widgets and (b) 3D widgets. Note that the images were slightly brightened for publication.
The 62 cm x 112 cm multi-touch enabled active stereoscopic tabletop system uses rear diffuse illumination [ SHB10 ] for the detection of touch points. Therefore, six high-power infrared (IR) LEDs illuminate the screen from behind. When an object, such as a finger or palm, comes in contact with the diffuse surface it reflects the IR light, which is then sensed by a camera. The setup uses a PointGrey Dragonfly2 camera with a resolution of 1024 x 768 pixels and a wide-angle lens with a matching IR band-pass filter at 30 frames per second. We use a modified version of the NUI Group′s CCV software for detection of touch gestures [ NUI14 ] with a Mac Mini server. Our setup uses a matte diffusing screen with a gain of 1.6 for the stereoscopic back projection. For stereoscopic display on the back projection screen we use an Optoma GT720 projector with a wide-angle lens and a resolution of 1280 x 720 pixels. The beamer supports an active DLP-based shutter at 60Hz per eye. For view-dependent rendering we attached wireless markers to the shutter glasses and tracked them with a WorldViz PPT X4 optical tracking system.
The vehicle visualization and configurator application is shown in Figure 4 and was implemented using the game engine Unity3D [ Uni14 ]. Unity3D provides a simple development environment for virtual scenes, animations and interactions. In order to synchronize virtual camera objects with the head movements of a user, we integrated the MiddleVR for Unity software framework [ Im14 ] , ensuring a correct perspective from the user′s point of view.
The application for vehicle configurations consisted of the registered view of the virtual “inside" of the wooden tabletop box (see Figure 3), in which virtual cars could be visualized.
The GUI widgets are displayed on the right of the virtual view with a base at zero parallax as illustrated in Figure 3. The widgets were labeled for users to change the visual appearance of the currently displayed vehicle. The vehicle was positioned on a large interactive plate (i. e., a control knob widget) in the center. In addition to the 3D stereoscopic version of the GUI widgets, we also developed 2D counterparts. This was done by scaling the 3D geometry of the 3D GUI widgets in depth direction to a 2D plane. Furthermore, small adjustments, e. g., adding color to buttons, were required to indicate different states of the widgets. With these widgets, users could turn on blinkers or headlamps, or change the height and orientation of the vehicle. As illustrated in Figure 4, this procedure results in a similar visual appearance of 2D and 3D GUI widgets. However, no stereoscopic or motion cues were provided for the 2D widgets, but on the other hand, no visuo-haptic conflicts arise when users interact with the 2D widgets on a touch surface. However, the question arises, which visual appearance, i. e., 2D vs. 3D, would be more effective.
In order to compare 2D with 3D GUI widgets, we performed a usability study. In this study we compare the 2D GUI widgets with their 3D counterparts in terms of usability and user behavior in conditions with or without stereoscopic display. From the results of [ ZBSL13 ] , we know already that head-tracking is beneficial for the setup described above and therefore we applied tracking of the head and view-dependent rendering in all conditions.
Figure 5. Mean scores from the AttracDiff questionnaire (higher is better). The vertical bars show the standard error.
9 male and 4 female subjects (ages 19-31, M=24.7, heights 163-190 cm, M=178.7 cm) participated in the user study. All subjects were students of the Department of Human-Computer-Media and obtained classed credit for participating in the experiment. All subjects had normal or corrected to normal vision. 6 subjects wore glasses and 1 subjects wore contact lenses during the user study. None of the other subjects reported known eye disorders, such as color weaknesses, amblyopia or stereopsis disruptions. We measured the interpupillary distance (IPD) of each subject before the experiment, which revealed IPDs between 6.4 cm and 7.2 cm (M=6.61 cm). 12 subjects reported experience with stereoscopic 3D displays, 3 reported experience with stereoscopic touch screens, and 5 had previously participated in a study involving touch surfaces. Subjects were allowed to take breaks at any time during the user study in order to minimize effects of exhaustion or lack of concentration. The total time per subject including questionnaires, instructions, conditions, breaks, and debriefing was about 30 minutes.
We used a 2 x 2 within-subject design. In particular, we compared the two conditions: 2D and 3D GUI widgets. The other independent variable was display modality, i. e., stereoscopic vs. monoscopic display. We randomized the order of all conditions between subjects. All subjects were informed about the widget panel on the right side and that the platform on which the vehicles rested was also a touchable and interactable area (e. g., a control knob widget).
Figure 6. Mean scores of the different components of the usability questionnaire (higher is better). The vertical bars show the standard error.
At the beginning of the trials, subjects were positioned in front of the tabletop surface for each condition (see Figure 3). Then they performed tasks given by the examiner and were asked to share their thoughts using the “think aloud" protocol [ Jø89 ]. The tasks varied in complexity, e. g., rotating the vehicle or turning on a single light could be solved straight-forward. In contrary, tasks like lighting and positioning the favorite vehicle to the user′s pleasing, required the subjects to make use of multiple widgets. Additionally the subjects were given the opportunity to explore the application on their own. We captured the subjects with a webcam during these phases.
After each condition the subjects were asked to complete the AttracDiff [ HBK03 ] and a general usability questionnaire, in which we asked subjects to judge the technique according to the criteria learnability, efficiency, memorability, errors and satisfaction on 5-point Likert scales. AttracDiff is a questionnaire used to analyze the overall attractiveness of an interactive product. The questionnaire splits attractiveness (ATT) into pragmatic and hedonic qualities. The pragmatic quality (PQ) describes the estimated ability of a product to achieve action goals by providing useful and usable features. The hedonic quality (HQ) is composed of the HQS and the HQI. HQS (hedonic quality of stimulation) describes the product′s ability to satisfy one′s need for knowledge and skill improvement by providing creative, novel or challenging features. HQI (hedonic quality of identity) describes the product′s ability to communicate self-providing messages to relevant others with connecting and professional features.
In this section we summarize the results from the user study. Results were normally distributed according to a Shapiro-Wilk test at the 5 % level. We analyzed these results with a repeated measure ANOVA and Tukey multiple comparisons at the 5 % significance level (with Bonferonni correction). Degrees of freedom were corrected using Greenhouse-Geisser estimates of sphericity when Mauchly′s test indicated that the assumption of sphericity had been violated.
The results for the AttracDiff questionnaire are illustrated in Figure 5. We found no significant main effects of the conditions on PQ (F(3,36)=.909, p=.45, η2 p=0.70), HQ (F(3,36)=1.635, p=.20, η2 p=.120), HQI (F(3,36)=1.500, p=.23, η2 p=.111), HQS (F(3,36), p=.24, η2 p=.108), or the overall ATT (F(3,36), p=1.915, η2 p=.138).
However, we observed a significant difference in overall ATT between monoscopic and stereoscopic display with 3D widgets (t(12)=-2.390, p=.03). Moreover, we observed a trend for a difference in HQI between monoscopic and stereoscopic display with 3D widgets (t(12)=-1.802, p=.10), as well as a trend for a difference in HQ between monoscopic and stereoscopic display with 3D widgets (t(12)=-1.938, p=.08).
We found no significant main effects of the conditions on learnability (F(3,36)=1.565, p=.22, η2p=.115), efficiency (F(3,36)=.559, p=.65, η2p=.045), memorability (F(3,36)=2.327, p=.09, η2p=.162), satisfaction (F(3,36)=.806, p=.50, η2p=.063), errors (F(3,36)=1.000, p=.40, η2p=.077), or the overall usability score (F(3,36)=1.070, p=.37, η2p=.82).
However, we observed a significant difference in overall usability between 2D and 3D widgets with stereoscopic display (t(12)=-2.269, p=.04). Moreover, we observed a trend for a difference in learnability between monoscopic and stereoscopic display with 2D widgets (t(12)=-1.897, p=.08), as well as a trend for a difference in memorability between 2D and 3D widgets with stereoscopic display (t(12)=-1.897, p=.08).
We observed that all subjects immediately understood the functionality of the 2D as well as 3D GUI widgets and could quickly solve the given tasks. In line with our previous findings [ ZBSL13 ], when users tried to “touch" the 3D GUI widgets, they often adapted their actions to the affordances provided by the widget. For instance, when they changed the platform height (small slider, see Figure 2) some users used the pincher grip to perform the task and all subjects touched the switch at its lifted part, although we did not distinguish between touch positions on the surface. Furthermore, we observed that most subjects tried to rotate the platform widget in the center on which the vehicle rested by using multiple fingers or even both hands.
Figure 7. Observed differences in touch behavior for the task to rotate the lowered platform. The two circles (red and green) indicate where the users touched the surface.
Similar to [ ZBSL13 ] we observed a tendency that users behaved differently with or without stereoscopic display. In particular, for the task to rotate the platform widget if the vehicle was lowered into the box, i. e., away from the interactive surface (see Figure 7), we observed the following general strategies:
-
Without stereoscopic display, the majority of the subjects touched towards the green circle displayed in Figure 7, indicating the projected on-screen area of the lowered platform. The remaining subjects used the corresponding widgets on the right.
-
With stereoscopic display, many subjects touched towards the red circle displayed in Figure 7, indicating the on-screen area after orthogonal projection of the lowered platform towards the surface. The remaining subjects refrained from touching the platform, and used the corresponding widgets displayed on the right.
Our results show a difference of overall attractiveness between monoscopic and stereoscopic display with 3D GUI widgets, which may be explained by a difference in hedonic quality and, in particular, the hedonic quality of identity (cf. Section 4.2). This difference suggests that stereoscopic display resulted in a more positive subjective experience than monoscopic display in the presence of 3D GUI widgets. In contrast, we observed no such difference between monoscopic and stereoscopic display for 2D GUI widgets. The overall quite high attractiveness scores suggest that the attractiveness was judged as considerably good for both 2D as well as 3D GUI widgets. Hence, if the application makes use of 3D GUI widgets we suggest to use stereoscopic display to increase the perceived attractiveness.
Our results show a difference in usability between 2D and 3D GUI widgets with stereoscopic display, which may be explained by the trend we observed for judged memorability in these conditions. Moreover, our results suggest a difference between monoscopic and stereoscopic display for 2D widgets. The scores for the usability metrics were all quite high, similar to what we observed for attractiveness, which suggest that the usability of the 2D and 3D GUI widgets is sufficiently high over different display environments and is not heavily impacted by monoscopic or stereoscopic display. However, for tabletop setups with stereoscopic display we suggest to use 3D widgets to increase the perceived usability.
The video data indicated that the 2D as well as 3D GUI widgets were all easy to understand and use, and user behavior suggests that the physical affordances of the 3D GUI widgets were usually perceived as dominating (e. g., all subjects touched towards the lifted part of a switch widget). The recordings also revealed differences between touch behavior with and without stereoscopic display. In particular, we observed that subjects touched different areas if virtual objects were displayed detached from the interactive surface. The results suggest different mental models used to resolve the conflicts that arise when touches are restricted to a 2D surface, but objects are displayed stereoscopically at positive parallax.
In this article we introduced and investigated the use of different 2D and 3D GUI widgets for stereoscopic multitouch displays. We analyzed 2D widgets of current operating systems and identified four categories of widgets ( action, choice, status, and data), which we used to design a set of 3D GUI widgets with strong mental models of real-world interactions. In the design process we maintained the skeuomorph nature of the corresponding real-world objects for their 3D GUI counterparts. From the 3D widgets we derived corresponding 2D widgets with very similar visual appearance and behavior.
Since the 2D and 3D GUI widgets have different affordances these may cause different user behavior, which we evaluated in a user study. Therefore, we implemented the widgets in a prototype based on a vehicle visualization application. We designed our experimental setup based on a stereoscopic tabletop such that users could use direct interaction to manipulate the 2D and 3D GUI widgets with (multi-)touch input. The setup allowed us to compare user behavior, attractiveness, and usability of these widgets.
The results of our user study show that stereoscopic display is important for 3D GUI widgets, for which users judged monoscopic display as less attractive, which may be explained by the hedonic qualities of the application. Moreover, our results show that the usability was judged as higher with 3D than 2D widgets with the stereoscopic tabletop setup, whereas we found no difference between the widgets with monoscopic display. In general, our results show that both the developed 2D and 3D GUI widgets for stereoscopic touch displays are easy and effective to use.
Furthermore, we observed an effect of the 3D nature of the widgets on user behavior if stereoscopic display was activated, which differed from behavior in case of monoscopic display. In particular, users adapted their actions to the perceived affordances of the widgets, which should be evaluated in more detail in future work. While our setup did not support 3D tracking of the user′s hands, this may be accomplished with Leap Motion [1] sensors or the 3Gear SDK [2] in future work. We believe that such movement trails during interaction will reveal differences in interaction between the different types of widgets. Moreover, we aim to investigate different user groups and mental models to explain the different interaction strategies between users.
This work was partly funded by the German Research Foundation (DFG) and the T-Systems Multimedia Solutions GmbH.
[ACJH13] Mockup Builder: 3D modeling on and above the surface Computers & Graphics, 2013 3 165—178 DOI 10.1016/j.cag.2012.12.005, 0097-8493
[Bou99] 1999 paulbourke.net/stereographics/stereorender, last visited September 9th, 2014. Calculating Stereo Pairs
[BSS13a] To Touch or not to Touch? Comparing 2D Touch and 3D Mid-Air Interaction on Stereoscopic Tabletop Surfaces SUI '13 Proceedings of the 1st symposium on Spatial user interaction, ACM Press New York, NY, USA 2013 pp. 9—16 DOI 10.1145/2491367.2491369, 978-1-4503-2141-9
[BSS13b] Touching the Void Revisited: Analyses of Touch Behavior On and Above Tabletop Surfaces Human-Computer Interaction - INTERACT 2013, Springer Berlin/Heidelberg 2013 Lecture Notes in Computer Science Vol. 8117, pp. 278—296 DOI 10.1007/978-3-642-40483-2_19, 978-3-642-40482-5
[CSH92] I3D '92 Proceedings of the 1992 symposium on Interactive 3D graphics, ACM New York, NY, USA 1992 pp. 183—188 DOI 10.1145/147156.147199, 0-89791-467-8.
[Fit54] The Information Capacity of the Human Motor System in Controlling the Amplitude of Movement Journal of Experimental Psychology, 1954 6 381—391 DOI 10.1145/2331714.2331737, 978-1-4503-1243-1
[HBCR11] Toucheo: multitouch and stereo combined in a seamless workspace UIST '11 Proceedings of the 24th annual ACM symposium on User interface software and technology, 2011 ACM New York, NY, USA pp. 587—592 DOI 10.1145/2047196.2047273, 978-1-4503-0716-1
[HBK03] AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität [AttracDiff: A questionnaire to measure perceived hedonic and pragmatic quality] Proc. of Mensch & Computer, Gerd Szwillus Jürgen Ziegler (Eds.) Vieweg+Teubner Verlag Wiesbaden 2003 pp. 187—196 DOI 10.1007/978-3-322-80058-9 19, 978-3-519-00441-7
[HHC09] Exploring tangible and direct touch interfaces for manipulating 2D and 3D information on a digital table ITS '09 Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, ACM, New York, NY, USA 2009 pp. 77—84 DOI 10.1145/1731903.1731921, 978-1-60558-733-2
[Im14] 2014 www.imin-vr.com/middlevr-for-unity, Last visited July 22nd, 2013. MiddleVR for Unity
[Jø89] Using the "thinking-aloud" method in system development Proceedings of the third international conference on human-computer interaction on Designing and using human-computer interfaces and knowledge based systems, Gavriel Salvendy Michael J. Smith (Eds.) Elsevier Science Inc., New York, NY, USA 1989 pp. 743—750 0-444-88078-X
[MBH09] Medieninformatik: Eine Einführung (in German) Pearson Studium München 2009 978-3-8273-7353-3
[NUI14] 2014 ccv.nuigroup.com, Last visited July 22nd, 2013. CCV - Multi-touch technologies
[SHB10] Collaborative Augmented Reality Tabletops - Horizontal Interactive Displays, Christian Müller-Tomfelde (Ed.) Springer London 2001 Human-ComputerInteraction 27—49 DOI 10.1007/978-1-84996-113-4_2, 978-1-84996-112-7
[Uni10] 2010 imuts.uni-muenster.de, Last visited July 22nd, 2013. iMUTS - Interscopic Multi-Touch-Surfaces
[Uni14] 2014 unity3d.com, Last visited July 22nd, 2013. Unity3D Game Development Software
[VSB10] Touching floating objects in projection-based virtual reality environments EGVE - JVRC'10 Proceedings of the 16th Eurographics conference on Virtual Environments & Second Joint Virtual Reality, 2010 pp. 17—24 DOI 10.2312/EGVE/JVRC10/017-024, 978-3-905674-30-9
[VSBH11] 2D touching of 3D stereoscopic objects CHI '11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM New York, NY, USA 2011 pp. 1353—1362 DOI 10.1145/1978942.1979142, 978-1-4503-0228-9
[ZBSL13] Design and Evaluation of 3D GUI Widgets for Stereoscopic Touch-Displays Proceedings of the GI-Workshop VR/AR, 2013 pp. 37—48.
Fulltext ¶
- Volltext als PDF ( Size 4.7 MB )
License ¶
Any party may pass on this Work by electronic means and make it available for download under the terms and conditions of the Digital Peer Publishing License. The text of the license may be accessed and retrieved at http://www.dipp.nrw.de/lizenzen/dppl/dppl/DPPL_v2_en_06-2004.html.
Recommended citation ¶
David Zilch, Gerd Bruder, and Frank Steinicke, Comparison of 2D and 3D GUI Widgets for Stereoscopic Multitouch Setups. Journal of Virtual Reality and Broadcasting, 11(2014), no. 7. (urn:nbn:de:0009-6-39934)
Please provide the exact URL and date of your last visit when citing this article.