Music Haptic : Musical Harmony Notions for All with A Force Feedback Mouse And A Spatial Representation

Bertrand Tornil and Nadine Baptiste-Jessel

Institut de Recherche en Informatique de Toulouse
Université Paul Sabatier - Toulouse 3
31062 Toulouse Cedex, France
bertrand.tornil@gmail.com

Abstract

This paper describes Music Haptic, an application which enables users (especially blind users) to train of musical harmony notions due to a spatial representation and a force feedback mouse. This mouse enables the progressive rebuilding of the mental image of a graphical document through knowing the positions of its elements. We present technical context of our prototype and the possible alternatives. Then, we describe our prototype and discuss directions for future research.

1 Introduction

With their capabilities, computers have often been used in musical education. Several approach exists in this research field : graphical visualization [8] [4], artificial intelligence [9], computer musical analysis and composition [2].However, these approaches often forget users with visual disabilities. In a traditional situation, a blind user or a low vision user uses a keyboard to interact with a computer. The computer answers him via a voice synthesis and/or a braille display. This method is completely adapted to the access to textual documents. With graphical information, a textual description may be proposed. However, the problem persists on the one hand because these descriptions are not always present, and on the other hand because textual descriptions can quickly prove to be long and tiresome to consult. Accessibility to the music notation for blind users is based on linear transcription of musical data into braille notation [1] [17]. It implies knowledge in musical and harmonical notions, and the comprehension of the Braille. The use of force feedback devices was thus studied in accessibility research [3] [5] [13] [14] [19]. Indeed, these peripherals authorize a more direct interaction based on the sensory capacities. Music Haptic uses a force feedback mouse in order to present some harmonic informations. First of all, thus, we present how we use the force feedback modality : we propose the concept of "relative localization". Then we present the Steedman musical representation which enables us to use the relative localization for harmony training. In the next part, we describe the architecture which we retained for our prototype. Then, we present the features of Music Haptic at its stage of development. Finally we will finish by presenting the research orientations which we consider for the future.

2 Haptic interaction

2.1 The force feedback mouse

We use the "Wingman Force Feedback Mouse" (Figure 1) created by Immersion Corporation [10] and marketed by Logitech. The handling surface of this mouse is small : 1.9cm by 2.5cm. Forces can reach up 1N. Originally, Wingman mouse was conceived for video-games. However, some research in accessibility [20] [7] have used this device.

Figure 1: Wingman force feedback mouse

Immersion Corporation proposes a plug-in for Internet Explorer. The plug-in give the possibility of controlling the mouse via javascript programming.

2.2 Haptic perception with a force feedback mouse

Tactilo-kinesthetic or "haptic" system [15] consists of :

cutaneous perception : it allows the perception of the temperature, the pressure or the pain. The sensory receptors are located under the skin.
The kinesthetic perception: it makes it possible to feel the position and the movements of the body. For instance, it enables us to know the weight, the shape and the position of an object which we are handling. It is relayed by sensory receptors located in the muscles and the tendons[3]

The handling of a force feedback pointing device, as a mouse, is based on the kinesthetic perception of the arm and the hand. The cutaneous perception is not stimulated. Thus, in our approach, it is not question of feeling a texture, but of perceiving the places where our hand is located during the handling of the mouse.

2.3 Relative localization

Due to the sensory memory associated with our kinesthetic perception, we can mentally represent the positions of the objects. It is what we call the "relative localization". For instance, in the figure 2,a blind user will recognize the relative disposition of some french counties on a map (top example); or, on the bottom example, he will know where the left arm of the skeleton is.

Figure 2: Relative localization

Coupled to an audio feedback, this approach will enable blind users to rebuild a mental image of an object starting from the elements of this object. [18] We present now how to apply this approach to the music learning field.

3 Seedman representation

In order to use the same approach for music, we need a spatial representation of musical data. All of the intervals in tonal music can be represented as combinations of the fundamental intervals the octave, the perfect fifth and the major third [11] et [12]. This creates a three dimensional harmonic representation although in practice we can represent the space of notes in two dimensions by collapsing the octave dimension into a single layer (Figure 3). There is a further enhancement to this space that can be made and this was proposed by Mark Steedman [16]. We can offset the rows of the space by half units to create a space that allows any particular note to have six adjacent members arranged hexagonally (Figure 3).

Figure 3: Longuet-Higgins (left) and Steedman (right) harmony representation

We have chosen this representation because the 6 neighboring notes of an hexagon are equidistant from the original note. The Fitts law [6] give an equal movement time to access to these 6 neighborhood hexagons. Moreover, these 6 notes may be considered as the "most harmonically closed" notes: minor and major thirds, and perfect fifth. Compared to the original representation, we carried out a rotation of 60 degrees in order to preserve the intuitive correlation between pitch height of the note and direction at the screen: the movements toward the bottom of the screen will always make the sound to go up (Figure 4). The main characteristics of this representation are :

vertical axis consists of perfect fifths;
axis diagonal-right consists of major thirds; and
axis diagonal-left consists of minor thirds.

Figure 4: Steedman representation

The most important property is the following one: a chord consist of several connected hexagons which represent a specific shape, whatever the tonality. The figure 5 shows the shapes of the implemented chords.

Figure 5: Implemented chords

Due to this property, a blind user (and even a sighter user) only have to learn a chord specific shape, in order to recognize it. Compared with the traditional musical notation, this is a advantage in a training task. For instance, a diminished 7^th in E and in F\flat are shown on figure 6. Chords shapes are different, alterations are different, but it is the same chord. With the Steedman representation, the diminished 7^th chord consists of four hexagons, whatever the scale.

Figure 6: Different shapes in traditional notation; the same chord in Steedman representation

Moreover, if two chords, in traditional musical notation, seems to have the same "shape", their nature may be different. For instance, on the figure 7, the two chords are similar. But, in fact, one is a in major mode, while the other is in minor mode. The Steedman representation distinguishes a minor triad chord from a major triad chord, whatever the scale, due to the orientation of a isosceles triangle : left-oriented for a major triad and right oriented for a minor triad.

Figure 7: Same shape in traditional notation, but F Major on the left and E Minor on the right

4 Music Haptic : technical aspects

We have chosen a WEB applications context. We have based our prototype on a client-server architecture. On the server side, we use Apache Web server. On the client side, we use Microsoft Internet Explorer. Nowadays, it is the only browser which support the Immersion Web Plug-in. The figure 8 shows how our system works.

Figure 8: overview of the system

5 Music Haptic : features of our prototype

In our prototype, we use the Steedman representation in order to present the different chords to the user. Two mode are available : Exploration mode and chord mode. In exploration mode, the user can freely explore the checkerwork of hexagons. Each note is "displayed" with a force feedback (Figure 9) and the note is played in the same time.

Figure 9: one note and its force feedback

Keyboard enable to switch in the different chord mode (major and minor triad, augmented triad, and so on ; see figure 5). In this configuration, the user only explores the notes of this chord. This mode enables a blind user to learn the specific shape of a chord. For example a right oriented isosceles triangle is a major triad chord (Figure 10). The last feature enable a sighted user to ask for the traditional musical notation of the chord. Information sharing between sight and blind user is easier thank to this feature.

Figure 10: Major triad : force feedback hexagonal display and traditional notation dynamically generated

6 Outlooks and conclusion

We currently set up a test protocol to evaluate our approach near different public : blind people, children and autists. The first blind user feedbacks we got was encouraging. We will also continue the development of our prototype dedicated to the training of the harmony. In particular we will propose the possibility of recording its own music; the creation of a chord could be done for example via a gesture recognition with the mouse.

References

[1]: Braille Music Editor. web site at : http://www.dodiesis.com/.
[2]: Brandao, P. . W. Computers in music education. In Proceedings of the AISB'99 Symposium on Musical Creativity, (1999), AISB.
[3]: Burdea, G. Force and touch feedback for virtual reality. Wiley-Interscience; 1 edition (August 3, 1996), 1996.
[4]: Castaing, J. Musicdraw : to make music learning attractive. In Forth MUSICNETWORk Open Workshop : http://www.interactivemusicnetwork.org/ (Barcelona, Spain, September 2004), Universitat Pompeu Fabra.
[5]: Colwell, C. Haptic virtual reality for blind computer users. In Proceedings of ASSETS'98, Third International Conference on Assistive Technologies (1998), pp. 92-99.
[6]: Fitts, P. The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47 (1954), 381-391.
[7]: Gardner, J. A., and Bulatov, V. Smart figures, svg, and accessible web graphics. In Proceedings of Technology And Persons With Disabilities Conference 2001, Los Angeles (2001).
[8]: Holland, S. Direct manipulation tools for novices based on new cognitive theories of harmony. In Proceedings of 1987 International Computer Music Conference (1987), pp. 182 -189.
[9]: Holland, S. Artificial intelligence in music education: a critical review. In Readings in Music and Aritficial Intelligence, Contemporary Music Studies, E. Miranda, Ed., vol. 20. Harwood Academic Publishers, The Netherlands, 1999.
[10]: Immersion Corporation. web site at : http://www.immersion.com, 2005.
[11]: Longuet-Higgins, H. Letter to a musical friend. Music Review (August 1962), 244-248.
[12]: Longuet-Higgins, H. Second letter to a musical friend. Music Review (Nov. 1962).
[13]: Offen, D., and Thomlinson, B. Good vibrations: Using a tactile mouse to convey page layout information to visually impaired computer users. In Proceedings of CSUN'S Sixteenth Annual International Conference :"Technology and Persons with Disabilities", Los Angeles (2001).
[14]: Ramstein, C., Martial, O., Dufresne, A., Carignan, M., Chassé, P., and P., M. Touching and Hearing GUIs: Design issues for the PC-Access System. In ASSET'96, ACM/SIGCAPH. In 2nd Annual ACM Conference on Assistive Technologies, Vancouver, BC, Canada (1996), pp. 2-9.
[15]: Revesz, G. Psychology and art of the blind. New York: Longmans, 1950.
[16]: Steedman, M. The formal description of musical perception. Unpublished phd, University of Edinburgh, University of Edinburgh, 1972.
[17]: Toccata. Optek systems. web site at : http://members.optusnet.com.au/ terryk/toccata.htm.
[18]: Tornil, B., and Baptiste-Jessel, N. Use of force feedback pointing devices for blind users. In User Interfaces for All (2004), pp. 479-486.
[19]: Wies, E. F., and al. Web-based touch display for accessible science education. In Haptic Human-Computer Interaction (2001), pp. 52-60.
[20]: Yu, W., Ramloll, R., and Brewster, S. Haptic graphs for blind computer users. Lecture Notes in Computer Science 2058 (2001), 41-49.