Webcam: Answering a Multiple Choice question by sliding the correct answer to the corner. 

Context in source publication

Context 1

... VM prototype game was created to compare the experience with a traditional mouse interface and a physical interface that uses the webcam. The game can be played with either of the input devices. A framework using a combination of Adobe Flash and C++ was developed to be able to quickly create games that include animations, sound and vector graphics, while at the same time maintaining the op- portunity to use advanced image analysis. The game was developed in cooperation with the Bibliotheek van Almere, the main library of the city Almere, Netherlands. Beforehand a list of important aspects and re- quirements of the game was put together. These require- ments originated from meetings with the library, existing game heuristics from literature and personal gaming experience. In the next section (5.1), the game itself is described, as well as some design choices. In section 5.2 the technical implementation of the game is discussed. Finally, some library specific aspects are discussed in section 5.3. We have chosen to create a book quiz that asks questions about popular books for children. Children are famil- iar with quizzes and the goal of the game can be easily understood: giving the correct answers to the questions. Questions were related to well known books like Harry Potter, Pippi Langkous, De Kameleon, and (Dutch) book authors like Paul Biegel and Carry Slee. The game was developed for boys and girls from the age of 7 to 11 and the questions were adjusted to that age group. There are two types of questions, Multiple Choice (MC) questions and Sort questions . The game consists of 12 MC questions and 3 sort questions. Most of the questions were created by the library of Almere. The number of MC questions is greater than the number of sort questions, because they are faster to answer. The questions were mixed in ad- vance to give some variation. With the MC questions, the player has to pick one of the four possible answers. An answer is chosen by sliding it to the side of the screen, as shown in figure 2 (webcam input). In figure 3 a screenshot is shown of the version that uses mouse input. The Sort questions ask the player to match 4 images and words. The images have to be dragged to the designated drop zone be- neath the corresponding words. The images originate from books. An example can be seen in figure 4. The webcam version requires the player to move across the screen. Often ducking or crawling is needed to give an answer. The player stands in front of the webcam and sees himself on the projected image on the screen. The game graphics are added to the webcam image. The mouse version works by just clicking or dragging the objects with the mouse. The game is aimed to take about 5 to 10 minutes to play. That should be enough to give the child a good idea of what the game is like and at the same time will not put too much stress on the child. For each question a bar is displayed in the top right corner to show the remaining time of the current question. When the time is up, the player can still give the right answer but receives only half of the points he would normally get. For the MC questions the player has two tries to get it right. Giving the correct answer in the first attempt within the time limit gives 100 points, a correct answer the second try gives 75 points. When two wrong answers are given the correct answer is shown and the game continues with the next round. The sort questions give 100 points. When an image matches the word it is hovering at that moment, the image locks to the word and can not be moved anymore. For example, the bottom right image in figure 4 is locked to the white coloured dropping zone. The progress of the game is shown in the top of the screen. The current round and total number of rounds gives an idea of how long the game will be. The current score is given to show the progress of the player and to increase the motivation to get a higher score. For most questions a small sound sample is played that relates to the current question. Examples are the sound of a howling wolf with horror questions and the Harry Potter tune with a question related to Harry Potter. Giving the wrong or correct answer plays respectively a buzzer or trumpet sound to give clear feedback of the players actions. When all questions are answered the game is finished and the final score is displayed on screen. A framework was created to allow rapid development and expansion of the current prototype. The framework consists of a front-end game client (Adobe ActionScript 3) and a server (C++) that analyses the image stream from the webcam. The analysis is done using the OpenCV computer vision library [3]. See figure 5 for a graphical representation of the application components. To minimise network delay, all applications are run on the same computer. The client takes care of the game graphics, sounds and actual gameplay. For the client the Adobe ActionScript language was chosen because it is a high level scripting language with support for many multimedia formats and is very suitable for game creation, animation and vector graphics. The latest version, ActionScript3 was used because it is faster and has better functionality for access- ing raw image and network data than previous ActionScript versions. ActionScript code is compiled into a executable Flash file (.swf) that can be run in the browser. The server has to be started separately. Additionally a simple PHP 3 script was used to log the game progress and scores to give insight in the amount of time that is needed to complete certain questions and the game as a whole. The webcam input is read by the Flash client and sent to the server for analysis. To have smooth movement feedback, it is necessary to send image data very frequently. For face detection the image data is sent every 50ms. For the optical flow calculation the data is sent every 30ms. On a computer with a 2.8Ghz Pentium 4 processor, the average time needed for face detection is 20ms. The optical flow takes about 8ms to calculate. On top of this, there is an overhead of 10-15ms for transferring the content between the applications. Each frame is sent in a package that contains the following data (total of 77609 ...