Posts tagged Multitouch
Yesterday I did the last step in my Media Informatics master endeavor: I presented my master thesis. After quite a few months of development on the Microsoft Surface, the application that resulted was a good start for a (possible) commercial product.
We started from the idea that a multitouch table device would offer more space, possibilities and opportunities to improve the collaboration and communication in the brainstorming phase of any project. This evolved very much during the development and conceiving the master thesis. It became a tool which not only supported and enhanced collaboration, but also, to name a few:
- offered clues gathered from Twitter and Flickr images,
- free drawing and sketching,
- text input using the virtual keyboard,
- audio recognition features,
- save / load of session,
- integration with a CRM, serving as repository;
- user-friendly navigation in sessions using also the CRM repository interface.
This article presents an overview of my Master thesis. It summarizes the work by presenting the contributions to the research field and also the future work based on the already created prototype. The Breiny application is meant to support the Brainstorming process by using computers in a modern way – Natural User Interface. After creating the prototype, a User Evaluation was used to prove or disprove the theoretical hypotheses.
In the Related work is presented the relevant work on which my master thesis relyies on and the systems that are the basis for comparison with the developed prototype. It considers several already available collaboration and brainstorming systems and compares their features with the proposed Framework for analysis. As a conclusion of the chapter, none of the considered systems fulfill all the requirements from the proposed Framework. (more…)
The Users for the first session were 5 males and 1 female, with age ranging from 27 to 49. For the second, there were 4 males with age ranging from 26 to 52, with significant Brainstorming sessions experience, using Mind-mapping software, whiteboard computer based systems. The room setup consisted from a large space around Surface, with chairs surrounding it. All users were in the Contributor role and they had the opportunity to switch to Session Master any time. The application was started, with no Brainstorming Ideas on the tabletop.
The Evaluation session started with the Prototype presentation. This was targeted only on the User Interfaces, explaining how and which actions can be performed with each area of interest (UserControls and Main SurfaceWindow from Implementation Chapter). The users were encouraged to experience the interface and ask questions during the presentation, in order to understand better the interaction with the prototype.
Following the presentation phase was designed as a game, in order to exercise the prototype’s interface and evaluate the system and its features. The game was built around a imaginary, newly created, travel agency. The participants should create the company’s portfolio in a Brainstorming session, composed from the four phases (scenarios) presented in Typical use cases. They were disclosed one by one, in order to not affect the brainstorming process.
The theoretical evaluation of the thesis work starts with by assessing in which measure the research questions were answered by the proposed concept. Then, the analysis will focus on the comparison of the implemented prototype’s features with the ones presented in Related work, in order to see which ones were implemented and how the prototype’s features map on the array of desired ones. Finally, this chapter will conclude with a matrix in which the implemented functional requirement are marked, giving a final overview on the capabilities of the prototype to fulfill all the requirements stated in Requirement elicitation. (more…)
The prototype’s architecture implementation
The architecture was already presented in the Prototype application architecture and now we’ll see the actual implementation. The prototype implementation tries to follow as close as possible the recommended architectural patterns and guidelines. To support the argumentation, a class diagram will be presented to give a visual orientation through the prototype implementation presentation.
The built prototype supporting Brainstorming Sessions on Microsoft Surface, further named Breiny, was developed as part of this thesis and uses all the above framework features. Thus, it is creating the basis for future work by being flexible in development yet fully packed with features.
Breiny’s class diagram
This section will present the Breiny application’s user interface and the available interactions with it, features which will be evaluated both theoretically and by the users. It is composed from a main SurfaceWindow named “Main screen” in figure below, on which all the interaction occurs. Following the requirements for extensibility, modularity and flexibility, the subsequent visible objects were separated into several user controls.
In the following subsections the main screen will be presented first along with its specific interactions, then the discussion will continue with the available item types and their particular available features.
The main window acts as an aggregator for all the interactions, providing also support for the other controls and their interactions. It offers access to the main contextual menu, to the tag recognition and visualization, offers the data context through binding against the Session singleton and, because binding is propagated to the children controls, allows subsequent binding context to all of them.
In the figure from the left, the Breiny application was started and a new Brainstorming session is created. In this specific case the Brainstorming Items Repository didn’t already contain Brainstorming Items, thus none of these appeared on the application screen. But the users may have prior access to other devices that can interface with this thesis proposed system (iPhone, iPad or Adobe Air client) as mentioned in Architecture chapter and send they initial suggestions. If so, their Brainstorming items will show on the newly created Brainstorming Session. (more…)
This article presents the system architecture and the underlying decisions that were taken to achieve the above requirements. The system requires a repository for sessions, a repository for Brainstorming items and a native Windows application for achieving the baseline of desired features. Thus, a distributed client – server architecture is considered, exemplified below.
Session repository architecture
The above schema represents the Session repository architecture. Its main purpose is to offer two main features:
- communication over network
- means for storing, retrieving and sharing the sessions
The server and the native application can be mixed together, but with limited sharing options. Because of this reason, we will use an already created system which not only that offers through its network module of API (Application Programming Interface) both means of storing and retrieving of the session data, but also allows advanced features of sharing the data in an already familiar interface.
Brainstorming items repository architecture
The architecture for the Brainstorming items repository is similar with the one for Session repository, but the most important feature it brings is the possibility to interface with other devices, not only with the tabletop.
Another reason for creating this secondary architecture is the fact that it easily enables an offline scenario, in which the repository server is not available because of the network availability issues. Nevertheless, the mobile devices can connect to this repository through creating a local wireless network without the need to be connected to the internet. Another advantage is that, because it’s small complexity, can (but it is not required to) be installed on the same tabletop. The offline scenario is not covered in this paper, but rather proposed as future work.
We already covered the proposed general usage scenario and particular use cases of a Brainstorming system. In order to evaluate their implementation, in this section the application requirements will be presented. They are split into functional and non-functional ones. The first refers to all the features needed to be implemented, while the second category refers to performance, extensibility and scalability of the system.
|FR.1||The system will allow multiple users to interact with the system simultaneously|
|FR.2||The system will allow multiple devices to interact with the Microsoft Surface simultaneously|
|FR.3||The users will be able to send ideas to the tabletop using mobile clients|
|FR.4||The users will be able to generate and input ideas in the system|
|FR.5||The Contributors will be able to input Brainstorming items directly on the tabletop|
|FR.6||The Contributors will be able to Group Brainstorming items|
|FR.7||The Contributors will be able to Edit Brainstorming items|
|FR.8||The Contributors will be able to Relate Brainstorming items|
|FR.9||The Contributors will be able to Print Screen|
|FR.10||The Contributors will be able to promote Twitter or Flickr items to Brainstorming items|
|FR.11||The Session Master will be able to login into its BSCW account|
|FR.12||The Session Master will be able to Save a Brainstorming Session|
|FR.13||The Session Master will be able to Load a Brainstorming Session|
|FR.14||The Session Master will be able to set Session Goal|
|FR.15||The Session Master will be able to start the application using his Tag|
|FR.16||The Session Master will be able to search on Flickr using a keyword|
|FR.17||The Session Master will be able to search on Twitter using a keyword|
|NR.1||The application will be designed to be extensible and maintainable|
|NR.2||The application will be highly responsive|
|NR.3||The application will be reliable|
|NR.4||The application will use responsibly hardware resources|
Now let’s move to the Architecture of the application!
Because of the specifics of the brainstorming activities, there are two sets of features that must or can be implemented: the first ones which regulate the brainstorming activity as a set of rules; the second ones add productivity to the process, enabling additional scenarios that improve the user experience. In the first phase of requirements elicitation for such a system, the Microsoft Surface was chosen as a direct, face to face collaboration. For enabling the sharing of Brainstorming session, BSCW system was chosen as it enables shared workspace paradigm, allowing easier collaboration and cooperation among users with common interests.
Detailed usages of typical brainstorming and idea generation tools were presented in Related work. What all this papers have in common are few scenarios which are very important for the idea generation process:
- Creation of ideas and inserting them into the system (e.g. Creating text items with brainstormed ideas)
- Editing (e.g. Correcting, improving and augment with graphical elements the already existing items)
- Grouping through visual clues (e.g. creating logically grouping structures with the aid of visual elements)
- Relating (e.g. creating a tree-like relation that gives a structured form to the ideas generated through the process towards a specific goal)
A typical user scenario is presented to clarify the use and features provided by the system.
Anton, Kai and Vladimir are three students developing applications in Fraunhofer FIT under the supervision of a research assistant, Greg. They set up a meeting in the User Experience lab, where the Microsoft Surface is, and set as their goal to create a new system for helping people with Alzheimer.
Because Greg is the coordinator of the brainstorming session, he uses its card to login into his BSCW system which will be later used as a repository and central sharing hub for the outcome of this session. Then he inputs the text “Alzheimer project” into the central control on the application, allowing all of them to keep focus on their goal. They start adding to the Surface the ideas that they have, as the brainstorming process is about quantity. When they feel that a small dose of inspiration is required, they use the application’s feature of searching in Twitter or Flickr for fresh ideas or hints. Since they work on the same device, the awareness of each one presence is complete, conversations start over some interesting features, raising more and more ideas for their project.
After the main part of the brainstorming is finished and all ideas are inputed on Microsoft Surface application, the next phase starts – reviewing their ideas and correct the small mistakes that they made in typing. Anton also quickly adds some more visual hints to ideas, like arrows pointing to the text or underlines some important keywords. Kai enjoys the dictation feature of the application to quickly add some more information to some ideas that he feels that aren’t very well explained.
With all ideas correct and complete, they go into the next phase of requirements elicitation for their application start grouping them into functional and non-functional features for their system, using the background colors to create visual hints. This enables them to have a quick overview on the ideas they gathered and relate them to a specific part of their software planning.
The nest step in the process is to link the ideas into a tree like structure, as all good software developers do, identifying the dependencies between the ideas. This will prove useful later, when they will create the necessary UML diagrams for the software.
Since they will go afterwards to their own workstations and they need to review and create the appropriate documents for their project, Greg uses the saving feature of the application, which will create a folder in his Workspace on the BSCW system. In it he will later find a screenshot of the application’s screen to have a quick visual of the session, the entire session metadata saved as XML for later use in other applications and a collection of all ideas saved as HTML documents.
Because all work needs to be shared and the session repository is on the BSCW system, Greg can share his session with his colleagues by inviting them to the session directory. Thus, not only that all of them will have access to the session, but they can choose later to present the outcome of their work to their Professor, by using the Load session capability of the Microsoft Surface application.
An additional scenario can be used with this application, if there is time for Brainstorming session preparation:
If they have time to prepare for the meeting, some rough ideas may have been conceived before and they want to share with each other. Because using the Microsoft Surface is not possible with prior reservation, they may use the additional clients of the system to record their initial ideas: Adobe Air application (available for Windows, Linux and Apple) and an iPad and iPhone client for the mobile devices. Anton and Vladimir use the Adobe Air client to quickly create and send their ideas to the Surface default user, where the brainstorming items will be recorded and made available in the next Brainstorming session. Kai uses its iPhone to do the same, as he gets its best ideas while commuting.
Next let’s see how the requirements look like!
The present article will present possible answers to the research questions presented in already existing work. The thesis must further investigate their correctness and evaluate the implementation of these solutions with the help of users.
A framework for analysis was used to search for the solutions proposed by other brainstorming applications and six direct categories aroused from the research questions:
- Additional features to improve the usability of a brainstorming supporting application;
- Input devices / mechanisms to register ideas into the system;
- Post brainstorming organization of ideas
- Mechanisms to support the brainstorming session;
- Solutions to address the physical limitation of the tabletop;
- Is the brainstorming session result used by other systems?
My thesis claims as a basic concept that an ideal tabletop application supporting the brainstorming process must have an implemented answer to the issues raised above.
Considering the first category of additional features which makes an application usable in the context of the brainstorming process, some properties were already mentioned and looked for in different applications: organization of ideas, relationships between ideas, sessions save and session restart for being continued. A new solution proposed by the thesis is the rating of ideas. Of course, the brainstorming session must not be influenced by categorizing the ideas in any way. However, there might be some ideas on which everyone to agree that they need to persist and need to be considered after the session end with high priority. In order to remember those very good ideas which meet the approval of everyone a rating mechanisms should be implemented.
The rating mechanism is intended to help the brainstorming session members to remember important ideas generated in the process. Nevertheless, if it is discovered that the rating of ideas is more an impediment to the creative process than a benefit, the rating mechanism can be enabled only in the evaluation phase of the brainstorming session.
The software touch keyboard is the usual mean of registering the ideas in an application running on the tabletop. This might be considered a difficult way to input the ideas by some participants. This is why the usability of a brainstorming supporting application would increase if more input means can be provided. Additional to typing on the keyboard on the table, a normal microphone could be used as an input device allowing the participants to give their ideas by normal speech. This method requires a speech recognition program in order to get the idea textually registered and consequently written on the table.
Moreover, the thesis claim is that graphical ideas can be important to the brainstorming process additional to the textual ones. Thus another way of registering a graphical idea into the application is to draw it with the finger on the table.
A brainstorming supporting application should address the scenario in which the participants get stuck and no new ideas are generated in a predefined time interval. As a mechanism to support the brainstorming session, the thesis proposes meshing with another system to bring up content related to the topic of the brainstorming session. For example bringing 5 – 10 tweets related to the topic could positively influence the creative process and get the session on track.
The system which supports the brainstorming process around a tabletop should address the scenario in which there are more participants than can physically fit around the table. This faucet of the brainstorming application can be the feature of adding other devices to the same idea generation process, as the tabletops have finite space that people can occupy in order to access it. A practical solution in this scenario is to allow people, using their own mobile phones, to interact with a mobile client of the application. They can use it to textually write their idea on the mobile device and sending it afterward on the table. In this way, they are not required to sit at the actual table to be able to participate with ideas.
Actually, in the scenario of having too many participants at the brainstorming session, it might be a good idea that everyone uses the mobile devices to input ideas and none of them to sit at the tabletop. This way no one has a preferential sitting place. The tabletop device can be used only to organize the ideas in a next phase when the direct manipulation is essential.
After the brainstorming session has ended and analyzed, the result should be persisted. It is possible and advised by the present thesis that the ideas resulted from the brainstorming session to become registered ideas in another system which can further make use of them. For example, the result might be saved as a list of Idea objects in a permanent repository system (like a CMS) in order to be used in a project or just to become available in a shared workspace.
Let’s see how the typical use case looks!