Department of Computer Science
Central Washington University
400 East University Way
Ellensburg, WA 98926-7520
Department of Computer Science
Central Washington University
400 East University Way
Ellensburg, WA 98926-7520
January 2003
Department of Computer Science
Central Washington University
This report describes the development of alternative input devices and software used with the rehabilitation therapy of a severely disabled community member. The challenge was to enable its intended user to interact with a computer program despite his poor motor control, resulting in erratic pointer movements and inability to use left and right input buttons, and his reduced cognitive functioning. Two alternative input devices were constructed out of wood with substantial bulk to limit erratic arm movements and protective coverings to prevent accidental clicking of input buttons. The Choice Program software was developed that allows its user to indicate selection by hovering the pointer cursor over an item for a few seconds eliminating the need for input button clicks. The goal of the project is to enable its user to gain sufficient computer skills to use custom communication, entertainment, and education software planned as follow-on projects.
This technical report documents the development of software designed to aid with the therapy of the motor and cognitive skills for a severely disabled Ellensburg community member. An alternative input approach was required since the intended user lacks the motor and cognitive skills to operate a typical mouse-driven, GUI application.
Work on the project began in September 2002 as part of a CS 496 independent study course offered by Ed Gellenbeck with three students: Lindsay Appel, Jason Brown, and Kelly Crooke. The software was completed and installed on the user's computer in December. A follow-on project developing Augmentative/Alternative Communication software for the same community member has begun with an expected completion date of March 2003.
The content of this report begins with a background description of the problems addressed. It includes background information on the intended user of the program followed by a review of off-the-shelf solutions. The next section of the report describes the project goals. Following that are sections describing the alternative input devices and Choice Program software developed. The document ends with two sections: the first describing preliminary results from the use of the software and the second drawing conclusions on the success of the project in terms of satisfying a community service need and contributing to an undergraduate academic experience.
Jean Lofy, an Ellensburg physical therapist, first posed the problem. She desired technical help setting up a computer and software to help in the therapy of a severely disabled community member. It quickly became apparent that, due to the severity of the disabilities, off-the-shelf solutions would not work and a custom solution would need to be developed.
Ron (last name omitted by request) is an Ellensburg community member in his mid 30’s who was injured in a motor vehicle accident in 1996 while serving in the U.S. armed forces in Germany. He incurred multiple injuries, including fractured arm and leg bones and traumatic brain injuries to his head.
As a result of his injuries, he suffers from spastic quadriparesis meaning that he has difficulty moving his arms and legs and has uncontrollable muscle spasms. He has a pump inserted internally to provide medicine to control these spasms. He also receives ongoing Botox injections for the spasms in his hands. While greatly impaired, Ron has experienced modest improvement in his motor skills over the last year. Medical providers expect continued incremental improvement.
In addition to impaired motor skills, Ron's cognitive skills are also greatly diminished as a result of the accident. Formal testing at the University of Washington was inconclusive; Ron's lack of communication skills makes it is hard to determine what level Ron is functioning at cognitively.
Ron lives with his parents, who provide much of his caretaking. Under good conditions, they have full-time nursing help. Between 1996 and 2002, Ron received physical, speech, and occupational therapy from the University of Washington, Providence Rehabilitation Center in Yakima, and Ellensburg therapists appropriate to his progress. While Ron still occasionally receives therapy and re-assessment from Yakima and Ellensburg area therapists, his primary caretakers supply the bulk of his rehabilitation therapy.
Ron's employment before his accident was in information technology. In addition to his professional work, he also enjoyed playing computer games and tinkering with computer technology. It was hoped that his physical, speech, and cognitive therapy could be enriched by tapping into his memories and interest in computers.
The original idea was to set up a computer system for Ron to use with off-the-shelf hardware components and software programs. Several attempts were made during the summer of 2002 with this in mind. It became apparent that off-the-shelf solutions would not work.
Three significant problems needed to be overcome. One, Ron's lack of motor skills made the use of traditional pointing devices, such as a mouse, trackball, and joystick, infeasible. Two, his cognitive abilities made reading of traditional GUI menus, buttons, and text labels unrealistic. Third, the subject matter accompanying software developed for pre-reading children was inappropriate for an adult male. Each of these three problem areas is described below.
Ron lacked the motor skills required to position and click a pointing device to provide input to a software program. Attempts were made to teach Ron to use a traditional mouse, trackball, and joystick. After a summer of training, Ron never was able to successfully position the pointing device or even keep the pointing device upright on his wheel chair tray. Furthermore, Ron lacks the ability to perform the small isolated muscle movements necessary to control his clicks on the mouse button.
The greatest success with the pointing devices came when an enabled assistant guided Ron's hands and provided the necessary clicks and double-clicks. However, this was not a desirable solution as one of the goals for using computers in therapy was to empower Ron to choose between alternatives as a way to diagnose and improve his cognitive abilities.
Given Ron's lack of communication skills, it is very difficult to determine his reading ability. Ron is able to read simple words, but because he is unable to speak aloud or communicate what he reads, it is difficult to determine his reading level. Attempts at using software that required Ron to read labels (menus or buttons) for interaction was not successful, both because of lack of mouse skills and, presumably, because of lack of reading skills.
Several games and software designed for children at grade level 2 met with the greatest success. The iconic and graphical designs of this software, along with the beginning reading vocabulary, seemed to engage Ron and hold his interest. However, the content level was juvenile and not appropriate for an adult male. In addition, even the simplified interaction was too difficult for Ron to master.
This section discusses the underlying project goals and objectives that resulted in two major implementation efforts: the construction of an alternative input device and the development of the Choice Program designed to work with the alternative input device.
A project team was assembled to discuss the problem, consider alternatives, and develop project goals. In addition to the CS 496 participants (Ed Gellenbeck, Lindsay Appel, Jason Brown, and Kelly Crooke), two additional students (Roy Avery and Tammie Correll), and three additional specialists joined the project team. First, Jean Lofy, a physical therapist with Kittitas Valley Community Hospital, joined the project team to help with therapeutic issues. Second, Ian Campbell from Disability Support Services on the CWU campus joined the team, bringing his expertise in dealing with disability accommodations. The third addition was Diana Springer-Lund, an adjunct faculty member in the Computer Science Department at CWU. Ms. Springer-Lund, in addition to computer science, provided insights on psychological issues drawing from her degrees and career in psychological counseling.
The overall goal of the project was to develop software to help Ron develop the motor and cognitive skills required to use a computer. Adaptive computer technology is often a key component to empowering disabled individuals to gain communication skills, exercise decision-making, provide entertainment, and foster education. All of these were desirable long-term goals with Ron's therapy.
The project team developed four objectives. The first objective was to design and construct a robust input device that Ron could use to interact with the computer. Second, develop software to help diagnose Ron's current cognitive functioning within the context of interacting with a computer. Third, enable Ron to exercise simple decision-making by choosing between alternatives displayed on the computer. Fourth, provide entertainment to make Ron's interaction with the computer rewarding. Subsections follow detailing these four objectives.
A robust, alternative input device needed to be developed that Ron could use to interact with the computer since previous attempts with Ron using traditional pointing devices had been unsuccessful. Two major hurdles needed to be overcome. Ron's spastic quadriparesis resulted in erratic pointer movements and difficulty in precise positioning. Second, Ron lacked the fine motor control in his fingers or thumb required for clicking the buttons on the pointing device.
The team developed the following requirements:
The second objective for the Choice Program was to provide diagnostic help assessing Ron's cognitive capabilities. Previous attempts to assess his cognitive capabilities through computer usage had been unsuccessful, primarily due to Ron's inability to interact successfully with traditional input devices.
Since there was no baseline to begin at, the team decided to use both graphical and text labels with selection items. At a future time, changes to the program's input parameters are possible that would remove the graphics and present text-only labels for selection. If Ron's pattern of usage with text-only labels is consistent with his usage with graphic labels, assessment of Ron's reading ability is possible.
Changes in the positioning of the selections items on the computer screen will also help diagnose Ron's cognitive capabilities. During Ron's interaction with the Choice Program, the program automatically updates statistics that record frequency of selection of the individual items. Once sufficient data has been collected to analyze Ron's preference patterns, the position of the items on the screen may be changed. Ron's consistency of selection preferences, despite the changes in position, is evidence of cognitive and decision-making capabilities.
Given the severity of Ron's disabilities and his lack of communication skills, most of his decisions are, by necessity, made by his caregivers. This is not unique to Ron, but is a common concern among the severely disabled. Software is often effective in empowering disabled individuals to exercise some independent choices in their day-to-day routine. The team developed the Choice Program with the objective of empowering Ron with the ability to choose among alternatives and feel in control.
Ron's prior interaction with computers required a collaborative exercise with his caregiver directing his hand movements and providing the clicks. In reality, it was hard to determine Ron's desires given his poor motor skills and communication difficulties. Eventually, the caregiver would interrupt or end the computer session by the caregiver removing (prying in some cases) the pointing device from Ron's hands.
The team decided that Ron would feel more empowered if he had sole control of the pointing device. However, it was unrealistic to expect program interaction to proceed solely from Ron's interaction. Therefore, all program interactions would be accessible through two input devices concurrently. Ron would use the alternative input device; his caregiver would use the computer keyboard. In this way, Ron, as much as possible, will provide the interaction with the program. However, should he need help, his caregiver may assist via the keyboard without the need to direct Ron's hands or remove the pointing device from Ron's grasp.
The fourth objective of the project was to provide entertainment. The project team wanted to make Ron's computer therapy enjoyable and rewarding. Since Ron enjoys looking at photo albums and pictures, the project team designed the Choice Program using the metaphor of a photo album. Included in the program are sixteen multimedia photo albums on various topics that interest Ron. During program use, Ron's goal is to select the graphic/text item label corresponding to the photo album he wishes to view.
Two alternative input devices were constructed and tested with Ron. The first device, a slant board, revealed problems prompting construction of the second device, a mouse box. In addition to these two alternative input devices, Ron has also used a modified trackball and an off-the-shelf trackball and mouse for input devices with the Choice Program. Details on each of these input devices follow.
Figure 1a: Slant Board
Figure 1b: Ron with Slant Board
A slant board, designed to stabilize Ron's forearm during computer interaction, was constructed. The idea was to reduce Ron's erratic arm movements by providing sufficient bulk and support for his right forearm. An off-the-shelf mouse is used at the bottom of the slant board to provide the computer input. Figure 1a illustrates the slant board. While the mouse is visible with the slant board in Figure 1a, during use the mouse has a protective cover over it. Figure 1b shows Ron's use of the slant board.
While the design of the slant board was successful in reducing Ron's erratic arm movements during use, it proved ineffective as an input device. Ron's inquisitive nature resulted in him raising, inverting, and eventually dropping the slant board from his wheel chair tray. In addition, the protective covering for the mouse and buttons was not sufficiently rigid to prevent accidental mouse clicks. A third problem with the slant board was its size, which proved too large for the wheel chair tray it rests on.
Using insights gained from the testing of the slant board with Ron led to the design and construction of the second alternative input device: the mouse box. The mouse box consists of a wooden box that completely encloses an off-the-shelf mouse. Wooden handles attach to the mouse box to provide two-armed stability to Ron's movements. Figure 2a illustrates the mouse box. Figure 2b shows Ron's use of the mouse box.
Figure 2a: Mouse Box
Figure 2b: Ron with Mouse Box
At this point in the project, Ron's use of the mouse box is considered successful. As with the slant box, Ron will raise, invert, and eventually drop the Mouse Box. However, the mouse box's construction is more rugged and the off-the-shelf mouse more securely attached to the box so that this is less of a problem compared with the slant board. The project team is hopeful, that through regular use, Ron will grow more proficient at using the mouse box as the computer's input device.
An off-the-shelf trackball was modified to disable its left and right input buttons. With the buttons disabled, the trackball proved a viable input device for Ron. After seeing some success with the use of the modified trackball, the project team re-wrote the Choice Program to ignore completely accidental button clicks. The Choice Program now permits Ron to interact with the program using an off-the-shelf pointing device, as his accidental clicking of the buttons during interaction causes no complications within the program.
Figure 3a: Modified Trackball
Figure 3b: Trackball and Keyboard
Figure 3a shows Ron's use of the trackball. Figure 3b shows the simultaneous use of the trackball along with his caregiver's use of the keyboard during an interaction with the Choice Program.
Ron's caregivers are helping Ron practice using the Choice Program with an off-the-shelf mouse. Since the Choice Program ignores Ron's accidental clicks of the mouse buttons, some success with Ron using the mouse has been achieved.
Enabling Ron to use an off-the-shelf input device, either a mouse or a trackball, with custom-developed software designed to ignore accidental clicks is a long-term goal of Ron's therapy. Once this goal has been achieved, therapy to develop Ron's muscle skills to manage controlled mouse clicks may be considered.
This section discusses the architecture, design, and implementation of the Choice Program software. The project team chose a Model-View-Controller (MVC) software architecture using object-oriented design principles and implemented in the programming language C# using Microsoft's Visual Studio.NET environment and framework. Subsections on the software architecture, design, and implementation follow.
Since the Choice Program is a specialization of an event-driven GUI application, the project team chose the Model-View-Controller software architectural pattern. Invariably, the Choice Program will undergo modifications as Ron's capabilities improve or new content is developed. The MVC pattern accommodates these types of modifications by isolating changes to one of the three components.
The model component of the MVC pattern manages the program's data. With the Choice Program, the model knows what image files to display as selection items and what multimedia photo albums to play that correspond to user selections. In addition, the model component records the statistics on Ron's usage patterns.
The program's data is stored in as an XML document text file and encapsulated in the software as a class with methods providing access to the data. The project team may change the program's data using a text editor with the XML file. At this point, there is no easy-to-use tool or set of instructions for non-programmers to modify the program's data. Appendix A shows a listing of the program's XML data file.
The view component of the MVC pattern handles the display of selection items on the computer screen. It queries the model component for the image filenames to display as selection items and positions them on the screen. Since output of sound files is not built-in to C#, the view component includes a Sound Utility class to perform the necessary Window's API calls to produce sounds from the computer speakers.
The controller component of the MVC pattern responds to user input. With the Choice Program, the controller component notifies the view component whenever Ron hovers the input pointer over selection items that require changes to the computer screen. Along with hover events, the controller component notifies the view component about keyboard events whenever Ron's caregiver, watching over Ron's shoulder, decides to intervene in the progress of the program by using the keyboard.
The project team developed sixteen multimedia photo albums using Microsoft PowerPoint. Upon Ron's selection of a photo album, the controller component displays the photo album on the screen by invoking a Microsoft COM component that plays a PowerPoint presentation.
As is typical with MVC patterns, the project team used an object-oriented design and took advantage of the rich object hierarchy provided by the .NET framework. Appendix B of this report shows an UML class diagram for the Choice Program. As is also typical of many implementations of MVC patterns using Microsoft technology, there is not always a clean separation between the view and controller components.
The model component is represented as class RonXmlDocument, a derived class of .NET's XmlDocument. RonXmlDocument contains Getter methods to provide the view and controller classes with access to the program's data. RonXmlDocument contains transformer methods to allow the controller classes to increment the counters kept on Ron's selection choices.
The view component consists of three types of computer views: (1) a Splash Screen that displays the program's start-up image, (2) a Choice Screen that displays four photo album categories to chose from, and (3) a Photo Album Screen that displays four photo albums, all in the same category. These three views are represented as three separate classes inside the program. However, since these three views all share similar characteristics, that is, a full-screen display with a large custom mouse cursor, they are derived from a common parent class: FullScreenForm. FullScreenForm is derived from .NET's Form class.
The controller component uses derived button classes to respond to two types of user events: the mouse OnHover event and the keyboard OnKeyDown event. OnKeyDown does its work essentially by invoking the corresponding OnHover event handler method. OnHover is a polymorphic method whose actions depend on its object type: ChoiceButton, BackButton, or PowerPointButton.
Appendix C shows two UML sequence diagrams for the Choice Program. The first sequence diagram, Appendix C-1, shows an interaction of the program with Ron making selection choices by hovering his pointer device over an item. The second sequence diagram, Appendix C-2, shows how Ron's caregiver may interact with the program using the keyboard. Most of the time, the expected interactions with the program will come from both Ron and his caregiver through simultaneous use of Ron's alternative pointing device and his caregiver using the keyboard.
The project team implemented the Choice Program in C#, an object-oriented programming language developed for Microsoft's .NET framework. One advantage of using C# was access to .NET's rich object hierarchy provided within the framework.
Implementation began with the development of a prototype of the program using Microsoft Visual Studio.NET's drag-and-drop designer capabilities. Once the prototype was tested and approved, the team re-wrote the software from scratch using Visual Studio's text editor.
Appendix D shows a program storyboard consisting of annotated screen snapshots showing the user actions required to progress through the program.
The Choice Program was completed in December 2002, installed on Ron's computer, and tested that same month. Appendix E contains the one-page minimalist user manual prepared for Ron's caregivers. Upon installation, the project team spent several hours with Ron and his caregivers familiarizing them with the program and testing the software during actual use.
As mentioned previously in this report, initial problems with the slant board led to (1) its quick re-design as the mouse box and (2) the modification of an off-the-shelf trackball to disable the input buttons. In addition, the Choice Program was modified to ignore accidental mouse clicks that produced problems with overlapping windows in the program and to provide additional sounds that serve as feedback to user input actions.
Ron uses the Choice Program on a daily basis. Ron is able to use the mouse box with moderate success. Eventually, however, Ron's inquisitive nature will end up inverting the mouse box and attempting to remove the secured mouse from inside its protective housing. His caregivers hope this will become less of a problem over time.
With the modifications to the software to ignore accidental button clicks, Ron has used off-the-shelf pointing devices with the Choice Program with some success. Ron's poor motor control and erratic arm movements still present problems; however, since the need to click buttons has been eliminated and all the selection target areas are quite large, off-the-shelf pointing devices are feasible.
Because of Ron's lack of communication abilities, determining his cognitive function during interaction with the Choice Program is difficult. His caregivers believe that Ron is having some difficulty mapping his input actions with the program's responses to his actions. However, it is clear that Ron enjoys using the program and viewing the multimedia photo albums it contains.
Ron has verbalized some of the text labels present on the Choice Program, a clue to his cognitive and reading capabilities. He has also shown preference patterns in his viewing.
Since Ron is still learning to interact with the input devices and Choice Program software, the statistics being kept on Ron's selections are of little use at this stage. The project team is hopeful that, over time and with sufficient practice, the program's statistics will be reset to zero and meaningful data can be obtained.
The success of this project provided a valuable service to an Ellensburg community member and his family. It also sparked a follow-on CS 481 project developing Augmentative/Alternative Communication (AAC) software for Ron to use to communicate with others. The goal is for Ron to use the Choice Program on a daily basis to get him to the point where he is proficient at using a pointing device to make selections by hovering over items in a program. By March 2003, the expected delivery date of the CS 481 project, Ron will be ready for the new challenges and opportunities provided by the AAC software. Furthermore, an independent study CS 496 course is planned for spring quarter 2003 developing accessible entertainment and educational software for Ron to use, building off his progress with both the Choice and AAC programs.
The project also provided a valuable supplement to the undergraduate computer science curriculum offered at Central Washington University. It exposed students to a non-traditional CS problem area with unique challenges and state-of-the-art solutions. Students spent many hours researching commercially available alternative input devices and accessible software applications. The alternative input devices constructed by the students were creative and well engineered. The software design and implementation methods used state-of-the-art design concepts and tools. Testing the solutions developed with Ron provided insights and rewards not found in traditional programming assignments.
The project leader, Ed Gellenbeck, wants to thank all the project participants. All five students (Lindsay Appel, Jason Brown, Kelly Crooke, Tammie Correll, and Roy Avery) worked hard and made valuable contributions. Ian Campbell, the Disability Support Specialist at CWU, contributed his in-depth knowledge about using technology to enable the disabled. Diana Springer-Lund provided ideas and insights into accessing, accommodating, and developing cognitive capabilities. Jean Lofy was invaluable to the project team, carefully explaining the therapy background information to team members, working with Ron to assess the prototypes and solutions developed, and carefully proofreading and contributing to this Technical Report.
Furthermore, Ron's parents and caregivers were generous with their time and commitment to making this project succeed. Lastly, thanks to Ron for his patience in putting up with us and our watching and evaluating of his progress.