Throughout the course, you will be involved in a group project. Each project team should consist of two or three students, although exceptions may be made in consultation with the instructor. The purpose of the project is to gain experience in applying concepts and techniques from the class to a real-world problem. What is most important to keep in mind is that your project marks are not determined by a wealth of functionality, nor by the complexity of what you tackle. Remember: this is a course on human-computer interaction.
Our project theme this year of "Making Life Easy" is described by the Usability Professionals' Association. Apart from the general guidelines offered on UPA webpages, the only criterion we will insist upon for your projects is that they are not centered around a conventional graphical user interface interaction paradigm. As the term progresses, your system will evolve from a paper description, to a simple prototype, and finally to a functional physical system.
All reports and information on your project are to be collected into a Web-based notebook, maintained separately by each project team. You may wish to review some of the notebooks from previous years for examples.
Each group will designate one "webmaster" who will be responsible for maintaining the notebook on a publicly accessible web server. The top-level URL for your notebook should be communicated to the instructor no later than January 24 to ensure that you have an opportunity to verify the results of a test grab. All of your project information must be stored under that URL. For important information about the web notebook see these helpful guidelines prepared by previous TAs. Be forewarned: just because your web pages look right in your account does not mean they will appear correctly when transferred, and this may result in you losing marks.
A snapshot of your notebook will be downloaded by wget at 16h00 on the due date of each term project component and graded as per the criteria described below. Each component of the project must be associated with a link from the main page of your notebook. Its content should be organized in such a manner that a concise overview is presented to a reader who only has approximately five minutes to gain an understanding of your work. You may provide as much supplemental material as you wish, linked to each entry, but the main entry must make sense on its own.
While your work will be assessed as a team effort, individual marks may be assigned based on the graders' assessment of a disproportionate contribution of effort or significantly varying quality of work by one or more members. As the evaluation of group work is a contentious process, we are requiring that you include in your project notebook a summary of individual contributions for each component, signed by all members of your team. If you prefer, your team may submit this form in-class, as a hardcopy, on each due date. The TAs will not grade any material for which this contribution sheet has not been received.
It is understood that various elements (e.g., web page setup, sensor processing, user testing) of the project will involve the combined efforts of multiple group members; nevertheless, the individual contributions of each member must be described in reasonable detail. This is being done both to reward exceptional efforts as appropriate and also to ensure that group members maintain clear communications with each other throughout the term.
This assignment is intended to acquaint you with the user community for whom your group will be designing and developing a prototype technology. It will provide you with an opportunity to observe real users engaging in everyday activities that may offer opportunities for technology to facilitate their tasks.
The intent of this first component is for you to gain a first-hand understanding of the users for whom your system is intended. This is where observation sessions are invaluable. From your observations, you will develop personas and use case scenarios associated with the activities they will perform. These materials will guide your overall design. As such, this part of the project is perhaps the most important, as it constitutes the basis for your work throughout the rest of the semester.
As you think about the problem and your proposed solution, keep in mind the high-level question we will be asking throughout the remainder of the project: "How will we judge the HCI quality of your solution?" Remember that the project is not intended to require massive engineering implementations.
A tentative grading scheme (subject to minor revision) appears below:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| Personas | Who are the users of your system and what are their goals, skills, abilities, and technical experience? What do your users need? What is the problem they are trying to solve? |
Persona descriptions, Defining the user group who stands to benefit from your project. | 30% |
| Use Case scenario | What is the expected interaction between users and the system as they try to accomplish a particular task? | Identifying a real need with reasonable impact. Must be a story about the user performing a task The story must describe the important user actions and system reactions in reasonable detail. Scenario must convincingly demonstrate the use of the proposed system functions to solve the proposed problem. |
30% |
| Related products | What are the existing solutions (products) to this problem? | Evidence of research into relevant solutions | 10% |
| Product comparison | Why is your solution different? Better? | Superiority. Must demonstrate convincing justification that some aspect of solution makes it a rational choice for the target user, given the alternatives presented above | 10% |
| High Level Design | What building blocks are you going to use? What existing components will you rely upon to realize its functionality |
Clearly communicated block diagram and explanation Use of existing building blocks |
10% |
| Feasibility justification | What additional requirements need to be developed? What are their functional requirements? |
Clarity of vision into ensuing stages of work A concise description of the anticipated development effort necessary on your part Sufficient information to convince your TAs that this effort is feasible. |
10% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
You may find it useful to consult last year's project notebooks to get a sense of expectations, both in terms of organization and depth of content, but please keep in mind that the projects may differ significantly year to year in terms of their themes and formal requirements.
The objective of this part of the project is for you to design and implement a low-fidelity mockup of your proposed system, based on team discussions that led to the completion of the proposal, instructor/TA feedback, and your experience in working through a previous prototyping effort.
Note that if your group has not been given the "go-ahead" to proceed with your proposal from Defining the Project, you will first need to revise those materials and obtain approval from the instructor.
Build a paper prototype, physical mock-up, or other low-fidelity (specifically, not computer-based) prototype of your new design, as you deem appropriate for investigating and validating high-level decision options that your team is considering. Remember that your prototype should not be overly detailed, in particular as you want to focus on the "big picture" at this stage. It will likely be useful (if not imperative) to employ "Wizard of Oz" techniques, with members of the design group animating the live operation of the envisioned system.
Your deliverables are a set of illustrations, photographs, or videos that explain the operation of your low-fidelity prototype. This should be sufficient to illustrate the "working flow" of your system and explain how each task will be carried out.
A tentative grading scheme (subject to minor revision) appears below:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| Prototype | Did you build a paper prototype? How well does it represent the system you described in your project proposal? |
Paper prototype was developed. All major functionalities of the planned system are represented within the prototype. |
20% |
| Illustration/Videos | Did you document your prototype with pictures? Did you capture functionalities that rely on Wizard of Oz techniques with videos? |
Illustrations and pictures provided demonstrate all major system functionalities adequately. Videos showcase Wizard of Oz techniques used to animate paper prototype. Be sure to organize your illustrations and videos in proper order to demonstrate the working flow of your system. |
40% |
| Design Decisions | Why was your paper prototype designed the way it was? Can you explain the logic behind some of your design choices? | Representation of all major system functionalities within the paper prototype were well justified and explained. Include a list of your high-level goals and explain how these have led you to implement specific functionalities for your system. |
40% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
You will now prepare and carry out an evaluation plan that will inform your work in the next part of this project.
First, determine a set of usability goals that will be used for evaluation purposes. Usability goals are the overall characteristics that you use to define the success of your system. For instance, if you would like to ensure that your system is easy to learn, then your usability goals could be something along the lines of "New users should be familiar with all aspects of the system within 1 hour of use". There are several metrics you can use to measure such success, such as task completion time, ratio of successes to failures, percentage of tasks completed successfully, or user satisfaction on a Likert scale.
What you are interested in determining is whether these usability goals are being met. To do so, you test your prototype on a set of benchmark tasks, which should:
Note that a benchmark task can (and often will) consist of several smaller subtasks, such as navigation, selection, confirmation/cancellation, etc.
Then, design the test materials as relevant to evaluating your prototype. These may include an observer briefing, user introduction, pretest questionnaire, user or training documentation, a set of test tasks, data collection sheet, post-test questionnaire, and a test script. Some helpful examples can be found in this excellent overview on Usability Evaluations and (although oriented specifically to the evaluation of web sites), the W3C Web Accessibility Initiative Usability testing questions.
Apply the usability testing procedure on at least two potential users (from an actual expected user population) and document the results. You may wish to make a recording (audio, photographic, or video) of your test sessions so that you can analyze these later. The recordings may also be useful to illustrate something you learned about the design (whether good or bad), as described below in the grading scheme. You must first obtain consent from the participant(s) before making any recording. The results from these early test sessions will likely prove helpful as you modify the design of your system throughout the term.
A tentative grading scheme (subject to minor revision) appears below:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| Benchmark tasks and usability goals |
What are the tasks users can perform with your system? How will you evaluate the success or failure of these tasks? What usability goals are most relevant to your solution? Do the tasks you have chosen demonstrate the fulfilment of these usability goals? Have you described how your solution might fail to meet these goals? How might your solution fail to meet the usability goals? |
Identification of usability goals Identification of tasks Identification of failure conditions |
30% |
| Samples of test materials |
Given that users have never seen your prototype, what information will you need to provide to help them understand and use the system? What materials will you need to collect this data for later reviewing? |
Tailoring test materials appropriately to the benchmark tasks. Inclusion of all necessary and relevant test materials |
20% |
| Brief extract or transcript of test session | Does the example or transcript effectively illustrate interaction with your low-fidelity prototype? | Illustration of interaction with low-fidelity prototype Make sure that your choice of format is appropriate for the content and viewable on a range of computer platforms without requiring non-standard codecs (MP4 is generally a safe bet), and is of reasonably small file size (we don't need uncompressed HD). |
15% |
| Brief summary of test results | What lessons did you learn from each of your benchmark tasks? From the test of your data collection documents and procedures on two users, are there any revisions (to the script and your prototype) to be made? |
Identification of good aspects of design Discussion of changes to the design suggested by the test results Table commenting on results for each of the benchmark tasks |
35% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
At this point, you are ready to give users a clearer sense of the look and feel of your application. This means that your prototypes will evolve from paper, cardboard, and styrofoam, to a second version that runs on a computer. For clarity, "computer" in this case means that the system performs some type of information processing, whether through a full-fledged computer, a mobile processor, an embedded microcontroller, or similar.
However, you are equally concerned with learning about any design problems at an early stage, before you have invested an inordinate effort in development. Thus, in addition to implementing your second prototype, you will now formulate a detailed usability test plan, expanding on your effort from the earlier test plan. Rather than conduct the usability testing yourself, this will be the task of another team during the next phase of the project.
For this assignment, your team must implement a computer-based prototype that allows users to evaluate the "look and feel" of your system with minimal or no use of Wizard of Oz techniques. While this prototype should be based on your description from earlier parts of the project, it is understood that the design will include modifications in light of TA feedback and initial user testing. Keep in mind that your prototype does not need to be functionally complete, nor is there any requirement that it use the final anticipated hardware at this stage. Rather, it should simply be sufficient to give a reasonable (albeit imprecise) impression of the intended interaction to candidate users. In this regard, an initial user manual should be prepared unless such a manual is inappropriate for your user group.
Since your prototype will likely need to be taken "on the road" for testing with actual users, you should ensure that it runs on commodity hardware, is ideally agnostic of operating system, and requires no special operating skills to install and launch. If this is not the case, it will be your responsibility to make available the necessary components to the testing group providing the formative feedback on your prototype.
You must also provide a usability evaluation and testing plan for your prototype. Note that the evaluation will be conducted with no assistance from you, so you must be very explicit as to exactly what steps should be carried out.
One of the evaluation activities must consist of a formal "laboratory" experiment, i.e., a "usability test", in which some quantitative measurement (e.g., learning time before user is able to use the system as intended, number of mistakes users make before invoking a specific function, as instructed) can be made. Target numbers should be justified.
A second activity may consist of either of the following evaluations:
The point of such an exercise is is to simulate a user's problem-solving process at each step in the the dialogue to see whether the desired/expected outcome occurs. In this case, you must describe in detail several common tasks that a user might want to perform with your system so that the evaluation team can "walk through" the appropriate operations. You should not have to provide instructions of the form: "now press this button and wait for the green light to flash"!
For those groups who received feedback from the TAs indicating that their usability goals for the low-fidelity prototype were unacceptable, you must first include a section in your notebook where you clearly define these goals and justify them. You should then refer to those goals as you explain the rationale behind your evaluation plan.
A tentative grading scheme (subject to minor revision) appears below:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| User Manual |
Can a user find necessary and sufficient instructions in the manual to run your system without your assistance? Is the presentation of content in the manual suitable for the intended audience? |
completeness and clarity of instructions | 20% |
| Prototype Development |
How has your vision of the project changed from the time of conception? For every change, new decision made, how do you rationalize your decisions from a standpoint of usability? ... feasibility? What was certain functionality critical to implement at this stage? |
Development decisions justified. Successful prototype implementation Functionality satisfying objectives |
40% |
| Usability Test |
What usability goals will be tested? What types of users are required for the test sessions? How many examiners are required? What equipment will the examiners need? How should your prototype be handled? (manual/documentation) How will you instruct your examiners to proceed? How should the examiners treat the test subjects, what should they tell them? What should the examiners avoid doing? What should the examiners avoid telling the test subjects? What should the examiners pay special attention to? What/How/When should the examiners measure? How should the examiners record and report information to you? What do you expect the results to be? (Best/worst case expectations) Tentatively, how will the results affect your next decisions about the project? |
20% | |
| Usability Evaluation | (See lecture slides and Usability Heuristics) | Completeness Clarity |
20% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
The objective is to give you experience in evaluating another group's design, as implemented in their high-fidelity prototype. This will be done by carrying out the exercises they have specified in the evaluation plan of their project notebook. You will also document and analyze your evaluation results, commenting on whether the group has chosen reasonable exercises to assess the usability of their system for the given benchmark tasks. If you feel that the exercises do not adequately serve to evaluate the strengths and weaknesses of the project, you may carry out additional exercises, provided that you justify their importance. Note that this does not mean you can ignore the exercises proposed by the development team (whose project you are evaluating). Finally, based on your analysis of the evaluation results, suggest what changes should be considered for the prototype system and describe how these would improve the design.
The assignment of projects to evaluation teams is as follows:
To assist you in carrying out these exercises, the development team should designate one of its members to act as a liaison to your group. This is the individual you should contact for any assistance, as required, related to the setup or operation of the prototype for evaluation. It is critical that the usability test be conducted on a number of representative users for whom the system is being developed. The liaison is thus responsible for putting you in touch with these users. Ideally, you will have at least three users available for testing, although it is understood that in certain cases, this may not be feasible. Please discuss with the instructor if you are having trouble identifying three representative users.
The results and analysis of the evaluation exercises you conduct, along with discussion of the relevance of each exercise to the intended tasks, should be included in your Web notebook.
A tentative grading scheme (subject to minor revision) appears below:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| Testing | Have all the documented tasks and test cases been completed with target users? | completion of all usability testing and evaluation plans documentation of results of testing and evaluation |
20% |
| Results and Analysis | What did you observe during the tests? What comments did users provide? Did any of the tests fail? If so, why? |
discussion of system usability on benchmark tasks as assessed through quantitative measurements or heuristics | 40% |
| Test Plan Critique | Did the test cases address the usability goals of the system? What improvements were made to the test plan to better suit the design goals? |
rationale for any additional exercises you carried out, or modifications made to the originally proposed test plan assessment of the quality of the test plan you were provided in terms of how well it fit with the usability goals of the design team |
10% |
| Design Critique | What design improvements can you suggest to better suit the target audience's needs? | formative feedback to the design team suggestions or recommendations for improvement |
30% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
Now that you have completed an initial prototype, (hopefully) carried out your own evaluation of the interface and obtained the results of a formal evaluation conducted by another team, you are now ready to implement another iteration of your design.
The required components and grading scheme for this assignment are as follows:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| description of interface evolution from previous prototype | Have you summarized the observations from testing and compared these with your earlier testing results? Have you explained the motivation for each change and its effect in terms of satisfying your design critieria or overcoming any limitations exposed by the evaluation? |
feedback provided by the evaluation team major strengths and weaknesses of your interface identified by their report changes made in response to your own observations or feedback from the evaluation team |
40% |
| the revised prototype | functionality how well it fulfills the objectives of the course project |
40% | |
| an updated user manual | Have you clearly indicated (ideally with a suitably defined colour scheme) the relevant changes that were made as a result of the feedback from (a) the TA (b) evaluation team (c) added functionality? | summarizes how to install/run your software and how players interact with the system | 20% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
The objective of this project component is to give you experience in evaluating the evolution of the development team's design from its earlier computer prototype to a functional alpha system. The intent in doing so is to provide HCI-specific feedback from your perspective as to how well the alpha system has met the objectives of the development team, and evaluate whether their design decisions and changes appear to have been reasonable.
The development team should begin by considering whether any changes are warranted to their test plan that were employed previously, in light of the evolution of their design as it progressed to the current alpha system. If so, these should be added to your notebook, and communicated right away (through your liaison) to the evaluation team. Part of your grade for this part of the project will come from your updates, as warranted, to these exercises.
The evaluation team should begin by reviewing the web notebook of the development team to see how they responded to your earlier suggestions and verify whether they have made changes to their test plan, as discussed above. You are also free to suggest your own revisions to the test plan, but if so, these should be discussed with the development team's liaison to make sure they are agreeable. You may also wish to make revisions to your own procedures based on feedback from the TAs.
Once your testing approach has been finalized, you should proceed, working with the same volunteer(s) as you did in formative feedback, insofar as possible, to carry out another round of usability test(s) on the alpha system. As before, your assessment should also include a usability evaluation, conducted without needing to involve a target user.
As before, the development team should designate one of its members to act as a liaison to your group. This is the individual you should contact for any assistance, as required, related to the setup or operation of the prototype for evaluation.
The results and analysis of the evaluation exercises you conduct, along with discussion of the relevance of each exercise to the intended tasks, should be included in your Web notebook.
A tentative grading scheme (subject to minor revision) appears below:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| Test and evaluation revision | What changes did you make to your own usability test and usability evaluation as a result of the evolution of your system? | changes and justication of changes | 20% |
| testing | Have all the documented tasks and test cases been completed with target users? | completion of all usability testing and evaluation plans | 10% |
| results and analysis | What were the results of your performed tests? What did you observe during the tests? What comments did users provide? |
documentation of results of testing and evaluation discussion of system usability on benchmark tasks as assessed through quantitative measurements or heuristics |
40% | design critique | What design improvements can you suggest to better suit the target audience's needs? | formative feedback to the design team, suggestions or recommendations for improvement | 30% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
The in-class presentations will be an opportunity for you to showcase the results of your work. The presentation should begin with a brief (maximum five minutes) overview, explaining the rationale behind the system you chose to implement. You should also provide a summary of your experiences testing and refining the system through its various stages of development, commenting on how the system evolved as a result of the evaluation feedback. The remaining time will be allocated for other class members to try out the system you have implemented and (simultaneously) a question and answer period.
Plan for demo syndrome: Make sure you have multiple "fall-back" plans available should any problems arise with your "live demo".
Grading will be based on the following four factors:
Based on the peer critique, your own testing, and further TA feedback, you will certainly have a number of refinements that appear important to make. Remember that you're not building a ready-for-market product, so your effort is intended to be focused on usability rather than functionality.
The required components and grading scheme for this assignment are similar to that of the alpha system, as follows:
| Activity | Question to answer | Breakdown | Weight |
|---|---|---|---|
| description of interface evolution from alpha system | Have you summarized the observations from the peer critique and compared these with your earlier testing results? Have you explained the motivation for each change and its effect in terms of satisfying your design critieria or overcoming any limitations exposed by the evaluation? |
feedback provided by the evaluation team major strengths and weaknesses of your interface identified by their report changes made in response to your own observations or feedback from the evaluation team |
40% |
| the revised prototype | functionality how well it fulfills the objectives of the course project |
40% | |
| an updated user manual | Have you clearly indicated (ideally with a suitably defined colour scheme) the relevant changes that were made as a result of the feedback from (a) the TA (b) evaluation team (c) added functionality? | summarizes how to install/run your software and how players interact with the system | 20% |
| Summary of contributions | What did each team member contribute to this component? | Don't forget the signed summary of individual contributions |
Last updated on 27 March 2011