What Is The Course About?

This course presents a wide variety of methodologies for evaluating technological interfaces with users. It is aimed at providing engineering students with the necessary knowledge on working with users, statistical techniques for analyzing the information gathered from the users, and how to interpret the results from the user studies.

Logistics

When And Where?

Social Sciences 1, Room 145, MWF 2:40-3:45 PM

Who To Contact?

Sri Kurniawan: Instructor

Tentative Schedule

W1: Introduction and course logistics. Data and the nature of measurement (scales, variables, errors). User evaluation and empirical research concepts. The role and place of user evaluation in interactive system development.

W2: Formulating "testable" evaluation criteria (hypothesis forming and testing, operationalizing variables). Basic experimental design (between vs. within subject; snapshot vs. longitudinal studies, etc). Components of user evaluations: participants, apparatus/stimuli, procedure, design, results and discussion.

W3: Types of user evaluation: accessibility vs. usability vs. acceptance testing, ethical consideration, IRB.

W4: Inquiry methods: interview, focus group, ethnography, contextual inquiry, survey  - collocated and remote

W5: Inspection (cognitive walkthrough, heuristics evaluation) and testing (thinking aloud protocol, retrospective testing, co-discovery learning) methods - collocated and remote

W6: Qualitative data analysis: grounded theory, content analysis of text and videos

W7: Quantitative data analysis: validity and reliablity, descriptive statistics, t-test, ANOVA, regression

W8: Using statistical tools - introduction and demonstration of the statistical tools of R, Excel and SPSS

W9: Qualitative student presentation: students dry run their qualitative data collection,  analysis and interpretation. Class discussions.

W10: Quantitative student presentation: students dry run their quantitative data collection,  analysis and interpretation. Class discussions.

Assessments

Homework: 30%

This homework requires you to perform data snooping on the quantitative and qualitative data from a survey that we conducted. The survey is in the Notes page. Just like in real life, the data is messy, some people probably filled in the survey more than once (there was a draw for two $100 Amazon gift cards for those who completed the survey), and we probably did not phrase the questions appropriately. The deliverable is a 5 minutes presentation + 1-2 minutes Q&A that has the following sections:

  1. Some literature survey on shopping experience, especially for people with disabilities.
  2. Descriptive statistics. Please present statistics that make sense in the context of shopping experience for blind people, e.g., don't present a correlation between age and gender. While that is interesting, it has nothing to say about shopping experience.
  3. Comparative statistics on at least 3 variables. You can do t-test, ANOVA, Mann-Whitney, Kruskall-Wallis, X-square, etc. If you are discarding some data, please say your justification for doing so.
  4. Qualitative data analysis, aka content analysis on at least 3 questions. You can sample a subset of the data but tell us your sampling strategy.
  5. At least 3 charts presenting themes from content analysis on one axis and one or more quantitative data on the other axis. 
  6. Discussion and conclusion. For discussion, I expect to see an interpretation of your findings beyond just "there is a significant difference between legally blind and totally blind in terms of frequency of shopping". It would be ideal if you could relate quantitative and qualitative data. I also would like you to critique the original survey (the phrasing of the question, the options, the order of questions, or anything else you feel should be different). For the conclusion, I would like you to summarize your findings, and if relevant, the take home messages for relevant researchers or industry. 

Student-Led Class Presentations: 20%.

  • One group of students (most likely of 2-3 students) will be assigned to learn more about a certain topic. The group may be asked to synthesize the chapters in one of the textbooks, suggested book or papers; when appropriate collect/analyze some data as a proof of concept of the topic; and present the synthesis/data analysis in class. This presentation should be presented in lecture format, around 30 minutes long, followed by 15 minutes of class discussion. The ppt or pdf file needs to be sent to me the day before the presentation so that I can upload it to the class website.

Student-Led Class Discussions: 10%.

  • Another group of students will be asked to choose and read the papers that accompany that week's presentation. This group will present a summary of the two studies and then will critique the papers for "flaws" in terms of user evaluation (e.g., the experimental methods, the sampling approach, the data analysis used). The rest of the class is then expected to start the discussion/debate - you have 20 minutes for this exercise, it is up to the group how to use this time slot. The ppt or pdf file needs to be sent to me the day before the presentation so that I can upload it to the class website.

Individual Project: 40%.

For this project I need you to recruit an appropriate number of users to evaluate a technological interface of your own choosing. You need to collect both qualitative and quantitative data, analyze the data, interpret them and provide conclusions in terms of improvements to the technological interface to make it more user-friendly. You must use one of the IDEO's Try methods and two methods from either the Ask, Look or Learn categories. 

    1. A report in the format resembling the example paper above.
    2. 10 minutes presentation covering: motivation (why did you choose that particular interface), methodology (participants, procedures, measures), results, discussion and conclusion. You must show at least one movie with a user doing the evaluation. 
    3. 5 minutes Q&A.

Policies And Accommodation

  • All students enrolling in this class are advised that Academic Integrity will be strictly enforced. Academic misconduct includes but is not limited to cheating, fabrication, plagiarism, or facilitating academic dishonesty or as further specified in campus regulations.
  • If you qualify for classroom accommodations because of a disability, please submit your Accommodation Authorization from the Disability Resource Center (DRC) to me in a timely manner, preferably within the first two weeks of the quarter. Contact DRC at 459-2089 V, 459-4806 TTY.

Readings/Resources (Subject To Change)

  1. Textbook: Cairns, P. & Cox, A.L. (2008). Research Methods for Human-Computer Interaction. Preface is here. Chapter 1. Chapter 2. Replacement of Chapter 3. Chapter 4. Chapter 6. Chapter 7.
  2. Textbook: Graziano, A. M., & Raulin, M. L., 8th ed. Research methods: A process of inquiry.
  3. Lazar, J., Feng, J. & Hochheiser, H. (2010). Research Methods in HCI.
  4. Usability Evaluation Methods
  5. Emerging research methods for understanding mobile technology use. In Proceedings of OZCHI, pp. 1-10.
  6. Various papers assigned in class.
  7. IRB.