User Research on the Center for Learning and Undergraduate Enrichment (CLUE) Math Section

090421_Clue_BysomerKreisman_013_500px.jpg

The Center for Learning and Undergraduate Enrichment is an academic support service that provides tutoring for current students in subject ranging from Chemistry to Writing.

The details:

Type of Project: User Research for the course HCDE 313: Introduction to User Research

Duration: March 2017 - June 2017

Skills Learned: Qualitative research methodologies like pure observation, interviews, and surveys;  affinity analysis, and report writing.

an introduction to user research

During Spring 2017, I took Introduction to User-Centered Research (HCDE 313), which my first class in the Human Centered Design and Engineering major, and my first time working on an independent research project.  I had previously conducted interviews for articles that I wrote for The Daily of the University of Washington, the school newspaper. Through this work, I realized that the only way to really know what someone needs is to ask them directly, and to be a good listener. I learned this early on in my interview stage: if you try to fill in the blanks for someone, you lose you the chance at getting a quote.

This class was an opportunity to see myself as a sounding board by creating space for people to voice their needs and make specific and actionable design recommendations. It also provided:

  • Exposure to research methodologies – field observations, interviews, and surveys.

  • The opportunity to work on a project that I really cared about for 10 weeks.

  • Constant opportunities for feedback on my writing and research process from the course instructor and TAs

The problem space:

I wanted my research project to bridge my interest in journalism, research, and higher education, which is why I focused on the Center for Learning and Undergraduate Enrichment (CLUE), an academic support program at the University of Washington that provides 1:1 drop-in tutoring in subjects ranging from Chemistry and Physics to Foreign Languages and Writing. 

As a CLUE tutor. I love contributing to CLUE’s mission of providing an inclusive community that supports students in their academic and personal growth, but I wanted to know if we were meeting students’ needs and expectations. 

CLUE had never done section-wide evaluation of how effective we are at fulfilling their expectations. This project was a chance to identify these needs and potential pain points, and identify actionable design recommendations that could be implemented in subsequent quarters. I did this by observing students as they used the space, and interviewing students about their experience, needs, and pain points.

Because of my interest in improving CLUE, my research question was:

What factors contribute to student satisfaction with the CLUE Math tutoring service? How is this influenced by 1:1 interactions and the set-up of the physical space?

I focused specifically on the Math section and gauging how well it serves students’ academic needs (e.g. answering their questions pertaining to homework or clarifying concepts) or desire for collaborative spaces, and how conducive the physical space and interactions with tutors are in achieving this goal. 

field observations: recognizing my assumptions and biases

I sat at the tables designated for the Math section to get detailed observations.

For the first stage of the study, I used pure observation as a way to understand how students actually utilized the math 1:1 tutoring services offered at CLUE. To narrow my focus, I took notes on the 1:1 tutoring interaction between tutors and students, and how students modified the physical space to collaborate with others.

While conducting observations, I strived to recognize my own biases or preconceived notions about what I think they need. As a tutor, I wanted to students to positively perceive our tutoring service, but I had to step back and focus on students' behavior in as much detail as possible. It was difficult to just be a fly on the wall in the space because I knew a lot of the Math tutors, so they would often come over and ask about my research, but I managed to get through 3, 40-minute observation sessions.

For the data analysis, I identified themes by identifying recurring topics that were salient across all three of my observation sessions. I grouped together similar ideas under the themes of 1) tutor-student dynamics, 2) use of physical space, 3) use of resources, and 4) traffic flow because these items showed up in the greatest frequency across my observation sessions. Additionally, all of these factors influenced a student’s experience of using the CLUE Math section:

Asking clarifying questions: Students who asked clarifying questions or reiterated the content in their own words tended to verbally self-report that they understood the material bette.

Physical interactions between students and tutors: Students and tutors who were on the same physical level e.g. both sitting at the table, tended to interact for longer, and the students often expressed their thanks or understanding more explicitly in these scenarios.

Variability in use depending on the weekday: There was a lot of variability in business depending on the day of the week. On Sunday, there were 5 students in the Math section, and I later found that two of those people were Math tutors on duty that night. Contrastingly, Tuesday was the busiest day I observed.

Interviews

Since I've worked as a journalist for the past 2.5 years, I had a lot of experience conducting interviews, establishing rapport, and asking open-ended follow-up questions. For this reason, I really felt like I was in my wheelhouse, which made me substantially more confident in my work.

For this research,  I conducted 3 semi-structured interviews with current CLUE Math users, and I focused on asking about their motivations, rationale for their behavior, and expectations of the CLUE Math tutors. To understand student needs and avoid inserting my own biases, I asked open-ended questions such as, "What you make a tutoring interaction helpful for you?" or "Walk me through your approach to a 1:1 tutoring interaction. What kind of questions do you ask?" I also strived to protect the confidentiality of each of my interviewees and asked for consent before recording each interview. In addition to recording the interview on my phone, I took notes on a word document and typed asterisks every time the student said something noteworthy. 

Key for thematic analysis of transcribed interviews

With these interviews, I strived to be an active listener and let students articulate their needs and expectations in their own words. Based on their self-identified needs, I identifies key themes about their expectations based on what showed up across all three interviews + aligned with findings from my field observations.

surveys

Screenshot of my survey introduction text.

This section of research was the most difficult for me because I had no prior experience designing, piloting, or analyzing a survey, so this was a learning process for me. When designing survey questions, my goal was to validate, explore, and extend the findings I had identified in interviews. I wanted to see if these findings were represented of a larger population of CLUE users. My questions pertained to aspects of the 1:1 tutoring interaction, how frequently students used the space, and their overall satisfaction rating. I tried to pick variables to compare and identify potential relationships.

Survey request form that I distributed in the CLUE Math section to recruit participants

Contrary to popular belief, good surveys don’t start with a question. It must start with a persuasive introductory statement that explains the purpose of the survey and why their participation is crucial. It should also address potential concerns about confidentiality and anonymity and explain how the measures used to protect their privacy - this is exactly what I strived to do in my own introduction (see above) in an effort to highlight the goal of my research to improve CLUE as well as my gratitude for their particpation.

Since a survey can be easily disseminated to students who have a variety of opinions about CLUE, I recruited a broader sample of students who may have different levels of satisfaction about CLUE and can provide suggestions for improvement about CLUE.

Recruitment: In an effort to increase the survey response rate, I posted my survey on the Facebook page for the current HCDE cohort page, 2015 – 2016 Resident Advisers, and Odegaard Writing and Research Center staff because I am heavily involved in these communities, and I thought a personal appeal would make people more likely to respond. The post stayed up for 1 week (May 18 through May 25). The post stayed up for 1 week (May 18 through May 25), and participants were able to complete and submit the survey anonymously.

10 participants took my survey and although this sample was certainly too small to draw statistically significant conclusions, it was a step in the right direction because I collected quantitative data that could be compared to other methodologies' findings. 

key themes

Findings from my User Research

“If there’s a tutor that will help you out and understands the undergraduate toils of Math, why not use that service? – That’s how I developed a preference for CLUE.”
– Interviewee D

Value of 1:1 empathetic tutoring: In interviews and surveys, students indicated that they values having an empathetic tutor who could provide academic support and validate the challenge of difficult Math classes at the UW. This was further corroborated in my surveys, where 70% of participants agreed or strongly agree that they wanted their tutor to be empathetic. This suggests that students want tutors to provide robust academic help related to Math concepts, but they also want to feel supported as individuals.

Survey data from a series of Likert scale questions about 1) the value of empathy, 2) desire for 1:1 help, 3) overall satisfaction, and 4) preference for CLUE as a study center

 

Tutoring as a bidirectional process: Through my observations and interviews, I found that tutoring is a bidirectional process where students and tutors are actively engaged in the process. This was important because CLUE strives to support students in becoming independent learners, and CLUE tutors were available to support these conversations and help students identify questions they can ask themselves.

design recommendations

Lesson plan on empathy for new tutor training: During fall quarter training, new and returning tutors have a mandatory training. I proposed to a lesson where tutors identify moments of empathy with past tutors/mentors/teachers, and why this was valuable for them. Then, they can apply these principles when interacting with students.  

Conceptual worksheets: Since I found that most students come into the CLUE Math section to understand broad concepts or clarify their own logic, I proposed that we create a binder with worksheets that have common formulas, graph paper, and concepts, and the students and tutors can refer to these worksheets when working together and understanding concepts. 

what I would do differently next time?

If I conducted this research again, I would spend more time recruiting students so I could have a representative sample of interviewees and survey participants. This would make my data statistically significant and a more accurate reflection of students' expectations. 

future research

Although my findings focused on the conversations between tutors and students, and broader traffic patterns, more research can be done about aspects of the 1:1 tutoring interaction such as body language, the impact of wait times, and set-up of the physical space and how it supports collaboration. More research could also be done about the use of technology and outlets since I observed that most students worked on Math homework on their laptop.

Conclusion

This project was the perfect way to bridge my current love of higher education with my growing understanding of user research and user-centered design. Through this project, I was able to identify actionable recommendations that I will pass on to my supervisor, and I hope that they can inform changes or improvements as early as the 2017 - 2018 academic year. I hope that these changes can ensure that CLUE is an even more inclusive and supportive space for students both academically and professionally.

Regardless of the industry that I enter, research will always come first and inform my final product, article, or deliverable. Now, I have developed the technical skills to start identifying these needs and quantify them, and I’m excited to pursue user research in future projects.

OTHER PROJECTS | USER RESEARCH @ CLUE | UX WRITING & RESEARCH @ MICROSOFT NEWSPAPER EDITING @ THE DAILY OF THE UNIVERSITY OF WASHINGTON | FEATURE WRITING AT THE CENTER FOR SENSORIMOTOR NEURAL ENGINEERING | SOCIAL MEDIA @ SMAHRT