Math Teachers’ In-class Information Needs and Usage for Effective Design of Classroom Orchestration Tools (Still working on this page...)
Receiving real-time information about students’ learning can enhance classroom teaching effectiveness. Learning dashboards are the most common means of communicating such information to teachers, however, research indicates they often fail to deliver actionable and context-specific information. In response to these challenges, researchers have emphasized the importance of involving teachers as end-users in the design process of teacher-facing learning dashboards to identify their authentic needs. However, this involvement is a complicated process and requires more innovative research methods. In this project, we conduct in-person classroom observations and interviews with elementary and middle school math teachers teaching in Western Massachusetts' public schools to identify their information needs. From our results, we propose design guidelines for HCI and Learning Analytics communities to enhance the effectiveness (i.e., actionability) of learning dashboards.
Problem Definition
Literature Review • Problem Definition • Forming the Research Team
Literature Review
To kickstart our research study, we initiated an in-depth exploration of relevant research papers, focusing on teacher-facing learning dashboards. This thorough literature review aimed to uncover the current state-of-the-art in this domain and pinpoint previously unexplored research gaps.
Problem Definition
After a comprehensive examination of prior research and consultations with experts in the field, we identified a critical gap within the area of teacher-facing learning dashboards. Based on our literature review and the identified gaps, we've refined the scope of our research as follows:
-
Our study exclusively focuses on math educators instructing grades 3-8 (covering upper elementary and middle school levels) within the public school system of western Massachusetts.
-
Our attention is on learning dashboards designed for real-time decision-making during in-person math classes.
The pivotal issue we've identified is the inadequacy of current learning dashboards in providing actionable and context-specific information to teachers. This deficiency has given rise to several challenges in the widespread adoption of such systems, which we will delve into further in subsequent sections.
Forming the Research Team
Recognizing the multifaceted nature of this challenge, we understood that tackling the problem would necessitate expertise spanning various domains, including education, computer sciences, human-computer interaction, psychology, and more. To this end, we created a dedicated team, including me (Hadi!), serving as the lead researcher alongside four accomplished advisors, each specializing in diverse research areas.
One of the key challenges we faced was ensuring seamless collaboration within our team to ensure that decisions were made based on input from all team members. In this regard, the lead researcher played a pivotal role in facilitating communication and synergy among team members throughout the entire research process, from inception to execution.
General Information
School
Laboratory
Project Duration
Team
Tools
Skillset
University of Massachusetts Amherst
Advanced Learning Technologies (ALT)
September 2022 to September 2023
Collaborated with advisors across fields of Education, HCI, and Computer Science. Team members: Mohammad Hadi Nezhad, Francisco Castro, Prof. Beverly Woolf, and Prof. Ivon Arroyo.
Google Suite, Zoom, Adobe Photoshop, Adobe Premiere
Interviewing, Field Observations, Contextual Inquiry, User Studies, Research, Project Management, Team Work, Thematic Analysis, Qualitative Research
Project Process
The focus of this project is on solving a problem that exists for actual users of learning dashboards. Therefore, to make sure that we are solving the right problem for the right users, we are taking a user-centered approach.
Empathize
Problem & Motivation
Research Questions
Methodologies
Participant Recruitment
Field Observations
Semi-Structured Interviews
Data Analysis
Thematic Analysis
Interpretation Sessions
Results
Thematic Analysis Results
Discussion & Design Guidelines
Phase 1: Empathize
Problem & Motivation • Research Questions
Problem & Motivation
Todo
Research Questions
In this project, we delved into three research questions aimed at addressing the specific problem at hand.
Research Question 1
What data do grade 3-8 math teachers want to collect about their students during class time?
Research Question 2
How do grade 3-6 math teachers intend to use student data?
Research Question 3
How can understanding these data needs and usage support the design of orchestration tools?
Phase 2: Methodologies
Participant Recruitment • Field Observations • Semi-Structured Interviews
Participant Recruitment
We distributed a recruitment poster across public elementary and middle schools in various Massachusetts school districts. The poster redirected participants to a short screening survey where we asked about the grades they are teaching, their teaching experiences, and their schools. After filtering the responses for the purpose of diversity, we recruited five teachers of grades 3 and 6 from four public schools and four school districts with varied teaching experiences.
Field Observations
We conducted field observations to record in-person classroom activities, such as teachers’ instructions, teacher/student
dialogues, student presentations, group work, individual problem-solving, and the use of educational technologies. The
observations provided us with context for refining the interview questions around our observations of what teachers
were doing within the classroom.
[TODO]
Semi-Structured Interviews
A few days after the observations, we interviewed each teacher in a remote session for around 45 minutes. Our interviews targeted teachers' motivations, frustrations, and explanations for the in-class activities we observed, and to gather information about other activities and practices that were not necessarily part of our single-class observation.
[TODO]
Phase 3: Data Analysis
Thematic Analysis • Interpretation Sessions
Thematic Analysis
We analyzed video recordings, transcripts, and observation notes from the interviews and observations through collaborative Thematic Analysis and Interpretation Sessions. We conducted a bottom-up (Inductive) coding approach to extract codes and themes from the data.
We had multiple approaches for reducing the inherent subjectivity of thematic analysis.
[TODO]
Interpretation Sessions
Todo
Phase 4: Results
Thematic Analysis Results • Discussion & Design Guidelines
Thematic Analysis Results
Our analysis resulted in a Theme Hierarchy related to the types of information that grade 3-6 math teachers want to collect about their students during class time and how they intend to use this data. The results include five high-level themes each having a number of lower-level sub-themes.
In our report, we described each theme and included sample representative participant quotes to bridge the perspectives of both teachers and our research team.
[TODO]
Discussion & Design Guidelines
In our design guideline, we considered the most prominent constraints of teacher-facing learning dashboards during class time, and the results of our data analysis answering the first two research questions. By combining them, we proposed four design guidelines that could assist HCI and Learning Analytics communities to improve the effectiveness of learning dashboards.
[TODO]
Constraints
#1 Todo
Todo
#2 Todo
Todo
#3 Todo
Todo
Reflection
#1 Todo
Todo
#2 Todo
Todo
#3 Todo
Todo
#4 Todo
Todo
