Development of an Assessment Tool To Measure Students’ Meaningful Learning in the Undergraduate Chemistry Laboratory

Development of an Assessment Tool To Measure Students’ Meaningful Learning in the Undergraduate Chemistry Laboratory

Kelli R. Galloway and Stacey Lowery Bretz

Journal of Chemical Education Article

Publication Date (Web): March 11, 2015

DOI: 10.1021/ed500881y

10 pages

SUPPORTING INFORMATION (6 PAGES):

The authors provided a written summary of how certain questions were revised, the assignment of cognitive versus affective to each item, and the actual pre- and post-surveys used in the full study.

ABSTRACT: http://pubs.acs.org/doi/abs/10.1021/ed500881y

The authors, both from the Department of Chemistry and Biochemistry at Miami University, begins this paper with the claim that while chemistry educators swear by the importance of the laboratory to student learning, there is very little published evidence supporting this claim. This leads some to question the resources, time, and money put into laboratory courses.

Motivated by this, the authors set out to develop an assessment tool to measure meaningful learning in the laboratory.  They point to already existing assessments that aim to measure some aspect of laboratory learning but they believe that they are limited in comprehensively assessing whether meaningful is taking place. 

They used as a starting point Joseph Novak’s Theory of Meaningful Learning and Human Constructivism to develop an assessment tool which they call Meaningful Learning in the Laboratory Instrument or MLLI. The goal of the MLLI is to “understand students’ perceptions of learning in their undergraduate chemistry courses”.  As the authors noted, there is a lot of emphasis on “doing” a lab from start to finish and they hope to find out “the extent to which students think and feel about their laboratory work and the connections between thinking and feeling”.

The research was guided by the following questions taken verbatim from the article:

1. Are the cognitive expectations of students fulfilled by their experiences in an undergraduate chemistry laboratory course?

2. Are the affective expectations of students fulfilled by their experiences in an undergraduate chemistry laboratory course?

3. In what ways do these expectations and experiences change as students learn more chemistry, that is, move from general chemistry to organic chemistry?

This article describes the development and the reliability and validity tests of the MLLI and the findings of their research.

The Methods section describes how the authors developed the items in the MLLI, the administration of the pilot study in terms of how students were asked to complete the survey and when, and what revisions were made and on what basis based on statistical analysis of the results of the pilot study. One of the considerations is that wording should be specific so students know that they are to evaluate only their time spent doing an experiment in the lab and not outside activities such as completing the lab report.

The revised version used for the FULL study had 31 questions, 16 cognitive, 8 affective, 6 cognitive/affective, and 1 indicator.  The response format was replaced by a slider bar from 0 to 100% indicating their agreement with the stem statement (was a 4-point Likert system ranging from strongly disagree to strongly agree). 

Some notable results of the full study conducted at the beginning and end of Fall 2013:

·         800 students took the pre- and post-test using Qualtrics survey software; 614 students had matched responses (436 GC, 178 OC).

·         Some lab instructors offered extra credit to increase response rate

·         Before the researchers extracted conclusions from the results, they carried out validity studies through student interviews to find out how students interpret the survey items and if they support their cognitive/affective assignments.

·         Overall results were presented in terms of change between courses (going from GC to OC) and between time (pre- and post-responses).  In the author’s original words:

o   For the between factor Course, this significant result means that GC students scored higher than OC students on cognitive (η2 p = 0.015), affective (η2 p = 0.03), and cognitive/affective scales (η2 p = 0.05).  

o   For the within factor Time, this significant result means that all students decreased in scores from pre- to post-test with η2 p = 0.44 for cognitive, η2 p = 0.04 for affective, and η2 p = 0.18 for cognitive/ affective, indicating small to large effects.

o   Stated another way, students had higher expectations for their chemistry laboratory courses that went unmet by the end of the semester.

o   The only significant interaction was for the cognitive/ affective scale but with a trivial effect size (η2 p = 0.01). This significant result indicates that GC students had a greater decrease in their cognitive/affective scores than OC students. The effect size, however, indicates that any difference in how general chemistry students changed compared to the organic chemistry students is too small to be meaningful. The nonsignificant interactions for the cognitive and affective scales are interesting to note because these results indicate that a student is likely to decrease in their ideas about learning in the lab no matter if they are in general or organic chemistry lab.

o   Comparing the ANOVA results to the scatterplots (Figure 2) reveals that the small effect for the decrease in affective scores is actually the result of an almost equal number of students increasing in affective scores as the number of students who decreased. This same phenomenon is not seen, however, with the cognitive scores. Rather, the sample appears to shift to lower scores as a whole. When laboratory curricula are developed, the focus is on what students will do and think in lab but typically does not consider how students will feel in lab. As a result, an entire course may have similar cognitive and psychomotor experiences but have a diversity of feelings toward what they’re thinking and doing.

In the Conclusion section, the authors have the following statements:

Results from the inferential statistics show that GC and OC students change in a similar way from pre- to post-test for cognitive, affective, and cognitive/affective experiences. The OC students consistently scored lower on the MLLI than GC students, but students in both courses responded that their experiences that failed to meet their expectations.

The MLLI has been shown to be a valid and reliable way to assess how students’ cognitive and affective experiences change before and after the semester and between GC and OC.  Instructors can use these results to improve the lab curriculum and/or the pedagogy to improve meaningful learning.  With the established learning goals in mind, instructors can reflect on how students may respond to the items in the MLLI.  Actual student responses may reflect how the curriculum is operationalized by both students and instructors.  This will help inform where the gaps are and where improvements can be made.

See the last section for a discussion of limitations of data for interpretation purposes.