A Process for Developing Introductory Science Laboratory Learning Goals To Enhance Student Learning and Instructional Alignmen

A Process for Developing Introductory Science Laboratory Learning Goals To Enhance Student Learning and Instructional Alignment

Jennifer M. Duis, Laurel L. Schafer, Sophia Nussbaum, and Jaclyn J. Stewart

Journal of Chemical Education 2013 90 (9), 1144-1150

DOI: 10.1021/ed4000102

Supporting Information:

Lab report guide and instructions for completing a tutorial on notebook documentation using “Interactive Tutorials and Virtual Lab Activities” (5 pages)

ABSTRACT: http://pubs.acs.org/doi/abs/10.1021/ed4000102

Reader’s note:  The process described by the authors below for the development of learning goals and assessment methods may serve as a model to organize out program’s efforts to develop more student learning outcomes for our laboratory curriculum.  The steps seem well-defined but require an extensive amount of time and discussion.  This is the second article I have read describing the process of developing assessment methods and the work required to validate them.  This article reinforces the idea that to come up with meaningful learning outcomes and assessment methods and analysis is not a trivial process! This is something that I would like to do for our lab curriculum but it requires a considerable amount of time, effort, and coordination from all participants.  This paper also introduced me to the existence of a set of interdisciplinary lab learning objectives developed at Rice University: http://www.owlnet.rice.edu/~labgroup/assessment/lab_objectives.html

This article describes the process and results of the development of learning goals and student assessment for the introductory chemistry lab curriculum at a large university with over 1700 chemistry students.  The researchers include an instructional research coordinator, the course instructor, a science educator, a chemical education postdoc researcher, and undergraduate researchers. 

The goals of the research and development team are to, verbatim:

·         optimize instructional use of limited laboratory contact hours

·         help students achieve learning gains in basic (e.g., maintaining a laboratory notebook, selecting measurement tools with appropriate precision, and completing “common” techniques properly) and transferable (i.e., valuable in other sciences) laboratory learning content, and

·         cultivate departmental consensus, a small research

The 2-semester course is taken by students who have completed what in Canada is Grade 12 Chemistry and is required most Faculty of Science majors. [They used the term “introductory” but I suspect this is equivalent to General College Chemistry.].  “Furthermore, approximately 1700 students complete the full, two-semester course series and complete five, mixed-mode guided inquiry style activities (supported by readings, technique guides, online tutorials, and virtual laboratories in off weeks) over each 13-week semester.”

The researchers used a set of 5 program-wide learning objectives developed by science and engineering instructors at Rice University as a basis for developing the learning goals:

1. Basic laboratory skills

2. Communication and record keeping

3. Maturity and responsibility

4. Context

5. Integration and application of knowledge/experience

Some key things about the development of the learning goals:

Chemistry department members came up with a draft of learning goals for each lab activity using the learning objectives above.  These were synthesized by a chemistry education researcher.  Prioritizing basic and transferable lab skills helped focus the learning goals.  These were then reviewed by the instructor and other members of the research and development team and rewritten in the active voice. 

A few challenges arose.  When reviewing members disagreed, the group went back to the learning objectives to rewrite or eliminate a learning goal.  Efforts were also made to clarify and specify unambiguous articulation (see example in article about “titration technique”).

A final draft of no more than 2 pages of learning goals per lab activity was completed after iterative discussions reached consensus.  This consisted of clearly articulated and consistently aligned learning goals organized into a chosen global framework.  The first main section of the framework is basic laboratory skills: safety, laboratory procedures and techniques, laboratory records, data analysis, and laboratory procedure troubleshooting.

Some key things about the development of assessments:

The first step the group undertook was to look at quizzes, laboratory notebooks, report expectations to see what learning goals are already adequately assessed within the course’s grading scheme.  Those that are not assessed in any of these laboratory deliverables were prioritized for assessment development.  These developed learning goals assessments were then validated with student and experts.

The learning goals were organized into categories of possible assessment methods: should be observed, can be in written form, can be tracked online, and/or part of currently gathered and graded material.

An example given for the assessment development process involves titration.  To assess the learning goal of using correct procedures during titration, students are observed using a rubric provided in the lab manual for the titration procedure.  To assess ability to list the general steps of a titration procedure, students were originally given an open-ended question but they found that they could not elicit the desired level of detail.  They then analyzed data collected from the open-ended questions and observations of procedure to create a set of multiple-choice and true-false questions to assess aspects of titration most commonly not conducted or described correctly.  The authors state, that, “In a best-case scenario, however, development of suitable assessments is delayed until student observations and think-aloud interviews are completed and analyzed. This prevents many false starts and reduces the lost effort when a large portion of collected data must be set aside because the assessments did not reveal

sought information.”

Assessment validation

Validation of an assessment is required to

ensure that students interpret questions correctly as intended

ensure that experts, namely experienced chemistry professors agree on whether the assessment is congruent with the corresponding learning goal

ensure validation by experts from outside the team improves construct and face validity

ensures that there is a single agreed-upon correct answer to each multiple-choice item and that all distractors are correct

To further improve face validity, assessments were validated with students to identify wording that they interpret correctly, to ensure that the correct answer was not chosen for the wrong reason, and to understand student reasoning.  This was done through personal interviews of the target population (a small financial incentive was offered to student participants).  See detailed description of the interview process in the article.

*See example of how the group arrived at revisions of assessment question and elimination of jargon in both the question and the procedure, with the “realization that such terminology was beyond the scope of the first-year course”.

Examination of laboratory activities and evaluation materials for course success for alignment with learning goals revealed some disconnects.  These disconnects were addressed either through a re-working of the course materials or realignment of the grading scheme (e.g., the points assigned to notebooks were raised and point for the lab report lowered as this was deemed more appropriate for alter in the curriculum).

The development of the learning goals and the assessment ultimately also functions to evaluate the success of the course and not just the student. To ensure that students’ responses are “honest” and not motivated by getting points, the authors suggest carrying out these assessments as tasks outside the grading rubric for the course. To encourage “sincere effort” from the students, We have

found that providing a very small bonus point allowance for

completing the test, explicit knowledge that test results will be

used to improve the course, and a final question where students

indicate how seriously they regarded the test (shown below in

Figure 3) resulted in considerable levels of serious participation and adequate controls to minimize frivolous responses.

Here is what the authors have to say on the outcomes of the learning goals and assessment development process (verbatim):

“With data provided by the process and these course assessments, we have

·         Cultivated departmental consensus regarding what students should learn in the course

·         Provided students with clear descriptions of what they should learn (LGs)

·         Emphasized basic and transferable skill development (such as suitable notebook maintenance and correct volumetric glassware manipulation) through enhanced assessment of these skills

·         Better aligned course grading policies with identified LGs

·         Better aligned instructional materials and TA orientation with established LGs

·         Provided credible feedback regarding student attainment of LGs to target course improvements for optimizing instructional use of limited laboratory contact hours

A SUMMARY OF THE STEPS, TAKEN VERBATIM FROM THE ARTICLE:

The potentially challenging task of identifying learning goals or performance objectives for an introductory science laboratory course can be accomplished systematically and productively. To begin, resolve to deal with inherent challenges: start with current laboratory instructional materials, identify an overarching theme or focus for the course, and engage invested parties throughout the process. Next, build initial LGs around an existing framework.23 If possible, engage an experienced person in discipline-based educational research to help in LG synthesis, alignment with course goals, employing clear, operational language, and organizing the resulting LGs into the agreed-upon framework. With iteratively refined LGs identified, re-examine current laboratory materials to avoid duplicating existing assessments and prioritize remaining LGs to focus assessment-development efforts. Decide on the best way to assess particular learning goals or types of learning goals, compare this to assessments currently used in the course, and develop new assessments to fill the gaps. Validate these assessments with experts (to enhance consensus, construct validity, and face validity) and students (to ensure assessments are correctly interpreted and further support face validity). Administer the assessments using a pretest−posttest strategy to assess the impact of new goal-centered course improvements.

Finally, ensure that current student assessments are aligned with the LGs and make any needed adjustments.