What Do Conceptual Holes in Assessment Say about the Topics We Teach in General Chemistry?

What Do Conceptual Holes in Assessment Say about the Topics We Teach in General Chemistry?

Cynthia J. Luxford and Thomas A. Holme

Journal of Chemical Education Article

Publication Date (Web): March 11, 2015

10 pages

DOI: 10.1021/ed500889j

ABSTRACT: http://pubs.acs.org/doi/abs/10.1021/ed500889j

This article begins by pointing out that the General College Chemistry series(referred to in the article as an introductory class) serves as a service course to many other students who are not chemistry majors.  These are students in the life science, physics and engineering, and in the allied or professional health sciences.  Because of this service nature, the researchers thought it important to investigate whether the content covered in the General Chemistry courses meets the needs of life science and other science students.

This article looks at the coverage of concepts in General Chemistry based on an analysis of American Chemical Society Exams. The researchers believe that the nature of the development of these exams “allows each exam to be used as a historical artifact reflecting current topics taught in chemistry classrooms.” Exam content is developed by members of a committee based on their classroom experience and, therefore, the authors deem that the exam questions reflect the content domain coverage in a large fraction of general chemistry courses taught.  [As I was reading this article, I had to remind myself frequently that the authors are using chemistry exam questions as a proxy for information about content coverage in general chemistry.  It could be true that content items that do not garner or garner very few questions may not necessarily be ignored in the actual courses but rather simply not covered well enough for fair assessment.]

Each exam item from a database of the ACS General Chemistry full term, conceptual, first term, and second term exam questions was aligned with the anchoring concepts map.

Demographic data source indicate that only about 11% of students who took the ACS Gen Chem Exams between 2000 and 2012 were chemistry or chemical engineering majors with about 67% from other sciences, pre-med, nursing, or agriculture majors.

Analysis by the authors showed that exam question items were not uniformly quantitatively distributed across the 10 anchoring concepts as shown in the graph below:

It appears that there is a heavy emphasis, based on the number of aligned questions, on atoms, intermolecular forces, reactions, and energy. 

The author did a more nuanced analysis by looking at how the questions are distributed along the enduring understanding items for each anchoring concept.  The graphs for these are given in the article. Some notable findings:

I. ATOMS – Periodicity and electrons get the most coverage and ions the least. The authors also note that the majority of questions on atoms are found in the first term exam consistent with when this topic is usually heavily covered.

II. BONDING – Electrostatic forces, orbital overlap, and bond number are topics with the highest number of questions while metallic bonding and energy required for breaking bonds get very few.  The authors also note that bonding got the fewest number of questions overall out of the 10 anchoring concepts.  They also noted a concern regarding lack of coverage for energy for breaking bonds as this is a fundamental concept in understanding the role of ATP.

III. STRUCTURE AND FUNCTION – Predicting shape and atoms combining to form compounds are the most covered topics while the energy minimization model, symmetry, and chirality earned no coverage. The latter three are normally covered in more advanced courses like organic chemistry, at the earliest, but with more depth in physical chemistry.  Functional groups also don’t get much coverage but this is unsurprising to me as, in my case, while we cover a very brief survey of organic chemistry, it is normally toward the end of the second semester and really is typically the purview of organic chemistry.

IV. INTERMOLECULAR FORCES – The weaker nature of IMF relative to chemical bonds and their effect on physical properties get the most covered, followed closely by dipole categorization.  Noncovalent forces in large molecules are not addressed at all while relationship to reaction energy gets less than 5 questions. The lack of coverage of noncovalent forces in large molecules concerns the authors because, again this “hole” in the assessment may reflect lack of coverage of material important to those taking biology courses.

V. REACTIONS – This anchoring concept big idea gets the most coverage out of the 10 based on number of questions.  Conservation of matter and balancing equations and reaction categories get the most question coverage, followed by chemical change control using catalysts.  The following get no or very few questions: enduring understanding level items B (formation and breaking of bonds), C (particulate versus macroscopic level), E (periodic trends), and G (synthesis) (see Anchoring Concept map).

VI. ENERGY AND THERMODYNAMICS – Types of energy and chemical change get the biggest coverage followed closely by energy and electrostatic forces and entropy and energy distribution. Net energy change, energy requirement for chemical reactions, energy changes in nuclear reactions, using devices to harness energy, and bond breaking requiring energy get no or very few questions.

VII. KINETICS – The dependence of rates on concentration and temperature gets the most coverage.  The time scale variation of chemical reactions, reaction mechanisms, components of collision theory, and role of catalysts get about equal treatment at between 15-20 questions.  The influence on products by rate control gets very few questions, <5.

VIII. EQUILIBRIUM – Application of equilibrium concepts to pH, buffers, etc. gets the highest coverage followed by description of the equilibrium state and constant.  The reversibility of reactions gets no coverage and only very few questions are given on the implications of the size of the equilibrium constant and the net zero change that results from opposing processes occurring at the same rate.

IX. EXPERIMENT – Nomenclature gets the most coverage followed closely by quantitative observations of matter, consideration of the representativeness of samples, fidelity of inferences from data.  The need for experimental controls gets no coverage.  It is also surprising that safety gets only about 10 questions along with the measurement basis (mass, temperature, charge, etc) which gets less than 10.

X. VISUALIZATION – This anchoring concept gets the least attention in the exam coverage. Of the Enduring Understanding items, the application of theoretical models at the particulate level and empirical observations for macroscopic levels gets the highest coverage followed very closely by the use of graphs to visualize and interpret quantitative reasoning. Zero to few questions exist for the use of a mole and statistical methods to bridge particulate and macroscopic levels and the use of mathematical equations to visualize physical and chemical processes.

The authors provide a table of the enduring understanding items that get the least or no coverage in the ACS exam surveyed in Table 2.

In the conclusion section, the authors reiterate this analysis and interpretation depend on the premise that topic coverage in the ACS exams correlates with content coverage in the actual teaching of these courses. The authors suggest that the deficiencies in coverage of more biochemistry-related material may stem from most general chemistry instructors having an inorganic or physical chemistry background. They also do point out however that some of these missing questions may have been due to reliability and validity process of eliminating questions that have been tested. They repeat their concern about the lack of “coverage” of applications to large molecules important in biochemistry and are clearly advocating for its inclusion at this level.