In order to understand the potential usefulness of a science program learning outcomes evaluation instrument, a pilot test was conducted on a non-random sample of students.
This run of the instrument was purely an experimental trial run. The instrument is unusual in its nature and structure, the instrument is nothing related to a standard multiple choice fact and content oriented test. On the contrary the instrument seeks to explore the students thinking about phenomenon and asks the students to engage in the process of science. The first part asks the students to observe a demonstration and explain why an event occurred. The second part has the students performing a simple experiment.
The result was that students have difficulty using the concepts of science to explain events, but are fairly competent at taking data, filling in tables, and making graphs. The students are evenly split on making a simply calculation, more advanced calculations that require a chain of calculations and understanding completely elude the students.
The 21 students were members of the SC/SS 115 Ethnobotany class spring 2004. The students are a mix of freshmen and sophomores from a broad variety of majors. The students were permitted as much time as they needed to perform the experiment in part two, but the whole instrument took less than thirty minutes for the students to complete.
Due to equipment limitations, students worked in pairs with one group working at a triple. Thus the instrument says nothing about individual abilities.
The concept of having students work in pairs should not necessarily be automatically discarded if more equipment were made available. Twenty-first century science is done in teams. The team results still inform the division as to what students, in the plural, take away from our programs. Science laboratories are done in pairs, surgeons work as a member of a team, scientists send papers for review to trusted colleagues, researchers function as members of inter-institutional teams. Collaboration, cooperation, and communication are crucial to successful science.
Students will be able to...
There was no design intent to evaluate each and every outcome, nor to comprehensively evaluate all the facets of each outcome.
The evaluation instrument consisted of two parts. The evaluation instrument can be viewed at http://www.comfsm.fm/~dleeling/physics/progoutscience.html Understanding these comments virtually requiers having this document in hand as this report is read.
In part one a drinking glass was filled with water and a piece cut from a file folder was placed on top of the full glass. The class was asked to predict what would happen when the instructor held the paper, turned the glass over, and then let go of the paper. Students answered this section as individuals.
The event was designed to act as a discrepant event for the students. The system behaves in a manner that surprises those who have not seen this demonstration. The intent was to elicit from the students why this system behaves as it does. Would any student hit upon the correct explanation? What various incorrect concepts would the students fall back upon in this novel situation?
This section was intended to have a look at program learning outcome six above: would the students explain observations of new phenomenon using the theories of science?
In part two the students were asked to perform a very quick and simple experiment. The intent was to have students perform an experiment (3), gather data (2), utilize appropriate procedures (4), express results of the experiment and make predictions from their data (5), and then show a knowledge of some of the fundamental concepts of science (1). A single experiment of limited scope can only provide a glimpse into the students scientific capabilities, but a worthy glimpse.
Of 21 students, only five predicted that the water would remain in the glass. Sixteen predicted the water would fall out. This is not unusual, the event is generally considered counter-intuitive, a discrepant event. Tik K. Liem has authored a series of science laboratory guides that build curriculum around discrepant events. These events challenge a students world view and often force students away from stock, memorized answers and solutions.
Of 21 students, no student hit upon the correct answer. The following table lists in abbreviated form the students answers:
| Why |
|---|
| because of the kind of paper |
| card sticky when wet |
| force and pressure |
| force of water sucks up paper |
| I don't know |
| no air in water |
| no air in water |
| no air pressure in cup |
| no air pressure in cup |
| no energy in the cup |
| paper is getting wet |
| paper is strong w/ a force |
| paper sealed by moisture bond |
| paper sticky |
| pressure in cup suck |
| pressure in the cup hold cardboard |
| pressure in the cuphold cardboard |
| pressure inside suck |
| water absorb paper |
| water pressure pull on paper |
| water stick paper |
A number of students, at least six, relied on a "suction" theory. Unfortunately "suction," like "centrifugal force" is a phantom force. Despite all of our experience with vacuum cleaners, nothing actually "sucks." The energy and force are, on the contrary, on the other side: an area of high pressure attempts to move towards or press towards an area of lower pressure. A vacuum cleaner motor removes air from a canister, reducing the air pressure. The outside air then seeks to find a way into the canister. The force, the pressure, is from the outside in, a push. Not a suctional pull.
Four students believed that lack of air in the glass was critical. Actually, the experiment is rather tolerant of air in the glass. In a future run of the demonstration there might be information profit in running the demonstration again with air in the glass and going for a second round of predictions.
Some will interpret the failure of the students to hit upon the correct explanation as a call for teaching the concepts surrounding pressure better. On the contrary, look past the specifics of this demonstration. The students relied on previously held concepts or concepts made up at the moment of observation. The key result is that no student fell back upon learned knowledge. Not all students had been in physical science or chemistry, but some students had been in each.
The meta-failure is that our students do not think scientifically or analytically. This is not unusual, studies in the United States have found the same result. I have seen students with degrees in engineering who fall back on non-scientific explanations when faced with new and puzzling phenomenon. This is has been linked by some to the willingness of Americans educated in science in the schools to also believe in astrology and pseudo-science.
Students showed a consistent ability to gather data in a simple experiment, plot that data, and make basic inferences from their data. All 21 students gathered essentially correct data. Seventeen students charted the points and put a line through their data. Fourteen students were able to correctly look up a y-value (bounce height) from a given x-value (drop height) using their line. Ten students were able to perform the inverse operation: correctly look up an x-value (drop height given a y-value (bounce height).
This skill, of reading a graph and interpreting the result, is a basic inferential skill. With success rates of 67% and 48%, student success on this basic skill is similar to overall performance rates in mathematics – an underlying skill to graph reading. Overall success rates for mathematics program learning outcomes hover around 55%.
Twelve of 21 students were able to calculate the coefficient of rebound. This shows a slightly better than half capacity at following a set of directions that ask to derive a value from experimental data. It should be noted that stability of performance on such a measure year-to-year may be complicated by the small sample size and the student sample. The class is overrepresented in marine science and HCOP students with respect to the general student population. This may skew the year-to-year results were the class to see a more representative student sample.
When asked to use the coefficient of rebound to try to determine the bounce height for a drop of 678 centimeters only a single student succeeded. The student was not, as might be anticipated, a HCOP, marine science, or business major (who showed high rates of success in mathematics). The single student was a 54 year old female of unstated major who notes that she has been out of school for years.
In the final section the students were asked what type of energy the ball possessed when held aloft just prior to being dropped, what type of energy the ball possessed just prior to hitting the floor, the name of the theory that suggests the energy aloft should be equal to energy with which the ball hits the floor, and why doesn't the ball bounce as high as it is dropped. The answers in raw form are instructive.
| sq | energy aloft | energy bottom | theory name | why not as high? |
|---|---|---|---|---|
| 1 | kinetic | gravitational energy | Newton's first law | amount of fource that was put on the ball equals to the amount it was dropped. |
| 2 | kinetic | gravitational energy | Newton's first law | amount of fource that was put on the ball equals to the amount it was dropped. |
| 3 | kinetic | gravitational energy | ||
| 4 | kinetic | gravitational force | Newton's theory | |
| 5 | kinetic | gravitational force | energy between height and drop | |
| 6 | kinetic | gravitational force | ||
| 7 | kinetic | potential | Newton's theory | |
| 8 | kinetic | potential | gravity | |
| 9 | kinetic | potential | gravity | |
| 10 | magnetic | electromagnetic | plastic not have any space in it | gravity push it down |
| 11 | mass | velocity | relativity | In every change of energy some is lost |
| 12 | potential | kinetic | accelerated | gravity |
| 13 | potential | kinetic | accelerated | gravity |
| 14 | potential | kinetic | force | kinetic and potential energy |
| it has more height | ||||
| gravity |
Nine students mistakenly thought the energy aloft was kinetic energy, only three correctly identified that the ball possessed potential energy when aloft. Only the same three correctly noted that the potential energy had become kinetic energy a the bottom of the ball fall. Note that none of these three then went on to correctly explain why the ball does not return to its drop height. Only number eleven above correctly reasoned the loss of energy at interchange points – energy is conserved, but that conservation include energy converted to heat and sound.
More disturbing is that not a single students could name the theory of the conservation of energy, one of the most fundamental guiding theories of the universe and everything. Conservation of energy, mass, linear momentum, and angular momentum are bedrocks of Newtonian physics. Dozens of problems are solved only with the understanding of these principles.
In general our students have very poorly developed understandings of the concepts of science. The vocabulary and language of science does not come easily to our students, and they are certainly confused about the difference between kinetic and potential energy.
As a first look at the pilot instrument the trial has been successful. The instrument did yield insight into what students can do and what they know even in the relatively unstructured environment of an open answer evaluation. The lack of multiple choice answers is necessary in order to not constrain the students thinking with limited answers or to prompt memories of encountered ideas. The open answer instrument provides a better idea of what students can do or what they know as a part of who they are as they walk through life. No student studied for this evaluation, the evaluation was a surprise to the students.
The students solutions will be retained for future reference. The recommendation would be to run the same instrument spring 2005, although year-to-year comparisons should be meaningless given the non-random nature of the sample.
The students undoubtedly leave our institution with basic charting and chart reading skills, plus some basic computational skills. Obviously these students would make highly confused and uncertain science teachers, but then few of them are heading directly into a classroom.