SC 130 Physical Science assessment report and grid

15 Dec 2009

College mandated assessment report format

Review of performance: SC 130 Physical science fall 2009. 32 students enrolled in course. Submitted by Dana Lee Ling

For the first three outcomes, student counts based on number who passed the class. For subsequent outcomes, student counts based on submission of laboratory reports for that outcome.

SC 130 Physical science assessment

Some files linked from this assessment grid use XHTML+ MathMl + SVG and require the use of browsers such as FireFox which can render these technologies. Sample laboratory reports are in Adobe Acrobat (.pdf) format.

Program Learning Outcomes

The relevant program learning outcomes are from the general education core.

• Define and explain the concepts, principles, and theories of a field of science.
• Perform experiments that gather scientific information and to utilize, interpret, and explain the results of experiments and field work in a field of science

Course Learning Outcomes

Learning themes cut across all activities and explorations in the course. These learning themes are the first three items on the course outline. The first theme centers the course on exploration and science as a process as expressed by the scientific method.

The second theme places mathematics at the core of the course. Many of the laboratories lead to linear relationships, linear regressions, with the attendant concepts of slope and intercept.

The third theme brings writing across the curriculum into the course. In addition to being marked for content, the laboratories are marked for grammar, syntax, vocabulary, spelling, format, cohesion, and organization.

The laboratories and the laboratory reports are the primary evidence that students explored physical science systems using scientific methodologies, that they generated mathematical models, and that they communicated their results.

Students will be able to...

Specific learning is centered on the laboratory experiences. Laboratory reports cited above are considered a primary measure of student performance in the course. Through the weekly laboratory reports, the course integrates writing into the core of the course curriculum.

Documenting actual activity during the laboratories is difficult. The laboratories are intended and designed to engender cooperative learning in scientific teams of exploration. One of the design intents is that the acquisition of scientific knowledge is a journey, not a destination. Science is not about a set of accumulated and memorized facts. Science is a process of discovery, careful thought and analysis. Science is about finding and testing explanations for systems, in physical science those explanations are typically mathematical models.

Tests can document acquired facts. Documenting the journey, as opposed to the acquired facts, is difficult. The table further below uses links to photo documentation as indirect evidence of science as an exploration.

The final includes single numeric problems typifying that area of study. Each final utilized a spreadsheet to randomly generate data values. A mail merge was used to produce individually unique final examinations for each and every student. The merge also generated an answer sheet for each student final, these were separated from the final and used to mark each unique exam paper.

The results of an item analysis for the spring 2008, fall 2008, spring 2009, and fall 2009 final are in the table further below. The item analysis is the percent of students answering the question in that area correctly.

Specific Learning			Final item analysis
Outcomes	Laboratory	Photo documentation	Sp 2008 n = 29	Fall 2008 n = 28	Sp 2009 n = 31	Sp 2009 n = 32
Explore dynamics of motion including performing calculations of velocity, acceleration, momentum, and kinetic energy, generating appropriate mathematical models such as linear regressions, making calculations of the conservation of momentum and energy	Linear	Rolling balls	0.62	0.89	0.61	0.56
	Acceleration	Falling balls	0.62	0.39	0.45	0.03
	Momentum	Marbles	0.59	0.57	0.48	0.31
Experiment with and determine the heat and electrical conductivity of materials	Heat	Conduction	0.90	0.86	0.81	0.91
Determine latitude, longitude, and find the mathematical relationship with standard linear measures; determine universal time	Lat Long	Lat Long	0.17	0.36	0.23	0.75
Observe and identify clouds, be able to describe precipitation processes in Micronesia such as collision-coalescence, Bergeron, and orographic precipitation; list the phenomenon associated with El Niño and La Niña	Clouds	Cloud formation and shape	0.48	0.79	0.61	0.22
Determine the speed of sound and perform experiments with sound	Sound	Echoes	0.21	0.14	0.48	0.22
Explore reflection and refraction, determining the mathematical relationships for reflected image depths, angles, and refracted image angles	Optics	Optics	0.72	0.36	0.65	0.53
List the primary and secondary colors of light, generate other colors from primary colors, explore systems of specifying colors	Colors of light	Spectra	0.34	0.36	0.29	0.61
Develop a mathematical model using measurements of current versus voltage across a resistance; determine and sketch open, short, and closed circuits	Electricity	Circuits	0.72	0.64	0.71	0.50
Determine whether substances are acids or bases using locally available pH indicator solutions	Chemistry	Acids and bases	0.52	0.86	0.61	0.72
Average:			0.54	0.57	0.55	0.49

Comments and notes fall 2009

The core of the course are the activities and laboratories. The laboratories involve a write-up using spreadsheet and word processing software. The laboratories are marked using a rubric. The course focuses on physical science as a process and method, an exploration in search of mathematical models of system behavior. The final examination is not a well aligned measure of process, method, and exploration. The final examination fall 2009 was a set of sixteen questions, one per laboratory, usually centered on the central mathematical relationship of the corresponding laboratory. As such, the final is an exercise both in remembered knowledge and calculations.

Fall 2009 the laboratories ranged from 20 to 60 points with an average of 41 points. With quizzes, tests, homework, and attendance, the course generated 909 points. The final was worth 24 points, less than 3% of the overall mark. While the final could be weighted, the true focus of the course are the laboratories. Laboratory 14 is a better measure of the achievement of outcomes than the final examination. The students are aware that the final has little impact on their grade, and this is reflected in the performance seen in the table above.

As an additional confounding variable I changed both the pretest review and the content of the final examination. This term I deleted all of the formulas from the final examination. To prepare the students for the greater memorization load that this represented, I ran off a randomly generated copy of the final examination and handed it out on the last day of class. Thus the students knew exactly what questions would be on the final examination, and what formulas they would need to have memorized, only the numeric values would remain unknown.

One student told me after the test, "I did not study, no need." The students knew the final could not make a significance difference. The combination of the increased memorization component and the comprehension that the final could not move their grade significantly apparently underlie the drop in performance on the final examination.

Although I do not intend to shift the focus of the course from the laboratories, I am rethinking how the final might be better handled spring term 2010. As memorization has never been an intent of the course, I will be adding a sheet of formulas on a separate sheet of paper. The sheet will have all formulas from the term, thus students will still have to know which formulas to use. I will also have to play down the relatively low impact of the final examination.

The item analysis success rate is related to the complexity of the problem. Questions involving multiple calculation steps had lower rates of success. The table includes success rates for the spring 2008, fall 2008, and spring 2009 terms on the final examination based on question difficulty.

^a Based on in-term item analysis of inference questions on quizzes and tests.
^b Based on in-term item analysis of inference questions on quizzes and tests.
^c Based on question six of the final examination.

Fact based recall rebounded from the prior term, but this may also reflect the impact of students being more particularly aware of what exactly would be on the final examination.

The ability to calculate results involving a remembered formula and a single calculation has remained remarkably stable term-on-term. Of interest is that the student's had to produce the correct formula from memory, thus this question includes the lower level of memorized fact.

Complex calculations, while difficult for the students, have shown strong variabilty in percentage of success rate across terms.

Comment on linear regressions

Spring 2008 the laboratories were rewritten to focus on mathematical relationships, specifically linear relationships. Students are introduced to using spreadsheet software in laboratories one and two to assist them in making xy scatter graphs and in finding the slope and intercept. The course requires access to a computer laboratory for the second half of the first two laboratories during a term in order to present this material to the students.

One might think that given the placement of linear relations at the core of the course, the course should have a math pre-requisite. The lack of a math pre-requisite is intentional. The course does not presume the student has anything more than high school level contact with algebra one material. The course not only undertakes to teach students to run linear regressions, the instructor presumes that using a computer to find slopes and intercepts is wholly new material for the students. The course intentionally seeks to introduce the algebra of linear equations to the students through the vehicle of physical science.

Analysis of mechanics of writing

Each laboratory is marked for grammar, vocabulary, organization, and cohesion. A rubric is used to mark these areas. The rubric is similar to the rubric used by the college to mark entrance test essays. Modifications include the addition of spelling to the vocabulary section and the organization section being based on the laboratory including all required sections.

Fall 2009 the scores for grammar, vocabulary, organization, and cohesion were compared for laboratory two and laboratory twelve. There was no statistically significant gain seen in any of the four areas. The gain in vocabulary, however, came closest to statistical significance with a p-value of 0.06. The complication is that the students write fairly well as measured by that rubric. Admission to the college usually means that a student can score a four or five on the essay rubric. Put another way, there simply is not a lot of upside room on that rubric.

Although I have the sense that writing ability improves over the term, I cannot as yet document that. An OpenOffice.org data sheet with the original data is available.

Liked and disliked laboratories

A survey (results) was run fall 2009 to determine the most liked and most disliked laboratories. The students were also asked why they liked or disliked that particular laboratory.

Laboratory 11 was the most liked laboratory. The lead reason for the popularity of the laboratory is that it was perceived as being easy and interesting. Also gaining votes as the most liked laboratories were laboratory two and laboratory seven.

The most disliked laboratory was laboratory nine. The lead complaint was that the laboratory was physically difficult. The lab required walking across campus, as some students noted, under the meltingly hot sun. Laboratory seven, although liked overall, was disliked by some students for the same reasons as nine. Other reasons cited for disliking particular laboratories included confusion, difficulty, and complicated procedures.

Laboratory eight was the most divisive in the voting, with six students picking the lab as the number one most detested laboratory and six other students choosing eight as their all time favorite laboratory. I had known that eight was hated by some students, but the realization that equal number enjoy the laboratory was a surprise. Every term I look at replacing eight, and each term I have backed off and run eight in its original form. Images from laboratory eight show engaged students focusing on the task.

Post-script: Final examination item analysis details

Details on the final examination item analysis fall 2009. Number of students answering a particular question correctly. Fact refers to the question being a recall of a fact, memorized knowledge. Single refers to a single calculation required to determine the correct answer. Complex refers to a chain of two or more calculations required to obtain a correct answer.

Question three used to include the formula, a complex formula that includes a square. The absence of the formula led to a complete collapse in students ability to answer the question. Question six was entirely new and covered different material than prior final examinations. Question seven was rewritten to focus on calculating an average and was less confusing than in prior terms. Question eight was based on a cloud quiz given on the last day of class. Question ten was changed to a more difficult calculation and performance fell twelve percent term-on-term.

Physci current term • Physci • Assessment • Courses • COMFSM