Assessment update physical science

02 September 2007

The physical science outline is not in a form that lends itself to the aggregation approach I have used in MS 100 and MS 150. The present outline is 108 specific student learning outcomes - a broad, shallow river of many facts and few deep understandings. Worse, the course level outcomes, though only five, are oddly specific. For example, the only one of the five to reference chemistry is, "Give the importance of organic chemicals in everyday life."  Given that I have already predetermined that the course will not reach organic chemistry, the outcome is not going to be attained. And even if it were attained, it is rather tangential to the thrust of that part of the course (and the present text) which is introductory chemistry.

Thus in terms of test instruments I can only report on the results of item analysis. With week two lecture focused on linear motion and the week two laboratory also being focused on linear motion, quiz two borrowed lab data in its questions. The quiz noted that:

Science is built on experimentation. Science is a field in which a theory is not considered tested until experiments have been unable to prove the theory false. In lecture I present theories and we work homework problems. But the only thing you know for sure based on experimentation is that there is a relationship between distance and time for a ball rolling. And the 8:00 lab section should still have doubts about the nature of the relationship due to some of the data they collected.
The 8:00 science team reinvented the laboratory, allowing a subsequent team of scientists working at 11:00 to collect more data more efficiently. The following data [in this quiz comes from the revised experiment performed by the 11:00 team].

The results of the quiz item analysis were:

Src Q Description Corr Corr%
q02 1 calculate speed 25 81%
q02 1 units 21 68%
q02 1 significant digits correct 15 48%
q02 2 calculate speed 24 77%
q02 2 units 23 74%
q02 2 significant digits correct 20 65%
q02 3 draw a conclusion from quantitative data 19 61%
q02 4 identify units for slope 10 32%
q02 5 identify units for intercept 1 3%
q02 6 interpret physical meaning slope 6 19%
q02 7 predict d given t 8 26%
q02 7 units 5 16%
q02 7 significant digits correct 7 23%
q02 8 predict t given d 7 23%
q02 8 units 5 16%
q02 8 significant digits correct 7 23%

Note that students can often make basic calculations and remember to include the units, but few understand the matter of significant digits. The 65% correct rate for question two is superfluous - the correct number of significant digits was two decimal places, the default rule the students use on every problem. Problems 7 and 8 where two decimal places was not correct reveal far lower rates of success in this area. Students also stumbled badly on determining the correct units. 

Questions one and two involved simply division of two numbers to get an answer, seven and eight required using a first degree equation with a non-zero intercept.

The first laboratory was marked using a rubric revised from that proposed in the course syllabus. The rubric borrows from existing rubrics found on line, the COMET rubric, and modifications I made as I read the first laboratory. The points are in the first column.

1. Question/Problem statement

Given. Not scored.
2. Hypothesis/Prediction
2 Clearly stated with any associations among the variables described, complete
1 Unclear statement, or variables left unaccounted for, incomplete
0 Omitted
3. Procedure (not marked in lab 01 due to its being given in the lab)
2 Sufficient detail such that lab could be reproduced/duplicated. All activities listed.
1 Insufficient detail, lab could not easily be reproduced, unclear, missing activities
0 Omitted
4. Data tables
3 Clear and complete, laid out clearly with labels and units
2 Missing one or more labels or units or other minor issues
1 Unclear, confusing, poorly done, incomplete
0 Omitted
5. Data analysis
2 Appropriate statistics calculated and/or appropriate data analysis including units
1 Missing units or other minor issues
0 Omitted
6. Data display: graphs and charts. None in lab 01
2 Complete
1 Incomplete
0 Omitted
7. Conclusions
4 Strong reasoning based on the data, logically reasoned, complete and thoughtful
3 Moderately well-reasoned, but tangential to the data or based on misconceptions
2 Overly brief or compromised and impaired by grammar and syntax errors
1 Unclear, not well reasoned, highly incomplete
0 Omitted
Grammar and Syntax
3 No errors of grammar or word order.
2 Some errors of grammar or word order but communication not impaired.
1 Frequent errors
0 Errors of grammar or word order so severe as to make comprehension virtually impossible.
3 Appropriate terms used consistently, clear command of vocabulary with a focus on correct usage of physical science vocabulary, no misspelled words.
2 Occasionally uses inappropriate terms or relies on circumlocution; expression of ideas not impaired; or a few misspelled words.
1 Frequent errors in vocabulary or spelling, ability to communicate limited by vocabulary
0 Vocabulary limitations so extreme as to make comprehension virtually impossible.
3 Ideas flow logically. Connector words assist the reader.
2 Ideas are disconnected and the conclusion reads more like an outline or answers to a list of questions without connector words. There is a choppy and disjoint sense to the writing style.
1 Communication impaired by a hodgepodge of inappropriate and misused cohesive structures
0 Incomprehensible collection of disconnected ideas and words.

[This was subsequently modified such that the tables subsection is also marked on a two point scale. Making this symmetric made using the rubric easier.]

Each individual lab includes the individual scores for each section of the rubric. Coupled with the rubric, the hope is that the scoring will help a student understand where their specific weaknesses are and then address those weaknesses. Explaining this to the students and helping them understand how to improve will be a process that will take time during the early part of this term.

Although I could record in a large grid each student's individual scores in each section, which would lead to an item analysis-like assessment of student performance on each of the ten sections, the amount of work that would entail is simply prohibitive. This early in the term this seems more work than the value to be gained from the information generated.

Some students are submitting their laboratories by email, providing the opportunity for me to gather sample laboratories for use in "course portfolio" assessment. i would note for the benefit of the other science instructors that this approach does not utilize a "lab notebook" of any sort in which students collect all their labs. Each lab report is an individual Microsoft Word document often using Microsoft Excel data tables and charts.

At present the conclusion "essays" for most students have a ways to go before they meet the writing expectations I have for the laboratories.

I would also note that I do not know if the above rubric based system is more work than whatever systems of lab marking other instructors in the division are using.  I was able to mark about 20 labs in a period of about two to three hours, I may get more efficient in future runs of the rubric. I also write comments on the labs and note grammar/vocabulary issues which adds to the marking time. In some sense I've given myself the load of a writing class instructor for what is the equivalent of a writing section and a half.

The course has a homepage, quizzes, tests, labs, and other material will be linked to that page. Many of the materials are in formats.