[assessment] MS 150, SC 130, SC/SS 115 update

30 September 2007

MS 150 Statistics

A difficult quiz on probability which included drawing conclusions from data saw the overall program level learning decline on aggregate. While students continued to do well on simple calculations, problems that involved analyzing data in a table and making inferences proved difficult for the students. Based on what I've come to understand of the student's comprehension of written English in SC 130 Physical Science, I am confident that the complexity of the some of the questions caused some confusion.

A question such as, "Consider just the second data row, the data on those who chew with tobacco. For those who do chew, what is the mode?" asked students to examine a data table and determine a mode for a subset of that data. Although this topic had been broached in class, the question itself requires synthesizing information from two sentences. The linguistic complexity is a very real barrier for many of our students. Only two of forty-nine students answered this question correctly. The quiz is available online.

The item analysis for the quiz is as follows, with the course level student learning outcomes column derived from extant outline. These course level student learning outcomes are the same outcomes that were specified on the now withdrawn proposed outline.

Q Description l Outref SLO Corr Corr%
1 probability 1 1 Calculate basic statistics 36 73%
2 level of measure 1 1 Calculate basic statistics 14 29%
3 mode 1 1 Calculate basic statistics 3 6%
4 probability 1 1 Calculate basic statistics 25 51%
5 mode 1 1 Calculate basic statistics 3 6%
6 mode 1 1 Calculate basic statistics 2 4%
7 inference from statistics 1 1 Calculate basic statistics 4 8%
8 mean 1 1 Calculate basic statistics 18 37%

Table one: Item analysis

Aggregating all item analysis to date yields the following course level performance:

Outref Students will be able to: Sum Count Avg
1 Calculate basic statistics 23.76 48 50%
2 Represent data sets using charts and histograms 2.96 8 37%
3 Solve problems using normal curve and t-statistic distributions including confidence intervals for means and hypothesis testing 0 0 0%
4 Determine and interpret p-values 0 0 0%
5 Perform a linear regression and make inferences based on the results 16.3 25 65%
PSLO define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas.


Table two: Performance on course and program level learning outcomes (withdrawn outline)

Aggregated to the program level, course level performance has dropped slightly.


Given my decision to withdraw a proposed outline for MS 150, I thought I ought to report how the item analysis aggregates against the present thirty-nine outcome outline. This proved difficult due to the number of outcomes. Due to technical issues, the result is not a true one-to-one mapping to the extant outline. Table two above would become:

Description Percent correct
bin width 66%
calculate a p-value 0%
calculate error tolerance 0%
calculate percentage 46%
calculate t-critical values 0%
calculate z-critical values 0%
calculate z-score 16%
coef var 12%
coefficient of determination 90%
completed homework 70%
correlation r 97%
determine a confidence interval using normal curve 0%
determine a confidence interval using t-distribution 0%
determine max conf interval from p-value 0%
direct variation calculation 53%
frequency table 35%
histogram chart 31%
hypothesis testing with confidence interval; 0%
inference from statistics 16%
intercept 64%
level of measure 32%
linear, non-linear, or random 79%
max 87%
mean 41%
median 59%
midrange 53%
min 91%
mode 21%
normal curve characteristics 0%
perc var x that explains var y 47%
perform percent multiplication 94%
point estimates for mean 0%
positive or negative relation 99%
predict x given y 38%
predict y given x 44%
probability 56%
range 59%
sample size 37%
shape of distribution 16%
slope 66%
slope-intercept eqn 2%
standard dev 48%
strength of relationship 85%
three ways to deter. prob. 13%
z ordinary/extraordinary 10%
Overall average achievement (pSLO?) 39%

Table three: Performance on course and program learning outcomes extant outline

Note that the difference in the aggregated program learning outcome performance of nine percent is due in part to the two mappings not being identical (this is the issue of table three not being a true one-to-one mapping back to the extant outline).

SC 130 Physical Science

This past week the lecture focused on time and place. The quiz also focused on time and place, with some interesting results. A traditional physical science quiz on time and place would tend to focus on having the students answer specific questions of fact. This quiz sought to examine how the students know something and how they think they should know that something. So the quiz started with the odd question, "What time is it?" From the student's perspective this is an odd question. Students tend to focus on getting the "right answer" but this question has no single right answer.

The second question was "Is that the right time?" Here I am asking the student to engage in metacognition, to engage in thinking about their thinking in the form of thinking about whether their answer is correct. The lecture section should have prepared the thoughtful student to answer no if they used a local time source such as their watch or the wall clock.  A couple students knew that Telecom was synchronized to a global time standard and used their cell phones to obtain the time. In this case the answer would be yes, their time is the "right" time for planet earth. Yes, students in the class are encouraged to use technologies including their cell phone, provided that usage is appropriate to the task at hand in the class.

Question three asked, "How do you know?" This question was the one that let me know if a student had checked telecom. One thing that I learned along with the class is that the time on a cell phone is not necessarily the correct time. Many cell phones have internal clocks that are set by the user. To get the "Telecom" time one texts to get their balance and that the returning message includes a time stamp.

The quiz continued in this vein of asking a fact and then asking how the student knows that fact. This proved very illuminating. A student might get a fact right but for the wrong reason. Individual quizzes showed clearly that students were able to parrot back a fact with either no understanding of how that fact is actually obtained or with a faulty understanding.

This meant that the right answers to the traditional "just the fact" questions at the end of the quiz cannot be taken as proof of learning. To the question "What causes the seasons on the planet earth," 22% of the students correctly answered that the tilt of the earth is responsible for the seasons. Yet some of those 22% are likely just parroting back a fact without true understanding.

This disjunction between memorized content and the actual model of a system being held in the brain of the student is made most clear in the recent laboratory on heat conduction where many student's explanations of the phenomenon is confused at best.

The distinction between memorized content and conceptual understanding is clearest to me in the series of the following series of questions:

5 Where are you right now in terms of latitude and longitude? 7 26%
6 Is that the right latitude and longitude? 8 30%
7 How do you know? 12 44%
8 How can you determine the right latitude and longitude? 16 59%

Only 26% of the students knew their latitude and longitude.  Bear in mind that this was covered on two separate occasions, Monday and Wednesday, by going outside with a GPS, determining our latitude and longitude, and writing this on the white board. I was only looking for answers to the nearest degree, not the nearest minute. Yet only 26% could answer that we are at 6º North latitude and 158º East longitude. However, 59% could tell me HOW to determine the correct latitude and longitude. For that question I allowed answers that referred to the GPS unit and answers that referred to using technologies such as Google Earth. The students do better on how to get an answer than the answer itself, and that is for me part and parcel of the impact of deep learning versus surface learning. My hope is that the students will learn how to learn. Yes, content is important, but it is also ultimately what is most forgotten.

The content level is always a source of some humor on any science quiz, and SC 130 is no exception. There is some content I do want my students to know. Like the number of days in the year. Three people missed this question, citing 356 days (obvious digit reversal), 163 days, and 135 days. That any college student would not know how long the year is in days has to be indicative of the lack of contextual knowledge our students bring to any science course.

Other interesting student theories included that a solar day is "A day that changes things around and it is also a day of partying and enjoying." Actually, a solar day is 24 hours, noon to noon technically.

Many of my labs use the simplest of equipment. Yes, there is some fancy physics equipment in the laboratory. I always fear, however, that students will think that the physical law being illustrated holds only for some fancy piece of special equipment. Science as magic. For heat labs what I really want is an uhm, but the campus lacks one of these. Students really understand an uhm. As an example of technology as magic, one student noted, "Using the GPS, you face out and let the solar system out in space to give you the approximate latitude and longitude of where you are." 

One student clearly lives in a very uncertain pre-Copernican world noting that the cause of the seasons is "the sun is orbiting the earth. Sometimes the sun rise east and sometimes it set west." Only sometimes. Not only is science magic, their whole daily world runs on a sometimes, a maybe the sun will rise in the east. Of course, with the sun orbiting the earth, lots of things are possible.

The item analysis for all items on the quiz is as follows:

Q Description Corr Corr%
1 what time is it? 26 96%
2 Is that the right time? 20 74%
3 How do you know? 18 67%
4 How can you determine the right time? 18 67%
5 Where are you right now in terms of latitude and longitude? 7 26%
6 Is that the right latitude and longitude? 8 30%
7 How do you know? 12 44%
8 How can you determine the right latitude and longitude? 16 59%
9 What is the latitude and longitude of your home island? 9 33%
10 How many days in a year? 24 89%
11 What is a solar day? 13 48%
12 What causes the seasons on the planet earth? 6 22%

Laboratory five was turned in this past week, the heat conduction laboratory. Images from this laboratory can be seen at lab05photos.html

Overall performance on the laboratory reports continues to rise, with the average on the rubric rising to 80% at present. The students have, by and large, mastered the format I have specified. Students did very well on the basic analysis of what happened in the heat conduction laboratory. Their tables and charts were generally spot on. Their conclusions, however, where they tried to explain what happened, betrayed a mix of many different mental models. I do not try to correct all of these models as that would be simply my word against their deeply held understanding. I could opt to design experiments that knock down the incorrect explanations, but we would spend a significant part of the term on one single concept (heat conduction in this case). I may one day decide that is what should be done, but that would lead to very deep and very narrow learning and even I am not ready to go there quite yet.

On the writing side grammar issues that are probably connected to L1 remain problematic. Students simply have a very hard time with subject-verb agreement between the singular and the plural. For example, the constructions "Copper conducts heat" and "The metals conduct heat." The L1s which I am familiar with do not have have singular and plural verb constructions, so this remains an area of difficulty for the students. I am not an English instructor, so my approach may be wrong, but I often try to shake the students loose from their ideas by saying that "s" is not plural, it is simply required in all present tense sentences. For the singular "s" attaches to the verb, for the plural "s" attaches to the noun. This is of course not universally true, but it is a starting point.

The other advise I will be giving to the students is to simplify their sentences. Stick to noun-verb-noun sentences. Avoid commas. Avoid conjunctions. Too often the student seems to get lost in a sentence, "The reason why the convection [sic] of heat starts from the hot water to the cool water is because the hot water is bigger than the cool water but, if the hot water is smaller than the cool water, we have to put the thermometer in the hot water."

Bear in mind that the student may be referring to either the amount of water or the temperature of the water when the student says "bigger" and "smaller." As to why the thermometer would be put in the hot water I remain unclear as the lab procedure did not call for monitoring of the hot reservoir.

I have to admit to enjoying the explanation by one student, "The heat is transfer by conduction of the hot water in the foam cup through the two copper wires to the other foam cup." The copper wires were solid six gage copper wire, a thick solid cylinder of copper used in electrical systems.

While marking the laboratories remains a significant effort each and every weekend, I find I am gaining too much insight into the abilities and thinking of my students to give up this approach. A fill-in the blank recipe approach would mark quickly and easily, but would be as meaningful as the dry facts spit back on a test. As one alumni said to me, "I do not remember what we did or learned, but I remember filling in the blanks in the laboratory." The alumni noted that they did well in the course, they put the right numbers in the right blanks, but they never really understood what the numbers meant.

The laboratories are also a lot of work for the students - a paper a week. The turn-in rate for laboratory five was 83% of the students who participated in the laboratory. There was one absence on lab day, there is also now apparently a withdrawal from the course by another student. The laboratories continue to be marked for grammar, vocabulary, organization and cohesion. On lab five 50% of the 24 points were on content items in the rubric and 50% were on grammar, vocabulary, organization, and cohesion. Writing remains and will remain a crucial component of the laboratories.

SC/SS 115 Ethnobotany

On Tuesday the class visited the Pwunso botanic garden to observe and learn about both gymnosperms and economic botany. Images from this field trip can be seen at http://wwww.comfsm.fm/~dleeling/ethnobotany/e73/botanic_garden.html

The trip went well, and the students were surprised at the variety of trees in the garden. They are also always surprised at my penchant for grabbing leaves and berries and sticking them in my mouth. I always have fun with the black pepper. Our black pepper is very very hot. The fresh peppercorns are on the order of the small cayenne peppers in terms of heat. And yet many students do not even recognize that the plant is black pepper. I love really hot peppers and kim chi base is like ketchup to me, so I have no problem chewing a fresh peppercorn. The flavor is wonderful. And each term a couple students get curious and want to try a pepper corn. I warn them not to, that the berry is hot, but they almost always insist on trying the berry. The look on their faces is always priceless. They will not soon forget that Pohnpei's black pepper berries are uniquely hot, no memorization required.

The tour now ends at the Calophyllum inophyllum tree. This tree is found on both the high islands and the atolls alike. A student from Pollap stood there in front of the tree, looked down at the muddy ground, and exclaimed, "Rekich cannot grow here... " then her voice trailed off as another student began to giggle. "It is an atoll tree!" she said turning towards the giggling student. I reassured her that the tree was not growing there and that it is only ever found on atolls, never on Pohnpei.

I then turned to the Pohnpei students and contradicted myself by asking them the name of the tree. A discussion ensued, with two students familiar with the tree but unable to recall the name. The Chuukese students all knew the name of the tree - a reversal of the usual situation in the class. The Kosraen students had no idea whether the tree was found on Kosrae or not, although one student thought it might be found on the island. I then gave the name of the tree in Pohnpeian and Kosraen (isou and ituc). A Pohnpeian student asked me to repeat it, and I did. The oldest Pohnpeian student at this point said, yes, that's right, that's the name. At this point a student further away came up to ask me the name and I said, "Mehnwai sohte kak lokaia en Pohnpei, i soasoa lokaia en Pohnpei."  This brought laughter and the eldest Pohnpeian student took over teaching the younger ones the name of the tree.

The Yapese students were not certain the tree is found on Yap proper, but it was not clear that they agreed with each other on this assessment.

Ethnobotany does not generate item analyses, only moments of rapt attention and fascination. Significant learning opportunities. Content that is not soon forgotten.