The following pretest assessment data is
based conceptually on aggregation of course level student learning
outcomes, the model proposed in the Student Learning Outcomes
Assessment Plan. In this model, course level measurable skills and abilities are
aggregated to report on accomplishment of program learning outcomes.
Course level internal outcome assessment data was always deemed to be insufficient and required triangulating data in order to determine whether a program was producing students who could perform the skills post-graduation. The present design of programs, courses, and articulation issues currently preclude the type of in situ assessment that schools such as Kapiolani Community College are deploying. I suspect that environmental differences will always complicate in situ assessment. Kolonia is simply not as large nor as diverse a community environment as Honolulu.
Until such time as alternate tools may be developed to perform assessment at the course level in the classroom, I will continue to employ item analysis as one way to gather data on student performance.
Students entering MS 150 Statistics include repeating students. The pretest covers basic statistics and linear relationships. In the table below, the fourth column (description) describes the concept tested by the pretest question. The seventh column (SLO) notes the outline outcome to which the question maps. The last two columns note the number of students who answered correctly and the percentage of students answering correctly. For example, 81% of the students correctly determined the minimum value in a given data set.
|0||Pre||P1||level of measure||1||1||Calculate basic statistics||2||5%|
|0||Pre||P2||sample size||1||1||Calculate basic statistics||5||12%|
|0||Pre||P3||mean||1||1||Calculate basic statistics||9||21%|
|0||Pre||P4||median||1||1||Calculate basic statistics||13||31%|
|0||Pre||P5||mode||1||1||Calculate basic statistics||4||10%|
|0||Pre||P6||min||1||1||Calculate basic statistics||34||81%|
|0||Pre||P7||max||1||1||Calculate basic statistics||34||81%|
|0||Pre||P8||range||1||1||Calculate basic statistics||10||24%|
|0||Pre||P9||standard dev||1||1||Calculate basic statistics||3||7%|
|0||Pre||P10||coef var||1||1||Calculate basic statistics||0%|
|0||Pre||P11||slope||5||5||Perform a linear regression and make inferences based on the results||4||10%|
|0||Pre||P12||intercept||5||5||Perform a linear regression and make inferences based on the results||4||10%|
|0||Pre||P13||slope-intercept eqn||5||5||Perform a linear regression and make inferences based on the results||1||2%|
|0||Pre||P14||predict y given x||5||5||Perform a linear regression and make inferences based on the results||6||14%|
|0||Pre||P15||predict x given y||5||5||Perform a linear regression and make inferences based on the results||4||10%|
The above questions map to the outline and thus a report on average performance on the proposed outline can be determined. This is shown in the following table:
|Outref||Students will be able to:||Sum||Count||Avg|
|1||Calculate basic statistics||2.71||10||27%|
|2||Represent data sets using charts and histograms||0||0||0%|
|3||Solve problems using normal curve and t-statistic distributions including confidence intervals for means and hypothesis testing||0||0||0%|
|4||Determine and interpret p-values||0||0||0%|
|5||Perform a linear regression and make inferences based on the results||0.45||5||9%|
|PSLO||define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas.||7.24%|
The last row above is the current mathematics program student learning outcome and average aggregate performance on that outcome.
SC 130 Physical science will utilize basic statistical concepts during
the course. Uncertainty in a particular measurement can be reduced by
repeated measurements. Precision of a measurement is often specified by
the standard deviation of a set of measurements about their mean. Many
simply physical systems obey linear relationships and a linear
regression can be used to find the relationship. All of these concepts
will play a role in the course.
While some stateside college students might arrive at college unable to calculate a standard deviation, many are now familiar with mode, median, and mean due to their inclusion in textbooks since the revision of school standards arising from the 1989 NCTM standards for mathematics documents. I cannot make this presumption, hence the pretest focused more on statistical calculation skills and not science skills. The pretest also included two problems that involved mixed time units (hours and minutes) which had to be converted to decimal hours to be solved. This was intended to provide insight into the students' abilities to work with units. The results are in the following table.
|0||Pre||P1||determine minimum in data set||22||71%|
|0||Pre||P2||determine maximum in data set||23||74%|
|0||Pre||P3||determine range for a data set||7||23%|
|0||Pre||P4||determine the mode for a data set||8||26%|
|0||Pre||P5||calculate the median for a data set||8||26%|
|0||Pre||P6||calculate the mean for a data set||3||10%|
|0||Pre||P7||calculate the standard deviation for a data set||0||0%|
|0||Pre||P8||calculate the slope for two-variable data with a linear relationship||0||0%|
|0||Pre||P9||calculate the intercept/initial value for two var data with linear relationship||2||6%|
|0||Pre||P10||write the slope-intercept function for linear data||1||3%|
|0||Pre||P11||predict d given t||1||3%|
|0||Pre||P12||predict t given d||0||0%|
|0||Pre||P13||calculate speed given data requiring a time unit conversion||2||6%|
|0||Pre||P14||calculate speed given data requiring a time unit conversion||4||13%|
|0||Pre||P15||draw a conclusion from quantitative data||2||6%|
Of some interest in the above table is the high performance since on basic minimum and maximum questions, mirroring the statistics results. Students appear to grasp the meaning of the words minimum and maximum. The remaining results are weak and indicate little contact with the statistical concepts that physical science laboratories presume.
Note that the physical science assessment does not include mapping to the physical science outline. At present the current outline is not aligned with the syllabus nor the envisioned design of the course. The present outline consists of 108 specific skills grouped into five course level outcomes. The later are not well aligned with the present syllabus. I have slated a reworking of the outline for spring 2008 at the earliest. There is much work to be done in what I hope will be a shift from a wide and shallow curriculum to deeper, narrower curriculum that handles science more as an activity and a way of thinking rather than as vast set of memorized factoids. I suspect it is the later that has led to the rise of fields such as intelligent design: science is presented as a giant book of facts to be taken on faith, so why cannot books of faith be taught as science? Approaching science as a purely constructivist process, however, would lead to little coverage of the material in the present outline. Given the probable need to maintain articulation and the spirit of the present title of the course, I will have to struggle my way forward in a balance between constructivism and direct instruction.
Although the aggregation approach of
the student learning outcomes assessment plan does not map directly to
the introduce, practice, and demonstrate/mastery model, ultimately
whether a particular outcome was introduced, practiced, or mastered
will have to be measured. The education division seems to be the
farthest along with the I,P,D/M model.
The education division has clearly chosen tests external to courses to determine student performance on their program student learning outcomes. How measurements internal to courses will interact with the external measurements remains something I am interested in seeing. I suspect that over time instructors in education will find their course performance implicitly measured by how well their students due on the external tests. The tests will eventually drive the courses. This is not necessarily a bad thing. If the tests are well-aligned to skills and content knowledge needed in the classroom, then this drive can be a positive one.
Physical science is primarily a general education core course. At best it can only introduce concepts. Many majors will never again take a laboratory science, hence there is no possibility of an assessment "stack" where students go on to practice and show mastery. The skills may also not be of obvious value post-graduation. Determining whether a business major graduate working at midtown video renting out DVDs is "Performing experiments that gather scientific information and utilizing, interpreting, and explaining the results of experiments and field work in a field of science," is not be something that will be observable let alone measurable.
SC/SS 115 Ethnobotany has evolved into a course that is influenced by thinking in the northeast region where significant learning opportunities are a key goal. The course is participatory and experiential, mixing direct instruction with student presentations and essays. The outline was redesigned a little over a year ago to align to a course level assessment structure.