Assessment preview fall 2007

19 August 2007

The following pretest assessment data is based conceptually on aggregation of course level student learning outcomes, the model proposed in the Student Learning Outcomes Assessment Plan. In this model, course level measurable skills and abilities are aggregated to report on accomplishment of program learning outcomes.

Course level internal outcome assessment data was always deemed to be insufficient and required triangulating data in order to determine whether a program was producing students who could perform the skills post-graduation. The present design of programs, courses, and articulation issues currently preclude the type of in situ assessment that schools such as Kapiolani Community College are deploying. I suspect that environmental differences will always complicate in situ assessment. Kolonia is simply not as large nor as diverse a community environment as Honolulu.

Until such time as alternate tools may be developed to perform assessment at the course level in the classroom, I will continue to employ item analysis as one way to gather data on student performance.

MS 150 Statistics

Students entering MS 150 Statistics include repeating students. The pretest covers basic statistics and linear relationships. In the table below, the fourth column (description) describes the concept tested by the pretest question. The seventh column (SLO) notes the outline outcome to which the question maps. The last two columns note the number of students who answered correctly and the percentage of students answering correctly. For example, 81% of the students correctly determined the minimum value in a given data set.

i Src Q Description l Outref SLO Corr Corr%
0 Pre P1 level of measure 1 1 Calculate basic statistics 2 5%
0 Pre P2 sample size 1 1 Calculate basic statistics 5 12%
0 Pre P3 mean 1 1 Calculate basic statistics 9 21%
0 Pre P4 median 1 1 Calculate basic statistics 13 31%
0 Pre P5 mode 1 1 Calculate basic statistics 4 10%
0 Pre P6 min 1 1 Calculate basic statistics 34 81%
0 Pre P7 max 1 1 Calculate basic statistics 34 81%
0 Pre P8 range 1 1 Calculate basic statistics 10 24%
0 Pre P9 standard dev 1 1 Calculate basic statistics 3 7%
0 Pre P10 coef var 1 1 Calculate basic statistics
0 Pre P11 slope 5 5 Perform a linear regression and make inferences based on the results 4 10%
0 Pre P12 intercept 5 5 Perform a linear regression and make inferences based on the results 4 10%
0 Pre P13 slope-intercept eqn 5 5 Perform a linear regression and make inferences based on the results 1 2%
0 Pre P14 predict y given x 5 5 Perform a linear regression and make inferences based on the results 6 14%
0 Pre P15 predict x given y 5 5 Perform a linear regression and make inferences based on the results 4 10%

The above questions map to the outline and thus a report on average performance on the proposed outline can be determined. This is shown in the following table:

Outref Students will be able to: Sum Count Avg
1 Calculate basic statistics 2.71 10 27%
2 Represent data sets using charts and histograms 0 0 0%
3 Solve problems using normal curve and t-statistic distributions including confidence intervals for means and hypothesis testing 0 0 0%
4 Determine and interpret p-values 0 0 0%
5 Perform a linear regression and make inferences based on the results 0.45 5 9%
PSLO define mathematical concepts, calculate quantities, estimate solutions, solve problems, represent and interpret mathematical information graphically, and communicate mathematical thoughts and ideas.


The last row above is the current mathematics program student learning outcome and average aggregate performance on that outcome.

SC 130 Physical Science

SC 130 Physical science will utilize basic statistical concepts during the course. Uncertainty in a particular measurement can be reduced by repeated measurements. Precision of a measurement is often specified by the standard deviation of a set of measurements about their mean. Many simply physical systems obey linear relationships and a linear regression can be used to find the relationship. All of these concepts will play a role in the course.

While some stateside college students might arrive at college unable to calculate a standard deviation, many are now familiar with mode, median, and mean due to their inclusion in textbooks since the revision of school standards arising from the 1989 NCTM standards for mathematics documents. I cannot make this presumption, hence the pretest focused more on statistical calculation skills and not science skills. The pretest also included two problems that involved mixed time units (hours and minutes) which had to be converted to decimal hours to be solved. This was intended to provide insight into the students' abilities to work with units.  The results are in the following table.

i Src Q Description Corr Corr%
0 Pre P1 determine minimum in data set 22 71%
0 Pre P2 determine maximum in data set 23 74%
0 Pre P3 determine range for a data set 7 23%
0 Pre P4 determine the mode for a data set 8 26%
0 Pre P5 calculate the median for a data set 8 26%
0 Pre P6 calculate the mean for a data set 3 10%
0 Pre P7 calculate the standard deviation for a data set 0 0%
0 Pre P8 calculate the slope for two-variable data with a linear relationship 0 0%
0 Pre P9 calculate the intercept/initial value for two var data with linear relationship 2 6%
0 Pre P10 write the slope-intercept function for linear data 1 3%
0 Pre P11 predict d given t 1 3%
0 Pre P12 predict t given d 0 0%
0 Pre P13 calculate speed given data requiring a time unit conversion 2 6%
0 Pre P14 calculate speed given data requiring a time unit conversion 4 13%
0 Pre P15 draw a conclusion from quantitative data 2 6%

Of some interest in the above table is the high performance since on basic minimum and maximum questions, mirroring the statistics results. Students appear to grasp the meaning of the words minimum and maximum. The remaining results are weak and indicate little contact with the statistical concepts that physical science laboratories presume.

Mapping to the SC 130 outline

Note that the physical science assessment does not include mapping to the physical science outline. At present the current outline is not aligned with the syllabus nor the envisioned design of the course. The present outline consists of 108 specific skills grouped into five course level outcomes. The later are not well aligned with the present syllabus. I have slated a reworking of the outline for spring 2008 at the earliest. There is much work to be done in what I hope will be a shift from a wide and shallow curriculum to deeper, narrower curriculum that handles science more as an activity and a way of thinking rather than as vast set of memorized factoids. I suspect it is the later that has led to the rise of fields such as intelligent design: science is presented as a giant book of facts to be taken on faith, so why cannot books of faith be taught as science? Approaching science as a purely constructivist process, however, would lead to little coverage of the material in the present outline. Given the probable need to maintain articulation and the spirit of the present title of the course, I will have to struggle my way forward in a balance between constructivism and direct instruction.

Mapping to the I, P, D/M model

Although the aggregation approach of the student learning outcomes assessment plan does not map directly to the introduce, practice, and demonstrate/mastery model, ultimately whether a particular outcome was introduced, practiced, or mastered will have to be measured. The education division seems to be the farthest along with the I,P,D/M model.

The education division has clearly chosen tests external to courses to determine student performance on their program student learning outcomes. How measurements internal to courses will interact with the external measurements remains something I am interested in seeing. I suspect that over time instructors in education will find their course performance implicitly measured by how well their students due on the external tests. The tests will eventually drive the courses. This is not necessarily a bad thing. If the tests are well-aligned to skills and content knowledge needed in the classroom, then this drive can be a positive one.

Physical science is primarily a general education core course. At best it can only introduce concepts. Many majors will never again take a laboratory science, hence there is no possibility of an assessment "stack" where students go on to practice and show mastery. The skills may also not be of obvious value post-graduation. Determining whether a business major graduate
working at midtown video renting out DVDs is  "Performing experiments that gather scientific information and utilizing, interpreting, and explaining the results of experiments and field work in a field of science," is not be something that will be observable let alone measurable.

SC/SS 115 Ethnobotany

SC/SS 115 Ethnobotany has evolved into a course that is influenced by thinking in the northeast region where significant learning opportunities are a key goal. The course is participatory and experiential, mixing direct instruction with student presentations and essays. The outline was redesigned a little over a year ago to align to a course level assessment structure.