Student Learning Outcomes Assessment

Division of Natural Sciences and Mathematics

Fall term 2006

Executive summary

Faculty in the Division of Natural Sciences and Mathematics each chose methods to measure performance on student learning outcomes in their courses during the fall of 2006. The data gathered was aggregated using a system described in the January 2006 system-wide assessment conference presentation on a proposed Student Learning Outcomes Assessment Plan.

Key findings:

1. Performance on course and program learning outcomes is typically below 70%

2. Faculty who engaged in the effort expressed gaining some insight into formative assessment information for their courses.

3. Coverage of program learning outcomes is not comprehensive.

4. Many institutional learning outcomes remain unmeasured.

5. The work involved proved long and difficult. The process is neither sustainable nor will the process scale-up well across the system.

6. When asked about the cost/benefit of the information gained versus the work done, faculty noted that the information gained served to confirm that which they already knew at the end of the course.ⁱ Thus the effort exceeded the benefit for the individual faculty member.

Report

Each faculty member was asked to determine ways to measure and report on student performance on the student learning outcomes for one of their courses. For the most part faculty chose to use item analysis of comprehensive final examinations to generate information on accomplishment of student learning outcomes. This approach, while logical and straightforward, naturally tends to limit the type of outcomes that can be measured.

The Student Learning Outcomes Assessment Plan described in the January 2006 system-wide assessment conference provided a conceptual framework by which to aggregate individual student learning outcomes.ⁱⁱ

For simplicity and manageability, each course outcome was mapped to a single program learning outcome and each program learning outcome was in turn mapped to a single institutional learning outcome. This "one-to-one" approach causes information loss wherein one course learning outcome might serve multiple program and institutional level learning outcomes. A "one-to-many" approach, however, would require an exponentially greater effort . In general, course learning outcomes best serve a particular program learning outcome, which in term best serve a particular institutional learning outcome. There is a limit to how many higher level outcomes a single course level outcome can conceivably serve, the "one-to-one" model is implementable and produces useful information.

The structure of this report is to move from the goal level down to the institutional learning outcomes, down to the program learning outcomes, and finally down to the course level outcomes.

Goal level

There is a story about a king who sent three wise counselors out into the world to gather all the world's information into a single book. The resulting book so pleased the king, that he sent the three wise counselors out to condense all the world's information into a single sentence. The result again pleased the king, so he asked the wise counselors to condense the world's knowledge to a single word. Decades later the now ancient counselors returned and said, "The word is 'maybe'." Aggregation ultimately reaches a point where the answer to the question of whether the number attained has any meaning is "maybe."

At the goal level, performance on goal one is given in table one.

Table 1: Learning attainment for goal one

Institutional level Student Learning Outcomes

Aggregation at this level remains problematic. Part of the difficulty is the issue of the loss of information that aggregation causes as noted above. Another complication is that the predominant choice of item analysis of comprehensive final examinations tends to emphasize the first institutional learning outcome, knowledge. Skills, communication, and critical thinking can conceivably be measured in a final assessment instrument, there is however a tendency to write questions at the lower knowledge levels of a learning pyramid.

Assessment of the other institutional learning outcomes, however, may be via methods that do not produce numbers which can be quantitatively aggregated. Qualitative aggregation would be needed. No attempt was made to explore qualitative aggregation, but this report will include discussion of an alternate assessment done in one science course that yields qualitative, indirect measurement of student learning.

Table 2: Learning attainment on institutional learning outcomes

Note that all of the mathematics course learning outcomes were mapped to the single mathematics general education core learning outcome. This is in turn mapped to a single institutional learning outcome, creativity. This was proposed in January 2006 and is based on "seek, find, and create solutions."

Program level Student Learning Outcomes

All of the mathematics courses, as noted above, are taken as serving the mathematics general education learning outcome. Thus the 64% reported in table three below represents the overall performance across all level of mathematics delivered by the division, including developmental mathematics. This level of performance on the student learning outcomes agrees well with the known pass rates in the mathematics courses in the division.

Illustration 1: Fall 2004 - Spring 2005 data

Although the possibility of a statistical coincidence cannot be ruled out, the system-wide pass rate for mathematics courses was 67% for fall 2003 and spring 2005. Work done in the previous century suggests that pass rates in mathematics are fairly stable in the 60% to 70% range. The 64% success rate for the mathematics general education student learning outcome is based wholly on item analysis of final examinations where items were mapped back to course level student learning outcomes.

This concurrence between typical, stable, historic pass rates and student learning outcome performance provides no grounds on which to disprove the theory that "qualified faculty delivering courses based on outlines built of student learning outcomes produce trustworthy grades reflective of actual learning by students."

Table 3: Learning attainment on program student learning outcomes

The HCOP performance is based only on a single course. The assessment was an item analysis of a final consisting solely of matching questions. The result was measurement only of cognition of definitions of terms, with a heavy focus on the structure, function, and basic pathologies of the human body.

A couple of terms of work on assessing the HCOP program indicates that some program learning outcomes are not specifically targeted.

Other HCOP program learning outcomes are not measured directly but are considered to be accomplished by the structure of courses in the program. An example of this is the program learning outcome related to working effectively in groups. All of the laboratory science courses have students working on lab teams. The outcome is being met via the structure of the activities in the course.

A future need of the HCOP program is a re-examination of these program level outcomes to determine whether specific program outcomes should be retained, altered, or eliminated.

The marine science outcomes were also measured by only a single course, and the assessment report provided only a global performance average. That said, the assessment done in the marine science course was a superb example of a formative assessment of whether the instruments used to assess the course matched the student learning outcomes on the outline.

Formative assessment is complex and multi-faceted. Using examinations to measure learning which is then mapped back to outcomes on an outline is only possible if the examination questions align well with the outcomes on the outline. The analysis done in SC 111 Environmental Science explored this question of alignment across a number of instruments in the course. The result will be changes to the outline, and probable modification of some of the assessment instruments. ⁱⁱⁱ

Course level Student Learning Outcomes

The following table reports on the success rate for each course level students learning outcome. Where an outline has general and specific objectives, the general objectives were used. The one exception was for SC 101 where the outline consists of a bank 19 general and 69 specific student learning outcomes. The final examination, in using a matching structure, did not map easily to the 19 general objectives which use language such as explain, describe, and list. The item analysis was aggregated against a topic list derived from the outline.

The construction of table four required a tremendous effort on the part of all faculty in the division. Table four, which underlies the data in table three, does not represent a sustainable mode of assessment. A debt of gratitude is owed to the many faculty in the division who grasped the model proposed in January 2006 and worked long and hard on producing data which could be aggregated.

A number of faculty were familiar with the assessment effort in MS 100 in Kosrae this past summer and went beyond the item analysis by mapping and aggregating their results to the outcomes on the outline. This represented a tremendous effort with faculty working well into the winter holiday break. Observing the labor hours required reinforced the realization that an aggregating approach is not sustainable nor will it scale up across all divisions at all sites.

Table 4 on the next page lists the course, sections from which the data derives, the source of the data (fx is final examination), the number of students (n) and the percentage of questions answered correctly that mapped to the shown course student learning outcome. Note that the number of questions underneath each percentage is not reported.

Table 4: Learning attainment on individual course level SLOs

A further note would be that absence of a course does not necessarily mean that the faculty did not submit assessment information. Some faculty were still working on their assessment analysis late into the winter holiday, more than a week after graduation. As noted, the effort the above analysis required was tremendous and unsustainable.

Year-to-Year

A year-to-year analysis is only possible for a single course, MS 150 Statistics, that performed a similar assessment in December 2005. The overall percentage difference year-to-year was not statistically significant (p-value = 0.71). Student performance on basic statistics remains strong, graphing skills required of histrograms remains weak as do interpretation of p-value and the making of inferences based on linear regressions. This data suggests that students have the ability to learn basic facts and perform fairly strongly tasks that are at the bottom of the learning pyramid. Students appear to have difficulty with interpretation, synthesis, and other higher level thinking skills. Tests that focus on "just the facts and nothing but the facts" often do not well assess these higher thinking skills.

The December 2005 analysis was a part of a study wherein the instructor first listed the areas of student weakness and then performed an item analysis on all examination questions. The instructor's intuition prior to the analysis was compared to the item analysis results and very strong agreement was found. Faculty who mark their own assessment instruments over the course of the term have an accurate internal compass in terms of areas of student strengths and weaknesses.

Specific year-to-year performance results are shown in table five.

Table 5: Year-to-year learning attainment in MS 150 statistics

Alternative Approach to Assessment

In SC/SS 115 Ethnobotany a different approach to assessment was taken. The outline was recently rewritten to facilitate the use of a course level portfolio which documents accomplishment of student learning. The portfolio consists of material produced during the fall 2006 running of the course. All material is presented on web pages linked from a single table that acts as an access "portal" to the portfolio. Material includes tests and photographic documentation of activities. The on-line web page table is reproduced in table six minus the embedded hyperlinks. The table only makes "sense" when accessed on line at:

http://www.comfsm.fm/~dleeling/ethnobotany/scss115_assessment63.html

The decision to utilize a portfolio approach was due in part to the lack of a comprehensive final examination to item analyze and map back to the outline. While the analysis is ultimately indirect, the portfolio documents that the course delivered a significant learning opportunity in full alignment with the outline.

Table 6: A table from a web page portal to on-line course portfolio items

The portfolio provides direct evidence of the learning opportunities for students but no direct evidence of learning. The portfolio also does not provide an obvious way in which to accomplish aggregation. Yet the portfolio can confirm that students accomplished the stated outcomes. Did the student communicate and present? There is photographic and notes evidence that the students did so. Did students identify local plants? Have a look at http://www.comfsm.fm/~dleeling/ethnobotany/e63mx.html, noting the first section of that midterm and the images of the plants present in the laboratory at the bottom of that same page. That the outcomes on the outline were accomplished at a course level is attested to by the evidence in the portfolio.

Plans for Improvement of Learning

Faculty were not asked to produce a plan for improvement in conjunction with the data submitted for this report. For most faculty, simply generating the data required a significant amount of time and effort. Anecdotally, a number of faculty reported on plans for improvement.

The intent of assessment of student learning is to produce a process by which learning in courses is improved. Assessment should lead to changes in outlines, approaches to material in courses, assessment instruments, and other instructional variables. Assessment should reveal learning weaknesses in the students and challenge the instructor to improve the course in those areas.

If every motivated student is successfully learning each and every specified student learning outcome on an outline, then that course can remain stagnant and unchanging decade after decade. The reality is that measures of learning typically reflect the overall average for all course learning outcomes in this study, 57%. Pass rates for most mathematics and science courses are also under 70%.

There is room for improvement in most if not all courses. A course that does not ever change is not taking the necessary risks of curriculum experimentation. Some changes will improve learning – whether measured formally or by the internal understandings of instructors. Other changes may have a neutral or even negative effect on learning. Selecting for changes that improve learning leads to courses which evolve, adapt, and survive.

Next Steps

While individual faculty may have found enough value in this exercise to repeat this process within their own individual courses, the division as a whole is unlikely to re-attempt this form of analysis. Bear in mind that there were faculty in the division who were unable to turn in any assessment results. Ideally assessment is faculty driven, the reality is that ongoing, sustained assessment will require the efforts of support personnel designated and paid specifically to do assessment.

Program assessment via aggregation of actual student learning attainment in the classroom is not viable over the long haul. Programs will have to be assessed by external measures. External measures such as data derived from alumni and employer surveys will provide information on whether graduates have mastered the knowledge sets required of them. Conceivably the college may need to do sampling with standardized tests. Measurement of learning will come from triangulation.

The above does not mean course outlines should not use student learning outcomes. Student learning outcomes, whether micro-skill specific or broader, more comprehensive statements of measurable learnings, provide a way to produce a focused statement of expected learning in a course.