Unified Assessment Plan: Flowing assessment up the plan

> ...how soon can we have [a] clear picture of whether our PLO's, CLO's and SLO's are viable or need further assessment...To be honest our PLO's are clear,... - BM

Bear in mind that the "unified assessment plan," or whatever one wishes to call it (someone suggested Fairly Obvious and Objectively Logical plan), takes those "clear PLO's" and provides a framework from which to generate inferential information on achievement of program and institutional level assessments.

In the world of mathematics we cannot prove that the square root of two is irrational. We can only prove that it is not rational. The only other choice left is that the root is irrational.  Inference works in this manner.  I may never be able to show that the communication program learning outcome is directly achieved by measuring learning at the program level. I can, however, show that the program learning outcome can never be achieved if I am able to show that no course in the program has an outline with a communication student learning outcome.  The matrices in that spreadsheet show distributions of institutional effort: where we are putting our emphasis.

We have fine PLOs, no need to alter these.

Obviously the holy grail, that which everyone desires, is to know what a graduate knows, can do, and values as they walk down the graduation aisle.  Like the holy grail, this may prove ethereally elusive. And in the end it will turn out not to be a gold chalice but a simple carpenter's cup.

In pursuit of knowing what a graduate knows, one's mind is likely to leap to exit tests. Yet if I gave the entrance test to the faculty, my guess is that many would place into developmental mathematics if not MS 090 - the lowest developmental level at this time.

I suspect that ultimately we will look at the achievement of course level student learning outcomes and use those to infer accomplishment of program learning outcomes, which can in turn be used to infer accomplishment of institutional learning outcomes. I have no idea if this is acceptable, but it simplifies things in my small mind. One measures student learning outcomes at the course level. The higher levels are inferred. Faculty would only have to focus on what students are mastering and not mastering in their own courses and then reporting this information.

Permit me to digress in order to come back to my point.

I was asked today whether we would eventually be able to generate numbers such as "70% of the students communicate effectively through writing, speaking, performing, exhibiting, or other forms of expression." Or to use such a statement as an institutional objective, "70% of the students will communicate effectively through writing, speaking, performing, exhibiting, or other forms of expression."

Before I share my response, think for a moment what one would have to do to be able to say this. One would have to do as one instructor in my division attempted to do this term: to measure every outcome for each and every student individually. It looked achievable in August and we discussed a possible presentation on the system in December. The faculty member has asked to defer the presentation until after the spring term, I gather he has found a need to modify the system.  My understanding is that the data entry load was ferocious. Bear in mind each outcome was to be assessed ten times during the course of the term for each and every student to ensure reliability of the assessment.

I have long argued that one's ability to handle the data entry and tracking load varies with the sheer number of outcomes and students. If the outcomes are few and the class sizes small, this is a feasible approach. Learning can be proven and documented on a student-by-student basis.

In larger classes with large batteries of outcomes record keeping and management become problematic. Proving learning becomes difficult.

Gödel proved the unprovability of "sufficiently complex systems" once and for all in mathematics - the home court of quantification. Any sufficiently complex system will have theorems that can neither be proven true nor shown to be false. I have long asserted that proving achievement of student learning, especially that which the student knows, can do, and values at graduation, will have Gödelian limitations.

One thing I suspect we can do is the internal inferential model such as that proposed by the so-called unified assessment plan and other inferential models such as those used at Mesa College. At a recent workshop the president was given the Mesa model. I noted that their "program assessment model" was, at its core, a set of three surveys.  The students, the community, and employers. The surveys sought to determine the value of a particular program to each of those stakeholders.  Again, inferential.

My response to the question of whether we would eventually be able to say,
"70% of the students communicate effectively through writing, speaking, performing, exhibiting, or other forms of expression," was that I thought this is something the faculty should probably decide. That is, whether or not we would want to attempt to generate such data or set some such goal.
Before I digressed, I said that I wanted to infer the higher levels. So how might this work, beyond the analysis work I've done in the "unified assessment plan" or "grand unified theory"1 as I dubbed it in my first emails?  Although I do not know, what follows is one possibility.  The following presumes one is familiar with the unified assessment spreadsheets. The spreadsheets are available at:
http://www.comfsm.fm/~dleeling/assessment/islos.xls

Take a small piece of the p versus c HCOP spreadsheet tab in the islos.xls spreadsheet I sent to the accreditation working group and the curriculum committee on the 11th of December:
 Students will be able to: analyze foundations human body problem solve Define the concept of total fitness and the essential characteristic and consideration of a physical fitness program. 1 Describe the process of conception as well as the influence of heredity and environment to the developing human organism. 1 Describe what it means to be healthy. 1 Discuss the effect of drugs to the human body, human behavior and its consequence to family and society. 1 Enumerate the causes and prevention of communicable diseases. 1 Explain how the environment can affect health. 1 Explain the body functions and the relation of each system to one another. 1 Explain the effect of the electrical signals and chemical messengers to human physiology and behavior. 1

Suppose I choose this year to focus on the foundations program learning outcome. Having ruled out the direct approach above, I would sit down with the faculty teaching each of the courses and "survey" them on each course level outcome that "matrixed" against foundations. The results for the small piece above might look like:

 Students will be able to: analyze: 2005 foundations: 2006 human body: 2008 problem solve: 2009 Define the concept of total fitness and the essential characteristic and consideration of a physical fitness program. Students have a good understanding of the concept based on tests and in class discussions, according to the instructor. Yet few exhibit behaviors that show they actually value fitness. Describe the process of conception as well as the influence of heredity and environment to the developing human organism. Students have a good understanding of this concept. Describe what it means to be healthy. The students are able to respond correctly when asked this question. Discuss the effect of drugs to the human body, human behavior and its consequence to family and society. The students are very capable at expressing themselves on these effects, whether orally or in writing. 1 Enumerate the causes and prevention of communicable diseases. Students are able to enumerate the causes and prevention. Explain how the environment can affect health. The students exhibit a number of misunderstandings and misconceptions concerning this topic. Explain the body functions and the relation of each system to one another. 1 Explain the effect of the electrical signals and chemical messengers to human physiology and behavior. 1

Is this direct assessment? No. In "Benchmarking to Improve Departmental Systems for Managing Quality and Standards," James Tannock and Norman Jackson include the following quote from their research, "A major theme in deciding on the type of assessment to be used the amount of marking time involved. Staff will tend to resist more student-centred [sic] formative assessment methods when these involve more marking time."  In other words, getting the actual data of how many students mastered each of the above is likely to fail, especially when we have some faculty in open opposition to the whole "student learning outcomes" approach.

The proposed hybrid returns the workload to the chair to work with faculty and generate information that can then be aggregated and reported.  After surveying all of the outcomes in all of the courses that impact the targeted program learning outcome, the chair or instructional coordinator could then summarize the findings in the cross-matrix between the institutional student learning outcomes and the program student learning outcomes.

Thus the i versus p matrix sheet at row 63:
 Prgrm program Student Learning Outcome: students will be able to: iSLO: analysis iSLO: communication iSLO: creativity iSLO: intellect iSLO: knowledge iSLO: skills HCOP Demonstrate a foundation in basic biology, chemistry, microbiology, anatomy, nutrition, health, and physiology. 1

Becomes:
 Prgrm program Student Learning Outcome: students will be able to: iSLO: analysis iSLO: communication iSLO: creativity iSLO: intellect iSLO: knowledge iSLO: skills HCOP Demonstrate a foundation in basic biology, chemistry, microbiology, anatomy, nutrition, health, and physiology. Students show solid foundations in these fields, with the exception of nutrition. In general, students tend to be capable of reciting the facts but show evidence they value the information as evidenced by a lack of changes in their behaviors.

Here my reach is exceeding my grasp, I am uncertain that terse summaries are possible.2 If they are, then the i versus p matrix would become, after five to seven years, a "single sheet" overview of how well the institution is meeting program and institutional learning outcomes.

The above approach seems to me to be achievable and to work with some of the practical realities. Work loads on higher level assessment remain with program coordinators (chairs and instructional coordinators) rather than on faculty.

General education would require each "area coordinator" - each chair and instructional coordinator - "chipping" in on their area, or something like that, contributing to the i versus p matrix for their area.

My apologies if I have overstepped my bounds or am simply too far off the beaten track with these ideas. This is definitely a model that I am developing as I go. Each new piece leads me on around a bend to another concept, but at some point I may indeed be "around the bend."  The unified assessment plan laid out in islos.xls, however, feels useful because the structure itself creates options and possibilities for assessment of the structure.

Dana

1 Borrowed from high-energy physics, the acronym also connoted that the theory makes "gut sense" to a physicist. Occam's razor simply demands that this is how the universe works, at least according to the proponents of the theory.  I cannot remember if super symmetric theory is predecessor to, a part of, or successor to GUT.

2 I am reminded of the story I first heard told by Pete Seeger about the three wise men. The king had an heir and wanted the heir to learn all that there is to learn in the world prior to ascending to the throne. The king set his three wise men out into the world to compile all the knowledge in the world into a single book.  Three years later they return with a massive tome and the king presents the volume to the royal tutors and the heir.

The king, having nothing better for the three wise men to do, said, "You did such a fine job on the book, go back out into the world and boil down all the knowledge of the world into a single sentence." Ten years later the wise men returned, looking increasingly aged and worn.  "Sire," reported the wise men, we have done as you asked. Then the wise men said, "Seek and you shall find, knock and the door will be opened, ask and you shall receive."

The king was thoroughly impressed. So impressed he asked, "Could you boil all the world's wisdom down to a single word?"  The three wise men committed to try.

The years past and no one heard from the wise men, the king thought perhaps age had caught up with them and they would not be returning. As the king lay in bed, his health fatally failing, the equally weakened wise men returned.  Ancient beyond reason, the three wise men collapsed by the king's bed. The king, his voice reduced to a whisper, said "Well?"  The wise men said, "Maybe."