IB • IB1 • IB2 • IB3 • IB4 • IB5 • IB6 • IB7

IB. Improving Institutional Effectiveness
 IB1. Dialog
 IB2. Goals
 IB3. Progress
 IB4. Evidence
 IB5. Communication
 IB6. Planning
 IB7. Mechanisms

IB. Improving Institutional Effectiveness

IB. Descriptive Summary

Production and Support of Student Learning Outcomes

The core of the effort to produce and support student learning consists of utilizing a highly qualified and professional faculty who are given the necessary resources and support to deliver courses built around measurable student learning outcomes. Measurement of that learning is primarily done by faculty.

Courses are designed around outlines consisting of measurable student learning outcomes. These outlines are prepared by faculty who teach the courses. These course outlines are reviewed by division chairs and the curriculum committee. The curriculum ensures that outlines are written using measurable student learning outcomes. Production and review of outlines often involves referring to similar courses at comparable institutions.

The responsibility for the process of assessment, analysis, and redevelopment to improve student learning rests first and foremost on faculty. Each is a specialist in the area they are teaching, either via academic qualifications or through a combination of academic qualifications and experience.

The faculty who teach a course develop the student learning outcomes for the course, usually referencing both college-level texts and, in many cases, equivalent courses at other colleges.

Over the past year a model of assessment has been developed. Originally developed in one division, the model is merely a diagram that shows visually how a division achieves and measures learning.

The original core of the model consists of qualified faculty delivering courses based on student learning outcomes. The actual model exists as a web portal, a web page that contains links to supporting documents. This document can only display the web page as a static, non-interactive entity. The actual web portal is, however, interactive.

IB.1 Diagram of Web Portal with Faculty and Outline links

Courses based on Student Learning Outcomes		Qualified faculty
Outlines	Sample Syllabi	Instructor
MS 150	MS 150	Dana Lee Ling

Although the above diagram is static, the online version includes links to outlines and faculty qualifications. The Outlines item links to a list of outlines that are available online. The MS 150 links to the MS 150 outline. The name Dana Lee Ling links to resumé for that member of faculty. The actual portal includes other faculty and course outlines.

The link between faculty and the courses that they teach are their course syllabi. In the actual portal there are links to online syllabi. This permits anyone either inside or outside of our organization to access information on our faculty, our courses, and how those courses are implemented. This purports to produce results, student achievement data in the form of grades, that can be trusted.

IB.2 Diagram of Web Portal with Faculty, Outline, and Grade links

Courses based on Student Learning Outcomes		Qualified faculty		Grades
Outlines	Sample Syllabi	Instructor
MS 150	MS 150	Dana Lee Ling

On the actual portal page Grades links to a report on grades given in the division.

This system has two shortcomings. The first is that this system does not provide a way to assess program level student learning outcomes. The second is that this system relies wholly on measures that are internal to the classroom and within the scope and control of single individual. The College hires qualified faculty and trusts the faculty implicitly. An outsider, however, might want some form of external evaluation of the system. An outsider would probably ask what a given grade might mean.

Each division is in the process of making its own decisions on how to validate their grades to an outside observer. In the example shown above, the mathematics division has developed and deployed an in-house test that is to be given to a sample of post-MS 100 students.

IB.3 Diagram of Web Portal with Faculty, Outline, Grade, and Program Evaluation links

Courses based on Student Learning Outcomes		Qualified faculty		Grades		Program outcomes evaluation
Outlines	Sample Syllabi	Instructor
MS 150	MS 150	Dana Lee Ling

On the actual portal page, the item Program outcomes evaluation links to a report on pilot run of the mathematics program evaluation instrument in the Spring of 2003. The program outcomes evaluation is referred to as an external measure for the system. The development and subsequent assessment of program level student learning outcomes is only just beginning at the College. The intent is that the external validation of grades will support program level student learning outcomes.

The following is the web portal with the program learning outcomes depicted.

IB.4 Diagram of Mathematics Web Portal with internal and external support for Program Learning Outcomes

					Math Program Learning Outcomes Link to Performance Based Budget
Up to degree level outcomes, goals, and mission
		Internal Measures				External Measures
Courses based on Student Learning Outcomes		Qualified faculty		Grades		Program outcomes evaluation
Outlines	Sample Syllabi	Instructor
MS 150	MS 150	Dana Lee Ling

The above is merely a diagram, a way of depicting the inter-relationships among the systems that measure student learning at the College. The genesis of the diagram was in discussions in the Fall of 2003 on the inter-relationships between course level student learning outcomes, program level student learning outcomes, and grades. Of course the heart of any college is the students, the diagram can only depict the structures that are involved with measuring student learning. The online portal provides easy access to the documents that report on student learning. The diagram has been referred to in some documents as a student learning outcome nexus or student learning outcomes hub.

While other divisions may not physically deploy a web portal, all divisions have systems that could be depicted in such a manner. Each division is making their own decisions on what constitutes external validation of student learning outcomes. Some divisions are publishing rubrics and sample student work with the associated grade. Other divisions are planning to use binding exit examinations. Discussion of external measures includes dialog on portfolios, projects, capstone experiences, and other ways of knowing students have achieved program level student learning outcomes.

The web portal does not depict the full assessment cycle, the feeding back into the development phase of assessment results. The diagram would become unnecessarily complex and unwieldy with the addition of arrows circling back from the program level to the course level.

Support for student learning also includes a number of divisions and programs that are outside of the classroom.

The Learning Resource Center is an important support facility providing both texts and Internet access. This facility is covered in Standard IIC.

Other programs, some institutionalized and others supported by grants, provide support to students. These programs include tutoring programs and college-run grants that serve pre-college students.

We have been made aware that WASC is less interested in what one plans to do than in what one is doing, however building our program level student learning infrastructure has only just begun.

IB. Key Processes and Resource Allocation

Resource allocation is separately and independently covered in Standard III later in this document.

Resource allocation is decentralized, including facilities design, facilities usage, and financial resources.

The facilities were designed with input from a number of constituencies, both at the national campus and at the state campuses. The exception to this is the Chuuk State Campus which does not currently own its own buildings.

The classrooms at the national campus, built in the mid-1990's, were designed with input from the chairs of the respective divisions of the College at that time. The Learning Resource Center was designed under the aegis of the library committee and guided by a vision statement prepared by that committee. Design details down to and including the location and type of shelves and computers were specified by a faculty-staff team.

Facilities usage is directed by facility directors and division chairs. The division chairs prepare schedules of classroom usage which are then approved by the Director of Academic Programs. Future facility plans are made in consultation with faculty and staff. Committees such as Curriculum Committee, the weekly chairs meetings, and ongoing email communications are all involved in providing input to these decisions.

Individual facility directors and division chairs prepare budgets for their area. These budgets are submitted to the finance committee, a committee consisting of faculty, staff, and administrators, for inclusion in the College budget.

The extent to which an individual facility director or division chair shares their budget planning and preparation with others in the facility or division varies from unit to unit. Concern has been expressed as to the transparency of budgets in the state campuses, especially with respect to the faculty and staff who work in the state campuses.

IB. Demonstrations of Effectiveness

All courses are based on course outlines that utilize measurable student learning outcomes. Measurement of these outcomes rests with faculty. At present there are no external mechanisms that provide evidence of achievement of student learning outcomes. All evidence of achievement resides with individual professors and instructors.

The methods of measurement vary from course to course, with each instructor specifying the measurement systems most appropriate to their course. Measurement methods include, but are not limited to, homework, quizzes, tests, midterm and final examinations, projects, presentations, papers, and performances.

The College has used student achievement data and other indicators to perform what have been called program health reviews. Key indicators have included enrollment and student seat costs. None of these indicators, however, carries any information as to whether learning is occurring in the classroom. Demonstrating that program's reviewed by indicators are effective programs from the perspective of student learning is not possible.

IB. Self Evaluation

1. Develop student learning outcomes above the course level and assess those outcomes.
2. Develop a process for formally approving program and institutional student learning outcomes involving the curriculum committee and the assessment committee.

The College has begun to build program level assessment of student learning outcomes. This effort has begun with a dynamic and active dialog on what constitutes important program level outcomes in each program. This is an ongoing conversation that is built on the knowledge of what works in our classrooms, the learning styles of our students, and on what is seen as important for our students to take away from a program. Program learning outcomes have been adopted in mathematics. Program learning outcomes for the English general education core, science, and kinesiology have been proposed.

The effort to deploy program evaluations that do reflect learning, the external evaluations noted in the web portal diagrams above, has only just begun. The College has not had a year-to-year cycle of any external evaluation system by which to make true program effectiveness measures.

Assessment of learning at the program level has only begun at the national campus and in the tourism and hospitality program at the Pohnpei State Campus. Efforts at the other state campuses have yet to begin, although workshops have been planned to help these campuses perform program level assessment.

IB. Planning Agenda

The College is conducting a review of course outlines to determine whether there are courses using out of date outlines or outlines that do not utilize measurable student learning outcomes.

The College is continuing to develop program level student learning outcomes and assessment instruments. The VPIA has proposed a timeline for this work, although as of the writing of this document the timeline has yet to be discussed by the curriculum committee.

Even while the curriculum committee discusses timeline issues, divisions are forging ahead with the development of program level student learning outcomes development. This development work is lengthy as it must involve dialog inside each division leading to general agreement on the learning outcomes for that particular program.

IB1. The institution maintains an ongoing, collegial, self-reflective dialogue about the continuous improvement of student learning and institutional processes.

IB1. Descriptive Summary

The college had recommended the use of student learning outcomes in course outlines for many years. Under the leadership of the then new Vice President for Instructional and Academic Affairs, all outlines produced after Fall 1992 had to be in a student learning outcome format.

In the Fall of 1995 the College embarked on the development of program health indicators as a way to measure the performance of all programs, not just academic programs. Many of these program health indicators were just that: single point in time indicators or measurements such as faculty to student ratios, freshmen intake to graduation rates, numbers of students counseled, number of books checked out, or number of air conditioners cleaned. While many indicators were internal to the College, a few were external. An accounting program might look at the number of graduates who successfully obtained jobs in their field.

In [1996-1997], as part of the development of a five year strategic plan the College developed program goals, objectives, and outcomes. These outcomes were not necessarily student learning outcomes, and in many divisions the previously developed program health indicators drove the development of the outcomes. That is, outcomes were developed that would be measured by the program health indicators.

Two examples of this structure can be seen below.

The structure seen above of goal-objective-indicator was developed for each division or area of the College by representatives from that division or area. These were developed not just for academic divisions but also for the Financial Aid Office, Office of Admissions and Records, the Business Office, Maintenance and Facilities, the Learning Resource Center, and all areas of the College.

This structure or portions of it can be seen in a variety of documents including the strategic plan and some of the indicators reported as part of the performance based budget (See Strategic Plan 2001-2006 HI 4 page 38 and Performance Based Budget Institution-Level and Program Outcomes and Outcomes Measures Fiscal Year 2004 214.4.1 page 82). Indicators or indicators modified into outcomes can still be found in a number of documents.

There is not, however, a complete congruence between the Strategic Plan of 2001-2006 and the above reported table. This has in part to due with the evolution and construction of the documents. The above table began life out of workshop on program health indicators conducted in 1995. At that workshop divisions developed program health indicators by which to evaluate the health of their programs. These were student achievement data indicators. This effort was a "bottom-up" effort and included the generation of program goals and objectives.

The Strategic Plan 2001-2006 arose out of mission statement review and the setting of ten institutional goals done by the Planning Council from 1998 to 1999. This process did not include all of the division chairs, those who had worked on the program health indicators effort three years earlier. Thus while the chairs were utilizing a program health indicators the Planning Council worked on a superstructure that was not directed by the existing structures. This was a "top-down" effort and generated goals from a different basis.

Work on bringing the program health indicators and their associated goals and objectives in line with the work done by the Planning Council was done by the Institutional Research and Planning Office in internal documents and by work done in the Assessment Committee in 2001 (Assessment Committee minutes 07 July 2001 through 11 October 2001).

The goals and objectives (now renamed activities) would be further modified in 2002 and 2003 principally by the Acting Director of Research and Planning 2002-2003 and published May 2003 in the Performance-Based Budget Institution-Level and Program Outcomes and Outcomes Measures Fiscal Year 2004 (Performance Based Budget 2004 or PBB 2004).

This document would differ from the Strategic Plan 2001-2006 due to work done in the Assessment Committee and due to work done by the Acting Director of Research and Planning. A program-by-program analysis of differences has not been formally performed, but change is inevitable in an organism as dynamic as the College. Notable differences include the omission of the Micronesian Studies Program from the PBB 2004 document and the inclusion of Physical Education in the PBB 2004 document. The PBB 2004 document represents the most current published statement of the status of the College's program goals.

To understand the nature of the changes from the above program health indicator based model seen in Table IB1.1, the following is an excerpt from the physical education section of the Performance Based Budget 2004 document. The budget data is omitted from this table.

Table IB1.2 reveals a hybrid nature that includes student learning outcomes as measured by performance indicators. The next step in the evolution of assessment at the college is seen as measuring the student learning outcomes directly rather than using indicators. The Performance Based Budget, however, had to use measures that are already extant and not those the College is in the process of developing and deploying.

The evolution from a goal-objective-indicator structure to a student learning outcomes structure at the program and institutional level is now occurring. This evolution is self-reflective dialog, and not necessarily the quiet reverie that the word self-reflection conjures up. This is an energetic discussion that permeates the campus. Specific models, formats, and implementations remain under discussion. Any examples presented have to be noted as being unofficial and unapproved at this point in time.

The following example derives from a physical education course outline and includes only a subset of all the student learning outcomes.

In the above model the institutional and program outcomes are measured through accomplishment of the course outcomes. Separately there would need to be survey assessments done to provide external measurements of the institutional and program outcomes.

The above outline is the result of a dialogue during the Fall of 2003 that is documented in part by the following reports and documents:

These reports trace an evolution in thinking as a result of conversations in Curriculum Committee and the ongoing conversations held on the sidewalks and in the cubicles of the College. The College does not set aside a specific time or place to discuss specific topics or assessment cycles. That was in part the downfall of the Assessment Committee (detailed in IB3 and IB6) - it was not seen as being relevant to what was happening out in the hallways and classrooms. The College is engaged in an ongoing, continuous discussion of learning in our classrooms.

Our small island environment means that faculty and administrators meet and talk nearly daily in a series of ongoing impromptu meetings. There are only a limited number of stores on Pohnpei - impromptu meetings occur in the hallways of the College, in the aisles of a supermarket, between tables at a local restaurant, or on a Saturday at a picnic. This is probably unique to small island schools and to our environment.

The result is that faculty are more aware of what is happening in other classrooms and other divisions than one would likely be at a continental school.

IB1. Self Evaluation

The College is engaged in a discussion of measuring student learning. This discussion is centered at the national campus, but does include state campus personnel as well. The discussion includes the issues surrounding the difficulties of including remote island state campus personnel. Many of these personnel are part-time employees literally passing through our system, teaching for but a term or two. In other instances there are permanent faculty, but communication with them can be difficult.

Due to the distances, and the cost of something as simple as a telephone call, communication with remote island faculty relies on email. Part-time faculty, however, rarely have email addresses, and when they do have an email address, the national campus and its division level personnel are rarely aware of the email address. Some campuses, notably Chuuk, appear to have difficulty providing faculty with sufficient access to Internet connected computers, complicating the communication effort immensely.

Thus the ongoing, self-reflective dialog is missing voices. Faculty from the remote state campuses are less likely to be a part of the discussion, and low level staff at remote state campuses are likely to have no ability to join the dialog via email.

Thus while there is an ongoing dialog on student learning at the core, the periphery is less participative. The provision of computers and Internet access would not necessarily solve this communication problem: part-time personnel, as are often found teaching at the state campuses, are but temporary employees. They are not likely to have any particular commitment to the institution. Involving them in the conversation may be difficult at best. Since such an employee might be literally gone by next term, there is little leverage to force them to contribute and, quite frankly, little they are likely to be able to contribute due to the briefness of their time with the College.

There are also communication issues even at the national campus. The College is quite unique in its extreme remoteness and in the size of the community in which the College exists. Coupled with no tenure track available, a large expatriate faculty, and no possibility under FSM law to remain in the FSM except under a work permit, the College will always have a high turn-over. There are faculty who are here for a three-year contract, a brief adventure abroad, and no intention of remaining beyond that contract. Other faculty might work a contract or two, but have no long term commitment to the institution or to living out their lives in the FSM.

The result is that the College has both institutional memory problems and institutional commitment problems. Faculty who are here for a brief adventure may not share the same level of commitment to working on institutional goals. The faculty may be doing a superb job in their classrooms, but have little interest beyond the four walls of their classroom. Due to the combination of contracts that can be non-renewed without cause being given, and the restrictions of the work regulations in the FSM, the College cannot guarantee any faculty more than a four year contract. This is true for all positions at the College, administrative, faculty, and staff. The result is both a lack of commitment to the institution by some and an inability of the institution to utilize tenure track faculty as key leaders in the institution.

IB1. Planning

Timelines for the deployment of program learning outcomes will be developed and adopted by the curriculum committee during the fall of 2003. The mathematics program and the Micronesian studies program will be conducting a second annual review of their programs during the Spring of 2004. Depending on timeline negotiations, other programs will also have program learning outcomes evaluations during 2004.

Communication with the remote state campuses has gone from zero prior to the existence of Internet connectivity to extant. At this point further improvement depends in part on facilities improvement at the state campuses. Kosrae State Campus built a faculty office which permitted the campus to provide each and every faculty member with a desk and a computer continuously connected to the Internet. The result is that personnel at the national campus can be in regular contact with personnel at the Kosrae campus. Pohnpei State Campus also provides desks and Internet connected computers to each faculty member, providing communication resources for their faculty. At this point, better communication with Chuuk and Yap would require similar improvements in their facilities.

IB2. The institution sets goals to improve its effectiveness consistent with its stated purposes. The institution articulates its goals and states the objectives derived from them in measurable terms so that the degree to which they are achieved can be determined and widely discussed. The institutional members understand these goals and work collaboratively toward their achievement.

IB2. Descriptive Summary

The goals of the College are stated on page five of the 2003-2005 College catalog:

• Instill and maintain pride in the unique heritage of linguistic and cultural diversity in Micronesia.
• Promote national economic self-sufficiency/self-reliance
• Develop individual's capacity for reflection, vision, critical thinking, and analytic problem solving in relation to local, national and global issues.
• Provide a safe, healthy, and attractive environment that promotes a sense of achievement and belonging for students, faculty, and staff.
• Obtain and adequate, stable for of financial support for COM-FSM.
• Demonstrate fiscal, management, and programmatic accountability.
• Provide relevant, up-to-date, quality, technical, and vocational degree and certificate programs, continuing education activities, and services enhanced by state-of-the-art technology to facilitate satisfying careers or transfer to schools in any geographic setting.
• Attract, employ, and retain highly competent faculty and staff who pursue professional development and contribute to a positive institutional culture which encourages honesty, appropriate moral values, and respect for human and cultural diversity.
• Establish partnerships with educational institutions, national and state governments, businesses, and other community agencies to identify and address common goals.
• Conduct necessary and appropriate research.

In the Performance Based Budget Institution Level and Program Outcomes and Outcomes Measures Fiscal Year 2004 (PBB 2004) document the institution reported five institutional level outcomes that were measured by performance indicators. The five institutional outcomes are effectively another set of goals for the College:

• The College will provide educational opportunity to all qualified students within the FSM as demonstrated by the diversity of students representative of the demographic make up of the general population and ethic groups within the FSM.
• Students will be academically successful as measured by retention rates, completion rates, persistence rates, and graduation rates by College major system-wide
• Students will demonstrate their readiness for further studies as measured by transfer rate to other educational institutions of higher learning.
• Graduates will demonstrate their job readiness and competency through gaining employment within six months of graduation.
• The College will maintain the high standards of institutional capacity and educational effectiveness as demonstrated by maintaining its accreditation with the Western Association of Schools and Colleges.

While the above are forms of overall institutional goals, programs also have goals. The goals given in the Strategic Plan 2001-2006 and those in the Performance Based Budget Institution Level and Program Outcomes and Outcomes Measures Fiscal Year 2004 (PBB 2004) are not identical. The table further below is only a small sample to show the shift in the goals and the focus of the goals between the Strategic Plan 2001-2006 and the Performance Based Budget for Fiscal Year 2004 that was produced in May, 2003.

There are other difficulties in mapping these two documents against each other. Items that were goals in the Strategic Plan are, in some instances, demoted to an activity in the later document. There are more activities in the PBB 2004 document than are listed below. In some instances the Strategic Plan reflects the top-down heritage that created that document with the Director of Research and Planning central to the creation of that document. The Performance Based Budget 2004 was developed in consultation with the program heads and reflect their work on developing program outcomes for their respective divisions. Thus there is far more ownership in the Performance Based Budget document.

This table is not meant to replace a more careful comparison of the two documents by a future Director of Research and Planning.

Note that program level goals are dynamic and continue to evolve. When the mathematics program found that the program goals for developmental mathematics and college level mathematics were essentially the same, the division opted to consider all of their math courses as part of a single program. The original division was artificial in that faculty regularly teach across the two areas. The result was a compression from five program level goals to four program student learning outcomes banks. That particular division has since mapped their Performance Based Budget assessments by indicators against their program level student learning outcomes that are assessed using an external instrument and a sample of students.

The above table represents the continued development and evolution from an indicator based system of assessment to a student learning based system of assessment in a single division.

Description of Strategic Plan Development

Institutional goals and purposes are captured in the strategic plans that are redeveloped on roughly a five year basis. Although the Strategic Plan 2001-2006 was developed primarily by the Institutional Research and Planning Office, the modified plan found in the Performance Based Budget for Fiscal Year 2004 was developed with extensive input from the program leads. Members of the College and other constituencies all have had input, through various mechanisms, into the plans of the College found in the Performance Based Budget. Thus these plans are not unknown to the members of the College, they typically represent their collective thinking. Through a variety of formal and informal mechanisms, these goals and purposes are then communicated back to the members of the College.

There remains work to be done in shifting this process from a present orientation of looking at indicators to the use of measurable student learning outcomes. Much of this data is couched in the language of numbers of students in sections, pass rates in courses, and other spot indicators.

Utilization of measurable student learning outcomes at the program and institutional level will require the development of new methods and instruments of measurement. The College is only beginning to engage in an internal conversation of what measurable skills a student will carry away from the College at a program and institutional level. Skills at the course level are well defined by course outlines grounded in measurable student learning outcomes that are developed by the instructors who teach those courses.

IB2. Self Evaluation

Performance Based Budget 2004

This document was prepared by the departing Director of Research and Planning in coordination with a small but representative team of lead administrators and line administrators at the college. There are specific goals and performance targets that the College has committed to meeting. The departure of the author of this document prior to a full year's cycle of living with the document and prior to any assessment demanded by the document leaves the College in a difficult position.

The College must report on each and every indicator, yet there is probably no one other person who fully comprehends this document in its entire scope and breadth. Any new Director of Research and Planning will have to become intimately familiar with this document. The new Director of Research and Planning will bear the primary responsibility for coordinating the follow-on reports to this document.

The upside is that the Performance Based Budget for Fiscal Year 2004 was developed out of input from those who are leading programs. Each program or unit lead had input on the contents of the document. With many of the same leads still in place, each lead should be able to handle evaluation of the goals they specified. This argues for the importance of a decentralized assessment system.

Bachelor's Degree in Business

References to developing a Bachelor's degree in Business are made in the Strategic Plan and in the Performance Based Budget 2004. The College, however, is now aware that taking a second program to four year status would require full accreditation with the Accrediting Commission for Senior Colleges and Universities. The College will have to revisit these goals with this new knowledge in mind.

Micronesian Studies Program

In the Strategic Plan for 2001-2006, under the single goal "Deliver a social science curriculum that meets the needs of FSM citizens," objective two was to "successfully implement the Micronesian Studies Program" (Strategic Plan 2001-2006, page 40). Although the Performance Based Budget for Fiscal Year 2004 did not use the term "objective," the Micronesian Studies Program was omitted from the activities of the Social Science division. The cause for this omission is unclear and does not reflect the absence of this program. The College presently offers an Associate's degree in Micronesian Studies.

IB2. Planning Agenda

Growth from Strategic Plan to Performance Based Budget and Beyond

Any future Director of Research and Planning will have to thoroughly understand the changes made in goals and activities between the Strategic Plan and the Performance Based Budget 2004. The later document represents part of an evolution from an indicator, institution centered perspective, to a student learning outcome, student centered perspective. This evolution is not yet complete as some of the goal and outcomes in the Performance Based Budget are still being measured by indicators as opposed to outcomes.

Assessment of goals specified in the Performance Based Budget will have to be done by the program leads who helped generate those goals.

Bachelors in Business

The College will have to revisit the goal of developing a Bachelor's degree in business with the new knowledge that this would require a change in accreditation. This discussion will begin within the business division and the curriculum committee. Decisions made will have to reported for advice and consent from the cabinet and the Planning Council. Note that the proposed process begins internally and then reports results to the Planning Council. This would allow the Planning Council to meet annually to consider changes being proposed at the strategic level at the College.

Micronesian Studies Program

The Micronesian Studies Program should be included in the activities of the social science division in the Performance Based Budget for Fiscal Year 2004.

IB3. The institution assesses progress toward achieving its stated goals and makes decisions regarding the improvement of institutional effectiveness in an ongoing and systematic cycle of evaluation, integrated planning, resource allocation, implementation, and re-evaluation. Evaluation is based on analyses of both quantitative and qualitative data.

IB3. Descriptive Summary

The overall effectiveness and evaluation of programs or institutional sectors remains an area in which improvement can occur.

Some areas regularly evaluate themselves, such as the Learning Resource Center. Through surveys and indicators, the Learning Resource Center evaluates the extent to which it is meeting or not meeting the needs of its constituencies and then modifies its plans to better meet those needs. The Learning Resource Center has gone from an institutional weak point in 1994 to an institutional strength at present.

Evaluation of faculty occurs on a regular basis, both by supervisors and by students. This data is then shared with the affected faculty member in order to assist them in their own self-evaluation.

While some areas regularly evaluate themselves through established instruments or processes, other areas are less well evaluated. The result is often discussions as to the effectiveness of a program without clear data.

One example of this is the Intensive English Program (IEP). The IEP was begun in 1996 as an English as a Second Language program run at the state campuses. The program was designed to replace a remedial English program that suffered from an atrocious loss of students.

Although the IEP assisted in retaining students, questions have arisen at the national campus over the perceived uneven quality of the students coming from the different campuses. Some instructors feel that there are quality problems in all four state campuses. The difficulty with the discussion is that there is little in the way of non-anecdotal data to look at, nor any comparison data to equivalent programs at other institutions.

A study of the first IEP students at the Pohnpei campus in 1996, that had unavoidable design flaws, suggested that retention and subsequent graduation rates were higher for IEP alumni versus students in the remedial English program. This study was unable to come to any statistically significant conclusion as the sample sizes were too ultimately too small.

Part of the complication was a high loss rate of students in the so-called control group who had taken remedial English courses instead of enrolling in the IEP program. There were too few students left to produce statistically significant comparisons of student performance in College level English courses. Other complications included the non-random nature of the study.

Virtually all of the IEP students were Pohnpei state students who lived at home and returned to their home environment and mother tongue at night. The students in the remedial English courses were principally Chuukese, Kosraen, and Yapese students (students from remote islands) who stayed in the College dormitory. These student were returning to a multi-lingual environment in which English provided a potential medium of common communication.

Ultimately the higher apparent retention rate for the IEP students drove its adoption.

Initially students were placed directly into college level English post-IEP. Instructors in those courses noted that many of these IEP students were not capable of handling college level reading and writing courses. As a result, students were later placed into what used to be remedial English courses post-IEP. This has led to questions as to whether the IEP is performing as designed or whether it could ever perform as designed. Yet discussions remain at the level of opinion and anecdotal observation in the absence of comprehensive effectiveness data or formal studies.

Another complication with this program is that, due to the organizational structure of the College, no one other than the Director's of the state campuses and the President of the College has any authority over the IEP programs. When the chair of the Language and Literature Division was asked to be the coordinator for the IEP programs, he rightfully refused noting he would have no authority over nor input with respect to key aspects of the program including hiring, number of students admitted at each campus, and local curricular choices. These are all issues within the program and the program as implemented clearly does not meet the current American Association of Intensive English Programs standards (AAIEP Standards).

Evaluative studies are often driven by anecdotal observations. As a result, many studies are done outside of the office of research and planning. When a new placement system that was introduced in 2000 appeared to be non-predictive of success, an informal study was undertaken by the chair of the division (Unpublished internal document. No longer extant). The result showed that the new system was no better than random at placing students. The anecdotal observations of the developmental math instructors were confirmed: students were in the wrong courses. Some students were placed to high and some too low.

This led to the development of a new placement instrument in 2001. The instrument was piloted in Spring 2002. A study of the results showed high rates of student success. Coupled with the observations of the instructors involved, the new system was deemed a success.

The instrument was used in Fall 2002 and the results were studied again later that Fall. Preliminary results of that study confirm the test is placing students at a level in mathematics at which they can succeed. There was also the suggestion of a slight underplacement of students in the Fall (COMFSM [Math] Placement Study Fall 2003. http://www.comfsm.fm/~dleeling/math/placement/placement23.html). The mathematics division is now making minor modifications in the instrument to correct the underplacement effect. Modifications in the process and timing are also being done.

In a small two-year college the task of evaluating effectiveness, following up on anecdotal data, and redeveloping systems will often depend on individual units engaging actively in self-study.

When the business community indicated that the accounting graduates did not have the skills to be effective employees in local businesses, the business division undertook the development of a third year in accounting program to provide the necessary skills.

When the need arose to develop a social sciences program that would focus on the cultures and traditions of Micronesia, the social science division developed the Micronesian Studies program.

When the dormitory needed evening opportunities for recreation and shopping; recreation and maintenance teamed up to provide bus service three times a week to town for the dormitory students.

Each unit, formally or informally, identifies needs, assesses solutions, and implements improvements on an ongoing basis.

All units, however, could benefit from better data on the effectiveness of programs implemented.

IB3. Self Evaluation

The standard implies a body that an assessment mechanism exists. The assessment mechanism that was deployed by the College in 1997 was the Institutional Effectiveness committee. This committee was tasked with, among other things, looking at the processes began as Program Health Indicators two years earlier.

The Institutional Effectiveness Committee and its successor the Assessment Committee would have periods of regular meetings punctuated by long periods of dormancy. More fundamentally, the membership of the committee was not the same as the Curriculum Committee. While the Institutional Effectiveness committee included members of the Curriculum Committee, the committee did not duplicate the membership of the Curriculum Committee in full. The result was that analysis of assessment and data based decisions made in the Institutional Effectiveness and later the Assessment Committee were not being fed back into the Curriculum Committee. During dormant periods assessment efforts tended to wane.

Records from 1997 and earlier are sketchy at best, only some of the minutes remain extant. Then Director of Research and Planning Marie Abram was not on board during the critical 1995 workshop in which the division chairs first learned about program health indicators.

In 1998 Director Abram left the College. This provided an opportunity for the College to send non-administrative faculty to a critical conference on assessment and student learning outcomes led by James Ratcliff, Director National Center Post-Secondary Teaching and Assessment, on Guam during May of that year (Transcriptional Report from the Conference on Institutional Assessment, 1998, http://www.comfsm.fm/comfsm/ladera.html). With turn-over at the Director of Research and Planning, this conference would pass information on assessment to rank and file faculty. This concept, of decentralizing assessment and moving assessment closer to the students, to the division chairs and their faculty is a central theme in this document.

During 1999 Greg Myers would take over as the Director of Research and Planning. Director Myers would come on board without the experience of having taken the institution through a self study and self study report. The new director would face a heavy load of daily tasks as well coming up to speed on the recommendations made in 1998.

For reasons this author cannot ascertain, the Institutional Effectiveness committee appears to have gone dormant in the spring of 2000. There exists the possibility that intense work associated with the planning council and development of the mission statement and college goals caused the Institutional Effectiveness committee to fail to convene. Institutionally, there are so many tasks delegated solely to the Director of Research and Planning that the position becomes a critical single point failure point in the college system.

The committee would be reborn in the fall of 2000 as the Assessment Committee. Whether other meetings occurred in 2000 has yet to be documented by this author. As of the writing this report there is no indication there were other meeting during this year.

At the 20 February 2001 meeting of the Assessment committee the committee was told to be prepared to discuss the purpose of the committee, with one definition offered, "Gather data and analyze all COM-FSM programs and recommend actions for improvement." The minutes for this minute appear to be missing except for a single last page including an item six. The committee membership included six directors, four of whom have since left the College (Email address list for committee members, 20 Feb 2001).

In 2002 the committee members were all provided with a copy of the recently approved Strategic Plan 2001-2006. The minutes also noted the development of a distance education plan for the College. Attendance, however, was not good. The meeting on 24 January had only 10 of 21 members present, although one absence was an off-island member. Many committees include members from the remote state campuses. Their participation has always been constrained by the complications of distance. In the past attempts were made to use the PeaceSat radio satellite network to include state directors in critical meetings. There was also a brief attempt in the 1999 time frame to use instant messenger services to include remote state campus personnel in a meeting. A transcriber typed the conversation in the meeting in real-time, while in another window the remote state campus personnel could respond or chime in. The transcriber would have to stop to read the input from the remote state campus. Neither system worked sufficiently well that either experiment was continued.

Today the only inclusion of remote state personnel in meetings in via the email traffic prior to a meeting during which issues that will come up in the meeting are discussed. This is not only imperfect, but it excludes the state campus member from the actual decision making vote on an issue.

During 2002 the Director of Research and Planning would depart. Prior to his departure he would form four committees to work on the self study. These committees would meet on three occasions in mid-2002 (Personal conversations with a participant). In August 2002 an Acting Director of Research and Planning would call a college-wide meeting and call for the formation of four new committees to work on the self study. The four new committees were not in any way related to the former four committees. Turn-over in a critical position, possibly unavoidable turn-over, once again impacted an assessment effort of the College.

Attendance is also impacted by the sense of institutional commitment or lack thereof. In an institution where all faculty are on limited term contracts that can be non-renewed with 60 day notice and without cause being given for non-renewal, full-time faculty are in the same position as a part-time faculty at other institutions. Coupled with the remote location of the College, a pay scale half that of equivalent positions in the United States, and the difficulties for many expatriates in living on small island, obtaining a sense of institutional commitment is difficult. This has translated, in part, to poor attendance for some committees. Assessment committee has had its attendance problems.

During 2002 Susan Moses began presentations and one-on-one training sessions with the academic chairs on student learning outcomes. This is cited in the above table in part because it constitutes evidence that work on assessment was continuing amongst the faculty during a time when the assessment committee was dormant. This is further evidence that decentralized assessment can proceed even in the absence of a functioning assessment committee. In section IB6 an alternative structure for assessment that fits with this new decentralized reality will be proposed.

In spring 2003 the Assessment committee reformed. The Acting Director of Research and Planning, however, would depart in early August 2003.

Complicating the ability of these committees to function was the perception by 2001 that the committee work was unfocused and primarily a talking session for the then Director of Research and Planning (Personal observations of a member and author of this section). At the 07 July 2001 meeting of the Assessment Committee only six of seventeen members were present. The following meeting had only five of seventeen members. Even once the term started attendance rose to ten of nineteen appointed members.

Assessment committee meetings in 2001 and 2002 left one participant with the impression that they were addressing the need of the moment for the Director of Research and Planning (Personal observations of a participant). A sense that the committee work would lead to an annual cycle to regular assessment was not evident.

Communication of decisions made within the assessment committee to those affected by the decisions was not accomplished. The 10 March 2003 meeting of the Assessment Committee passed a resolution that the development of standards for program level outcomes would be the responsibility of the Assessment Committee. This pronouncement, however, lacked input from critical non-members who were, at the same time, developing program learning outcomes based on dialog in their divisions and based on the recommendations of national organizations in their field. This isolation and detachment of the assessment committee from planning and deploying committees such as curriculum is addressed in the planning section below and in the planning section for IB6.

IB3. Planning Agenda

Although the obvious recommendation might be to recommend that the Assessment committee meet regularly, ultimately even regular meetings will not solve the fundamental problem that planning and deployment of courses and programs occur in the Curriculum Committee while assessment is reported into the Assessment Committee. This builds into place a permanent separation of planning and deployment from assessment. If the assessment cycle is to be closed, assessment must be reported where planning and deployment occur: the Curriculum Committee.

Another factor is driving assessment away from an assessment committee led by the Institutional Research and Planning Office. This factor is that program assessment is being done out in the academic divisions. Where student achievement data was the forté of the Director of Research and Planning, the new assessment regime is being built in each division. This decentralization of assessment, and the planning recommendations that these appear to entail, are detailed in the IB6 planning agenda.

The issue of a lack of system-wide assessment of the Intensive English Program, as mentioned above, will have to be tackled by the President's office through the Institutional Research and Planning Office due to the peculiar organization of the College wherein neither the Vice President for Instructional Affairs nor the academic division chairs have any authority over academic programs in the state campuses.

There is a need to find a way to preserve records and minutes of meetings. The effort to find minutes for the Institutional Effectiveness committee and the Assessment committee involved searching file cabinets, notebooks, hard drives, and floppy disks, all in multiple locations.

One option would be to post all minutes to the web server for the college, deleting out those materials that are strictly internal or are governed by privacy policies and regulations. Web server based storage would provide access to these materials on all six campuses and appears to be a stable and useful way to store information over the term. Unlike paper based products, web pages do not physically degrade nor are they easily lost or misplaced. As long as the item remains linked to the rest of the College web site, an Internet search engine can find the item with relative ease.

New efforts must be made to find ways to include the active participation of remote state campus personnel assigned to committees. The college continues to work toward an infrastructure that would support full video conferencing. This will be covered in the section on technological resources.

IB4. The institution provides evidence that the planning process is broad-based, offers opportunities for input by appropriate constituencies, allocates necessary resources, and leads to improvement of institutional effectiveness.

IB4. Descriptive Summary

The planning process at the College is driven by a wide variety of formal and informal structures.

Ongoing strategic planning and guidance is provided by the planning council, a council consisting of members of the College system at all levels, members of the community at large including traditional leaders, and members of government (Appendix A, Strategic Plan 2001-2006). The present College of Micronesia-FSM Strategic Plan 2001-2006 was developed by the planning council beginning in 1998 (Page 2, Strategic Plan 2001-2006).

Degree granting academic programs at the College also have a program advisory council consisting of members of the College who work under that program and community members who work in that field. Business/Accounting, Computer Information Systems, and Tourism and Hospitality program advisory councils have been formed and have met to provide input on those degrees. These councils are not yet formed for every degree, divisions in charge of degrees without councils are being tasked with forming councils.

Planning also occurs as a direct function of the Director of Research and Planning.

Other input comes from a variety of committees that tackle issues on a more tactical level. Occasionally, however, these committees drive planning on a larger scale. Data that indicated the students at the College were unfit in terms of body fat is one factor driving the development of a physical education program (Report on the Reasons for PE 101j, Body Fat Data ). The development of this program came from internal processes at the College. The College recognizes that physical education alone cannot be shown to impact body fat levels. In a nation with high rates of life style diseases such as diabetes and heart disease, physical education is one of many approaches that educational institutions ought to implement.

Planning can also be driven by outside constituencies. The College is the only post-secondary institution in the Federated States of Micronesia. Hence when the national government calls on post-secondary institutions to train citizens in tourism, marine sciences, and agriculture, it is the College that must respond. Each of these calls arose from national or state strategic plans, and each call led to the development of a program at the College. The most recently added was the Hotel and Restaurant Management program.

Planning and the processes that develop new programs also occurs informally as members of the College interact with the broader community. Individual members of the College often see community needs and as a result develop programs to meet those needs. These programs are then routed through formal channels for approval, but the start of the planning process was a conversation held off campus in the evening between a member of the College community and the broader community at large.

Once a need enters the system, resource deployment depends on the personnel tasked with meeting that need. For the most part the College develops and deploys programs or processes to meet the need.

There are limitations on resources at the College. If meeting a need has high costs and the demand is low, the College might not be able to meet that need.

In some cases where the demand is low and the cost is high, but the benefits are real, the process will proceed but at a slower and more affordable pace. For a number of years there has been the realization that our software systems in the Office of Admissions and Record, the Financial Aid Office, and the Business Office need to be integrated. The original designer of the database used in the Office of Admissions and Records since 1995 was informally informed of this potential need by an outside consultant in 1997. (Oral conversation between Dana Lee Ling and Paul Young, 1997). Software that could handle all three areas has proven prohibitively expensive to implement, and may not remove all double entry of data for the units involved. Not all of this is within the control of the College: the Financial Aid Office uses required United States Department of Education software for their financial aid work. This software does not currently appear to be designed to integrate with other systems software. While the demand for integrated systems is low, there would be a real benefit both to students and to constituencies such as the FSM national and state scholarship officers.

Other limitations involve the difficulties the College faces with long term financial planning. The College remains dependent on an annual allotment from the congress of the Federated States of Micronesia. This allotment is subject to change and introduces uncertainty in the College's long term financial planning.

Another major source of funding is through the United States Pell grant to students. This funding is dependent on the continuing inclusion of the citizens of the Freely Associated States in the Pell grant program. The United States Congress has made no long term commitment as to the inclusion of the citizens of the Freely Associated States. For further and more current information on this, please refer to the section on Financial Resources.

While at the time of the drafting of this document the College lacks a Director of Research and Planning, key administrative sectors of the College are aware of the need to start planning for the 2007 - 2012 Five-Year Strategic Plan.

IB4. Self Evaluation

The college planning process has been broad based and has offered opportunities for input by appropriate constituencies. Resources have been and will continue to be allocated to the planning process, both strategic plans and tactical plans.

Plans have changed the programs offered at the college, but evaluation of the effectiveness of those programs remains to be done. For example, the nation has placed a high priority on local agriculture, hence the College offers a degree in agriculture. The number of students who choose this program is small, the number who graduate is also small. The College is responding to a call in the national plans, but whether the students want to be farmers is another matter.

More recently the effort to develop an integrated Performance Based Budget has led to modifications of the Strategic Plan. The Performance Based Budget is to be based on outcomes, while the 2001 - 2006 Strategic Plan was based on objectives and health indicators. This disjunction meant that in order to submit an acceptable Performance Based Budget, the College had to reword, and in some cases restructure the strategic plan.

This happened during a period of transition for the College. Both the Presidency and the Director of Research and Planning positions were in transition at one time or another during this effort. The upshot is that the May 2003 Performance Based Budget Institution Level and Program Outcomes and Outcomes Measures Fiscal Year 2004 included a de facto rewrite of the strategic plan.

IB4. Planning Agenda

The College will have to continue to work on bringing future performance based budgets into alignment with the effort to assess programs based on student learning outcomes as opposed to student achievement data. This effort will have to be coordinated by the Institutional Research and Planning Office. Wherein this effort impacts the strategic plan, the College will have report on that impact for the advice and consent of the planning council.

IB5. The institution uses documented assessment results to communicate matters of quality assurance to appropriate constituencies.

IB5. Descriptive Summary

This standard was taken as referring to communication with external constituencies. The only formal mechanism by which the College communicates results at this time is via an Annual Report. The annual reports were begun in 1994, with the first one written by the Director of Research and Planning (Personal recollection of Executive Assistant to the President, 1994 Annual Report). The most recent annual report was a combined two-year annual report for October 1, 1999 to September 30, 2000 and October 1, 2000 to September 30, 2001 (College of Micronesia Annual Reports). The most recent annual reports were completed by the College President (Personal communication with Executive Assistant to the President).

The annual reports began as a four page highlighting of events over the past year. The last report, for fiscal years 2000 and 2001, was twenty pages and included reports on instructional programs, program costs based on student seat costs from divisional budgets, enrollment statistics, program evaluation based on completion rates and graduation rates, student services, accreditation, finances, and facilities (College of Micronesia Annual Reports October 1, 1999 - September 30, 2000 and October 1, 2000 - September 30, 2001).

The Annual Report was distributed to national and state leadership, both political leadership and educational leadership. The report is also a requirement of the FSM National Government.

Communication with the nation as a whole is complicated by the distances, a lack of a developed media infrastructure, and language barriers. The only media that is distributed across the FSM is a single newspaper produced on Pohnpei in English. Radio is limited to a single island, and then only some areas of the island can receive the radio signal. Cable television is presently limited in distribution, both by island and on the islands on which it is installed. All of these factors contribute to a difficult in communicating.

Some divisions are posting student achievement data and student learning outcomes assessment results to the College web site. Although Internet access remains limited, and the location of this information is not obvious, this does represent an effort to expose internal assessment data to a broader global community (Division of Natural Science and Mathematics assessment web portal).

IB5. Self Evaluation

The Annual Report is the only broadly communicated output of the College. The amount of assessment information in it is minimal and is limited to broad student achievement data. This report was not completed for fiscal year 2002: October 1, 2001 to September 30, 2002. According to a former President of the College, another report is due within 120 days of October 1, 2003 for fiscal year 2003.

As few units at the College have yet assessed program or degree level student learning outcomes, the Annual Report cannot at this point report on learning to external constituencies. At this point the Annual Report can only report student achievement data. For those programs that have initial student learning outcomes assessment systems and data, the systems and the data can be reported.

IB5. Planning Agenda

The College will complete an annual report for fiscal year 2003. The College President will take lead responsibility for the production of this report.

IB6. The institution assures the effectiveness of its ongoing planning and resource allocation processes by systematically reviewing and modifying, as appropriate, all parts of the cycle, including institutional and other research efforts.

IB6. Descriptive Summary

At the center of the effort to produce a systematic cycle of process review is the Director of Research and Planning and the Assessment Committee.

As the effort to shift from program or objective health indicators to program level and institutional level student learning outcomes has only begun, systematic review of these higher level outcomes have yet to occur.

Two issues have impacted the ability of the Assessment Committee to function. The first is a lack of data, even with respect to indicators. The College used manual record systems up until 1995. In 1995 the College began using a home grown database for tracking basic student indicators, primarily grades.

Researching indicators has involved being able to construct queries on the database.

The second issue has had an impact on the ability of the Director of Research and Planning to construct those queries: turnover in the position. The College now has its fourth Director of Research and Planning since 1994. Each of the last three Directors has either been new to the College and to Micronesia, or relatively new to the College system. Setting up and institutionalizing cycles of planning and assessment requires multiple years of effort.

Turnover in the lead position, and the only College employee specifically tasked with research, has meant continuous change in the direction and thrust of planning and assessment. Each new director requires on the order of a year to comprehend the annual cycle of research and assessment needs. The director then usually needs a second year to implement changes, modifications, and additions. Obtaining buy-in from units also requires time and effort. By the time a regular cycled of planning and assessment is in place, the director has often moved on and a new director with new ideas and desires has come on board.

Another factor in the past has been that the paucity of data coupled with unfamiliarity with who actually has the necessary data, or what data is actually held in a given system, has led to a nonproductive blame game. There has been a tendency to blame personnel or systems the inability to generate data. Or to make demands on units that handle data which the unit cannot meet. Data requests have been made in the past without regard to the present load being carried by the unit or to the amount of labor involved in the data entry necessary to generate the desired data.

Stability, continuity, and institutional memory in the director of research and planning position, possibly couple with expansion of the position into a multi-employee office, would benefit the ability of the College to close the circular cycles of planning, implementation, assessment, and redevelopment.

IB6. Self Evaluation

Although both the most recent Acting Director of Research and Planning and the previous Director spoke of a need to develop a published research calendar for the Institutional Research and Planning Office, no formally published calendar is extant at this time.

IB6.Planning Agenda

A proposal was made in September 2003 to integrate the functions of the presently dormant Assessment committee into the existing and active Curriculum committee and Student Services committee. This was as a result of the realization that to close the academic assessment loop that is made up of curriculum planning, development, approval, deployment, assessment, and redevelopment, the Assessment committee would have to be identical to the Curriculum committee in composition.

The loop is not currently closed by design: curriculum and programs are discussed in and initially approved by the curriculum committee. The assessment is reported into the Assessment committee. This creates a physical separation of the events that no amount of communication effort can fully bridge.

The proposal would also include having the student support services assessments reported into the existing student services committee.

At the core to the effort to improve courses and programs is assessment. Assessment permits data and fact driven decisions at the course level, program level, degree level, and institutional level. These assessments are assessments of student learning outcomes. Yet at the course level, and the program level, as envisioned in Language and Literature, Education, Social Science, and the Natural Science and Mathematics division, student learning outcome assessment will be done either by the faculty or by the chair.

This proposed change helps redistribute load away from an over burdened central researcher and planner, without necessarily overloading the chairs. Assessment would move from a centralized research office out into the corridors and offices of academia.

Student achievement data - referred to in the past as Program Health Indicators (retention, graduation data, and the like) - would also be reported to the Curriculum committee. The curriculum committee all ready has representation from each academic unit.

This proposal would mean that the Director of Research and Planning would sit on the Curriculum and Student Services committee as a voting member. The Director could also report data relevant to assessment and assist units in making data based decisions.

The existing separate Assessment committee would no longer exist under this proposal.

This proposal should also address an institutional weakness in the institution's difficulty in retaining Director's of Research and Planning capable of performing the many tasks the College asks of them. The Director of Research and Planning is presently responsible for organizing strategic planning, heading the Planning council, institutional research, budget development, performance based budget reports, grant writing, accreditation liaison, sponsored programs, and the annual IPEDS report. The Director also plays a key role in the production of the annual report.

Due to issues such as the remote location of the College, budget constraints that in turn constrain salary, and the complexity and load of the job, the College has seen instability in the Director of Research and Planning position.

At the salary offered the College either gets people who are not competent at every single one of these of these skills, or the College attracts people who are competent and then they do not stay long as they can obtain better paying positions elsewhere. Directors have remained on board no more than two to three years. This creates instability in the assessment cycles as these cycles can take many years to complete.

Neither the Curriculum committee, nor the Student Services committee have had the same stability over time difficulties. Both are headed by Vice Presidencies, positions that have remained stable over time at the College.

The College's current structure consists of three committees:

This proposal would result in there being two committees with assessment integrated into the existing committees:

Another problem that accompanies turn-over at the Director of Research and Planning position is that each departure carries away critical information. The Director of Research and Planning is tasked with attending crucial conferences and workshops on assessment and accreditation. With the departure of a Director, much of the training and knowledge departs as well.

This is evidenced in part by what happens in the time gaps between directors. In 1998 the College sent a combined administrator-faculty team to an assessment conference in Guam (Transcriptional Report from the Conference on Institutional Assessment, 21 May 1998 to 23 May 1998, Ladera Tower, Guam. http://www.comfsm.fm/comfsm/ladera.html). The team consisted of the Director of Academic Programs and three faculty members. At that time there was no Director of Research and Planning, otherwise the College would likely have sent only the Director of Research and Planning and the Director of Academic Programs.

Five years after this conference the Director of Academic Programs has moved on, and the College has had a Director of Research and Planning come and go along with an Acting Director of Research and Planning come and go. Yet all three faculty members are still on board, carrying the knowledge imparted by that conference.

As noted above, the College is located on a remote island and can only offer salaries that are best described as limited compared to larger or better endowed stateside institutions. If the College succeeds in attracting top flight talent to a position such as the Director of Research and Planning, such a capable person is likely to move on - either locally or internationally. The other possibility is that the College does not attract the highest calibre of talent, and the College winds up exercising an option to not renew the initial contract. Either way the Director of Research and Planning is a position that is subject to turn-over. With each turn-over another critical hunk of institutional memory is lost.

To help distribute institutional memory, a recommendation was made to separate the duties of the Accreditation Liaison Officer from the Institutional Research and Planning Office. The Accreditation Liaison Officer position would be moved back to faculty, where it has resided in the past. The proposal would not limit the Accreditation Liaison Officer position to faculty - the office could be held by other positions outside of Planning and Research.

There would be natural advantages to the Accreditation Liaison Officer being a faculty member. Faculty would be in the best position to share accreditation knowledge with other faculty on a daily basis. Under the new standards, student learning and its assessment are the core to accreditation. The Accreditation Liaison Officer would be in an ideal position to share knowledge of how other schools are handling student learning outcomes, program assessment, and other areas of growth and change in other schools in the Western Association of Schools and Colleges.

A faculty based Accreditation Liaison Officer who taught classes would also be seen as a teaching colleague, as opposed to an administrator. Despite the best efforts of institutions and people, those who teach in a classroom will likely always distinguish between others who teach and those who administer without ever teaching. A faculty Accreditation Liaison Officer would be able pilot new concepts in their own classrooms first, and then bring to other faculty a proven-to-work prototype rather than merely a hypothetical concept. The College is small enough that the Accreditation Liaison Officer could continue to teach and handle their accreditation duties.

The likelihood that the College would lose both an Accreditation Liaison Officer and a Director of Research and Planning at the same time would be reduced if the Accreditation Liaison Officer were separated. Currently the College loses an Accreditation Liaison Officer at the exact same instant it loses a Director of Research and Planning.

IB7. The institution assesses its evaluation mechanisms through a systematic review of their effectiveness in improving instructional programs, student support services, and library and other learning support services

IB7. Descriptive Summary

Program evaluation has meant, until only recently, a review of student achievement data. Although this data has helped some units improve the effectiveness of a unit, the evaluation mechanism itself, the student achievement data, has not been assessed. The effort in the academic programs is to define program student learning outcomes, to evaluate those outcomes, and then later to examine whether the assessment methods used were effective. The later is a meta-analysis that will best be done after a couple of evaluation cycles.

The faculty, through the auspices of the curriculum committee, did take a look at the faculty evaluation system. The present form used by personnel which involves being rated on a scale from one to ten has been viewed by faculty as not only having nothing to do with effectiveness but with being irrelevant to faculty. Some, however, view the proposed replacement system as equally irrelevant to improving effectiveness. Others feels improving the effectiveness of faculty is not the goal of the evaluation form or the faculty evaluation by supervisor system and prefer to handle improving faculty effectiveness outside of the realm of supervisor evaluation. There remains a division of opinions as to whether faculty evaluation should be used as a stick to drive improvements in effectiveness or whether evaluation should remain summative while other mechanisms are deployed to improve instructor effectiveness.

IB7. Self Evaluation

Although the College has been conducting program review based on student achievement data, and, in some cases, advisory councils, evaluation from a program student learning outcomes perspective has only just begun. This effort represents a de facto assessment of our present evaluation mechanisms and a decision that these evaluations could be improved by student learning data.

IB7. Planning Agenda

Although a number of units have been working on program student learning outcomes as part of their program evaluation effort, the Vice President for Academic Affairs circulated two policy statements in September 2003 to formalize this process. The two polices, Policy on Instructional Programs Evaluation and Action Plan for the Program Evaluation for School Year 2003-2004 p