COMET Admissions Examination Statistical Summaries Spring 2008

09 June 2008 Notes: This document should be taken as draft work in progress. Consider these results to be preliminary and subject to revision and correction at a later date.

In this statistical summary those who took the College of Micronesia-FSM Entrance Test (COMET) are referred to interchangeably as "students" and "candidates." There is no distinction intended by these two terms.

Basic Statistics

The COMET admissions examination consisted of four main sections. The four main sections included an essay, vocabulary, comprehension, and a math section consisting of four subsections. The basic statistics for these sections and subsections are presented in the table one below.

Table 1: Basic statistics

Essay is the essay score out of 50.

Voc is the vocabulary score out of 45.

Comp is comprehension out of 48.

95 is the first math subsection and contains ten arithmetic level problems.

96 is the second math subsection and contains ten prealgebra level problems.

99 is the third math subsection and contains ten elementary algebra problems.

100 is the fourth math subsection and contains ten college level algebra problems.

msum is the sum of the math subsections and is out of forty.

Correlations

Although the first two subsections of the math test were rewritten in 2006 to require less reading ability, the math sum remains, given the sample size, correlated moderately to the essay and comprehension sections of the examination.

Table 2: Correlations

At present the college seeks to admit 400 students into associates degree programs. The college accepts the top 400 students into associates degree programs. The rank order is generated by a sum of z-scores detailed later in this report. The 0.92 correlation for the essay to the z-score reflects the 50% weight of the essay score in the z-score. The rank order is supposed to be strongly correlated to the essay performance by design. The results above support that this goal was achieved.

The moderate correlation for mathematics indicates that mathematics is linked to language skills, at the same time the mathematics score provides independent information on the abilities of the candidate. Thus the mathematics score is valuable for a more comprehensive picture of the skills the candidate possesses.

The question is often asked, would student "x" have been admitted if their math score was stronger. In other words, does the math section keep students from attaining associates degree admission. The above correlations should provide statistical assurance that the math section alone did not "sink" a student – the essay was the strongest driver of the rank order. In addition, there is a correlation between math and English, so a really weak math score for a given set of language scores may suggest the student is weaker than the language test results indicate. Again, the mathematics score is important to providing a broader picture of the student's academic abilities.

Distributions

The distribution of scores on each section is given in illustration one. The score is on the horizontal axis, the vertical axis is the number of students with that score.

Illustration 1: Frequency Distribution

With the exception of the writing section, the other sections distributed in a manner which is considered statistically reasonable. That is, the frequency (the number of students) rises with the score towards the mean, and then falls as the score continues to rise past the mean to the maximum possible score. If the distribution (the shape of the curve) is a symmetrical heap, the most common score is also the mean score as well as the median score. This distribution is sometimes called a "bell curve" and is mathematically described by the normal or student's t-distribution.

In the past, the writing section, the essay, has in performed in a statistically reasonable fashion except for the number of students who score a sum of zero. This was true again this year with 130 scores of zero. In the spring of 2006 there were 235 scores of zero for 1783 candidates (13%). In the spring of 2007 there were 144 scores of zero for 1608 candidates (9%). Spring 2008 there were 130 scores of zero for 1601 candidates (8.1%). This suggests that the percent of students scoring zero is declining.

In the spring of 2006, the most common score on the essay, except for zero, was the then maximum possible score of 40. There was anecdotal evidence that the rubric needed to be adjusted as papers that varied in quality were "maxing out" at 40. A change in the rubric permitted a maximum score of 50 in 2007. In both 2007 and the current run of the test, the right tail of the essay distribution now falls steadily from the mean to the maximum possible. With the adjustment in the maximum possible in 2006, the essay continues to behave in a statistically reasonable manner.

Graders did raise questions on whether individual essay prompts behaved in a statistically equivalent manner. The lack of machine coding of the essay prompts and scores precluded analyzing the performance of individual prompts.

Year-to-Year Mean Scores by Section

The following table provides the mean score by entrance test subsection. Note that use of the reading section was discontinued in 2006. The grammar section was discontinued in 2007. A vocabulary and comprehension section based on the Gates-MacGinitie Level AR reading tests was added in 2008. The essay section was worth only 40 points in 2006, in 2007 and 2008 the essay section was worth 50 points.

Table 3: Year-to-year mean performance by section

The essay and math subsection scores suggest that the mean performance is improving. The improvements seen could be due to improved educational processes or better specific preparation for the COMET. In either case the COMET could be seen as a curricular impetus. The college might consider broadening the content scope of the COMET to provide an impetus for schools to improve in areas beyond English and Mathematics. The natural and social sciences are areas that might be explored for content oriented sections of the COMET.

Rank order and Program Placement

The admissions board approved a rank-order formula using z-scores. In spring 2008 the formula placed a 50% weight on the essay, 16.7% weight on vocabulary, 16.7% weight on comprehension, and a 16.7% weight on mathematics. The individual weighted z-scores were added to generate a sum of z-scores. The resulting z-score was linearly transformed to produced the final COMET score.

A COMET score of 700 was approved as the cut-off for regular admission to an associate degree program. A COMET score of 300 was established as the minimum score for admission to a certificate program. Students who have scored between 300 and 700 will be eligible to enroll in a certificate program.

Bear in mind that the college now operates based on fixed resources and specific projections of the number students that the college can serve. Each site has an target enrollment number and an maximum enrollment capacity limit. Like many smaller colleges, the college opts to admit a specific number of students.

The result is that the target admissions number is a fixed number. Put another way, there is not the possibility at present of all high schools increasing the number of students attaining regular admission to the college. If one high school sees an increase in the number of students who have attained regular admission, then some other high school (or high schools) must have seen a decrease in the number of students who gained regular admission. Thus admission is competitive and is mathematically termed a zero-sum game.

High schools that are using increases in regular admissions as an indicator do so at their own peril. If another high school improves by a larger amount, then that first high school could see a drop in admissions despite real improvement in their own educational processes. The college does not recommend using the entrance test as an performance indicator. The entrance test is designed solely for the internal admissions purposes of the college. That said, school that still choose to use the entrance test as a performance measure should look at improving average performance by section. This data is reported later in this report.

The next table records the distribution by state and high school of the students into the three categories delineated above.

Table 4: Program placement by high school

Some schools in the table above provided section level class lists. In recognition that the different high school programs may have different individual goals for their students, this section level data is broken out above. The college thanks the high schools and administrators who have provided this information. Better information from the high schools allows the college to provide more accurate feed back to those schools.

Note that the table above reports the high school name using the names provided to this author by those working on behalf of the admissions board.

The average COMET score seen in table four represents a measure of the relative performance across all sections as weighted by the z-score. High school names are as in the original data, refer to table three above for clarification.

Table 5: Average COMET score by high school

A COMET mean below 400 is exceptionally weak and below 300 represents essentially random performance on all included subsections. Two high schools had an average COMET score below certificate admission level. This should be looked upon as an opportunity for improvement.

Bearing in mind that admissions is zero sum, it is not possible for all schools to score above 700. That said, schools or sections with averages above 700 deserve a "well done" for their strong performances.

While the individual high school scores are of interest to individual schools, state educational leadership might consider the mean performance of all of the schools in their state.

Where a performance is strong, no one effort can be credited with the success. Where a performance is weak, no one lack can be faulted as the cause of the poor performance. Education is a set of canoes with many people paddling. Students, parents, teachers, administrators, state leadership, and special programs should all share in the credit where credit is due, in the blame where performances are weak.

Essay section

The writing section is considered by both the language and literature division and by those from other divisions who mark this section to possibly be singularly indicative of the capacity of students to academically succeed at the college – if there is such a thing as a single indicator. The maximum possible score is a fifty. Scores of twenty and under are considered to be significantly weak and raise questions as to whether these students can tackle any program that requires the ability to communicate in written English.

Determining whether one high school is statistically significantly stronger in essay rank order than another high school requires determining whether the mean essays scores are sufficiently separated.

In illustration 2, the vertical bars show the extent of the 95% confidence interval for the mean essay score. Where a bar for one school overlaps the mean for another school, the former is not statistically separated from the mean of the later. The mean is marked by the horizontal line at the midpoint of the vertical bar.

The vertical axis in illustration two is the essay score, the high schools are listed in descending rank order.

Illustration 2: Essay mean by site with 95% confidence interval for the mean

Note that the mean for NMHS is for all sections, while NMHS a1 would be for only the a1 section at NMHS. Splitting out the academic sections is important as it makes more realistic comparisons possible.

The mean essay scores suggest that Kosrae, Pohnpei, and Yap states are, to differing degrees, accomplishing the task of instructing students in the skills necessary to compose written English essays.

Looking at the individual public high schools in Chuuk indicates an almost complete lack of ability to accomplish a written composition in the English language.

An average of 20 is the equivalent result of two essay readers rating the essay as being two in all categories. Averages below 10 would be the result of being awarded a one in all categories. Below 10, the essay has errors of grammar that are so severe as to make comprehension virtually impossible, the vocabulary is limited and misused, and the answer bears little relation to the task set by the essay question. The rubric is included at the end of this report.

While some fields such as science have significant support costs, English language and literature is likely the least expensive program on a per student basis. This is not likely a failure due to a lack of funding alone.

Also of note this year are a couple of new arrival in the list of schools, Our Lady of Mercy Vocational Technical school and Southern Noumeneas High School at Fefan. Our Lady of Mercy has done well for a rookie year.

To be all too blunt, the performance each newer high school in Chuuk casts doubt on the wisdom of splitting out the high school students from Weno back to their home islands in the lagoon. Each newer lagoon high school is weaker than the original high school from which the students had been split out. This is in contrast to Pohnpei where the new high schools perform on par with or better than the original "parent" high school.

Changes in essay performance 2007 to 2008

With the essay marked against the same rubric in 2007 and 2008, year-to-year differences can be determined. Small differences are not statistically significant. With only two years worth of comparable data, the year-to-year variation for any one school cannot be known. If the same rubric is used in spring 2009, then any three-year trends are more likely to be significant than small changes in the mean.

The table is in roughly alphabetic order.

Table 6: Changes in mean essay score 2007 to 2008

Although year-to-year variation in the mean for schools is not know, changes of five or more may prove significant. For example, a study of the actual Saramen essay scores for spring 2007 and spring 2008 indicates that the gain of 8.31 is statistically significant. The drop in score for the NMHS academic sections on the essay is also statistically significant.

Mathematics section

Year-to-year mathematics subsection performance

Table five records the year-to-year performance from 2005 to 2008 on the mathematics subsections. The table also shows the averages for the top four-hundred rank order students on each subsection for 2006, 2007, and 2008

Table 7: Changes in mean math score by math subsection

Year-to-year the averages for each subsection improved slightly for all sections. For the top 400 students the averages improved in three of the four subsections, one subsection remained static. These changes are not individually significant. There is, however, an underlying trend of increasing averages. Those trends are encouraging, although an average of two on a section is no better than random.

Mean Mathematics Scores by High School

The mean score by high school for the mathematics section represents the average number correct out of forty for a high school.

A student blindly guessing on all forty questions, each with five multiple choice options, should get at least eight correct. Given that the first few problems involve basic arithmetic, scores around twelve or under are extremely weak performances.

Illustration three on the next page depicts the 95% confidence interval for the mean mathematics scores, with the high schools in descending mathematics score order. The vertical bars are the extent of the 95% confidence interval for the sample mean. Where a vertical line for one school overlaps the mean (the center line) of a another school, the first school is not statistically significantly separated from the second school's mean.

Illustration 3: Mean math sum by high school with 95% confidence intervals

Of note on the chart above are the strong performances by the academic sections at Kosrae High School and Nanpei Memorial High School. The Kosrae High School advanced a1 section was statistically separated from other schools and took first rank.

Nanpei Memorial's a1 section ranked second in the nation, with Xavier rounding out the podium finishes. Also of note is that Kosrae High School's a2 section edged out Calvary Christian Academy. Nanpei Memorial's a2 came in just behind Pohnpei SDA. All of these averages are high and encouraging.

At under 12 were both Southern Noumeneas High School at Fefan and Faichuuk High School. These performances are so close to random that one could effectively argue there would be no significant impact from the cessation of the teaching of mathematics. Shift the resources into language acquisition skills.

Mathematics Placement Data

Placement of students by high school and the program into which they were admitted is shown in the following table.