COMET Admissions Examination Statistical Summaries Spring 2007

Basic Statistics

The COMET admissions examination consisted of a grammar and vocabulary section, an essay, and a math section consisting of four subsections. The basic statistics for this sections and subsections are presented in the table one below.

Table 1: Basic statistics

Grammar is the grammar and vocabulary score out of 80.

Essay is the essay score out of 50.

95 is the first math subsection and contains ten arithmetic level problems.

96 is the second math subsection and contains ten prealgebra level problems.

99 is the third math subsection and contains ten elementary algebra problems.

100 is the fourth math subsection and contains ten college level algebra problems.

Σmath is the sum of the math subsections and is out of forty.

The essay this year was marked on a 50 point scale as opposed to the 40 point scale used last year.

Correlations

Although the first two subsections of the math test were rewritten to require less reading ability, the math subsections sum remains, given the sample size, correlated moderately to the grammar and essay sections of the examination.

Table 2: Correlations

The lowest inter-subject correlations are seen for performance on college level mathematics and English. The correlation of the 100 subsection of the mathematics examination is low at 0.28 for both the grammar and the essay section. The mean for the 100 subsection is near random, hence the low correlation. The low correlation is simply a reflection of the underlying random performance on this subsection of the mathematics examination.

Distributions

The distribution of scores on each section is given in illustration one. The score is on the horizontal axis, the vertical axis is the number of students with that score.

With the exception of the writing section, the other sections distributed in a manner which is considered statistically reasonable. That is, the frequency (the number of students) rises with the score towards the mean, and then falls as the score continues to rise past the mean to the maximum possible score. If the distribution (the shape of the curve) is a symmetrical heap, the most common score is also the mean score as well as the median score. This distribution is sometimes called a "bell curve" and is mathematically described by the normal or student's t-distribution.

In the past, the writing section, the essay, has in performed in a statistically reasonable fashion except for the number of students who score a sum of zero. This was true again this year with 144 scores of zero. Although in the spring of 2006 there were 235 scores of zero on the essay, the number of students who wrote essays also fell from 1783 in spring 2006 to 1608 in the spring of 2007. That is 175 fewer students who wrote essays and 91 fewer essays with a score of zero. The drop in the number of students writing essays was not equal across all four states, with Chuuk seeing the largest drop in the number of students who took the admissions examination.

In the spring of 2006, the most common score on the essay, except for zero, was the then maximum possible score of 40. There was anecdotal evidence that the rubric needed to be adjusted as papers that varied in quality were "maxing out" at 40. A change in the rubric permitted a maximum score of 50 in 2007 and the result is that the right tail of the essay distribution now falls steadily from the mean to the maximum possible.



Rank order

The admissions board approved a rank-order formula using z-scores on the mathematics section, the grammar and vocabulary section, and the writing (essay) section. The formula placed a 50% weight on the essay, 25% on grammar and vocabulary, and a 25% weight on mathematics. The individual weighted z-scores were added to generate a sum of z-scores. The resulting z-score was linearly transformed to produced the final COMET score.

A COMET score of 700 was approved as the cut-off for regular admission to an associate degree program. A COMET score of 300 was established as the minimum score for admission to a certificate program. Students who have scored between 300 and 700 will be eligible to enroll in a certificate program.

Bear in mind that the college now operates based on fixed resources and specific projections of the number students that the college can serve. Each site has an target enrollment number and an maximum enrollment capacity limit. Like many smaller colleges, the college opts to admit a specific number of students.

The result is that the target admissions number is a fixed number. Put another way, there is not the possibility at present of all high schools increasing the number of students attaining regular admission to the college. If one high school sees an increase in the number of students who have attained regular admission, then some other high school (or high schools) must have seen a decrease in the number of students who gained regular admission. Thus admission is competitive and is mathematically termed a zero-sum game.

High schools that are using increases in regular admissions as an indicator do so at their own peril. If another high school improves by a larger amount, then that first high school could see a drop in admissions despite real improvement in their own educational processes. In 2007 the admissions board selected an intake of the top 400 students into associate degree programs.

Table three records the distribution by state and high school of the students into the three categories delineated above.

Table 3: Admissions status

Note that the table above reports the high school name using the names provided to this author by those working on behalf of the admissions board.

The average COMET score seen in table four represents a measure of the relative performance on the essay section, the grammar section, and the mathematics sections.

Table 4: High School COMET means

A COMET mean below 400 is exceptionally weak and below 300 represents essentially random performance on all included subsections.

While the individual high school scores are of interest to individual schools, state educational leadership might consider the mean performance of all of the schools in their state. Note that Kosrae reported results from only a single school, thus the state mean is the mean for that school.

Where a performance is strong, no one effort can be credited with the success. Where a performance is weak, no one lack can be faulted as the cause of the poor performance. Education is a set of canoes with many people paddling. Students, parents, teachers, administrators, state leadership, and special programs should all share in the credit where credit is due, in the blame where performances are weak.

Essay section

The writing section is considered by both the language and literature division and by those from other divisions who mark this section to possibly be singularly indicative of the capacity of students to academically succeed at the college – if there is such a thing as a single indicator. The maximum possible score is a fifty. Scores of twenty and under are considered to be significantly weak and raise questions as to whether these students can tackle any program that requires the ability to communicate in written English.

Determining whether one high school is statistically significantly stronger in rank order than another high school requires determining whether the mean essays scores are sufficiently separated.

In illustration 2, the vertical bars show the extent of the 95% confidence interval for the mean essay score. Where a bar for one school overlaps the mean ("the ball") for another school, the former is not statistically separated from the mean of the later.

The vertical axis in illustration two is the essay score, the high schools are listed in rank order.





The mean essay scores suggest that Kosrae, Pohnpei, and Yap states are, to differing degrees, accomplishing the task of instructing students in the skills necessary to compose written English essays.

Celebrating their first graduating class in the history of their high school, Nanpei Memorial School led the public high schools on the essay in 2007. As noted earlier, success is the result of many factors including supportive parents, a professional faculty, a capable and dedicated administration, and motivated students. While Nanpei Memorial High School has faced facilities challenges including older buildings and other facilities issues, the school clearly rose above those challenges on the essay section.

Looking at the individual public high schools in Chuuk indicates an almost complete lack of ability to accomplish a written composition in the English language.

While some fields such as science have significant support costs, English language and literature is likely the least expensive program on a per student basis. This is not likely a failure due to a lack of funding alone.

Changes in essay performance 2006 to 2007

This section of the 2007 report cannot be completed as the marking rubric was substantively changed. The addition of a fifth category would, theoretically, lift scores for good essay writers even wherein there was no actual improvement in the ability of that school's students to write essays. This impact was greatest in the highest ranks schools on the essay.

Mathematics section

Year-to-year mathematics subsection performance

Table five records the year-to-year performance from 2005 to 2007 on the mathematics subsections. The table also shows the averages for the top four-hundred rank order students on each subsection for 2006 and 2007.

Table 5: Longitudinal math means

Two subsection means showed a small improvement while two subsections showed a small decline. There were no substantive changes in the test from 2006 to 2007, so the means are comparable. The first two sections were substantively revised, with the first section being completely rewritten, between 2005 and 2006.

Mean Mathematics Scores by High School

The mean score by high school for the mathematics section represents the average number correct out of fifty for a high school.

A student blindly guessing on all forty questions, each with five multiple choice options, should get at least eight correct. Given that the first few problems involve basic arithmetic, scores around twelve or under are extremely weak performances.

Illustration three on the next page depicts the 95% confidence interval for the mean mathematics scores, with the high schools in descending mathematics score order. The vertical bars are the extent of the 95% confidence interval for the sample mean. Where a vertical line for one school overlaps the mean (the center ball) of a another school, the first school is not statistically significantly separated from the second school's mean.

The performances at Nukuno and Faichuuk were statistically indistinguishable from a random score of eight. This is the second year in a row Nukuno is statistically indistinguishable from random. Joining Nukuno and Faichuuk is Yap SDA. Weno, Chuuk High School, Nukuno, Southern Noumeneas, and Faichuuk performed so weakly in both mathematics and English that there is merit to an argument which would shift all of whatever resources are being spent on mathematics education to the teaching of English. These schools will not do significantly worse by not teaching mathematics – there is almost no where "down" for them to go from the their current status.



Mathematics Placement Data

Placement of students by high school and the program into which they were admitted is shown in table six below.

Table 6: Placement by high school and admissions status

Comments

The college admissions examination is an instrument that has been refined over the years towards the sole purpose of determining admissions at the college. The examination is not intended as either a diagnostic or summative measure of the effectiveness of high schools or state departments of education.

The admissions test can indicate, at the extreme ends, whether a school is performing strongly on these admissions measures or performing in a manner indistinguishable from random. Comments above have been made where performances are extraordinarily weak.

Diversity

The differences in the performances by state have repercussions on the make-up of the regular admissions. Table seven compares the composition of the regular admissions to the state shares of the national population. The table reflects the state of the high school, not the cultural identity of the individual test takers.

Table 7: Diversity

Thus in terms of the high school's state share of the national population, students from high schools in Kosrae are the most over-represented group in the regular admissions category with 2.29 times their share of the national population, up from 1.81 the previous year. Those from high schools in Chuuk are under-represented at a factor of 0.30 of their share of the national population, a drop from 0.35 the year before. Each high school has students from every state, however the data is still likely roughly reflective of the underlying cultural diversity the college can expect among freshmen in the fall of 2006.

Note that as one state gains an increasing share of the 400 associate degrees slots, other states lose share. As noted earlier, this is a zero-sum game. Gains by Kosrae and Pohnpei were mirrored by losses for Yap and Chuuk. Yap saw some of their strongest losses in the outer islands of Yap.

Subgroups Analysis

The college admissions examination is neither a diagnostic nor comprehensive measure of learning. The college admissions examination is designed for a single purpose: admitting students to the appropriate programs for their abilities in English and mathematics. Despite the focus of the examination, many sponsored programs utilize the admissions results and statistics in their reports. Other programs in the high schools also choose to use the college admissions examination as one of their many indicators. The following are analyses of the essay and mathematics section with some of these subgroups broken out separately.

This author is deeply indebted to those who work in sponsored programs and the high schools who provide lists of student names in programs and sections.

Essay section

Table eight details the performance of the high schools in descending mean essay score order, repeating an earlier table but including some of the subgroups. Explanation of the various subgroups is given after the table. The writing section had a total possible of fifty. The sample size n, sample standard deviation sx, lower bound for the 95% confidence interval for the sample mean, upper bound ("high") for the 95% confidence interval for the mean, and the sample mean are provided.

This section intentionally includes basic statistics for the subgroups as this is useful to many programs who have to report statistical data.

Many programs use admissions to an associates degree program as a indicator of success. This practice is not recommended. The college admits roughly the top 400 students into associates degree programs as ranked by a z-score using the grammar, essay, and mathematics test. Many of these students will be placed into developmental and not college level programs. Thus admission to an associates degree program is not a statistically meaningful measure. As a result this report does not include admissions status of the subgroups.

The rubric used to mark the essay is provided in an appendix. Two graders mark each essay. If the graders differ by five points, a third grader marks the essay. The college currently considers a score of forty to be a strong essay. Averages of forty or above are exceptionally good results and are likely difficult to improve upon year-to-year.

Table 8: 95% confidence interval for essay including subgroups

Note that the Kosrae High School advanced section and the Upward Bound Kosrae program are not independent subgroups. Ten of the twenty-two advanced section students are also in the Upward Bound program.

The Upward Bound Pohnpei participants are all enrolled at PICS on Pohnpei.

The subgroups indicate that the success seen at Nanpei Memorial this year was not simply a strong performance by only the academic sections. Nanpei Memorial's business sections performs on par with PICS, YHS, and Ohwa. Nanpei Memorials vocational sections outranked private schools such as Ohwa, Mizpah, and outranked two state campuses in the college system.

A note of caution is in order when comparing results between programs and sections. Clearly the Nanpei Memorial academic sections did well on the essay. Section lists for the academic sections at other high schools were not available to this author at the time of the writing of this report. PICS academic, Madolehnihmw academic, and Yap academic may have all done equally well or possibly better than Nanpei academic.

In illustration four, the mean essay score is marked by the central ball. The extent of the 95% confidence interval for the mean is given by the vertical bars. Where the 95% confidence interval for one school or subgroup overlaps the ball of another school or subgroup, then the scores are not sufficiently separated to ensure statistical significance. 

Mathematics Section

The following table details the performance of the high schools in descending mean mathematics score order, repeating an earlier table but including some of the subgroups of interest. The mathematics score is the sum of the four subsections, the total number correct out of forty.

Bear in mind that the the final column of ten problems are set at the level of college algebra, high school students are not expected to succeed at that column. Therefore averages above 26 (80% success in the first three columns of the test and a random two correct in the last column) would be a very strong performance. Lifting a school or a program's average above 30 is likely to be difficult.

Table 9: 95% confidence interval for math including subgroups

Of note in the above table is that for the second year in a row the top ranked programs by mean math sum are Upward Bound programs and the Kosrae High School advanced section. From the perspective of preparing students to pursue college programs in science, technology, engineering, and medicine, the Upward Bound programs remain very important to this nation.

Also of note is that these programs lift their students to the level of the private schools and above. This provides important opportunities to students in the public school system. This is important to ensure equity in access to a quality college prepatory education. If the only access to a quality education is through private schools, then the nation faces a future where those with the resources potentially generate an intellectual elite that would likely become a governing elite. Down this road lies potential loss of the proverbial "middle class" that seems so crucial to democratic stability and the rise of oligarchic governmental structures.

Programs such as Upward Bound, Talent Search, and others are currently beyond the financial wherewithal of the local governments. Continued United States support for these programs is important to this young nation.

Illustration five depicts a portion of table nine on the next page.



Mathematics Placement Data

The following table shows the break-down of the math placement by the subgroups. In that the mathematics section functions as a placement test, the placement of students has real meaning. There is no "400" limit involved. Theoretically, all students could place into college algebra classes. The class into which students are placed are indicated in the top row.

MS 095 PreAlgebra

MS 096 Elementary Algebra (Algebra I)

MS 099 Intermediate Algebra (Algebra II)

MS 100 College Algebra

MS 101 Algebra and Trigonometry (or MS 150 Statistics)

Students who place into MS 100 or MS 101/101 are deemed college level students.

Author and contact information:

All errors are solely those of the author. Please contact Dana Lee Ling at dleeling@comfsm.fm or 691-320-2480 extension 228 if you have questions or corrections.



Appendix A.