COMET Admissions Examination Statistical Summaries Spring 2006

Basic Statistics

The COMET admissions examination consisted of a reading section, grammar and vocabulary section, an essay, and a math section consisting of four subsections. The basic statistics for this sections and subsections are presented in the table one below.

Table one.

m1 is the first math subsection and contains ten arithmetic level problems.

m2 is the second math subsection and contains ten prealgebra level problems.

m3 is the third math subsection and contains ten elementary algebra problems.

m4 is the fourth math subsection and contains ten college level algebra problems.

SM is the sum of the math subsections and is out of forty.

Reading is a reading score out of 40.

Writing (essay) is the essay score out of 40.

Grammar is the grammar and vocabulary score out of 80.

The reading section average is separated from random by less than one standard deviation. While significant due to the sample size, the low average reinforced analysis in the language and literature division that the reading section might be problematic.

Correlations

The correlations between the subsections is given in table two below.

Table two.

Although the first two subsections of the math test were rewritten to require less reading ability, the math subsections sum remains, given the sample size, correlated moderately to moderately-strong with the English sections.

Writing correlated more strongly with the grammar section that with the reading section. Past experience in the language and literature division indicated that the writing section, the essay, was a trustworthy indicator of student ability. The strong correlation between the trusted writing section and the grammar section and the lower correlation of both to reading led the language and literature division to request that the reading score not be used in the final admissions decision this year. The division plans to redesign this section.

Distributions

The distribution of scores on each section is given in chart one. The score is on the horizontal axis, the vertical axis is the number of students with that score.

Chart one.

With the exception of the writing section, the other sections distributed in a manner which is considered statistically reasonable. That is, the frequency (the number of students) rises with the score towards the mean, and then falls as the score continues to rise past the mean to the maximum possible score. If the distribution (the shape of the curve) is a symmetrical heap, the most common score is also the mean score as well as the median score. This distribution is sometimes called a "bell curve" and is mathematically described by the normal or student's t-distribution.

In the past, the writing section, the essay, has in performed in a statistically reasonable fashion except for the number of students who score a sum of zero. This was true again this year with 235 scores of zero. Yet the removal of those 235 scores would not return the distribution this year to a bell curve.

The most common score on the essay, except for zero, was the maximum possible score of 40 with seventy-four students attaining a score of 40. The frequency rises steadily from eight students with a score of 2 to fifty-nine students with a score of 20. From a score of 20 to 35 the distribution is fairly flat. Then the distribution drops slightly until a score of 40.

The behavior of the essay section suggests that the section did not distinguish well among the seventy-four students with a score of 40. Bear in mind that a score of forty represents a paper with some errors and omissions of grammar, vocabulary, organization, cohesion, and content coverage. The distribution argues that the rubric should now include a score of five in each of the five areas. This should spread the seventy-four scores of 40 across the ten-point range from 40 to 50. This might also provide enough information to potentially place students in a writing course at the college.

Rank order

With the recommendation to remove the reading section from the calculations, the admissions board approved a rank-order formula using z-scores on the mathematics section, the grammar and vocabulary section, and the writing (essay) section. The formula placed a 50% weight on the essay, 25% on grammar and vocabulary, and a 25% weight on mathematics. The individual weighted z-scores were added to generate a sum of z-scores. The resulting z-score was linearly transformed to produced the final COMET score. The basic statistics for the z-score sum and the COMET score are given in table three.

Table three.

A COMET score of 700 was approved as the cut-off for regular admission to an associate degree program. A COMET score of 250 was established as the minimum score for admission to a certificate program. Students who have scored between 250 and 700 will be eligible for either the Intensive English Instruction sequence or a certificate program based on their individual position on a rank order by state. Decisions on the exact nature of the IEI admissions mechanism are still to be determined.

Bear in mind that the college now operates based on fixed resources and specific projections of the number students that the college can serve. Each site has an target enrollment number and an maximim enrollment capacity limit. Like many smaller colleges, the college opts to admit a specific number of students.

The result is that the target admissions number is a fixed number. Put another way, there is not the possibility at present of all high schools increasing the number of students attaining regular admission to the college. If one high school sees an increase in the number of students who have attained regular admission, then some other high school (or high schools) must have seen a decrease in the number of students who gained regular admission. Thus admission is competitive and is mathematically termed a zero-sum game.

High schools that are using increases in regular admissions as an indicator do so at their own peril. If another high school improves by a larger amount, then that first high school could see a drop in admissions despite real improvement in their own educational processes.

Table four records the distribution by state and high school of the students into the three categories delineated above.

Table four.

Note that the table above reports the high school name reported by those taking the test. The decision on the IEI admissions mechanism was still pending at the time this report was written.

The non-admitted status is not equivalent to the the "limbo" status discussed in prior years. "Limbo" was an arbitrary term used to refer to students who had a z-score sum on reading, writing, and grammar which was less than negative three. The nearest equivalent this year would be students below a COMET score of 328. Bear in mind that the calculation this year involves mathematics, grammar, and a weighted essay score.

The average COMET score seen in table five represents a measure of the relative performance on the writing (essay) section, the grammar section, and the mathematics section.

Table five.

A COMET average below 400 is exceptionally weak and below 328 represents essentially random performance on all included subsections. As this is the first year with a non-admission category, a conservative cut-off at 250 was established.

While the individual high school scores are of interest to individual schools, state educational leadership might consider the mean performance of all of the schools in their state. Note that Kosrae reported results from only a single school, thus the state mean is the mean for that school.

Where a performance is strong, no one effort can be credited with the success. Where a performance is weak, no one lack can be faulted as the cause of the poor performance. Education is a set of canoes with many people paddling. Students, parents, teachers, administrators, state leadership, and special programs should all share in the credit where credit is due, in the blame where performances are weak.

Table five is depicted graphically in chart two.

Chart two.

95% Confidence intervals for the COMET mean score by high school

Determining whether one high school is statistically significantly stronger in rank order than another high school requires determining whether the mean COMET scores are sufficiently separated. The following table provides the exact values for the 95% confidence interval for the COMET means by high school.

Table six.

Thus, for example, while PICS has a higher mean COMET score than KHS, KHS high school's 95% confidence interval spans 22 points on either side of the KHS mean COMET of 650.08. The PICS mean of 657.22 is only 7.14 points away. As the separation does not exceed 22 points, the two schools are in a statistical dead heat. Table six above is graphically shown in chart three on the next page.

The vertical bars show the extent of the 95% confidence interval for the mean score. Where a bar for one school overlaps the mean ("the ball") for another school, the former is not statistically separated from the mean of the later. Note that the upper limit for the possible mean for Faichuuk high school is less than 328, thus confirming that in terms of the admissions examination, Faichuuk scored no different from students randomly choosing answers.

Chart three.

Writing (essay) section

The writing section is considered by both the language and literature division and by those from other divisions who mark this section to possibly be singularly indicative of the capacity of students to academically succeed at the college – if there is such a thing as a single indicator. The maximum possible score is a forty, and there is a general sense that a score above 32 indicates a potential to succeed at the college. Scores of twenty and under are considered to be significantly weak and raise questions as to whether these students can tackle any program that requires the ability to communicate in written English. Table six records the mean essay scores by state and by high school.

Table seven.

The data in table seven is shown in chart four on the next page.

Chart four.

The mean essay scores suggest that Kosrae, Pohnpei, and Yap states are, to differing degrees, accomplishing the task of instructing students in the skills necessary to compose written English essays. As a state, despite the impact of Xavier and Saramen Chuuk Academy, the Chuuk state average is abysmal. Looking at the individual public high schools in Chuuk indicates an almost complete lack of ability to accomplish a written composition in the English language.

While some fields such as science have significant support costs, English language and literature is likely the least expensive program on a per student basis. This is not likely a failure due to a lack of funding alone.

The 95% confidence interval for the mean writing (essay) score allows one to determine whether the means for two schools are statistically significantly separated.

Table eight.

Chart five depicts the 95% confidence interval for the mean writing (essay) scores, with the high schools sorted by state and then in descending writing score order. The vertical bars are the extent of the 95% confidence interval for the sample mean. Where a vertical line for one school overlaps the mean (the center ball) of a another school, the first school is not statistically significantly separated from the second school's mean.

Chart five.

Changes in writing (essay) performance 2005 to 2006

During the marking process some graders noted that the essays appeared to be slightly stronger this year versus last year. Although year-to-year comparisons are always risky due to potential differences in the grading teams and other confounding factors, showing that there was no significant change would disconfirm the possibility the essays improved.

The mean scores by high school were compared year-to-year. The results are seen in table nine. NS indicates that the year-to-year change is not significant.

Table nine.

The only changes which were large enough to attain significance were positive changes. No school dropped by a statistically significant amount.

Mathematics section

Year-to-year mathematics subsection performance

Table ten records the year-to-year performance from 2005 to 2006 on the mathematics subsections. The table also shows the averages for the top four-hundred rank order students on each subsection.

Table ten.

Each subsection showed a small improvement, with the largest improvement seen in the first subsection. This improvement is thought to be due to the removal of English, to the extent that it could be removed, from the first subsection. Failure by a student to succeed in the first subsection is now a strong indicator of a failure of the student to perform basic arithmetic operations correctly.

The redesign of the mathematics subsections with the accompanying improvement in scores, coupled with a curriculum realignment within developmental mathematics, led to a redesign of the mathematics placement function.

Mean Mathematics Scores by High School

The mean score by high school for the mathematics section represents the average number correct out of forty for a high school.

Table eleven.

A student blindly guessing on all forty five-option multiple choice questions should get at least eight correct. Given that the first few problems involve basic arithmetic, scores around twelve or under are extremely weak performances. Table eleven is depicted graphically in chart six.

Chart six.

The performance at Nukuno was statistically indistinguishable from a random score of eight. Weno high school and Faichuuk high school performed so poorly in both mathematics and English that there is merit to an argument which would shift all of whatever resources are being spent on mathematics education to the teaching of English. These schools will not do significantly worse by not teaching mathematics – there is no where down for them to go from the their current status.

Chuuk high school, while exceptionally weak in performance, still outperformed the other public high schools in Chuuk on the mathematics section. Given the poor performance of Faichuuk on the essay and mathematics sections, there is scant evidence at present that dispersing the students away from Weno will lead to improved academic results.

Chart seven on the next page depicts the 95% confidence interval for the mean mathematics scores, with the high schools sorted by state and then in descending mathematics score order. The vertical bars are the extent of the 95% confidence interval for the sample mean. Where a vertical line for one school overlaps the mean (the center ball) of a another school, the first school is not statistically significantly separated from the second school's mean.

Chart seven.

Mathematics Placement Data

Incoming freshmen placement over the past three fall terms is given in the table below:

Table twelve.

The mathematics placement by high school is given in the table below:

Table thirteen.

Comments

The college admissions examination is an instrument that has been refined over the years towards the sole purpose of determining admissions at the college. The examination is not intended as either a diagnostic or summative measure of the effectiveness of high schools or state departments of education.

The admissions test can indicate, at the extreme ends, whether a school is performing strongly on these admissions measures or performing in a manner indistinguishable from random. Comments above have been made where performances are extraordinarily weak.

Diversity

The differences in the performances by state have repercussions on the make-up of the regular admissions. Table fourteen compares the composition of the regular admissions to the state shares of the national population. The table reflects the state of the high school, not the cultural identity of the individual test takers.