VOL. 10 NO. 3 THE MAGAZINE OF THE UNIVERSITY OF UTAH WINTER 2000

Continuum Home Page - University of Utah Home Page - Alumni Association Home Page
Questions, Comments - Table of Contents

THE RANKINGS GAME

Another year, another list: U.S. News & World Report releases its national rankings of colleges. Where is the U, and what does it all mean?

BY SUZANNE DEAN

It happens in the fall at most universities. The team readies itself. The fans weigh the opposition. The final scores are analyzed by Monday-morning quarterbacks, for many Mondays to come.

Your average college football game? Well, yes—but also the annual release of U.S. News and World Report's rankings of America's colleges. And when the 2001 rankings were issued Sept. 1, the U generally counted this one as a win.

For the first time since the 1997 rankings, the University was listed in the second tier of national universities, which put it somewhere between 52nd and 115th place among 228 institutions. There has long been a consensus among U officials that the second tier is where the institution belongs.

Last year, the U was in the third tier, which went from 121st to 176th place. "But we felt we were on the cusp of the second tier," says Paul Brinkman, associate vice president for budget and resource planning. The U adjusted some freshman retention data to account for students leaving on LDS missions. Meanwhile, its performance on other U.S. News measures improved. Apparently, those changes enabled the U to rise into the second tier.

For some years, education experts have prescribed a grain of salt when digesting the rankings, and just as the 2001 rankings were being released, Washington Monthly leaked a report that lent credence to that viewpoint. In 1997 the National Opinion Research Center (NORC) at the University of Chicago was commissioned to evaluate the ranking methodology. The NORC essentially concluded that the methodology has been concocted by journalists who are not experts in education and does not adhere to accepted standards of educational and social science research. In particular, the NORC questioned the validity of the statistical approach U.S. News uses to determine the rankings.

But no matter. The rankings have become one of the most influential sources of information about American universities. During the week they are released, the U.S. News Web site records millions of hits. A supplement based on the ratings is the nation's best-selling college guide. Locally and nationally, the rankings are front-page news.

Where the U Ranks

U.S. News always leads its "Best Colleges" issue with its choices for the top 50 national universities. This year, because of a three-way tie for 49th place, the list contained 51 institutions. As usual, prestigious Ivy League institutions headed the list. Princeton was no. 1, Harvard no. 2, and Yale no. 3.

After the top 50, national universities—defined as schools that have a broad range of majors and that award doctorates—are grouped into tiers. The tier lists are not in rank order; institutions are simply listed alphabetically. Tier 2 includes schools from 52nd to 115th place; tier 3 includes those down to 171st place; and tier 4 to 228th place.

No institution in the eight-state Mountain West (New Mexico, Arizona, Colorado, Utah, Nevada, Wyoming, Idaho, and Montana) was in the top 50. Only six Mountain West schools made Tier 2—University of Utah, Brigham Young University, University of Arizona, University of Colorado, Colorado School of Mines (a state technological institution), and University of Denver (a private, nonsectarian college).

U.S. News publishes only a fraction of the data that determine the rankings. But it does publish, by institution, each college's score for academic reputation. The score is derived from a survey in which university presidents, provosts, and deans of admission are asked to rate institutions on a 1-5 scale, with no. 1 being marginal and no. 5 distinguished.

The U's average rating was 3.2, which is a good score. If private colleges are deleted from Tier 1 and Tier 2, leaving only public institutions, the U ranks 38th.

Most of the rankings are designed to evaluate undergraduate education in general. This year, U.S. News also ranked undergraduate business and engineering programs, and specialties within those programs.

The David Eccles School of Business at the U was ranked 50th among 338 accredited business schools. The U's College of Engineering was ranked 67th among 181 accredited engineering schools that award Ph.D.s. The U was also ranked 16th in the specialty of bioengineering.

How U.S. News Ranks Colleges

U.S. News bases its rankings on seven data categories, each designed to reflect a dimension of institutional quality. Some category scores are based on a single data item while others are based on up to six items. There are 16 measures across all categories. Various measures are assigned various weights in determining category scores. Likewise, category scores are weighted in the final tabulation.

Following are the categories, their weights, and how the U performs, as best can be determined based on the information U.S. News makes available.

Academic Reputation (25 percent)

This score comes from a single measure—an institution's average rating from the survey of academic officials described earlier. For the 2001 rankings, U.S. News says it sent out about 4,000 surveys and received a 67 percent response.

When ranked against all Tier 1 and Tier 2 colleges, including private schools, the U is 74th. Its 3.2 average was the highest among the three Utah schools that are classified as national universities. In contrast, BYU's rating was 3.1 and USU's 2.7.

Faculty Resources (20 percent)

This score is based on six measures:1

Average faculty compensation;
Percentage of faculty with terminal degrees in their fields (generally a Ph.D.);
Student/faculty ratio;
Percentage of classes with 1-19 students;
Percentage of classes with 50 or more students;
Percentage of faculty who are full-time.

U.S. News does not publish any data on the first three measures (although national figures indicate that faculty salaries at the U are below average). But the U ranks well on the last three items. In 1999, the year used in scoring, 95 percent of faculty were full-time, which put the U 26th among Tier 1 and Tier 2 schools. The percentage grew 10 percent from the previous year.

Forty-three percent of classes had 19 or fewer students, which translates to 66th place among Tier 1 and 2 schools. Just 15 percent of classes had 50 or more—another favorable score.

Retention and Graduation (20 percent)

This score is based on two measures:

Freshman retention, which is the percentage of freshmen who enter one fall and are also registered the next fall, averaged over three years;
Graduation rate, which is the percentage of freshmen who receive degrees within six years, averaged over three years. This year's rankings are based on freshmen who entered from 1990 to 1993.

The U's freshman retention rate, adjusted for the "missionary factor," is 72 percent, which is at the bottom of Tier 1 and 2 institutions. Its graduation rate, 52 percent, is 112th among Tier 1 and 2 institutions.

The U is working aggressively to improve freshman retention, according to David Pershing, senior vice president for academic affairs. In recent years, Undergraduate Studies, the residence halls, and some departments have initiated programs—such as LEAP, the Liberal Education Accelerated Program—to link students upon arrival with a small group of fellow students.

In addition, the University has set up an early intervention program to identify and counsel students who may experience problems during their freshman year, and is seeking scholarship money to help freshmen who are in danger of dropping out for financial reasons.

But Pershing also notes that the six-year time frame used for determining the graduation rate is too tight for evaluating an urban commuter institution such as the U. "Many of our students work more than 30 hours per week to pay tuition and support themselves," he says. "Naturally, they are going to take longer to graduate than at a university where the parents are paying the expenses. We are trying to meet the needs of our students, and many simply cannot afford not to work."

Both freshman retention and graduation rates reflect the abilities of the students the U admits in the first place. U.S. News itself calculates that based on entering student abilities and the money the U spends on students, it should expect to graduate 51 percent. The U exceeds that.

"Our mission is to provide higher education for the state's citizens," says Pershing, "and therefore we do accept a wider range of students than would be typical, for instance, of an Ivy League university or even BYU. Does this mean the quality of education is poor? Certainly not."

Student Selectivity (15 percent)

The measures for this category are:

The ACT or SAT score at the 25th and 75th percentile of the freshman class;
The percentage of freshmen who were in the top 10 percent of their high school classes;
The acceptance rate, or percentage of applicants who are admitted;
The yield, or the percentage of admittees who elect to attend an institution.

It appears that the University ranks below the median of Tier 1 and 2 schools in this category—but the institution is climbing. In 1999 the U's 25th and 75th percentile ACT scores were 21 and 28. Both scores were up one point from the previous year. Most Tier 1 and 2 schools use the SAT, but among 26 institutions that use the ACT, the U ranks 17th.

Twenty-six percent of the 1999 entering class came from the top ten percent of their high school classes, a five percent jump from the previous year. The U ranks 93rd among Tier 1 and 2 schools on this measure.

The U accepted 93 percent of applicants in 1999, which was very liberal relative to other Tier 1 and 2 schools, but the acceptance rate fluctuates from year to year.

U.S. News does not release figures on the last measure, the percentage of admitted students who actually enroll.

"The best students at the U, the top quartile, are very good," notes Pershing. "An ACT score of 28 is above the Tier 2 average and is comparable to some top-tier institutions like the University of Illinois and UC Irvine."

Financial Resources (10 percent)

This score is based solely on expenditures per student averaged over 1998 and 1999. Although U.S. News does not publish the dollar amounts used in ranking, national figures show that the U's state support and its tuition are well below the national average for schools of its type. Therefore, the U probably ranks low on this measure.

Graduation Rate Performance (5 percent)

This is another single-measure category. Using entrance test scores adjusted for expenditures per student, U.S. News calculates a predicted graduation rate. It compares that with the actual graduation rate. The difference, plus or minus, is the category score. The U's predicted graduation rate was 51 percent and its actual rate 52 percent, which translates to a +1. As the rankings go, that score is neutral to good. While some Tier 1 schools have scores as high as +15, others have minuses, such as a -14 at fourth-ranked California Institute of Technology.

Alumni Giving Rate (5 percent)

This score is determined by the percentage of graduates who donate in a year, averaged over two years. For the two target years covered in this year's rankings, an average of 12 percent of U graduates donated, which put the institution at 102nd among Tier 1 and 2 schools.

While the score is designed to measure alumni regard for their university, it also reflects development practices, such as how frequently an institution solicits alumni and how much emphasis the school places on large gifts vs. small ones. "The U emphasizes both broad participation and large gifts, but it needs to expand the number of alumni who contribute at all levels," says J. Michael Mattsson BS'60, vice president for development.

Are the Rankings Valid?

Even critics of the U.S. News methodology stop short of calling it totally worthless. Don Hossler, vice chancellor and professor of educational leadership at Indiana University, writes that several of the magazine's measures "have been used for years by scholars as indirect indicators of academic quality." If a school has been at the top of the U.S. News rankings for a decade, he writes, "it may be a reasonably good indicator that this college or university is better than many other institutions."2

In 1998 U.S. News conducted a survey in which it asked college presidents, provosts, and deans of admission to rate the importance of its measures to academic quality. Depending on the measure, 60 to 70 percent rated items such as reputation, freshman retention rate, class size, percent of faculty with terminal degrees, and student/faculty ratio as "important" or "very important."

Most of the criticism centers not on the measures themselves but on how U.S. News adds them up to determine final rankings. The magazine says that it assigns weights to various items (such as making ACT scores count for 40 percent of the student selectivity category and making student selectivity count for 15 percent of the final score) according to its "nonpartisan view of what matters in education."

That, the NROC study says, is the central problem. The foundation of the ranking system is the personal preferences of the U.S. News staff. If, as the magazine contends, the rankings reflect the "academic quality" at the nation's various colleges, U.S. News needs to define that term. Then, says the NORC, the magazine needs to run the necessary statistical tests to make sure its ranking system actually measures academic quality.

In addition, Hossler and other critics say the whole psychology of ranking tends to leave false impressions. When items are ranked, people assume that there are big differences between the items at the top and those at the bottom of the list. But in the U.S. News rankings, institutions tend to be very close to each other—and are often tied—on various measures. Even a one percent change in one or two measures can change a school's ranking. Yet such a shift is imperceptible to students on the campus.

Where does that leave the U? To the extent the rankings are meaningful, a Tier 2 ranking suggests it is solidly in the mainstream of major U.S. universities. The U is above many Tier 2 schools on some measures, such as reputation and percent of faculty who are full-time, and is moving up on others, such as ACT scores and percentage of freshmen who had top grades in high school.

And there's something more, says Pershing. The U.S. News rankings "almost completely ignore the question of value vs. cost, and that is a very important issue to many young people and their parents. Our tuition and fees, about $2,500 per year, are below almost all of our peer schools. We now have the best residential living facilities in the West and comparatively small class sizes for a major university. Considering all of these factors, the U is an amazing value for the students of this state."

Footnotes

1 U.S. News does not count teaching assistants as faculty. The number of classes with more than 50 students is an inverse measure; the higher the number, the lower the institution's score for that item.

2 Hossler, Don, "The Problem with College Rankings," About Campus, Indiana University, 5:1, March/April 2000, pp. 20-24.

—Suzanne Dean BS'71 works in marketing communications for a national health-care company.

Continuum Home Page - University of Utah Home Page - Alumni Association Home Page
Questions, Comments - Table of Contents