The University of Texas’ affirmative action admissions policy, and perhaps even a 2003 landmark ruling on race, appeared in serious trouble Wednesday during arguments in the U.S. Supreme Court.

Eight justices heard from lawyers for the university, a white woman denied admission in 2008, and the United States in Fisher v. University of Texas at Austin. Justice Elena Kagan did not participate, apparently because of experience with the litigation in her prior role as solicitor general of the United States.

Constitutional questions about race generally have divided the Roberts court and, unsurprisingly on Wednesday, the court’s moderate-liberal members aggressively challenged Abigail Fisher’s counsel, Bert Rein of Wiley Rein, while the conservative justices aimed their firepower at the university’s advocate, Gregory Garre of Latham & Watkins.

The backdrop to the challenge by Fisher is the court’s 2003 ruling in Grutter v. Bollinger involving affirmative action by the University of Michigan School of Law. That 5-4 decision reaffirmed that diversity in higher education is a compelling government interest. Under the ruling, universities may use race as one of many factors in an individualized assessment of student applicants in order to achieve a critical mass of underrepresented students. Justice Sandra Day O’Connor wrote the majority opinion in Grutter.

Under state law, the University of Texas is required to admit all students who graduate in the top 10 percent of their classes. The “10 percent plan,” as it is known, fills nearly 90 percent of the university’s freshman class and has contributed to increased diversity on campus. The remaining student applicants are judged on the basis of academic scores and “personal achievement” scores, which include a dozen factors, such as leadership qualities, extracurricular activities, family status, socioeconomic status and race. The university added race in 2005 after the Grutter decision permitted a limited use of that factor.

Fisher did not graduate in the top 10 percent of her high school class and claims she was denied admission under the alternative plan because of her race.

The difficulty of defining and measuring when a university has achieved “critical mass” dominated much of the justices’ questioning, and answers by Rein and Garre did not seem to satisfy the justices asking the questions.

As a visiting O’Connor and the widow of Justice Thurgood Marshall watched from guest seats in the courtroom, Justice Stephen Breyer told Rein that the two lower courts had found that the university’s plan satisfied Grutter. “There is no quota. It is individualized. It is time limited. It was adopted after the consideration of race-neutral means. Each applicant receives individual consideration, and race did not become the predominant factor,” he said and then asked why this plan did not satisfy Grutter.

Rein replied, “There was no effort in this case to establish even a working target for critical mass. They simply ignored it. They just used words and they said, ‘we’ve got to do more.’ So they never answered the predicate question which Grutter asks: Absent the use of race, can we generate a critical mass?”

Picking up on the critical-mass issue, Chief Justice John Roberts Jr. asked Garre, “What is the critical mass of African-Americans and Hispanics at the university that you are working toward?” Garre responded, “We don’t have one.” And Roberts rejoined, “So how are we supposed to tell whether this plan is narrowly tailored to that goal?”

Garre told the court that it should look at the criteria listed in the Grutter decision. Grutter, he added, rejected the notion that there is a fixed percentage.

Roberts returned several times to the question of measuring critical mass. “Grutter said there has to be a logical end point to your use of race. What is the logical end point?” he asked Garre again. Garre said to determine critical mass, the university looks at enrollment data, feedback from students on feelings of racial isolation on campus and in the classrooms, and the racial climate on the campus.

The “bottom line” to her colleagues’ question, said Justice Sonia Sotomayor, is: When does the court stop deferring to the university’s judgment that race is still necessary?

Garre replied, “I think the standard you would apply is the one set forth in Grutter, and it comes from Justice Powell’s opinion in Bakke [Regents of the University of California v. Bakke (1978)], that you would look to whether or not the university reached an environment in which members of underrepresented minorities, African-Americans and Hispanics, do not feel like spokespersons for their race — an environment where cross-racial understanding is promoted, an environment where the educational benefits of diversity are realized.”

Garre received a boost from Solicitor General Donald Verrilli Jr., who said the university’s 10 percent plan does produce ethnic diversity but the university cannot control that diversity in the way that it can with its holistic review of the remaining student applicants.

“So my understanding of what the university here is looking to do, and what universities generally are looking to do in this circumstance, is to make individualized decisions about applicants who will directly further the educational mission,” Verrilli said. “For example, they will look for individuals who will play against racial stereotypes just by what they bring: the African-American fencer; the Hispanic who has mastered classical Greek.”

Verrilli re-emphasized Garre’s point that critical mass is not a number, but, he added, as the number of minority enrollees gets higher, the burden on the university to justify its use of race gets harder.

In rebuttal, Rein returned to the chief justice’s questions, saying, “Where is the end point? If you have nothing to gauge the success of the program, if you can’t even say at the beginning we don’t have critical mass because we don’t know what it is and we refuse to say what it is, there is no judicial supervision, there is no strict scrutiny and there is no end point to what they are doing.”

Rein stressed that he was not asking the court to overrule Grutter, but “to clearly state what it is that allows the use of this odious classification” and to state it as a “narrow window.”

Sotomayor interjected, “So you don’t want to overrule Grutter; you just want to gut it. Now you want to tell universities that once you reach a certain number, then you can’t use race anymore.” Rein denied that charge, saying, “The only way one could reach that conclusion is to assume that Grutter is an unlimited mandate without end point to just use race to your own satisfaction and to be deferred to in your use of race. That is unacceptable.”

Justice Anthony Kennedy, considered the key to the case’s outcome, asked few questions and revealed little of his thinking about the case. With Kagan recused, there is the possibility of a 4-4 split, which would leave in place the ruling favoring the university by the 5th U.S. Circuit Court of Appeals.

The Fisher case has drawn roughly 90 amicus briefs, with more than 70 supporting the university.

Marcia Coyle is the chief Washington correspondent for The National Law Journal, a Texas Lawyer affiliate. Her email address is mcoyle@alm.com.