Manhattan can be an intimidating place for an outsider. The buildings are huge, the people are aggressive, and the cab drivers are just plain crazy. The legal market isn’t the most welcoming, either. Just ask the Magic Circle.

Four of the five members of this elite club of U.K. firms—Allen & Overy, Clifford Chance, Freshfields Bruckhaus Deringer, and Linklaters—have each had offices in New York for between 27 and 41 years, but they are still fighting just to make their presence felt. These pioneers came to America with grand intentions: not just to service the U.S. needs of existing clients, but also to compete head-on with the top Wall Street firms for the biggest-ticket transactional work—the sort of complex and cross-border deals for which the Magic Circle is known around the world. (The fifth Magic Circle firm, Slaughter and May, has generally eschewed international expansion in favor of a global alliance network that in the United States includes Cravath, Swaine & Moore and Wachtell, Lipton, Rosen & Katz.)

The foreign quartet has enjoyed some success in the U.S., gaining traction in niche practices such as aviation finance, real estate funds, and international arbitration. But despite significant investment—Allen & Overy’s U.S. offices have lost a staggering $250 million over the past 20 years, according to two former partners with knowledge of the situation—they continue to lack the scale, profile, or major client relationships to effectively challenge America’s leading firms in the key areas of litigation, mainstream finance, and M&A. The Magic Circle’s rigid lockstep compensation systems and lower profitability relative to the leading U.S. firms has also severely hampered their recruitment efforts.

While the Magic Circle has long found life across the Atlantic to be a frustrating and costly slog, marked at times by a surprising lack of strategic clarity, a flood of senior departures in recent months suggests that their problems in the U.S. may have become acute. As of early September, the four U.K. firms have seen seven U.S. practice heads leave in 2013 alone. A&O has lost four, including two in the past two months: U.S. leveraged finance head Mark Wojciechowski, who joined Morrison & Foerster in August, and Americas international arbitration practice head Benno Kimmelman, who moved to Sidley Austin in September. (An A&O spokesman points out that, even after the recent glut of senior-level exits, the firm still has around 10 more partners in the U.S. than it had in 2010.) Also in September, Linklaters U.S. litigation head Joseph Armao left for Reed Smith, alongside fellow disputes partner Paul Alfieri.

Both A&O and Linklaters have recently completed strategic reviews in which they effectively admit defeat in their attempts to crack the New York market. At the beginning of the year, A&O’s global managing and senior partners, Wim Dejonghe and David Morley, respectively, traveled to New York to restructure the firm’s New York office. In meetings, partners were told that global capital markets head David Krischer was being parachuted in from London to replace Kevin O’Shea as the firm’s U.S. managing partner. (O’Shea was subsequently named A&O’s first U.S. senior partner—an external, client-facing role with less responsibility for the strategy or administration of the practice.) Several partners and associates were asked to leave the firm following that meeting, according to two individuals who were present at the meeting, while other equity partners were asked to take a pay cut—some of whom then decided to leave anyway. The partners were also hit with a truly shocking revelation: A&O’s U.S. offices had lost a quarter of a billion dollars over the past two decades, including $20 million in 2011 alone.

“We were all pretty stunned,” says one A&O partner who was in attendance—one of 39 U.S.–based law firm partners, recruiters, and consultants interviewed for this article. “Part of it was down to fluctuations in currency exchange rates, and there was some pretty screwy accounting over things like how global overheads were allocated, but there’s no escaping the fact that it was a massive, massive loss.”

In an interview, Dejonghe declined to comment on the financial performance of the U.S. offices (the firm has a base in Washington, D.C., as well as in New York) or the pay cuts, saying only that New York, like all of the firm’s offices, is subject to “routine equity management.” While Dejonghe denies that the U.S. practice was restructured, he admits that the firm has “refocused” its strategy there. It will now be geared toward servicing the U.S. needs of existing firm clients and targeting international work from U.S. clients, rather than trying to compete directly with top American firms for locally sourced domestic work. (The same strategic shift has occurred at Linklaters, which presented the findings of a yearlong U.S. strategy review, led by London-based projects and finance head Jim Rice, at the firm’s annual partner conference in April.)

“If you look at our global offering, we can claim to have a leading practice in every jurisdiction in which we operate—apart from the U.S.,” Dejonghe says. “We initially had an ambition to become one of the leading [firms in the U.S.], but we’ve become quite a bit more realistic. It’s a tough market—tougher than we first thought. We’ve learned that the hard way.”

Ask a partner at any of the top U.S. firms about the Magic Circle in New York, and the answer tends to be: “We don’t see them.” Ask one of the Magic Circle firms about their U.K. rivals in New York and they’ll say they don’t see them either: “We only compete with U.S. firms.” To more objectively quantify the relative strength of the Magic Circle’s U.S. practices, we analyzed their performance in legal directories Chambers & Partners and The Legal 500, deal data from Mergermarket Ltd, and client-sourced brand strength indexing from global research company Acritas. We found that, far from being nonentities in the U.S., the Magic Circle firms have genuine strength in certain niche areas, such as aviation finance, real estate funds, and international arbitration. But they also have no meaningful presence in the core practices of mainstream banking and finance, commercial litigation, and big-ticket M&A.

In the directories, Clifford Chance comes out on top among the U.K. firms, with the most overall rankings and by far the highest number of leading lawyers. While just six Magic Circle lawyers are named as “leading individuals” by the U.S. Legal 500—itself an indication of the firms’ lack of standing in the market—five are at Clifford Chance: U.S. corporate finance chair Jay Bernstein and Americas tax head Richard Catalano, who are both experts in real estate investment trusts (REITs); aviation finance heads John Howitt and Zarrar Sehgal; and tax partner Donald Carden. (The sixth is Nigel Blackaby, head of the U.S. international arbitration team at Freshfields.)

Clifford Chance, which in the fiscal year ended April 30, 2013, generated revenues in the Americas of $225.7 million—11 percent of its total firmwide revenue of £1.27 billion ($1.99 billion)—is a recognized market leader in aviation finance and REITs, achieving top directory rankings in both areas. The firm also has a strong, 22-lawyer U.S. project finance team, with Americas practice cochair Christopher McIsaac the only Magic Circle projects partner ranked in the first band by Chambers.

A&O also scores well, matching Clifford Chance in its number of both firm and lawyer rankings in Chambers. The firm possesses a leading public-private partnerships group—one of only three Magic Circle practices with a tier-one ranking in Chambers. U.S. senior partner O’Shea—ranked in the second band by Chambers for real estate finance—has developed a niche advising hedge funds on distressed property deals, while Pamela Chepiga is highly regarded for both commercial litigation and white-collar criminal investigations.

Linklaters, meanwhile, features in Chambers in only four practice areas, three of which are in the bottom band, and is the only U.K. firm to have no leading or top-tier lawyers in either Chambers or the U.S. Legal 500. Linklaters is the sole Magic Circle firm to be ranked by Chambers for U.S. banking and finance, however—a surprise, given that it is better known globally as a corporate firm, as opposed to the more finance-centered A&O and Clifford Chance. The firm also has the strongest funds practice of any U.K. firm in the U.S., according to Chambers, although that department suffered a blow last year with the departure of U.S. hedge funds leader Stephen Culhane, who joined Kaye Scholer.

“We’re not going to judge success by head count—we’re not trying to build a U.S. law firm,” says Linklaters U.S. co–managing partner Jeff Norton, who adds that the revenue generated by the firm’s U.S. offices has increased annually throughout the recession. “We could build domestic practices and penetrate some of those markets, but if it doesn’t align with our global strategy, what’s the point?”

Among the Magic Circle firms, Freshfields has arguably adapted to the American market most successfully. Its practice mix most closely resembles that of a U.S. firm, with a 16-partner, 80-lawyer litigation and disputes group accounting for more than 50 percent of its business in the States. That is twice the proportion of litigators in Freshfields’s London headquarters, where 25 percent of partners are litigators, and three times bigger than the litigation arms of the U.S. offices of A&O, Clifford Chance, or Linklaters, each of which has fewer than 20 percent of their attorneys in disputes.

As it is elsewhere in the world, Freshfields is a recognized market leader in the U.S. for international arbitration, achieving top-tier rankings in both Chambers and the U.S. Legal 500—one of just two Magic Circle practices to do so, alongside Clifford Chance’s REITs team. The firm has also been perhaps the clearest and most consistent in its strategy. So while the other Magic Circle firms have grown or contracted over time—Clifford Chance had 450 attorneys in the U.S. immediately following its combination with Rogers & Wells but just 230 today, for example—Freshfields has been stable. Just one partner has left the firm in the U.S. so far in 2013: Washington, D.C.–based tax partner Gregory May, who retired. You’d have to go back over a year for the last time Freshfields lost a U.S. partner to a rival, when projects specialists Kent Rowey and Dolly Mirchandani moved to A&O. Only five other partners have left the firm in either New York or Washington, D.C., over the past seven years.

It could be said that Freshfields has been too cautious, however. Its New York office is by far the smallest of the Magic Circle, with just 120 attorneys, and as a result it lacks a profile outside litigation and disputes. Julian Pritchard, U.S. regional managing partner at Freshfields, acknowledges that the firm may have been too circumspect in its approach to America, but says that the risks associated with more aggressive investment outweigh the potential rewards. “We could have grown more quickly in the early days, but felt that quality should be the guiding principle,” he says. “If being cautious has cost us some market share, that’s a price we’re willing to pay.”

While each of the four international Magic Circle firms has gained traction in certain practice areas in the U.S., they are nowhere to be seen when it comes to M&A. The four do not feature at all in the top U.S. M&A categories in either Chambers or The Legal 500, and it’s a similar story with deal data. As of late September, each of the firms sit outside the top 20 in Mergermarket’s 2013 M&A adviser rankings by aggregate deal value on transactions with a U.S.–based target. Linklaters languishes in 70th place.

The U.K. firms are simply not in the running for large U.S. public company M&A deals, and still typically have to team up with American firms on global transactions with U.S. components. Freshfields recently worked alongside Munger, Tolles & Olson in advising Warren Buffett’s Berkshire Hathaway Inc. on its $28 billion acquisition, with 3G Capital Inc., of U.S. food processing giant Heinz Company—the second-largest U.S. M&A deal of the year so far. Similarly, Clifford Chance paired with Simpson Thacher & Bartlett to represent private equity firm Silver Lake Partners on the $24.4 billion (£15.5 billion) buyout of computer manufacturer Dell Inc.—the largest leveraged buyout since the beginning of the financial crisis in 2008.

Clifford Chance’s Cohen says the firm wants to get to the point where it doesn’t have to partner with U.S. rivals, but there are two significant barriers to it and the other Magic Circle firms achieving that goal: scale and client relationships. Major M&A deals require significant lawyer resources—not just attorneys practicing in corporate law, but also in specialist support areas, such as tax, labor and employment, environmental law, antitrust, IP, real estate, finance, and securities. “If you’re going to do huge public M&A [deals], you can’t just take a few people—you need a large, fully formed team, which is a serious commitment,” Freshfields’s Pritchard says. “That makes building M&A more challenging—it takes a concerted effort.”

But even if the Magic Circle firms had the head count necessary to handle the biggest-ticket M&A transactions, they typically lack the client relationships needed to win the work in the first place. Earlier this year, global research company Acritas contacted 815 senior in-house counsel at global corporations with annual revenues of at least $1 billion, and asked them to name the first five law firms they thought of when it came to awareness, favorability, and those they would be most likely to use for multijurisdictional transactions and litigation. While more than 99 percent of European respondents named at least one of the Magic Circle firms, just one in eight U.S.–based in-house counsel named them. A mere 5 percent of U.S. corporate counsel mentioned the Magic Circle when asked which firms they liked best. Surprisingly, the Magic Circle firms are more than twice as well-known in Asia as they are in America.

“[The Magic Circle firms] have all grown through the [practice] areas of least resistance, but in every market, success is ultimately defined by your ability to attract top corporate work,” says Conrado Tenaglia, U.S. co–managing partner at Linklaters. “For us to be able to claim that we are making significant progress in the U.S. market, we would definitely need to have better capability and track record in M&A. It’s fair to say that we haven’t gotten to that level yet.”

In fairness to the Magic Circle, even U.S. firms based outside New York have found it difficult to enter the Manhattan market. “To be successful in New York, you need to either have a tremendous strength in a particular practice area, or a strong relationship with one of the top clients in town, and one that is committed to sending you work,” says Robert Dell, chairman of Latham & Watkins, one of the few out-of-town firms widely considered to have cracked New York. “If you don’t have at least one of those things—or at least a good amount of luck—you’re going to find it very hard.”

When Latham opened its New York office in 1985, it did so purely to advise two existing clients in two practice areas: private equity deals for buyout firm KKR & Co. and high-yield bond work for the now-defunct U.S. investment bank Drexel Burnham Lambert. Both private equity and high-yield debt were relatively new practice areas at the time, but they were growing rapidly. Latham quickly established itself at the top of those markets and used that strength as a base upon which to develop a full-service offering, with New York now the firm’s largest office worldwide. Kirkland & Ellis, which launched in Manhattan in 1990, followed a path similar to Latham’s by building out from an initial focus on private equity. Today, New York is Kirkland’s second-largest office.

The Magic Circle firms believe there may be an opportunity for them to pursue a similar strategy using the fast-evolving practice of global investigations and regulation, where their greater international presence compared to most U.S. firms could give them an advantage. With regulations being enforced more aggressively and agencies working more cooperatively across borders, it’s a growing market, and the duration and scope of multijurisdictional investigations makes it extremely profitable work.

“The U.S. has been very extraterritorial in its approach to regulation,” says Clifford Chance’s Cohen. “Clients all over the world are now having to look at how regulatory changes in the U.S. affect their business. We think that plays well to our strengths.”

The U.K. firms have been actively investing in Washington, D.C., opening offices and bringing in laterals to bolster their expertise in areas such as regulatory enforcement, white-collar crime, and Foreign Corrupt Practices Act (FCPA) work. A&O launched its base in 2011 with the hire of a two-partner team from O’Melveny & Myers, while Linklaters reestablished a D.C. presence last year. (Linklaters was the first Magic Circle firm to launch in Washington, in 1992, but the firm closed the office a decade later, telling U.K. publication Legal Week at the time that it no longer made economic sense to remain in the capital.)

Freshfields, which opened its doors in Washington, D.C., in 1998, has the top-rated white-collar crime and government investigations practice of the Magic Circle firms in the U.S., placing it in band two in Chambers. This April, it further deepened its practice with the hire of Boies, Schiller & Flexner partner Matt Friedrich, a former acting head of the U.S. Department of Justice’s criminal division. A Freshfields spokesman says the firm is currently working on more than 30 FCPA investigations.

A&O’s O’Shea says the U.K. firms have found it easier to recruit partners in Washington, D.C., than in New York: “Frankly,” he says, “it’s more affordable.”

Despite this, the Magic Circle remain small players in the U.S. capital. Even the largest of the group in Washington—Clifford Chance, whose office has been open 15 years—still has fewer than 50 lawyers in the city, less than a tenth of the number at Hogan Lovells. (As with its New York office, this is another sharp decrease: In 2000 Clifford Chance had more than 80 lawyers in D.C.)

Building scale has been a consistent problem for the Magic Circle in the U.S. Much of this is due to the firms’ lockstep compensation systems, in which equity partners are paid on the basis of seniority, with their remuneration rising each year as they move up the ladder. Lockstep is a deeply engrained part of the Magic Circle ethos. Each of the four U.K. firms have in recent years modified their firmwide remuneration systems to provide some much-needed flexibility, adding salaried partner ranks, bonus pools, and the ability to allow management to alter an individual’s position on the lockstep. But despite this shift toward the sort of performance-related pay that is typical among American firms, lockstep remains the single greatest factor that distinguishes the U.K. firms in New York.

Freshfields’s Pritchard says this is a good thing. “People might think that our remuneration system would prevent us from hiring the right people, but in fact it’s one of the main reasons they join,” he says. “We have a great culture, and that is driven by lockstep. There are some partners for whom money is the deciding factor [in joining a firm], but for most, it’s not what they focus on.”

Pritchard points to hires such as his firm’s recruitment in 1998 of a highly regarded four-partner group from Milbank, Tweed, Hadley & McCloy—including project finance partner Ted Burke, who is currently Freshfields’s global managing partner, and structured finance partner Brian Rance, whom Pritchard succeeded as U.S. managing partner—as evidence that lockstep firms can attract star names in New York. (Freshfields recently announced that Burke will leave in early 2014 to become general counsel at private equity firm Arclight Capital Partners.)

That’s not the message we received from the partners, recruiters, and consultants we spoke to for this feature, however. Their overwhelming response was that, for the Magic Circle in New York, lockstep is a hindrance, not a help. “You can make a lot of positive arguments for lockstep, but you can’t make them in a market that really doesn’t embrace it,” says law firm consultant Bradford Hildebrandt. “Lockstep is just not in the DNA of U.S. lawyers, so it’s very hard to convince them to work under that philosophy, particularly when you can only afford to pay them 60 percent of what they’d get elsewhere.”

Some of the top Wall Street firms, including Cravath; Davis Polk & Wardwell; and Paul, Weiss, Rifkind, Wharton & Garrison, also utilize lockstep systems. But those firms are much more profitable than the Magic Circle, so they can still pay top dollar for talent. Cravath’s average profits per equity partner in its most recent fiscal year, for instance, was $3.44 million—more than twice that of Clifford Chance ($1.5 million) or A&O ($1.6 million).

Clifford Chance’s compensation system does contain a provision that allows the firm to pay star performers above the top of its lockstep, but the firm hasn’t used what it calls “super-point” packages for more than a decade in the U.S. market, last doing so in 2000 in an attempt to match the earnings of top rainmakers at Rogers & Wells following the firms’ transatlantic merger. Indeed, four current and former partners say Clifford Chance would be unlikely to offer such lucrative deals again, since they failed to stop Rogers & Wells stars from leaving the combined firm over compensation, despite some being paid up to three times more than the top of the Clifford Chance lockstep.

A&O went to even more extreme lengths to attract Cravath derivatives guru Daniel Cunningham in 2001. Unlike Clifford Chance and Link­laters, A&O has no way to break its lockstep for marquee hires. Instead, the firm put together a onetime package, including the maximum permissible 50 points of equity, at that time worth around £1.5 million ($2.4 million), supplemented by five annual payments of £750,000 ($1.2 million) to match Cunningham’s Cravath pension entitlement.

Most radical of all, Cunningham was at that time awarded a 15 percent stake in a new stand-alone derivatives company, Derivative Ser­vices LLP, which would net him up to $15 million. Combining the three elements gave him a total compensation that the U.K. publication Legal Business, in a feature titled “Mr Bling,” calculated to be $6 million per year—more than double that of any other partner at the firm. (As is the case with Clifford Chance, A&O sources say the firm does not intend to pay partners outside its lockstep again in the U.S. A&O declined to comment, and Cunningham did not respond to requests for comment.)

One of the other weaknesses of lockstep is that when activity levels drop, firms can be left with highly paid but underutilized partners. Such tension was evident at A&O. When the derivatives market was hit hard during the recession, Cunningham’s compensation placed him under increasingly intense scrutiny from London. In 2009 he left the firm to join Quinn Emanuel Urquhart & Sullivan—now The Am Law 100′s second most profitable firm.

The U.S. market—and New York in particular—has been a long and sometimes painful learning experience for the Magic Circle. In their other international ventures, the U.K. firms were able to follow their formidable institutional client bases into countries where they were competing with smaller and less sophisticated practices, whom they were able to outbid in the recruitment market. In the U.S., the opposite was true. As powerful as the Magic Circle brand is in almost every jurisdiction in which they operate, that currency has proven to be significantly less valuable in the U.S.

The New York legal market is at once extremely sophisticated and frustratingly provincial. A deep conservatism runs through the culture of the city’s lawyers and law firms—and perhaps even more significantly for incoming firms trying to win work, through the buying habits of clients. As outsiders encroaching on this insular community, the odds were stacked against the Magic Circle. “We were always going to be viewed as British firms coming over and planting a flag,” says Linklaters’s Norton.

The U.K. firms clearly have not done as well as they would have liked, given their own high standards. But they’ve not been alone in their struggles—it’s been hard for other outsiders, too. “It’s a really tough market to break into—it takes time,” says Kenneth Doran, managing partner at Gibson, Dunn & Crutcher, which in 1982 became the first major U.S. firm not based in New York to open in the city. (New York is now Gibson Dunn’s largest office by head count.) It is true that whenever firms enter a new market, perception—in the eyes of rivals, clients, and legal directories—lags behind reality. But each of the the U.K. firms have now been in New York for more than a quarter of a century.

Still, pulling out of the U.S. isn’t an option. The global significance of the U.S. economy—and more generally of U.S. law, which is increasingly challenging the dominance of English law in markets such as Hong Kong—makes it too important a market for any international firm to ignore. Norton says that while just 6 percent of Linklaters’s 2,700 attorneys are based in the U.S., more than 40 percent of firmwide revenue—around £500 million ($810 million)—is generated by work involving a U.S. component.

The Magic Circle firms may be battle-worn, but they remain determined to finally solve the U.S. conundrum. And with their newly honed strategies, there are signs that the firms may finally be adapting to the American market, rather than simply expecting the market to adapt to them. “It’s fair to say that it has taken some time for us to refine what our goal really is,” says Linklaters U.S. co–managing partner Tenaglia. “We know we can’t do everything—our clients don’t want us to—but we definitely want to be doing more. It’s the most important battle we can fight at this stage.”