By Rob Copeland

Sept. 10, 2015 12:15 a.m. ET

BOSTON—Hedge-fund managers have long clung to a doctrine of high fees in good years and bad, minting billionaires and riling investors along the way.

A pair of former Harvard University endowment executives have built the world’s largest stock-focused hedge fund with the opposite approach. Robert Atchinson and Phillip Gross let investors in their $28 billion Adage Capital Management LP keep almost all of their trading gains—and promise refunds if the fund’s performance falters.

By hedge-fund standards, the practice is nearly unheard of, and this year it may be particularly costly. Adage has underperformed the S&P 500 through August. If they can’t turn things around by the end of the year, Messrs. Atchinson and Gross may hand out the biggest refunds in their firm’s history.

Hedge-fund managers typically collect about 2% of investors’ assets under management every year and about 20% of profits—whether they beat the markets or not.That has put the industry under increasing pressure for what is viewed by critics as a heads-I-win, tails-you-lose proposition. Even managers who fall short of their benchmarks can earn hefty paychecks simply by riding markets higher.

At Adage, Messrs. Atchinson and Gross charge 0.50% of assets, less than at even many actively managed mutual funds. They also pay themselves only for trading profits that beat the S&P 500, including dividends—they take 20% of the returns above the S&P. And they pay back up to half of those fees if they fall short of the benchmark in the subsequent year.

Its system pays off for Adage in one important way: It collects its full incentive fees even if it loses money, as long as it beats the benchmark. Most fund managers collect performance fees only if they produce positive absolute returns.

Adage is getting harder to dismiss as an anomaly. The firm has nearly doubled in size over the past four years and is now one of the world’s 10 largest hedge funds. Its approach also reflects a bigger structural change in the industry. Many of the public retirement plans and other institutional investors that have poured money into hedge funds for years have hit their limits on what they can allocate to the sector. To keep the gas pedal on growth, hedge-fund managers are increasingly recasting themselves as something closer to staid money managers like Fidelity Investments. In many cases, that means taking a different tack on fees.

For instance, a new wave of single-bet deals, known as co-investments when sold by firms like Magnetar Capital LLC and Trian Fund Management LP, often carry half-off prices on performance fees compared with traditional hedge funds.

“That’s how I think the world should look,” said Jack Meyer, the former head of Harvard’s endowment. Mr. Meyer’s $10 billion Convexity Capital Management LP only collects performance fees when it exceeds a market benchmark. “I’m surprised it has taken the world so long to get there,” he said.

Despite the drumbeat for change, hedge funds as a group haven’t altered their practices. Money continues to pour into the $2.9 trillion industry as plenty of wealthy investors still appear willing to pay high fees for access to complicated strategies and the promise of protection against market swoons. The average hedge fund charges a 17.7% annual incentive fee, down from 19.3% at the start of 2008, according to HFR Inc.

Still, the industry has come under an unusual amount of pressure that threatens its ways of doing business long term. The California Public Employees’ Retirement System, the nation’s largest public pension, said last fall it would eliminate its entire $4 billion investment in hedge funds due in part to concerns about high costs, and other public pensions have followed suit.

From offices high in Boston’s John Hancock Tower, Adage’s founders preside over an operation that bears little resemblance to most hedge funds. There is no central trading desk or bullpen area to swap ideas, people familiar with the firm said.

Investment analysts, 26 in total including the founders, work silently in separate offices ringing the floor. Six golf putting holes dot the space, people familiar with the firm said, though they are rarely used except for a half-hour tournament each December in which the winner gets no money. A person who has been inside described it as a “library.”

Adage’s founders still make plenty of money. Last year, the founders and staff split an estimated $400 million in gross earnings after the firm’s main fund rose 18.4%, people familiar with the matter said. Still, that is less than a third of the more than $1 billion they would have collected under a typical hedge-fund fee structure.

The founders break from tradition in small ways, too. Mr. Atchinson drives a seven-year-old Nissan Altima, and Mr. Gross takes the Peter Pan bus to his summer home on Cape Cod.

“That’s just how they approach life in general,” said Timothy Peterson, founder of Regiment Capital Advisors LP and a former Harvard colleague of the founders. “They feel very comfortable not being the center of attention.”

Because of its unusual practices, Adage is viewed by its large investors as its own breed. The University of California system’s $98 billion endowment has $1.4 billion riding on Adage—more than in any other stock investment except for three low-cost index funds, which it considers to be in the same category. Endowment Senior Managing Director Scott Chan said, “a lot of people would call them a hedge fund,” but that he isn’t particularly concerned about the distinction.

Since it was started with $3.8 billion in October 2001, Adage has produced an average annualized return, after fees, of about 9.7%, beating the S&P 500’s 6.4% pace, including dividends. The average stock-focused hedge fund averaged 5.3%. Adage is down 3.53% this year through the end of August, worse than the benchmark’s drop of 2.88%, including dividends.

Until this year, Adage fell short of the benchmark and passed out refunds only twice. That happened in 2002 and 2008, after it lagged behind the S&P 500 by 0.18 percentage point and 0.75 percentage point, respectively.

Unlike most hedge funds, which juggle a variety of offsetting bets, Adage holds far more bullish bets than bearish ones and keeps its proportion of the wagers static. Bets on industries must be equal, percentagewise, to that sector’s representation in the S&P 500. That means the firm’s more than 1,500 stock positions tend to ride the ups and downs of the benchmark. Any difference comes from individual stock picks, such as the firm’s roughly $500 million position in Puma Biotechnology Inc., which has more than quadrupled in value since the start of 2013.

Messrs. Atchinson and Gross met as investment analysts in the mid-1980s at Harvard’s endowment. Mr. Atchinson, 57 years old, traded mostly aerospace and defense firms, while Mr. Gross, 55, focused on drug companies and other health-care stocks.

When Mr. Meyer took charge of the endowment in 1990, he put his traders on notice: Beat the benchmark or find a new job. Many left. Messrs. Atchinson and Gross took on the challenge, initially topping the benchmark by 0.50 percentage point and more in subsequent years. They got rich—Harvard paid them a bonus of around 4% of their outperformance, at peak amounting to more than $10 million apiece—but drew scorn from some alumni and professors unhappy about the hefty paydays for an academic institution.

In part because of the scrutiny, Mr. Atchinson persuaded Mr. Gross to leave in 2001. They took with them an 18-person team and $1.8 billion day-one investment from Harvard in exchange for an agreement, since phased out, to tithe a portion of their firm’s future earnings to the school.

The founders’ move from academia to hedge funds came with an added bonus: a chance to refashion themselves as symbols of frugality after years of criticism for their pay. Adage’s unique position, with one foot inside the lucrative hedge-fund industry and one foot out of it, is perhaps most plain on weekends during the summer.

Like many hedge-fund managers, Mr. Atchinson, who goes by Bob, heads to his vacation home on Nantucket. Unlike his peers, Mr. Atchinson usually hops on a commercial Cape Air flight. When he does fly private, it is on “Air Mom,” his name for the single-engine turboprop in which he owns a one-eighth share and often ferries around his mother.

Back at Boston’s Logan Airport on Monday mornings, he waits for a taxi to take him into the office.

Write to Rob Copeland at rob.copeland@wsj.com