Hedge funds looking for higher payouts in corporate buyouts scored a win this week.

In a case stemming from the 2012 buyout of Ancestry.com Inc., a judge on Monday ruled shareholder Merion Capital LP didn’t have to prove it voted its shares against the family-tree website’s buyout to challenge the deal’s $1.6 billion price tag in court.

The decision keeps open what has become an increasingly popular strategy—known as “appraisal arbitrage”—for these investors: buying up shares of companies on the cusp of a takeover, opposing the deal and then seeking more in court in a legal process known as appraisal.

At issue in the Ancestry.com case was whether Merion could prove its shares weren’t voted in favor of the sale to European private-equity firm Permira. Appraisal seekers must abstain or vote “no” on a deal.

Merion had bought its 3% stake too late in the process—just days before the buyout vote—to be eligible to vote them itself.

Instead, its shares were nominally held by Cede & Co., a centralized warehouse for stock certificates. In a buyout, Cede acts as an aggregator, collecting ballots from shareholders and then voting its stock in bulk accordingly. Cede held 29 million shares of Ancestry.com, of which about 10 million either voted “no” or abstained, according to court filings.

Ancestry.com said Merion couldn’t prove its shares were among them, and indeed, when asked in a deposition, a Merion executive said he couldn’t be sure.

But the judge said there were enough Cede votes against the buyout to “cover” Merion, which only own 1.3 million shares. Historically, courts didn’t scrutinize the issue as long as the total number of shares seeking appraisal didn’t exceed the number of shares that abstained or voted “no”—as was the case here.

The ruling’s takeaway also applies to a similar defense mounted by BMC Software Inc. against Merion in pending appraisal litigation over BMC’s 2013 buyout. The judge on Monday let Merion’s claims, valued at some $350 million based on the buyout price, go forward.

A decision the other way would have complicated appraisals by forcing hedge funds to buy their shares far earlier in the process. That increases their risk of the deal going bust and crimps their annualized returns by tying up their money longer.

The appraisal strategy has gained popularity in recent years on the heels of several big wins for shareholders. A record 33 appraisal claims stemming from public-company takeovers were filed last year in Delaware, the legal home to most U.S. listed firms, according to a Wall Street Journal review of court documents.

And the strategy is attracting new and larger players. Pennsylvania-based Merion has nearly $1 billion under management, according to a person familiar with the matter, and some $750 million tied up in pending lawsuits. Magnetar Financial LLC, Fortress Investment Group LLC and Gabelli Funds all have active appraisal cases.

“Sophisticated investors are seeing significant valuation gaps in certain deal prices,” said Kevin Abrams, a lawyer for Merion, adding that he expects more to come.

About 81% of Delaware appraisals that went to trial since 1993 have yielded higher prices, according to law firm Fish & Richardson PC. Merion has averaged an 18.5% annualized return across five completed appraisals, four of which settled, according to documents reviewed by the Journal.

Write to Liz Hoffman at liz.hoffman@wsj.com