Hedge funds lost their bid for a higher price in the 2012 buyout of family-tree website Ancestry.com, showing the limits to what’s become an increasingly popular investing strategy.

A Delaware judge on Friday awarded Merion Capital LP and funds advised by Ancora Advisors LLC $32 a share for their Ancestry.com stakes, the same price other shareholders got when private-equity firm Permira bought the company for $1.6 billion.

The funds had filed so-called appraisal claims, using an increasingly popular legal tool to get more money from corporate buyouts. Historically, most such cases have succeeded in getting at least a small bump from a judge.

Merion, which owned about 3% of Ancestry.com, and Ancora, which owned less than 1%, had argued the company was worth at least $42.81 a share, and possibly as much as $47. The company’s experts countered the fair value was only $30.63 a share.

Finding both arguments “problematic” and noting that Qatalyst Partners had run a “robust” sales process, the judge said the $32-a-share deal price was “the best indicator of Ancestry’s fair value,” according to a court filing.

“This was a swing and a miss,” said Minor Myers, a professor at Brooklyn Law School who has studied appraisals. “Not every appraisal case is a winner, and this opinion makes clear that picking the good cases is hard.”

The funds will receive 5.75% interest on their stakes, accruing from when they filed the suits.

A lawyer for the funds didn’t immediately respond to a request for comment.

Hedge funds are increasingly using appraisal claims as an investing strategy. A record 33 appraisal claims challenging public-company takeovers were filed in 2014 in Delaware, the legal home to most U.S.-listed firms, according to a Wall Street Journal review of court documents. They have challenged the buyout prices in deals for companies including Dell Inc., BMC Software Corp. and Dole Food Co. Most of the recent cases are still pending.

And the strategy is attracting new and larger players. Merion, among the most active appraisal investors, has nearly $1 billion under management. Magnetar Financial LLC, Fortress Investment Group LLC and Gabelli Funds have all filed appraisal cases in recent months.

The next big test for the strategy is coming next month, when a trial is set to begin over Dole’s 2013 buyout. Funds including Merion and Fortress are challenging the $13.50-a-share price paid by the company’s founder.

Write to Liz Hoffman at liz.hoffman@wsj.com