By Liz HoffmaN and Sarah Krouse

June 2, 2016 7:29 p.m. ET

After a nearly $200 million blunder, one of America’s largest mutual-fund companies is racing to find a way to compensate thousands of clients.

T. Rowe Price Group Inc. is likely to announce as early as next week a plan to reimburse clients who lost out when it accidentally voted in favor of the 2013 buyout of Dell Inc. by its founder Michael Dell, according to people familiar with the matter.

T. Rowe had vocally opposed the deal, saying it undervalued the 30 million Dell shares held by its mutual funds. But a series of administrative errors caused the firm to vote in favor of the deal, disqualifying it from suing for more money. Other holders successfully sued and won compensation from a judge, who ruled earlier this week that Mr. Dell and his partners underpaid by $6 billion.

Such a misstep is rare and embarrassing for the Baltimore-based firm, one the biggest managers of U.S. retirement assets and a major holder of S&P 500 companies. T. Rowe had $764.6 billion in assets under management at the end of March.

Trading and other operational missteps can happen at mutual-fund firms, but voting errors on major deals like this one are unusual. T. Rowe’s costly mistake highlights how the largely automated proxy voting process is vulnerable to human error. The company launched a review of its internal voting procedures, according to a person familiar with the matter.

There is no guarantee that the compensation plan will be completed, and the final form it could take is still being discussed, the people added.

“We are in the process of reviewing the opinions and considering our options,” a T. Rowe spokesman said Thursday.

T. Rowe investors stand to miss out on roughly $190 million—about $4 a share, plus interest, on some 30 million Dell shares held across several of its funds. The payment package could head off potentially costly lawsuits from disgruntled fund investors and any potential inquiries from regulators.

The T. Rowe fund investors are arguably worse off than Dell investors who knowingly voted to accept the $13.75-a-share deal consideration back in 2013. Those shareholders were paid immediately when the deal closed in October 2013. But because T. Rowe sued to challenge the deal price, its fund investors had to wait more than two years for payment.

The money manager has blamed its vote in favor of the deal on a back-office error. The default stance of T. Rowe in merger votes—like many large investors—is to support management, which in Dell’s case recommended in favor of the deal. So T. Rowe Price’s computerized system spit out instructions to vote “yes” that weren't manually overridden before the final vote, according to court filings.

T. Rowe held Dell shares in a number of its mutual funds including the Equity Income fund and the Science and Technology fund in 2013. The Equity Income fund is up 6.27% over three years, compared with an 11.11% return for its benchmark, the S&P 500, according to Morningstar Inc., while the Science and Technology fund returned 15.85% over three years, ahead of a 5.17% benchmark return.

T. Rowe publicly filed a record of some 2013 fund proxy votes, including the fateful Dell ballot, in August of 2014, as all large investors must do annually. But T. Rowe said in court that it didn’t become aware of the voting error until two months later, when an analyst asked why it had voted in favor of the deal, according to court filings.

In February 2013, just a week after the buyout was announced, T. Rowe joined a chorus of investors who argued it was unfair. While T. Rowe is typically more vocal than its competitors, mutual funds rarely speak out publicly on corporate mergers. Its opposition lengthened the deal’s odds of passage.

Mr. Dell eventually agreed to raise the price by 10 cents a share, and shareholders approved the deal. Soon after, T. Rowe and a handful of other disgruntled investors filed for a so-called court “appraisal,” asking a judge to determine the fair value for Dell’s stock.

Dell challenged T. Rowe’s standing to sue after the voting snafu was uncovered. T. Rowe argued in court that it had never taken “deliberate, affirmative action” to vote in favor of the merger. And documents produced during the trial showed that Dell’s advisers had marked T. Rowe down in the “against” camp as they tallied votes in the lead-up to a hotly contested meeting.

T. Rowe’s mistake is partly attributable to the complexities of both the way shareholder votes are cast and the twists and turns of the Dell deal itself. Ahead of the first scheduled vote, in July 2013, T. Rowe governance chief Donna Anderson manually overrode the default “for” vote, and T. Rowe’s votes were logged as against the transaction, according to court filings.

But when the deal was renegotiated, the vote was postponed, and T. Rowe’s instructions were effectively vacated. The firm did not override the default “for” election when the vote was later held, according to court filings.

“Asset managers in general, particularly active managers, typically take a hands-on approach to these shareholder votes,” said Todd Rosenbluth, director of exchange-traded-fund and mutual-fund research at S&P Global Market Intelligence. “Voting mistakes like this are few and far between.”

One of the biggest beneficiaries of this week’s court ruling was silent as the debate over the buyout raged. Hedge fund Magnetar Capital LLC stands to pocket about $25 million in profits.

Most of the investors who voiced opposition to the buyout either voted to accept the deal price or sold their shares before voting. They include billionaire activist Carl Icahn, who led a charge for an alternative buyout plan, and his partner Southeastern Asset Management Inc., a longtime Dell holder.

Mr. Icahn said he opted not to sue, a decision that cost him as much as $1 billion, because he had “better uses” for his cash. Around the same time, Mr. Icahn took a stake in Apple Inc., a trade the outspoken investor said ultimately delivered around $2 billion in profits.

Write to Liz Hoffman at liz.hoffman@wsj.com and Sarah Krouse at sarah.krouse@wsj.com