An activist investor wants publisher Houghton Mifflin Harcourt Co. to further turn the page from its recent bankruptcy and hand back more cash to shareholders.

Q Investments LP, which owns about 3% of Houghton’s stock, is seeking a shareholder vote to pressure the company to either institute a dividend or boost its share repurchases, according to a letter it released Thursday.

Houghton, which specializes in educational books and digital learning, has a “fortress balance sheet” that is far too protective in a market where debt is cheap, Q Investments said. The investor’s push comes as at least one competitor in educational publishing, Cengage Learning, turned to the debt markets this week.

A spokeswoman for Houghton Mifflin Harcourt declined to comment.

The company, whose primary focus is preschool through 12th grade, also has a small consumer arm whose authors include Philip Roth, Jonathan Safran Foer and Yann Martel. Houghton emerged from a brief stay in bankruptcy in 2012 and went public in 2013 at $12 a share. Its shares closed Thursday at $20.30, giving the company a market capitalization of $2.87 billion.

On a Nov. 6 earnings conference call, Houghton Chief Financial Officer Eric Shuman said the company expects to end the year with a cash balance of between $675 million and $725 million, up from $425 million at the end of 2013. Houghton’s board authorized the repurchase of $100 million in common stock over a two-year period because of its expected strong cash position.

In an interview last month with The Wall Street Journal, Houghton Chief Executive Linda Zecher emphasized the importance of delivering value for shareholders. “You want to make sure you’re focused on growth and that you’re spending your capital judiciously,” she said. “And you want to make sure that anything you’re doing will accrue to the stock price and to the value of the company.”

Houghton has been on something of an acquisition spree. It has purchased three digital education-related companies in 2014 as it looks to build its digital portfolio and increase its opportunities for selling directly to consumers. Ms. Zecher also said Houghton would be open to a larger deal if it would expand the company’s footprint.

In the letter to the board, Q Investments said hoarding cash for a major acquisition is unnecessary. While the investor said it would support a deal, if merited, it believes the company should return some of the cash on hand because the debt markets could finance any significant transaction.

Q Investments didn’t specify how much cash it believes should be returned to investors, but suggested a dividend could signal that Houghton has confidence in its long-term prospects. In the letter, the activist fund voiced support for management and noted the company’s progress since bankruptcy.

Earlier this week, Cengage, another educational publisher, said it was planning to use debt to pay a dividend only eight months after it emerged from bankruptcy.

Write to David Benoit at david.benoit@wsj.com and Jeffrey A. Trachtenberg at jeffrey.trachtenberg@wsj.com