By Timothy W. Martin

Updated Sept. 2, 2015 9:16 p.m. ET

The nation’s second-largest pension fund is considering a significant shift away from some stocks and bonds, one of the most aggressive moves yet by a major retirement system to protect itself against another downturn.

Top investment officers of the California State Teachers’ Retirement System have discussed moving as much as 12% of the fund’s portfolio—or more than $20 billion—into U.S. Treasurys, hedge funds and other complex investments that they hope will perform well if markets tumble, according to public documents and people close to the fund. Its holdings of U.S. stocks and other bonds would likely decline to make room for the new investments.

The board of the $191 billion fund, which is known by its abbreviation Calstrs, discussed the proposal at a meeting Wednesday. A final decision won’t be made until November.

A wave of deep selloffs over the past two weeks has shattered years of steady gains for U.S. stocks. Calstrs isn’t reacting directly to those sharp price swings, but they are a reminder of the volatility in stocks and how exposed Calstrs is when markets swoon.

“There’s no question,” Calstrs Chief Investment Officer Christopher Ailman said in an interview. The recent market volatility “has been painful.”

Calstrs currently has about 55% of its portfolio in stocks. The fund’s investment officers began discussing the new tactic—called “Risk-Mitigating Strategies” in Calstrs documents—several months ago as they prepared for a regular three-year review of how Calstrs invests assets for nearly 880,000 active and retired school employees.

Mr. Ailman, who has been chief investment officer at the fund since 2000, said he hopes a move away from certain stocks and bonds could help stub out heavy losses during future gyrations. This could include moving out of some U.S. stocks as well as investment-grade bonds.

Pension funds across the U.S. are wrestling with how much risk to take as they look to fulfill mounting obligations to retirees, and the fortunes of most are still heavily linked with the ebbs and flows of the global markets. State pension plans have nearly three-quarters, or 72%, of their holdings in stocks and bonds, according to Wilshire Consulting.

Many retirement systems are now looking to get even more defensive as they lower their investment ambitions and acknowledge a multiyear rally in U.S. stocks and bonds may be nearing an end. Public pensions earned just 3.4% in the year that ended June 30, according to Wilshire, one of their worst showings since 2008 and well below many internal targets.

“We’re not going to shoot for the moon for returns, because we could lose,” said Annette St. Urbain, chief executive of the $2.5 billion San Joaquin County Employees’ Retirement Association, which is also evaluating a risk-mitigating strategy.

In recent decades many funds plowed into real estate, commodities, hedge funds and private equity holdings as a way of offsetting losses from their primary investments, but that strategy faltered during the 2008 financial crisis when many pension suffered heavy losses.

Their countermoves in recent years—which include a bigger shift into non-U.S. equities—have left funds more exposed to problems around the world. State pension fund investments in foreign stocks have grown to 21% of their portfolios on average from 18.8% from 2008, according to the Wilshire. During the same period, holdings of U.S. stocks dropped to 27.9% of all public pension holdings from 38.1%. Holdings of real estate and private-equity funds rose.

The proposal at Calstrs differs from the defensive strategy unfolding roughly a mile away at the larger California Public Employees’ Retirement System, which decided last year to exit all hedge-fund investments as a way of reducing its reliance on complex holdings. Other pensions seeking to diversify have beefed up stakes in bonds or international stocks.

One fund taking an approach similar to Calstrs is the Hawaii Employees’ Retirement System, which later this month could approve a shift of between 10% and 20% of its $14.4 billion in assets out of stocks and certain bonds into what it considers to be safer bets of U.S. Treasurys and so-called liquid-alternative funds that mimic hedge-fund strategies.

“You take away some of the human judgment or hubris,” said Vijoy Paul Chattergy, the pension fund’s chief investment officer.

The Calstrs documents propose a range of commitments to the new strategy, from zero to 12%. In addition to Treasurys and hedge funds, the new holdings could include liquid-alternative funds, according to people close to the fund. The goal is to find investments that don’t track as closely with market swings.

“I’ll equate this to the cost of insurance,” Mr. Ailman said. “It’s the idea of adding more of a hedging asset class.”

Commitments to infrastructure projects would also climb, according to the documents. Holdings of foreign stocks could rise or fall, depending on what the board approves.

Calstrs hasn't made any major adjustments to its portfolio as world markets seesawed. Mr. Ailman said he knew there would be turbulence after Asian markets tumbled last month but said Calstrs chose to stay put because it views itself as a long-term investor.

Write to Timothy W. Martin at timothy.martin@wsj.com