Who's getting rich off the stock market?

Only 49% of Americans own stocks.

NEW YORK (CNNMoney)

Only 49% of Americans have any money in stocks at all, according to the latest data from the Federal Reserve. That figure includes everyone invested in retirement funds (think pensions and 401(k) plans) as well as those who take the time to buy specific stocks such as Apple (AAPL, Tech30), Ford (F) and Facebook (FB, Tech30).

"Part of the reason there is so much discussion about income inequality is there is a group of people who participated in the stock market over a period where it nearly tripled, and there's another group of people that didn't," said Mark Grinblatt, a professor of finance at UCLA's Anderson School of Management.

Related: More families own cats than stocks

Even among the half of America that has money in the market, there are disparities. The top 10% of American households have roughly $282,000 each in the market, if you take the median value of their holdings.

Compare that to the middle class, which has a median value of a mere $14,000 a household.

The reality is the more money you put into the market, the more you stand to gain (or lose).

Putting a dollar into the popular S&P 500 index that tracks the largest companies traded on U.S. stock exchanges in March 2009 would leave you with $3 today. That's a nice 200% return, but obviously if you had invested $1 million in the market over the same time period, you would now have $3 million.

The stock market has been especially kind to Caucasians and college graduates.

White households typically have about three times the amount of money invested that non-white families have, according to the Fed data. That's held true since the 1990s when the market shot up during the dotcom era.

Education also plays a major role. Only 35% of households headed up by someone with a high school diploma have any money in the market. Compare that to homes led by someone with a college degree -- 72% of them have investments in equities.

Professor Grinblatt has shown through research that people with higher IQs are more likely to put money into stocks.

"Even among two siblings from the same family, the sibling who has the higher IQ is more likely to participate in the stock market," he told CNNMoney.

On the one hand, some of these disparities should be expected. Rich people have more money lying around in savings that they can invest than those who are struggling to pay their bills. But the issues go deeper than that.

Even among Americans who are working and likely to have savings, they aren't always choosing to invest.

Related: How we made nearly $1 million on Apple stock

In its latest survey, the Employee Benefit Research Institute found that only 64% of workers save for retirement or have a spouse that does so.

"It's much easier for the average American to buy a smartphone that commit to a retirement plan," says Dan Greenshields, CEO of Capitol One ShareBuilder.

In 2007, stock ownership peaked at just over 53% of American households owning any equity investments. The financial meltdown and housing crisis hurt peoples' pocketbooks, but it also shook their faith in the system.

There are calls for more education about money matters to try to close some of the stock market investing gap.

"If we give individuals responsibility for their own retirement savings, we have to make sure that they first of all save enough and that they know how to invest," said Annamaria Lusardi, a professor at George Washington University who specializes in financial literacy.

"We need to start in K-12 schools," Lusardi added. "Any year we delay by not adding financial education is one more generation out out of high school without the skills and knowledge they need."