Is it time to exit stocks?

NEW YORK (CNNMoney)

It's great advice, except for one problem: it's incredibly hard to tell the peak or the nadir.

Billionaire Carl Icahn has made a lot of money off his investments. Even he says, "No one is smart enough to know when to buy at the bottom."

After the worst stock market drop of the year, plenty of people wonder if it's time to sell. The argument goes something like this: The S&P 500 is up nearly 200% since it bottomed out in early 2009. There hasn't been a correction since 2011, let alone a true "bear market" where things really drop. Surely it's time to exit...

But here's the truth: Most of us are better off doing nothing.

Yes, you read that right. Think about these points:

1) Put the recent market dip in perspective. The market is down, but only a tad off its all-time highs from September.

"This is not earth shattering," says Jurrien Timmer, director of Global Macro for Fidelity. "Volatility happens. A couple of percent is not a big deal. The market will correct by 10% almost any time. Over the course of history, it shows up as really nothing more than noise."

Related: Stocks should fall more. You shouldn't worry

2) Never sell in a panic. Selling when you're anxious often means you sell at the low point and end up buying back in at a higher one -- the exact opposite of what you hope to do.

"The worst time to decide to get out of the stock market is when it's falling or right after a big drop," says Kate Warne, an investment strategist at Edward Jones. "If you look at the days with the best market performance, a lot of them follow days with the worst market performance."

Even if you do manage to get out a the right time, it can still hurt you in the end.

"If you decide to get out, you have to make two decisions correctly: both when to get out and when to get back in," Warne says.

Think of it like rolling two dice and having to guess BOTH numbers correctly.

3) Let history be your guide: Many of the biggest stock market upswings happen within days of a downturn. You could be off celebrating that you exited the market perfectly only to look like a fool for not catching the surge.

Consider this: Fidelity looked at the returns of the S&P 500 from 1980 through June 30, 2014. They found that an investor who missed only the best five days in the market would end up with a portfolio worth about 35% less than the one that had been fully invested the whole time.

I ran a similar analysis just for 2014 for the S&P 500.

Your return if you stayed in the market and did nothing: +3%

Your return if you managed to miss the 10 biggest down days: +21%

Your return if you managed to miss the 10 biggest positive days: -9%

This is totally hypothetical. No one is that good or bad at market timing.

But it illustrates the key point: Everyone wants to be the superman or superwoman with the 21% return and miss all the worst days. But what you really want to avoid is being the person who missed out on the gains.

Related: Will you have enough to retire? Check the calculator

4) Get a long-term plan and stick to it. If you are worried about the stock market right now, the best use of your time is to make sure you have the right plan.

Do you have the right mix of stocks and bonds and other assets?

"The typical long-term investor who has most of his or her investments in a retirement account of some sort needs to a) have a plan and b) stick with the plan," says Timmer.

From time to time, investors should look at their portfolios and ask if they need to rebalance. Given the phenomenal rise of equities since March of 2009, it's possible that you need to trim your stock allocation a bit.

But be wary of throwing out the entire game plan. Stocks still give the best return by far over the long run.

As humans, we worry a lot about those down days. But we forget that the bigger risk is missing the upswings.