In this article, I'll explore why most investors underperform the market, specifically the S&P 500 Index. Many investors, from novices to seasoned professionals, aspire to outpace this benchmark. However, only a handful consistently achieve this over the long term. We'll delve into the reasons behind this widespread underperformance and present the statistics and research that support these claims. Moreover, considering that retail investors often don't have the same resources and expertise as full-time professionals, who themselves struggle to beat the market, we'll discuss whether attempting to outperform the market is truly worth the effort.
Evaluating Underperformance Statistics
The term "beating the market" refers to achieving an investment return that surpasses the performance of the Standard & Poor's 500 Index, commonly known as the S&P 500 Index. The S&P 500 Index, representing about 80% of U.S. market capitalization, is the leading benchmark for large-cap U.S. equities. It comprises 500 top companies listed on U.S. stock exchanges.
Professional Equity Funds Performance
Based on findings from the SPIVA U.S. Scorecard Mid-Year 2023 (which is run by the S&P Dow Jones Indices), over the 10-year, 15-year, and 20-year periods, all large-cap U.S. equity funds underperformed their benchmarks (the S&P 500) by 85.61%, 92.19%, and 93.58% respectively. So, while professional equity fund managers already find it challenging to beat the market, statistics show that their likelihood of underperformance increases as time horizons lengthen (as illustrated in the chart below):
It's also worth examining the chart below, to understand the historical U.S. large-cap equity fund underperformance percentage on an annualized basis:
Notably, in 2022, even with conditions leaning in favor of stock pickers, 51% of actively managed large-cap U.S. equity funds failed to outperform the market. To put this in perspective, you'd have to go back to 2009 to find a lower underperformance rate, which was at 48%.
As The New York Times reports, the S&P 500 saw a 19.4% decline in 2022, largely driven by mega-cap tech stocks like Google, Amazon, Apple, Meta (Facebook), Microsoft, Nvidia, and Tesla. These giants, which constitute a significant portion of the market-weighted S&P 500 Index, declined in value while the broader market fared better. This is evident when comparing the S&P 500 Index to the S&P 500 Equal Weight Index: the latter outperformed the former by ~7% in 2022 (as the chart below shows).
In short, this suggests that a random selection from the S&P 500 would likely have surpassed the index itself, especially considering the broader market's resilience in the face of underperforming mega-caps. Yet, despite a landscape ripe with promising options for stock pickers, 51% of active large-cap U.S. equity fund managers failed to outperform the market.
Average Retail Investor Performance
Now, shifting our focus from seasoned financial professionals, let's consider the typical retail investor.
The following chart provides a snapshot of the average annualized performance of retail investors over a 20-year period:
While the data spans from 1998 to 2017 and might seem dated, its 20-year period offers enduring insights. As illustrated, the average investor's return during this timeframe was 2.6%, marginally above the average annualized inflation rate of 2.1%. The typical retail investor barely achieved positive real returns from their investments. More critically, they lagged the S&P 500's average annual return by a substantial 5.2%.
In comparison, Dalbar's annual Quantitative Analysis of Investor Behavior (QAIB) report reveals that over a 30-year span (from January 1, 1993, to December 31, 2022), the average U.S. equity fund investor returned 6.81%, underperforming the S&P 500 Index by 2.81%. When projecting these average annual returns over the 30 years, a $100,000 investment would yield a striking difference of $864,138 in favor of the S&P 500.
It's hardly surprising that professional investors, whose primary job is to manage others' funds, would significantly outperform the average retail investor (with returns of 6.81% compared to 2.60%, though these figures stem from different time periods). The typical retail investor, perhaps dedicating only a few hours weekly without any significant market advantage, would understandably find it challenging to surpass the returns of the S&P 500, much less the average U.S. equity fund.
4 Key Reasons on Why Most Investors Underperform the Market
Below are 4 key reasons on why most investors underperform the market.
Reason #1: Efficient Market Hypothesis
The Efficient Market Hypothesis (EMH) is a foundational concept in finance that originated in the early 20th century, but it was more formally developed in the 1960s and 1970s. This theory is often attributed to Eugene Fama, who, in his 1970 paper "Efficient Capital Markets: A Review of Theory and Empirical Work," presented a comprehensive analysis of the existing research on market efficiency.
At its core, the EMH states that stock prices fully reflect all available information. This means that at any given time, stock prices are a fair representation of what is known about a company and its environment. The hypothesis is typically divided into three forms based on the extent and type of information considered:
- Weak Form: Stock prices reflect all past trading information, including stock prices and volume. Under this form, technical analysis cannot be used to achieve superior returns.
- Semi-Strong Form: Stock prices adjust rapidly to all new public information. This suggests that neither technical nor fundamental analysis can provide superior returns consistently.
- Strong Form: Stock prices reflect all information, public and private. This means that even insider information cannot help an investor consistently achieve superior returns.
So, the EMH suggests that stock prices reflect all available information, making it tough to consistently out-perform the market. If every stock is fairly priced, identifying undervalued or overvalued stocks becomes nearly impossible. The EMH also indicates that stock prices adjust to new, unpredictable information, leading to random future price movements. While there might be instances of investors beating the market, the EMH largely attributes such successes to luck rather than skill, especially over long periods of time.
While it seems logical that a market with more participants would become more efficient, as increased competition brings diverse information to influence prices, the EMH remains a fairly controversial topic.
Behavioral finance argues that psychological biases can introduce market inefficiencies. Historical financial bubbles, certain active managers' consistent out-performance, and stock return anomalies challenge the EMH's core thesis. Furthermore, issues like information asymmetry, limits to arbitrage, market manipulation, and market microstructure effects, such as liquidity concerns, highlight the complexities of real-world markets that might not always align with the EMH's predictions of perfect efficiency.
Generally, the weak form of EMH is the most widely accepted, even though it has its nuances and exceptions. Despite its criticisms, EMH still offers a persuasive reason, at least to some extent, for why beating the market is such an impossible task for investors.
Reason #2: Investor Psychology and Emotions
Investor psychology undeniably influences the stock market, often making the task of consistently outperforming it a formidable challenge. Emotions like fear and greed, which are central to investment decisions, can lead to irrational behaviors. For instance, these emotions can fuel a herd mentality, causing investors to buy stocks when prices are high and sell when they're plummeting. This reactive 'buy high, sell low' approach is counterproductive. In contrast, disciplined investors who manage their emotions and adhere to a long-term strategy are better positioned to outperform the market.
The Prospect Theory, introduced by Nobel laureates Kahneman and Tversky, delves deeper into these emotional biases. It reveals that the psychological pain of losing $100 is more intense than the pleasure derived from gaining the same amount. Such a bias towards loss aversion can lead investors to make decisions that are not in their best financial interest. Moreover, the enticement of potential gains can blind investors to inherent risks, making them susceptible to long-term financial setbacks.
One key indicator that captures the sentiment of fear in the market is the VIX, often referred to as the "fear index". The VIX measures the market's expectation of volatility over the next 30 days, and spikes in the VIX often correlate with periods of market turmoil. For reference, the live VIX chart is shown below:
A rising VIX indicates that investors expect increased volatility and are likely feeling uncertain or fearful about the market's future direction. By monitoring the VIX, investors can gain insights into the collective emotional state of the market, emphasizing the profound influence of investor psychology on market dynamics.
Reason #3: Investment Fees and Taxes
Consistently outperforming the market is no simple task, with investment-related costs being a primary obstacle. Even when investing in an S&P 500 index fund, which aims to replicate the performance of the S&P 500, the associated fees inevitably diminish the returns. While these fees for passive S&P 500 tracking ETFs might seem minimal, ranging from ~0.03% to ~0.2% annually, their impact compounds over time, subtly eroding returns and causing your performance to lag the S&P 500.
Imagine you allocate $100,000 to an S&P 500 ETF with a 0.05% annual expense ratio for 30 years. If the S&P 500 Index yields an average 10% return annually, mirroring its historical trend, and the ETF perfectly replicates the S&P 500 without any tracking errors — a feat even top-tier ETFs struggle with — we can then isolate and examine the sole impact of the expense ratio on the 30-year returns.
After 30 years, without additional contributions to the initial investment, a mere 0.05% expense ratio leads to a difference of $23,638.47. This underscores the influence of even a minimal expense ratio over time because of compounding. At the same time, it illustrates the appeal of passive ETFs to investors: a relatively modest cost of just over $23K across three decades, while still realizing a return of over $1.7M (gross of fees and taxes).
Tax implications further complicate the quest for superior returns. Capital gains tax, levied on profits from selling investments, can range from 15% to 20% depending on one's income bracket. Short-term holdings, or stocks held for less than a year, face taxation as ordinary income, which can escalate to rates as high as 37% for top earners. This tax structure means that the frequent trading strategies employed by hedge funds, mutual funds, actively-managed ETFs, and other active traders can result in heftier tax bills.
It's also worth mentioning that the act of frequent trading itself incurs costs, albeit to a lesser degree over the last few years, due to advancements in technology and the rise of commission-free investment platforms. These fees can potentially consume a portion of your trading capital, necessitating even higher returns to match the market's performance. Further, the more frequent the trading, the more fees accumulate, raising the bar for the returns needed.
Reason #4: Difficulty in Selecting Great Stocks
While the three reasons mentioned above may not be a significant concern for seasoned investors, the task of curating and managing a stock portfolio to surpass the broader market is daunting, especially when most stocks struggle to match market returns.
In the study "Do Global Stocks Outperform US Treasury Bills?", an analysis of the compound returns of nearly 62,000 global common stocks from 1990 to 2018 revealed that 56% of US stocks and 61% of non-US stocks trailed behind one-month US Treasury bills, which are effectively risk-free investments. Additionally, the study points out that only 1.3% of the best-performing global firms accounted for the total net gain of the stock market over the years, which highlights the inherent difficulty in stock selection and the importance of strategic diversification in an investment portfolio.
By diversifying, investors risk excluding one of the few stocks that could significantly boost their annual returns. As J.B Heaton elaborates in "Why Indexing Works", in a hypothetical scenario with five stocks where four yield a 10% return and one yields 50%, the market's cumulative performance would be 18%. However, any portfolio lacking that high-performing stock would inevitably fall short.
Should Retail Investors Attempt to Pick Stocks to Beat the Market?
While some renowned investors have achieved remarkable success in this endeavor, the broader data suggests that both retail and professional investors often struggle to surpass market benchmarks. The intricate dynamics of market returns further complicate this task. As we know, only a small fraction of stocks significantly outshine their counterparts over extended periods, suggesting that the distribution of stock returns is skewed. This skewed pattern indicates that a majority of stocks don't generate returns comparable to the market average.
Thus, the task of identifying those few stocks that will surpass the market becomes analogous to searching for a needle in a haystack. The compounded effect of returns further complicates this endeavor, as a couple of missteps in stock selection could substantially hinder an investor's overall returns.
Clearly, stock picking and making a worthwhile return is no easy endeavor. However, given that beating the market is possible, it's worth comparing when it may or may not make sense to pick stocks in an attempt to beat the market.
When It Might Make Sense
- Educational Value: Stock picking can be an invaluable learning tool for beginners, providing insights into company evaluations, understanding market dynamics, and navigating the emotional aspects of investing.
- Passion and Expertise: Deep knowledge within one's "circle of competence" can offer a distinct advantage in stock selection. For instance, someone with a comprehensive understanding of renewable energy may have a unique edge when evaluating stocks within that sector.
- Access to Information: While trading on non-public, insider information is illegal and unethical, obtaining exclusive insights not accessible to the general investor pool can enhance the potential for superior returns over time.
- Risk Tolerance: Individuals with a higher risk tolerance are better positioned for stock picking, recognizing the potential for significant long-term gains while acknowledging the inherent volatility and uncertainties associated with individual stock selections.
When It Might Not Make Sense
- Time Commitment: Comprehensive stock analysis requires significant time and dedication. For those unable to commit the necessary hours, passive investment strategies might be more appropriate.
- Emotional Bias: Cognitive biases can influence decision-making. Investors who are easily swayed by emotions, rather than relying on data-driven analysis, may find that stock picking adversely affects their portfolio's performance.
- Diversification: Selecting individual stocks can expose an investor to unsystematic risks. Ensuring a diversified portfolio is crucial to counteract risks tied to specific industries or companies.
- Costs: Active trading often results in higher transaction fees and tax implications, which can erode overall returns. Additionally, active traders must be fully informed, and the need to screen and analyze a variety of stocks might necessitate additional expenses for stock-picking tools.
Even seasoned professionals, dedicating extensive hours to market analysis, often find consistent out-performance elusive. The best in the business still face a significant margin of error in their stock picks.
The unpredictability of the stock market is a testament to the interplay of skill and luck in investment success. Renowned investors like Peter Lynch and Warren Buffett stand as outliers in a sea of professionals who grapple with consistent market out-performance. Some experts even suggest that luck plays as significant a role as skill, as evidenced by the research of economists like Eugene Fama and Kenneth French, which you can explore further here.
In conclusion, while the potential to outpace the market is alluring, the challenges are manifold. A pragmatic approach for many investors might be a hybrid strategy: allocating a majority of their portfolio to passive index funds for broad market coverage and reserving a portion for individual stock picks. This balanced approach allows investors to dabble in stock picking while ensuring a diversified portfolio foundation.
The Bottom Line
Achieving consistent returns above the market average is a sophisticated challenge. It encompasses not only direct costs such as investment fees and taxes but also the subtle dynamics of investor behavior and the unpredictable interplay between skill and chance. For the majority of investors, a strategic focus on long-term investments, particularly through passive investing in cost-effective index funds, provides a practical approach to building lasting wealth.
Yet, the fundamental question remains: Can one consistently surpass the market's performance? While it's achievable, it requires a careful combination of expertise, timely judgment, and strategic financial planning. For many investors, finding this balance remains elusive, highlighting the unforgiving nature and uncertainties inherent in the stock market.