Why index investing is so hard to beat  

A traditional index fund invests in every company in a share market, with larger companies receiving a greater allocation. This means investors earn the market’s return, less a small fee. Importantly, because an index fund owns the entire market, it also owns the relatively small number of shares that generate most of the returns. 

The case for index investing is often supported by research such as S&P’s SPIVA reports, which consistently show that most actively managed funds underperform the market over the long term. This evidence is compelling – it tells us what happens, rather than explaining why. 

In this blog, I want to explore what actually drives index returns, why a relatively small number of companies account for so much of the market’s wealth creation, and how volatility and investor behaviour affect long-term outcomes. Understanding these factors helps explain why index investing has been so difficult to beat. 

The shares you don’t own matter more than the ones you pick 

Hendrik Bessembinder, a finance professor at Arizona State University studied more than 64,000 shares across 43 markets between 1990 and 2020. They measured how much wealth these companies created for shareholders above what investors could have earned by simply holding cash. 

His findings are remarkable. Just 2% of companies accounted for all US$75.7 trillion in net wealth created. The remaining 98% collectively generated no more wealth than cash. In fact, 55% of US shares and 57% of shares outside the US underperformed cash over their lifetimes. 

Australia was no different. Of the almost 3,000 Australian shares included in the study, the median share lost 43% of its value over its listed life, and only 36% outperformed cash. Remarkably, just eight companies, representing less than 1% of all Australian shares, accounted for almost 48% of the wealth created. These figures are measured in US dollars and cover each share’s listed lifetime, which was typically around eight and a half years, rather than the full 30-year study period. 

We can see the same concentration of returns over much shorter periods. For example, in the 2024 financial year, Commonwealth Bank, Westpac, NAB, Goodman Group and Wesfarmers represented just 19% of the ASX 300 index at the beginning of the year, yet contributed around 8 percentage points of the index’s 11.9% return (i.e., contributed two-thirds of the total return). The remaining 295 companies, representing 81% of the index, returned less than 4% collectively. Commonwealth Bank alone accounted for approximately 21% of the index’s total return that year, despite representing less than 8% of the index at the beginning of the period. 

But could an investor have identified these five companies in advance? The evidence suggests it would have been very difficult. Between the 2016 and 2023 financial years, these five shares delivered roughly the same return as the other 295 companies, sometimes outperforming and sometimes underperforming. All their excess returns came after 30 June 2023. In other words, they only became obvious winners with the benefit of hindsight. 

We see a similar pattern in the US too. According to J.P. Morgan, the so-called Magnificent 7 (Alphabet, Amazon, Apple, Meta, Microsoft, NVIDIA and Tesla) accounted for 63% of the S&P 500’s share price gains in calendar year 2023, 55% in 2024, and 46% in 2025 (but only 1% in the first half of 2026).  

However, identifying tomorrow’s winners is much harder than recognising yesterday’s. Research by Dimensional, covering US shares from 1927 to 2025 (98 years), found that while companies typically performed strongly before becoming one of the 10 largest in the market, on average, they did not continue to outperform after reaching that position – that data isn’t a good sign for current market darling, NVIDIA.  

This helps explain John Bogle’s famous advice: rather than trying to find the needle in the haystack, simply buy the whole haystack. 

Bessembinder reached a similar conclusion. Unless an investor has a genuine ability to identify the relatively small number of companies that will generate most of the market’s wealth (unlikely), broad diversification through index investing makes a lot of sense.  

Falls of 10% or more happen in most years  

An intra-year decline measures the largest fall in the share market from its highest to its lowest point during a calendar year, regardless of where it finishes. J.P. Morgan has tracked this for the ASX 200 since 1994. Over those 32 years, the market experienced an average intra-year decline of 14% yet still finished the calendar year higher in 23 of those years, or 72% of the time. These returns exclude dividends, which would improve the overall result. 

In fact, more than half of those 32 years experienced a decline of at least 10%. The important point is that a fall of 10% or more is a normal part of investing in shares, not necessarily a sign that something has gone wrong, so investors should just sit tight.  

The US share market tells a similar story. According to Charlie Bilello’s research, the S&P 500 has experienced an average intra-year decline of approximately 16% since 1928, despite generating an average annual return of around 10%. In 2025, for example, the index fell almost 19% by April, before recovering to finish the year with a positive total return of almost 18%. 

The longer-term Australian evidence is equally compelling. Since 1900, the Australian share market, measured using the All Ordinaries Accumulation Index, has delivered positive returns in approximately 80% of calendar years. 

Of course, this doesn’t mean every market decline will be followed by a recovery before the year ends. Some downturns last much longer. However, it does demonstrate that significant share market declines are entirely consistent with strong long-term investment returns. 

Missing the market’s best six months can cost you almost a third of your wealth  

Some of the strongest share market returns occur over surprisingly short periods, often immediately after a significant decline. Missing these periods can have a substantial impact on long-term wealth. 

Dimensional analysed this using the ASX 300 index. An investment of $1,000 at the beginning of 2001 would have grown to $7,583 by the end of 2025. However, the outcome would have been very different if an investor had missed just one of the market’s strongest periods: 

  • Best week: $6,653 – 12% lower (week ending 30 March 2020). 
  • Best month: $6,331 – 17% lower (month ending 6 April 2009). 
  • Best three months: $5,753 – 24% lower (period ending 23 June 2020). 
  • Best six months: $5,179 – 32% lower (period ending 1 September 2009). 

In other words, missing the market’s best six months, representing just 2% of the 25-year investment period, would have reduced the investor’s final balance by approximately 32%. 

Importantly, all these strong periods occurred immediately after significant market declines, either following the global financial crisis in 2009 or the COVID crash in 2020. 

This highlights one of the biggest risks of trying to time the market. Investors who sell during periods of uncertainty, intending to reinvest once conditions improve, risk missing the very recovery that drives a substantial proportion of their long-term returns. 

Of course, Dimensional’s calculations are hypothetical and assume the investor held cash while out of the market. Nevertheless, they demonstrate why remaining invested through periods of volatility is so important. 

US research tells a similar story when we look at individual trading days. According to J.P. Morgan, US$10,000 invested in the S&P 500 at the beginning of 2006 would have grown to US$80,619 by the end of 2025, representing an annualised return of 11%. However, missing just the 10 best trading days over those 20 years would have reduced the final balance to US$35,866, equivalent to just 6.6% p.a. 

Remarkably, 6 of those 10 best days occurred within two weeks of the market’s 10 worst days. This highlights how closely the market’s strongest recoveries can follow its sharpest declines. 

Australian research produces similar results. AMP’s Shane Oliver found that the All Ordinaries Accumulation Index returned 9.6% p.a. between 1995 and May 2025. Missing the 10 best days would have reduced that return to 7.6% p.a. Conversely, avoiding the 10 worst days would have increased it to 12.2% p.a. 

Of course, avoiding the worst days would be just as beneficial as capturing the best ones. The problem is that nobody can reliably predict when either will occur. Because the best and worst trading days tend to cluster together and investors who sell to avoid further losses risk missing some of the strongest gains when the market recovers. 

This is why attempting to time the market can be so costly. It’s not enough to correctly predict when to sell. You also need to know when to reinvest, often when uncertainty is at its greatest. 

A falling market is a poor reason to stop investing  

The same principle applies to making regular investments. When share markets fall, each dollar invested buys more shares. And as the research above demonstrates, some of the strongest market recoveries occur shortly after significant declines. 

This means investors who continue making regular contributions during market downturns can benefit from buying shares at lower prices and participating in the subsequent recovery. Conversely, those who stop investing when markets fall risk missing these opportunities. 

There is some evidence to support this. Morningstar found that over the 10 years to 2025, the average dollar invested in US managed funds earned 8.7% p.a., compared with the funds’ actual returns of 9.9% p.a. Morningstar attributes this 1.2 percentage point difference to the timing of investors’ contributions and withdrawals. 

The practical lesson is: If you are investing regularly and your financial circumstances have not changed, a market downturn is generally not a good reason to stop. In fact, continuing to invest when share prices are lower can improve your long-term outcome. 

This is not an argument that investing regularly is necessarily better than investing a lump sum immediately – in fact, the reverse as discussed here. Rather, it’s a reminder that market volatility should not dictate when you make your regular investments. 

Where traditional index investing is less compelling  

Active management can deliver somewhat better results in two areas of the share market, although most funds have still underperformed over the long term. 

The first is small companies. According to S&P’s 2025 SPIVA scorecard, 52% of actively managed Australian mid and small-cap funds underperformed their benchmark over three years, 60% over five years, 80% over 10 years and 60% over 15 years. These results are comparatively better than other equity categories, but even the fund ranked in the top 25% over 10 years outperformed by just 0.12% p.a. 

One possible explanation is that smaller companies receive less analyst coverage, creating more opportunities for active managers to identify undervalued shares.  

The second is emerging markets. Morningstar’s US Active/Passive Barometer found that the proportion of active emerging market funds that survived and outperformed comparable passive funds increased from 22% in 2024 to 64% in 2025 and 70% over the 12 months to June 2026. 

Because traditional market-cap-weighted indexing has a somewhat weaker track record in small companies and emerging markets, we tend to favour factor-based index investing in these asset classes, using funds that incorporate quality or value filters. 

The distinction is important. In developed markets, it’s often what you don’t own that hurts your returns, because a small number of companies generate most of the gains. However, in small companies and emerging markets, it’s often what you do own that hurts your returns. That’s why we prefer a more selective, rules-based approach that seeks to avoid lower-quality or overvalued companies. 

So why do most active funds (and individual stock investors) fall short? 

In developed share markets, it’s often the shares an active fund or individual investor does not own that hurt its performance most. 

The first reason is market concentration. A relatively small number of shares typically generate most of the market’s returns. Active managers need to own these winners, and hold them during their strongest periods, simply to keep up.  

The second reason is market timing and asset allocation. Managers who hold too much cash or are underweight shares during a recovery risk missing the strongest returns, which often follow significant declines. 

The third reason is cost. William Sharpe demonstrated in 1991 that, before costs, the average actively managed dollar must earn the same return as the average passively managed dollar, because collectively they make up the market. After fees and trading costs, the average active dollar must underperform.  

The evidence reflects these challenges. According to S&P’s 2025 SPIVA Australia scorecard, 87.6% of Australian equity funds underperformed the S&P/ASX 200 over 10 years and 86.9% over 15 years. For global equity funds, the figures were 95.2% and 96.0%, respectively. 

Leave a Comment