"My understanding is that stock markets are expected to generate returns above inflation in the long run because companies are paying out dividends and because investors demand compensation for accepting risk"
Your logic here is not exactly correct. As pointed out by @DStanley in the comments to your question, the dividends themselves do not increase the net value of a stock; dividends are merely one method of returning value to shareholders [the ultimate owners of the company]. See more on that here: If a stock doesn't pay dividends, then why is the stock worth anything?.
To press this point further - the 'built in' expectation for returns from equity investments, is that companies produce things that increase total economic value. That production increases value of owning those companies, because as a result of creating value, a net inflow of money is received by the company. As a shareholder, you would literally own a piece of that company, which you would receive as a dividend or future liquidation value. As long as the economy grows, value is being created on a net basis, increasing the future expected returns of all companies as a whole.
Now compare that with an asset class which does not have an inherent 'growth logic': Foreign Currencies.
If I am Canadian, and I decide to buy USD on speculation, there is absolutely no inherent logic which states that the USD will always grow stronger, and the CAD will always grow weaker. Me buying USD would be a gamble on the relative value of the two currencies. Compare this to buying a global stock portfolio, where I would be relying partially on the underlying premise that the global economy will continue to strengthen. Even if the US economy strengthens faster than the Canadian economy, my Canadian corporate investments will still increase in value overall, if the Canadian economy strengthens.
You can again compare the difference between owning an empty plot of land, or owning a storefront that charges rent. An empty plot of land does not inherently increase in value; it increases in value on the basis of the ebbs and flows of the local property market. Comparatively, if you own a storefront that charges a higher amount of rent than your carrying costs [mortgage, property tax, etc.], then you have a built in assumption that as long as it remains valuable as a storefront, your land will continue to return value. The catch is that there is often so much speculation in real estate, that the potential value of the land in the future outweighs the rent earned in the interim.
And you can't simply rely on the expectation that investors are getting the return that they 'should be', to compensate for risk. For example, look at the speculative growth that hit the cryptocurrency market particularly over the last year. There is, in my opinion, literally 0 inherent value in any current cryptocurrency [except as a tool to evade taxes and trade illicit items], and yet the volatility of the market implies a massive amount of risk. Just because the risk exists, however, would not convince me that the 'underlying value' exists.
