Most U.S. mutual funds include terms that restrict their sale to U.S. residents only. This has always been the case, but hasn’t always been enforced. I believe that not abiding by those terms could potentially get the broker into U.S. legal trouble.
Interestingly, most U.S. ETFs have no such restrictive terms, but as you said, U.S. brokers often have concerns about running afoul of the local laws in your country of residence if you are not U.S. based.
Keeping a U.S. address on your U.S. brokerage account, and not informing your broker about your non-U.S. country of residence is generally the best way to avoid these issues, provided you are U.S. person.
I've done a little more research on this.
The general consensus is that there's nothing in U.S. law prohibiting a non-U.S. person from investing in a U.S. mutual fund. And many foreigners have successfully done it, notwithstanding the tax consequences. However, in practice, many brokers' compliance departments are frequently reluctant to facilitate these particular investments because they're concerned about running afoul of possibly foreign (non-U.S.) laws. Thus it's hit or miss whether you can find a broker that'll help out.
There are still some scattered reports that even U.S. citizens living outside the United States face some difficulties. Same thing: it's hit or miss.