I don't want to get involved in deals and private equity etc, I just want to invest the money and have it available for me in 20 years when I retire. Since the fees have such a significant impact on the performance long-term I'm wondering why would I go the private banking route, what am I missing?
1. preferred rates in mortgage space and others. These rates can sometimes be 0.25-0.5% lower than your best open market rate.
2. borrow against your portfolio. Your pay some points over SOFR rate. This way, you don't trigger tax event or liquidate your portfolio.
3. access to structured products that have upside and downside protection. this is something you most likely don't care about.
#1 & #2 are very useful when you have decent assets. #3 is very useful when you have sizeable assets.
For <1,000,000, you would not be eligible for what is generally called "private banking". Usually, that's for 8 figures in liquid assets or more, although exceptions are made for people likely to be there in the next few years (e.g., non-liquid startup founder of a unicorn).
With a "private bank", you'll have access to financial experts well-versed in the tax consequences of your investments. They'll be familiar with trusts and other things like that. They might give you access to exclusive deals. And you'll have a phone number that you can call any time of day and speak to someone intelligent (at least compared to the person answering the phone at your local Chase branch). Most importantly, you'll have people who can make you customized deals. For example, when Jeff Bezos wants $500m to spend on a yacht but doesn't want to sell his Amazon shares, his bank will happily give him a loan on good terms for that. If you own a Picasso and want to get a cash loan with that as collateral, your banker will at least consider that request. For most people, the only asset they can borrow against is their home. But at the private banking level, they are more likely to consider other assets. They will at least humor the request to look into it.
What you are talking about is more of a "financial advisor". 1% is typical. If you are simply interested in investing in index funds and have the discipline to not touch your funds when the market has ups and downs, then you may not need/want one. But I would pose the question to a financial advisor and see what services they offer that might make the fee worthwhile for you. Typically, things like tax planning start to get more important when you cross the $10m mark and inheritance tax becomes a thing.
With regard to investing, an "all-world passive index tracker" may not be what you want. I'm not sure where you are located, but assuming it's the US, you might want to allocate a good amount of your money to US indices. The problem with all-world is that you are most likely not located "all over the world". You are most likely tied to the US and so you probably want assets closely tied to the US economy, which is where you will probably be spending most of your money.
I was never sure if that was because the clients needed more sophistication, or because there was more money in it for the bank.
Not very exciting, but as long as you have a concept of enough, 7-8 figures should be enough, and you don't need anything exciting. With high 6-figures, it may or may not be enough, but it should be enough to let you choose what you work on and how; and there's no sense gambling that away or paying 1% fees, either. Somewhere between 20/80 and 80/20 stocks/bonds should work well enough (and if it doesn't, there's likely a much bigger problem).
Index funds are probably a good idea. It wouldn't hurt to put 10-20% in something more active, blue chip, etc. Some of the online brokerages offer various levels of advice for very small fees (30ish basis points maybe). They could help you select appropriate index funds and other investments based on macro trends etc.
> have it available for me in 20 years when I retire.
Just 1 random person's superficially informed advice: I bet a US index is going to do better than all-world over the next 20 years.
It's true that you can't really beat the market before tax, but it might be worth optimizing for your own after-tax outcomes instead.