After 25 years on Wall Street and another 10 years as a fee-only fiduciary RIA, Gil Baumgarten knows all the brokerage tactics that make your portfolio inefficient and put you at a disadvantage. He also understands the common self-destructive tendencies that make every investor vulnerable to brokerage firm schemes.

His new book, FOOLISH pulls no punches and Wall Street brokerage firms won’t be happy about it. That’s because brokerage firms have built their business on profiting in the shadows and they surely don’t want the lights to come on.

In the book, you’ll discover the staggering differences between brokerage and advisory systems and walk away with actionable advice to help you stay on guard. Most importantly, you’ll take an introspective look at your investing style and learn how to walk away from the foolish routes investors often take.

Drew Appelbaum: Hey listeners. My name is Drew Appelbaum and I’m excited to be here today with Gil Baumgarten, author of Foolish: How Investors Get Worked Up and Worked Over by the System. Gil, thank you for joining, welcome to The Author Hour Podcast.

Gil Baumgarten: Thank you, glad to be here.

Drew Appelbaum: Let’s kick this off. Can you give us a rundown of your professional background?

Gil Baumgarten: I’ve been in the investment management business for 36 years, the first 25 of those were with major brokerage firms such as Smith Barney and UBS, and the last 11 years have been at my own fiduciary registered investment advisory firm called Segment Wealth Management.

Drew Appelbaum: Why was now the time to share the stories in the book? Was there something inspiring to you? Did you have an “Aha!” moment? Is it something as simple as, you had a little bit more time on your hands because of COVID?

Gil Baumgarten: No, it’s just the culmination of about 20 years worth of “Aha!” moments, and taking notes about the way things work, and the way things should work, and the way things don’t work. The investment industry, when delivered by brokerage firms, probably doesn’t do the best job for clients and maybe not what clients perceive about it.

I wanted to shine a bright light on how those machinations actually work and what problems that creates for investors. And secondarily, investors create a lot of problems for themselves with fear and greed and second thoughts, and all the demons that they have from the issues that they’ve got and the relationship that they have with their money.

I really wanted to write a book about how those two issues conspire to destroy investment performance. Some, the fault of the industry, and some the fault of investors for not having a proper perspective about their money.

Drew Appelbaum: When you decided that you were going to write a book, a lot of authors will have the idea of the book rattling around in their head, they might even have an outline of the book but then, during the writing process and digging deeper into some of the subjects, you’ll come through some major breakthroughs and learnings. Did you have any of these major breakthroughs or learnings on your writing journey?

Gil Baumgarten: That’s exactly what happened for me. I at first thought I would start out and write my version of how investors should successfully invest, and the more I thought about it, the more the issues seem to be with the industry and how it capitalizes on the lack of knowledge of the typical investor, and the behavioral issues that people tend to display over and over and over again, most of which are self-harm tendencies.

Rather than just be a tell-all about, “I think you should do this and I don’t think you should do that,” I really wanted to focus on those two areas, which seemed to be the most prevalent problems that a typical investor encounters.

Drew Appelbaum: Now, while you were writing the book, in your mind, who are you writing it for? Is this a book for millionaires? Is this a book for people with current modest investments? Is this a book for GameStop shareholders?

Gil Baumgarten: You know, the GameStop shareholder’s probably not going to find anything of value in my book. The moderate investor probably won’t have enough experience with the issues to really have the head-slap moment that would lead them to believe that the book is profound. The 10-million-dollar investor or hundred-million-dollar investor who has been banging their head against the wall and listening to stories and pitches from hedge funds and mutual fund guys and wealth managers and everybody who is trying to deliver services to those guys, they’re going to have the head-slap moment because they will surely have encountered many of the issues that we bring up in FOOLISH.

Drew Appelbaum: Now, I want to dive into the book a little bit, and you kind of talk about this right at the beginning of the book. You say that you’ve been out of the game for about 10 years at a brokerage firm and so, can you confirm that all the practices that you talk about and that you outline the book, are still in place today?

Gil Baumgarten: You know, I can’t confirm for sure because so much of this is cloak and dagger. My brokerage buddies tell me that a lot of it is still in play and that some of the egregious behavior has been attempted to be legislated out of existence and unsuccessfully so because the regulation best interest is really not the best interest standard at all.

I think the vast majority of the things that I’ve discussed in the book do still go on and I haven’t been proven otherwise.

Regulation Best Interest

Drew Appelbaum: Yeah, I wanted to dig one layer deeper into what you just talked about. In 2020, there was this regulation best interest brought forward by the SCC but did it actually change anything?

Gil Baumgarten: I think it’s just going to make things more confusing because now, you’re going to have brokerages that would have an incentive to proudly prance around the fact that they have this best interest, which is going to falsely gain trust from investors, when in fact at the core of best interest, you’re right back to suitability and disclosure of the lack of fiduciary responsibility. I think it’s only going to give the brokerages more ammunition to mislead clients about where their allegiances actually lie.

Drew Appelbaum: Now, you mentioned in the book that the house always wins, just like at a casino. Can you explain, because you go to really shocking depth in the book about how some of these firms make their money? Can you talk about how you could equate a casino to an investment firm and how that house will always win?

Gil Baumgarten: Well, in a casino, every game that you play has a certain amount of statistical odds associated with it. Every one of the games has the odds that lean towards the favor of the house. You can, in fact, go into a casino and put a quarter into a slot machine and make a million dollars. The odds are just really, really low.

The casino is betting on the fact that a million other people like you are going to put those quarters in there and they’re going to pull the lever hundreds of times a piece and over enough pulling of that one-armed bandit, they can even pay out a million dollars and still make money. And that’s the way the odds work inside the brokerage firm too.

They have all the rules of the game set up for their brokers so that no matter what the clients do, a certain amount of money is going to accrue to the house and, after they pay the broker, the firm is going to make out. Now, will the client make out? They don’t know and frankly, I don’t think they much care because the standard of care at a brokerage is still suitability and the rules that are governing the brokerage industry are laid down by FINRA, which is a self-governing body.

FINRA is the Financial Industry Regulatory Authority. FINRA is regulated by the SCC but the SCC gives FINRA wide latitude to allow its member firms to take advantage of clients and the interest of the clients do not have to come first.

The brokerage firm has to see rules of engagement set-up with their brokers and the relationships that they have with their clients so that they can always hide behind how things turned out. “Hey, if what we recommended was not suitable, you’re on your own.” That’s not totally objectionable, there has to come a point at which if you’re doing a transaction with the brokerage firm, you can’t return the stock that you bought yesterday, and get your money back. Now, you can resell it in the marketplace.

There is only a certain amount of wiggle room that a brokerage firm should allow a client to have, and I don’t take issue with the fact that when a client says, “Yes, buy me a thousand shares of ABC,” you’re going to own it and whatever happens to it, good, bad or otherwise, you’re going to own that as the investor.

What I have a problem with is when a brokerage tells his client, “I’m looking out for you. You should be doing this for this reason.” The underlying reason why the advisor is doing that is because he’s trying to get a commission or a fee, or somehow grease the skids of the brokerage firm based on the rules that they have laid down that are within the FINRA rules of engagement with the client. And to say that he’s going to bat for the client is almost always a misnomer.

Drew Appelbaum: You talk in the book about how difficult the financial ecosystem is to navigate and now we just heard that sometimes even a financial advisor isn’t an end-all solution. What are some questions you can ask your advisor before signing on the dotted line to make sure that you’re avoiding some of these pitfalls?

Gil Baumgarten: Well, I think the bigger question might be, whether you want to hire a broker to be your advisor. Any kind of a commissioned salesman is going to have allegiances to the way their system is operating with regard to how they’re going to get paid, and it puts the broker in a position that every single prospect that he meets, the first thing that has to go through his mind is, how am I going to get paid on this?

The ecosystem is built so that the broker is going to align that client with particular strategies for which he’s going to get paid in particular ways, and that’s not necessarily problematic but underlying that, the brokerage firm is going to get paid in more ways than that. There are so many layers to the onion about the way the brokerages get paid to swap trades on our mutual fund for order flow, we can swap your managed account for some back-office support of this or that.

The brokerage firm will swap stock research for stock trading revenue for the mutual fund. You have all of these products and these layers that have cost after cost after cost. Most of which is opaque, and the client cannot see it, and many clients believe that if they can’t see a fee that it surely can’t exist.

Let me tell you how that works, a mutual fund takes its fee from the client, every single day, and they post the value of the mutual fund online or on the client’s statement after their fee has already been withdrawn. A client assumes that as a bank account unless there’s a $300 entry on my statement, a fee cannot exist. Well, wait a minute, before we calculate the value of your $11 unit, we took our nickel a share, our fee out the night before and when we put that on your statement, it’s already deleted and therefore the client perceives that they’re not paying a fee for that, when in fact, this is embedded in the cost of the unit and that is never even put on the statement.

Misplaced Trust

Drew Appelbaum: Where’s the major problem? Is it the market in general, is it the brokers, the advisors, or is it just the entire system?

Gil Baumgarten: Well, the first half of the problem is the ecosystem at the brokerages and suitability as a standard of care for clients, if in fact what’s being discussed with the client is something more than suitability.

I don’t have any problem with brokerages getting paid, I don’t have any problem with brokerages getting paid well. I don’t have any problem with brokerages getting paid on multiple layers inside of a client’s account. You just have to make sure that the client understands it. When you look at the typical relationship between a broker advisor and his client, I can assure you the client takes for granted that the expenses are far lower than what is in fact true, and when you hire a broker to be your advocate, you are probably misplacing your trust in a system that is not designed to hold the client’s interest out first.

The interest of the brokerage firm is held out first and the client complies with that through the regulatory environment laid down by FINRA in which suitability is the standard of care, and the brokerages will be quick to throw up their shield of defense and say as long as that vehicle was suitable for you, you own the results. Again, not a problem as long as the client knows what they are getting into.

Drew Appelbaum: I want to go again one more layer deeper because this was eye-opening in the book. What are the typical fees that large brokerage firms are taking from their clients and things that you wouldn’t normally see or think of as an average investor?

Gil Baumgarten: Well, there is no way to say what ‘typical’ is because the level of client activity, the types of products that they buy, the frequency with which they turn them over varies a lot. So, there is no standard. It wouldn’t be unusual for a client with lots of activity who buys highly expensive products to pay more than 4% per year of the value of their accounts, and if you look at it from a high commission, high velocity, lots of activity, client’s total cost could exceed 4% per year.

Drew Appelbaum: Let’s move over to the individual investor. When you really break it down, how are people, and why are they making the investment decisions that they’re making?

Gil Baumgarten: Where most often decisions on the part of an investor are made from one of two decision or one of two priorities. One, they’re either being greedy and they want some of the action that everybody else seems to be getting. Or two, they are fearful and they want to hide when things get tough, so fear and greed are the two things that drive the emotions and ultimately, drive the decisions of most investors. The problem is that, as Warren Buffett said, “People get fearful when they should be greedy and they get greedy when they should be fearful.”

That’s exactly what happens right now, you’re seeing a lot of velocity in things like what you mentioned with GameStop. People have no idea what kind of risk is involved in trading a one dollar stock, and getting it by one way or another to trade it two or three hundred dollars per share. Somebody is going to get hurt, I just don’t know who that’s going to be, but it gets the juices flowing that I want some of the action. I want something to brag about to my buddies and it creates all kinds of problems.

Fear and greed are what drive decisions, but in the wrong way. Fear and greed should be driving decisions but only inverse of the way the typical person does it.

Drew Appelbaum: Now, is it actually okay? Can it work out if you make some of these decisions based on emotions?

Gil Baumgarten: It normally does not work out well because of how we are wired as humans. There are many reasons, but some of it is that we want to turn everything into a game. We see patterns that don’t exist. You could ask people if they believe that one day of the week was a better day of the week to invest than any other day. That disproportionate number of investors would say yes that surely somebody has developed some kind of a system to identify that on 2 PM on Tuesday afternoons, if you did this or if you did that, it was more profitable.

Of course, you’re going to have hedge funds that they are going to say that they’ve built an algorithm that can predict what those are going to be like, and they can exploit that for your advantage, but the hedge fund industry has produced overall some of the worst results for investors primarily because the fees and expenses are high, and the taxes are high, and the liquidity is also not good. It’s just a minefield of people out there trying to provide services and in the end, they tend to not deliver the best results for investors, versus a buy and hold or an index strategy or just a low-cost low-tax, low-friction methodology tends to compound much better numbers over longer time periods than hiring some crackpot who’s got some kind of a strategy that he wants to deploy to your theoretical advantage.

Drew Appelbaum: Do you have any tips for the listeners out there for better investment results?

Gil Baumgarten: Well, combating fear and greed would be number one. Having a contrarian methodology would be number two. Try to do what everybody else is not doing, thinking long term would be another, not thinking about what you can get out of this today or tomorrow, thinking about the tax effect. One, do I understand the tax code and what it allows, what type of activity is advantageous or disadvantageous within the tax code? Understanding that is a must. What kind of fees and expenses do I pay for deploying various strategies?

You can clearly understand what the odds of success are if you can take the cost and the tax drag from traditional trading methodologies and turn that into an investment policy, personally, that would drive longer hold periods with much more consistent results, and ultimately compound to much better numbers over time by reducing fees and taxes and letting the marketplace do it’s action, as opposed to us getting involved in trading methodologies that are normally unsuccessful.

What Is Diversification?

Drew Appelbaum: I think most investors understand that diversification is safety and so when someone is constructing their portfolio, what are some of the basic building blocks that folks should follow?

Gil Baumgarten: Well, I would take issue with your statement that diversification is safety. Diversification is safety if a person owns a broad spectrum of asset classes, but one of the problems with the perception that diversification equals safety is that if I own 15 tech stocks that I’m somehow diversified or even if I own 15 mutual funds, I am somehow diversified, because people tend to have a bias in the way they select those things and they tend to stack their bets in ways that they don’t understand. Then when the tech stocks all fall apart, you find out that you have a 45% loss and, “Oh my, I thought I was diversified!”

Diversification if you’re on a broad spectrum of asset classes does in fact equate to more safety but you know, your safety and my safety might be two different things, and so I don’t believe that just simply owning more positions necessarily makes you more diversified.

Drew Appelbaum: Now, in your mind, what was the end goal of the book for readers? What do you want them to walk away really thinking about, or some action items you’ll hope they’ll take?

Gil Baumgarten: I just wanted to challenge the traditional way of doing things. The traditional way of hiring investment advice is to go with a stock brokerage because there are brokers everywhere from all firms. Their market penetration, so to speak, is very high and it is most likely that if you’re going to deploy investment strategy, you’re going to be working with a brokerage firm and, heck, even I work with a brokerage firm!

We custody up the vast majority of our client money either at Schwab or at Vanguard, so really all we’re looking for is a cheap and reliable place to park our client money because I am not in the brokerage business, but I, unfortunately, need to have a brokerage firm attached to my clients so that I can execute class strategy. So, that’s kind of how it works.

Drew Appelbaum: Are there any resources you would suggest to readers other than the book? Where does Gil go after we finish this conversation to go get investment advice or news?

Gil Baumgarten: Well, you know being in the institutional side of the business, we have people providing us with information all the time but the vast majority of information is really just noise. You have a lot of people out there clamoring to give you investment advice, to give you their opinions, people stay glued to their screen, they come home every afternoon and they watch CNN and CNBC and MSN and all of these news sources.

The vast majority of everything that they talk about on those shows is non-action-oriented news. It is just hyperbole and it is for the most part just noise. The stock market when properly deployed owning large companies and in broad diversification will generate whatever rate of return it’s going to be, which is positive in 81% of all time periods, it’s a very successful and compelling bet if you do it correctly. Most investment advice generally wants to sell the client on the concept of getting something more than a market return for which they’re going to charge vastly higher fees than the clients would have paid if they had just accepted the market return.

The vast majority, over 90% of all investment programs, mutual funds, or whatever, underperform their benchmark. The friction from the cost and the taxes causes a substandard rate of return to accrue for the majority of those packaged products and clients are just unaware that they could have made 17% if they’d owned an index fund. They got 14% because they had some stock mutual fund. Well, their investment advisor dude is going to say, “Hey look, you made 14%, aren’t we smart?”

Well, the reality is, you could have done better if you had just done it a different way and that’s really the information that I am trying to get out is that the industry isn’t necessarily looking out for the client, and the clients need to look out for themselves. On top of that, if you are looking out for yourself, be careful because you have self-harm tendencies that you’re probably unaware of.

Drew Appelbaum: Gil, you know we just touched on the surface of the book here but I want to say that writing a book is really going to help so many people open their eyes to the reality of the financial world is no small feat, so congrats on having your book published.

Gil Baumgarten: Thank you.

Drew Appelbaum: Now, I do have one question left and it is the hot seat question. If readers could take away only one single thing from the book, what would you want it to be?

Gil Baumgarten: The only person that’s going to look out for the investor is the investor themselves, and they need to arm themselves with enough knowledge of how the system works, what they pay for things, how the tax system works, and what those friction costs are in order to develop an investment policy for themselves.

In the absence of the desire or willingness to do that, they are probably better off hiring a fiduciary advisor who would be registered as a registered investment advisor not regulated by FINRA, not going to earn a commission, not able to earn a commission, who can never trade against the client and only has a fiduciary duty to every single client because that’s what true best interest looks like.

If you’re going to end up hiring an advisor, which I recommend that people do, you’re probably going to be better off with a fully fiduciary advisor than you’re going to be by operating your business in the brokerage ecosystem.

Drew Appelbaum: Gil, this has been a pleasure and I am really excited for people to check out this book. Everyone, the book is called Foolish and you can find it on Amazon. Gil, besides checking out the book, where can people connect with you?

Gil Baumgarten: They can sign up for my blog, my website is segmentwm.com, so www.segmentwm.com/blog. I write a blog on a bi-monthly basis and people can sign up there and get updates on our writings about these. I know the topic is similar to investor advocacy and what that looks like.

Drew Appelbaum: Gil, thank you so much for coming on the show today, and best of luck with your new book.

Gil Baumgarten: Thanks so much. I appreciate it.