Thinking about planning for retirement makes most people’s heads spin. We don’t even want to think about it now, much less at all, but financial planner James Miller has some really powerful motivation for you, a way to get basically free money. In his new book, Divorce the IRS: How to Defuse Your Biggest Tax Time Bombs Before You Retire, he explains, in plain language, all the different taxes that are waiting for us at retirement and how they work.

Then he tells us what we can do today to avoid paying some of them and save thousands and thousands of dollars as a result. On Author Hour today, he shares some of his biggest tips and how to take advantage of them.

Jane Stogdill: Hi Author Hour listeners, I’m here today with James Miller, author of Divorce the IRS: How to Defuse Your Biggest Tax Time Bombs Before You Retire. James thank you so much for being with me today.

James Miller: Thanks for having me.

Jane Stogdill: Okay, first of all, will you tell us a little bit about your line of work and how this became your specialty?

James Miller: Sure, I’m in financial services, I’m a financial advisor and I’ve been a financial advisor for about 20 years now. As I and my practice matured over the years, I really realized what a big role taxes play in creating and retaining real wealth for people who are trying to create a good financial plan and reach all their financial goals in life.

A lot of people pay way too much in taxes over the course of their lifetime and it became a real passion of mine to help people pay less to the IRS, which really improves their financial plan.

Jane Stogdill: Now, I feel like when people think about their retirement, they’re not always thinking about the IRS, is that part of the problem?

James Miller: That is part of the problem. We pay taxes for a lot of things in life, not just to the IRS. You pay taxes whenever you pump gas in your car, or you pay for a license for your profession. There are lots of ways we pay taxes but the one that this book focuses on and the one I help people with the most is the tax bill they pay to the IRS.

Three Tax Buckets

Jane Stogdill: I was really interested in this book, not least because it is not the advice that I had previously received and not everyone understands this. Before we get into all of it, tell us first about the concept of the three tax buckets?

James Miller: Sure. One thing I tell people is that I’m not your mother and father’s financial advisor. I give modern financial advice. Things over the years always change and money is changing just like anything else–in the way we think about money. This book does, to your point, give a lot of advice that people haven’t heard before. It’s a new way of thinking about money. It’s not the old way of thinking about money.

Money investments specifically, always fall into one of the three tax buckets. The tax me now, tax me later, or tax me never bucket. Most people are familiar…

Jane Stogdill: We’re talking specifically about retirement plans, sorry, I should have said.

James Miller: Retirement plans or honestly any money that you have that you save or invest is going to fall into one of those three buckets.

The most common one that everybody has is the tax me now bucket. If you have a checking account or a savings account, that’s a tax me now bucket. The interest you earn in your savings account at the bank, you get a 1099 for it at the end of the year and you have to pay taxes on any interest.

When I say tax me now, I mean, tax me each year.

The tax me later or tax me never buckets, now we’re moving into retirement specific accounts. Americans have a choice about how they get taxed on their retirement accounts and it boils down to those two buckets. Tax me later or tax me never.

The most popular bucket, the one most people are familiar with, is the tax me later bucket. That’s the traditional 401(k) and the IRA. Those are the two most common. There are many others that fit in there–457 plans, simple IRA’s, things like that.

The traditional IRA and the 401(k) are the two most common. When I say tax me later, what I really mean is if we put money into that bucket, we don’t have to pay any taxes on what we put into that bucket now, but the IRS is going to tax us later on that bucket. Any money that we take out of that bucket later in life or pass on to a beneficiary someday is going to get taxed. It’s going to be taxed at a later point in time.

The final bucket, the tax me never bucket is a newer bucket and those are generally accounts that have the word Roth around it. A Roth IRA, a Roth 401(k), those you don’t get the tax break up front, but they grow tax-free, and you never pay taxes down the road on that money or its growth. That’s the tax me never bucket.

Jane Stogdill: Because the money is going in after you’ve already paid taxes on it?

James Miller: Correct and then growing tax-free.

Jane Stogdill: Wow, even the interest you earn does not get taxed?

James Miller: Correct.

Jane Stogdill: Wow, these are relatively new, these kinds of Roth accounts, is that correct?

James Miller: Roth has been around for about 20 years now, but most employers have only started adding the Roth 401(k) option in the last 10 years. They are fairly new to a lot of people but about 85% of the 401(k) plans in America now have a Roth 401(k) option that people can choose.

A lot of people don’t choose because they were already on a set it and forget it plan with the traditional 401(k), maybe they didn’t understand all the benefits of the Roth 401(k). There are lots of reasons that people haven’t switched over to the Roth.

A lot of people just don’t understand how fully the Roth 401(k) works or why it would benefit them to switch to it.

Borrowing Money from the IRS

Jane Stogdill: Now, you talk about how tax deductions and tax-deferred accounts are maybe not in our best interest, at least not as our only retirement savings strategy. What is this concept of borrowing money from the IRS that you discuss?

James Miller: This is a key concept in the book. What I find is most people are happy to put the pain of paying taxes off to later. They just don’t understand the compounding negative effect that that has. When you put money into an IRA or your traditional 401(k), almost anybody will tell you, “I love doing that because I don’t have to pay tax on the money.” That’s not really accurate.

Really, it’s more accurate to say, “I don’t have to pay tax on that money this year.” You still have to pay tax on that money. You’re going to pay tax on it later. Plus, you’re going to pay tax on the growth. Really, what you’re doing, when you put money into a tax-deferred bucket like your 401(k) or an IRA, is you’re taking a small loan from the IRS.

Now, a simple example to explain that is, if you put a hundred dollars into your 401(k), you don’t pay any taxes on it. Let’s just say you’re in the 20% tax bracket, that saves you today, that 20% or $20 in this example.

Your take-home pay goes up by $20. Now, people think that’s great. I got extra 20 bucks in my pocket, but really all you did was borrow $20 from the IRS because you still owe that $20 to the IRS. You’ve just put it off to a future date.

Now, the problem with that little loan is, like all loans, there is a date they come due and there’s usually interest. If that hundred dollars has grown by the time you take it out someday, just to say $200 to get the example very simple, now you owe taxes on the $200.

If you’re still in that same 20% tax bracket, now you owe $40 back in taxes when you take it out. That’s a $20 loan that you originally got, you now have to pay back $40 in retirement. Putting money into a tax-deferred account really is just borrowing from the IRS today so you can pay it back and then some down the road.

Most people have never made that connection, they just think they’re putting their taxes off till later and that’s a good deal and for most people, it’s not. The other reason that it’s not is because it’s not just the tax that you’re going to pay. When you take that money out later in life, the IRS has set up a system of triggers that will create other taxes for you as well, like taxes on your social security, extra taxes on your Medicare payments, and things like that.

Jane Stogdill: Okay, those are some of the bombs that I want to get into in a second. First of all, whenever I hear about opportunities that seem so great like this. My question is why do Roth’s exist, why would the IRS let accounts like this exist and not charge us tax on the interest and the gains?

James Miller: They have limits on the amount of money you can put into a Roth account because the tax benefits are so great. A lot of people have different ideas on why this was set up the way it was set up. Some people think that the Roth accounts are really set up as a way for the wealthy to get wealthier. The truth is, most Americans don’t spend a lot of time becoming financially literate–understanding how money works. I mean, that’s the reason I have a job as a financial adviser is because it’s just not interesting to some people.

If you’re a doctor, an attorney, you’re busy being a doctor and attorney, not learning all the different IRS codes and how to save your money the most tax-efficient way possible. It is a great way to save for the future and I think also, the government did it to encourage people to save more for the future, so they said, “Hey, we’ll even give you tax-free accounts if you save for the future.” I think the government did it with good intention to try to encourage more people to save for their future in a more tax-efficient way.

Jane Stogdill: Since you’ve talked a little bit about your work as a financial planner. What’s the difference between a tax planner and a tax preparer?

James Miller: Great question. Most tax services are tax preparers. In my book, I talk about the difference between a tax planner and a tax preparer. A tax preparer is somebody who is going to fill out and file all your tax forms for you. This is what most Americans are used to dealing with.

Even in our block, liberty tax, all the big national tax places, what they want to do is get you back the most money on your taxes in any one year, and I understand why they want to do that. I understand also why people want that. I’ve never met anybody who goes to see the tax person when the tax person says, “Hey, I can get you an extra $2,000 of return this year.” Nobody says, “I don’t want that, I’d love to give that extra money to the government.”

People are happy about that and that’s why tax preparation services try to get you the most money back. They want to make their clients happy. You might go back to them year after year and refer people. “Hey, my tax guy got me an extra $2,000 back, you should go use his service.”

What most people don’t realize is that thinking about their financial plan on a year-to-year basis might not be what’s best for them in the long run. They’re not taking their lifetime tax plan into account. Sometimes it’s not best to get the most taxes back in any one year.

The point of tax-deferred or tax-free savings isn’t to get the most back in any one year. It’s to maximize the amount of retirement income you’re going to have someday. I think a lot of people miss the point when they’re filing their taxes, with a tax preparer who is just looking at a single year and not their overall lifetime tax plan.

The Myth of the Lower Tax Bracket

Jane Stogdill: Tell me, what is the myth of the lower tax bracket?

James Miller: That’s a good question. The main reason is I’m talking to somebody that they believe in tax-deferred investing. A good point is they say, well, wouldn’t I want to save on my taxes now since I’m working in probably my highest income-earning years and save at this high rate because I’m going to pay fewer taxes in retirement?

Now, this is the myth that’s out there. It’s good to save now because I’m in a higher tax bracket and when I get to retirement, I’m going to be in a lower tax bracket when I take that money out. It makes sense at first when you hear that but if you actually look at the examples in my book, it really is a myth for most people.

The reason is that, first of all, none of my clients want to be in a lower tax bracket. I am basically saying I’m going to have less money in retirement than I do now. My standard of living is going to go down.

Most people hire me as a financial advisor because they want to maintain or increase their standard of living over time. That’s the first part of that myth.

The second part of that myth is really tied into tax time bomb one. The tax rates–right now we’re at historically low tax rates, we’re at the lowest tax rates we’ve ever seen in America.

If you just go back a couple of decades, we had tax rates in the 60 and 70% for the top bracket. Today, the top tax bracket is 37% and we already know that the current tax rates are going to go up in 2525 when these tax rates sunset in the tax laws.

To save money now at these historically low rates, just to probably pay at a higher tax rate because tax rates have gone up over time and you might be 10, 20, 30 years from retirement.

Who knows what tax rates are going to be in 30 years? Probably higher than they are now would be my guess, especially with the problems with Social Security the way the government is spending for the pandemic currently. Really, the only way we’re going to be able to pay for these things is higher taxes. That’s the second part of the myth.

The third part of the myth is even if you are in a slightly lower tax bracket or even a considerably lower tax bracket, it is probably not going to be low enough to actually benefit you.

The reason for that, we’ve already mentioned in this podcast, is the fact that that account is growing over time. If you put $100 away for retirement and you’ve got 20 years to go for retirement, you are probably hoping that $100 is going to grow before you get to retirement and not still be a $100. If you’ve saved $100 and by the time you get to retirement, it could be two or three or $400, hopefully, if you have done well with your investing.

As we mentioned earlier, you’re going to have to pay taxes on all of that money, not just the $100 you originally saved. In my earlier example, $100 growing to $200, if you’re in a 20% tax bracket, you are going to save 20 upfront to pay 40 in retirement. Now, when you get to retirement in that same example that we are following even if your tax bracket has gone down from say 20 in my original example to 15 now that you are in retirement, 15% tax on that now $200 you have in retirement is still $30 of tax.

If you remember, you saved 20 and now you are paying the IRS 30 even though you’re in a lower tax bracket. You are still paying more taxes to the IRS than you ever got in a tax deduction. For most people, it’s a complete myth that being in a lower tax bracket in retirement is actually going to benefit them with tax-deferred savings accounts like a 401(k) or an IRA.

Jane Stogdill: Yeah, it’s not what I expected to hear but I am so glad to hear it. So, we just started talking about tax time bomb one, exploding tax rates. Tax time bomb two is early withdrawal penalties. I think some people know a little bit about this but let’s talk about it in the context of your larger argument.

James Miller: Sure, this is another tax time bomb that blows up for a lot of people. Life happens, I understand that as a financial planner, and life throws us curve balls. A lot of people don’t save properly in an emergency fund, and they decide at some point in their life before they are 59 and a half to access their retirement accounts with the 10% early withdrawal penalty.

Now, some people do this because it is a real emergency. They need to put a new roof on their house, they need to get a new car because they don’t have any transportation. Other people do it voluntarily to pay for a new boat or a vacation, but if you take money out of a tax-deferred retirement account before you’re 59 and a half, you’re going to not only owe taxes on the money, you’re going to also owe 10% early withdrawal penalty.

That’s a tax time bomb that is hard to swallow and a lot of people have experienced it. Although it is totally avoidable with good financial planning.

Benefits of a Roth Account

Jane Stogdill: Okay and the way to avoid it is to have money in a Roth, which the penalties for pulling it out of a Roth are not as steep, or to have an emergency fund, which we should all have anyway, right?

James Miller: Or both. If you saved in a Roth account, the money that you put in you can always take out with no taxes or penalties. If you put $500 in a Roth and three months from now, let’s just say it’s grown to $501, you can take out the 500 you put in with no taxes or penalties. It’s only the one dollar of interest that it’s grown that if you take that out before 59 and a half, you are going to owe taxes and the 10% penalty.

If you needed to get that money, it doesn’t hurt nearly as bad to pay taxes and penalties on one dollar of growth. You can always get back your 500 without taxes and penalties. It gives you a lot more flexibility if life does throw you a curveball. It still hurts because once you take that money out, you can never put it back in so you’ve kind of robbed your future retirement income but at least you are not going to blow up that tax time bomb on yourself by taking it out of the tax-deferred account.

Jane Stogdill: Yeah, okay. All right, the third time bomb is sharing your retirement with the IRS. What do you mean by that?

James Miller: We’ve already hit on this one. This is if you save in a tax-deferred account like your 401(k) or an IRA, you are going to share that account with the IRS. They are going to tax you. All those loans that they gave you overall those working years, they’re not going to collect on all of those loans. If you get to retirement and you’ve saved a million dollars in an IRA or 401(k), you should be proud of that balance, but you should also understand that you don’t have a million dollars.

The IRS has been in partnership with you for years at this point, giving you all of those little tax deduction loans along the way. They are going to tax that entire million bucks, so again at 20% tax bracket, out of that million dollars, only 800,000 is yours. 200,000 of it already belongs to the IRS and they are going to collect on it. You’re in partnership with the IRS on that tax-deferred money.

Jane Stogdill: Okay, number four is the Social Security tax and you say there is a way to avoid that or some of it.

James Miller: There is and this is a big issue. Nobody is thinking when they are in their working years about things like Social Security tax. I mean Social Security is so far out there for a lot of people. They are not thinking about how the actions they take today could affect something like Social Security later but it does and the IRS has set this up strategically knowing that people don’t think about this.

If you poll people, most people think Social Security is tax-free. I mean we paid for it with taxes, right? We would think that it would pay us back without taxes but that is not the case for most people. The way it works, briefly, in America if the only thing you received is Social Security, well, you are fairly poor, to be honest. Social Security is not that much. In 2021, the average Social Security check is a little more than $1,400 a month. You could survive on it but it is not a lot of money.

If that is all you’ve received, there is no tax on it but the government set up thresholds. If you have other income in retirement besides Social Security and you breach these thresholds, Social Security does become taxable, and the thresholds aren’t that high either. For single people, the first threshold starts at 25,000, and for married filing jointly, so couples, that threshold starts at $32,000. For a quick example, if you are single and you get your Social Security and you take out more than $25,000 from say your 401(k), you are going to trigger taxes on your Social Security as well.

Had you put that money into a Roth IRA, Roth withdrawals don’t count towards making your Social Security taxable at all and you can take as much as you want out from a Roth without triggering your taxes on your Social Security. The average American pays $5,600 a year in taxes on just their Social Security. Had they done some planning earlier in life and utilized the Roth accounts like I am advocating for, they could have avoided that tax on their Social Security.

Jane Stogdill: Wow, there are a few other time bombs and generally so much good advice in this book. I want to get to the last section but first, can you tell us a little bit more about provisional income and why that’s important when you reach retirement age?

James Miller: Sure, another term that most people have never heard of is provisional income. This is the income that the IRS uses to calculate your taxes when you get to retirement. Those thresholds I just mentioned, the 25,000 and the 32,000 are provisional income thresholds. Provisional income is half of your Social Security, so if you are going to get $30,000 a year from Social Security, 15,000 of accounts is provisional income.

Any money that you take out of a tax-deferred account like an IRA or 401(k) is going to count as provisional income. Any tax-free interest you earn, like from municipal bonds, is also going to count. The final category is rental income. A lot of retirees like to own properties for diversification purposes, but any rental income is also going to count as provisional income. When you add that provisional income together, if it breaks those thresholds that I mentioned earlier, that’s what’s going to trigger the taxes on your Social Security and determine how much you’re going to pay for Medicare as well.

Jane Stogdill: Okay, as you mentioned earlier, Roth accounts have limits. You can only put in a certain amount a year. Obviously, people should be aiming to save more a year than the Roth allows. How do we do the calculus here to come out with the best outcome?

James Miller: There are legal ways for just about anybody to get significant sums of money into Roth-type accounts. Everybody is eligible to do a Roth IRA up to certain income limits and then if you are above those income limits, you can still do a Roth IRA by the backdoor Roth IRA strategy, where you put money into a nondeductible IRA and then convert it to a Roth IRA. Most people do have access to a Roth 401(k) as well, which allows you to put this year 19,500 extra dollars into a Roth.

On top of that, you can always convert anything that’s in a traditional account like a 401(k) or an IRA to a Roth IRA, and when you do that, you do have to pay the taxes on that money but then it becomes tax-free going forward. I call this refinancing your IRA. When I talk to most people about, “Well, why don’t we take your 401(k) and convert it to a Roth?” They say, “No way, I’d have to pay taxes on all of that money.” Even though they know they’re going to have to pay taxes on that money someday.

The more that account grows, the more taxes they’re going to have to pay someday. People don’t like the feeling of paying all of those taxes now, which is funny to me because people are willing to do this in another area of their life in their mortgage. We are at historically low-interest rates today and I don’t know many people that own a home that wouldn’t be interested in locking in a new lower interest rate in their home for the next 30 years.

Even though it is going to probably extend their mortgage out to 30 more years and they’re probably going to pay thousands of dollars in fees to the mortgage company in order to refinance their loan and lock in that lower rate. They want that lower rate.

That’s what I’m an advocate of with your IRA. Let’s convert it and let’s pay taxes today at today’s known historically low tax rates, and then make that account tax-free forever going forward, not just so you never have to pay taxes on it again but at least you can do it now on your terms at a known tax rate, not some future tax rate someday that you don’t even know what it’s going to be. Plus, by doing it now, the growth going forward will all be tax-free and you’ll be building a Roth account that won’t trigger other taxes like Social Security and Medicare as well.

Jane Stogdill: Wow, there is so much to think about in this book. It’s been such a pleasure speaking with you. Congratulations again on the book. Listeners, it’s called Divorce the IRS: How to Defuse Your Biggest Tax Time Bombs Before You Retire. In addition to reading the book, where can people go to learn more about you and your work?

James Miller: The book has a website. It’s There are lots more resources there, you can download the first chapter of the book for free there as well if you are interested in getting a preview of it, and there are calculators and all sorts of other information on all of the things we discussed today that can be found there on that website.

Jane Stogdill: Great, thanks so much.

James Miller: It’s my pleasure.