James Miller: Divorce the IRS: How to defuse your biggest tax time bombs before you retire

Transcript

[0:00:25] JB: Thinking about planning for retirement makes most people’s head spin. We don’t even want to think about it now, much less at all but financial planner James Miller has some really powerful motivation for you, a way to get basically free money. In his new book, Divorce the IRS: How to defuse your biggest tax time bombs before you retire, he explains in plain language, all the different taxes that are waiting for us at retirement and how they work. Then he tells us what we can do today to avoid paying some of them and save thousands and thousands of dollars as a result. On Author Hour today, he shares some of his biggest tips and how to take advantage of them. Hi Author Hour listeners, I’m here today with James Miller, author of Divorce the IRS: How to defuse your biggest tax time bombs before you retire. James thank you so much for being with today?

[0:01:22] James Miller: Thanks for having me.

[0:01:23] JB: Okay, first of all, will you tell us a little bit about your line of work and how this became your specialty?

[0:01:29] James Miller: Sure, I’m in financial services, I’m a financial advisor and I’ve been a financial advisor for about 20 years now and as I and my practice matured over the years, I really realized what a big role taxes play in creating and retaining real wealth for people who are trying to create a good financial plan and reach all their financial goals in life. A lot of people pay way too much in taxes over their course of their lifetime and it became a real passion of mine to help people pay less to the IRS which really improves their financial plan.

[0:02:01] JB: Now, I feel like when people think about their retirement, they’re not always thinking about the IRS, is that part of the problem?

[0:02:08] James Miller: That is part of the problem. You know, the IRS and taxes or Uncle Sam is fairly synonymous I find. We pay taxes for a lot of things in life, not just to the IRS and we pay taxes whenever you pump gas in your car or you pay for a license for your profession. There’s lots of ways we pay taxes but the one that this book focuses on and the one I help people with the most is the tax bill they pay to the IRS.

[0:02:32] JB: Okay, now I was really interested in this book, not least because it is not the advice that I had previously received and that’s something you write about that we – not everyone understands this. Before we get into all of it, tell us first about the concept of the three tax buckets?

[0:02:53] James Miller: Sure. One thing I tell people is that I’m not your mother and father’s financial advisor. I give modern financial advice. Things over the years always change and money is changing just like anything else in the way we think about money and this book does to your point, it gives a lot of advice that you people haven’t heard before. It’s a new way of thinking about money. It’s not the old way of thinking about money. Money investments specifically, they always fall into one of the three tax buckets. The tax me now, tax me later or tax me never bucket. Most people are familiar –

[0:03:28] JB: We’re talking specifically about retirement plans, sorry, I should have said.

[0:03:32] James Miller: Sure, we could – retirement plans are honestly any money. It’s going to have any money that you have that you save or invest is going to fall into one of those three buckets. The most common one that everybody has is the tax me now bucket. If you have a checking account or a savings account, that’s a tax me now bucket. The interest you earn in your savings account at the bank, you get a 1099 for it at the end of the year and you have to pay taxes on any interest that you are tax me now. When I say tax me now I mean, tax me each year. The tax me later or tax me never buckets, now we’re moving into retirement specific accounts. Americans have a choice on how they get taxed on their retirement accounts and it boils down to those two buckets. Tax me now or tax me later or tax me never. The most popular bucket, the one most people are familiar with is the tax me later bucket. That’s the traditional 401(k) and the IRA. Those are the two most common. There’s many others that fit in there, 457 plans. KO, simple IRA’s, things like that. The traditional IRA and the 401(k) are the two most common. When I say tax me later, what I really mean is if we put money into that bucket, we don’t have to pay any taxes on what we put into that bucket now, but the IRS is going to tax us later on that bucket. Any money that we take out of that bucket later in life or pass on to a beneficiary someday is going to get taxed. It’s going to be taxed at a later point in time. The final bucket, the tax me never bucket is a newer bucket and those are generally accounts that have the word Roth around it. A Roth IRA, a Roth 401(k), those you don’t get the tax break upfront with but they grow tax free and you never pay taxes down the road on that money or its growth. That’s the tax me never bucket.

[0:05:21] JB: Okay. Because the money is going in after you’ve already paid taxes on it?

[0:05:25] James Miller: Correct and then growing tax free.

[0:05:27] JB: Wow, even the interest you earn does not get taxed?

[0:05:33] James Miller: Correct.

[0:05:33] JB: Wow, these are relatively new, these kind of Roth accounts, is that correct?

[0:05:38] James Miller: Roth has been around for about 20 years now but most employers have only started adding the Roth 401(k) option in the last 10 years. They are fairly new to a lot of people but about 85% of the 401(k) plans in America now have a Roth 401(k) option that people can choose. A lot of people don’t choose because they were already on a set it and forget it plan with the traditional 401(k), maybe they didn’t understand all the benefits of the Roth 401(k). There’s lots of reasons that people haven’t switched over to the Roth. A lot of people just don’t understand how fully the Roth 401(k) works or why it would benefit them to switch to it.

[0:06:20] JB: Now, you talk about tax deductions and tax deferred accounts are maybe not in our best interest, at least not as our only retirement savings strategy. What is this concept of borrowing money from the IRS that you discuss?

[0:06:36] James Miller: This is a key concept in the book. Every time you take a tax deduction. What I find is most people are happy to put the pain of paying taxes off to later. They just don’t understand the compounding negative effect that that has. When you put into an IRA or your traditional 401(k), almost anybody will tell you, “I love doing that because I don’t have to pay tax on the money.” That’s not really accurate. Really, it’s more accurate to say, “I don’t have to pay tax on that money this year.” You still have to pay tax on that money. It’s just you're going to pay tax on it later. Plus, you're going to pay tax on the growth. Really, what you’re doing, when you put money into a tax deferred bucket like your 401(k) or an IRA is you're taking a small loan from the IRS. Now, a simple example to explain that is, if you put a hundred dollars into your 401(k), you don’t pay any taxes on it. Let’s just say you’re in the 20% tax bracket, that saves you today, that 20% or $20 in this example. Your take-home pay goes up by $20. Now, people think that’s great. I got extra 20 bucks in my pocket but really all you did was borrow $20 from the IRS because you still owe that $20 to the IRS. You’ve just put it off to a future date. Now, the problem with that little loan is like all loans, there is a date they come due and there’s usually interest. If that hundred dollars has grown by the time you take it out someday, just to say $200 to get the example very simple, now you owe taxes on the $200. If you’re still in that same 20% tax bracket, now you owe $40 back in taxes when you take it out. That’s $20 loan that you originally got, you now have to pay back $40 in retirement. Putting into a tax deferred account really is just borrowing form the IRS today so you can pay it back and then some down the road. Most people have never made that connection, they just think they’re putting their taxes off till later and that’s a good deal and for most people, it’s not. The other reason that it’s not is because it’s not just the tax that you’re going to pay. When you take that money out later in life, the IRS has set-up a system of triggers that will create other taxes for you as well, like taxes on your social security. Extra taxes on your Medicare payments and things like that.

[0:09:03] JB: Okay, those are some of the bombs that I want to get into in a second. First of all, I just – whenever I hear about opportunities that seem so great like this. My question is like, why do Roth’s exist, why would the IRS let accounts like this exist and not charge us tax on the interest and the gains?

[0:09:28] James Miller: They have limits on the amount of money you can put into a Roth account because the tax benefits are so great. A lot of people have different ideas on why this was set-up the way it was set-up. Some people think that the Roth accounts are really set-up as a way for the wealthy to get wealthier. The truth is, most Americans don’t spend a lot of time becoming financially literate. Understanding how money works. I mean, that’s the reason I have a job as a financial adviser is because it’s just not interesting to some people. If you’re a doctor, an attorney and you're busy being a doctor and attorney, not learning all the different IRS codes and how to save your money the most tax efficient way possible. It is a great way to save for the future and I think also, the government did it to encourage people to save more for the future so they said, “Hey, we’ll even give you tax free accounts if you save for the future.” I think the government did it with good intention to try to encourage more people to save for their future in a more tax efficient way.

[0:10:29] JB: Okay. Since you’ve talked a little bit about your work as a financial planner. What’s the difference between a tax planner and a tax preparer?

[0:10:38] James Miller: Great question. Most tax services are tax preparers. In my book, I talk about the difference between tax planner and a tax preparer. A tax preparer is somebody who is going to fill out and file all your tax forms for you. This is what most Americans are used to dealing with. Each in our block, liberty tax, all the big national tax places, what they want to do is get you back the most money on your taxes in any one year and I understand why they want to do that and I understand also why people want that. I mean, I’ve never met anybody who goes to see the tax person when the tax person says, “Hey, I can get you an extra $2,000 of return this year.” Nobody goes, “I don’t want that, I’d love to give that extra money to the government.” People are happy about that and that’s why tax preparation services try to get you at the most money back. They want to make their clients happy. You might go back to them year after year and refer people. “Hey, my tax guy got me an extra $2,000 back, you should go use his service.” What most people don’t realize is that thinking about their financial plan on a year-to-year basis might not be what’s best for them in the long run. They’re not taking their lifetime tax plan into account. Sometimes it’s not best to get the most taxes back in any one year. The point of tax deferred or tax-free savings isn’t to get the most back in any one year. It’s to maximize the amount of retirement income you’re going to have someday. I think a lot of people miss the point when they’re filing their taxes, whether a tax preparer who is just looking at a single year and not their overall lifetime tax plan if that makes sense.

[0:12:20] JB: Tell me, what is the myth of the lower tax bracket?

[0:12:24] James Miller: That’s a good question. The main reason if I’m talking to somebody that they believe in tax deferred investing. It’s a good point is they say, well, wouldn’t I want to save on my taxes now since I’m working in probably my highest income earning years and save at this high rate because I’m going to pay less taxes in retirement. Now, this is the myth that’s out there. It’s good to save now because I’m in a higher tax bracket and when I get to retirement, I’m going to be in a lower tax bracket when I take that money out. It makes sense at first when you hear that but if you actually look at the examples in my book. We’ll talk about a couple here that it really is a myth for most people. The reason is because first of all, I don’t know, none of my clients, I don’t want to be in a lower tax bracket. I don’t – I mean, basically saying I’m going to be – I’m going to have less money in retirement than I do now. My standard of living is going to go down. Most people hire me as a financial advisor because they want to maintain or increase their standard or living over time. That’s the first part of that myth. The second part of that myth is really tied into tax time bomb one. The tax rates, right now we’re at a historically low tax rates, we’re at the lowest tax rates we’ve ever seen in America. If you just go back a couple of decades, we had tax rates in the 60 and 70% for the top bracket. I mean today, the top tax bracket is 37% and we already know that the current tax rates are going to go up in 2525 when these tax rates sunset in the tax laws. To save money now at this historically low rates, just to probably pay at a higher tax rate because tax rates have gone up overtime, you might 10, 20, 30 years from retirement. Who knows what tax rates are going to be in 30 years? Probably higher than they are now would be my guest especially with the problems with social security the way the government is spending for the pandemic currently. Really, the only way we’re going to be able to pay for these things is higher taxes. That’s the second part of the myth. The third part of the myth is even if you are in a slightly lower tax bracket or even a considerably lower tax bracket, it is probably not going to be low enough to actually benefit you. The reason for that, we’ve already mentioned in this podcast is the fact that that account is growing overtime. If you put a $100 away for retirement and you’ve got 20 years to go for retirement, you are probably hoping that $100 is going to grow before you get to retirement and not still be a $100. If you’ve saved a $100 and by the time you get to retirement, it could be two or three or $400 hopefully if you have done well with your investing. As we mentioned earlier, you’re going to have to pay taxes on all of that money not just the $100 you originally saved. In my earlier example, a $100 growing to $200, if you’re in 20% tax bracket, you are going to save 20 upfront to pay 40 in retirement. Now, when you get to retirement in that same example that we are following even if your tax bracket has gone down from say 20 in my original example to 15 now that you are in retirement. 15% tax on that now $200 you have in retirement is still $30 of tax. If you remember, you saved 20 and now you are paying the IRS 30 even though you’re in a lower tax bracket. You are still paying more taxes to the IRS than you ever got in tax deduction. For most people, it’s a complete myth that being in a lower tax bracket in retirement is actually going to benefit them with tax deferred savings accounts like a 401(k) or an IRA. Does that make sense?

[0:16:15] JB: Yeah, it’s not what I expected to hear but I am so glad to hear it. Okay, let’s go through, so we just started talking about tax time bomb one, exploding tax rates. Tax time bomb two is early withdrawal penalties. I think some people know a little bit about this but let’s talk about it in the context of your larger argument.

[0:16:40] James Miller: Sure, this is another tax time bomb that blows up for a lot of people. Life happens, I understand that as a financial planner and life throws us curve balls. A lot of people don’t save properly in an emergency fund and they decide at some point in their life before their 59 and a half and they access their retirement accounts without the 10% early withdrawal penalty to withdraw some money from their retirement accounts. Now, some people do this because it is a real emergency. They need to put a new roof on their house, they need to get a new car because they don’t have any transportation. Other people do it voluntarily to pay for a new boat or a vacation but if you take money out of a tax deferred retirement account before you’re 59 and a half, you’re going to not only owe taxes on the money. You’re going to also owe 10% early withdrawal penalty. That’s a tax time bomb that is hard to swallow and a lot of people have experienced it. Although it is totally avoidable with good financial planning.

[0:17:36] JB: Okay and the way to avoid it is to have money in a Roth, which the penalties for pulling it out of a Roth are not as steep or to have an emergency fund, which we should all have anyway, right?

[0:17:46] James Miller: Or both. If you saved in a Roth account, you can always take your money out, the money that you put in you can always take out with no taxes or penalties. If you put $500 in a Roth and three months from now, let’s just say it’s grown to $501, you can take out the 500 you put in with no taxes or penalties. It’s only the one dollar of interest that it’s grown that if you take that out before 59 and a half, you are going to owe taxes and the 10% penalty on. If you needed to get that money, it doesn’t hurt nearly as bad to pay taxes and penalties on one dollar of growth. You can always get back your 500 without taxes and penalties. It gives you a lot more flexibility if life does throw you a curveball. It still hurts because once you take that money out, you can never put it back in so you’ve kind of robbed your future happiness of retirement income but at least you are not going to blow up that tax time bomb on yourself by taking it out of the tax deferred account.

[0:18:45] JB: Yeah, okay. All right, the third time bomb is sharing your retirement with the IRS. What do you mean by that?

[0:18:53] James Miller: We’ve already hit on this one. This is if you save in a tax deferred account like your 401(k) or an IRA, you are going to share that account with the IRS. They are going to tax you. All those loans that they gave you overall those working years, they’re not going to collect on all of those loans. If you get to retirement and you’ve saved a million dollars in an IRA or 401(k), you should be proud of that balance but you should also understand that you don’t have a million dollars. The IRS has been in partnership with you for years at this point, giving you all of those little tax deduction loans along the way. They are going to tax that entire million bucks, so again at 20% tax bracket, out of that million dollars only 800,000 is yours. 200,000 of it already belongs to the IRS and they are going to collect on it. You’re in partnership with the IRS on that tax deferred money.

[0:19:44] JB: Okay, number four is the social security tax and you say there is a way to avoid that or some of it.

[0:19:50] James Miller: There is and this is a big issue. Nobody is thinking when they’re in their working years about things like social security tax. I mean social security is so far out there for a lot of people. They are not thinking that the actions they take today could affect something like social security later but it does and the IRS has set this up strategically knowing that people don’t think about this. If you pull people, most people think social security is tax free. I mean we paid for it with taxes right? We would think that it would pay us back without taxes but that is not the case for most people. The way it works briefly in America if the only thing you received is social security, well, you are fairly poor to be honest. Social security is not that much. In 2021, the average social security check is a little more than $1,400 a month. You could survive on it but it is not a lot of money. If that is all you’ve received, there is no tax on it but the government they set up thresholds. If you have other income in retirement besides social security and you breach these thresholds, social security does become taxable and the thresholds aren’t that high either. For single people, the threshold, the first threshold starts at 25,000 and for married filing jointly, so couples, that threshold starts at $32,000. For a quick example, if you are single and you get your social security and you take out more than $25,000 from say your 401(k), you are going to trigger taxes on your social security as well. Had you put that money into a Roth IRA, Roth withdrawals don’t count towards making your social security taxable at all and you can take as much as you want out from a Roth without triggering your taxes on your social security. The average American pays $5,600 a year in taxes on just their social security. Had they done some planning earlier in life and utilize the Roth accounts like I am advocating for, they could have avoided that tax on their social security.

[0:22:00] JB: Wow, there are a few other time bombs and also just generally so much good advice in this book. I want to get to the last section but first, can you tell us a little bit more about provisional income and why that’s important when you reach retirement age?

[0:22:16] James Miller: Sure, another term that most people have never heard of is provisional income. This is the income that the IRS uses to calculate your taxes when you get to retirement. Those thresholds I just mentioned, the 25,000 and the 32,000 are provisional income thresholds. Provisional income is half of your social security, so if you are going to get $30,000 a year from social security, 15,000 of accounts is provisional income. Any money that you take out of tax deferred account like an IRA or 401(k) is going to account as provisional income. Any tax-free interest you earn like from municipal bonds is also going to count and the final category is rental income. A lot of retirees like to own properties for diversification purposes, any rental income is also going to count as provisional income. When you add that provisional income together, if it breaks those thresholds that I mentioned earlier, that’s what’s going to trigger the taxes on your social security and determine how much you’re going to pay for Medicare as well.

[0:23:18] JB: Okay, as you mentioned earlier, Roth accounts, there is a limit. You can only put in a certain amount a year. Obviously people should be aiming to save more a year than the Roth allows. How do we kind of do the calculus here to come out with the best outcome?

[0:23:39] James Miller: There are legal ways for just about anybody to get significant sums of money into Roth type accounts. Everybody is eligible to do a Roth IRA up to certain income limits and then if you are above those income limits, you can still do a Roth IRA by the backdoor Roth IRA strategy, where you put money into a non-deductible IRA and then convert it to a Roth IRA. Most people do have access to a Roth 401(k) as well, which allows you to put this year 19,500 extra dollars into a Roth. On top of that, you can always convert anything that’s in a traditional account like a 401(k) or an IRA to a Roth IRA and when you do that, you do have to pay the taxes on that money but then it becomes tax-free going forward. I call this refinancing your IRA. When I talk to most people about, “Well, why don’t we take your 401(k) and covert it to a Roth?” they go, “No way, I’d had to pay taxes on all of that money.” Even though they know they’re going to have pay taxes on that money someday. The more that account grows, the more taxes they’re going to have to pay someday. People don’t like the feeling of paying all of those taxes now, which is funny to me because people are willing to do this in another area of their life in their mortgage. Were also historically low interest rates today and I don’t know many people that own a home that wouldn’t be interested in locking in a new lower interest rate in their home for the next 30 years. Even though it is going to probably extend their mortgage out to 30 more years and they’re probably going to pay thousands of dollars in fees to the mortgage company in order to refinance their loan and lock in that lower rate. They want that lower rate. That’s what I’m an advocate of with your IRA. Let’s convert it and let’s pay taxes today at today’s known historically low tax rates and then make that account tax-free forever going forward for you. Not just so you never have to pay taxes on it again but at least you can do it now on your terms at a known tax rate, not some future tax rate someday that you don’t even know what it’s going to be plus by doing it now, the growth going forward will all be tax-free and you’ll be building a Roth account that won’t trigger other taxes like social security and Medicare as well.

[0:26:01] JB: Wow, there is so much to think about in this book. You are giving away all of these great advice basically for free to me. Why? No, I’m kidding. It’s been such a pleasure speaking with you. Congratulations again on the book. Listeners, it’s called Divorce the IRS: How to diffuse your biggest tax time bombs before you retire. In addition to reading the book, where can people go to learn more about you and your work?

[0:26:28] James Miller: The book has a website. It’s divorce-the-irs.com. There is lots more resources there, you can download the first chapter of the book for free there as well if you are interested in getting a preview of it and there are calculators and all sorts of other information on all of the things we discussed today can be found there on that website.

[0:26:54] JB: Great, thanks so much.

[0:26:57] James Miller: It’s my pleasure.

[0:26:58] JB: Thanks for joining us for this episode of The Author Hour Podcast. You can get James Miller’s book, Divorce the IRS: How to diffuse your biggest tax time bombs before you retire, on Amazon. You can also find a transcript of this episode as well as previous episodes on our website, authorhour.co. Make sure to subscribe to The Author Hour Podcast for more interviews and insights into life-changing books.