Have you ever wondered why pollsters can’t seem to predict who the next president will be? With the sheer volume of data that encircles our lives, why can’t pollsters detect the signal through the noise? John Geraci’s new book, POLL-ARIZED examines, what has gone wrong with US pre-election polls, written from the unique perspective of a market research industry insider. Lending actual data from polls, interviews with leading pollsters and a proprietary survey conducted specifically for the book.
POLL-ARIZED reframes the narrative on what’s wrong with our polling system and how pollsters should move forward. Accurate polling is essential to any democracy and America needs pollsters who re-establish trust, simplify the polling process and nudge their methodologist out of the way. The book delves deeply into these issues and provides a clear roadmap through which pollsters can once again become trusted arbiters of American public opinion.
Hey Listeners, my name is Drew Applebaum and I’m excited to be here today with John Geraci, author of POLL-ARIZED: Why Americans Don’t Trust the Polls and How to Fix Them Before It’s Too Late. John, thank you for joining, welcome to The Author Hour Podcast.
John Geraci: Thank you, thank you for having me.
Drew Applebaum: Help us kick this off, can you give us a brief rundown of your professional background?
John Geraci: Sure. I’m recording this from my home, which is just south of Rochester, New York. I grew up in the Finger Lakes, went to college in Rochester and I’ve sort of lived in this general area my whole life but I started working in survey research and opinion polling about 30 years ago in 1989. I’m currently the president of Crux Research, which is a firm I founded in 2006.
We’re a small research agency and we conduct polls. We conduct polls for some of the country’s best-known companies and nonprofit organizations and brands. Before I started the firm that I’m with now, I worked for more than 15 years at Harris Interactive, which is one of the largest and best-known polling firms in the world.
Probably best known for The Harris Poll. Many people may have heard of The Harris Poll before and I sort of felt like I was in a unique position to write this book. I got some experience as an election and opinion pollster but that’s actually a small part of what I’ve done in my career. I tend to focus on survey research projects for companies and the company – the projects that I do, they’re actually way more complicated than election polls but they use the exact same methods.
I sort of know how polls are done. I have a lot of pollsters as friends and methodologists that I know really well. I sort of have this insider’s perspective and I kind of looked into the polls more and more and how they’re doing, I realize that this is a perspective that isn’t out there much. Sometimes you get academics that will look at, how the polls have done and they look at it in one kind of lens and the media will kind of criticize them.
But I didn’t really see anything out there that was written by anyone that actually does the polls and so I really thought this was fairly important to do. I have this sort of vested interest in the polls getting better but I really got nothing riding on the current methods of conducting them. I have a lot of pollster friends and I don’t have a problem criticizing their work but I’m trying to keep it very sort of constructive criticism along the way as well.
So, I’m a long-term sort of researcher, I’ve been in the field for more than 30 years. I like to brag to people, I’ve probably authored more questionnaires and polls than almost anybody in the country I can think of at this point
Drew Applebaum: Wow. So, you’ve clearly been in the business a long time but when you decided to write the book, just by maybe digging in deeper to some of the subjects, by thinking back, you know, everyone sees their past with clearer eyes and the future, do you have any major breakthroughs or learnings or did you feel like you had to pivot along your writing journey?
John Geraci: Yeah, it was very interesting. There were definitely some surprises along the way for sure. One was just to understand more and more how the polling industry and how they react after every election has become pretty formulaic. They will sort of, when they don’t predict the election perfectly or as they might intend to, they sort of rely on the same excuses every time. They – first they’ll talk that there were late-breaking voters that we weren’t able to catch with the polls.
Drew Applebaum: Sure.
John Geraci: They did that in 1936 in a big report that came out after the 1936 election. They tried to do it again now and I was like, things that would surprise me is how can you possibly claim that when you’re doing polls up to the night before the election yet somehow there are these late-breaking voters that you know, caused it?
There were some surprises along the way. I think the smartest thing I did when writing this book is, I decided to look into this. I didn’t really set out to write a book, I just kind of got curious because after the election of 2020, I had these like sleepless nights and I was really worried because – let’s back up a little bit.
The night before the election in 2020, I put out a blog post on our corporate site where I basically said, “I’m looking at all the adjustments pollsters have made since 2016 where most people think they blew it,” right? I kind of looked on now and what they had done, what they hadn’t done, I put this blog posting out and said, “I have great confidence the pollsters are just going to nail it this time.”
It was really clear on election night, that didn’t happen and I was, “Oh, this is so embarrassing.” I go, “What am I going to do when my clients call me?” the right question for my clients to call me with is like a corporate client will be to say, “How the heck can I trust you to do this really complicated project that uses the same methods as these polls when these polls can’t even get the president right?”
I didn’t know how to answer that question and that’s what led me to this book. I started reading everything I could and the smartest thing I did was stop reading, get all the polling data for myself and just pretend it was a research project. I’m going to analyze this for myself and try to figure out what’s going on here. It’s the smartest thing I ever did and I stop looking at what other people wrote when I wrote this book.
Drew Applebaum: So, in your mind, who exactly are you writing this book for? Is this anyone who is interested in the topic or really for professionals?
John Geraci: Yeah, I think it’s a mix. I definitely think anyone working in marketing, survey research, polling would definitely be very interested in this book but I’m hoping it has a little wider audience. I would say, anyone who is very politically engaged and interested in politics but is very open-minded.
You know, someone that actually will look at facts and change their mind and I sort of say, it’s sort of the middle rest of us you never hear about on the news anymore. I think people that – it’s a non-political book. If I’ve done a good job with this book, you should have no idea where I stand politically after reading this book.
I really hope anyone that reads this has no idea. I criticize pollsters on the left, I criticize pollsters on the right. I give credit to the pollsters on all sides of everything but I really think, if you’re kind of interested and sort of engaged in politics and current events, I think you’d be interested in this book. I think you would find it interesting.
Drew Applebaum: If you don’t mind setting just the foundation of everything for us, why do we do these polls, why do they matter?
John Geraci: Yeah, I think that’s what struck me, talk about things that are surprising when I looked into it more and more. I realized, they mattered a lot more than I thought they did. We should really all care – the conversation around the polls is almost entirely negative. I’d really think that’s a mistake. It doesn’t serve anybody well to be bashing the polls and pollsters and I really believe I looked in this more and more is that public opinion is sort of like the electricity that powers a democracy, especially in America.
There was this – in the early 1900s, you may or may not know this but there was this French aristocrat that came over to America. His name was Alexis de Tocqueville, right? The French nobility sort of sent him over here to find out what is going on with this American experiment, you know? Because France was not a democracy yet at the point and this guy traveled around the country for quite a long time, he wrote, this thing called Democracy in America that really has been cited – it’s really one of the greatest books ever written, right?
But it’s all about what makes America special and in the midst of that, what he said was, compared to any other nation on earth or any other democracy or anything in history, he thought that political power in America was more tethered to public opinion than any other country in the world.
That our leaders, our sense of the public opinion where you don’t see it in other areas of the world and that’s what makes America special. I would say, even despite the polarization you see in our country, that’s still the case. Public opinion in America, I think it electrifies government power in a way you don’t even see in other countries. That’s why we should care that the polls get better because we should care there are leaders that are tethered to public opinion.
Like it or not, the polls are our best way of measuring and conveying that to our officials currently. I mean, they still do a pretty good job, I don’t want to sound over critical to them and I’m sure we’ll get into that a little bit but you know, we stop trusting the polls, we start – that’s a real, that’s an actual threat to the democracy and you don’t have to do much more than look at what’s going on in Russia right now, right?
See what can happen if the leaders just start disregarding the opinions of the people they rule, you can kinds of see what happens. I actually think it’s a – the word existential threat is thrown out there all the time in political conversations. I actually think the polls are an existential threat to American democracy If we can’t find a way to convey public opinion to our leaders in a way that everybody trusts, that’s a big deal.
That’s surprising more than anything, I didn’t think I would get into political science. I thought I was just going to look at the nuances of how polls are conducted but I actually realized along the way that they’re way more important than I ever thought they were.
The Accuracy of Polls
Drew Applebaum: So, I know it’s hard to give a number or generalize all polls but how accurate are these polls and let’s – I guess, focus on the ones that really measure public opinion.
John Geraci: Yeah, the hard part about that question, it’s the central question of the book, right? How good or bad are polls and it’s the most unsatisfying answer ever because the answer really is, it depends, it depends on what perspective you’re coming from and what criteria you want to use to judge them.
That in itself is the problem, is the pollsters judge their own work, using this technical statistical criteria that the general public does not use at all and they don’t see that their own judgment of how well they’re doing doesn’t matter. I’ll give you an example. Pollsters will say, there’s these kind complicated terms, I used one that’s called mean absolute error.
It’s really how far are the polls off, right? Again, I had all the polls from 2000 onwards, six cycles that I could use. When I looked at it, I realized that from 2000 on, all the final polls that have been done in the last six cycles, if you look at all those polls, three-quarters of them have a lower error rate than polls have had historically, right?
I think it’s perfectly technically fair to say, “Hey, the polls are – you know, polls are maybe never as accurate as you really thought they were but they’re actually getting a little bit better over time” and that’s technically a correct term and it’s not a way to describe this but it’s not a way that anyone cares about, right?
In the midst of doing this, because I work in this field, when I was all done writing this book, I went and did a poll, right? I asked the American population on what they thought of the pollster, you know? One of the things I ask them is, “What’s the purpose of an election poll?” and they came back, it was very clear.
Here’s what the American public wants in an election poll, they want to accurately tell them who’s going to win the election and they want to know how big the gap is, so you know? That’s what they want to know. Well, you can see why I said how the pollster judge themselves, it wasn’t on either those criteria, right?
Drew Applebaum: Right.
John Geraci: I looked into those criteria, right? When I look into those criteria and I found out, maybe they didn’t do so good, right? Maybe they didn’t do so good after all. Since 2000, 75% of the polls have picked the winner correctly. One in four polls hasn’t been able to get the winner right since 2000. That’s not too good.
The other thing, the other thing that a lot off people, I think the media does this more and I’ll say that again, it depends on who you are and your perspective and what criteria you want to use, they’ll use sampling error. They’ll say, “A poll is good if it’s within a sampling error” right?
Drew Applebaum: Right.
John Geraci: Well, this is the one that I think is the most clear mathematically. The final polls I looked at from 2000, 54% have results within their sampling errors, right? Now, anyone who is listening to this was at a college statistics course probably in about the third lecture you would know that 95% of the polls should be within their sampling errors, that’s literally what sampling error means, right?
Fifty-four percent did, so there is something going on, so you know very quickly when I kind of look how good or bad are they, they’re like, “Wow, you know they’re probably are worse than they have been historically. We may be asking too much of the polls” but at the end of the day, we have high standards for the polls and the pollster really are not meeting them currently.
Drew Applebaum: So what are some of the key causes of polling failures?
John Geraci: Well, there’s a lot. I’d say the bulk of the book goes through those. I actually start the book with a quote from Humphrey Taylor and Humphrey is the former chairman of The Harris Poll, a very famous pollster perhaps the one I maybe respect the most in the world and I should say on the side of been lucky enough to have worked with him in my career as well. A very well-respected pollster and I am going to paraphrase it.
But I have a quote right at the beginning of the book, it says, he says something like, “Whenever the media asked me what the potential for error is on this poll, the only honest answer I can give to them is that the potential for errors are infinite” right? This is really one of the best pollsters in the country saying this. I go, “Wow” so you can see why the bulk of the book comes up and talks about polling errors and how they arise but there’s really two broad categories of things that are going on.
There are things that are going on in the world in general and then there are the things that pollsters are doing to themselves, right? I’ll start with the first one, the things that are going on in the world. There are some things that just change over time, right? I mean, one, you can say that we’re a more polarized country now than we were 20 years ago but that doesn’t hold a lot of water with me because the more polarized opinion is, actually the easier it is to measure.
So I don’t think that has anything to do with it. Strong opinions are easier to measure than weak opinions but the thing, then there is a lot of external causes but the one to really think about is just technological change. We’re the only field that I can think of, we’re just a data-oriented field. It is all about data, it is all about computers, everything we do is computerized, data flying all around, we’re the only field that I can think of like that.
Every inch of technological progress I can think of in telecommunications has worked to make our jobs harder to do. I’ll give you – I think I’d said somewhere, I started in 1989. In 1989, I think it was 1991 when it peaked that landline telephone use in America peaked at like 93%, right? The average poll, telephone poll, got a response rate of about 60% and it was not uncommon the polls that I would write that we would get 70 to 80% of the people to respond.
So you know it wasn’t that hard, we had a phone bank that we called people. They were happy to take part in the polls, got a good sample, all easy and then you think of they have technologically. We moved from landlines to cellphones. Well now, there is no directory. I don’t know how to random digit dial a cellphone to begin with. Caller ID came up, now people knew we’re calling and they wouldn’t answer the phone, right?
We got the “do not call” registry came up, even though surely research was exempted from it, nobody knew that. I signed up for the registry, I am not answering this thing but the big thing was the iPhone. As soon as smartphones came up, that was in 2007, all of a sudden we just kind of guaranteed everyone we’d ever interview would be distracted. On our poll, it was 48% of the people on our poll were doing some sort of distracting activity while we’re trying to interview them.
You know either eating in a car, you have the kids watching TV, whatever they were doing right? So the really big changes happened sort of externally not the polls were done but it is really how we source respondents. We were able to just random digit dial phones, get 60, 70% of the people answer, it was wonderful, right? Now, we rely on these large panels of people who have raised their hand beforehand that they’ll take polls.
The average response in my poll had taken eight polls in the two weeks before I even interviewed them for this ones. You get this sort of professional – and the response rates are pathetic. Our response rates, the last really good study on response rate was in 2018, the response rates then were under 5% and I would tell you they’re between three and 5% right now. So if you think of that, the technology has just really killed us, right?
To get a thousand – most polls will go with a sample size of about a thousand. To get a thousand people to answer a poll, you have to send out between 30 and 40,000 emails and you’ll get about 1,500 will respond and you’ll have to throw out at least a third of them for bad quality because they are messing with the poll and so you wonder, well, in the face of all of that pollsters are doing kind of good as they’ve always done.
It is pretty good, it is kind of big rate really. You know, I am pretty impressed with them so the biggest external threat is really what is happening with technology and telecommunications and the pollsters have done a lot to themselves as well. I bring up things I just don’t think will be naturally brought up, one is I watched along the way election polls and opinion polls become very unprofitable to do.
Drew Applebaum: Right.
John Geraci: So most pollsters that are doing poll –, doing opinion polls, they are not making money off their opinion poll work. You know, if you have seen then driving good cars, they were paid for by their corporate clients not by their opinion poll clients and that’s actually – I own a small agency and I actually feel like the small agencies do some of the best survey research work in this country but none of us do opinion polls because we can’t afford the loss leader of doing that, right?
So all the incentives are off because you really can’t make money doing opinion polls, so that is one thing and then I think the central pieces of the book is in terms of what the pollsters are doing to themselves but first we torture respondents, you know our polls are too long, they are poorly written. I mean, I don’t know, sign up for a research panel and take a study and you’re like, “Who wrote this?” right?
But really, you know I could probably rattle off 50 issues that polls have that effect their ability to make a good prediction. I rattle a lot of them in the book but the thing that pollsters are doing is they only focus on one of them to the exclusion of all others. Now, Humphrey Taylor said there is infinite errors, these guys we all focus one, it is sampling error, right? So they have these teams of methodologists.
They figure out these enormously complex ways to deal with the fact that people that take polls don’t perfectly match the population. They do things you can’t understand what they’re doing and it would be really beautiful if there was any evidence that it was working but it is not and I will give you one more example before I pause. There was a New York Times polling expert, Nate Cohn. In 2016, he did a pre-election poll in Florida and he took the data.
He heads off to four pollsters, right? Just took the data from the poll, gave it to four pollsters and said, “Tell me who is going to win the election” right? He got four different answers. There was a five-point spread in their calculations off the same data, right? You know, different teams of methodologists look at the same data and they come up with different answers, so if polling is an art and a science, this just comes right there.
I think the biggest irony in polling is that – the biggest irony in polling to me is that the polling becomes an art when you hand it off to the scientists to deal with the data. That’s the weirdest thing I can think of to say. It should be a science but the most scientific people in the polling firm are the methodologists and it is when you give the data to them that all of a sudden polling becomes an art and that makes no sense to me whatsoever.
What You Can Do When You’re Surveyed
Drew Applebaum: Is there anything that a reader of the book can do or someone who gets that phone call to take that survey. What can they do to help out to make sure that they are giving the most accurate data or that they’re giving data that when you do hand it off like that? It is clearly analyzed.
John Geraci: Yeah, I think I don’t know if there is anything on the respondent end because there – you will hear – the problem with polls rise, the information gets out there and then it gets used in a political fashion and it became kind of fashionable in the last election to say, “Oh people are lying on polls, there is a shy Trump voter effect” and there is actually a lot of studies have been done and there’s no evidence whatsoever that it happens.
There is no evidence right now that people are being dishonest on polls and I don’t think I could do what I do for a living if I thought people are dishonest on my surveys. I think people give honest answers but if it is a poorly constructed question, you don’t know what their answers mean. I think people give it their best shot, I really do. Now the problem is that I think the industry should worry about is taking part in polls can become a political act in itself.
I think we got to worry about this for the future. In the past, people relying on polls, I think you have a hard time making a case if that [actually happens] but it could happen in the future if taking part in polls is seen as a political act and I think we should worry about that. So from a response standpoint, I know that there is much you can do on the researcher side of it, the pollster side, I think there’s a lot they can do to make things better.
Drew Applebaum: What impact do you hope the book will have on a reader and are there any immediate steps that you’ll hope they’ll take after reading the book?
John Geraci: Yeah, I just hope that one is I think I am hoping the readers will understand the complexity of how a poll actually gets done. You know, I would ask me it’s somewhere between 30 and 50 people handle a poll within a company and I go through that in the book and so there’s a lot. It is sort of pretty nuanced thing, so you sort of have an appreciation for how polls are actually done but also going back up to what I said upfront is an appreciation of how important they are to our society, that we get them right.
I am really hoping the book will help kind of kindle a more constructive dialogue around the polls because it is negative right now and I don’t think the pollsters are doing themselves any favors because the last chapter of the book is like the ten things we could do to make this better and all of my ten things focus on two issues, trust and simplicity. I mean, one is if you just step back and imagine this world where a pollster approached to take a poll, agreed to take a part and answer it honestly, I don’t think we’d have any trouble predicting who the president is going to be.
So why does the industry not work towards that? We don’t at all, instead, we just layer on a new wave of methodologists and try to see if we can deal with the data we have. We’re sort of putting a band-aid on a problem rather than focusing on not getting the injury to begin with and so if anything, I am hoping to reframe the narrative around the polls, make people understand just how important they are that they really aren’t that bad.
Maybe we are expecting a little bit too much of them but hopefully, the future is bright. I think for the time being, this is still going to be the best way we can express the public’s desires to our elected leaders, maybe they will change in the future. Technology moves quickly but I am not sure.
Drew Applebaum: Now you also offer additional resources on your website, would you mind telling us what that site is and what readers and listeners can find there?
John Geraci: Yeah, our corporate site is cruxresearch.com and that just explains what our company does and our clients, et cetera. The book will be available everywhere, Amazon, all other places and also pollarized.com with two L’s because the book is pollarized.com that will be the webpage for the book, which will just give you an overview of the book, some sample pages and a link to bio and a way to contact me as well if you’d like.
Drew Applebaum: Well, John, we just touched on the surface of the book here. There are so many more details inside but I just want to say that sharing your expertise and really educating folks in and outside of market research and the polling industry is no small feat, so congratulations on having your book published.
John Geraci: Yeah, I appreciate it. Thank you, Drew.
Drew Applebaum: John, this has been a pleasure and I am excited for people to check out the book. Everyone, the book is called, POLL-ARIZED, and that’s with two L’s and you could find it on Amazon. John, besides checking out the book, besides checking out your website, is there anywhere else where people can connect with you?
John Geraci: No, I am actually not that much on social media. I guess LinkedIn, you can find me on LinkedIn. I am maybe just old enough to not do Instagram and Twitter and all of that but yeah, you can find me on LinkedIn if you’d like to particularly if you work in this field, yeah, I would love to connect with you.
Drew Applebaum: Well, thank you so much for coming on the show today and best of luck with your new book.
John Geraci: Thank you. Thank you so much, Drew.