Earlier this evening, I surfed to PolitiFact to ascertain the truthfulness of a comment made by a well-known politician. Along the way, I decided to make a submission of my own. I so doing, I recognized a number of biases at play in the way PolitiFact solicits inputs. Even if all three editors are die-hard, apolitical objective observers, neither their Scorecards nor Truth-O-Meters can be very accurate given the constraints surrounding their submissions process.
Thus, I had to ask, “Just how reliable are fact-checking organizations like Snopes and Politifact?” The answer staring me in the face was not at all reassuring. Their process is seriously flawed.
Join me as I set out via a simple submission to determine their integrity and expertise:
“PolitiFact seeks to present the true facts, unaffected by agenda or biases.”
WHO SAID IT
I compared your current scorecard for both Donald Trump (https://www.politifact.com/personalities/donald-trump/) and Joe Biden (https://www.politifact.com/personalities/joe-biden/).
On the face of it, it appears Biden is mostly a truth-teller while Trump is mostly a liar.
I created a simple spreadsheet to calculate weighted averages and derive a stanine. This resulted in a 64.9% truthfulness rating for Biden and a 46.9% truthfulness rating for Trump. Thus, your own measures indicate Trump is about half and half truthful while Biden is about two-thirds truthful.
As I mentioned earlier, this is on the face of it.
In reality, Democrats and liberals have been both demonstratively and measurably more outspoken than Republicans and conservatives. It goes without saying the former support Biden more than Trump while the latter support Trump more than Biden.
Furthermore, it has long been established controversial incumbents draw considerably more negative comments than challengers.
Additionally, it has also long and often been measured the Top 3 mainstream news agencies — ABC, CBS and NBC, along with CNN, fall left of middle (particularly CNN) in their reporting while FoxNews falls right of middle. It’s also widely known Democrats and liberals receive their news to a significantly greater extent from the former than the later while the reverse is true of Republicans and conservatives.
Thus, I submit to you that roughly two-thirds of your submissions are from Democrats and liberals, mostly anti-Trump, while roughly one-third of your submissions are from Republicans and conservatives, mostly pro-Trump.
In fact, I propose you treat the above statement as an Hypothesis, conduct hypothesis testing, determine the actual breakdown, and adjust your inputs from submissions accordingly.
I submit that if you were to do so, you would find the breakdown between Biden and Trump is not 64.9%/46.9% as the two linked Scorecards suggest, but rather, the numbers would be closer to the same, if not reversed.
You may not believe this would be the case, but until you crunch the numbers, employing proper statistical mensuration techniques and procedures, you cannot know one way or the other.
Since your own description of your process clearly indicates you do no such thing, and in light of all of the above, I submit to you that your claim, “PolitiFact seeks to present the true facts, unaffected by agenda or biases,” ranges between Mostly False and Half True.
I submitted the above just moments ago, and rest assured, I’ll be watching PolitiFact’s response.
If they bury it, I cannot, as a statistician, trust anything PolitiFact publishes ever again, as it would be prima facia evidence they’re dishonest and have something to hide.
If they agree with my assessment and publish it as such with either a Mostly False or Half True rating on their claim that “PolitiFact seeks to present the true facts, unaffected by agenda or biases,” then I see room for great improvement in their future! They and the entire world would then be very well served if they used proper statistical procedures to minimize Response Bias and other factors which significantly skew PolitiFact’s Scorecards, Truth Meters, and pretty much all other conclusions at which PolitiFact’s three editors arrive.
If they conclude their claim is either True or Mostly True, then again, proper statistical rigor requires all objective examiners to question either their integrity or their expertise — possibly both — as each and every one of the statements in the submission I provided them above is back by decades of rock-solid statistical research performed by many leading statistical research firms.
I believe one of their greatest flaws may be they do not consider their submissions process to be a survey. Asking others to voluntarily provide submissions of suspected untruthfulness most certainly is a survey, and the way they do it is absolutely fraught with biases of many kinds. There is no statistical rigor to identify much less minimize bias in their process.
But there could be, if they made some much-needed changes.
Bottom Line: Time will tell! The TRUTH will OUT! How will PolitiFact’s integrity and/or expertise stand up against proper statistical rigor???