What Contributes to Misinformation & Disinformation?

More and more, it feels like social media favors misinformation over the truth, and it’s getting harder to sift through the barrage of posts to decipher what’s going on in the world.

The big issue with that is that as the general public’s distrust of legacy media grows, more are turning to platforms like TikTok, Instagram, X, and Threads for their news and information on current events. Unfortunately, there are no guardrails for how that information gets sourced, created, and disseminated, which can and has led to some troubling consequences.

This movement towards getting information and news from social media has made it easy for bad actors to spread disinformation effectively and quickly. It’s our responsibility as social media users to stay aware of what misinformation and disinformation are and to avoid spreading it further.

In our How to Recognize Disinformation on Social Media article, we outlined some questions to ask yourself if content feels suspicious.

1. Convenient: Does this feel super convenient? Does this make something that I already know suddenly feel explained? Does it conveniently connect dots I didn’t have connected?
2. Emotionally Charged: Does this generate a lot of emotion for people? How engaging/viral is it? If the info feels very convenient and the info feels very graphic/emotional and generates a strong response, it’s likely fabricated to do exactly that.
3. Easily Fought Over, Not Easily Proven: Are people arguing over it? The information is usually really easy for fights to break out with people taking one side or the other, but it can be difficult to disprove the information.

But misinformation doesn’t just get spread around because it’s information that sounds good, convenient, emotional, etc. There are certain variables at play that help enable its spread.

If we want to fight misinformation spread, it’s important to stay aware of how it has become so prevalent, and being conscious of the factors that contribute to it will help us recognize and potentially regulate and police its spread going forward.

We’ve created a list of a few of the catalysts to misinformation spread.

Engaging and Shareable Content

Misinformation and disinformation tend to be clippable, easily repeated, and very emotionally-charged so that it’s easy to share and repeat across platforms. Social media has shaped the public’s attention spans, making people prefer shorter, quicker, easily-digested content over long form writing or videos. This has led to misinformation spreading easier, because lying or stating false information is easier in one sentence, if you don’t have to prove it to be true because the viewer would prefer short, quick content.

An example is the current misinformation going around, “The US military is merging with Italy.” This one quick sentence is easy to repeat, and immediately paints the whole (misinfo) picture, and gets people easily and quickly riled up at the idea. The truth is not so simple, making it more difficult to refute.

Good misinformation is easy to share and understand the gist of in a brief sentence or headline. The frustrating thing about this is that the truth is typically much more nuanced and complex. That means misinformation can be stated in a sentence, but oftentimes to refute the misinformation, much more than a single sentence is needed. That makes the truth harder to share and spread.

Confirmation Biases + Recommendation Engines

Confirmation bias is a person’s inclination to seek data that supports rather than refutes his or her past opinions. Social media is a place where people follow friends and accounts they agree with, further perpetuating their exposure to confirmation bias.

Social media platforms and search engines provide readers with personalized recommendations based on past preferences and search history. The platforms want people to engage with what they see, so they’re going to push content that fits each person’s confirmation biases because it increases the likelihood a person will engage with the post.

According to an article titled, “Confirmation Bias: Roles of Search Engines and Search Contexts“, “Results of two studies show that search engines may exacerbate confirmation bias by generating results that consist only of confirming evidence for search contexts where disconfirming evidence is identified using different terms or phrases.” (1)

Visible Engagement Metrics

Social media feeds prioritize content using engagement metrics, pushing content further in the algorithm to more viewers based on how much engagement it receives from viewers. Accuracy of the post’s information is not a factor. If the content receives a lot of engagement, it has a high likelihood of getting more eyes on it, whether or not the content was true or from a reliable source. (2)

Additionally, when people see high engagement on a post, their natural inclination is to believe it, because it appears as if thousands of others have “endorsed” the thought by liking it.

This concept has been manipulated by bad actors for disinformation through the use of troll farms.

Troll Farms

According to incyber.org, “[a] troll farm (or “troll factory”) is an organized structure, public or private, that coordinates individuals—and often bots or automated scripts—to produce, disseminate and amplify content (posts, comments, rumors, videos, memes) on social media in order to influence opinion, destabilize opponents or promote a specific narrative.”

Investigations over the years have shown that bad actors like Russian intelligence agencies have used troll farms to create the appearance of public opinion towards or against specific political candidates during elections. A great example of this is the Internet Research Agency and how it operated during the 2016 US presidential election, as documented in the Mueller Report. (3)

Troll farms don’t just disseminate the info, but also can be used to like posts en masse to give the appearance of public support for content, even though much of it may be artificial.

We’ve studied this happening, ourselves, in specific disinformation campaigns that we are actively investigating. When disinformation went viral, we checked the likes and shares to find a large majority of early engagement came from suspicious accounts that seemed to be regularly used to push similar information inauthentically. We’ll hopefully be able to share more on these investigations in the future.

Influencers, Paid and Unpaid

Influencers have become large players in the game of shaping public opinion through the past decade, as they do exactly what their name implies and influence public behavior and opinion through their content and large audiences.

Influencers have been known to spread misinformation, because they make the same mistakes everyone else does and see an engaging trend and decide to hop on by sharing the sentiment, themselves. They also are oftentimes hired to spread disinformation and propaganda intentionally.

Earlier this year, Advocators.ai were outed as an agency you can hire for your PR campaigns. They activate influencers of any requested size to spread a directed message. In 2024, they won an award for a campaign that “reignite[d] public awareness and support for Ukraine amidst the ongoing conflict. […] The campaign targeted key demographics across three crucial states: Pennsylvania, North Carolina, and Michigan. Each sub-campaign was uniquely developed to resonate with its specific audience.”

Again, there are few, if any, guardrails to stop disinformation or propaganda from being spread with these types of campaigns. Advocators.ai does not provide any terms of service on their site that would lead us to believe that they vet campaigns for misinformation first.

AI / LLMs like ChatGPT, Gemini, and Claude

AI systems such as Learning Language Models (LLMs) are being used regularly to promote disinformation. AI can create realistic fake material based on the target audience. It can also be used to create bots to impersonate human users and spread disinformation.

LLMs source information from less than reliable sources often, including message forums and Reddit, where the information the LLMs are providing like fact are just posts by random strangers.

We Asked an LLM About Misinformation and it Lied to Us

We recently decided to test LLMs on misinformation by asking it about a Substack article that was actively promoting what we knew to be misinformation.

ChatGPT confidently asserted that the article had many “Verifiable Facts” in it that were “objectively checkable”, however when the listed “facts” were challenged, it was found that they were not verifiable at all.

The more concerning part of the information it provided was that the sources for the information it pulled and was stating confidently were posts on a message forum that was generally a liberal-leaning platform, which was likely to confirm my own biases that it guessed at, but meant that the information it provided was not inherently neutral.

In this case, the comment in the message forum was correct, but that doesn’t mean the LLM should have cited it as fact. At no point was the credibility of the anonymous source called out as something to keep in mind when weighing the truth of the information provided. This means that information could have also been false information, maybe even purposely injected into the message forums by anonymous posters.

False Information Gets Purposely Injected Into Platforms for LLMs to Read

This tactic of injecting bad information into platforms gets co-opted by disinformation operations often. They can post disinformation on Reddit or Wikipedia, and use this as a way to get the information seeded into LLM summaries. Our prediction is that we’ll see this tactic abused more and more as the prevalence of LLM use for research increases, unless there are specific regulations and guardrails put in place to prevent this.

(We’ll talk more in future articles about 1. specific steps to take to vet information for misinfo/disinfo and 2. how to write prompts for research using LLMs in a responsible way, so as not to spread more misinformation. Subscribe to stay tuned.)

Staying Aware is the First Step to Fighting Misinformation

It’s getting rough out there. It’s harder and harder to discern the truth from false information and lies. And it’s on us, the viewers and social media participants to sort out truth from lies and make the conscious effort to not perpetuate misinfo and disinfo on the platforms.

We’ll continue to write about this topic in depth, including sharing Weekly Fact Checks that analyze misinformation that trends every week. If that’s something you’re into, make sure to subscribe and stay tuned. We’re all in this together.

I love to support the creative artists that help me create. This article was written while listening to Hugh Ropp.

If you have some good lo-fi focus music artists you listen to, please share them! I’m always looking for new artists to support.

Sources