6mo ago

Media Bias Fact Check - Automation

Hello World, As many of you have probably noticed, there is a growing problem on the internet when it comes to undisclosed bias in both amateur and professional reporting. While not every outlet can be like the C-SPAN, or Reuters, we also believe that it's impossible to remove the human element from the news, especially when it concerns, well, humans.

To this end, we've created a media bias bot, which we hope will keep everyone informed about WHO, not just the WHAT of posted articles. This bot uses Media Bias/Fact Check to add a simple reply to show bias. We feel this is especially important with the US Election coming up. The bot will also provide links to Ground.News, as well, which we feel is a great source to determine the WHOLE coverage of a given article and/or topic.

As always feedback is welcome, as this is a active project which we really hope will benefit the community.

Thanks!

FHF / LemmyWorld Admin team 💖

154 comments

I think having this post isn't a great idea because you are just assuming the websites bias are legit. At the very least there needs to be a lot of warnings in the bots post about the websites biases and the methodology they use so the reader can come to their own conclusion.
Just looking over the methodlogy it's clear that it has it's own biases:
American Bias
The website itself says it’s distinctions of left and right are US based which is very skewed from the rest of the world. There should be a disclaimer or it shouldn't be used in any world news communities.
Centrist Bias
The website follows the idea of “enlightened centrism” since if it determines a website has a left/right lean (again arbitrary) it affects the factual ratings of the sources.
Examples of this are: FAIR only getting the 2nd highest rating despite never having failed a fact check.
The Intercept getting only a “mostly factual” rating (3rd highest) despite their admittance it has never failed a fact check.
Despite my personal opinions on the pointlessness of using a US based left/right bias criteria I'd feel better if it was at least kept it it's own section but when you allow it to affect the factual rating of the source it's just outright wrong. The factual accuracy of the website should be the sole thing that affects this rating.
Questionable Fact Checking
Even just checking some of their ratings raises doubts on the websites credibility.
The ADL is rated as high (2nd highest) and wasn’t found to fail any fact checks.
The ADL was found to be so unreliable on it's reporting of the Israel-Palestine conflict it is considered an unreliable source by Wikipedia.
“Wikipedia’s editors declared that the Anti-Defamation League cannot be trusted to give reliable information on the Israel-Palestine conflict, and they overwhelmingly said the ADL is an unreliable source on antisemitism.”
Maybe Wikipedia editors are a good arbiter of truth and maybe they aren’t but as people can see there isn’t a consensus and so by choosing Media Bias/Fact Check you’re explicitly choosing to align your “truth” with this websites biases.
- I'll add UN Watch to the list.
  MBFC rates it as "highly credible" despite it publishing laughably bad hit-pieces on UN officials who openly criticize Israel.
  I did a debunk on one of their articles that was removed from this very community due to disinformation, but I've posted a screenshot of my critique here for anyone who is interested.
- This is a really well-reasoned response... Which probably means the mods will ignore it
- The Intercept getting only a “mostly factual” rating (3rd highest) despite their admittance it has never failed a fact check.
  This is literally in bold at the top of the page:
  Overall, we rate The Intercept progressive Left Biased based on story selection that routinely favors the left. We also rate them as Mostly Factual in reporting rather than High due to previous fabricated work and censorship of writers.
  Fabricated work.
  Is there anything that's more of a capital crime in journalism than fabricating quotes? Surely we can all agree that publishing fiction as news is the opposite of factual reporting? They may not have failed a fact check in the last five years but it just isn't possible for them to have published fabricated news without ever failing at least one. By their own admission they failed five in that incident alone.
  
  I’m not going to die on the intercept hill here I’m fine with the fact that even though they fired the person it’s a stain on their record so sure let’s say that rating is fine.
  It was one of the first 3 I checked so I’m sure I’ll find more that are problematic when I have a chance to look because it’s their methodology that’s biased. Also the other 2 I pointed out are clearly not correct.
  Got rebuttals for any of my criticisms about the methodology?
- A standard of factuality needs to include a provision of avoiding emotionally-loaded, manipulative language. Otherwise you can pump unlimited amounts of propaganda with full factuality simply by "asking questions".
  
  I wont disagree that there should be a ranking for using loaded language but combining it with the factuality ranking twists what the ranking means since to the average person they’re going to read that as how accurate the facts are.
  It should be its own separate rating from factuality. Again if we’re going to have to have a bot like this put clear disclaimers and ideally find a better one than this.

What a terrible idea.
MBFC is already incredibly biased.
It should be rejected not promoted.
- Ok then tell me an alternative we can use in the scale for free.
  None? Then pls dont just complain complain complain... And dont suggest improvements.
  
  So much for "feedback is welcome" I guess
  
  You don’t.
  There doesn’t exist a site to magically do what you want.
  Likewise it’s not needed. It doesn’t add to the quality of discussion on the community. All it’s going to do is cause conflict as we now have to constantly point out to people how garbage the source is so that they don’t let it influence them.
  
  As the other poster says we don’t need to have something like this at all.
  If you’re adamant about it then make a post where people can suggest which one we use and vote on it. We can also adjust the bots comment to clearly call out the chosen ones biases and methodology. As it is now it’s actively harmful as I mention in my other comment.
  
  Every newspaper has its bias. MBFC heavily favors western liberal perspectives. It is often fine on domestic policy but not reliable when it comes to foreign policy.
  As this is worldnews and not Americanews, MBFC ratings are not reliable. Articles should be judged by the evidence they provide.
  
  Boooo. Running a community as a mod-dictator and not being able to hear feedback and react to it like an adult. Just because you thought of something, doesn't mean it is a good idea or that people will like it. The approach of "better than nothing" is naive and plain wrong - misinformation isn't "better than nothing" it actively hurts the community.
  
  Please remove this bot.
  
  Seeing as this is the stance the admins stance on decisions that are majority hated by the community, I'm just gonna leave this instance and go to one with admins that are more user-focused.
  I expect community leaders to take reasonable feedback from the community respectfully even if they disagree, rather than doubling down on very unpopular decisions. Especially when said community funds the platform.
  The majority of the bots posts have more downvotes than upvotes. The community has voiced its dislike for this bot as a majority.
  
  I find the only people that say MBFC is biased, are just saying they themselves have biased opinions so they don't agree with the MBFC rating
  I'm 1000% with you on this

Choosing one organization to be the arbiter of truth and bias gives them way too much power. I think fact checking should be the responsibility of whoever reads the article.
- Yes but have you considered that by using a fixed source you can shift the Overton window to where you want it to be?
  At least I acknowledge that the Overton window on lemmy.ml leans to the left. This is just slowly tilting the Overton window on lemmy.world to the right.
- Yes, everyone should always do all their own work every time. Trust nothing! Formula of gravity? Newton and Einstein might be liars, and all the science textbooks could be complicit. Do your own research. Conduct your own experiments. Is the Earth flat? Grab a sailboat and find out!
  /parody
  
  You think this organization's judgement is some objective algorithm and doesn't contain its own subjective biases?

I'm just gonna drop this here as an example:
https://mediabiasfactcheck.com/the-jerusalem-report/
https://mediabiasfactcheck.com/the-jerusalem-post/
The Jerusalem Report (Owned by Jerusalem Post) and the Jerusalem Post
This biased as shit publication is declared by MBFC as VEEEERY slightly center-right. They make almost no mention of the fact that they cherry pick aspects of the Israel war to highlight, provide only the most favorable context imaginable, yadda yadda. By no stretch of the imagination would these publications be considered unbiased as sources, yet according to MBFC they're near perfect.
- Interesting how @Rooki is still a day later active in this post responding to all the comments supporting their bot, but manages to avoid replying to all the legitimate criticisms on display.
  Really shows the mods don't value feedback, which begs the question why even bother making a thread to get feedback if you've already made up your mind.
- yet according to MBFC they’re near perfect
  Here are some quotes from the link you posted:
  They often publish factual information that utilizes loaded words (wording that attempts to influence an audience by appealing to emotion or stereotypes) to favor conservative causes. These sources are generally trustworthy for information but may require further investigation.
  After Conrad Black acquired the paper, its political position changed to right-leaning, when Black began hiring conservative journalists and editors. Eli Azur is the current owner of Jerusalem Post. According to Ynetnews, and a Haaretz article, “Benjamin Netanyahu, the Editor in Chief,” in 2017, Azur gave testimony regarding Prime Minister Benjamin Netanyahu’s pressure. Current Editor Yaakov Katz was the former senior policy advisor to Naftali Bennett, the former Prime Minister and head of the far-right political party, “New Right.”
  During the 2023 Israel-Hamas conflict, the majority of stories favored the Israeli government, such as this Netanyahu to Hezbollah: If you attack, we’ll turn Beirut into Gaza. In general, the Jerusalem Post holds right-leaning editorial biases and is usually factual in reporting.
  Overall, we rate The Jerusalem Post Right-Center biased based on editorial positions that favor the right-leaning government. We also rate them Mostly Factual for reporting, rather than High due to two failed fact checks.
  Based on MBFC's methodology, they can't have more than 6 points (out of 10) toward credibility, which is the floor for high credibility. They're one lost point from being listed as a medium credibility source, not "near perfect." They've also failed two fact checks in news reporting (not op-ed), which is seriously non-perfect. No one reading that page could walk away thinking that jpost isn't biased toward both the current Israeli government and conservative causes. MBFC calling them "right-center" is also consistent with how they're rated just about everywhere else. AllSides rates them as "center" (with a note that community feedback in disagreement believes they "lean right") and even Wikipedia describes them as "center-right/conservative".
  What exactly are you angry about here?
  
  MBFC Credibility Rating: HIGH CREDIBILITY

It has been pointed out multiple times that mbfc is ran by a Zionist.
There is no way the mod team is not aware of this by now so it must be on purpose.
- The mod team is absolutely aware of the criticisms - they're censoring them.
  I just got a comment deleted just for telling OP to engage with the criticism instead of hiding away with people who agree with them.
  I wonder if they were former Reddit power mods?
  
  Your comment was deleted for the insult, I'm guessing. I can still see it in the mod log.
- There's a reason lemmy.ml mocks MBFC daily lmao

Oh, lovely. Ministry of Truth Bots...
This is predicated on the assumption that those organizations are neutral arbitrators of facts, but they aren't.
They might have a better gauge on reality than OAN, or PatriotEagleNews.ru, but that doesn't mean platform moderators should present them as if they are a source of universal truth.
People can be critical of posts, comments, and their sources, without the heavy hand of moderators using a privatized Ministry of Truth.
We don't even have to look very far back to see how platform level "fact checking" systems are used and abused to silence and suppress information that goes against mainstream narratives or is viewed as politically damaging.
- MiniTrue would just remove wrongthink, so that's hyperbolic.
  I don't love relying on this one source of fact/bias checking so much, but the general idea of not allowing unrestricted use of whatever source without warning is good.
- Its better to have some "fact checking" than the "trust me bro" system.
  We all know all "fact checking" systems have humans behind it, those humans can have biases, dislikes or do mistakes. But thats the reason why we should not have such system is not good. Its the viewers discretion to believe into the fact/bias checks of the given page. We are just giving our best effort to simplify the view.
  Then i give you the recommendation to block the bot, if you dont like it.
  
  No it’s not.
  Bad fact checking is more harmful than not.
  
  You're putting your moderators hands on the scale and that far outweighs any community/user input into the validity of information discussed here.
  On a completely unrelated note, did you know that Hamas went on a baby beheading spree on Oct. 7?
  I know this because I read it on MSN.com, and your MediaBiasFactCheck said that MSN.com has a HIGH FACTUAL RATING
  Anyone is free to rip apart my comment, and that source, but that task becomes more difficult when bots that have been anointed as bias and fact checkers, contradict them in any way, or are themselves biased.

I love this, but I would like to state that Media Bias Fact Check seems to have a pro-Israel bias.
https://mediabiasfactcheck.com/mondoweiss/
Overall, we rate Mondoweiss as Left Biased and Questionable due to the blending of opinion with news, the promotion of pro-Palestinian and anti-zionist propaganda, occasional reliance on poor sources, and hate group designation by third-party pro-Israel advocates.
I feel like “blending of opinion with news” and “occasional reliance on poor sources” is all that really need be said.
- We don't allow Mondoweiss links either.
  
  I independently checked Mondoweiss using Media Bias a few months ago because it was posted elsewhere and I had not heard of it before, but was disturbed to see the extra reasoning behind the rating.
  It’s for sure questionable at best, the Wikipedia discussion someone else posted was enlightening on that, but “designation as a hate-group by pro-Israel” sources doesn’t really mean much when sources like the ADL equivocate anti-Zionism and anti-Semitic rhetoric in bad faith.
  https://www.washingtonpost.com/nation/2024/06/26/wikipedia-adl-jew-zionism-israel/
  Again, I love the bot, but wanted to state something to be conscious of

That's just introducing 2 more sources of bias
- Both sides? Geeet outta here.
  
  That's literally what the other source being added called Groundnews attempts to do.
- Yes, that everyone make a better picture. Instead of getting shouted at by one manipulative entity.
  
  Have you looked into who runs Media Bias Fact Check? It's pretty much as opaque as it gets for a website that claim to have an authoritative list of biases for hundreds of websites. Just because it's a meta source does not make it any more credible than any other random website.

Given the overwhelmingly negative response from the community, what is the justification for leaving the bot in place? Is it because the moderators think they know better than everyone else?
- Well, it's more that the mods know that people don't have an alternative
- Overwhelmingly negative? Those are the 24/7 negative users. We do anything: Those guys: THIS IS IS A THREAT TO DEMOCRACY
  So you stand alone in that statement. See the post vote score.
  We give you the option to block it. Block it.
  
  Numerous comments contain thoughtfully researched, balanced and reasonable criticisms, and your reaction is to basically call them just a bunch of negative nellies, rather than to consider maybe whether they have a point.
  If I made a bot that shared fake news in comments on every single news story, would you say that having the option to block that bot is sufficient? I can block anyone, yet you still ban people for breaking the rules here.
  You’re getting way too defensive, and digging your heels in - criticism isn’t always bad faith.
  
  Got it, only enthusiastic yes men are actually counted as valid members of the community.
  Interesting take, gotta admit.

could you have the bot automatically unvote its posts (make it 0) so it goes under new comments when sorted by votes?
the spoiler thing doesn't work on eternity and it kinda hides everything under it being so long
- I wish bot comments didn't count toward the comment count, too. It's annoying to see "1 comment" and then you look and it's just this or the summary bot.

Remove that. It’s too US centric. I don’t want that here.

Why does the bot spend so much space asking for donations to mediabiasfactcheck.com and thanking them for an api? Especially when it's one of the few areas not in a spoiler block so it's always shown?
- Because it's free money for MBFC
  The mods on this community have always had a rather unhealthy relationship with MBFC

Bot: Hmm this article reflects reality, thus it is biased to the left.
Using charged language like that constitutes disinformation and is reprehensible. Imagine if viewers started disregarding a source on account of your bot declaring it biased.
Shameful.

So is it time for a new news community then if the admins don't want to listen?
- You can even be better than that! You can make a community that fact check news article / news pages. Then we can add the threads from that community to the bot and have there news page specific discussion.

Please get rid of it. I'll figure my own truth from facts I descern are true. I don't need someone else telling me what to believe. Especially with the election coming up...

Media Bias Fact Check is totally meaningless in world news since the overwhelming majority of international news coverage seen in the west is filtered through just three global agencies, AP, AFP and Reuters and they always toe a pro US/Nato line.

Hmm. It's not a perfect way of measuring source bias, and bias is only correlated with truthfulness as I think they themselves admit, but I applaud the spirit.
I worry that people will put too much stock in it's assessment, and as far as I can tell propaganda posting is already pretty controlled, on .world specifically. Did you code this yourselves? Is there some way one of us could request to push to the source, like if I figure out some way it could be better? In particular, it would be good to add notes on the specific sources commenters have described as having issues not covered by MBFC.
- Currently the code is for now private. We will see if we can make it public in the future. The mods wont moderate posts because of the MBFC result.

Good, one should always refer to the Ministry of Truth before deciding what is true or false for The Party.

I appreciate having this bot, and I also think that it can be tweaked to be better. Are there other services that do something similar (ex. I see ground.news in the bot comments). What might be better is if there was a bot that linked to a few different options, so that people can benefit from the extra information. I seem to remember a Lemmy bot that was doing something like that last year, but I can't find it now.
For example, a format like this might get the benefits of the bot while also addressing the concerns people have:
Information for News Source Name
Media Bias Fact Check: Centre-Left Bias, Mostly Factual, Germany
Wikipedia: Left Bias, Privately Owned, High Factuality
See the Ground News page for this article to compare bias and credibility with other sources
See this page to learn about this bot, and how you can support the tools above.
If the bot was open sourced somewhere, then people could contribute improvements to formatting and add/remove sources as appropriate. It doesn't need to be a fully democratic process, as the maintainers would get the final say, but it would make people trust the tool a lot more.
Other small tweaks / bugs
The links need an https:// at the start, else it breaks and shows https://instance/LINK
If the data can be condensed some more, with inline links as opposed to full ones. Yes we should recommend that developers fix their apps/frontends, but with federation it's likely that there will be breakages in a lot of places. Improvements to comment format will help.
I'm not sure if the thank you and donation link is appropriate in the comment, since it feels like an advertisement / endorsement. Having that information on a separate link would be more fair. For example, ground.news also has a donation page, but it's not in the comment.
- Thanks for the feedback. With the new format we will think about it, but i think this is pretty good.
  We will discuss this and come back to you. We would love to open sourc ethe bot but the code quality for reading is not in a good state. We will have to clean the code up. But we will be working for that.
  
  So the reasons against open sourcing the bot is because you’d be embarrassed?
  
  Sounds great, I'll keep an eye out :)

But seriously, this is pretty great. Thanks for putting that together.

While I'm not as concerned with MBFC as many others are, why not use Wikipedia's RSP as the datasource? Made by the most reliable user-generated platform in the world, it's a great list of controversial sources and is completely open. Changes are also infrequent enough so that adding to the database by hand would be quite easy.
I also echo the concerns raised below on the uselessness at a glance due to the accordion hiding the only information and purpose the bot was created to serve.
- Because the mods on this community would rather choose a source that they agree with than a source that's reliable.

This about about to spawn so many sidebar threads xD
It's either going to be awesome, or hilarious. Probably both.
Any guesses for how long until the "we've disabled the bot for further testing and review" post? My bet is a month.

I think it's a great addition, but it sure does eat up a lot of space. Any way it can be condensed to the absolute basic information?
This is what it looks like for me on Boost:
- It is. You just need a client that can handle the formatting most likely?
- See reply in support post

What has it come to - bots will be hunting bots. I hope you see this too.

A whole lot of people here don't read MBFC each day and it shows. They tend to take a single and testable claim and make a decision. It's really easy to see if the claim is true or false if the claim is specific. They don't have a habit of taking a big claim and ruling it false because of one small detail like Snopes does.
- lol, look at the failed fact checks of the Guardian UK and tell me that
  
  See, this is what I'm talking about. They don't fact check articles by specific publishers. They fact check a claim. "Is this statement true", "did X Y", etc. they don't do "is this this article by the guardian true." That's a whole separate thing not done by them.
  They offer a separate service where they rate the general trustworthiness and bias of a publication but that's not the same as doing a specific article, is it?
  Your comment makes me wonder if you might be confusing them with someone else or are intentionally saying something about them that isn't accurate. Because your comment is incompatible with what they actually do.

Mods, I appreciate this bot!
Deciphering media bias is tough, and finding 1 site that will 'perfectly' identify biases is an impossible task, but at the minimum having this bot show up on posts 'gets people thinking' about the credibility of their news sources.
MBFC doesn't have to be the ultimate arbitrator either. If it is missing something about a specific article people can call it out in the comments. At the end of the day, the worst thing it does is add more data about a news source and I'm not gonna complain about that.
- Actually, I’ve checked, and you’re an unreliable commenter. Sorry.

I actually like it. Thanks.

This will be great to have. Thanks!

::: spoiler Ground News Media Bias Fact Check Credibility: [High] (Click to view Full Report)
Ground News is rated with High Creditability by Media Bias Fact Check.
Bias: Least Biased
Factual Reporting: Mostly Factual
Country: Canada
Full Report: https://mediabiasfactcheck.com/ground-news/
Check the bias and credibility of this article on Ground.News:
- https://ground.news/find?url=https%3A%2F%2Fground.news%2F%29%2C
:::
::: spoiler Media Bias/Fact Check Media Bias Fact Check Credibility: [High] (Click to view Full Report)
Media Bias/Fact Check is rated with High Creditability by Media Bias Fact Check.
Bias: Least Biased
Factual Reporting: Very High
Country: United States of America
Full Report: https://mediabiasfactcheck.com
Check the bias and credibility of this article on Ground.News:
- https://ground.news/find?url=https%3A%2F%2Fmediabiasfactcheck.com%2F%29
:::
::: spoiler Media Bias/Fact Check Media Bias Fact Check Credibility: [High] (Click to view Full Report)
Media Bias/Fact Check is rated with High Creditability by Media Bias Fact Check.
Bias: Least Biased
Factual Reporting: Very High
Country: United States of America
Full Report: mediabiasfactcheck.com
Check the bias and credibility of this article on Ground.News:
- https://ground.news/find?url=https%3A%2F%2Fmediabiasfactcheck.com%2F%29
:::
Media Bias Fact Check is a fact-checking website that rates the bias and credibility of news sources. They are known for their comprehensive and detailed reports.
Thanks to Media Bias Fact Check for their access to the API.
Please consider supporting them by donating.
Beep boop. This action was performed automatically. If you dont like me then please block me.💔
If you have any questions or comments about me, you can make a post to LW Support lemmy community.
- Media Bias/Fact Check is rated with High Creditability by Media Bias Fact Check.
  Good, I guess.
- This is a god-tier shitpost.
- I don't like this bot but I love how blatant it is lol

Thanks for this Rooki!

A lot of the criticism I've seen thus far falls into two categories:
Users complaining that their favorite source is scored poorly
Users complaining that the ratings have various sources of statistical bias
The ones in the first group I think should take it as a wakeup call that they are either headline shopping or missing out on other perspectives of current events. This is especially important on the international stage where armed conflicts will naturally produce two opposing accounts (and lots of propaganda).
The second group have a point - MBFC isn't the end all be all, but it's certainly better than nothing. Having meaningful bias measurements for each relevant scale would be impressive but way beyond what MBFC aims to do.
So all in all - I see this as a very positive change
- Thanks! Your points are perfectly on target.
  If we had any other api with parity of media bias / fact check, then we would have included it, but we only see paid, no api available.
  But for now we have added a ground.news search link so that everyone can see a third opinion on it.

Great idea

While I love the idea, I KNOW that there are certain groups that will refuse to accept that factual information. Tankies, for instance, will refuse to accept any criticism of their preferred sources. (As will Russian-asset Jimmy Dore.) Far-right conservatives will do the same, only on the other end of the spectrum.
- MBFC is not factual.
  It’s subjective. The opinion of one random man on the internet and his supposed volunteers.
  I’ve seen it rate Indian papers low and add comments like “Never once reported anything false.” Meanwhile some US garbage will be ranked as reliable and the comments are an essay on all the times they’ve been busted lying.
  
  Got an example of a US source being rated reliable despite failed fact checks? I'd be interested in seeing that.

For those reporting the bot:
We know! We worked with the Admins to enable it. :)
- Then I guess we'll continue until you remove it.
- Thanks for this!
  Edit: And happy cake day!
  
  Thanks! I can't believe it's been a year since Reddit imploded! 15 years there and never looked back!

I for one support this. Sure it's not perfect and the bias checker had its own bias, but it's merely am advisory, you can disregard it if you want.
- You got it. Thats the point. Its just advisory, not if its Bad creditability you will get banned.

154 comments

Ground News is rated with High Creditability by Media Bias Fact Check.

Media Bias/Fact Check is rated with High Creditability by Media Bias Fact Check.

Media Bias/Fact Check is rated with High Creditability by Media Bias Fact Check.

:::