Soliciting Feedback for Improvements to the Media Bias Fact Checker Bot
Soliciting Feedback for Improvements to the Media Bias Fact Checker Bot
Hi all!
As many of you have noticed, many Lemmy.World communities introduced a bot: @MediaBiasFactChecker@lemmy.world. This bot was introduced because modding can be pretty tough work at times and we are all just volunteers with regular lives. It has been helpful and we would like to keep it around in one form or another.
The !news@lemmy.world mods want to give the community a chance to voice their thoughts on some potential changes to the MBFC bot. We have heard concerns that tend to fall into a few buckets. The most common concern we’ve heard is that the bot’s comment is too long. To address this, we’ve implemented a spoiler tag so that users need to click to see more information. We’ve also cut wording about donations that people argued made the bot feel like an ad.
Another common concern people have is with MBFC’s definition of “left” and “right,” which tend to be influenced by the American Overton window. Similarly, some have expressed that they feel MBFC’s process of rating reliability and credibility is opaque and/or subjective. To address this, we have discussed creating our own open source system of scoring news sources. We would essentially start with third-party ratings, including MBFC, and create an aggregate rating. We could also open a path for users to vote, so that any rating would reflect our instance’s opinions of a source. We would love to hear your thoughts on this, as well as suggestions for sources that rate news outlets’ bias, reliability, and/or credibility. Feel free to use this thread to share other constructive criticism about the bot too.
My personal view is to remove the bot. I don't think we should be promoting one organisations particular views as an authority. My suggestion would be to replace it with a pinned post linking to useful resources for critical thinking and analysing news. Teaching to fish vs giving a fish kind of thing.
If we are determined to have a bot like this as a community then I would strongly suggest at the very least removing the bias rating. The factuality is based on an objective measure of failed fact checks which you can click through to see. Although this still has problems, sometimes corrections or retractions by the publisher are taken note of and sometimes not, leaving the reader with potentially a false impression of the reliability of the source.
For the bias rating, however, it is completely subjective and sometimes the claimed reasons for the rating actually contradict themselves or other 3rd party analysis. I made a thread on this in the support community but TLDR, see if you can tell the specific reason for the BBC's bias rating of left-centre. I personally can't. Is it because they posted a negative sounding headline about Trump once or is it biased story selection? What does biased story selection mean and how is it measured? This is troubling because in my view it casts doubt on the reliability of the whole system.
I can't see how this can help advance the goal (and it is a good goal) of being aware of source bias when in effect, we are simply adding another bias to contend with. I suspect it's actually an intractable problem which is why I suggest linking to educational resources instead. In my home country critical analysis of news is a required course but it's probably not the case everywhere and honestly I could probably use a refresher myself if some good sources exist for that.
Thanks for those involved in the bot though for their work and for being open to feedback. I think the goal is a good one, I just don't think this solution really helps but I'm sure others have different views.
One problem I’ve noticed is that the bot doesn’t differentiate between news articles and opinion pieces. One of the most egregious examples is the NYT. Opinion pieces aren’t held to the same journalistic standards as news articles and shouldn’t be judged for bias and accuracy in the same way as news content.
I believe most major news organizations include the word “Opinion” in titles and URLs, so perhaps that could be something keyed off of to have the bot label these appropriately. I don’t expect you to judge the bias and accuracy of each opinion writer, but simply labeling them as “Opinion pieces are not required to meet accepted journalistic standards and bias is expected.” would go a long way.
I think it should be removed
My personal view is that the bot provides a net negative, and should be removed.
Firstly, I would argue that there are few, if any, users whom the bot has helped avoid misinformation or a skewed perspective. If you know what bias is and how it influences an article then you don't need the bot to tell you. If you don't know or care what bias is then it won't help you.
Secondly, the existence of the bot implies that sources can be reduced to true or false or left or right. Lemmy users tend to deal in absolutes of right or wrong. The world exists in the nuance, in the conflict between differing perspectives. The only way to mitigate misinformation is for people to develop their own skeptical curiosity, and I think the bot is more of a hindrance than a help in this regard.
Thirdly, if it's only misleading 1% of the time then it's doing harm. IDK how sources can be rated when they often vary between articles. It's so reductive that it's misleading.
As regards an open database of bias, it doesn't solve any of the issues listed above.
In summary, we should be trying to promote a healthy sceptical curiosity among users, not trying to tell them how to think.
Who fact-checks the fact-checkers? Fact-checking is an essential tool in fighting the waves of fake news polluting the public discourse. But if that fact-checking is partisan, then it only acerbates the problem of people divided on the basics of a shared reality.
This is why a consortium of fact-checking institutions have joined together to form the International Fact-Checking Network (IFCN), and laid out a code of principles. You can find a list of signatories as well as vetted organizations on their website.
MBFC is not a signatory to the IFCN code of principles. As a partisan organization, it violates the standards that journalists have recognized as essential to restoring trust in the veracity of the news. I've spoken with @Rooki@Lemmy.World about this issue, and his response has been that he will continue to use his tool despite its flaws until something better materializes because the API is free and easy to use. This is like searching for a lost wallet far from where you lost it because the light from the nearby street lamp is better. He is motivated to disregard the harm he is doing to !politics@Lemmy.World, because he doesn't want to pay for the work of actual fact-checkers, and has little regard for the many voices who have spoken out against it in his community.
By giving MBFC another platform to increase its exposure, you are repeating his mistake. Partisan fact-checking sites are worse than no fact-checking at all. Just like how the proliferation of fake news undermines the authority of journalism, the growing popularity of a fact-checking site by a political hack like Dave M. Van Zandt undermines the authority of non-partisan fact-checking institutions in the public consciousness.
To clarify what MBFC considers "MIXED" factual reporting (the same rating they give known disinformation factory Breitbart):
Further, while The Guardian has failed several fact checks, they also produce an incredible amount of content; therefore, most stories are accurate, but the reader must beware, and hence why we assign them a Mixed rating for factual reporting.
They list like five fact checks, while The Guardian puts out basically quintuple that every day. And moreover, this is the sort of asinine nitpick that they classify as a "fact check".
"Private renting is making people ill." "Private renting is making people ill, but maybe this happens with other housing situations too, we don't know, so we rate this as false."
MBFC's ratings for "factual reporting" are a joke.
Remove it.
No need for a bot. Obvious misinformation should be removed by the mods. Bias is too subjective to be adjudicated by the mods. Just drop it already. It's consistently downvoted into oblivion for a reason. The feedback has been petty damn obvious. This whole thread is just because the mods are so sure they're right that they can't listen to the feedback they already got. Just kill the bot.
The bot is basically a spammer saying "THIS ARTICLE SUCKS EVEN THOUGH I DIDN'T READ IT" on every damn post. If that was a normal user account you'd ban it.
In literally every thread I've seen it post in, it gets downvoted to hell.
It has been helpful and we would like to keep it around in one form or another.
Bull fucking shit. The majority of feedback has been negative. I can't recall a single person arguing in its favor, but I can think of many, myself included, arguing against it. I hope you can find my report of one particular egregious example, because Lemmy doesn't let me see a history of things I reported. I recall that MBFC rated a particular source poorly because they dared to use the word "genocide" to describe what's going on in Gaza. Trusting one person, who clearly starts from an American point of view, and has a clearly biased view of world events, to be the arbiter of what is liberal or conservative, or factual or fictional, is actively harmful.
No community, neither reddit nor Lemmy nor any other, has suffered for lack of such a bot. I strongly recommend removing it. Non-credible sources, misinformation, and propaganda are already prohibited under rule 8. If a particular source is so objectionable, it should be blacklisted entirely. And what is and is not acceptable should be determined in concert with the community, not unilaterally.
Edit: And another thing! It's obnoxious for bot comments to count toward the number of comments as shown in the post list. Nobody likes seeing it and thinking "I wonder what people are saying about this" and it's just the damn bot again. But that's really a shortcoming in Lemmy.
The bot is basically loud as fuck in a way that disrupts the comment feed.
Imagine how comments should create and add to a conversation. Imagine how various lemmy clients feed or service that conversation….
Now imagine how a double dropdown big as fuck post says “fuck you” to that conversation.
Just please consider how the form of your shit can be just as imposing as the content, which I really appreciate.
Yet somehow your posts always have me thinking “shut the fuck up” which seems antithetical to building a community.
I'm gonna be Left-Center on this with reliable credibility that the bot is useless at best.
It is reporting on the source, not the content, of what is posted which is already going to be a problem for discourse.
If there are media sources that are known or proven to be a problem, I would find it preferable the bot just alert that and ignore anything else.
- Please, move the bias and reliability outside of the first accordion/spoiler. This is the sole purpose the bot was meant to provide. If we can't see that at a glance, it's bad. I don't see how these few words are "too long" either. I feel like a lot of the space could be cleared by turning the "Search Ground News" accordion into another link in the footer.
- While I personally don't see the point of the controversy, it wouldn't be too hard to manually enter Wikipedia's Perennial Sources list into the database that the bot references, especially with MediaWiki's watchlist RSS feed. This would almost certainly satisfy the community.
- Open source the database and the bot. Combined with #2, this could also offer an API to query Wikipedia's RSP for everyone to use in the spirit of fedi and decentralization.
Get rid of it entirely. In another one of your comments you acknowledged that it "seemed" like the bot is an extension of the mods telling everyone else what to think. You are close. It doesn't seem that way, it is that way.
Also, bot is annoying AF. If you really are in love with so much, make it an opt-in service and it can DM all the psychos who want to be spammed by it.
@jeffw@lemmy.world Why did you stop replying to posts here? Most people is telling you the bot is bullshit. You stopped commenting in this thread while being active elsewhere, are you going to take action or not?
I blocked it straight away so I don't have a dog in this fight but I'm instantly skeptical of any organization that claims to be the arbiter of what is biased and to what degree.
How do you rate bias without bias? What is the bot's definition of left or right? How did you build your ratings?
It's all bullshit, man.
How much more feedback do you need to gather on the subject to understand that a bot with a garbage datasource is no use to anyone? Even opening this thread is an insult and a sign of how little you recognize and care for your community. Remove the shit bot already instead of fishing for excuses to keep it active.
I'm frankly rather concerned about the idea of crowdsourcing or voting on "reliability", because - let's be honest here - Lemmy's population can have highly skewed perspectives on what constitutes "accurate", "unbiased", or "reliable" reporting of events. I'm concerned that opening this to influence by users' preconceived notions would result in a reinforced echo chamber, where only sources which already agree with their perspectives are listed as "accurate". It'd effectively turning this into a bias bot rather than a bias fact checking bot.
Aggregating from a number of rigorous, widely-accepted, and outside sources would seem to be a more suitable solution, although I can't comment on how much programming it would take to produce an aggregate result. Perhaps just briefly listing results from a number of fact checkers?
Im sorry but the sole concept of the bot is bullshit and as many have said already the idea is biased per se. I wish i lived in the same world as mbfc where it seems like all media is left-center.
If anything, what would be needed would be a bot that checked if the information on that article has any known missinformation or incorrect/wrong facts. And that would be extremely hard to maintain and update as a lot of news are posted before any fact checking can be done.
Here's the comment reply from when I first asked what was wrong with MBFC. Gotta say. I agree with that comment. I'm surprised more people haven't posted similar examples here.
https://lemmy.dbzer0.com/comment/12328918
Edit: here is the text from the linked comment.
I'm just gonna drop this here as an example:
https://mediabiasfactcheck.com/the-jerusalem-report/
https://mediabiasfactcheck.com/the-jerusalem-post/
The Jerusalem Report (Owned by Jerusalem Post) and the Jerusalem Post
This biased as shit publication is declared by MBFC as VEEEERY slightly center-right. They make almost no mention of the fact that they cherry pick aspects of the Israel war to highlight, provide only the most favorable context imaginable, yadda yadda. By no stretch of the imagination would these publications be considered unbiased as sources, yet according to MBFC they're near perfect.
Tell the bot to never be the first comment. I find it very frustrating when I see "a comment on this post" and it's just the bot. I'm here to read what people have to say so it is very annoying when I think someone said something and it's just the bot.
There was even a front page meme about this last year, but with another noisy bot. Lemmy doesn't bury new comments like Reddit does, so there's no real penalty to making the bot wait.
Ban it and all bots honestly. I hate seeing a comment on a thread just to find out it's a bot. If not use like this continues we might see a fresh post with 6 new comments, all of them bots that don't add to the discussion.
The bot has no purpose. Either an article can be posted or not there's no reason for the bot prompt. It just looks like thought policing using a bias checker which 'coincidentally' prefers what the current Democrats position is.
I can hardly imagine the bot stopping any fake news from being posted either.
While I think it's important to have some sort of media bias understanding, I dislike the bot being the first (and sometimes only) comment on a post. Maybe it should be reserved only for posts that are garnering attention, and has a definitive media bias answer for (the no results comments are just damn annoying to see).
It also has the knock-on effect of boosting the post higher in whichever sorting algorithm users are using. So it often feels artificially controlled whenever something has 100+ upvotes and less than 10 comments, knowing the first comment is always a bot. Like, would it be fair for me to have 10 bots that comment factual information of posts I personally like, just to boost their visibility?
Unfortunately the bot is fatally flawed as long as it's just repeating MBFC information. I would be interested in a community program but I have the same end worry. What's the risk that we create an echo chamber? It might be better than an echo chamber based on MBFC ratings but it's still an issue worth worrying about.
That said I'm down to try a community approach.
Get rid of the bot
Credibility isn't subjective. It should be a hard value.
Orientation is indeed subjective and unless in the extremes should (imo) not be defined
This bot was introduced because modding can be pretty tough work at times and we are all just volunteers with regular lives.
Then maybe it can be an internal thing only. Let people do their own critical thinking. I believe that if you're on Lemmy, you can make informed decision.
Although I do see that that bot has a very slight right wing bias I like it. It prevents the normalization of the use of literal propaganda outlets as news sources.
I have a suggestion that might be a good compromise.
The bot only comments on posts that are from less factual news sources or are from extreme ends of the spectrum.
On a post from the AP the bot would just not comment.
On a post from Alex Jones or RT the bot would post a warning.
That way there is less “spam”, but people are made aware when misinformation or propaganda is being pushed.
Also with such a system smaller biasses are less relevant and therefore become less important.
Bias ratings will always be biased. So aggregate or having multiple sources briefly used in a single small post would work best.
Not directly related to MBFC bot, but what's your opinion on other moderation ideas to improve the nature of the discussion? Something Awful forums have strawmanning as a bannable offense. If someone says X, and you say they said Y which is clearly different from X, you can get a temp ban. It works well enough that they charge a not-tiny amount of money to participate and they've had a thriving community for longer than more existing social media has been alive. They're absolutely ruthless about someone who's being tricksy or pointlessly hostile with their argumentation style simply isn't allowed to participate.
I'm not trying to make more work for the moderators. I recognize that side of it... the whole:
This bot was introduced because modding can be pretty tough work at times and we are all just volunteers with regular lives. It has been helpful and we would like to keep it around in one form or another.
... makes perfect sense to me. I get the idea of mass-banning sources to get rid of a certain type of bad faith post, and doing it with automation so that it doesn't create more work for the moderators. But to me, things like:
- Blatant strawmanning
- Saying something very specific and factual (e.g. food inflation is 200%) and then making no effort to back it up, just, that's some shit that came into my head and so I felt like saying it and now that I've cluttered up the discussion with it byeeeeee
... create a lot more unpleasantness than just simple rudeness, or posting something from rt.com or whatever so-blatant-that-MBFC-is-useful type propaganda.
Personally I'm in favor of the bot. One complaint I've seen that I agree with is that it doesn't need to float high up in the comments. If it was simply made to not upvote itself, it would stay nearer to the bottom naturally, which I think would be preferable.
Users: “This bot is absolute trash.”
Mods: "Well, I mean, it doesn't set your computer on fire, so that's a win, right guys?"
You just know the guy who coded this bot is probably holed up in his mom’s basement, unloved and ignored, tinkering away on this disaster while muttering to himself in a corner. Bet he’s got some rare genetic condition that makes him think using ChatGPT to write code is a good idea. He’s probably got a conversation open right now, asking for help on how to fix the mess he created, but let’s be real—he’s more likely to ask how to heat up Hot Pockets without burning his hands. This bot’s so bad, it’s like he’s using it as a cry for help.
Delete it, nobody likes it. It is useless and nobody likes it
Cripes, you really got thrown to the wolves here.
I appreciate the bot. I like to play a game of “guess what the bot will say” before checking. I usually win, but it’s cool to have.
I feel like bots on lemmy get way too much hate in general. There aren't that many and if you don't like you can block this one/all bots. I for one find it useful as it is.
This only applies to beautiful geniuses that include MBFC links in their posts, but the bot probably doesn't need to include the MBFC entry for MBFC. It's pretty useless and that could free up a little space. And, hey, that's something people are pretending to care about, right?
Holy moly, people seem to really be upset with this bot. I like it because it can call out when someone is doing something shady with their news sources when people like me (that don't know news sources by heart) read a posting.
We have a lot of repeat users in here that I personally feel (and I could be wrong) that have ulterior motives, like being a foreign actor spreading misinformation, trying to sew division, and lots of other foreign and domestic actors that are obsessed with one thing and throw the baby out with the bathwater (for example people obsessed with Gaza and Israel war just being nasty in general because they're angry - I'm not saying that scenario is not wrong and fucked, but this bot can help illuminate patterns in their behavior which can help us regular people tag them accordingly as a single issue participant so they are more informed when engaging that person)
My suggestion is to be very careful about crowd-sourcing the rating process. Nearly every post I go into this bot is super negative on its downvotes. Rather than just simply blocking the bot, people are retaliating against something they don't agree with. You would likely see that translate to your crowd-sourcing rating also at best. At worst you would see bad actors focused on division and misinformation making a fuckery of it all.
I'm not saying don't include the community, but brainstorm with this potential pitfall in mind.
I like this community, and want to see it continue to be as factually correct and represented fairly, and appreciate the mods and their ongoing challenges with the people that would seek to upset the apple cart at any opportunity.
I think the bot adds value and applaud the honest effort to make improvements.
The news source of this post could not be identified. Please check the source yourself. Media Bias Fact Check | bot support
I think the bot is incredibly useful. The criticism falls under a very specific group of users being very loud about their preferred source not ranking the way they expect.
Linking additional sources will improve it. Wikipedia maintains an active list and has an incentive to do so. Personally, I'd like to see a transparent methodology applied to a source: number of articles retracted silently, corrections issued in last 30 days, etc.
That having been said, I'd rather see efforts invested in other areas rather than inventing yet another "weighing" function for multiple ratings. Let us decide if mbfc is good enough or if we prefer ad fontes or Wikipedia or whoever. Give us two or three options and let us decide on our own.