1d ago

Working on MèngZǐ, the Federated Search Engine Alternative

Novel Federated Search Engine utilizing Web-rings | Under construction

About 20 days ago, I had made a blog post about an idea I had for a better federated search engine model.

It didn't take much time for it to develop into a thing I am fixated on.

I am putting the code out, its not ready or working, but it is something I am really happy to make and has filled my time with joy designing.

My current plan is the following:

Get the basic web-ring creation process down
Get scraping jobs functional
Provide a basic query system
Implement basic user accounts
Implement basic federation
Implement basic moderation

Once I am done with the core features that I have in mind, I will start working on adding more features and quality of life improvements.

Some features I want to work on to make this software more enticing to administrators:

The ability to customize what is publicly accessible.
The ability to edit the pages HTML style on the fly, without having to recompile.
Containers for easy deployment.

In regards to application design, I am taking pages from my book in developing Android applications, along with cherry-picking from projects @nutomic@lemmy.ml made.

MVC design, with static pages to provide the fastest loading experience for users
Bootstrap to make the pages responsive for any device
Diesel to abstract database interaction and migration.
Handlebars for view templating
Axum as the HTTP core

Hopefully these design decisions make my application as debt free as possible.

If you have any advice or suggestion, please do give, I want to know how I can do better or avoid common pitfalls for newcomers!

If you have criticisms, please be constructive and have empathy towards the fact of me doing this because it makes me happy.

27 comments

It’s 孟子 for anyone wondering

Really interesting proposal! To a degree the structure of Lemmy/Mbin/etc may be quite close to the categorising and moderating aspect, and might be a good place to start collecting URLs to crawl.
Each community could be considered analogous to a (rather chaotic) webring. When an instance doesn't meet your moderation expectation, defederate; if a MengZi user wants to see search results from different defederated segments, use a MengZi instance that federates with both, or just have both plugged into a searx instance.
The categorising side of MengZi could be (from an activitypub perspective) like a very cut down version of lemmy –each webring/category being a community, each website being a post, comments disabled or limited/filtered to hashtags.
A webring could be a specific sort of category/community, where a submitted website's url's page must contain specific metadata definining its membership in that ring or it is automoderated and removed. Such a category could automoderate the url and title to be the default page defined by its membership metadata. Existing webring html element standards could suffice.
A website could be crossposted to other categories, including to other instances, even to/from lemmy or other compatible activitypub sites. If a (cross)posted post is not a url returning the correct mime type for a category then it can be automoderated and deleted; same for other arbitrary criteria a category could define.
A website/post on MengZi could be accompanied by relevant crawling metadata, even full search database data available via the api for sharing to other MengZi instances to save duplication of crawling effort while distributing the database.

This sounds like a very interesting idea. I agree that Yacy doesnt work, when I checked it out years ago it was a completely bloated mess. Not sure how viable how your idea is, because Im not familiar with webrings, and not sure how the federation will work. Anyway the main challenge for this project will be to actually give useful search results, both early on when there are very few crawlers, and also later once spammers try to abuse it.
- What will abuse look like in your mind?
  
  Mainly SEO spam with text copied from other sites and lots of ads/referral links to make the owner a profit. But after thinking about it more, those would be rather easy to filter based on ad code in the HTML.
  A much bigger challenge will be the ranking of search results. When searching for a term and there are 100 pages in the index that contain it, which of these pages should be shown first? Google developed the Pagerank when they started out, so that might be a good starting point to research further.
  
  Some of the ways abuse can happen
  Crawling false data / misinformation on a topic
  Putting info on search as part of a scam / spam campaign
  Putting false news about events that are happening, or have not happened at all
  Putting false information about a business competitor
  Putting fake reviews about a product
  Just a few that I can think off.. existing websites have the issues too, but what is different is how existing sites decide relevance and how often said algorithms weed out the bad content . In my opinion a distributed search engine will have a harder time at combating those, and other potentials for abuse, because there is less control about what is getting scanned there is an open policy of who can join the distributed scanning.

Webring integration would be very cool so many queer people still use them
- The idea revolves around web-rings, but I feel like you're implying something different?

Keep us updated as you get more working.
- 🫡

An API that developers could use to integrate search in their projects would be nice. And that would also allow developers to create an app ecosystem.
This sounds very interesting, and I can’t wait to see what comes of it.
- Gotcha! Totally something to work on for the future.

any advice or suggestion, please do give
I haven't build one so I can't help as much... but I'd be honest from the start by comparing it, head-on, with alternatives (if I understood correctly) e.g. https://github.com/searxng/searxng and simultaneously, because it's federated, make it interroperable with them.
- Looking trough their docs I can't even find the word federate. From what I can tell it just refers to the part where the aggregate results

This sounds interesting. I would love to hear about how it could integrate with other platforms.
- What platforms do you have in mind?

I love the idea
I'm starting to look at airflow for my own project, not sure if you've heard of it or projects like it, but it seems like a great foundation for a scraper. I'm still evaluating options for that, but so far it's my pick
Hit me up if you get stuck or make a breakthrough, I've got a pretty good handle on activity pub and the lemmy API, and your project would add a lot to mine
- What is airflow? I was considering to use spider.
  
  Basically it's this system to do all kind of directional acyclic tasks, primarily based around data ingestion. It's very flexible and powerful, which also means there's a steep learning curve.
  To give an example, you could have a task that gatherers a list of instances and updates the database. It could also spawn a new task for each one to check if the server is up and get the version number, and you could even have it email you to create an account for new instances.
  Then from the task that made sure the server is up, you could spawn a new task that gets communities, which then spawns new tasks to ingest posts from it
  And when this whole process is done, you could have it kick off a new set of tasks to do the indexing or whatever else on the up to date data set
  It has some nice visualization of the process, you can allocate workers across devices, you can kick off the process through an API... You can use it to do anything from monitoring to scraping and doing map reduce on it. You could even federate and wire into activity pub directly, use their apis, or mix and match with scraping
  I've never worked with crawlers and I'm not sure what angle you're going to attack this from, but if normal crawlers don't play well with the fediverse this is an option

Did you ever stumble upon yacy? https://github.com/yacy The website seems down but the general direction of the project might be up your alley.
- I remember trying yacy over ten years ago and I really wanted to like it but it was functionally useless. The results I was getting would be unrelated to my query almost every time. I checked back periodically over the years but that never seemed to improve much.
  It's the first thing I thought of too though.
  
  Honestly, I was the same. I tried to run it multiple times but the results where never good enough to make the switch. Just wanted to point out it's existence to OP.
  
  I can relate to this. Tried it around a year back, really wanted to like it, but unfortunately the search results are pretty much useless..
- Do check my original blog, it explains somewhat the issue of YaCy.
  Linux reviews also has a good article on it. https://linuxreviews.org/YaCy
  
  I'll have a look at your blog. Thanks for the link, it pretty much illustrates my feelings towards yacy as well.

27 comments