Hi,
I know this is quite impossible to diagnose from afar, but I came across the posting from lemmy.world admins talking about the attacks they are facing where the database will get overwhelmed and the server doesn't respond anymore. And something similar seemed to have happened to my own servers.
Now, I'm running my own self-hosted Lemmy and Mastodon instances (on 2 seperate VPS) and had them become completely unresponsive yesterday. Mastodon and Lemmy both showed the "there is an internal/database error" message and my other services (Nextcloud and Synapse) didn't load or respond.
Login into my VPS console showed me that both servers ran at 100% CPU load since a couple of hours. I can't currently SSH into these servers, as I'm away for a couple of days and forgot to bring my private SSH key on my Laptop. So, for now I just switched the servers off.
Anyway, the main question is: what should I look at in troubleshooting when I'm back home? I'm a beginner in selfhosting and I run these instances just for myself and don't mind if I'd have to roll them back a couple days (I have backups). But I would like to learn from this and get better at running my own services.
For reference: I run everything in docker containers behind Nginx Proxy Manager as my reverse proxy. I have only ports 80, 443 and 22 open to the outside. I have fail2ban set up. The Mastodon and Lemmy instances are not open for registration and just have 2 users each (admin + my account).
Yes, the final setup would be to run Adguard on a docker container and have this container be in a VPN. I'm not sure yet, how I would do that without messing up the other things already running on that VPS. Maybe I will go for Raspberry to run adguard at home.
Edit: I have set the second DNS in Windows to the same IP, so it shouldn't have a fallback.
Hi there, I'm trying to set up AdGuard home and it doesn't seem to work properly. Maybe I'm getting it wrong on how it's supposed to work, but I'm kinda confused right now and it seems to me than Win11 is lying to me about my DNS entries ...
Here's my setup: as I have a VPS server already, I wanted to try and use it for Adguard as well. Installation there was straightforward enough and I have it up running and it has a static IP that I would use now as a DNS server, routing my traffic through it.
Now, all tutorials say that one should set the DNS entries on the router that connects to the Internet, but this option is not enabled on my router (more about this later on).
I thought, no worries, I will deal with the router situation later and just see how Adguard works with a single computer. So I went into network settings of my Win11 machine and configured my IP settings manually. Gave me a fixed IP in my home network and used the static IP from my adguard server for DNS entries. But this didn't seem to do anything. Still got ad's everywhere although my Adguard dashboard showed a lot of blocked domains (clearly identifiable as ad-servers by their name).
Ok, I went to troubleshooting and here's the first weird thing I noticed: When I sutdown Adguard (as in stopping the docker container it's running in on my server), I still can connect to the internet on my Windows machine. This shouldn't be happening, no? I set both DNS entries (main and fallback) to the same IP, where no DNS server should be running and I still got to browse the web?
So, is Windows lying to me and has a secret fallback DNS somewhere that get's used when the entries don't work? Do I not understand how this all should work?
Or - and here my specific router/modem comes into play - my hardware get's around DNS entries. I do have a "hybrid modem" which connects to the internet using both fiber DSL and LTE at the same time to get extra bandwith and speed. The customer support forum of my ISP revealed that due to the nature of this "dual line internet connection" DNS entries are fixed on the router and cannot be changed by the user.
I still think the settings in Windows should take precedence, but admittedly I have no real understanding how this is all supposed to work in detail.
So, question: how could I get Adguard to work on a VPS without being able to set DNS entries on my router? Would using a second router get around this (i.e. using the router of my ISP just as a modem and do my home network/wifi from this second router)? And why would Win11 still connect to the internet with supposedly broken DNS entries?
Edit: I fixed my problem by re-making my nginx reverse proxy and a do-over of my proxy hosts. I have yet to restart my server, though ...
I'm a beginner with all of this stuff, so I'm sure I'm not assessing correctly what's wrong with my setup. It's more of a methodical "trial and error" approach, that I have, where I change one thing at a time and see what happens ... quite time consuming but it helps me to figure things out along the way :)
However, if you have an idea, what could be wrong with my server, I'd appreciate any ideas: I run Nginx Reverse Proxy with nginx in a container within a custom network "my_network" and have assigned that container a fixed IP. I run other containers (portainer, mariadb, nextcloud, synapse) that all connect to the same custom network. The nginx container "see's" the outside web with ports 80 and 443 openend on the firewall for that container's fixed ip and routes traffic (and needed other ports) to my other containers. This is all working well and also works after restarting the server.
Now I tried to install a lemmy instance and got it up and running by bringing the lemmy containers in my custom network as well and proxy'img my nginx to the lemmy proxy. However, when I made a restart of the server, something broke and I cannot get the web-ui of NPM to load. I think somehow host names and/or IP adresses got mixed up somewhere. The containers start just fine, but I can't access it with web-ui anymore. Also reverse proxy-ing doesn't work, but if I open the needed ports on my firewall manually I can access the other services containers.
I hope this is even understandable, not sure if I'm using the correct terms ..
thanky you, this looks like exactly what I need.
I do run several webservices (nextcloud, matrix) behind the same reverse proxy (nginx prxy manager). In my setup I have one docker with nginx running, which is the only one to be exposed to the web. It proxy-ing for the other services relies upon them being in the same network. It all works well, however I ran into problems when restarting my server after a shutdown. I suspect that some of the services tried to get the same ip adress as my nginx service, which results in that service not running properly and my whole reverse proxy setup falls apart at that point.
I'm not certain, that this is really what happens but I want to try and assign the fixed ip's and see if that solves the problem.
Hi, I figured out how to get docker containers to join an existing network with putting "networks" into the respective sections of the docker-compose.yml
If I want to also give them fixed ip's on this network, what would the syntax look like in the docker-compose.yml?
thanx! I got it running now, not sure yet if federation is working, but at least I have my instance up and could register admin + standard user :)
Which domain name should I put in the nginx configuration from Lemmy? My intended domain (like lemmy.my-domain.tld) or do I put some internal IP (e.g. 172.20.0.1) and point to that IP from my host nginx?
In the configuration of the docker proxxy, do I define my domain name (like lemmy.my-domain.tld) or will I define some local IP (like 172.20.0.1) and let nginx proxy manager point to that?
This is a slow learning process for me and some of you already helped me a lot to figure out reverse proxies in general. However, I'm not there yet ... so:
How can I set up Lemmy (and Mastodon down the line) behind my existing reverse proxy? I'm trying to install from docker and the docker compose files come with templates for reverse proxy configuration, but these are (probably) only valid, if I'm installing on a dedicated server with nothing else running there.
I tried commenting out the stuff for the proxy configuration, but I can't seem to get it to work. The Lemmy install ends up with 5 docker containers (lemmy, lemmy-ui, ....) and I'm not sure which of them need to be adressed by my proxxy setup. Just getting the lemmy-ui container addressed by nginx didn't work out.
I'm probably way out of my league with what I'm trying here, but if any of you have some useful tips I'd be really grateful.
thank you, that clears things up a bit. Now it's to play around with it, until I get it up and running :)
Could you have a look at my answer to the poster above - would multiplexing mean, that I configure my internal IP 0.0.0.0:XXXA for one service and 0.0.0.0:XXXB for another?
This makes it clearer to my, would you mind helping me to understand all steps for my usecase. I want to run a lemmy instance and a mastodon instance on the same VPS, using the same domain but different subdomains - lmy.my-domain.tld and mstdn.my-domain.tld. I have my VPS IP address and setup the 2 subdomains with my domain provider (both subdomains are resolving the same IP).
I also did setup nginx on my server and can install SSL certificates for both of these domains. I'm now at the step where lmy.my-domain.tld should by directed to the lemmy service and mstdn.my-domain.tld to the mastodon service. As I understand it, both services listen to the ports 80 (http) and 443 (https). Do I now setup a room/building for Lemmy / Mastodon respectively where I tell nginx that lmy.my-domain.tld is at 0.0.0.0:3001 and mstdn.my-domain.tld is at 0.0.0.0:3002 for example. And in the config files for each of these installs I'd specify "0.0.0.0:300x" respectivly? (also have to make sure, that these docker installs don't mess with my nginx config by themselves, right?)
So, I have some idea on what a reverse proxy does and will be using nginx (with the neat proxy manager UI) for my setup.
However, I'm not completely clear what exactly I want it to do and how I cn use it to run different services on one machine. I'm especially unclear on the ports configuration .... tutorials will say things like "change the listening port to xxx for that service and to port yyy for the other service"
How does this work, which ports can I use and how do I need to configure the respective services?
EDIT: thanks everybody, your replies did help me a lot! I have my basic setup now up and running using portainer + nginx + fail2ban.
Hi, this is a follow-up on the 502 question earlier, which I think I got a step closer to solving. However, if I try to connect to my lemmy instance now, it results in a time out. Now, I have set up the ufw firewall to allow nginx http - do I need to allow anything else to get to connect? Or is my timeout error something else?
that seems to have been part of the problem, as I indeed had nginx running on the host as well. Now I get the error code "website cannot be reached" when I try to go to my instance in the browser.
I tried to follow the configuration for nginx as was in the template file on github, but I most probably have an error there. One thing confuses me, that's the ports for lemmy and the lemmy UI. I think they should be 8536 an 1235 respectively, but sometimes it says 1234 and 1236 for the UI port as well. Also in the template I'm using (https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/nginx.conf#L63) there is only one section to enter ports: proxy_pass http://0.0.0.0:{{lemmy_port}}; - which port do I enter here?
if you happen to know, please let me know :)
Hi, does anybody have an idea what the reason could be? I installed a lemmy instance on a VPS using the docker images. Beforehand I installed nginx and got a letsencrypt - certificate (which seems to have worked). I downloaded the nginx.conf file from github and made the configurations, also in the lemmy.config and docker-compose.yml files. However, I'm unsure if there's anything else I should look at. Any tips are welcome :)
Hi, I'm new with self-hosting but managed to set up my own Lemmy and Mastodon instances on a VPS recently. However, I ran into an issue with disk space quite rapidly (which I had way too few, because I started with the cheapest, smallest package for my VPS).
Now I prepare a new setup, where I'll be able to dynamically scale disk space as needed, but this can get expensive quickly. Therefor my question: How much disk space do I typically need for private (1-3 user) instances of Lemmy and Mastodon? Are there settings, where I can limit the disk space utilization (at the cost of older stored content being overwritten)?
I would be fine with needing up to like 30-40 GB, but any more than that would be getting kinda expensive ....
Out of curiosity I'm currently considering to self-host a Lemmy and a Mastodon instance. Just for me (and maybe 2-3 close friends) privately. The proposition of having full control over my social media sounds appealing to me.
However, I'm not a software developer and I have next to no experience in self-hosting anything. Also, I don't plan to make self-hosting a hobby of mine.
Given these circumstances - how much time investment do you think is needed to keep everything running smoothly. I wouldn't mind spending 1-2 hours a week, but if it's more like 1-2 hours a day, I would stay clear.
Also, are there resources for troubleshooting available? I found the installations guides and some seem to be quite good for a layperson, giving step-by-step advice, however where to go if it doesn't work?
I'm trying to make up my mind if it would be worthwhile to try or if I set myself up with wasting a lot of time :) So, any advise is welcome.