Running Generative AI Models Locally with Ollama and Open WebUI - Fedora Magazine
How to get started running Generative AI Models Locally with Ollama and Open WebUI
curl -fsSL https://ollama.com/install.sh | sh
Yeah... No. What's with the kids these days and shitty install scripts for Linux?
You can grab that shell file and examine it. You can also check it out from their public git. I agree that this is bad practices, but not exactly uncommon.
It being a bad practice and "not uncommon" is my complaint.
why no?
I feel like it should be self evident - but I'll outline a few of the specifics.
"Hey kids - just run this rando internet script directly without inspecting it first"
Firstly - you shouldn't copy/paste directly into the terminal at all. What you copy and paste may not be what you see: example. So even "simple" things could turn bad. Paste to an editor first, then your terminal - especially if you've used "sudo" recently. It's simply bad security hygiene to just run random code in a shell and to get people used to doing so.
Secondly - you're just running some rando un-structured shell script. They can, and do, do whatever they want and put things wherever the developer felt they should go. It can re-configure your system in ways you may not want. In fact in this specific case it will add repositories to your system without asking. Did you want EPEL setup on your system? Did you want to add an external NVidia repo to your system? Too bad, it's done. Hope you saw the "Installing NVIDIA repository..." message as it flew past because that's all the notice you'll get - and you only get that because the developers told you about it.
Thirdly - since these are completely unstructured there is no uninstall without sifting through the script (which you didn't keep because it's a "curl | sh"). Again lets use this one as an example:
How many things do you think that shell script installs?
And that's all I see on a cursory walk through the 300+ lines of script. All of that may be reasonable to get things working - but that's a lot for you to find and undo if you wish to remove this later.
There are better ways to distribute software and handle dependencies.
I tried it briefly, but its hot garbage if you dont have potent hardware.
The amount of iterations you have to do, to get proper answers and the time it takes to produce them is a waste of time.
You might instead just install the Alpaca flatpak. I found it a very easy and quick way to get started.
Personally I'd just recommend either Alpaca or GPT4All, both of which are on Flathub and much easier to set up (or at least GPT4All is; I haven't tested Alpaca yet).
I still hate AI from an environmental and artistic point but I'm at least glad they didn't mention AI 'art'.
I did try to use it on Fedora but i have a Radeon 6700 XT and it only worked in the CPU. I wait until ROCM official support reaches my older Model.
ollam runs on the 6700 XT, but you need to add an environment variable for it to work... I just don't remember what it was and am away from my computer right now
I have the same setup, you have to add the line Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0" for that specific GPU to the ollama.service file
Alternatively, you don't even need podman or any containers, as open-webui can be installed simply using python/conda/pip, if you only care about serving yourself:
https://docs.openwebui.com/getting-started/quick-start/
Much easier to run and maintain IMO. Works wonderfully.
And llamafile is a binary you can just download and run, no installation required. "Uninstallation" is deleting the file.