{"comments":[{"comment":{"id":13818712,"creator_id":9863411,"post_id":17020491,"content":"I'd say you're looking for something like a 80GB VRAM GPU. That'd be industry grade (an Nvidia A100 for example). \r\n\r\nAnd to squeeze it into 80GB the model would need to be quantized to 4 or 5 bits. There are some LLM VRAM calculators available where you can put in your numbers, like [this one](https://rahulschand.github.io/gpu_poor/). \r\n\r\nAnother option would be to rent these things by the hour in some datacenter (at about $2 to $3 per hour). Or do inference on a CPU with a wide memory interface. Like an Apple M3 processor or an AMD Epyc. But these are pricey, too. And you'd need to buy them alongside an equal amount of (fast) RAM.","removed":false,"published":"2025-01-10T14:10:27.535763Z","updated":"2025-01-10T14:17:55.837230Z","deleted":false,"ap_id":"https://palaver.p3x.de/comment/796957","local":false,"path":"0.13818712","distinguished":false,"language_id":37},"creator":{"id":9863411,"name":"hendrik","display_name":"hendrik","banned":false,"published":"2021-08-21T12:00:00Z","actor_id":"https://palaver.p3x.de/u/hendrik","bio":"A software developer and Linux nerd, living in Germany. I'm usually a chill dude but my online persona doesn't always reflect my true personality. Take what I say with a grain of salt, I usually try to be nice and give good advice, though. \r\n\r\nI'm into Free Software, selfhosting, microcontrollers and electronics, freedom, privacy and the usual stuff. And a few select other random things as well.","local":false,"deleted":false,"matrix_user_id":"@hendrik:p3x.de","bot_account":false,"instance_id":6205},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13818712,"score":7,"upvotes":7,"downvotes":0,"published":"2025-01-10T14:10:27.625166Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13832533,"creator_id":15968,"post_id":17020491,"content":"If you're running a consumer level GPU, you'll be operating with 24GB of VRAM max (RTX 4090, RTX 3090, or Radeon 7900XTX).\n\n90b model = 90GB at 8-bit quantization (plus some extra based on your context size and general overhead, but as a ballpark estimate, just going by the model size is good enough). You would need to drop down to 2-bit quantization to have any hope to fit it in a single consumer GPU. At that point you'd probably be better off using a smaller model will less aggressive quantization, like a 32b model at 4-bit quantization.\n\nSo forget about consumer GPUs for that size of model. Instead, you can look at systems with integrated memory, like a Mac with 96-128GB of memory, or something similar. HP has announced [a mini PC](https://www.hp.com/us-en/workstations/z2-mini-a.html) that might be good, and Nvidia has announced a [dedicated AI box](https://www.nvidia.com/en-us/project-digits/) as well. Neither of those are available for purchase yet, though.\n\nYou could also consider using multiple consumer GPUs. You might be able to get multiple RTX 3090s for cheaper than a Mac with the same amount of memory. But then you'll be using several times more power to run it, so keep that in mind. ","removed":false,"published":"2025-01-10T22:54:14.057040Z","updated":"2025-01-10T22:56:53.942833Z","deleted":false,"ap_id":"https://lemmy.sdf.org/comment/16992574","local":false,"path":"0.13832533","distinguished":false,"language_id":37},"creator":{"id":15968,"name":"GenderNeutralBro","banned":false,"published":"2023-06-15T14:19:07.207646Z","actor_id":"https://lemmy.sdf.org/u/GenderNeutralBro","local":false,"deleted":false,"bot_account":false,"instance_id":9},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13832533,"score":4,"upvotes":4,"downvotes":0,"published":"2025-01-10T22:54:14.057040Z","child_count":1},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13820708,"creator_id":672357,"post_id":17020491,"content":"The new $3000 NVidia Digit has 128 GB of fast RAM in an Apple-M4-like unified-memory configuration, reportedly. NVidia claims it is twice as fast as an apple stack at least at inference. Four of these stacked can run a 405B model, again according to NVidia.\n\nIn my case I want the graphics power of an GPU and VRAM for other purposes as well. So I'd rather buy a graphics card. But regarding a 90B model, I do wonder if it is possible with two A6000 at 64 GB and a 3 bit quant.","removed":false,"published":"2025-01-10T15:16:38.267344Z","deleted":false,"ap_id":"https://lemmy.world/comment/14397896","local":false,"path":"0.13820708","distinguished":false,"language_id":37},"creator":{"id":672357,"name":"tpWinthropeIII","banned":false,"published":"2023-07-22T19:16:07.253356Z","actor_id":"https://lemmy.world/u/tpWinthropeIII","local":false,"deleted":false,"bot_account":false,"instance_id":24},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13820708,"score":3,"upvotes":3,"downvotes":0,"published":"2025-01-10T15:16:38.267344Z","child_count":3},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13834162,"creator_id":14768598,"post_id":17020491,"content":"In case of LLM's you should look at [AirLLM](https://github.com/lyogavin/airllm).\nI suppose there is no conviniet integrations to local chat tools, but issue at Ollama already started.","removed":false,"published":"2025-01-11T00:06:20.724780Z","deleted":false,"ap_id":"https://lemmy.world/comment/14406092","local":false,"path":"0.13834162","distinguished":false,"language_id":37},"creator":{"id":14768598,"name":"Sylovik","banned":false,"published":"2025-01-10T23:46:01.269927Z","actor_id":"https://lemmy.world/u/Sylovik","local":false,"deleted":false,"bot_account":false,"instance_id":24},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13834162,"score":3,"upvotes":3,"downvotes":0,"published":"2025-01-11T00:06:20.724780Z","child_count":3},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13820094,"creator_id":791,"post_id":17020491,"content":"You have to specify which quantization you find acceptable, and which context size you require. I think the most affordable option to run large models locally is still getting multiple RTX3090 cards, and I guess you probably need 3 or 4 of those depending on quantization and context.","removed":false,"published":"2025-01-10T15:01:02.454586Z","deleted":false,"ap_id":"https://lemmy.world/comment/14397658","local":false,"path":"0.13820094","distinguished":false,"language_id":37},"creator":{"id":791,"name":"fhein","display_name":"ffhein","banned":false,"published":"2023-06-14T15:12:58.599060Z","actor_id":"https://lemmy.world/u/fhein","local":false,"deleted":false,"bot_account":false,"instance_id":24},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13820094,"score":2,"upvotes":2,"downvotes":0,"published":"2025-01-10T15:01:02.454586Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13817263,"creator_id":7355434,"post_id":17020491,"content":"The biggest issue will be your VRAM. If you don't have enough of it (which is very likely, even the 8B models I use need ~10gb), you'll have to use a GGUF model which will need to use your system RAM and CPU for the parts that don't fit in the VRAM, which will heavily slow it down. ","removed":false,"published":"2025-01-10T13:44:10.493279Z","deleted":false,"ap_id":"https://discuss.tchncs.de/comment/15533672","local":false,"path":"0.13817263","distinguished":false,"language_id":37},"creator":{"id":7355434,"name":"Fisch","avatar":"https://discuss.tchncs.de/pictrs/image/29f0da17-3330-47ee-97a1-b51ee98e2b48.jpeg","banned":false,"published":"2024-03-14T09:21:45.841947Z","actor_id":"https://discuss.tchncs.de/u/Fisch","local":false,"deleted":false,"matrix_user_id":"@nico:tchncs.de","bot_account":false,"instance_id":40},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13817263,"score":2,"upvotes":2,"downvotes":0,"published":"2025-01-10T13:44:10.493279Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13818548,"creator_id":9856829,"post_id":17020491,"content":"Round about that much vram. I.e. 90 GB ","removed":false,"published":"2025-01-10T13:54:25.230203Z","deleted":false,"ap_id":"https://feddit.org/comment/4063600","local":false,"path":"0.13818548","distinguished":false,"language_id":37},"creator":{"id":9856829,"name":"Gladaed","banned":false,"published":"2024-06-26T05:42:50.993656Z","actor_id":"https://feddit.org/u/Gladaed","local":false,"deleted":false,"bot_account":false,"instance_id":6177},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13818548,"score":2,"upvotes":2,"downvotes":0,"published":"2025-01-10T13:54:25.230203Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13849830,"creator_id":11040857,"post_id":17020491,"content":"all you need is 2 3090s and some good CPU and lot of ram, then you should be able to get like 10tps","removed":false,"published":"2025-01-11T09:02:27.433002Z","deleted":false,"ap_id":"https://lemmy.zip/comment/15855150","local":false,"path":"0.13849830","distinguished":false,"language_id":37},"creator":{"id":11040857,"name":"red","banned":false,"published":"2023-12-21T13:17:15.931296Z","actor_id":"https://lemmy.zip/u/red","local":false,"deleted":false,"matrix_user_id":"@red3172:matrix.org","bot_account":false,"instance_id":44},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13849830,"score":2,"upvotes":3,"downvotes":1,"published":"2025-01-11T09:02:27.433002Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13838789,"creator_id":7530831,"post_id":17020491,"content":"Huh so basicly sidestepping the gpu issue entirly and essentially just using some other special piece of silicon with fast (but conventional ram). I still dont understand why u cant distribute a large llm over many different processors each holding a section of the parameters in memory.","removed":false,"published":"2025-01-10T17:05:26.293713Z","deleted":false,"ap_id":"https://lemm.ee/comment/17404273","local":false,"path":"0.13820708.13838789","distinguished":false,"language_id":37},"creator":{"id":7530831,"name":"muntedcrocodile","display_name":"🇦🇺𝕄𝕦𝕟𝕥𝕖𝕕𝕔𝕣𝕠𝕔𝕠𝕕𝕚𝕝𝕖","avatar":"https://lemm.ee/pictrs/image/76b7cbf1-38ca-4b77-80a8-b37ffbac5777.png","banned":false,"published":"2024-03-20T09:22:01.255810Z","actor_id":"https://lemm.ee/u/muntedcrocodile","local":false,"deleted":false,"matrix_user_id":"@muntedcrocodile:matrix.org","bot_account":false,"ban_expires":"2024-09-22T17:33:57Z","instance_id":25},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13838789,"score":0,"upvotes":0,"downvotes":0,"published":"2025-01-10T17:05:26.293713Z","child_count":2},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13838790,"creator_id":672357,"post_id":17020491,"content":"Not exactly. Digits still uses a Blackwell GPU, only it uses unified RAM as virtual VRAM instead of actual VRAM. The GPU is probably a down clocked Blackwell. Speculation I've seen is that these are defective and repurposed Blackwells; good for us. By defective I mean they can't run at full speed or are projected to have the cracking die problem, etc.","removed":false,"published":"2025-01-11T03:27:57.231283Z","deleted":false,"ap_id":"https://lemmy.world/comment/14408084","local":false,"path":"0.13820708.13838789.13838790","distinguished":false,"language_id":37},"creator":{"id":672357,"name":"tpWinthropeIII","banned":false,"published":"2023-07-22T19:16:07.253356Z","actor_id":"https://lemmy.world/u/tpWinthropeIII","local":false,"deleted":false,"bot_account":false,"instance_id":24},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13838790,"score":3,"upvotes":3,"downvotes":0,"published":"2025-01-11T03:27:57.231283Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13919777,"creator_id":693814,"post_id":17020491,"content":"> I still dont understand why u cant distribute a large llm over many different processors each holding a section of the parameters in memory.\n\nBecause each weight in a layer influences each weight in the next layer, which means the bandwidth requirements are enormous and regular networking solutions are insufficient for that.","removed":false,"published":"2025-01-13T07:53:30.907768Z","deleted":false,"ap_id":"https://lemmy.world/comment/14443523","local":false,"path":"0.13820708.13838789.13919777","distinguished":false,"language_id":37},"creator":{"id":693814,"name":"breakingcups","banned":false,"published":"2023-07-25T21:15:00.562004Z","actor_id":"https://lemmy.world/u/breakingcups","local":false,"deleted":false,"bot_account":false,"instance_id":24},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13919777,"score":2,"upvotes":2,"downvotes":0,"published":"2025-01-13T07:53:30.907768Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13875445,"creator_id":461,"post_id":17020491,"content":"You can use system ram for when the GPU memory fills up. Alternatively you can run multiple GPUs. ","removed":false,"published":"2025-01-11T18:59:37.645701Z","deleted":false,"ap_id":"https://lemmy.zip/comment/15863565","local":false,"path":"0.13832533.13875445","distinguished":false,"language_id":37},"creator":{"id":461,"name":"possiblylinux127","display_name":"Possibly linux","avatar":"https://lemmy.zip/pictrs/image/7e90c2c8-ff29-4d59-998a-1abfa3551de4.webp","banned":false,"published":"2023-06-26T21:41:18.522794Z","actor_id":"https://lemmy.zip/u/possiblylinux127","bio":"### I know nothing!","local":false,"banner":"https://lemmy.zip/pictrs/image/9d0e54d1-65d6-4a3a-8423-18ad2a9c3cf5.webp","deleted":false,"bot_account":false,"instance_id":44},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13875445,"score":1,"upvotes":1,"downvotes":0,"published":"2025-01-11T18:59:37.645701Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13849773,"creator_id":11040857,"post_id":17020491,"content":"this is useless, llama.cpp already does that airllm does (offloading to CPU) but its actually faster. so just use ollama ","removed":false,"published":"2025-01-11T09:01:08.949165Z","deleted":false,"ap_id":"https://lemmy.zip/comment/15855143","local":false,"path":"0.13834162.13849773","distinguished":false,"language_id":37},"creator":{"id":11040857,"name":"red","banned":false,"published":"2023-12-21T13:17:15.931296Z","actor_id":"https://lemmy.zip/u/red","local":false,"deleted":false,"matrix_user_id":"@red3172:matrix.org","bot_account":false,"instance_id":44},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13849773,"score":1,"upvotes":1,"downvotes":0,"published":"2025-01-11T09:01:08.949165Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13847806,"creator_id":7530831,"post_id":17020491,"content":"That looks like exactly the sort of thing i want. Any existing solution to get it to behave like an ollama instance (i have a bunch of services pointed at an ollama run on docker). ","removed":false,"published":"2025-01-11T00:47:07.306670Z","deleted":false,"ap_id":"https://lemm.ee/comment/17412801","local":false,"path":"0.13834162.13847806","distinguished":false,"language_id":37},"creator":{"id":7530831,"name":"muntedcrocodile","display_name":"🇦🇺𝕄𝕦𝕟𝕥𝕖𝕕𝕔𝕣𝕠𝕔𝕠𝕕𝕚𝕝𝕖","avatar":"https://lemm.ee/pictrs/image/76b7cbf1-38ca-4b77-80a8-b37ffbac5777.png","banned":false,"published":"2024-03-20T09:22:01.255810Z","actor_id":"https://lemm.ee/u/muntedcrocodile","local":false,"deleted":false,"matrix_user_id":"@muntedcrocodile:matrix.org","bot_account":false,"ban_expires":"2024-09-22T17:33:57Z","instance_id":25},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13847806,"score":0,"upvotes":0,"downvotes":0,"published":"2025-01-11T00:47:07.306670Z","child_count":1},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false},{"comment":{"id":13847807,"creator_id":14768598,"post_id":17020491,"content":"You may try [Harbor](https://github.com/av/harbor/wiki/2.2.11-Backend:-AirLLM). The description claims to provide an OpenAI-compatible API.","removed":false,"published":"2025-01-11T08:20:31.054442Z","deleted":false,"ap_id":"https://lemmy.world/comment/14410431","local":false,"path":"0.13834162.13847806.13847807","distinguished":false,"language_id":37},"creator":{"id":14768598,"name":"Sylovik","banned":false,"published":"2025-01-10T23:46:01.269927Z","actor_id":"https://lemmy.world/u/Sylovik","local":false,"deleted":false,"bot_account":false,"instance_id":24},"post":{"id":17020491,"name":"How much gpu do i need to run a 90b model","body":"Do i need industry grade gpu's or can i scrape by getring decent tps with a consumer level gpu.","creator_id":7530831,"community_id":6229,"removed":false,"locked":false,"published":"2025-01-10T13:32:43.048107Z","deleted":false,"nsfw":false,"ap_id":"https://lemm.ee/post/52232292","local":false,"language_id":37,"featured_community":false,"featured_local":false},"community":{"id":6229,"name":"localllama","title":"LocalLLaMA","description":"Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.\n\nGet support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.\n\nAs ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.\n\nRules:\n\nRule 1 -\nNo harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.\n\nRule 2 -\nNo comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.\n\nRule 3 -\nNo comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as \"llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since .\n\nRule 4 -\nNo implying that models are devoid of purpose or potential for enriching peoples lives.","removed":false,"published":"2023-06-08T18:21:54.578976Z","updated":"2025-06-09T20:51:11.460711Z","deleted":false,"nsfw":false,"actor_id":"https://sh.itjust.works/c/localllama","local":false,"icon":"https://sh.itjust.works/pictrs/image/8e07ec26-c7e5-4411-a4d5-c0c7a292b773.gif","banner":"https://sh.itjust.works/pictrs/image/5d07b398-7675-4862-8277-5906400b666d.png","hidden":false,"posting_restricted_to_mods":false,"instance_id":32,"visibility":"Public"},"counts":{"comment_id":13847807,"score":2,"upvotes":2,"downvotes":0,"published":"2025-01-11T08:20:31.054442Z","child_count":0},"creator_banned_from_community":false,"banned_from_community":false,"creator_is_moderator":false,"creator_is_admin":false,"subscribed":"NotSubscribed","saved":false,"creator_blocked":false}]}