I’d like to self host a large language model, LLM.

I don’t mind if I need a GPU and all that, at least it will be running on my own hardware, and probably even cheaper than the $20 everyone is charging per month.

What LLMs are you self hosting? And what are you using to do it?

  • InverseParallax@lemmy.world
    link
    fedilink
    English
    arrow-up
    6
    ·
    edit-2
    9 days ago

    Ollama, llama3.2, deepcode and a bunch of others.

    Using a GPU but man they’re picky, they mostly want Nvidia gpus.

    Do NOT be afraid to run on the cpu. It’s slow, but for 1 user it’s actually mostly fine.