not_IO@lemmy.blahaj.zone to Programmer Humor@programming.devEnglish · 2 days agomight be a form of Jevons Paradoxlemmy.blahaj.zoneimagemessage-square256fedilinkarrow-up11.54Karrow-down111file-text
arrow-up11.53Karrow-down1imagemight be a form of Jevons Paradoxlemmy.blahaj.zonenot_IO@lemmy.blahaj.zone to Programmer Humor@programming.devEnglish · 2 days agomessage-square256fedilinkfile-text
minus-squareVlyn@lemmy.ziplinkfedilinkEnglisharrow-up16arrow-down1·1 day agoWith 32 and 64 GB systems I’ve never run out of RAM, so the RAM isn’t the issue at all. Optimization just sucks.
minus-squareSamskara@sh.itjust.workslinkfedilinkarrow-up1·10 hours agoHave you ever tried running a decent sized LLM locally?
minus-squareVlyn@lemmy.ziplinkfedilinkEnglisharrow-up1·10 hours agoDecent sized for what? Creative writing and roleplay? Plenty, but I try to fit it into my 16 GB VRAM as otherwise it’s too slow for my liking. Coding/complex tasks? No, that would need 128GB and upwards and it would still be awfully slow. Except you use a Mac with unified memory. For image and video generation you’d want to fit it into GPU VRAM again, system RAM would be way too slow.
minus-squareSamskara@sh.itjust.workslinkfedilinkarrow-up1·9 hours agoI use a Mac with unified memory, so that distinction slipped my mind.
With 32 and 64 GB systems I’ve never run out of RAM, so the RAM isn’t the issue at all.
Optimization just sucks.
Have you ever tried running a decent sized LLM locally?
Decent sized for what?
Creative writing and roleplay? Plenty, but I try to fit it into my 16 GB VRAM as otherwise it’s too slow for my liking.
Coding/complex tasks? No, that would need 128GB and upwards and it would still be awfully slow. Except you use a Mac with unified memory.
For image and video generation you’d want to fit it into GPU VRAM again, system RAM would be way too slow.
I use a Mac with unified memory, so that distinction slipped my mind.