Big tech boss tells delegates at Davos that broader global use is essential if technology is to deliver lasting growth
Nadella maybe knows a lot more than any of us about LLMs/GenAI tech, but one doesn’t need to know anything about LLMs (or even technology) to know that an oligarch like Nadella cannot be trusted (in any context).
I’m kind of more-sympathetic to Microsoft than to some of the other companies involved.
Microsoft is trying to leverage the Windows platform that they control to do local LLM use. I’m not at all sure that there’s actually enough memory out there to do that, or that it’s cost-effective to put a ton of memory and compute capacity in everyone’s home rather than time-sharing hardware in datacenters. Nor am I sold that laptops — which many “Copilot PCs” are — are a fantastic place to be doing a lot of heavyweight parallel compute.
But…from a privacy standpoint, I kind of would like local LLMs to be at least available, even if they aren’t as affordable as cloud-based stuff. And at least Microsoft is at least supporting that route. A lot of companies are going to be oriented towards just doing AI stuff in the cloud.
Is that true? I haven’t heard MS say anything about enabling local LLMs. Genuinely curious and would like to know more.
That’s why they have the “Copilot PC” hardware requirement, because they’re using an NPU on the local machine.
searches
https://learn.microsoft.com/en-us/windows/ai/npu-devices/
Copilot+ PCs are a new class of Windows 11 hardware powered by a high-performance Neural Processing Unit (NPU) — a specialized computer chip for AI-intensive processes like real-time translations and image generation—that can perform more than 40 trillion operations per second (TOPS).
It’s not…terribly beefy. Like, I have a Framework Desktop with an APU and 128GB of memory that schlorps down 120W or something, substantially outdoes what you’re going to do on a laptop. And that in turn is weaker computationally than something like the big Nvidia hardware going into datacenters.
But it is doing local computation.
Isn’t that the whole shtick of the AI PCs no one wanted? Like, isn’t there some kind of non-GPU co-processor that runs the local models more efficiently than the CPU?
I don’t really want local LLMs but I won’t begrudge those who do. Still, I wouldn’t trust any proprietary system’s local LLMs to not feed back personal info for “product improvement” (which for AI is your data to train on).
NPU neural processing unit
I wouldn’t trust a local LLM solution from a large American company. Not saying that they would try to “pull a quick one”, but they are unreliable and corrupt.
Microsoft wants developers to have local access to models but end users are 100% corralled into OneDrive and Copilot. I’m not sympathetic to them at all.
If Microsoft cared about privacy then they wouldn’t have made windows practically spyware. Even if they install AI locally in the OS, it’s still proprietary software that constantly sends data back to the mothership, consuming your electricity and RAM to do so. Linux has so many options, there’s really no reason not to switch.
Small LLMs already exist for local self-hosting, and there are open-source options which won’t steal your data and turn you into a product.
https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/
Bear in mind that the number of parameters your system can handle is limited by how much memory is available, and using a quantized version can increase the number of parameters you can handle with the same amount of memory.
Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up. But I’m no expert, so do some research and calculate for yourself what your system can handle.
Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up.
Eh…I don’t know if you’d call it “really serious hardware”, but when I picked up my 128GB Framework Desktop, it was $2k (without storage), and that box is often described as being aimed at the hobbyist AI market. That’s pricier than most video cards, but an AMD Radeon RX 7900 XTX GPU was north of $1k, an NVidia RTX 4090 was about $2k, and it looks like the NVidia RTX 5090 is presently something over $3k (and rising) on EBay, well over MSRP. None of those GPUs are dedicated hardware aimed at doing AI compute, just high-end cards aimed at playing games that people have used to do AI stuff on.
I think that the largest LLM I’ve run on the Framework Desktop was a 106 billion parameter GLM model at Q4_K_M quantization. It was certainly usable, and I wasn’t trying to squeeze as large a model as possible on the thing. I’m sure that one could run substantially-larger models.
EDIT: Also, some of the newer LLMs are MoE-based, and for those, it’s not necessarily unreasonable to offload expert layers to main memory. If a particular expert isn’t being used, it doesn’t need to live in VRAM. That relaxes some of the hardware requirements, from needing a ton of VRAM to just needing a fair bit of VRAM plus a ton of main memory.
See, you have more experience in the matter than I do, hence the caveat that I’m not an expert. Thanks for sharing your experience.
Then again, I’d consider 128GB of memory to be fairly serious hardware, but if that’s common among hobbyists then I stand corrected. I was operating on the assumption that 64GB of RAM is already a lot
All in all, 106 billion parameters on 128GB of memory with quantization doesn’t surprise me all that much. But again, I’m just going off of the vague notions I’ve gathered from reading about it.
The focus of my original comment was more on the fact that self-hosting is an option, I wasn’t trying to be too precise with the specs. My bad if it came off that way
They’re trying to leverage their windows platform to seek rent (sell premium cloud services like LLM access) for shit people don’t even want because they aren’t satisfied making very respectable money on licenses.
Power corrupts, we’ve know this since the dawn of time
So somebody or a few somebodies is/are regretting a few investments right about now I imagine
Not much of a “boom” then is it?
If it doesn’t have wide adoption, it isn’t a boom. It’s just sparkling speculation.
I didn’t read anything and I don’t know who this man is but the picture is just… I can’t… He looks like a baby with the headband eye glasses. That expression… He’s enjoying his puffed rice cereal bites while signalling that he wants milk.
🤞
If he wanted people to like it then he should have made it do things people want it to do.
It is the new metaverse.
Hell I’d almost settle for just “making it work”. No disclaimers, no bullshitting. Computers should be optimized and accurate. AI is neither.
Ai does work great, at some stuff. The problem is pushing it into places it doesn’t belong.
It’s a good grammar and spell check. It helps me get a lot of English looking more natural.
It’s also great for troubleshooting consumer electronics.
It’s far better at search than google.
Even then it can only help, not replace folks or complete tasks.
Fundamentally due to it’s design, LLMs are digital duct tape.
The entire history of computer science has been making compromises between efficient machine code and human readable language. LLM’s solve this in a beautifully janky way, like duct tape.
But it’s ultimately still a compromise, you’ll never get machine accuracy from an LLM because it’s sole purpose is to fulfill the “human readable” part of that deal. So it’s applications are revolutionary in the same way as “how did you put together this car engine with only duct tape?” kind of way.
Ai does work great, at some stuff. The problem is pushing it into places it doesn’t belong.
I can generally agree with this, but I think a lot of people overestimate where it DOES belong.
For example, you’ll see a lot of tech bros talking about how AI is great at replacing artists, but a bunch of artists who know their shit can show you every possible way this just isn’t as good as human-made works, but those same artists might say that AI is still incredibly good at programming… because they’re not programmers.
It’s a good grammar and spell check.
Totally. After all, it’s built on a similar foundation to existing spellcheck systems: predict the likely next word. It’s good as a thesaurus too. (e.g. “what’s that word for someone who’s full of themselves, self-centered, and boastful?” and it’ll spit out “egocentric”)
It’s also great for troubleshooting consumer electronics.
Only for very basic, common, or broad issues. LLMs generally sound very confident, and provide answers regardless of if there’s actually a strong source. Plus, they tend to ignore the context of where they source information from.
For example, if I ask it how to change X setting in a niche piece of software, it will often just make up an entire name for a setting or menu, because it just… has to say something that sounds right, since the previous text was “Absolutely! You can fix x by…” and it’s just predicting the most likely term, which isn’t going to be “wait, nevermind, sorry I don’t think that’s a setting that even exists!”, but a made up name instead. (this is one of the reasons why “thinking” versions of models perform better, because the internal dialogue can reasonably include a correction, retraction, or self-questioning)
It will pull from names and text of entirely different posts that happened to display on the page it scraped, make up words that never appeared on any page, or infer a meaning that doesn’t actually exist.
But if you have a more common question like “my computer is having x issues, what could this be?” it’ll probably give you a good broad list, and if you narrow it down to RAM issues, it’ll probably recommend you MemTest86.
It’s far better at search than google.
As someone else already mentioned, this is mostly just because Google deliberately made search worse. Other search engines that haven’t enshittified, like the one I use (Kagi), tend to give much better results than Google, without you needing to use AI features at all.
On that note though, there is actually an interesting trend where AI models tend to pick lower-ranked, less SEO-optimized pages as sources, but still tend to pick ones with better information on average. It’s quite interesting, though I’m no expert on that in particular and couldn’t really tell you why other than “it can probably interpret the context of a page better than an algorithm made to do it as quickly as possible, at scale, returning 30 results in 0.3 seconds, given all the extra computing power and time.”
Even then it can only help, not replace folks or complete tasks.
Agreed.
I find that people only think its good when using it for something they dont already know, so then they believe everything it says. Catch 22. When they use it for something they already know, its very easy to see how it lies and makes up shit because its a markov chain on steroids and is not impressive in any way. Those billions could have housed and fed every human in a starving country but instead we have the digital equivalent of funko pop minions.
I also find in daily life those who use it and brag about it are 95% of the time the most unintelligent people i know.
Note this doesnt apply to machine learning.
It only looks good in comparison to Google search because they trashed Google search.
Which of course, Google did just so you’d have to search more, so you’d see more ads.
We’ll have to agree to disagree. To go through your points, spell check I don’t find particularly impressive. That was solved previously without requiring the power demands of a small town. Grammer, maybe - but in my experience my “LLM powered” keyboard’s suggestions are still worse than old T9 input.
I’ve had no luck troubleshooting anything with AI. It’s often trained on old data, tries to instruct you to change settings that don’t exist, or dreams up controls that might appear on “similar” hardware. Sure you can perhaps infer a solution, maybe, but it’s rarely correct at first response. It’ll happily run you through steps that are inconsequential to fixing a problem.
Finally, it might be better than indexed search NOW - but mostly because LLMs wrecked that too. I used to be able to use a couple search operators and get directly to the information I needed - now search is reduced to shifting through slop SEO sites.
And it does all this half assing while using enough power to justify dedicated nuclear reactors. I cant help but feel we’ve regressed on so many fronts.
Don’t threaten us with a good time
So you’re saying that in order to win we must do nothing? Alright I’m pretty good at that.
SELL SELL SELL!
We’ve already adopted AI. Most people keep a Chatgpt bookmark or a permanent browser tab. Everything else is just slop, extra risk and privacy invasion.
“Most people?” Sounds like you’ve gotten yourself into a filter bubble, my friend. Only two people I know use it regularly, but adjusting for my own filter bubble, I think most people have played with it a couple of times, found it wanting, and maybe use Gemini in the Google search results when it comes up.
Oh no…anyway
Translation: We need more suckers now!
“The only way to save this shrinking ice floe is for more people to jump on”
