I was interested in these “light” phones for a bit, but they seem a bit gimmicky and expensive. I understand not having a browser on purpose, but for communication, none of them (AFAIK) support Matrix or even XMPP (even some old feature phones had a Java Jabber client). Punkt MP02 supports Signal though.
Android phone with custom ROM (Lineage, /e/, Graphene, DivestOS…) is a possibility, and would be usable until hardware is incompatible with the phone/wifi networks.
If you have a patience of a saint, PinePhone and Librem5 are Linux phones, both in fairly early stages.
Other than with language models, this has already happened: Take a look at apps such as Merlin Bird ID (identifies birds fairly well by sound and somewhat okay visually), WhoBird (identifies birds by sound, ) Seek (visually identifies plants, fungi, insects, and animals). All of them work offline. IMO these are much better uses of ML than spammer-friendly text generation.