I have a GNU/Linux phone I carry in my other pocket. Here are the biggest issues I can see:

1. Driver support for components in the mainline kernel (lets you install any distro and things like camera, Bluetooth just work)
2. Power management; turns out it is a hard technical problem to have your phone suspend to save energy, while being awake enough to know what and when to turn back on to receive chats/calls, playback music, etc
3. Cameras have a lot of stuff beyond drivers happening behind the scenes these days in software that would need to be developed, especially given it is a big reason people choose their phones for
4. Phone certification is tough, this has stopped even companies like Fairphone from shipping their devices worldwide, I imagine even harder for a device like the Purism Librem 5 where you can literally upgrade Wi-Fi, BT, and cellular generations like a gameboy cartridge
5. App ecosystems take a while to build up, it is a chicken/egg scenario. I think things are in a useable state for all the default apps an iPhone has, but if you want Uber, Uber Eats, you either have to draw even more power essentially running Android via Waydroid, or use a typically more janky web app that may be missing some features