FYI: local inference on ROCm
Moondream has provided what seems to be a really good local inference engine for NVIDIA/MPS users with kestrel. Unfortunately, there's no ROCm support.
I spent a couple days with LLMs and now I have a version of the kestrel engine running pretty fast on my R9700 with TheRock ROCm nightlies.
You need the minimal fork at https://github.com/sjuxax/kestrel/tree/rocm-support and the kestrel-kernels+ROCm package at https://github.com/sjuxax/moondream31/tree/main/kestrel-rocm. Installing those should get local things running fast. Then you can use kestrel as a backend on AMD GPUs.
I'll soon be releasing a vision-indexer project that makes use of moondream3.1-9B-A2B via your choice of cloud, transformers, or kestrel.
Just posting this as an FYI for interested parties.