Optional on-device LLM
Run the LLM enhancement pass locally via llama.cpp. No cloud, no key, no per-token cost. Pick a model to match your GPU memory.
By default the rewrite pass uses a cloud provider, or is Off. Optionally, Dimmy can run it on your own machine through llama.cpp. No key, no upload, no per-token cost. The catalogue is 15 GGUF models and the default is Phi-4 Mini Q4, 2.5 GB, 3.8B params.
Turn it on
- Open Settings → Output. On Windows the card is LLM MODE, with Local and Cloud radios. On macOS it is LLM provider → Mode, with On device and Cloud.
- Pick Local, or On device.
- Choose a model from the list and click Download. Files stay on disk until you remove them, and a partial download resumes where it stopped.
- Run a dictation. The pill shows the style icon while the model works on the text.
Every row in the picker tells you two things: a green check when the file is already on disk, and a coloured dot for whether it fits the memory Dimmy can see. The recap picker uses the same rows, so you can run a different local model for meeting recaps than for dictation. See Meeting recaps.
The models
Language matters more than size here
The -it in the file names means instruction tuned, not Italian. One model covers every language, which is why an Italian meeting comes back recapped in Italian without being asked. Two measured exceptions are worth knowing. TranslateGemma 4B came out best of every local model on the rewrite styles, with zero wrong-language and zero unchanged answers across 48 trials, where Gemma 4 E2B QAT handed the input straight back on Professional and Imbruttito. Qwen 3 4B extracts the most detail of any of them and took 108 seconds where Gemma 4 E2B took 44 on the same meeting, but it tends to answer in English even when you spoke Italian.
How much memory it really takes
- Dimmy budgets roughly the file size times 1.5, plus about 1 GB. That is deliberately generous: a 4 GB model measured 5.9 GB resident on a Mac.
- If loading the LLM would push past a third of your machine memory, Dimmy unloads the speech model first and brings it back afterwards.
- A model left idle for 5 minutes is unloaded, so the memory comes back to you between dictations.
- If the GPU has no room, Dimmy says exactly that and points you at a smaller model. It is not reported as a network or HTTP error.
Local recaps
A local model can write meeting recaps too, and it streams into the meeting window token by token rather than appearing all at once. The whole transcript stays on the machine. Local recaps are capped at 4096 tokens of output, which is plenty for a summary and not meant for a transcript dump.
Where cloud is still better
On-device is private and free, but it is not the equal of the cloud flagships. For Correct and Summarize the small models are fine. For Prompt, Elaborate and long recaps, GPT-5.6 Sol or Claude Opus 5 are noticeably better. The 12B and 26B builds close part of that gap and charge you memory and speed for it.