Large language models have the context window of a distracted child, hallucinate like in-laws, and can confidently tell you a lie like a true friend. They are almost indistinguishable from real humans.
In this tutorial, we will learn how to hone our conversational skills using LM Studio, a desktop application for running local LLMs on your computer.
Note: I’m using AMD 5600G APU, but most of what you see here also applies to discrete GPUs. Whenever something is APU specific, I will mark it as such. I use containers with ROCm, but Nvidia CUDA users should also find this guide helpful.
First, for LM Studio we need a container with installed ROCm (no need for PyTorch). It’s around 30 GB in size, so don’t be surprised. Tutorial for that is here. This is a GUI app, so we should use a GUI profile when creating it.
Let’s log in to the container using the default ubuntu user:
incus exec <container_name> -- sudo --login --user ubuntu
We need to install some required packages before we download LM Studio and make its AppImage file executable (check the homepage to see if a newer version is available):
sudo apt install libfuse2 libnss3 libatk1.0-0 libatk-bridge2.0-0 libcups2 libgtk-3-0 libgbm1 libasound2 libclblast1 libgl1 fonts-noto
wget https://releases.lmstudio.ai/linux/0.2.19/beta/LM_Studio-0.2.19.AppImage
chmod u+x LM_Studio-0.2.19.AppImage
Now we can start LM Studio with this command:
./LM_Studio-0.2.19.AppImage
Unfortunately for APU owners, the LD_PRELOAD trick doesn’t work and we need to assign fixed amount of VRAM to the GPU. It should be approximately at least 0,5 GiB more than the size of the downloaded model you will use. For more information see ROCm tutorial linked before. Discrete GPU owners don’t have to worry about it.
To learn more about LM Studio and its many features, check out its homepage and YouTube. For example watch this video by Matthew Berman, which shows how to enable GPU acceleration and move model layers to VRAM for faster inference.
That’s all. If you have any questions, feel free to ask. Feedback, corrections and tips are greatly appreciated.