Run your own AI with Ollama: what to expect on CPU

Ollama is a tool that runs open AI models on your own server. Seturack servers have no graphics card (GPU), so models run on the CPU. That means small models (1-4 billion parameters) are usable for chat and short tasks, 7-8 billion parameter models run but slowly, and larger ones aren't practical. A model's file size is the lower bound for the RAM you should set aside for it.

The speed comments in this article come from general experience, not from measurements on Seturack servers. The model sizes come from Ollama's own library.

The model has to fit in RAM

The file size is close to the part of the model that gets loaded into memory. On top of that come the context memory, which depends on conversation length, Ollama's own overhead and the operating system. So RAM equal to the file size isn't enough, and you should leave headroom. There is no exact multiplier, but measuring is easy: run free -h after the model has loaded. The library is updated constantly, so check Ollama's model library for the current list.

What size of server?

This table is a rule of thumb made by adding headroom to the file sizes above. It is not an official recommendation. If RAM runs out, the system starts using the disk as memory (swap) and speed drops sharply, or the process is killed. Before trying a large model, it is safer to scale the server up first.

What to expect on a CPU

• Text generation is limited largely by memory bandwidth. The bigger the model, the slower the answer streams.
• Compared with a graphics card, a CPU is noticeably slower. The answer can stream slower than you read.
• Processing a long text lengthens the wait for the first answer. On the first request the model is loaded from disk into memory, so it arrives later still.
• Several users at the same time split the speed between them.

What works well: short summaries, classification, email drafts, a personal assistant, experiments and development, small automations. What struggles: a live service for many users, large models, real-time replies, heavy code generation.

Setup (Ubuntu)

1. Run the official one-line install script: curl -fsSL https://ollama.com/install.sh | sh It is good practice to download and read the script first.
2. Start with a small model: ollama run llama3.2:3b
3. See where the model is running: ollama ps If the Processor column says only CPU, the model sits entirely in system memory.
4. Watch your disk space. On Linux the models live under /usr/share/ollama/.ollama/models and each takes gigabytes. Use ollama list to list them and ollama rm to delete one.

If you prefer Docker, the official command is docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. We suggest binding the port to the local address only: -p 127.0.0.1:11434:11434.

Which model should you pick?

Start with a 3 billion parameter model and try it on your own work. Small models are usually weaker in Turkish than in English, so test them with your own texts. If the answers fall short, step up one size and check with free -h that the RAM is enough.

The sizes in the table are for the library's default tags. Models usually come compressed (quantized) to around 4 bits. At 16-bit precision, 8 billion parameters take about 16 GB, and the compression brings that down to around 5 GB, as in the table. In return, a small amount of quality is given up.

Using it from applications

Ollama exposes a local API. From inside the server you can try it like this:

curl http://127.0.0.1:11434/api/generate -d '{"model":"llama3.2:3b","prompt":"Hello, introduce yourself.","stream":false}'

The answer comes back as JSON. Automation tools and your own applications can connect to the same API.

Managing memory

• Ollama keeps a model in memory for 5 minutes after the last request by default. It is normal for RAM to look full during that time. ollama stop MODEL_NAME unloads the model right away.
• OLLAMA_MAX_LOADED_MODELS limits how many models can stay loaded at the same time. The default on a CPU is 3. If RAM is tight, set it to 1, otherwise calling different models one after another leaves them all in memory.
• OLLAMA_NUM_PARALLEL is the number of parallel requests per model and defaults to 1. The more you raise it, the more RAM it needs.

Security

• By default Ollama listens only on port 11434 of 127.0.0.1. If you open it up with OLLAMA_HOST, anyone who knows the address can use your model, because the API has no authentication step by default. Don't expose it.
• An SSH tunnel is enough for remote use: ssh -L 11434:127.0.0.1:11434 user@SERVER_IP
• Docker skips ufw rules for the ports it publishes. That is why you bind the port to the local address only, as above.
• To chat from a browser you can install an interface such as Open WebUI. Run it behind HTTPS and protect it with user accounts.

Data and licenses

When you run a model on your own server, your conversations stay on your server. Ollama also offers models that run in the cloud. They don't run on your server and are outside the scope of this article. Every model has its own license, so read the license on the model page before commercial use.

Scaling the server

RAM needs change as you change models. If you start with a 3 billion parameter model and move up to an 8 billion one, you scale the server up in the panel. Scaling up doesn't delete the server, and your IP address and data stay the same. When you scale down, the disk doesn't shrink, and since models can fill a disk, factor the disk in as well.

You can work out your price on the pricing page.

Ready-made on Seturack: Ollama, AnythingLLM, Qdrant.