Configuring Ollama, Deepseek, and AnythingLLM on Linux

Target¶

By combining Ollama, Deepseek, and AnythingLLM, you can create a powerful local knowledge base that allows for efficient, context-aware querying and retrieval of information. While configuring Ollama, Deepseek, and AnythingLLM on Windows is straightforward, and numerous tutorials are available, this guide will focus on setting them up on Ubuntu.

Ollama + Deepseek Configuration¶

Step 1: Install Ollama¶

Method 1: Official script¶

curl -fsSL https://ollama.com/install.sh | sh

# Verification of successful installation
systemctl status ollama
ollama --version

The installation process on Ubuntu is designed to automatically configure everything for you. However, due to potential network-related issues, this automatic setup may sometimes fail to complete successfully.

Method 2: Manual Installation¶

Download and Extract the Package

curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama-linux-amd64.tgz
sudo tar -C /usr -xzf ollama-linux-amd64.tgz

Start Ollama Launch Ollama with the following command:
```
ollama serve
```
Verify Ollama is Running Open another terminal and check if Ollama is running with:
```
ollama -v
```

If the version number is displayed, Ollama is up and running successfully.

Step 2: Adding Ollama as a Startup Service¶

To ensure Ollama starts automatically upon system boot, follow these steps:

Create a User and Group for Ollama First, create a dedicated user and group for Ollama:

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)

Create a Systemd Service File Next, create a service file for Ollama at /etc/systemd/system/ollama.service:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=$PATH"

[Install]
WantedBy=default.target

Reload Systemd and Enable the Service Reload the systemd configuration and enable the Ollama service to start on boot:
```
sudo systemctl daemon-reload
sudo systemctl enable ollama
```
Start Ollama and Verify Finally, start the Ollama service and check if it's running properly:
```
sudo systemctl start ollama
sudo systemctl status ollama
```

If everything is set up correctly, Ollama will now start automatically whenever your system boots up.

Step 3: Configure Ollama for remote access¶

By default, Ollama is only accessible locally at http://127.0.0.1:11434. To enable external access, you need to modify its configuration file.

sudo nano /etc/systemd/system/ollama.service

Add the following content to the file:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_ORIGINS=*"
Environment="OLLAMA_MODELS=/data/ollama/.ollama/models"
ExecStart=/bin/ollama serve
User=root
Group=root
Restart=always
RestartSec=3

[Install]
WantedBy=default.target

After saving the changes, reload the systemd configuration and enable the service with the following commands:

sudo systemctl daemon-reload
sudo systemctl enable ollama

This configuration allows Ollama to accept connections from external sources by binding to 0.0.0.0:11434, which means that other computers on the same network can access it using the IP:11434 (e.g.: 192.168.5.190:11434).

# stop it
systemctl stop ollama.service

# disable it if you want
systemctl disable ollama.service

# confirm its status
systemctl status ollama.service

# e.g.: Specify to use the first two GPUs
export CUDA_VISIBLE_DEVICES=0,1

Step 4: Download Deepseek¶

Before deploying Deepseek, it's important to determine which version you need. Once you’ve identified the version, visit the Ollama website to find the corresponding download link.

In this guide, we will deploy the DeepSeek-R1-Distill-Qwen-7B version.

ollama run deepseek-r1:7b

Now, Ollama + Deepseek has been successfully deployed.

Step 5: Download Embedding Model¶

Next, we need to install the vector model and database, here we choose nomic-embed-text. Without this, the AI won't be able to embed our uploaded file into the embedding database. Search for nomic-embed-text on Ollama website. This model can convert text content into vector data, and the website provides an introduction to the model.

To install the model, you can run the following command in the terminal:

ollama pull nomic-embed-text

(Optional) Open Webui¶

Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution.

The intallation you can ref to another Wiki.

(Optional) Uninstalling Ollama¶

Remove the Ollama Service Stop and disable the Ollama service:

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm /etc/systemd/system/ollama.service

Remove the Ollama Binary Delete the Ollama binary from the system (usually found in /usr/bin, /usr/local/bin, or /bin):
```
sudo rm $(which ollama)
```
Delete the Model Files and Ollama User/Group Remove the downloaded models and the Ollama service user and group:
```
sudo rm -r /usr/share/ollama
sudo userdel ollama
sudo groupdel ollama
```
Remove Installed Libraries
Finally, remove any associated libraries installed by Ollama:
```
sudo rm -rf /usr/local/lib/ollama
```

AnythinLLM Configuration¶

Here, you can choose to run the Dockerized version of AnythingLLM or build it locally from source, with the former being recommended:

docker pull mintplexlabs/anythingllm

Then you can mount the storage locally and run AnythingLLM in Docker:

export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm

Important!

Here are some considerations. These can be ignored for now, but they should be kept in mind for future use.

If you are running another service on localhost like Chroma, LocalAi, or LMStudio you will need to use http://host.docker.internal:xxxx to access the service from within the docker container using AnythingLLM as localhost:xxxx will not resolve for the host system.
Requires Docker v18.03+ on Win/Mac and 20.10+ on Linux/Ubuntu for host.docker.internal to resolve!
Linux: add --add-host=host.docker.internal:host-gateway to docker run command for this to resolve.
eg: Chroma host URL running on localhost:8000 on host machine needs to be http://host.docker.internal:8000 when used in AnythingLLM."

Go to http://localhost:3001 or http://your_ip:3001 and you are now using AnythingLLM! All your data and progress will persist between container rebuilds or pulls from Docker Hub.

How to Use AnythingLLM¶

Not only can you deploy your own model locally, but you can also use APIs from other model providers to build a personal knowledge base with AnythingLLM.

Since there are many tutorials available on how to use AnythingLLM, we will only provide some key tips here.

Remember to choose Embedder provider in Setting - Embedder perference.
If you want your AI to only respond when it can retrieve information from the database, rather than providing random answers, you need to select "Query" instead of "Chat" in the workspace settings under "Chat Mode."

That's all. Enjoy your AI journey!