Info
Recently bought a Intel Arc B50 and decided to push it to the limits by trying to run some AI models on it
Setup
If you are using proxmox make sure you passthrough the GPU you wish to run AI on
1
2
3
4
5
6
7
8
9
10
11
|
features: fuse=1,nesting=1
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/card0 dev/dri/card0 none bind,optional,create=file
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.idmap: u 0 100000 262144
lxc.idmap: g 0 100000 44
lxc.idmap: g 44 44 1
lxc.idmap: g 45 100045 948
lxc.idmap: g 993 993 1
lxc.idmap: g 994 100994 261150
|
for rootless setup update
/etc/subgid
1
2
3
|
root:100000:262144
root:44:1
root:993:1
|
For simplicity and ease of upgrade I run this inside a docker compose container to keep it simple.
First lets prep the environment with the neccesary packages
1
2
|
apt update && apt install curl intel-gpu-tools
curl -L https://get.docker.com |sh
|
Here is the docker-compose.yml I used
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
services:
llama:
image: ghcr.io/ggml-org/llama.cpp:server-intel
container_name: llama
restart: unless-stopped
ports:
- 11434:8080
volumes:
- ./models:/models
devices:
- /dev/dri/:/dev/dri/
group_add:
- "993" # render
- "44" # video
environment:
# Hardware Acceleration
SYCL_DEVICE_FILTER: "level_zero:gpu"
ONEAPI_DEVICE_SELECTOR: "level_zero:gpu"
SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS: "1"
SYCL_PI_LEVEL_ZERO_USE_COPY_ENGINE: "1"
# Performance Tuning B50
GGML_SYCL_FORCE_MMQ: "1"
GGML_SYCL_F16_AS_F32: "0"
LLAMA_ARG_FLASH_ATTN: "on"
LLAMA_ARG_CACHE_TYPE_K: "q4_0"
LLAMA_ARG_CACHE_TYPE_V: "q4_0"
LLAMA_ARG_BATCH: 2048
LLAMA_ARG_UBATCH: 512
LLAMA_ARG_N_PARALLEL: "1"
# LLAMA_ARG_CTX_SIZE: "8000"
LLAMA_ARG_CACHE_RAM: 0
LLAMA_ARG_PORT: 8080
LLAMA_ARG_MODEL: /models/gemma-4-12b-it-Q4_K_M.gguf
LLAMA_ARG_MMAP: false
LLAMA_ARG_N_GPU_LAYERS: "99"
|
Before we start the container make sure you download a model you with to run.
llama.cpp only supports “GGUF” models
Download models uwint “wget” from https://huggingface.co/models
Once you obtained a model, update teh LLAMA_ARG_MODEL to point to your downloaded file.
then run
once it is running go to page http://your_ip:11434 try a prompt.
You can monitor your GPU using gputop