Instructions to use RedHatAI/granite-3.1-2b-base-quantized.w4a16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/granite-3.1-2b-base-quantized.w4a16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/granite-3.1-2b-base-quantized.w4a16")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/granite-3.1-2b-base-quantized.w4a16")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/granite-3.1-2b-base-quantized.w4a16", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/granite-3.1-2b-base-quantized.w4a16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/granite-3.1-2b-base-quantized.w4a16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-base-quantized.w4a16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RedHatAI/granite-3.1-2b-base-quantized.w4a16

SGLang

How to use RedHatAI/granite-3.1-2b-base-quantized.w4a16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/granite-3.1-2b-base-quantized.w4a16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-base-quantized.w4a16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/granite-3.1-2b-base-quantized.w4a16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/granite-3.1-2b-base-quantized.w4a16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RedHatAI/granite-3.1-2b-base-quantized.w4a16 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/granite-3.1-2b-base-quantized.w4a16
```

granite-3.1-2b-base-quantized.w4a16

Commit History

Update README.md

22b72eb
verified

shubhrapandit commited on Feb 28, 2025

Update README.md

afe8cc6
verified

nm-research commited on Jan 30, 2025

Update README.md

7b37ead
verified

nm-research commited on Jan 30, 2025

Update README.md

d7f242c
verified

nm-research commited on Jan 30, 2025

Update README.md

a807cbb
verified

nm-research commited on Jan 25, 2025

Update README.md

c677fdb
verified

nm-research commited on Jan 24, 2025

Update README.md

f9b4aef
verified

nm-research commited on Jan 20, 2025

Update README.md

c8ba441
verified

nm-research commited on Jan 17, 2025

Update README.md

8edc04a
verified

nm-research commited on Jan 17, 2025

Create README.md

f4730a4
verified

nm-research commited on Jan 17, 2025

Upload model files

395dc80

Shubhra Pandit commited on Jan 17, 2025

initial commit

fd34d5c
verified

nm-research commited on Jan 16, 2025

Commit History

Update README.md 22b72eb verified

Update README.md afe8cc6 verified

Update README.md 7b37ead verified

Update README.md d7f242c verified

Update README.md a807cbb verified

Update README.md c677fdb verified

Update README.md f9b4aef verified

Update README.md c8ba441 verified

Update README.md 8edc04a verified

Create README.md f4730a4 verified

Upload model files 395dc80

initial commit fd34d5c verified

Update README.md

22b72eb
verified

Update README.md

afe8cc6
verified

Update README.md

7b37ead
verified

Update README.md

d7f242c
verified

Update README.md

a807cbb
verified

Update README.md

c677fdb
verified

Update README.md

f9b4aef
verified

Update README.md

c8ba441
verified

Update README.md

8edc04a
verified

Create README.md

f4730a4
verified

Upload model files

395dc80

initial commit

fd34d5c
verified