Instructions to use Groq/Llama-3-Groq-8B-Tool-Use with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Groq/Llama-3-Groq-8B-Tool-Use with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Groq/Llama-3-Groq-8B-Tool-Use")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Groq/Llama-3-Groq-8B-Tool-Use")
model = AutoModelForCausalLM.from_pretrained("Groq/Llama-3-Groq-8B-Tool-Use", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Groq/Llama-3-Groq-8B-Tool-Use with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Groq/Llama-3-Groq-8B-Tool-Use"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Groq/Llama-3-Groq-8B-Tool-Use",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Groq/Llama-3-Groq-8B-Tool-Use

SGLang

How to use Groq/Llama-3-Groq-8B-Tool-Use with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Groq/Llama-3-Groq-8B-Tool-Use" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Groq/Llama-3-Groq-8B-Tool-Use",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Groq/Llama-3-Groq-8B-Tool-Use" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Groq/Llama-3-Groq-8B-Tool-Use",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Groq/Llama-3-Groq-8B-Tool-Use with Docker Model Runner:
```
docker model run hf.co/Groq/Llama-3-Groq-8B-Tool-Use
```

Excited about Llama-3-Groq-8B-Tool-Use! Question about Vietnamese language support 🐾

by himmeow - opened Jul 22, 2024

Discussion

himmeow

Jul 22, 2024

Congratulations to the Groq team for releasing the new Llama-3-Groq-8B-Tool-Use model! 🔥 The specs and benchmarks look very impressive.

I'm particularly interested in the model's tool-use capabilities. Could the Groq team share any insights into whether this base model can be fine-tuned for function calling in Vietnamese?

Vietnamese language support would be a huge step forward, allowing the Vietnamese AI community to access and leverage the power of Llama-3-Groq-8B-Tool-Use.

Looking forward to hearing from the Groq team! 😊

ricklamers

Groq org Jul 22, 2024

Thanks @himmeow ! The Llama 3 models themselves aren’t advertised as multilingual models, they are focused on English as per their model card.

Perhaps you can still get some performance out of this model for Vietnamese prompts, the best way to find out is to just try I think!

himmeow

Jul 22, 2024

Thanks @ricklamers , my intention is to use Llama-3-Groq as the base model and fine tune more Vietnamese function calling examples to improve performance. Do you agree and have any suggestions for me?

ricklamers

Groq org Jul 23, 2024

That may work! Make sure you add tool use examples in Vietnamese that follow the exact same tool use format as outlined in the text prompt example.

ricklamers changed discussion status to closed Jul 23, 2024

himmeow

Jul 24, 2024

thanks @ricklamers

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment