Instructions to use TinyPixel/Llama-2-7B-bf16-sharded with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TinyPixel/Llama-2-7B-bf16-sharded with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TinyPixel/Llama-2-7B-bf16-sharded")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("TinyPixel/Llama-2-7B-bf16-sharded")
model = AutoModelForCausalLM.from_pretrained("TinyPixel/Llama-2-7B-bf16-sharded", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use TinyPixel/Llama-2-7B-bf16-sharded with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TinyPixel/Llama-2-7B-bf16-sharded"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TinyPixel/Llama-2-7B-bf16-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TinyPixel/Llama-2-7B-bf16-sharded

SGLang

How to use TinyPixel/Llama-2-7B-bf16-sharded with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TinyPixel/Llama-2-7B-bf16-sharded" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TinyPixel/Llama-2-7B-bf16-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TinyPixel/Llama-2-7B-bf16-sharded" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TinyPixel/Llama-2-7B-bf16-sharded",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use TinyPixel/Llama-2-7B-bf16-sharded with Docker Model Runner:
```
docker model run hf.co/TinyPixel/Llama-2-7B-bf16-sharded
```

I trained a model using autotrain, how do I use it?

by MonishMeher - opened Aug 18, 2023

Discussion

MonishMeher

Aug 18, 2023

it's at https://huggingface.co/MonishMeher/catalyst-kiran, how do I chat with it now? I want to use google collab, I dont have a config.json.

TinyPixel

Owner Aug 19, 2023

Use this notebook
https://colab.research.google.com/drive/13IJGf0n8NyBahROCCBZApNE1cT2rdiSC?usp=sharing

MonishMeher

Aug 19, 2023

How do I know for sure its giving output from my trained model? Like the replies are somewhat similar but do not adhere exactly to the dataset, the dataset was just 100rows I know it very less but is there any way to know for sure?
Thankyou Already for your help

Thang199x

Aug 19, 2023

Im struggling with the same situation as you are. Hope someone help us. I expand my dataset to 300 rows but Im not sure if it is enough

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment