dranger003/GritLM-8x7B-iMat.GGUF

GGUF importance matrix (imatrix) quants for https://huggingface.co/GritLM/GritLM-8x7B
The importance matrix was trained for ~50K tokens (105 batches of 512 tokens) using a general purpose imatrix calibration dataset.

GritLM is a generative representational instruction tuned language model. It unifies text representation (embedding) and text generation into a single model achieving state-of-the-art performance on both types of tasks.

Layers	Context	Template (Text Representation)	Template (Text Generation)
32	32768	<s><|user|> {instruction} <|embed|> {sample}	<s><|user|> {prompt} <|assistant|> {response}