Hugging Face's logo

flexitok
/
bpe_deu_Latn_16000

German
tokenizer
bpe
flexitok
fineweb2
Model card Files
xet
Community

Byte-Level BPE Tokenizer: deu_Latn (16K)

A Byte-Level BPE tokenizer trained on deu_Latn data from Fineweb-2-HQ.

Training Details

Parameter Value
Algorithm Byte-Level BPE
Language deu_Latn
Target Vocab Size 16,000
Final Vocab Size 0
Pre-tokenizer ByteLevel
Normalizer NFC
Special Tokens <s>, </s>, <pad>, <unk>
Training Shards 2

Usage 

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_deu_Latn_16000")
tokens = tokenizer.encode("Hello, world!")

Files

  • tokenizer.json — Full HuggingFace tokenizer
  • vocab.json — Vocabulary mapping
  • merges.txt — BPE merge rules
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including flexitok/bpe_deu_Latn_16000