recipe-gen-mini-gpt / tokenizer_config.json
Shoriful025's picture
Create tokenizer_config.json
6085d51 verified
{
"add_prefix_space": false,
"model_max_length": 1024,
"tokenizer_class": "GPT2Tokenizer"
}