Synthyra

/

Profluent-E1-150M

like 1

Follow

Synthyra 40

Hey @GrimSqueaker ,

Thanks for pointing this out! Looks like one of the recent transformer updates broke our AutoModel compatibility - that has been fixed now. The examples are now working fine:

import torch
from transformers import AutoModelForMaskedLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = AutoModelForMaskedLM.from_pretrained('Synthyra/Profluent-E1-150M', trust_remote_code=True, dtype=torch.bfloat16).eval().to(device)

sequences = ['MPRTEIN', 'MSEQWENCE']
batch = model.prep_tokens.get_batch_kwargs(sequences, device=device)

output = model(**batch, output_hidden_states=True) # get all hidden states with output_hidden_states=True
print(output.logits.shape) # language modeling logits, (batch_size, seq_len, vocab_size), (2, 11, 34)
print(output.last_hidden_state.shape) # last hidden state of the model, (batch_size, seq_len, hidden_size), (2, 11, 768)
print(output.loss)
print(len(output.hidden_states)) # all hidden states if you passed output_hidden_states=True (in tuple)

torch.Size([2, 13, 34])
torch.Size([2, 13, 768])
tensor(3.4620, device='cuda:0', grad_fn=<AddBackward0>)
21

Also, there are additional requirements for some of the models. They are referenced in our READMEs and GitHub

The GitHub with the implementation and requirements.txt can be found here

Hope this is helpful!
Best,
Logan

Hi - what do you mean fixed as of now? (Currently, rerunning with the updated config - I'm still getting an error).

Traceback (most recent call last):
  File "train_model_v3.py", line 682, in <module>
    train_model(
  File "train_model_v3.py", line 414, in train_model
    word_embedding_model = models.Transformer(
                           ^^^^^^^^^^^^^^^^^^^
 in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "anaconda3/envs/hf/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 1185, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.Synthyra.Profluent_hyphen_E1_hyphen_150M.ffd710bd8e9104abcaae21fbe0db440698628883.modeling_e1.E1Config'> to build an AutoTokenizer.

Hi @GrimSqueaker ,

The tokenizer is not supported via AutoTokenizer.

You can prep sequences with the tokenizer via the code I pasted above:

sequences = ['MPRTEIN', 'MSEQWENCE']
batch = model.prep_tokens.get_batch_kwargs(sequences, device=device)

which you can easily send to the model with

output = model(**batch)