Update README.md
Browse files
README.md
CHANGED
|
@@ -9,16 +9,17 @@ tags:
|
|
| 9 |
- MoE
|
| 10 |
- unicode
|
| 11 |
---
|
| 12 |
-
|
| 13 |
# bvv241-abs: Unified Unicode Tokenizer (SOTA Intersection) with Frozen Embeddings and Extended Vector Dim (4096)
|
| 14 |
|
| 15 |
-
This model is a core component described in the paper [**Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate**](https://huggingface.co/papers/2507.07129).
|
| 16 |
-
|
| 17 |
-
This work explores a novel constructive approach to model development, built upon the foundation of non-trainable, deterministic input embeddings. It demonstrates that this fixed representational substrate acts as a universal "docking port," enabling seamless modular composition and progressive layer-wise growth of Transformer models.
|
| 18 |
|
| 19 |
## Tokenizer Description
|
| 20 |
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
This tokenizer is based on a hybrid vocabulary:
|
| 24 |
|
|
@@ -40,21 +41,14 @@ No training or adaptation; suitable for plug-and-play use in research on embeddi
|
|
| 40 |
## How to Get Started with the Tokenizer
|
| 41 |
|
| 42 |
```python
|
| 43 |
-
|
| 44 |
from transformers import AutoTokenizer
|
| 45 |
-
|
| 46 |
from huggingface_hub import hf_hub_download
|
| 47 |
-
|
| 48 |
import torch
|
| 49 |
-
|
| 50 |
tokenizer = AutoTokenizer.from_pretrained('Bochkov/bvv241-abs')
|
| 51 |
-
|
| 52 |
-
|
| 53 |
emb_path = hf_hub_download(
|
| 54 |
repo_id="Bochkov/bvv241-abs",
|
| 55 |
filename="normalized_embeddings_weights.pt"
|
| 56 |
)
|
| 57 |
-
|
| 58 |
embeddings = torch.load(emb_path)
|
| 59 |
```
|
| 60 |
|
|
@@ -72,7 +66,6 @@ If you use this model or the underlying concepts in your research, please cite o
|
|
| 72 |
primaryClass={cs.CL},
|
| 73 |
url={https://arxiv.org/abs/2507.04886},
|
| 74 |
}
|
| 75 |
-
|
| 76 |
@misc{bochkov2025growingtransformersmodularcomposition,
|
| 77 |
title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate},
|
| 78 |
author={A. Bochkov},
|
|
@@ -84,4 +77,4 @@ If you use this model or the underlying concepts in your research, please cite o
|
|
| 84 |
}
|
| 85 |
```
|
| 86 |
|
| 87 |
-
This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.
|
|
|
|
| 9 |
- MoE
|
| 10 |
- unicode
|
| 11 |
---
|
|
|
|
| 12 |
# bvv241-abs: Unified Unicode Tokenizer (SOTA Intersection) with Frozen Embeddings and Extended Vector Dim (4096)
|
| 13 |
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
## Tokenizer Description
|
| 16 |
|
| 17 |
+
This repository contains the tokenizer and associated resources from this paper:
|
| 18 |
+
|
| 19 |
+
[📚 Paper (Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations)](https://huggingface.co/papers/2507.04886) -
|
| 20 |
+
|
| 21 |
+
[💻 Code](https://github.com/AVBochkov/Embeddings)
|
| 22 |
+
|
| 23 |
|
| 24 |
This tokenizer is based on a hybrid vocabulary:
|
| 25 |
|
|
|
|
| 41 |
## How to Get Started with the Tokenizer
|
| 42 |
|
| 43 |
```python
|
|
|
|
| 44 |
from transformers import AutoTokenizer
|
|
|
|
| 45 |
from huggingface_hub import hf_hub_download
|
|
|
|
| 46 |
import torch
|
|
|
|
| 47 |
tokenizer = AutoTokenizer.from_pretrained('Bochkov/bvv241-abs')
|
|
|
|
|
|
|
| 48 |
emb_path = hf_hub_download(
|
| 49 |
repo_id="Bochkov/bvv241-abs",
|
| 50 |
filename="normalized_embeddings_weights.pt"
|
| 51 |
)
|
|
|
|
| 52 |
embeddings = torch.load(emb_path)
|
| 53 |
```
|
| 54 |
|
|
|
|
| 66 |
primaryClass={cs.CL},
|
| 67 |
url={https://arxiv.org/abs/2507.04886},
|
| 68 |
}
|
|
|
|
| 69 |
@misc{bochkov2025growingtransformersmodularcomposition,
|
| 70 |
title={Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate},
|
| 71 |
author={A. Bochkov},
|
|
|
|
| 77 |
}
|
| 78 |
```
|
| 79 |
|
| 80 |
+
This work demonstrates that transformer blocks, not token embeddings, carry the semantic burden in LLMs — a step toward modular, fusable, multilingual LMs.
|