Update README.md
Browse files
README.md
CHANGED
|
@@ -5,24 +5,24 @@ language:
|
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
tags:
|
| 7 |
- Arabic_Dialectal_Lemmatization
|
| 8 |
-
---
|
| 9 |
-
|
| 10 |
# Gulf-S2S-lemmatizer
|
| 11 |
-
|
| 12 |
-
# Model Description
|
| 13 |
-
|
| 14 |
-
The model is developed for Arabic dialect lemmatization, focusing on Gulf (GLF) Arabic. It follows a sequence-to-sequence formulation of lemmatization, where the model generates the lemma of a given word knowing 2 words before and 2 words after the current word rather than treating lemmas as fixed classification labels.
|
| 15 |
-
|
| 16 |
-
The model is evaluated using lemma accuracy as the main metric, with an additional normalized lemma accuracy metric that accounts for orthographic and diacritic variation. The full methodology, training setup, hyperparameters, and evaluation results are described in our paper [“Lemmatizing Dialectal Arabic with Sequence-to-Sequence Models”](https://aclanthology.org/2025.arabicnlp-main.10/)
|
| 17 |
-
|
| 18 |
-
|
| 19 |
# Standalone Usage
|
| 20 |
|
| 21 |
|
| 22 |
-
|
| 23 |
The model can also be used independently without the full lemmatization workflow on the GitHub repository (https://github.com/CAMeL-Lab/seq2seq-arabic-dialect-lemmatization). In this case, the input should contain the target word surrounded by the special token `<target>`, with up to two words before and two words after the target word.
|
| 24 |
-
|
| 25 |
-
|
| 26 |
```python
|
| 27 |
import re
|
| 28 |
import math
|
|
@@ -33,7 +33,7 @@ from transformers import T5Tokenizer, T5ForConditionalGeneration
|
|
| 33 |
|
| 34 |
DIALECT_MODELS = {
|
| 35 |
"glf": "CAMeL-Lab/GLF-S2S-lemmatizer",
|
| 36 |
-
}
|
| 37 |
def load_model(s2s_dialect: str):
|
| 38 |
model_name = DIALECT_MODELS[s2s_dialect]
|
| 39 |
tokenizer = T5Tokenizer.from_pretrained(model_name, use_fast=True, legacy=False)
|
|
@@ -82,16 +82,16 @@ def predict(tokenizer, model, texts: list[str], device=None, batch_size: int = 1
|
|
| 82 |
|
| 83 |
# Example:
|
| 84 |
# df should contain an input_text column with the target word marked using <target>
|
| 85 |
-
# Example input: "أنا
|
| 86 |
|
| 87 |
df["word"] = df["input_text"].apply(extract_target_word)
|
| 88 |
|
| 89 |
tokenizer, model = load_model("egy")
|
| 90 |
df["predicted_lex"] = predict(tokenizer, model, df["input_text"].tolist())
|
| 91 |
-
```
|
| 92 |
-
|
| 93 |
-
## 📖 Citation
|
| 94 |
-
|
| 95 |
If you use this model in your research, please cite the following paper:
|
| 96 |
|
| 97 |
```bibtex
|
|
@@ -103,5 +103,5 @@ If you use this model in your research, please cite the following paper:
|
|
| 103 |
address = {Suzhou, China},
|
| 104 |
url = {https://aclanthology.org/2025.arabicnlp-main.10/}
|
| 105 |
}
|
| 106 |
-
```
|
| 107 |
-
|
|
|
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
tags:
|
| 7 |
- Arabic_Dialectal_Lemmatization
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
# Gulf-S2S-lemmatizer
|
| 11 |
+
|
| 12 |
+
# Model Description
|
| 13 |
+
|
| 14 |
+
The model is developed for Arabic dialect lemmatization, focusing on Gulf (GLF) Arabic. It follows a sequence-to-sequence formulation of lemmatization, where the model generates the lemma of a given word knowing 2 words before and 2 words after the current word rather than treating lemmas as fixed classification labels.
|
| 15 |
+
|
| 16 |
+
The model is evaluated using lemma accuracy as the main metric, with an additional normalized lemma accuracy metric that accounts for orthographic and diacritic variation. The full methodology, training setup, hyperparameters, and evaluation results are described in our paper [“Lemmatizing Dialectal Arabic with Sequence-to-Sequence Models”](https://aclanthology.org/2025.arabicnlp-main.10/)
|
| 17 |
+
|
| 18 |
+
|
| 19 |
# Standalone Usage
|
| 20 |
|
| 21 |
|
| 22 |
+
|
| 23 |
The model can also be used independently without the full lemmatization workflow on the GitHub repository (https://github.com/CAMeL-Lab/seq2seq-arabic-dialect-lemmatization). In this case, the input should contain the target word surrounded by the special token `<target>`, with up to two words before and two words after the target word.
|
| 24 |
+
|
| 25 |
+
|
| 26 |
```python
|
| 27 |
import re
|
| 28 |
import math
|
|
|
|
| 33 |
|
| 34 |
DIALECT_MODELS = {
|
| 35 |
"glf": "CAMeL-Lab/GLF-S2S-lemmatizer",
|
| 36 |
+
}
|
| 37 |
def load_model(s2s_dialect: str):
|
| 38 |
model_name = DIALECT_MODELS[s2s_dialect]
|
| 39 |
tokenizer = T5Tokenizer.from_pretrained(model_name, use_fast=True, legacy=False)
|
|
|
|
| 82 |
|
| 83 |
# Example:
|
| 84 |
# df should contain an input_text column with the target word marked using <target>
|
| 85 |
+
# Example input: "أنا أبي <target>أروح<target> البيت الحين"
|
| 86 |
|
| 87 |
df["word"] = df["input_text"].apply(extract_target_word)
|
| 88 |
|
| 89 |
tokenizer, model = load_model("egy")
|
| 90 |
df["predicted_lex"] = predict(tokenizer, model, df["input_text"].tolist())
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
## 📖 Citation
|
| 94 |
+
|
| 95 |
If you use this model in your research, please cite the following paper:
|
| 96 |
|
| 97 |
```bibtex
|
|
|
|
| 103 |
address = {Suzhou, China},
|
| 104 |
url = {https://aclanthology.org/2025.arabicnlp-main.10/}
|
| 105 |
}
|
| 106 |
+
```
|
| 107 |
+
|