| ---
|
| language:
|
| - en
|
| library_name: transformers
|
| pipeline_tag: summarization
|
| license: apache-2.0
|
| tags:
|
| - chemistry
|
| - scientific-summarization
|
| - distilbart
|
| - abstractive
|
| - tldr
|
| - knowledge-graphs
|
| datasets:
|
| - Bocklitz-Lab/lit2vec-tldr-bart-dataset
|
| model-index:
|
| - name: lit2vec-tldr-bart
|
| results:
|
| - task:
|
| name: Summarization
|
| type: summarization
|
| dataset:
|
| name: Lit2Vec TL;DR Chemistry Dataset
|
| type: Bocklitz-Lab/lit2vec-tldr-bart-dataset
|
| split: test
|
| size: 1001
|
| metrics:
|
| - type: rouge1
|
| value: 56.11
|
| - type: rouge2
|
| value: 30.78
|
| - type: rougeLsum
|
| value: 45.43
|
| ---
|
|
|
| # lit2vec-tldr-bart (DistilBART fine-tuned for chemistry TL;DRs)
|
|
|
| **lit2vec-tldr-bart** is a DistilBART model fine-tuned on **19,992** CC-BY licensed chemistry abstracts to produce **concise TL;DR-style summaries** aligned with methods → results → significance. It’s designed for scientific **abstractive summarization**, **semantic indexing**, and **knowledge-graph population** in chemistry and related fields.
|
|
|
| - **Base model:** `sshleifer/distilbart-cnn-12-6`
|
| - **Training data:** [`Bocklitz-Lab/lit2vec-tldr-bart-dataset`](https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-tldr-bart-dataset)
|
| - **Max input length:** 1024 tokens
|
| - **Target length:** ~128 tokens
|
|
|
| ---
|
|
|
| ## 🧪 Evaluation (held-out test)
|
|
|
| | Split | ROUGE-1 | ROUGE-2 | ROUGE-Lsum |
|
| |------:|--------:|--------:|-----------:|
|
| | Test | **56.11** | **30.78** | **45.43** |
|
|
|
| > Validation RLsum: 46.05
|
| > Metrics computed with `evaluate`'s `rouge` (NLTK sentence segmentation, `use_stemmer=True`).
|
|
|
| ---
|
|
|
| ## 🚀 Quickstart
|
|
|
| ```python
|
| from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, GenerationConfig
|
|
|
| repo = "Bocklitz-Lab/lit2vec-tldr-bart"
|
|
|
| tok = AutoTokenizer.from_pretrained(repo)
|
| model = AutoModelForSeq2SeqLM.from_pretrained(repo)
|
| gen = GenerationConfig.from_pretrained(repo) # loads default decoding params
|
|
|
| text = "Proton exchange membrane fuel cells convert chemical energy into electricity..."
|
| inputs = tok(text, return_tensors="pt", truncation=True, max_length=1024)
|
|
|
| summary_ids = model.generate(**inputs, **gen.to_dict())
|
| print(tok.decode(summary_ids[0], skip_special_tokens=True))
|
| ````
|
|
|
| ### Batch inference (PyTorch)
|
|
|
| ```python
|
| texts = [
|
| "Abstract 1 ...",
|
| "Abstract 2 ...",
|
| ]
|
| batch = tok(texts, return_tensors="pt", padding=True, truncation=True, max_length=1024)
|
| out = model.generate(**batch, **gen.to_dict())
|
| summaries = tok.batch_decode(out, skip_special_tokens=True)
|
| ```
|
|
|
| ---
|
|
|
| ## 🔧 Default decoding (saved in `generation_config.json`)
|
|
|
| These are the defaults saved with the model (you can override at `generate()` time):
|
|
|
| ```json
|
| {
|
| "max_length": 142,
|
| "min_length": 56,
|
| "early_stopping": true,
|
| "num_beams": 4,
|
| "length_penalty": 2.0,
|
| "no_repeat_ngram_size": 3,
|
| "forced_bos_token_id": 0,
|
| "forced_eos_token_id": 2
|
| }
|
| ```
|
|
|
| ---
|
|
|
| ## 📊 Training details
|
|
|
| * **Base:** `sshleifer/distilbart-cnn-12-6` (Distilled BART)
|
| * **Data:** 19,992 CC-BY chemistry abstracts with TL;DR summaries
|
| * **Splits:** train=17,992 / val=999 / test=1,001
|
| * **Max lengths:** input 1024, target 128
|
| * **Optimizer:** AdamW, **lr=2e-5**
|
| * **Batching:** per-device train/eval batch size 4, **gradient\_accumulation\_steps=4**
|
| * **Epochs:** 5
|
| * **Precision:** fp16 (when CUDA available)
|
| * **Hardware:** single NVIDIA RTX 3090
|
| * **Seed:** 42
|
| * **Libraries:** 🤗 Transformers + Datasets, `evaluate` for ROUGE, NLTK for sentence splitting
|
|
|
| ---
|
|
|
| ## ✅ Intended use
|
|
|
| * TL;DR abstractive summaries for **chemistry** and adjacent domains (materials science, chemical engineering, environmental science).
|
| * **Semantic indexing**, **IR reranking**, and **knowledge graph** ingestion where concise method/result statements are helpful.
|
|
|
| ### Limitations & risks
|
|
|
| * May **hallucinate** details not present in the abstract (typical for abstractive models).
|
| * Not a substitute for expert judgment; avoid using summaries as sole evidence for scientific claims.
|
| * Trained on CC-BY English abstracts; performance may degrade on other domains/languages.
|
|
|
| ---
|
|
|
| ## 📦 Files
|
|
|
| This repo should include:
|
|
|
| * `config.json`, `pytorch_model.bin` or `model.safetensors`
|
| * `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, merges/vocab as applicable
|
| * `generation_config.json` (decoding defaults)
|
|
|
| ---
|
|
|
| ## 🔁 Reproducibility
|
|
|
| * Dataset: [`Bocklitz-Lab/lit2vec-tldr-bart-dataset`](https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-tldr-bart-dataset)
|
| * Recommended preprocessing: truncate inputs at 1024 tokens; targets at 128.
|
| * ROUGE evaluation: `evaluate.load("rouge")`, NLTK sentence tokenization, `use_stemmer=True`.
|
|
|
| ---
|
|
|
| ## 📚 Citation
|
|
|
| If you use this model or dataset, please cite:
|
|
|
| ```bibtex
|
| @software{lit2vec_tldr_bart_2025,
|
| title = {lit2vec-tldr-bart: DistilBART fine-tuned for chemistry TL;DR summarization},
|
| author = {Bocklitz Lab},
|
| year = {2025},
|
| url = {https://huggingface.co/Bocklitz-Lab/lit2vec-tldr-bart},
|
| note = {Model trained on CC-BY chemistry abstracts; dataset at Bocklitz-Lab/lit2vec-tldr-bart-dataset}
|
| }
|
| ```
|
|
|
| Dataset:
|
|
|
| ```bibtex
|
| @dataset{lit2vec_tldr_dataset_2025,
|
| title = {Lit2Vec TL;DR Chemistry Dataset},
|
| author = {Bocklitz Lab},
|
| year = {2025},
|
| url = {https://huggingface.co/datasets/Bocklitz-Lab/lit2vec-tldr-bart-dataset}
|
| }
|
| ```
|
|
|
| ---
|
|
|
| ## 📝 License
|
|
|
| * **Model weights & code:** Apache-2.0
|
| * **Dataset:** CC BY 4.0 (attribution in per-record metadata)
|
|
|
| ---
|
|
|
| ## 🙌 Acknowledgements
|
|
|
| * Base model: DistilBART (`sshleifer/distilbart-cnn-12-6`)
|
| * Licensing and OA links curated from publisher/aggregator sources; dataset restricted to **CC-BY** content.
|
|
|
