--- language: - en license: apache-2.0 library_name: sentence-transformers pipeline_tag: sentence-similarity base_model: ibm-granite/granite-embedding-small-english-r2 datasets: - daniel-gomm/sciseek-train - daniel-gomm/SciSeek tags: - sentence-transformers - feature-extraction - sentence-similarity - text-retrieval - scientific-papers - literature-search - arxiv - onnx - transformers.js model-index: - name: sciseek-nano results: - task: type: retrieval name: Retrieval dataset: type: daniel-gomm/SciSeek name: SciSeek (test) split: test metrics: - type: ndcg_at_10 value: 0.6461 - type: recall_at_10 value: 0.5418 - task: type: retrieval name: Retrieval dataset: type: princeton-nlp/LitSearch name: LitSearch metrics: - type: ndcg_at_10 value: 0.5119 - task: type: retrieval name: Retrieval dataset: type: doris-mae name: DORIS-MAE metrics: - type: ndcg_at_10 value: 0.6506 - task: type: retrieval name: Retrieval dataset: type: BeIR/trec-covid name: BEIR TREC-COVID metrics: - type: ndcg_at_10 value: 0.6304 - task: type: retrieval name: Retrieval dataset: type: BeIR/scifact name: BEIR SciFact metrics: - type: ndcg_at_10 value: 0.7413 - task: type: retrieval name: Retrieval dataset: type: BeIR/nfcorpus name: BEIR NFCorpus metrics: - type: ndcg_at_10 value: 0.3657 - task: type: retrieval name: Retrieval dataset: type: BeIR/scidocs name: BEIR SciDocs metrics: - type: ndcg_at_10 value: 0.2287 --- # sciseek-nano **sciseek-nano** is a highly efficient embedding model for **scientific literature search**. It is fine-tuned specifically for embedding search queries and title + abstract descriptions of research papers. We release it alongside the larger [sciseek-micro](https://huggingface.co/daniel-gomm/sciseek-micro); both are trained on the [sciseek-train](https://huggingface.co/datasets/daniel-gomm/sciseek-train) dataset. sciseek-nano powers the semantic literature search of [refract.science](https://refract.science). On SciSeek, both models are Pareto efficient regarding their model and embedding size, outperforming much larger models. With 47M parameters and 384 dimensions, sciseek-nano ranks third of all models we evaluated on the [SciSeek](https://huggingface.co/datasets/daniel-gomm/SciSeek) benchmark, behind sciseek-micro and Qwen3-Embedding-4B (85× its size). | model | base model | params | dim | SciSeek | LitSearch | 8-bit ONNX | |---|---|---:|---:|---:|---:|---:| | **sciseek-nano** (this model) | granite-embedding-small-english-r2 | 47M | 384 | 0.646 | 0.512 | 53 MB | | [sciseek-micro](https://huggingface.co/daniel-gomm/sciseek-micro) | granite-embedding-english-r2 | 149M | 768 | 0.715 | 0.584 | 167 MB | nDCG@10 on SciSeek (3,010 machine-generated queries over 200,000 arXiv papers) and [LitSearch](https://huggingface.co/datasets/princeton-nlp/LitSearch) (597 real researcher questions). ![nDCG@10 against parameter count on SciSeek and LitSearch](assets/overview.png) - **SciSeek nDCG@10 0.646**, +0.059 [+0.054, +0.064] over its base model. Only sciseek-micro and Qwen3-Embedding-4B score significantly higher; every other model we tested scores significantly lower. - **LitSearch nDCG@10 0.512**, +0.026 [+0.012, +0.040] over its base model. Here, sciseek-micro and four larger general-purpose models (Qwen3-Embedding-4B and -0.6B, jina-v5-small and -nano) score significantly higher; granite-r2, gte-modernbert and leaf-ir are tied, and the rest score lower. - **At equal index size it leads SciSeek:** truncated to 384 dimensions, the best Matryoshka model reaches 0.612 (Qwen3-Embedding-4B), significantly below sciseek-nano. ## Usage The models run in sentence-transformers, ONNX Runtime and transformers.js, and we provide docker compose files for deploying the models with [TEI](https://huggingface.co/docs/text-embeddings-inference) and [vLLM](https://vllm.ai/). Embed papers as `"Title:\n{title}\nAbstract:\n{abstract}"`; queries need no prefix or instruction. The CLS token is used for pooling; L2-normalise and compare with the dot product. The model is specialised for literature search; for other tasks, use a general model such as [granite-embedding-small-english-r2](https://huggingface.co/ibm-granite/granite-embedding-small-english-r2), the base model of sciseek-nano. ```python from sentence_transformers import SentenceTransformer model = SentenceTransformer("daniel-gomm/sciseek-nano") query = "methods for compressing large language models via knowledge distillation" papers = [ "Title:\nTinyBERT: Distilling BERT for Natural Language Understanding\n" "Abstract:\nWe propose a novel Transformer distillation method...", "Title:\nObservations of Type Ia supernovae in distant galaxies\n" "Abstract:\nWe report photometric measurements...", ] q = model.encode(query, normalize_embeddings=True) d = model.encode(papers, normalize_embeddings=True) print(q @ d.T) # cosine similarity ``` In the browser or Node with [transformers.js](https://huggingface.co/docs/transformers.js): ```js import { pipeline } from '@huggingface/transformers' const extractor = await pipeline('feature-extraction', 'daniel-gomm/sciseek-nano', { device: 'webgpu', dtype: 'fp16', // or device: 'wasm', dtype: 'q8' (53 MB) }) const q = await extractor('methods for compressing large language models via knowledge distillation', { pooling: 'cls', normalize: true }) // q.dims = [1, 384] ``` Not every browser exposes WebGPU with fp16, so fall back to WASM if loading fails, and run the extractor in a Web Worker. For a server, [`deploy/`](deploy/) has docker compose files for [Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference) and [vLLM](https://github.com/vllm-project/vllm); vLLM returns unnormalised embeddings, so normalise them yourself. ## Results We evaluated 14 open embedding models (23M to 4B parameters) and BM25 in the same setup. We use the query/document prompts and document format specified in each of the model cards. Search is implemented as the exact cosine over the full corpus. No setting was tuned on SciSeek or LitSearch, and every top-10 result of every run on SciSeek has a relevance label. | model | params | dim | SciSeek | LitSearch | |---|---:|---:|---:|---:| | [sciseek-micro](https://huggingface.co/daniel-gomm/sciseek-micro) | 149M | 768 | 0.715 | 0.584 | | Qwen3-Embedding-4B (8-bit) | 4.0B | 2560 | 0.682 | 0.598 | | **[sciseek-nano](https://huggingface.co/daniel-gomm/sciseek-nano)** (this model) | 47M | 384 | **0.646** | **0.512** | | granite-embedding-english-r2 | 149M | 768 | 0.623 | 0.497 | | jina-embeddings-v5-text-small | 596M | 1024 | 0.620 | 0.556 | | Qwen3-Embedding-0.6B | 596M | 1024 | 0.597 | 0.538 | | jina-embeddings-v5-text-nano | 239M | 768 | 0.587 | 0.541 | | granite-embedding-small-english-r2 | 47M | 384 | 0.587 | 0.486 | | gte-modernbert-base | 149M | 768 | 0.581 | 0.501 | | mdbr-leaf-ir | 23M | 768 | 0.554 | 0.490 | | embeddinggemma-2 | 270M | 768 | 0.550 | 0.481 | | bge-small-en-v1.5 | 33M | 384 | 0.499 | 0.428 | | BM25 | — | — | 0.471 | 0.394 | | all-MiniLM-L6-v2 | 23M | 384 | 0.468 | 0.355 | | SPECTER2 | 110M | 768 | 0.297 | 0.248 | | *queries* | | | *3,010* | *597* | nDCG@10 at each model's native embedding size. Qwen3-Embedding-4B was run with 8-bit weights to fit an 8 GB GPU. Exact settings and run files can be found in the SciSeek benchmark under [`baselines/systems.json`](https://huggingface.co/datasets/daniel-gomm/SciSeek). ### Effect of fine-tuning ![nDCG@10 per query type, base model and fine-tuned model](assets/query_types.png) Compared to the base model, fine-tuning has the largest effect on descriptive queries (questions, problem descriptions, citation contexts, literature reviews) where the semantic context is more relevant for determining relevance than direct overlap. Looking up the titles of known items, which BM25 already solves, drops slightly for both models. **Other scientific retrieval sets.** The fine-tuning keeps the base model's general ability on most sets (nDCG@10, change against the base model with 95% CI; \* = CI excludes 0): | benchmark | what it tests | n | sciseek-nano | sciseek-micro | |---|---|---:|---:|---:| | DORIS-MAE | complex multi-aspect literature queries | 100 | 0.651 (+0.003) | 0.630 (−0.028\*) | | BEIR TREC-COVID | biomedical literature search | 50 | 0.630 (−0.035\*) | 0.712 (−0.017) | | BEIR SciFact | claim verification | 300 | 0.741 (+0.001) | 0.748 (−0.001) | | BEIR NFCorpus | lay-language nutrition/medical queries | 323 | 0.366 (−0.004) | 0.370 (−0.008) | | BEIR SciDocs | a paper title as the query, citing papers as targets | 1,000 | 0.229 (−0.010\*) | 0.240 (−0.011\*) | ## Formats and quantizations | build | file | sciseek-nano | sciseek-micro | |---|---|---:|---:| | PyTorch | `model.safetensors` | 95 MB | 298 MB | | ONNX fp32 | `onnx/model.onnx` | 191 MB | 597 MB | | ONNX fp16 (WebGPU) | `onnx/model_fp16.onnx` | 96 MB | 298 MB | | ONNX 8-bit (WASM, `q8`) | `onnx/model_quantized.onnx` | 53 MB | 167 MB | The fp32 and fp16 ONNX builds give the same embeddings as PyTorch (minimum cosine 1.0000 on 256 texts). The 8-bit builds do not significantly decrease performance and achieve the same test results on both benchmarks. On SciSeek, nDCG@10 changes by −0.000 [−0.001, +0.001] (nano) and −0.001 [−0.001, +0.000] (micro), on LitSearch it changes by +0.004 [+0.000, +0.007] and +0.001 [−0.002, +0.004]. ## Training sciseek-nano is fine-tuned from [granite-embedding-small-english-r2](https://huggingface.co/ibm-granite/granite-embedding-small-english-r2) using the [sciseek-train](https://huggingface.co/datasets/daniel-gomm/sciseek-train) dataset, which consists of semi-synthetic training data sourced from arXiv publications and synthetic search queries. We train with InfoNCE with in-batch and mined hard negatives and false-negative masking, with 512 papers per step for one epoch. We select the best performing model from a separate validation set and blend the fine-tuned and base weights with WiSE-FT `(1 - α) * base + α * fine-tuned` with `α = 0.45` at step 1,000 of 3,248. We ensure that the training data is entirely independent of all papers from the evaluation benchmarks SciSeek, LitSearch, and DORIS-MAE to ensure there is no leakage. We will provide full details of the training process in an upcoming paper. ## Limitations The SciSeek embedding models are trained for literature search, thus the performance for generic retrieval tasks is not sufficiently assessed. Our training and evaluation are also limited to papers sourced from arXiv, which limits the domain of documents the model is trained on, primarily consisting of physics, mathematics, and computer science papers. Other research domains are covered to a lesser degree. Additionally, the training data is semi-synthetic and queries are generated. While evaluation results suggest generalization to human-written queries, we expect a narrower query distribution in our training data than in real search. ## License and citation Apache-2.0. The model builds on [IBM Granite Embedding R2](https://huggingface.co/ibm-granite/granite-embedding-small-english-r2). Thank you to arXiv for use of its open access interoperability. ```bibtex @misc{gomm2026sciseek, title = {SciSeek: Benchmarking and Training Embedding Models for Scientific Literature Search}, author = {Gomm, Daniel}, year = {2026}, url = {https://huggingface.co/datasets/daniel-gomm/SciSeek} } ```