Upload folder using huggingface_hub

README.md

@@ -1,95 +1,135 @@
 ---
-library_name: gliner2
+language: en
 license: mit
+library_name: gliner2
+tags:
+- gliner
+- gliner2
+- token-classification
+- ner
+- named-entity-recognition
+- world-bank
+- datasets
+- data-use
+- lora
+- adapter
+- forced-displacement
+- refugees
+- fcv
 base_model: fastino/gliner2-large-v1
 datasets:
-  - ai4data/datause-train
-tags:
-  - ner
-  - data-mention-extraction
-  - lora
-  - gliner2
-  - development-economics
+- ai4data/datause-train
+- ai4data/datause-holdout
+metrics:
+- precision
+- recall
+- f1
+---
+
+# datause-extraction-v12rerun
+
+This repository contains the LoRA fine-tuned adapter weights (`v12-rerun`) for dataset mention extraction. It is trained on top of the base model `fastino/gliner2-large-v1`.
+
+It classifies spans into three categories:
+1. `named_data`: Proper named datasets, surveys, censuses, or registries (e.g., *Demographic and Health Survey*, *LFS*, *UNHCR PRIMES*).
+2. `descriptive_data`: Data resources described by their producer or characteristics rather than a proper name (e.g., *World Bank household surveys*, *spatial socioeconomic data sets*).
+3. `vague_data`: General references containing a data noun but lacking enough specificity to identify the exact source (e.g., *administrative data*, *project statistics*).
+
+---
+
+## Rationale and Context: Forced Displacement, Refugees, and FCV
+
+In Fragile, Conflict, and Violence (FCV) settings, monitoring the utilization of datasets is crucial for coordinating developmental and humanitarian aid. Research on forced displacement and refugee integration relies heavily on specific household surveys, operational registries, and geographic vulnerability datasets. 
+
+By automating the extraction of these references from project documents, appraisal papers, and academic studies, this model helps map data usage, highlights under-analyzed areas, and evaluates the policy impact of statistical capacity investments.
+
 ---
 
-# datause-extraction
+## Data Sources & Domain Coverage
 
-Fine-tuned GLiNER2 LoRA adapter for extracting structured data mentions from
-development economics and humanitarian research documents.
+The model is specialized in the socio-economic development and forced displacement domains, with strong representation of:
+*   **Humanitarian Registries & Briefs:** UNHCR registration databases (PRIMES), Refugee Socio-Economic Inclusion Surveys (SEIS), Durable Solutions reports, and Protection Monitoring tools.
+*   **Development Economics & Surveys:** World Bank Project Appraisal Documents (PADs), Living Standards Measurement Study (LSMS), Demographic and Health Surveys (DHS), Multiple Indicator Cluster Surveys (MICS), and national censuses.
+*   **FCV/Geospatial Data:** Livelihood surveys, cash-based intervention tracking, and geographic data (e.g., Shuttle Radar Topography Mission, flood hazard mapping, population distribution layers).
 
-This is the production release of
-[rafmacalaba/gliner2-datause-large-v1-deval-synth-v2](https://huggingface.co/rafmacalaba/gliner2-datause-large-v1-deval-synth-v2).
+---
 
-## Task
+## Model Performance
 
-Given a passage of text, the model identifies every data source mentioned and
-classifies it across four dimensions:
+The adapter was evaluated on the canonical layout-aware, project-purged **Holdout v10** dataset (`flat_ner_holdout_v10.jsonl` / `ai4data/datause-holdout`) at a confidence threshold of **0.40** (Jaccard matching threshold = 0.50):
 
-| Field | Type | Values |
-|---|---|---|
-| `mention_name` | Extractive span | Verbatim text from the passage |
-| `specificity_tag` | Classification | `named` / `descriptive` / `vague` |
-| `typology_tag` | Classification | `survey` / `census` / `administrative` / `database` / `indicator` / `geospatial` / `microdata` / `report` / `other` |
-| `is_used` | Classification | `True` / `False` |
-| `usage_context` | Classification | `primary` / `supporting` / `background` |
+| Evaluation Set | TP | FP | FN | Precision | Recall | F0.5 Score |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Positive-Only Records** (465 chunks w/ mentions) | 576 | 51 | 152 | **91.9%** | **79.1%** | **0.8900** |
+| **All Records** (Full set of 1,149 chunks) | 576 | 136 | 152 | **80.9%** | **79.1%** | **0.8054** |
 
-## Inference — Two-Pass Hybrid
+---
 
-This model uses a **two-pass** architecture. A single-pass structured extract
-will not produce correct results.
+## How to Use
+
+You can load and use this model either via the direct `gliner2` library interface or using the high-level `ai4data` library wrappers.
+
+### Option 1: Using the `ai4data` Library (Recommended)
+
+The `ai4data` python package automatically handles base model initialization, adapter downloads, token chunking, and post-filtering:
 
 ```python
+from ai4data import extract_from_text
+
+text = (
+    "To analyze the impact of infrastructure spillovers, we combine data from the "
+    "2010 Ghana Living Standards Survey (GLSS) with production records for 17 "
+    "large-scale gold mines."
+)
+
+# Extract dataset mentions using this specific adapter
+result = extract_from_text(
+    text,
+    adapter_id="rafmacalaba/datause-extraction-v12rerun",
+    include_confidence=True
+)
+
+for ds in result.get("datasets", []):
+    print(f"Dataset:    {ds['dataset_name']}")
+    print(f"Confidence: {ds['dataset_confidence']:.3f}")
+    print(f"Section:    {ds['section_context']}")
+    print("-" * 30)
+```
+
+### Option 2: Using the Raw `gliner2` Interface
+
+If you are using the raw weights directly as a LoRA adapter, you must load the base model (`fastino/gliner2-large-v1`) first and apply the adapter:
+
+```python
+import torch
 from gliner2 import GLiNER2
 from huggingface_hub import snapshot_download
 
-# Install the patched GLiNER2 library:
-# pip install git+https://github.com/rafmacalaba/GLiNER2.git@feat/main-mirror
-
-BASE_MODEL = "fastino/gliner2-large-v1"
-ADAPTER_ID = "ai4data/datause-extraction"
-
-extractor = GLiNER2.from_pretrained(BASE_MODEL)
-extractor.load_adapter(snapshot_download(ADAPTER_ID))
-extractor.eval()
-
-CLASSIFICATION_TASKS = {
-    "specificity_tag": ["named", "descriptive", "vague"],
-    "typology_tag": [
-        "survey", "census", "administrative", "database",
-        "indicator", "geospatial", "microdata", "report", "other",
-    ],
-    "is_used": ["True", "False"],
-    "usage_context": ["primary", "supporting", "background"],
-}
-
-text = "We use the Demographic and Health Survey (DHS) 2020 as our primary data source."
-
-# Pass 1: entity extraction
-res_ent = extractor.extract_entities(text, ["data_mention"], threshold=0.3, include_confidence=True)
-spans = (
-    res_ent.get("entities", {}).get("data_mention", [])
-    if isinstance(res_ent, dict)
-    else res_ent
+# 1. Initialize base model
+kwargs = {}
+if torch.cuda.is_available():
+    kwargs["map_location"] = "cuda"
+elif torch.backends.mps.is_available():
+    kwargs["map_location"] = "mps"
+else:
+    kwargs["map_location"] = "cpu"
+
+model = GLiNER2.from_pretrained("fastino/gliner2-large-v1", **kwargs)
+
+# 2. Download and apply the LoRA adapter weights
+adapter_path = snapshot_download("rafmacalaba/datause-extraction-v12rerun")
+model.load_adapter(adapter_path)
+
+# 3. Perform inference
+text = (
+    "To analyze the impact of infrastructure spillovers, we combine data from the "
+    "2010 Ghana Living Standards Survey (GLSS) with production records for 17 "
+    "large-scale gold mines."
 )
 
-# Build classification inputs for each valid span
-results = []
-for span_data in spans:
-    span_text = span_data.get("text", "") if isinstance(span_data, dict) else str(span_data)
-    span_conf = span_data.get("confidence", 0.0) if isinstance(span_data, dict) else 1.0
-    if len(span_text) < 3:
-        continue
-    start     = text.find(span_text)
-    ctx_start = max(0, start - 150) if start != -1 else 0
-    ctx_end   = min(len(text), start + len(span_text) + 150) if start != -1 else len(text)
-    context_str = f"Mention: {span_text} | Context: {text[ctx_start:ctx_end]}"
-
-    # Pass 2: classify the span's context window
-    classes = extractor.classify_text(context_str, CLASSIFICATION_TASKS, threshold=0.3)
-    mention = {"mention_name": span_text, "confidence": span_conf}
-    for task, out in classes.items():
-        mention[task] = out[0] if isinstance(out, tuple) and len(out) == 2 else out
-    results.append(mention)
-
-print(results)
-```
+labels = ["named_data", "descriptive_data", "vague_data"]
+predictions = model.predict_entities(text, labels, threshold=0.40)
+
+for entity in predictions:
+    print(f"Text:  {entity['text']} | Label: {entity['label']} | Score: {entity['score']:.3f}")
+```

adapter_config.json

@@ -1,15 +1,54 @@
 {
-  "adapter_type": "lora",
-  "adapter_version": "1.0",
-  "lora_r": 16,
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": {
+    "base_model_class": "GLiNER2",
+    "parent_library": "gliner2.inference.engine"
+  },
+  "base_model_name_or_path": "fastino/gliner2-large-v1",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
   "lora_alpha": 32.0,
-  "lora_dropout": 0.1,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
   "target_modules": [
-    "classifier",
-    "count_embed",
-    "count_pred",
-    "encoder",
-    "span_rep"
+    "query_proj",
+    "value_proj",
+    "project_end.3",
+    "project_start.0",
+    "project_end.0",
+    "project_start.3",
+    "out_project.0",
+    "dense",
+    "key_proj",
+    "out_project.3"
   ],
-  "created_at": "2026-04-06T13:46:19.060075Z"
+  "target_parameters": null,
+  "task_type": null,
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
 }

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:219080a9707cbffafd5b4548b1fbe6723798ae2f622ea57d13d4a42f0ca02e51
+size 30387408