An Empirical Study of Path Feasibility Queries
Paper โข 1302.4798 โข Published
AI & ML interests
None defined yet.
Recent Activity
View all activity
codelionย
authored 4
papers 5 days ago
Post
140
# One Script, Every Benchmark
Every Bench Labs benchmark had its own eval files. Now there's one script, hosted in the leaderboard Space:
It runs the full suite with the official scoring โ Effortless (exact-match), Easy (hybrid category-aware), Mid (loglikelihood:
Output includes
bench-labs
Every Bench Labs benchmark had its own eval files. Now there's one script, hosted in the leaderboard Space:
curl -sLO https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard/resolve/main/script.py
pip install torch transformers
python script.py --model your/modelIt runs the full suite with the official scoring โ Effortless (exact-match), Easy (hybrid category-aware), Mid (loglikelihood:
acc, acc_norm, soft_score_norm) โ and reports every category and subcategory, not just one number.Output includes
leaderboard.json: a ready-to-paste models.json entry. Run the script, paste it, open a PR on the [leaderboard]( bench-labs/BenchLabs-Leaderboard). Done. 
- 30 replies
Post
3641
# PixelModel v1
Last month we released PixelModel โ a neural network whose weights are literally the pixels of a PNG. It was a toy: 202,752 parameters, welded to 32ร32 output, trained on six solid-color swatches. It scored FID 566.84 on the Tiny-T2I-Leaderboard, mostly by producing the same yellow noise for every prompt.
Today we're releasing PixelModel v1. It is 8.5ร smaller โ 23,747 parameters โ and it beats v0 on both benchmark metrics while being trained on 20,000 real MS-COCO caption/image pairs instead of six color swatches. The entire model now fits in a 160ร149 PNG.
That image is not a visualization of the model. It is the model. All 23,747 weights, one per pixel.
## links
bench-labs
Blog post [read it here โ]( bench-labs/blog)
See us on the [Leaderboard โ]( FlameF0X/Tiny-T2I-Leaderboard)
Model card [here]( bench-labs/pixelmodel-v1)
## The catch
A 23K-parameter model does not draw sandwiches. With ~1 parameter per training image, the loss-minimizing behavior is to output the average of all plausible images for a caption โ caption-conditioned color, light, and layout statistics. Food prompts come out warm and brown; sky prompts come out cool and bright. That is the ceiling for this size class, and we'd rather show it than crop around it.
# cherry on top ๐
The model generates 600 images (cpu) in 5 (five) seconds.
Thats 5000 images in 24 seconds on cpu.
The model trained on cpu for just 30 minutes.
Last month we released PixelModel โ a neural network whose weights are literally the pixels of a PNG. It was a toy: 202,752 parameters, welded to 32ร32 output, trained on six solid-color swatches. It scored FID 566.84 on the Tiny-T2I-Leaderboard, mostly by producing the same yellow noise for every prompt.
Today we're releasing PixelModel v1. It is 8.5ร smaller โ 23,747 parameters โ and it beats v0 on both benchmark metrics while being trained on 20,000 real MS-COCO caption/image pairs instead of six color swatches. The entire model now fits in a 160ร149 PNG.
That image is not a visualization of the model. It is the model. All 23,747 weights, one per pixel.
## links
Blog post [read it here โ]( bench-labs/blog)
See us on the [Leaderboard โ]( FlameF0X/Tiny-T2I-Leaderboard)
Model card [here]( bench-labs/pixelmodel-v1)
## The catch
A 23K-parameter model does not draw sandwiches. With ~1 parameter per training image, the loss-minimizing behavior is to output the average of all plausible images for a caption โ caption-conditioned color, light, and layout statistics. Food prompts come out warm and brown; sky prompts come out cool and bright. That is the ceiling for this size class, and we'd rather show it than crop around it.
# cherry on top ๐
The model generates 600 images (cpu) in 5 (five) seconds.
Thats 5000 images in 24 seconds on cpu.
The model trained on cpu for just 30 minutes.
PhysiQuantyย
posted an update 19 days ago
Post
4726
๐ง Arithmetic-SLM : A 30M model that manages to compute simple arithmetic better than a 3B model ๐
WhirlwindAI/Arithmetic-SLM
WhirlwindAI/arithmetic-slm
๐ Leaderboard ArithMark-2 ๐
๐ฅ Qwen/Qwen2.5-Math-1.5B = 82.08%
๐ฅ WhirlwindAI/Arithmetic-SLM = 78.60% (31.7M Params)
๐ฅ Qwen/Qwen2.5-3B = 78.44%
Example WhirlwindAI/Arithmetic-SLM =
0.5 * 0.5 = 0.25 โ
105 + 45 / 8 = 110 โ
(132 / 12) + (46 - 15) = 42 โ
(10 + 28) * 3 = 114 โ
1 * (16 + 28) = 44 โ
(21 + 27) * (14 - 7) = 336 โ
Bench =
AxiomicLabs/ArithMark-2.0
DataSet =
WhirlwindAI/Arithmetic
By Science AND FOR SCIENCE <3
WhirlwindAI/Arithmetic-SLM
WhirlwindAI/arithmetic-slm
๐ Leaderboard ArithMark-2 ๐
๐ฅ Qwen/Qwen2.5-Math-1.5B = 82.08%
๐ฅ WhirlwindAI/Arithmetic-SLM = 78.60% (31.7M Params)
๐ฅ Qwen/Qwen2.5-3B = 78.44%
Example WhirlwindAI/Arithmetic-SLM =
0.5 * 0.5 = 0.25 โ
105 + 45 / 8 = 110 โ
(132 / 12) + (46 - 15) = 42 โ
(10 + 28) * 3 = 114 โ
1 * (16 + 28) = 44 โ
(21 + 27) * (14 - 7) = 336 โ
leaderboard = """
| Model | Params | Score |
|----------------------------------|--------------|-----------|
| Qwen/Qwen2.5-Math-1.5B | 1.54B | 82.08% |
| WhirlwindAI/Arithmetic-SLM | 31.70M | 78.60% | <=
| Qwen/Qwen2.5-3B | 3.09B | 78.44% |
| Qwen/Qwen2.5-1.5B | 1.54B | 77.72% |
| Qwen/Qwen2.5-Coder-1.5B | 1.54B | 74.88% |
| HuggingFaceTB/SmolLM2-1.7B | 1.71B | 66.12% |
| Qwen/Qwen2.5-0.5B | 494M | 63.04% |
| facebook/MobileLLM-R1-140M-base | 140M | 53.88% |
| SupraLabs/Supra-50M-Base | 52M | 27.12% |
"""Bench =
AxiomicLabs/ArithMark-2.0
DataSet =
WhirlwindAI/Arithmetic
By Science AND FOR SCIENCE <3
- 3 replies
emanuelaborosย
authored a
paper 25 days ago
Shrijanagainย
posted an update 25 days ago
Post
192
Welcome Researcher and Developers!
SKT AI Labs, we are pushing the boundaries of AI architecture and researchโand today, we are thrilled to open our doors to the global research community!
โWe warmly welcome researchers, developers, and AI enthusiasts to join us and contribute to our R&D efforts.
โ๐งช What You Can Explore:
We invite you to experiment with our WMF (Weight Manifold Fusion) technology. You can test this high-dimensional fusion technique on smaller models to gain a deeper understanding of its behavior and token convergence.
---------- CHECK OUT:
SPACE : SKT-NRS/RD
EXPERIMENT : https://huggingface.co/sKT-Ai-Labs/SKT-SURYA-H
DIRECT TO MAIN DISCUSSION : SKT-NRS/RD#1
โ๐ค Your Feedback Shapes the Future :
โIf it works: Fantastic! Share your results with us and contribute directly to the core vision of SKT AI Labs.
โIf it doesn't work: No problem at all! Your critical feedback is just as valuable to us. Every experiment and anomaly helps us refine this architecture to make it more stable and robust.
โWe firmly believe that true innovation stems from community collaboration and transparent testing. Let's build the future of advanced AI together. Your ideas, test results, and feedback are always welcome!
You Can Still Research and Development On WMF Only SKT-SURYA-H Model is Dismissed.
โLet's innovate and build together! ๐ก
SKT AI Labs, we are pushing the boundaries of AI architecture and researchโand today, we are thrilled to open our doors to the global research community!
โWe warmly welcome researchers, developers, and AI enthusiasts to join us and contribute to our R&D efforts.
โ๐งช What You Can Explore:
We invite you to experiment with our WMF (Weight Manifold Fusion) technology. You can test this high-dimensional fusion technique on smaller models to gain a deeper understanding of its behavior and token convergence.
---------- CHECK OUT:
SPACE : SKT-NRS/RD
EXPERIMENT : https://huggingface.co/sKT-Ai-Labs/SKT-SURYA-H
DIRECT TO MAIN DISCUSSION : SKT-NRS/RD#1
โ๐ค Your Feedback Shapes the Future :
โIf it works: Fantastic! Share your results with us and contribute directly to the core vision of SKT AI Labs.
โIf it doesn't work: No problem at all! Your critical feedback is just as valuable to us. Every experiment and anomaly helps us refine this architecture to make it more stable and robust.
โWe firmly believe that true innovation stems from community collaboration and transparent testing. Let's build the future of advanced AI together. Your ideas, test results, and feedback are always welcome!
You Can Still Research and Development On WMF Only SKT-SURYA-H Model is Dismissed.
โLet's innovate and build together! ๐ก
Post
158
Decades before the modern scaling laws, this paper showed that neural networks behavior under scale follows remarkably predictable laws.
In 1993, researchers at Bell Labs were grappling with a constraint that feels entirely familiar (and contemporary): datasets were outgrowing the available hardware, and training a model to the end was becoming too expensive. To evaluate an architectural tweak to a state-of-the-art model (at the time it was LeNet) on 60,000 samples meant burning up to three weeks of compute time.
To save compute, people would train candidate architectures on small subsets of the data, assuming that the top performer at small scale would remain the top performer at full scale. But with our future wisdom, we know this is not the case.
In "Learning Curves: Asymptotic Values and Rate of Convergence (NeurIPS 93)", using insights from statistical mechanics, they proposed a practical and principled method for predicting the performance of classifiers trained on large datasets (at the time, models were assumed to be large enough). The method was based on a simple power-law modeling of the expected training and test errors.
It is often noted that many of today's breakthroughs in AI and deep learning are actually decades-old concepts that simply lacked the computational power to be tested at the time. While there is some truth to that, it highlights a more valuable lesson: there is immense worth in revisiting early literature and reflecting on foundational ideas we may have prematurely left behind.
So, go explore and find your own inspiration. The current trend has enough champions already!
In 1993, researchers at Bell Labs were grappling with a constraint that feels entirely familiar (and contemporary): datasets were outgrowing the available hardware, and training a model to the end was becoming too expensive. To evaluate an architectural tweak to a state-of-the-art model (at the time it was LeNet) on 60,000 samples meant burning up to three weeks of compute time.
To save compute, people would train candidate architectures on small subsets of the data, assuming that the top performer at small scale would remain the top performer at full scale. But with our future wisdom, we know this is not the case.
In "Learning Curves: Asymptotic Values and Rate of Convergence (NeurIPS 93)", using insights from statistical mechanics, they proposed a practical and principled method for predicting the performance of classifiers trained on large datasets (at the time, models were assumed to be large enough). The method was based on a simple power-law modeling of the expected training and test errors.
It is often noted that many of today's breakthroughs in AI and deep learning are actually decades-old concepts that simply lacked the computational power to be tested at the time. While there is some truth to that, it highlights a more valuable lesson: there is immense worth in revisiting early literature and reflecting on foundational ideas we may have prematurely left behind.
So, go explore and find your own inspiration. The current trend has enough champions already!
- 2 replies
Post
434
SPROG-9M โ a 9.37M parameter model trained from scratch to solve GSM8K-style math without using an LLM at inference.
The model, codelion/sprog-9m, predicts symbolic programs over number slots, then a deterministic executor does the arithmetic. With a simple verifier, it reaches ~11.8% on GSM8K test.
We also released the dataset: codelion/gsm8k-synth, 117K validated synthetic GSM8K-style problems.
Tiny model, no pretraining, no LLM at inference, runs on a laptop.
The model, codelion/sprog-9m, predicts symbolic programs over number slots, then a deterministic executor does the arithmetic. With a simple verifier, it reaches ~11.8% on GSM8K test.
We also released the dataset: codelion/gsm8k-synth, 117K validated synthetic GSM8K-style problems.
Tiny model, no pretraining, no LLM at inference, runs on a laptop.
Shrijanagainย
posted an update 29 days ago
Post
206
๐ Big News for the AI Community! ๐ฅ
Weโre excited to release NRS_QWEN_MYTHOS_1M โ a powerful reasoning model built on Qwen 3.5 9B!
At SKT AI LABS, weโve supercharged this 9B model with our proprietary Neural Reasoning System (NRS) to deliver next-level performance.
๐ฅ Why This Model is a Game-Changer:
โ 100x Reasoning Capacity โ Exceptional deep logical thinking and complex problem-solving
โ 1 Million Token Context โ Perfect for massive codebases, long documents, and multi-turn agentic workflows
โ Advanced Thinking Mode โ Native <think> tags for true step-by-step Chain-of-Thought reasoning
โ Tool-Use Ready โ Optimized for Python execution, Web Search, and self-correction
โ Blazing Fast โ Runs smoothly on consumer GPUs like RTX 3090/4090
Technical Highlights:
Base: Qwen 3.5 9B
Tuning: NRS-specific high-quality reasoning data
Context: 1M Tokens (YaRN Scaling)
License: NRS DOCS
Whether youโre a developer building coding agents, a researcher working with long-context data, or someone who loves powerful reasoning โ this model is built for you.
๐ Try it now on Hugging Face:
SKT-NRS/NRS_QWEN_MYTHOS_1M
Drop a comment: What will you build with it first? ๐
#AI #OpenSource #LLM #Qwen #ReasoningModel #HuggingFace #NewModel #AICommunity
Weโre excited to release NRS_QWEN_MYTHOS_1M โ a powerful reasoning model built on Qwen 3.5 9B!
At SKT AI LABS, weโve supercharged this 9B model with our proprietary Neural Reasoning System (NRS) to deliver next-level performance.
๐ฅ Why This Model is a Game-Changer:
โ 100x Reasoning Capacity โ Exceptional deep logical thinking and complex problem-solving
โ 1 Million Token Context โ Perfect for massive codebases, long documents, and multi-turn agentic workflows
โ Advanced Thinking Mode โ Native <think> tags for true step-by-step Chain-of-Thought reasoning
โ Tool-Use Ready โ Optimized for Python execution, Web Search, and self-correction
โ Blazing Fast โ Runs smoothly on consumer GPUs like RTX 3090/4090
Technical Highlights:
Base: Qwen 3.5 9B
Tuning: NRS-specific high-quality reasoning data
Context: 1M Tokens (YaRN Scaling)
License: NRS DOCS
Whether youโre a developer building coding agents, a researcher working with long-context data, or someone who loves powerful reasoning โ this model is built for you.
๐ Try it now on Hugging Face:
SKT-NRS/NRS_QWEN_MYTHOS_1M
Drop a comment: What will you build with it first? ๐
#AI #OpenSource #LLM #Qwen #ReasoningModel #HuggingFace #NewModel #AICommunity
Post
299
It has been more than a decade now since the knowledge distillation paper came out.
Knowledge Distillation (KD) is one of my favorite topics, but I have to confess that I'm not a huge fan of the term because I find it confusing (or at least, it has became so over time).
The idea behind KD is not novel; it was there almost a decade before the paper came out (and arguably even a decade before that, back to 1990-91). But this paper is the one that clicked, the one that made the topic much more popular and introduced it to a broader audience.
First, the timing and the authors played a big role: we have Geoffrey Hinton, Oriol Vinyals, and Jeff Dean here. And second, Geoffrey Hinton is really good at idea branding: Model compression?! No, no, no! Let's call it "Knowledge Distillation" and use evocative terms such as "Dark Knowledge" to describe what is being transferred.
It's a great name, but as time has passed, the term became a bit of a relic. KD is no longer solely about compression (KD used to be introduced as a method for model compression, but now model compression is just one application of KD). And the other thing is that the word "distillation" implies some sort of potency here, that the student is somehow more powerful than the teacher, which is not the case (but many counterarguments could be made, for example, more powerful compared to another model trained with no teacher)
Nevertheless, the paper is incredibly well-written, short, and fun to read. It's one of few papers that I read several times. Check it out, and maybe share your thoughts on the topic with us here!
If you had to choose another name for Knowledge Distillation, what would it be?
Knowledge Distillation (KD) is one of my favorite topics, but I have to confess that I'm not a huge fan of the term because I find it confusing (or at least, it has became so over time).
The idea behind KD is not novel; it was there almost a decade before the paper came out (and arguably even a decade before that, back to 1990-91). But this paper is the one that clicked, the one that made the topic much more popular and introduced it to a broader audience.
First, the timing and the authors played a big role: we have Geoffrey Hinton, Oriol Vinyals, and Jeff Dean here. And second, Geoffrey Hinton is really good at idea branding: Model compression?! No, no, no! Let's call it "Knowledge Distillation" and use evocative terms such as "Dark Knowledge" to describe what is being transferred.
It's a great name, but as time has passed, the term became a bit of a relic. KD is no longer solely about compression (KD used to be introduced as a method for model compression, but now model compression is just one application of KD). And the other thing is that the word "distillation" implies some sort of potency here, that the student is somehow more powerful than the teacher, which is not the case (but many counterarguments could be made, for example, more powerful compared to another model trained with no teacher)
Nevertheless, the paper is incredibly well-written, short, and fun to read. It's one of few papers that I read several times. Check it out, and maybe share your thoughts on the topic with us here!
If you had to choose another name for Knowledge Distillation, what would it be?
- 6 replies
Post
2669
What if you could train a model on just 10 images instead of 60,000 and still get close to the same performance?
Traditional machine learning requires thousands, even millions, of data points to achieve high accuracy. But what if we could "distill" the entire dataset into just a few synthetic samples?
This is what Dataset Distillation offers. Unlike traditional knowledge distillation, we keep the model fixed and distill the knowledge contained in a massive training set into a tiny set of synthetic distilled images.
The goal is to train a model on this ultra-small set and achieve performance that almost matches what the same model would get when trained on the massive original dataset.
For example, training on only 10 distilled MNIST images (this is equivalent to a single image per class) yields 94% accuracy, compared to 99% when training on the full 60,000 images.
Interestingly, these distilled images look significantly different (as you can see in the image below) from natural images because they are optimized for model training rather than for matching the correct data distribution.
But that's not all.
Most importantly, this same method opens the door to a potent form of data poisoning. Because distilled images are specifically optimized for rapid learning, an attacker can create a tiny set of adversarial distilled images to cause a well-trained model to forget or misclassify a specific category.
What I find fascinating about dataset distillation is this: it mimics human-like learning by letting a model grasp a concept from a single example, but it does so using alien synthetic images that mean absolutely nothing to a human eye!
What about you? What are your thoughts on it?
Traditional machine learning requires thousands, even millions, of data points to achieve high accuracy. But what if we could "distill" the entire dataset into just a few synthetic samples?
This is what Dataset Distillation offers. Unlike traditional knowledge distillation, we keep the model fixed and distill the knowledge contained in a massive training set into a tiny set of synthetic distilled images.
The goal is to train a model on this ultra-small set and achieve performance that almost matches what the same model would get when trained on the massive original dataset.
For example, training on only 10 distilled MNIST images (this is equivalent to a single image per class) yields 94% accuracy, compared to 99% when training on the full 60,000 images.
Interestingly, these distilled images look significantly different (as you can see in the image below) from natural images because they are optimized for model training rather than for matching the correct data distribution.
But that's not all.
Most importantly, this same method opens the door to a potent form of data poisoning. Because distilled images are specifically optimized for rapid learning, an attacker can create a tiny set of adversarial distilled images to cause a well-trained model to forget or misclassify a specific category.
What I find fascinating about dataset distillation is this: it mimics human-like learning by letting a model grasp a concept from a single example, but it does so using alien synthetic images that mean absolutely nothing to a human eye!
What about you? What are your thoughts on it?
- 2 replies
VedantPadwalย
authored a
paper about 1 month ago
UVSKKRย
authored a
paper about 1 month ago
UVSKKRย
submitted a
paper to Daily Papers about 2 months ago
Post
4991
It was supposed to be a failed experiment. Instead, it led to the discovery of one of the most intriguing phenomena in neural networks, simply because a researcher forgot to turn it off and left it running....for a week!
In 2022, researchers at OpenAI were studying how neural networks generalize from their training data. For this task, they were training small transformer models to perform modular arithmetic.
The thing is, neural networks are weird. When a model has an abundance of parameters (like neural nets), it can easily overfit. It essentially memorizes its training data, scoring a perfect 100% accuracy when tested on it, but remains completely clueless when faced with any new instances not present in the training set (close to 0 accuracy). It is like memorizing 1 + 2 = 3 without understanding the concept of addition, so if 2 + 3 wasn't in the training set, the model fails miserably!
Usually, when a model overfits like this, people just cut their losses, turn off the experiment, and move on with their lives.
But sometimes they forget. And that is exactly what happened to our researchers at OpenAI. A week later, they checked back in, and a miracle had happened!
They discovered Grokking (And no, this has nothing to do with xAI's Grok , the term was originally coined by sci-fi author Robert Heinlein to mean understanding something so deeply that it becomes part of you). Grokking is when a neural network suddenly and abruptly learns to generalize long after it has overfitted. Just take a look at the graph in the image below!
Spooky, right! I told you neural nets are weird!
In 2022, researchers at OpenAI were studying how neural networks generalize from their training data. For this task, they were training small transformer models to perform modular arithmetic.
The thing is, neural networks are weird. When a model has an abundance of parameters (like neural nets), it can easily overfit. It essentially memorizes its training data, scoring a perfect 100% accuracy when tested on it, but remains completely clueless when faced with any new instances not present in the training set (close to 0 accuracy). It is like memorizing 1 + 2 = 3 without understanding the concept of addition, so if 2 + 3 wasn't in the training set, the model fails miserably!
Usually, when a model overfits like this, people just cut their losses, turn off the experiment, and move on with their lives.
But sometimes they forget. And that is exactly what happened to our researchers at OpenAI. A week later, they checked back in, and a miracle had happened!
They discovered Grokking (And no, this has nothing to do with xAI's Grok , the term was originally coined by sci-fi author Robert Heinlein to mean understanding something so deeply that it becomes part of you). Grokking is when a neural network suddenly and abruptly learns to generalize long after it has overfitted. Just take a look at the graph in the image below!
Spooky, right! I told you neural nets are weird!
- 4 replies
Post
1030
Human brains don't recreate every pixel to understand the world!
Most current models in genomics, proteomics, and single-cell transcriptomics rely on generative objectives like masked language modeling or next token prediction. While effective, these architectures waste significant capacity reconstructing raw, noisy sequence details that may not carry functional biological meaning.
But a promising, more efficient alternative is emerging: Joint-Embedding Predictive Architecture (JEPA)
Originally introduced by Yann LeCun for computer vision, JEPA is a non-generative, self-supervised learning (SSL) framework. Instead of predicting raw inputs, it operates as a world model that predicts abstract semantic embeddings in latent space.
Recently, the JEPA framework (and its more efficient LeJEPA variant) has been adapted into the biological sciences to develop performing foundation models and to improve on already existing ones.
It's interesting how each adaptation modified and tailored JEPA to suit its specific biological domain, whether by experimenting with different backbones or complementing the objective with other loss terms.
For example, JEPA-DNA and ProteinJEPA used JEPA as a continual pre-training framework to enhance existing foundation models without training from scratch, while Cell-JEPA and JEPA-DNA employed a hybrid objective that combines the JEPA loss with a traditional language modeling loss.
The article below provides an overview of these implementations, along with others that came out this year. As always, your thoughts and feedback are welcome and highly appreciated!
Link to the article is in the first comment ๐
Most current models in genomics, proteomics, and single-cell transcriptomics rely on generative objectives like masked language modeling or next token prediction. While effective, these architectures waste significant capacity reconstructing raw, noisy sequence details that may not carry functional biological meaning.
But a promising, more efficient alternative is emerging: Joint-Embedding Predictive Architecture (JEPA)
Originally introduced by Yann LeCun for computer vision, JEPA is a non-generative, self-supervised learning (SSL) framework. Instead of predicting raw inputs, it operates as a world model that predicts abstract semantic embeddings in latent space.
Recently, the JEPA framework (and its more efficient LeJEPA variant) has been adapted into the biological sciences to develop performing foundation models and to improve on already existing ones.
It's interesting how each adaptation modified and tailored JEPA to suit its specific biological domain, whether by experimenting with different backbones or complementing the objective with other loss terms.
For example, JEPA-DNA and ProteinJEPA used JEPA as a continual pre-training framework to enhance existing foundation models without training from scratch, while Cell-JEPA and JEPA-DNA employed a hybrid objective that combines the JEPA loss with a traditional language modeling loss.
The article below provides an overview of these implementations, along with others that came out this year. As always, your thoughts and feedback are welcome and highly appreciated!
Link to the article is in the first comment ๐
- 3 replies
Post
432
๐ __Monostep v1__ is up โ[Monostep-v1 Demo}( wop/Cosmos-T2-Chat)
A tiny (~16.6M) experimental model that predicts 4 tokens per forward pass instead of one. A Transformer trunk pools the prompt into a single vector, then 4 sequential "slot" heads emit a block of tokens left-to-right โ a lightweight take on multi-token prediction.
Trained on GSM8K (GPT-2 tokenizer, 10 epochs). It's small and rough โ answers are often wrong โ but it's a fun little testbed for block decoding. Weights, config, training curves, and a self-contained inference snippet are all in the repo.
Also wired into the Cosmos T2-Accelerate chat demo, where it streams those 4-token blocks live. ๐งช
#multitokenprediction #gsm8k #smallmodels
A tiny (~16.6M) experimental model that predicts 4 tokens per forward pass instead of one. A Transformer trunk pools the prompt into a single vector, then 4 sequential "slot" heads emit a block of tokens left-to-right โ a lightweight take on multi-token prediction.
Trained on GSM8K (GPT-2 tokenizer, 10 epochs). It's small and rough โ answers are often wrong โ but it's a fun little testbed for block decoding. Weights, config, training curves, and a self-contained inference snippet are all in the repo.
Also wired into the Cosmos T2-Accelerate chat demo, where it streams those 4-token blocks live. ๐งช
#multitokenprediction #gsm8k #smallmodels