Benchmaxxed
The model fails at any real-world tasks. There are many things your model simply does not know or understand at all.
The model fails at any real-world tasks. There are many things your model simply does not know or understand at all.
What did you ask it to do? and what do you see as different?
The model fails at any real-world tasks. There are many things your model simply does not know or understand at all.
What did you ask it to do? and what do you see as different?
for example "what is inmp441?" (is mems micro). Any model, even a 4B, knows this. In other words, it is impossible to write programs for the ESP32 without websearching. The model hasn’t been trained for this; it lacks the necessary knowledge. I also recommend watching this video: https://www.youtube.com/watch?v=BjRTtHCJCgM . This suggests that the model was fine-tuned to pass the tests.
Thanks for the detailed feedback of the INMP441 example.
We tested this question: Nanbeige4.3-33B-Base answered it correctly, and Nanbeige4.3-33B answered it correctly when used in a harness with web search.
Agentic capabilities were a major focus of this release, but we recognize that the model should also handle factual questions better without search. We’ll work to improve that balance in future releases.
One more thing: please keep the bad cases coming and we want to find and fix them.
But we’re definitely not just optimizing for benchmarks. 🙂
use ProgramBench to test where it has no access to the internet to cheat....model SUCKS.
use ProgramBench to test where it has no access to the internet to cheat....model SUCKS.
At least it doesn't hallucinate and actually try to fact check. That's better than inventing an answer confidently. That's dangerous stuff
Damn bro, cope harder.
I don't use this model bro. I ain't coping lol just stating a fact. Just trying to educate the illiterate
I mean we are paying 33b worth of vram for more knowledge than 9b models. Better reasoning isn't the only thing we expect from bigger models. More knowledge is right to be expected too.
I mean we are paying 33b worth of vram for more knowledge than 9b models. Better reasoning isn't the only thing we expect from bigger models. More knowledge is right to be expected too.
It probably has more knowledge. Just not on the field you're expecting.
I'm pretty good at math but I eat shit on geography. But that doesn't make me dumb.
Also, the model is free to skip 😄. Doesn't quite work for you? Also free to be nice, leave a constuctive feedback and move on. You chose to test the model, that's on you.
I'm not sure I agree with the idea that a 33B model should always have X information. You're going to have to make tradeoffs at any size of model, and 33B is at the very small end. It's a good size for a local model, but it's 1/10th the size of a moderately sized open weight model, and 1/100/th the size of foundation models. If the benchmark performance holds up in real-world performance I absolutely expect a loss of detailed knowledge in trade for a model at this size. There is no free lunch.
Considering the fact that there are plenty of facts that YOU would need to look up because you don't know everything, I think it's unreasonable to expect a small local model to know everything. IMO it's a really, really good trade for the model to look easily searchable information up instead of trying to store the entire internet in its weights. Even if you want to do your ESP32 programming without access to the internet, it would be pretty trivial to compile a local library of documentation for the model to reference.
IMO this is how we should be using small agents, because coding and tool calling capabilities cannot be supplemented in the same way that knowledge can.
I wouldn't mind it knowing as few circumstantial facts as possible (that will probably become obsolete), rather pack more intelligence into it and have it look up what it needs. It sucks when your agent uses something thats not supported anymore or doesn't use a newer feature because it assumed everything it knows is up to date
I mean we are paying 33b worth of vram for more knowledge than 9b models. Better reasoning isn't the only thing we expect from bigger models. More knowledge is right to be expected too.
It probably has more knowledge. Just not on the field you're expecting.
I'm pretty good at math but I eat shit on geography. But that doesn't make me dumb.Also, the model is free to skip 😄. Doesn't quite work for you? Also free to be nice, leave a constuctive feedback and move on. You chose to test the model, that's on you.
What was destructive about it? Just because you don't agree doesn't mean my opinions are bad.
I mean we are paying 33b worth of vram for more knowledge than 9b models. Better reasoning isn't the only thing we expect from bigger models. More knowledge is right to be expected too.
It probably has more knowledge. Just not on the field you're expecting.
I'm pretty good at math but I eat shit on geography. But that doesn't make me dumb.Also, the model is free to skip 😄. Doesn't quite work for you? Also free to be nice, leave a constuctive feedback and move on. You chose to test the model, that's on you.
What was destructive about it? Just because you don't agree doesn't mean my opinions are bad.
I was really talking about this one. You just got in the crossfire
use ProgramBench to test where it has no access to the internet to cheat....model SUCKS.
Damn bro, cope harder.