promotedrejected (no real improvement)failed compile or correctnesschampion line
| iter | mean ms | min ms | decision | reason | model |
|---|---|---|---|---|---|
| 1 | n/a | n/a | rejected | compile failed: response contains prose around the code fence | nvidia/nemotron-3.5-lightning-30b-a3b |
| 2 | 76.248 | 74.725 | PROMOTED | mean improved 40.44% (> 2% threshold) | nvidia/nemotron-3.5-lightning-30b-a3b |
| 3 | 75.881 | 74.057 | rejected | mean improved only 0.48% (needs > 2%) | nvidia/nemotron-3.5-lightning-30b-a3b |
| 4 | 76.248 | 74.725 | rejected | mean improved only 0.00% (needs > 2%) | nvidia/nemotron-3.5-lightning-30b-a3b |
| 5 | 75.449 | 73.530 | rejected | mean improved only 1.05% (needs > 2%) | nvidia/nemotron-3.5-lightning-30b-a3b |
| 6 | 76.248 | 74.725 | rejected | mean improved only 0.00% (needs > 2%) | nvidia/nemotron-3.5-lightning-30b-a3b |