Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
It seems implementing transformers is the new "hello world" of AI assisted coding.
But that still not a breakthrough or anything for such a small active set of parameters that is quantized
Zero surprise it did it
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
https://architectlabs.com/blog/frostbite