Bug Byte puzzle here - https://bit.ly/4bnlcb9 - and apply to Jane Street programs here - https://bit.ly/3JdtFBZ (episode sponsor). More info in full descript...
A free running cellular automaton (CA) approach in hardware would work, but each cell would be a much souped up SRAM cell, the interactions would be all local and 2D. Considering Cerebras is 40 G SRAM on the 300 mm WSI and is about at the cooling limit I’m afraid you do not have 5 orders of magnitude. Perhaps reversible spintronics can help with the power draw, but you still have to splat a higher dimensional network so not just local interactions into a 2D array.
Current research points to memristors, which can work both as memory cells, and as weights in a n×m grid representing a fully connected n->m layer that executes in 1 clock. I forgot which company was showing prototypes since pre-covid… and now Google is so full of wannabes that I can’t seem to find it, oh well.
Cerebras is at the limit of SRAM, that’s true.
Spintronics could be the next step, but seems to be way less ready for production.
Higher dimensionality would be nice, but even at 2D, being able to push multiple processes at once, through multiple n×m layers, would already give those 5 orders of magnitude, at least for inference. Since training also involves an inference step, it would speed that too, just not as much.
Self-training would be the next step after that… I don’t think I’ve seen research in that regard, but maybe I’ve just missed it.
A free running cellular automaton (CA) approach in hardware would work, but each cell would be a much souped up SRAM cell, the interactions would be all local and 2D. Considering Cerebras is 40 G SRAM on the 300 mm WSI and is about at the cooling limit I’m afraid you do not have 5 orders of magnitude. Perhaps reversible spintronics can help with the power draw, but you still have to splat a higher dimensional network so not just local interactions into a 2D array.
Current research points to memristors, which can work both as memory cells, and as weights in a n×m grid representing a fully connected n->m layer that executes in 1 clock. I forgot which company was showing prototypes since pre-covid… and now Google is so full of wannabes that I can’t seem to find it, oh well.
Cerebras is at the limit of SRAM, that’s true.
Spintronics could be the next step, but seems to be way less ready for production.
Higher dimensionality would be nice, but even at 2D, being able to push multiple processes at once, through multiple n×m layers, would already give those 5 orders of magnitude, at least for inference. Since training also involves an inference step, it would speed that too, just not as much.
Self-training would be the next step after that… I don’t think I’ve seen research in that regard, but maybe I’ve just missed it.