Model description

Doge is an architecture that combines the advantages of state-space and self-attention. It solves the problem of self-attention getting lost in long sequences by computing dynamic mask from cached value states using zeroth-order holding. It can also use wsd_scheduler on top of dense weight checkpoints to additionally train a sparsely activated feedforward network expansion layer.

paper: https://arxiv.org/abs/2412.11834

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

Repository: https://github.com/LoserCheems/WonderfulMatrices
Weights: https://huggingface.co/collections/JingzeShi/doge-slm-677fd879f8c4fd0f43e05458