Add THL-150 model architecture implementation

[ErebusTN]

Description

• Implement core THL-150 architecture with sliding window attention
• Add configuration with RoPE embedding support
• Include fast/slow tokenizers using BPE
• Implement all model heads (CausalLM, Classification, QA, TokenClass)
• Add dynamic RoPE scaling and GQA support
• Validate attention mask generation for long sequences

Motivation

This implementation enables:

• Efficient 32k context window processing
• Flexible attention mechanisms (sliding window + full attention hybrid)
• Compatibility with HF Transformers pipelines
• Modern architecture features like Grouped Query Attention

Context

• Built for long-context NLP tasks
• Implements architecture similar to LLaMA with sliding window extensions
• Designed for easy integration with existing HF ecosystems

Dependencies

• Requires PyTorch >= 2.0
• Recommends flash-attn >= 2.3 for optimal performance

Checklist

• Updated documentation in docstrings
• Verified config validation
• Tests will be added in subsequent PR (current focus on core implementation)

Notes

• Implements EGen License v0.1
• Initial release focused on base architecture
• Special thanks to the reviewer for consultation

Authored-by: @ErebusTN [[email protected]]

[Rocketknight1]

hi @ErebusTN, we usually don't add architectures to the library until there's an existing pretrained model. Is there a relevant model repo or paper anywhere?

[ErebusTN]

Hi @Rocketknight1 i working on a model EGen V1 and it's my phD project and i going to update it frequently it's just in the first steps as v0.1 I still have a lot to work on to improve it so I haven't uploaded it yet but stay tuned the next update will be the first and I will upload the model to be open source

[Rocketknight1]

Cool! One thing we'd advise is that models can be uploaded as custom code using the steps here. This will let you share the model immediately, and it'll work exactly the same as a library model (except that users will need to set trust_remote_code=True)

This can be a lot faster than actually getting a PR into transformers, and it's a good way to validate the model and get users, which will help speed up the PR later!