This repository contains lecture materials and hands-on tutorials on fine-tuning a clinical domain Large Language Model (LLM).
- [2024-08-10] KoSAIM 2024 Summer School Hands-on Session III. Large Language Model [Slides] [Colab]
- [2024-11-14] KoSAIM 2024 개발자를 위한 의료 AI 심화교육 II [Slides] [Colab]
- [2025-01-11] KSR 2025 AIMC 4기 [Slides] [Colab]
- How to build a clinical domain Large Language Model (LLM)?
- (Large) Language Model Basics
- How to build a (Language) Langauge Model?
- Building an instruction-following LLM in the clinical domain
- Introduction to Asclepius (Gweon and Kim et al., ACL 2024 Findings)
- Hands-on Session: Fine-tuning a clinical domain LLM
- Environment Setup & Colab Practice
- LLM memory layout
- Parameter-Efficient Fine-Tuning: LoRA / QLoRA
- Extended Topics (Optional)
- Prompt Engineering Techniques
- Evaluation Metrics for Clinical LLMs
- https://github.com/huggingface/transformers
- https://github.com/huggingface/trl
- https://github.com/huggingface/accelerate
- https://github.com/huggingface/peft
- https://github.com/bitsandbytes-foundation/bitsandbytes
- https://github.com/starmpcc/Asclepius
- starmpcc/Asclepius-Synthetic-Clinical-Notes
BERTDevlin, Jacob. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018).GPTRadford, Alec, et al. "Improving Language Understanding by Generative Pre-Training."T5Raffel, Colin, et al. "Exploring the limits of transfer learning with a unified text-to-text transformer." Journal of machine learning research 21.140 (2020): 1-67.- Liu, Pengfei, et al. "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing." ACM Computing Surveys 55.9 (2023): 1-35.
GPT-3Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.FLANWei, Jason, et al. "Finetuned language models are zero-shot learners." arXiv preprint arXiv:2109.01652 (2021).LlamaTouvron, Hugo, et al. "Llama: Open and efficient foundation language models." arXiv preprint arXiv:2302.13971 (2023).RLHFOuyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.ChatGPThttps://openai.com/index/chatgpt/Llama 2Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).Llama 3Dubey, Abhimanyu, et al. "The Llama 3 Herd of Models." arXiv preprint arXiv:2407.21783 (2024).DPORafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." Advances in Neural Information Processing Systems 36 (2024).AlpacaTaori, Rohan, et al. "Alpaca: A strong, replicable instruction-following model." Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html 3.6 (2023): 7.AsclepiusKweon, Sunjun, et al. "Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes." arXiv preprint arXiv:2309.00237 (2023).LoraHu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).QLoraDettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in Neural Information Processing Systems 36 (2024).- https://github.com/starmpcc/KAIA-LLM-FT-2024
- Seongsu Bae ([email protected])
- Sujeong Im ([email protected])
This repository is licensed under the MIT License. See the LICENSE file for more details.
- All resources in this repository are provided for research and educational purposes only.
- No clinical or patient-identifiable data is included.
- Use responsibly and check relevant privacy, security, and ethical guidelines before applying in a real clinical setting.
- The contributors of this repository assume no liability for any clinical outcomes or misuse of the provided materials.