SOTA-II-WS24

Course materials for the SOTA-II course.

week 1

Group A

Topic: Data Sources

• Link: https://zoom-x.de/wb/doc/FgnLCEUMT9-GEkorLWWUxw/p/165815171781
• PDF: Data Sources

Group B

Topic: Filtering + De-Duplication

• Link: https://zoom-x.de/wb/doc/yVzHiJ0IRveHptYXsInz5Q

Group C

Topic: Scheduling

• PDF: Scheduling

Group D

Topic: Tokenizer

• Board: https://zoom-x.de/wb/doc/FgnLCEUMT9-GEkorLWWUxw/p/91485439788

week 2

Group A

Topic: Improving Attention

• Link: https://docs.google.com/presentation/d/1bRbcEWJXuob_3F5cF75SiiUNzQpzoJ7chOCrKhYdkUo/edit?usp=sharing
• PDF: Improving Attention

Group B

Topic: The GPT Familie

• Link: https://zoom-x.de/wb/doc/_iN7rdxoRNq4vrOzzFCGEw

Group C

Topic: The LLAMA Familie

• PDF: The LLAMA Familie

Group D

Topic: Normalization/Positional Encoding

• Board: https://zoom-x.de/wb/doc/M9NB8fkCRieVgkZqs0-RKA/p/128063118675

week 3

Group A

Topic: Mixed Precision Training

• Link: https://docs.google.com/presentation/d/1L69EkPAsCeRFr30z6Z6RodZIKs3f6Bht9tilpTLrFQg/edit?usp=sharing
• PDF: Mixed Precision Training

Group B

Topic: Parallel Training

• Link: https://zoom-x.de/wb/doc/SU41xh3tS0mvszjBEkgs3w

Group C

Topic: Scaling Laws

• PDF: Scaling Laws

Group D

Topic: Optimization

• Board: https://zoom-x.de/wb/doc/KjSvT7XURH6AF6QzlA8m9g/p/119067491936

week 4

Group A

Topic: Supervised Finetuning

• Link: https://docs.google.com/presentation/d/1qvLuo-tFolbqOw3WdLCF5yjYAQWsYYTzk_SmEWVFQqA/edit?usp=sharing
• PDF: Supervised Finetuning

Group B

Topic: Reinforcement Learning from Human Feedback

• Link: https://zoom-x.de/wb/doc/VXfKftCSRfuxEglbxkTNmQ

Group C

Topic: Self-Instruct

• PDF: Self-Instruct

Group D

Topic: Formatted Instruction Tuning

• Board: https://zoom-x.de/wb/doc/BmikYbyYSGaiM6hRp1bX3A/p/108385592003

week 5

Group A

Topic: Optimizing Prompts

• Link: https://docs.google.com/presentation/d/1FEd9Vfd6C256K6CiIHflzazyXAuP3cmGqrfPuV5CcVg/edit?usp=sharing
• PDF: Optimizing Prompts

Group B

Topic: In Context Learning (ICL)

• Link: https://zoom-x.de/wb/doc/VXfKftCSRfuxEglbxkTNmQ

Group C

Topic: Chain of Thought

• PDF: Chain of Thought

Group D

Topic: Planning

• Board: https://zoom-x.de/wb/doc/xiBaxIEvQDuABgtFUv7hqw/p/111749251518

week 6

Group A

Topic: Vector Database

• Link: https://docs.google.com/presentation/d/1F7OsAIHjl99-t6GfriBwixQ4Ax__US1Hy3hary9Y338/edit?usp=sharing
• PDF: Vector Database

Group B

Topic: RAG enhancement

• Board: https://zoom-x.de/wb/doc/kzohRNHVTfuu8AlY_D9F6A/p/96652574261248

Group C

Topic: RAG enhancement

• PDF: RAG enhancement

Group D

Topic: RAG

• Board: https://zoom-x.de/wb/doc/c_FkuPfLTDuVOWHZHd2Dww/p/110311143885

week 7

Group A

Topic: Quantization

• Link: https://docs.google.com/presentation/d/1ypTE9AEKwCYiv3RoKDdy4zY_NeERKSprwXHNAnEUJHs/edit?usp=sharing
• PDF: Quantization

Group B

Graph-RAG

• Link: Graph-RAG

Group C

Topic: Emerging Capabilities

• PDF: Emerging Capabilities

Group D

Topic: Low Rank Adaption (LoRa)

• Board: https://zoom-x.de/wb/doc/Vvzqj3YOTauSgYJq3s56SA/p/111500866661

week 8

Group A

Topic: Judging Models

• Link: https://docs.google.com/presentation/d/1YbTXZv45tcqlt8NaLGN-Hbi9F3FLbEcEgOXf-7427F4/edit?usp=sharing
• PDF: Judging Models

Group B

Benchmarks

• Link: Benchmarks

Group C

Topic: Benchmarking RAG

• PDF: Benchmarking RAG

Group D

Topic: Benchmarking Coding Models

• Board: https://zoom-x.de/wb/doc/-HtVBAuLSfqqgdOMDTjSsA/p/110806398618

week 9

Group A

Topic: Nerf-Studio

• Link: https://docs.google.com/presentation/d/14YoNxLij5wIA5-ZzY_vV41WKHwgOWRiYnjIBKCOn9VE/edit?usp=sharing
• PDF: Nerf-Studio

Group B

Group C

Topic: Fast NERFS

• PDF: Fast NERFS

Group D

Topic: Basic NeRF

• Board: https://zoom-x.de/wb/doc/NDlKS9sOSX2xkX1FN9Cc1g/p/96835091698

week 10

Group A

Group B

Group C

Group D

Topic: PPO

• Board: https://zoom-x.de/wb/doc/DswHhUjDSbqn7NipirFPlA/p/102011508926