Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add paged attention support #1355

Open
wants to merge 50 commits into
base: main
Choose a base branch
from
Open

Add paged attention support #1355

wants to merge 50 commits into from

Conversation

cyanguwa
Copy link
Collaborator

@cyanguwa cyanguwa commented Dec 4, 2024 •
edited
Loading

Description

This PR adds paged attention support for FusedAttention, FlashAttention, and UnfusedDotProductAttention.

  • KV cache is maintained in 'bshd' format, and it supports FP32/FP16/BF16, not FP8 yet
  • Context parallelism is not supported with KV cache
  • FusedAttention and UnfusedDotProductAttention support page_size >= 16, and FlashAttention supports page_size >= 256
  • All backends support both pure generation and mixed generation/context in the batch
  • FlashAttention supports paged attention through flash_attn_varlen_func, not flash_attn_with_kvcache, due to some numerical issues
  • UnfusedDotProductAttention supports paged attention by converting paged cache tensors to non-paged, before attention

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refractor

Changes

Please list the changes introduced in this PR:

  • Add paged attention support for FusedAttention, FlashAttention, and UnfusedDotProductAttention

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

cyanguwa and others added 10 commits December 3, 2024 17:01
@cyanguwa
add paged attention; test_kv_cache_accuray and test_paged_attn pass
44f6ff2 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
remove unnecessary change from last commit
06605e5 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
test_fused_attn pass
0b2eb88 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
Merge branch 'main' into paged_attention
d243b79 
Signed-off-by: Charlene Yang <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
b0a5da4 
for more information, see https://pre-commit.ci
@cyanguwa
remove unnecessary import in test_numerics
b4efd71 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
add license for test
e637a07 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
fix lint
767c8f5 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
add to L0 test
a3bb14f 
Signed-off-by: Charlene Yang <[email protected]>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
d65933c 
for more information, see https://pre-commit.ci
@cyanguwa
Copy link
Collaborator Author

cyanguwa commented Dec 4, 2024

/te-ci pytorch L0

@cyanguwa
Copy link
Collaborator Author

cyanguwa commented Jan 6, 2025

/te-ci pytorch L0

sudhakarsingh27
sudhakarsingh27 reviewed Jan 28, 2025
View reviewed changes
transformer_engine/pytorch/kv_cache_manager_non_paged.py Outdated
v_cache: torch.Tensor
The value cache tensor containing previous and the current tokens
"""
k_cache, v_cache = self.cache[layer_number]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
k_cache, v_cache = self.cache[layer_number]
assert layer_number in self.cache
k_cache, v_cache = self.cache[layer_number]

transformer_engine/pytorch/attention.py Outdated
def __init__(
self,
max_batch_size: int,
max_seqlen_kv: int,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

corresponding docstring says max_sequence_length, we should change one of those

transformer_engine/pytorch/kv_cache_manager_non_paged.py Outdated
seq_s = self.sequences[seq] - step_dict[seq]
seq_e = self.sequences[seq]
if qkv_format == "bshd":
new_k_cache[i, seq_s:seq_e, :, :] = k[i, : step_dict[seq], :, :]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k[i, : step_dict[seq], :, :]

k isn't supposed to have any tokens beyond step_dict[seq], right?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for v

transformer_engine/pytorch/kv_cache_manager_non_paged.py Outdated
Comment on lines 128 to 129
seq_s = self.sequences[seq] - step_dict[seq]
seq_e = self.sequences[seq]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These could potentially be moved into a method since this could be reused from outside like when getting the start positions of RoPE embeddings application

@cyanguwa
WIP: non-paged, thd, no CG
f9bd83c 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: padding + BRCM
ba5e333 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: restructure IP, clean up
bcef6b3 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: fix non-CG, fused
f3975f0 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: fix last commit
125548c 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: unfused, non-CG
9bf3204 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: flash-attn, non-CG
3060892 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: flash_attn_with_kvcache
11f15fd 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: thd_bshd_bshd
1c31b68 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa cyanguwa force-pushed the paged_attention branch 2 times, most recently from 6fcad33 to f5b91c6 Compare February 21, 2025 23:33
@cyanguwa
WIP: fix last commit
7331a4c 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: fix 1c31b68
0341de7 
Signed-off-by: Charlene Yang <[email protected]>
@cyanguwa
WIP: add bshd_2sbhd, sbhd_2bshd
6bd61a7 
Signed-off-by: Charlene Yang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sudhakarsingh27 sudhakarsingh27 sudhakarsingh27 left review comments

@ptrendx ptrendx Awaiting requested review from ptrendx

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cyanguwa @sudhakarsingh27