Skip to content

Commit

Permalink
Typo fix
Browse files Browse the repository at this point in the history
  • Loading branch information
simveit committed Apr 7, 2024
1 parent 5aa4209 commit 1dd193d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion _posts/2024-04-07-attention.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ We can combine these two masks by using a simple logical `OR`, i.e. we mask out
One more important point is the possibility of parallelizing to leverage multiple chips.
In case of attention it turns out that parallelizing is not difficult to reason about.

- We have `BATCH` dimension. Each batch is processed idepently, so we can always parallelize along this dimension.
- We have `BATCH` dimension. Each batch is processed independently, so we can always parallelize along this dimension.

- We have `HEADS` dimension. By definition each head is independent from the other heads so we can parallelize as well.

Expand Down

0 comments on commit 1dd193d

Please sign in to comment.