Code

P2p comms through Cuda Ipc #3883

Code

Draft

samnordmann wants to merge 60 commits intoNVIDIA:mainNVIDIA/Fuser:mainfrom samnordmann:gpu_comms/add_p2psamnordmann/Fuser:gpu_comms/add_p2pCopy head branch name to clipboard

+1,564-98

Commits

Commits on Oct 23, 2024

• working simple benchmark

  samnordmann

  committed
  2c04df9

  Copy full SHA for 2c04df9

Commits on Oct 25, 2024

• minor

  samnordmann

  committed
  af36cf1

  Copy full SHA for af36cf1

• test script

  samnordmann

  committed
  68b858a

  Copy full SHA for 68b858a

Commits on Oct 28, 2024

• minor

  samnordmann

  committed
  0c3493b

  Copy full SHA for 0c3493b

Commits on Oct 29, 2024

• add nsight profiling

  samnordmann

  committed
  b30b44b

  Copy full SHA for b30b44b

Commits on Oct 31, 2024

• nsight and tl/nccl/ sync mode

  samnordmann

  committed
  0592a13

  Copy full SHA for 0592a13

Commits on Nov 4, 2024

• add cuStreamWriteValue but linkage error

  samnordmann

  committed
  0037b1e

  Copy full SHA for 0037b1e

• multiple pgs

  samnordmann

  committed
  ec71e23

  Copy full SHA for ec71e23

• reenable cuStreamValue32

  samnordmann

  committed
  a15fdfc

  Copy full SHA for a15fdfc

• add tl/cuda and ec/cuda flags in bash test script

  samnordmann

  committed
  6682a33

  Copy full SHA for 6682a33

• add option to unfuse loops

  samnordmann

  committed
  b01f1f4

  Copy full SHA for b01f1f4

Commits on Nov 5, 2024

• add cuda graphs. Only working for NCCL and S1 bc there is a syncStream in nccl

  samnordmann

  committed
  ea7fd37

  Copy full SHA for ea7fd37

Commits on Nov 26, 2024

• write matmul to sliced output

  samnordmann

  committed
  9dddac2

  Copy full SHA for 9dddac2

Commits on Nov 28, 2024

• wip cuStreamWriteValue not working

  samnordmann

  committed
  faf8bbe

  Copy full SHA for faf8bbe

Commits on Dec 2, 2024

• dummy benchmark

  samnordmann

  committed
  a6b5fd7

  Copy full SHA for a6b5fd7

• add pre post comms option

  samnordmann

  committed
  8d927bf

  Copy full SHA for 8d927bf

• add pre post comms option

  samnordmann

  committed
  d9c581c

  Copy full SHA for d9c581c

Commits on Dec 6, 2024

• cleanup test script

  samnordmann

  committed
  bfc7fa6

  Copy full SHA for bfc7fa6

Commits on Jan 8, 2025

• update

  samnordmann

  committed
  1a1138c

  Copy full SHA for 1a1138c

Commits on Jan 16, 2025

• Merge branch 'overlap_bench/first_experiments' of github.com:samnordmann/Fuser into overlap_bench/first_experiments

  samnordmann

  committed
  743185d

  Copy full SHA for 743185d

• Merge branch 'main' of github.com:NVIDIA/Fuser into overlap_bench/first_experiments

  samnordmann

  committed
  a2b1650

  Copy full SHA for a2b1650

• test with stream parallel type and host IR

  samnordmann

  committed
  e037ee5

  Copy full SHA for e037ee5

Commits on Jan 20, 2025

• add support for other dtypes

  samnordmann

  committed
  8328c28

  Copy full SHA for 8328c28

Commits on Jan 22, 2025

• remove trace print

  samnordmann

  committed
  2fecf02

  Copy full SHA for 2fecf02

Commits on Jan 23, 2025

• add stub files

  samnordmann

  committed
  26f1f7a

  Copy full SHA for 26f1f7a

Commits on Jan 24, 2025

• first working example opening cuda ipc handles

  samnordmann

  committed
  03b0147

  Copy full SHA for 03b0147

• adding a non-working example with cudaDeviceCanAccessPeer

  samnordmann

  committed
  7625fab

  Copy full SHA for 7625fab

Commits on Jan 28, 2025

• cleanup

  samnordmann

  committed
  f703abd

  Copy full SHA for f703abd

• AllgatherThroughCudaMemcpyAsync

  samnordmann

  committed
  abf5c17

  Copy full SHA for abf5c17

Commits on Jan 29, 2025

• refactor to expose choice of backend

  samnordmann

  committed
  836d599

  Copy full SHA for 836d599

• add backend type to P2PCommunication

  samnordmann

  committed
  e09dd58

  Copy full SHA for e09dd58

• Merge branch 'main' of https://github.com/NVIDIA/Fuser into gpu_comms/AGMemcpyAsync

  samnordmann

  committed
  2fbea13

  Copy full SHA for 2fbea13

• Merge branch 'overlap_bench/AG_Matmul_with_stream_parallel_type' of github.com:samnordmann/Fuser into gpu_comms/AGMemcpyAsync

  samnordmann

  committed
  d87c8e6

  Copy full SHA for d87c8e6

Commits on Jan 30, 2025

• wip

  samnordmann

  committed
  6371746

  Copy full SHA for 6371746

• working chkpt

  samnordmann

  committed
  b700a31

  Copy full SHA for b700a31

• remove prints

  samnordmann

  committed
  1838d1e

  Copy full SHA for 1838d1e

• working chkpt

  samnordmann

  committed
  21eed4a

  Copy full SHA for 21eed4a

• reenable profiling

  samnordmann

  committed
  f455c70

  Copy full SHA for f455c70

Commits on Jan 31, 2025

• fix cache for ipc handles

  samnordmann

  committed
  5a27b7e

  Copy full SHA for 5a27b7e

Commits on Feb 3, 2025

• synchronize running stream with original stream at the beginning of pipeline for-loop

  samnordmann

  committed
  356feeb

  Copy full SHA for 356feeb

• lint

  samnordmann

  committed
  4c0736a

  Copy full SHA for 4c0736a

Commits on Feb 5, 2025

• wip. The send and recv Expr* need to be matched together for associating the buffer. Need to either use (non-P2P)Communication* or better have a Host Node to explicitely share the handles as someth…

  Show description for 7fca035
  samnordmann

  committed
  7fca035

  Copy full SHA for 7fca035

• working chkpt well prepared for two ranks

  samnordmann

  committed
  371554e

  Copy full SHA for 371554e

Commits on Feb 7, 2025

• change signature of P2Pcomms to accept src and dst

  samnordmann

  committed
  c7c0404

  Copy full SHA for c7c0404

Commits on Feb 11, 2025

• working chkpt with get zcopy

  samnordmann

  committed
  6c20a20

  Copy full SHA for 6c20a20

Commits on Feb 12, 2025

• working checkpt with many ranks

  samnordmann

  committed
  f7409b2

  Copy full SHA for f7409b2

• chkpt non blocking

  samnordmann

  committed
  08f8fe0

  Copy full SHA for 08f8fe0

• harden tests by removing hard syncs

  samnordmann

  committed
  de843bb

  Copy full SHA for de843bb

• use cudaMemcpyAsync

  samnordmann

  committed
  4dc9936

  Copy full SHA for 4dc9936

• clean and lint

  samnordmann

  committed
  4e05609

  Copy full SHA for 4e05609

• Move distributed tensors to separate file

  samnordmann

  committed
  326b683

  Copy full SHA for 326b683

• rename DistributedBuffer to IpcHandle

  samnordmann

  committed
  cf8991c

  Copy full SHA for cf8991c

• working chkpt. Added in the commit the new files that were forgotten before

  samnordmann

  committed
  541fe80

  Copy full SHA for 541fe80

• refactor

  samnordmann

  committed
  a496004

  Copy full SHA for a496004

• minor cleanup

  samnordmann

  committed
  263d95c

  Copy full SHA for 263d95c

• lint

  samnordmann

  committed
  106d295

  Copy full SHA for 106d295

• minor

  samnordmann

  committed
  ed69f75

  Copy full SHA for ed69f75

• Merge branch 'main' of github.com:NVIDIA/Fuser into gpu_comms/add_p2p

  samnordmann

  committed
  5672eca

  Copy full SHA for 5672eca

• minor

  samnordmann

  committed
  929ae0d

  Copy full SHA for 929ae0d

Commits on Feb 13, 2025

• move p2p runtime in separate file

  samnordmann

  committed
  359779d

  Copy full SHA for 359779d