simplification of CI: matrix strategy #3882
Triggered via pull request
February 18, 2025 14:49
Status
Failure
Total duration
10h 49m 39s
Artifacts
42
ci.yaml
on: pull_request
metadata
0s
Matrix: build-amd64-arm64
merge-new-manifest
0s
Matrix: publish-containers
Waiting for pending jobs
finalize
/
report
8s
finalize
/
publish-badge
4s
Annotations
9 errors and 2 warnings
build-amd64-arm64 (amd64) / test-jax / jax-A100-unit-test
The self-hosted runner: A100-197018b130b2 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.
|
build-amd64-arm64 (amd64) / test-nsys-jax / nsys-jax-A100-unit-test
Process completed with exit code 1.
|
build-amd64-arm64 (amd64) / test-nsys-jax / nsys-jax-A100-unit-test
Process completed with exit code 4.
|
build-amd64-arm64 (amd64) / test-triton / triton-A100-unit-test
Process completed with exit code 1.
|
build-amd64-arm64 (amd64) / test-upstream-t5x / test-upstream-t5x-outcome
Process completed with exit code 1.
|
build-amd64-arm64 (amd64) / test-rosetta-t5x / test-t5x-rosetta-outcome
Process completed with exit code 1.
|
build-amd64-arm64 (amd64) / test-maxtext / test-maxtext-outcome
Process completed with exit code 1.
|
make-publish-configs
The template is not valid. .github/workflows/ci.yaml (Line: 332, Col: 14): Error reading JToken from JsonReader. Path '', line 0, position 0.
|
CI
Error when evaluating 'strategy' for job 'publish-containers'. .github/workflows/ci.yaml (Line: 414, Col: 15): Error parsing fromJson,.github/workflows/ci.yaml (Line: 414, Col: 15): Error reading JToken from JsonReader. Path '', line 0, position 0.,.github/workflows/ci.yaml (Line: 414, Col: 15): Unexpected value ''
|
make-publish-configs
Encountered an error when evaluating display name ${{ format('declare -a FLAVORS=(
base
jax
triton
equinox
maxtext
levanter
upstream-t5x
t5x
gemma
)
declare -a STAGES=(
mealkit
final
)
## create JSON specs for a 1D matrix of container publication jobs
ALL_TAGS=$(
echo ''{0}'' \
''{1}'' |\
jq -s ''add''
)
PUBLISH_CONFIGS=''[]''
for stage in "${{STAGES[@]}}"; do
for flavor in "${{FLAVORS[@]}}";do
# collect images for different platforms, e.g. amd64 and arm64
matching_tags=$(
echo "$ALL_TAGS" |\
jq -c ".[] | select(.stage == \"${{stage}}\" and .flavor == \"${{flavor}}\" and .tag != \"\")"
)
# source_image is a list of all platform-specific tags
source_image=$(echo "${{matching_tags}}" | jq -c "[.tag]" | jq -s ''add'')
# if the build job failed without producing any images, skip this flavor
n_source_images=$(echo "$source_image" | jq ''length'')
if [[ $n_source_images -gt 0 ]]; then
echo "PUBLISH image $flavor with $n_source_images $stage containers"
# tag priority is the highest priority of all platform-specific tags
priority=$(echo "${{matching_tags}}" | jq -r ".priority" | jq -s ''max'')
# put all final images in the `ghcr.io/nvidia/jax` namespace
# and mealkit images in `ghcr.io/nvidia/jax-toolbox-mealkit` namespace
case ${{stage}} in
mealkit)
target_image=${{MEALKIT_IMAGE_REPO}}
;;
final)
target_image=${{FINAL_IMAGE_REPO}}
;;
esac
PUBLISH_CONFIGS=$(
echo ${{PUBLISH_CONFIGS}} | jq -c ". + [{{
\"flavor\": \"${{flavor}}\",
\"target_image\": \"${{target_image}}\",
\"priority\": \"${{priority}}\",
\"source_image\": ${{source_image}},
\"stage\": \"${{stage}}\"
}}]"
)
else
echo "SKIPPED image $flavor with 0 $stage containers"
fi
done
done
PUBLISH_CONFIGS=$(echo "$PUBLISH_CONFIGS" | jq -c ''{{"config": .}}'')
echo ${{PUBLISH_CONFIGS}} | jq
echo "PUBLISH_CONFIGS=${{PUBLISH_CONFIGS}}" >> $GITHUB_OUTPUT
', fromJson(needs.build.outputs['0'].DOCKER_TAGS), fromJson(needs.build.outputs['1'].DOCKER_TAGS)) }}. The template is not valid. .github/workflows/ci.yaml (Line: 332, Col: 14): Error reading JToken from JsonReader. Path '', line 0, position 0.
|
make-publish-configs
Encountered an error when evaluating display name ${{ format('declare -a FLAVORS=(
base
jax
triton
equinox
maxtext
levanter
upstream-t5x
t5x
gemma
)
declare -a STAGES=(
mealkit
final
)
## create JSON specs for a 1D matrix of container publication jobs
ALL_TAGS=$(
echo ''{0}'' \
''{1}'' |\
jq -s ''add''
)
PUBLISH_CONFIGS=''[]''
for stage in "${{STAGES[@]}}"; do
for flavor in "${{FLAVORS[@]}}";do
# collect images for different platforms, e.g. amd64 and arm64
matching_tags=$(
echo "$ALL_TAGS" |\
jq -c ".[] | select(.stage == \"${{stage}}\" and .flavor == \"${{flavor}}\" and .tag != \"\")"
)
# source_image is a list of all platform-specific tags
source_image=$(echo "${{matching_tags}}" | jq -c "[.tag]" | jq -s ''add'')
# if the build job failed without producing any images, skip this flavor
n_source_images=$(echo "$source_image" | jq ''length'')
if [[ $n_source_images -gt 0 ]]; then
echo "PUBLISH image $flavor with $n_source_images $stage containers"
# tag priority is the highest priority of all platform-specific tags
priority=$(echo "${{matching_tags}}" | jq -r ".priority" | jq -s ''max'')
# put all final images in the `ghcr.io/nvidia/jax` namespace
# and mealkit images in `ghcr.io/nvidia/jax-toolbox-mealkit` namespace
case ${{stage}} in
mealkit)
target_image=${{MEALKIT_IMAGE_REPO}}
;;
final)
target_image=${{FINAL_IMAGE_REPO}}
;;
esac
PUBLISH_CONFIGS=$(
echo ${{PUBLISH_CONFIGS}} | jq -c ". + [{{
\"flavor\": \"${{flavor}}\",
\"target_image\": \"${{target_image}}\",
\"priority\": \"${{priority}}\",
\"source_image\": ${{source_image}},
\"stage\": \"${{stage}}\"
}}]"
)
else
echo "SKIPPED image $flavor with 0 $stage containers"
fi
done
done
PUBLISH_CONFIGS=$(echo "$PUBLISH_CONFIGS" | jq -c ''{{"config": .}}'')
echo ${{PUBLISH_CONFIGS}} | jq
echo "PUBLISH_CONFIGS=${{PUBLISH_CONFIGS}}" >> $GITHUB_OUTPUT
', fromJson(needs.build.outputs['0'].DOCKER_TAGS), fromJson(needs.build.outputs['1'].DOCKER_TAGS)) }}. The template is not valid. .github/workflows/ci.yaml (Line: 332, Col: 14): Error reading JToken from JsonReader. Path '', line 0, position 0.
|
Artifacts
Produced during runtime
Name | Size | |
---|---|---|
artifact-base-build-amd64
|
566 Bytes |
|
artifact-base-build-arm64
|
566 Bytes |
|
artifact-equinox-build-amd64
|
570 Bytes |
|
artifact-equinox-build-arm64
|
568 Bytes |
|
artifact-final-report
|
5.02 KB |
|
artifact-gemma-build-amd64
|
560 Bytes |
|
artifact-jax-build-amd64
|
553 Bytes |
|
artifact-jax-build-arm64
|
552 Bytes |
|
artifact-levanter-build-amd64
|
571 Bytes |
|
artifact-levanter-build-arm64
|
572 Bytes |
|
artifact-maxtext-build-amd64
|
567 Bytes |
|
artifact-maxtext-build-arm64
|
567 Bytes |
|
artifact-maxtext-test
|
661 Bytes |
|
artifact-rosetta-build-t5x-amd64
|
584 Bytes |
|
artifact-rosetta-build-t5x-arm64
|
585 Bytes |
|
artifact-rosetta-t5x-mgmn-test
|
2.31 KB |
|
artifact-t5x-build-amd64
|
568 Bytes |
|
artifact-t5x-build-arm64
|
568 Bytes |
|
artifact-triton-build-amd64
|
565 Bytes |
|
artifact-upstream-t5x-mgmn-test
|
1.62 KB |
|
artifact-workflow-metadata
|
277 Bytes |
|
bumped-manifest
|
46.7 KB |
|
gemma-unit-test-A100
|
7.75 KB |
|
levanter-unit-test-A100
|
24.7 KB |
|
nsys-jax-unit-test-A100
|
30.5 MB |
|
rosetta-t5x-13393331622-1P1G_te-0
|
6.02 MB |
|
rosetta-t5x-13393331622-1P8G_te-1
|
5.58 MB |
|
rosetta-t5x-metrics-test-log
|
5.75 KB |
|
rosetta-t5x-vit-13393331622-VIT1G1N
|
21.6 KB |
|
rosetta-t5x-vit-13393331622-VIT1P8G
|
21.7 KB |
|
rosetta-t5x-vit-13393331622-VIT8G1N
|
33 KB |
|
triton-unit-test-A100
|
3.18 KB |
|
upstream-maxtext-13393331622-1DP1FSDP1TP1PP
|
16.6 KB |
|
upstream-maxtext-13393331622-1DP1FSDP8TP1PP
|
22.8 KB |
|
upstream-maxtext-13393331622-1DP2FSDP4TP1PP_single_process
|
17 KB |
|
upstream-maxtext-13393331622-1DP4FSDP2TP1PP
|
22.7 KB |
|
upstream-maxtext-13393331622-1DP8FSDP1TP1PP
|
22.3 KB |
|
upstream-maxtext-13393331622-2DP2FSDP2TP1PP
|
22.9 KB |
|
upstream-maxtext-metrics-test-log
|
2.95 KB |
|
upstream-t5x-13393331622-1P2G_fmha
|
7.24 MB |
|
upstream-t5x-13393331622-1P8G
|
7.24 MB |
|
upstream-t5x-metrics-test-log
|
7.79 KB |
|