[UI] Nomad behind a loadbalancer break the UI

[cberescu]

Nomad version

Nomad v1.9.4
BuildDate 2024-12-18T15:16:22Z
Revision 5e49fcd+CHANGES

Operating system and Environment details

Ubuntu

Issue

The issue seems to be that when having multiple servers behind a load balancer that can respond to UI requests if the assets are not requested from the same server they are returned as 404 pages.

So if the main page is loaded from Server A , and some of CSS and JS are loaded from Server B or Server C they return the 404 page for some reason. This did not happen prior of 1.9.3 version.

Reproduction steps

Having a domain dashboard.nomad.com and a load balancer that will forward every request to a different server(round‑robin).

Expected Result

The assets request from diffeent pages to not return 404 page

Actual Result

Some CSS or JS file contain the not found page

[philrenaud]

Hi @cberescu, thanks for raising this. I haven't been able to track down an obvious culprit in 1.9.4 or 1.9.3 that would indicate a new bug surrounding this, so I'd like to dig in a little more.

What was the last version you were running where you didn't notice a problem here?

Are there any resources that 404 other than css/js resources across servers? For example, do you find any XHR requests 404ing on page load?

[cberescu]

Hi @philrenaud , Version 1.9.3 works, issue is both 1.9.4 and 1.9.5 .

No XHR loads because of the issue with js/css

[cberescu]

ok, found the issue, need to upgrade all servers to the same version... i believe i forgot one and from there was the issue.

LE:
If i update to 1.9.4 all servers it seems to be working, however if i update it to 1.9.5 it does not load anymore and gives 404 : one example is this file :
/ui/assets/chunk.524.1247795429ad0ec5743c.js

[cberescu]

ok, so found the issue :
One of the servers has updates from
deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com jammy main
and the others from
deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com focal main

while node version gives exactly the same response the UI's assets have different names.

Maybe it should be added to the nomad version also a build version of the UI .. or something like that.. or in some way to always build the assets to the same name...

However the worst part is all servers need to be on the same version...

LE: it seems the issue is not even from the ubuntu server version... i have the same version on the both servers but get different js/css names...
is it possible to be different builds of the UI on your release servers ? ... the only explanation at this moment is that when installing on server S1 from apt.releases.hashicorp.com i am routed to a server and when installing on server S2 i am routed to another server with a different UI build...

LE2 :

I also get different sizes :

Package: nomad
Version: 1.9.5-1
Priority: optional
Section:
Maintainer: HashiCorp
Installed-Size: 121 MB
Depends: openssl
Recommends: dmidecode
Homepage: https://github.com/hashicorp/nomad
License: BUSL-1.1
Download-Size: 51.6 MB
APT-Manual-Installed: yes
APT-Sources: https://apt.releases.hashicorp.com jammy/main amd64 Packages
Description: Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications.

and

Package: nomad
Version: 1.9.5-1
Priority: optional
Section:
Maintainer: HashiCorp
Installed-Size: 116 MB
Depends: openssl
Recommends: dmidecode
Homepage: https://github.com/hashicorp/nomad
License: BUSL-1.1
Download-Size: 48.0 MB
APT-Manual-Installed: yes
APT-Sources: https://apt.releases.hashicorp.com jammy/main arm64 Packages
Description: Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications.

[cberescu]

Ok, so it seems I missed the CPU architecture , the issue is that the servers are either ARM or AMD and the UI build differs on them .

This did not happen until 1.9.5 .

Any way to fix this ?

[philrenaud]

Hi @cberescu , just a note to say we've started digging into this. We're building the JS bundle for each of our machine targets during our release process, but probably should be building it a single time and passing that build to each machine target, instead. Triaging internally.