Monitoring endpoint page failing

[antmeehan]

Describe the bug

Description

When going to a specific endpoint within Monitoring in ServicePulse, it is failing to display any data.

Expected behavior

The graphs and details are displayed

Actual behavior

Nothing is shown.

Versions

ServicePulse 1.40.0
ServiceControl: 5.4.1
SC Monitoring: 5.4.1

Steps to reproduce

1. Access the /#/monitoring/endpoint/MyEndpointName page
2. Details not shown

Relevant log output

2024-09-20 13:42:49.4497|36|Error|Microsoft.AspNetCore.Server.Kestrel|Connection id "0HN685MK31QHE", Request id "0HN685MK31QHE:00000001": An unhandled exception was thrown by the application.|System.NullReferenceException: Object reference not set to an instance of an object.
   at ServiceControl.Monitoring.Infrastructure.Api.EndpointMetricsApi.GetSingleEndpointMetrics(String endpointName, Nullable`1 history) in /_/src/ServiceControl.Monitoring/Infrastructure/Api/EndpointMetricsApi.cs:line 145
   at lambda_method10(Closure, Object, Object[])
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Additional Information

Additional information

Only happens for a specific endpoint, others work okay.

[DavidBoike]

Hey @antmeehan, you mention this only happens for a specific endpoint. Any way you could tell us what that endpoint name is? Or even better, also capture the logs from the browser's developer console so we can see what URLs are generated?

If any of this is protected information, please let us know by opening a support case so we can keep that info private.

[antmeehan]

Thanks @DavidBoike. I've raised a support case 👍

[DavidBoike]

Hey @antmeehan, we've been looking at the code for awhile and it seems that this is a (hopefully very rare) race condition, and of course as all the storage is in-memory, restarting the instance always fixes it.

While we're unable to actually make the code fail in the same way in any test (which strengthens our assumption of a race condition) we are going to go replace a bunch of Dictionary types (from the .NET Framework era of ServiceControl) with ConcurrentDictionaries and hope that takes care of the issue.