Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add three new ZMQ publishers for TP results / Retrieve TP results #2230

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Add three new ZMQ publishers for TP results / Retrieve TP results #2230

wants to merge 12 commits into from

Conversation

MichaelHuth
Copy link
Collaborator

@MichaelHuth MichaelHuth commented Aug 15, 2024 •
edited by t-b
Loading

Publishers added:
testpulse results live
testpulse results 1s update
testpulse results 5s update
testpulse results 10s update

The TP results from TP analysis are packed in a JSON in TP_TSAnalysis and published to ZMQ.
The time information when the next publishing is due is stored in the TUFXOP.

PUB_Publish was changed to be threadsafe and added an optional argument to control if the JSON should be released.

close #2157

@MichaelHuth MichaelHuth self-assigned this Aug 15, 2024
@MichaelHuth MichaelHuth force-pushed the feature/2230-Add_zeromq_pub_filters_for_TP branch 4 times, most recently from 33d4401 to 716f021 Compare August 20, 2024 17:09
@MichaelHuth

This comment was marked as outdated.

Sign in to view
@t-b

This comment was marked as outdated.

Sign in to view
@MichaelHuth MichaelHuth force-pushed the feature/2230-Add_zeromq_pub_filters_for_TP branch 2 times, most recently from 96f6592 to 9c33299 Compare August 22, 2024 11:51
@MichaelHuth MichaelHuth assigned t-b and unassigned MichaelHuth Aug 22, 2024
@t-b

This comment was marked as off-topic.

Sign in to view
@t-b t-b assigned MichaelHuth and unassigned t-b Aug 22, 2024
@MichaelHuth

This comment was marked as outdated.

Sign in to view
@MichaelHuth MichaelHuth force-pushed the feature/2230-Add_zeromq_pub_filters_for_TP branch from ef494e7 to e82cd88 Compare August 25, 2024 18:11
@MichaelHuth MichaelHuth assigned t-b and unassigned MichaelHuth Aug 25, 2024
@t-b

This comment was marked as outdated.

Sign in to view
@MichaelHuth MichaelHuth force-pushed the feature/2230-Add_zeromq_pub_filters_for_TP branch from e82cd88 to 82d748d Compare September 3, 2024 12:50
@t-b t-b assigned MichaelHuth and unassigned t-b Sep 3, 2024
@MichaelHuth MichaelHuth force-pushed the feature/2230-Add_zeromq_pub_filters_for_TP branch 2 times, most recently from 28dcb99 to e86dda8 Compare September 3, 2024 22:39
@MichaelHuth MichaelHuth assigned t-b and unassigned MichaelHuth Sep 3, 2024
@t-b

This comment was marked as outdated.

Sign in to view
@t-b

This comment was marked as outdated.

Sign in to view
@campagnola

This comment was marked as outdated.

Sign in to view
@ben-at-allen

This comment was marked as outdated.

Sign in to view
@timjarsky

This comment was marked as outdated.

Sign in to view
@t-b

This comment was marked as outdated.

Sign in to view
@t-b

This comment was marked as outdated.

Sign in to view
@t-b t-b added the PatchLink label Jan 8, 2025
@campagnola

This comment was marked as outdated.

Sign in to view
@t-b

This comment was marked as outdated.

Sign in to view
@t-b t-b assigned t-b and unassigned campagnola Jan 8, 2025
@t-b
Copy link
Collaborator

t-b commented Jan 10, 2025 •
edited
Loading

Conclusion from today's meeting:

  • Add new ZeroMQ pub message with the raw AD data attached as another multipart message block
  • Only every TP version
  • Keep JSON with metadata
  • Don't require to store every TP in MIES with that

@t-b t-b mentioned this pull request Jan 13, 2025
Merged
4 tasks
@mah1329 mah1329 mentioned this pull request Jan 23, 2025
Open
MichaelHuth and others added 12 commits February 18, 2025 15:31
@MichaelHuth @t-b
Util: Add two conversion function for DA/AD unit string return
72a12ac 
Added GetADChannelUnit, GetDAChannelUnit that return the unit string
depending on clamp mode.

Adapted GetChanAmpAssignUnit to use the new functions.
@MichaelHuth @t-b
TP: Add more information that is transferred to the TP analysis thread
bfd910b 
- extended the TPAnalysisInput structure

This is a preparation commit for adding zeromq publishers that include
some of that information.
@MichaelHuth @t-b
PUB: Preparation to add four zeromq publishers for TP data
b20ff77 
- the data is published from the TP analysis thread including additional
information available in the thread through the previous commit.
- The additional values are also returned by the thread and collected in
the async buffer as well then in TPResult and in TPStorage.
- The involved waves and their respective getters were adapted with new
elements that the additional data can be stored.
- As most of the elements store the same information, thus a constant
was introduced with a dimension label list that is used as helper for
the wave creation in the getter functions.
@MichaelHuth @t-b
PUB: Add four publishers to publish TP data
697253d 
- The four publishers publish the same json, just with a different period.
  There is a filter for live, 1s, 5s and 10s publishing interval.

- See PUB_TPResult for JSON description

- publisher is called from TP_TSAnalysis thread
@MichaelHuth @t-b
TP: Rename TP_GetStoredTPs to TP_GetConsecutiveTPsUptoMarker
56ae07d 
This prevent misleading naming and it more fitting to the functionality
the function actually implements
@MichaelHuth @t-b
TP: Added two functions that allow to retrieve info about TPs
dc6afe4 
Added TP_GetStoredTP and TP_GetStoredTPsFromCycle that allow to get
information about a TP by tpMarker or TPs by cycle id and headstage.

- both functions allow to recreate the DA wave for the TPs with the flag includeDAC
- the returned data includes the AD, DA data as well as metadata for
  each returned TP (from TPStorage).

- These TP functions use the same TP utility function.
@MichaelHuth @t-b
Tests: Adapt FetchPublishedMessage for more tries
5fda5f9 
With a running TP adding zeromq publishing messages for each TP
it appears that we have to look through more than the last 100 messages
after a test to find the requested.

- split logic into two parts: either read out upto 10000 existing messages
  or wait up to 10 seconds (100 trys with 100 ms sleep) if no message
  is available
@t-b
XOPs: Update ZeroMQ XOP
68a3306
@t-b
PUB_Publish: Port to *multi interface
f974513 
This prepares for a future commit where we want to sent additional data.
@t-b
FetchPublishedMessage: Port to zeromq_sub_recv_multi
c0f6321
@t-b
tests: Prefer FetchAndParseMessage
9948189 
And make all received data available from FetchPublishedMessage.
@t-b
Add ZMQ_FILTER_TPRESULT_NOW_WITH_DATA
5f00ccf 
Add a message type which not only publishes the TP properties but also the
raw AD data.
@t-b t-b force-pushed the feature/2230-Add_zeromq_pub_filters_for_TP branch from fc20936 to 5f00ccf Compare February 18, 2025 19:06
@t-b
Copy link
Collaborator

t-b commented Feb 18, 2025

@MoxenWolf Ready to test. The new filter name is testpulse:results live with data and has three frames. Filter, json message and then the raw AD data (single precision float). As you are only using one HS the raw AD data wave is 1D.

@t-b t-b assigned MoxenWolf and unassigned t-b Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@t-b t-b t-b left review comments

@timjarsky timjarsky Awaiting requested review from timjarsky timjarsky is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@MoxenWolf MoxenWolf

Labels
PatchLink
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Extend zeromq PUB filters for TP updates
6 participants
@MichaelHuth @t-b @timjarsky @campagnola @MoxenWolf @ben-at-allen