Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix never trigger #33989

Merged
merged 5 commits into from
Feb 15, 2025
Merged

Fix never trigger #33989

merged 5 commits into from
Feb 15, 2025

Conversation

reuvenlax
Copy link
Contributor

No description provided.

@reuvenlax
Copy link
Contributor Author

R: @kennknowles

@github-actions GitHub Actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

kennknowles
kennknowles reviewed Feb 14, 2025
View reviewed changes
.../worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingSideInputFetcher.java Outdated
@@ -320,7 +320,10 @@ private <SideWindowT extends BoundedWindow> Windmill.GlobalDataRequest buildGlob
.build())
.setExistenceWatermarkDeadline(
WindmillTimeUtils.harnessToWindmillTimestamp(
sideWindowStrategy.getTrigger().getWatermarkThatGuaranteesFiring(sideInputWindow)))
sideWindowStrategy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that triggers cannot actually give a tight bound. You either have to change all of them to return maxTimestamp + allowedLateness or just take the minimum here. I would suggest keeping it simple and doing the minimum here. In fact, I would even support deleting the method from Trigger and doing all of the logic here, since it is really just a one-off special case for AfterWatermark / DefaultTrigger

kennknowles
kennknowles approved these changes Feb 14, 2025
View reviewed changes
Copy link
Member

@kennknowles kennknowles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good to me. Might want a TestStream test that reproduces the issue and protects against future bugs.

@reuvenlax
Copy link
Contributor Author

All good to me. Might want a TestStream test that reproduces the issue and protects against future bugs.

A bit tricky to structure this? The problem with TestStream is that everything will be produced after advanceWatermarkToInfinity is called, and I believe will be produced in the correct windows. Will see what I can do.

@reuvenlax
Copy link
Contributor Author

@kennknowles Added a TestStream test

@reuvenlax reuvenlax merged commit 23ba9fc into apache:master Feb 15, 2025
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kennknowles kennknowles kennknowles approved these changes

Assignees
No one assigned
Labels
dataflow direct java jet runners
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@reuvenlax @kennknowles