Pose datasets start/end

[AmitMY]

It is most common to get data in subtitles format, or other time aligned formats

Therefore, when something like this:

• multimodalhugs/multimodalhugs/data/datasets/pose2text.py

  Line 89 in adcc3ea

  pose = Pose.read(pose_file.read(), start_frame=sample['source_start'], end_frame=sample['source_end']) # [t, people, d, xyz]

• multimodalhugs/multimodalhugs/processors/pose2text_preprocessor.py

  Line 61 in adcc3ea

  pose = Pose.read(pose_file.read(), start_frame=source_start, end_frame=source_end) # [t, people, d, xyz]

happen, it forces the user to do unnecessary calculation. (also, for videos for example)

Instead, please use start_time and end_time which are in milliseconds.

And either way, they are not floats, but ints

multimodalhugs/multimodalhugs/data/datasets/pose2text.py

Lines 35 to 36 in adcc3ea

	"source_start": Optional[float],
	"source_end": Optional[float],

[GerrySant]

This is something I have thought about a lot. The problem is that to make the conversion to frames (useful for some modalities) I need to know the fps of the input file. I would like to find a way to extract the fps directly from the file (be it mp4 or whatever).

I am aware that for poses, pose-format allows to specify start and end in milliseconds. The problem is currently in video (which is not fully implemented at the moment, but will extend image2text).

I know it's something to look at, but I'm focusing on more priority things at the moment. Any suggestions are welcome.

[AmitMY]

for poses - specifying start and end time uses the body.fps to calculate
https://github.com/sign-language-processing/pose/blob/e07ca689547e228952ca4dbad17dab591df33bc9/src/python/pose_format/pose_body.py#L239

for videos - depending on the use case - can perform the same thing by opening the video and asking for the fps - thing is, videos don't necessarily have constant fps, and then, it is way more important to work with times.

---

I feel like working with frames burdens the user with one more processing step (since most data is time aligned) while time is natural

[GerrySant]

Noted, upcoming commits will change default unit to seconds.

[AmitMY]

milliseconds is better*
they are integers, and easier to work with in general