Added touch gesture to companion protocol

[albaintor]

Hi, I have added the implementation of touch gestures and updated the documentation.

I have tested it successfully.
To test it :

atv = await pyatv.connect(config, loop)
companion = cast(FacadeRemoteControl, atv.remote_control).get(Protocol.Companion)
if companion:
    api: CompanionAPI = companion.api
    # Simulate a gesture from x=100, y=100 to x=1000, y=1000 in 1000 milliseconds
    await api.touch_gesture(100, 100, 1000, 1000, 1000)

[albaintor]

Hi, I have added the new interface and its implementation including the facadeAppleTV class and brought a fix to my code.
Now it works more simpler :

atv = await pyatv.connect(config, loop)
    feature = atv.features.get_feature(FeatureName.TouchGesture)
    print("Feature : "+feature.state.name)
    await atv.touchgestures.touch_gesture(900, 900, 10, 10, 1000)

I have added an additional method in the new interface. This method above is kind of a helper function. But if one wants to simulate a customized gesture there is the touch_event also (I'm thinking of a slider in a UI that would generate events constantly based on slider position)

[IsakTheHacker]

Hi, amazing work you've done here! Have been waiting for this for so long. Do you think it would be possible to have this working for real time gestures, just like the way it works for the Siri Remote and in control center on iOS? I have no idea how such an interface would work or even look like though, but that would be very awesome. Everything looks exceptionally good by the way and I hope @postlund can take a look at this soon

[albaintor]

Hi, I had a great help from @postlund or else I couldn't find out how to reproduce the streams.

As written above I implemented different methods :

1. Touch_gesture that is a helper function to simulate touch from point A to point B in a given time (shorter time means faster move as well as the distance between A & B). It should be used if you want to map common gestures (horizontal swipe from left to right or opposite) to a single command.
2. Touch_event is the base method for gestures : this would be used for more advanced usage. For example if you press a slider from left to right then right to left without releasing it, you would have to synchronize the range of the slider with the range of the touchpad every few milliseconds. Same approach for a virtual touchpad

I didn't implement tap/double tap although this is possible as I think that those commands are already available in the library

[albaintor]

Touchpad.tool.1.mp4

I have made a tool to simulate the touchpad gestures and the right algorithm behind
Available in the repository : pyatv/scripts/atvui.py

[postlund]

A few comment but it looks very good, nice addition! 👍

[postlund]

A section in the atvremote documentation with some examples would be great as well!

[postlund]

One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py

[albaintor]

A section in the atvremote documentation with some examples would be great as well!

Yes I will add that

[albaintor]

One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py

I don't think this is possible : touch events are sent with no feedback by Appletv

[postlund]

One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py

I don't think this is possible : touch events are sent with no feedback by Appletv

I have implanted fake "server sides" of each protocol. So you can just store the events and verify what you received. No interaction with an Apple TV is used (it's just unit tests).

[albaintor]

One last thing that I forgot about... Some kind of basic tests for the added API methods are needed. Only a "good case" is needed, just to ensure we verify something. Should be fairly easy to add in https://github.com/postlund/pyatv/blob/master/tests/protocols/companion/test_companion_functional.py

I don't think this is possible : touch events are sent with no feedback by Appletv

I have implanted fake "server sides" of each protocol. So you can just store the events and verify what you received. No interaction with an Apple TV is used (it's just unit tests).

Ok got it, I am on it

[albaintor]

Hi,
all done. I'll let you review. I have also updated the atvremote script to support touch calls

[albaintor]

As an example :

# Horizontal touch gesture from left to right in 200ms
atvremote.py --id <appletvid> touch=0,0,800,0,200
# Touch click
atvremote.py --id <appletvid> touch_click

[postlund]

I'll try to look at it tonight, great work so far! Also remember that we should add some developer documentation as well at some point, in case someone wants to use it (a subpage here https://pyatv.dev/development/).

[albaintor]

I'll try to look at it tonight, great work so far! Also remember that we should add some developer documentation as well at some point, in case someone wants to use it (a subpage here https://pyatv.dev/development/).

Yes I a little new to python's coding (since this year), however I found your code very well organized apart all the offered features for all-in-one library.
I got a little struggled though by the fact that I had to apply modifications in several files (don't remember exactly but : companion/api, companion/protocol, interfaces, and facadeappletv), so yeah it would deserve a howto guide to develop

[albaintor]

@albaintor There are a few linting stuff to fix.

[albaintor]

Hi,
it was sorting issues. Should be fixed now thanks

[postlund]

@albaintor I don't think you managed to push any changes?

[albaintor]

@albaintor I don't think you managed to push any changes?

Are you sure ? I have pushed everything on my side
For example : pyatv/interfaces.py:1284 --> method renamed click instead of touch_click (N-1 commit)
And linting fix pushed from albaintor@d307a8c
For example : pyatv/core/facade.py

[postlund]

@albaintor I just noted that there are no pushed changes after my last comment according to the PR. There are also still checks that are failing, you can check that below.

[albaintor]

Hi I think I know what is going on : something is wrong on the build action

The black command fails with the new "match case" code in tests/fake_device/companion.py
The target python should be added in the black : -t py310 or whatever

EDIT :
Addin black --target-version py310 --fast --check . within chickn.yaml would resolve the problem in order to accept python >3.10 match / case routines. But it also raises new linting issues. Just want to be sure what do you want to do from there
See ref psf/black#2242

[postlund]

Hi I think I know what is going on : something is wrong on the build action

The black command fails with the new "match case" code in tests/fake_device/companion.py The target python should be added in the black : -t py310 or whatever

EDIT : Addin black --target-version py310 --fast --check . within chickn.yaml would resolve the problem in order to accept python >3.10 match / case routines. But it also raises new linting issues. Just want to be sure what do you want to do from there See ref psf/black#2242

Ah, I don't want to drop support for older Python versions just because of this. Just revert to an if-statement instead. You can use ./scripts/chickn.py -t fixup to deal with linting properly (it's what's used by CI).

[albaintor]

Hi I think I know what is going on : something is wrong on the build action
The black command fails with the new "match case" code in tests/fake_device/companion.py The target python should be added in the black : -t py310 or whatever
EDIT : Addin black --target-version py310 --fast --check . within chickn.yaml would resolve the problem in order to accept python >3.10 match / case routines. But it also raises new linting issues. Just want to be sure what do you want to do from there See ref psf/black#2242

Ah, I don't want to drop support for older Python versions just because of this. Just revert to an if-statement instead. You can use ./scripts/chickn.py -t fixup to deal with linting properly (it's what's used by CI).

Hi, this is all done including linting
There are some linting warnings but not related to my modifications

[postlund]

As long as the linting fails, you must fix it. You can suppress too-many-branches and you probably have to move the existing suppression for logging.TRAFFIC.

[albaintor]

As long as the linting fails, you must fix it. You can suppress too-many-branches and you probably have to move the existing suppression for logging.TRAFFIC.

It should be okay this time

[albaintor]

All tests passed !

[postlund]

Yes! 🥳