Cant run test on real iPhone Device iOS 14

[jajo-shubham]

Hi Guys!
I am unable to run sitespeed with my mac and iPhone. It tells me to turn on remote automation even though it is already on.
It runs on the simulators but not on the real device.
I have tried safaridriver --enable as a popular suggestion but that also doesnt work.
OS X: Catalina 10.15.6
Phone: iOS 14.0 for iPhone XR and iPhone 6S

Command Used:
sitespeed.io -b safari --safari.ios --safari.useSimulator false --safari.deviceUDID "XXXXXXXXXXX" https://google.com
Output:

[2020-10-13 18:08:40] INFO: Versions OS: darwin 19.6.0 nodejs: v10.22.0 sitespeed.io: 15.4.0 browsertime: 10.4.0 coach: 5.1.1
[2020-10-13 18:08:41] INFO: Running tests using Safari - 3 iteration(s)
[2020-10-13 18:08:42] INFO: Browser failed to start, trying 2 more time(s): Could not create a session: Some devices were found, but could not be used:
- [unknown wireless device]: device is not paired
- Shubham’s iPhone (XXXX): Remote Automation is turned off (turn it on via Settings > Safari > Advanced > Remote Automation)
[2020-10-13 18:08:43] INFO: Browser failed to start, trying 1 more time(s): Could not create a session: Some devices were found, but could not be used:
- [unknown wireless device]: device is not paired
- Shubham’s iPhone (XXXXX): Remote Automation is turned off (turn it on via Settings > Safari > Advanced > Remote Automation)
[2020-10-13 18:08:44] INFO: Browser failed to start, trying 0 more time(s): Could not create a session: Some devices were found, but could not be used:
- [unknown wireless device]: device is not paired
- Shubham’s iPhone (XXXXX): Remote Automation is turned off (turn it on via Settings > Safari > Advanced > Remote Automation)
[2020-10-13 18:08:53] ERROR: BrowserError: Could not start the browser with 3 tries
    at SeleniumRunner.start (/Users/shubhamjajodia/.nvm/versions/node/v10.22.0/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/seleniumRunner.js:86:13)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-10-13 18:08:53] ERROR: No data to collect
[2020-10-13 18:08:53] ERROR: TypeError: Cannot read property 'alias' of undefined
    at Object.processMessage (/Users/shubhamjajodia/.nvm/versions/node/v10.22.0/lib/node_modules/sitespeed.io/lib/plugins/browsertime/index.js:148:50)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2020-10-13 18:08:54] INFO: HTML stored in /Users/shubhamjajodia/sitespeed-result/google.com/2020-10-13-18-08-40

[soulgalore]

Hi @jajo-shubham thanks for creating the issue. I think this could be that iOS 14 broke testing, let me try tonight (I haven't tried yet with an updated phone).

[jajo-shubham]

@soulgalore thanks for quick reply, FYI It works on the iOS 14 emulator (like iPhone SE second gen)

[soulgalore]

Yep I get the same. Lets wait for tonights event and see when next OS X is released and then I'll create an upstream bug at https://bugreport.apple.com/

[soulgalore]

Hi @jajo-shubham I tested again and now it worked. Are you sure your phone is paired? I synced my phone and after that it worked for me. When webdriver support was released, there was a bug that the phone needed to be paired with the device, my guess is that bug is back.

[soulgalore]

I tested again and now it doesn't work. It also looks like --safari.diagnose (verbose logging used when filing issues to Apple) has stopped to work. I'll look into it the coming days and file an issue.

[jajo-shubham]

Hey @soulgalore I am also facing similar issues,
On a real device it worked randomly 1 out of 20 times and even then it didn't complete the test, it timed out after opening the URL page.
Also, I tried with an iOS 13 device, getting the following error:

[2020-10-14 13:53:07] INFO: Versions OS: darwin 19.5.0 nodejs: v10.16.3 sitespeed.io: 15.4.1 browsertime: 10.4.1 coach: 5.1.1
[2020-10-14 13:53:07] INFO: Running tests using Safari - 3 iteration(s)
[2020-10-14 13:53:38] INFO: Browser failed to start, trying 2 more time(s): Could not create a session: 
The session timed out while connecting to a Safari instance.

Is there in general some issue with connecting to iOS safari using selenium?

[soulgalore]

Yep I think it started with iOS 14. I sent an email earlier today to the web-evangelist mail at Apple hoping they can help us.

[soulgalore]

Ah sorry I missed that you tried with 13. If so there's a webdriver issue. For me it was stable for 13.

[jajo-shubham]

Which OS X did you use for successful testing on iOS 13?

[soulgalore]

Catalina. Tomorrow is Big Sur released, I will have a go then again. If it still doesn't work I will create an upstream issue.

[soulgalore]

HI @jajo-shubham reported this upstream. They say they heard about the problem but they had a hard time to reproduce. I've been able to give them logs. The last time we had issue they where really helpful and fixed it, so I hope for the same thing this time. I'll update the thread when I get more news.

[soulgalore]

Hi @jajo-shubham if you also can submit a issue for Apple that would be great. Use the Feedback Assistant. Simplest use case to reproduce using pure Selenium for me is https://gist.github.com/soulgalore/a5478a1cc9937b19383f233443c1ce3f - I tried on two different Mac (Catalina and Big Sur) and one iPhone and iPad and get the same result.

[jajo-shubham]

@soulgalore Sure, I'll also submit an issue for the same!

[jajo-shubham]

Hey @soulgalore . I have not received any updates on this from apple, How about you?

[soulgalore]

@jajo-shubham they got back to me with a workaround that didn't work for me (I'll share it tomorrow see if it could work for you). I'll ping them again when I get back to work in a week or so. I also tried with a new clean phone but no luck.

[jajo-shubham]

Yeah sure, Let me know what they specified. Also what exactly seems to be the issue?

[soulgalore]

Starting in iOS 14, we’ve gotten reports that the Remote Automation setting may not be correctly read by the host computer. This seems to only happen if you’ve never looked at the Settings > Safari > Advanced screen on the iOS device/simulator. Try loading that screen, ensure Web Inspector and Remote Automation are enabled, and let me know if that resolves the issue.

However as you said it seems to be the same on 13 too. The thing is when I use Xcode I can see that the phone is paired. However WebDriver doesn't seems to know that it is.

[soulgalore]

@jajo-shubham I've finally got some good news. I got some help from the Apple dev team with a workaround, I'm gonna implement it the coming days. The fix is easy but there are some problem (bug) with the video recording I implemented some time ago so let me try to fix that too.

[jajo-shubham]

Hey That is amazing @soulgalore .. Do let me know the workaround too!

[soulgalore]

Yeah sorry, I was so tired last night :). The workaround is like this: start the Safaridriver standalone and configure to use that. I've pushed a fix for that that I'm gonna release the coming days. However video is not working at the moment, but I plan to release this fix first since its still valuable.

[soulgalore]

@jajo-shubham I've just pushed 17.8.1 with the fix. Video do not work though.

[jajo-shubham]

Thanks a lot.. will try it out soon!

[jajo-shubham]

Hey @soulgalore .. Thanks for fixing this.. I tested this out but there are some issues I am facing

• When coach is enabled, the process breaks, get the error

log.txt

• When visual metrics is enabled, video breaks with following error

[2021-07-13 11:51:07] INFO: Versions OS: darwin 20.5.0 nodejs: v14.16.1 sitespeed.io: 17.8.2 browsertime: 12.9.3 coach: 6.4.1
[2021-07-13 11:51:09] INFO: Running tests using Safari - 1 iteration(s)
[2021-07-13 11:51:13] INFO: Testing url https://www.browserstack.com iteration 1
[2021-07-13 11:51:13] INFO: Start IOS recorder.
[2021-07-13 11:51:22] INFO: Stop IOS recorder.
[2021-07-13 11:51:28] ERROR: Error: Command failed with exit code 1: ffmpeg -framerate 60 -i /Users/shubhamjajodia/sitespeed-result/www.browserstack.com/2021-07-13-11-51-07/tmp.h264 -c:v copy -f mp4 /Users/shubhamjajodia/sitespeed-result/www.browserstack.com/2021-07-13-11-51-07/pages/www_browserstack_com/data/video/1.mp4 -y
ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
built with Apple clang version 12.0.5 (clang-1205.0.22.9)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_2 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
/Users/shubhamjajodia/sitespeed-result/www.browserstack.com/2021-07-13-11-51-07/tmp.h264: No such file or directory
ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
built with Apple clang version 12.0.5 (clang-1205.0.22.9)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_2 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
/Users/shubhamjajodia/sitespeed-result/www.browserstack.com/2021-07-13-11-51-07/tmp.h264: No such file or directory
at makeError (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/execa/lib/error.js:60:11)
at handlePromise (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/execa/index.js:118:26)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async IOSRecorder.stop (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/video/screenRecording/ios/iosRecorder.js:61:5)
at async Measure.stop (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/command/measure.js:279:7)
at async /Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/run.js:9:9
at async Iteration.run (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/iteration.js:214:9)
at async Engine.runByScript (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine/index.js:277:20)
at async Object.analyzeUrl (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/lib/plugins/browsertime/analyzer.js:151:21)
at async Object.processMessage (/Users/shubhamjajodia/.nvm/versions/node/v14.16.1/lib/node_modules/sitespeed.io/lib/plugins/browsertime/index.js:150:26)
[2021-07-13 11:51:28] INFO: https://www.browserstack.com
[2021-07-13 11:51:29] INFO: HTML stored in /Users/shubhamjajodia/sitespeed-result/www.browserstack.com/2021-07-13-11-51-07

[soulgalore]

For video the problem is sitespeedio/browsertime#1598 ... waiting on an upstream fix for that. For the Coach sitespeedio/browsertime#1275 it seems to depends on how large the page is (it works for some page). I'm gonna have a look later this week and try to fix that.

[jajo-shubham]

Ah got it.. I saw your video implementation and I thought it was done.. No issues.. Thanks for the quick reply as usual 👍

[soulgalore]

I've updated the docs with instructions on how to disable the coach, gonna create an upstream issue when I get back from my vacation. Closing the issue since it works to run tests, and hope that the video gets fixed soon :)