NextCloud conferecing dropping connections

[jakubgs]

As reported by @jarradh our NextCloud instance was misbehaving during a call with 3 people on 27th September, at 8pm GMT+7.

https://grafana.infra.status.im/d/QCTZ8-Vmk/single-host-dashboard?orgId=1&var-host=node-01.do-ams3.nextcloud.office&from=1632745808858&to=1632753004260

[jakubgs]

We can see the call starting slightly before 8pm GMT+7:

Not a huge amount of traffic:

[jakubgs]

Peak load average on a 2 core machine was 1.83, while memory looked stable with little swapping:

But if we look at CPU usage itself it's mostly idle:

And containers themselves don't appear to be starving for CPU time:

[jakubgs]

The only error I found in the NextCloud logs at the time was this:

{"reqId":"iOYAjRvI9f0L5aX811XI","level":3,"time":"2021-09-27T12:56:42+00:00","remoteAddr":"172.17.1.1","user":"corey","app":"no app in context","method":"GET","url":"/ocs/v2.php/core/autocomplete/get?search=&itemType=call&itemId=new&shareTypes[]=0&shareTypes[]=1&shareTypes[]=7","message":"in_array() expects parameter 2 to be array, null given","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36","version":"22.1.0.1","exception":{"Exception":"Exception","Message":"in_array() expects parameter 2 to be array, null given","Code":0,"Trace":[{"file":"/var/www/html/lib/private/AppFramework/App.php","line":156,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\AutoCompleteController"},"get"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":301,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OC\\Core\\Controller\\AutoCompleteController","get",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"ocs.core.AutoComplete.get"}]},{"file":"/var/www/html/ocs/v1.php","line":62,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/ocsapp/core/autocomplete/get"]},{"file":"/var/www/html/ocs/v2.php","line":23,"args":["/var/www/html/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","Line":158,"Previous":{"Exception":"TypeError","Message":"in_array() expects parameter 2 to be array, null given","Code":0,"Trace":[{"file":"/var/www/html/apps/circles/lib/Collaboration/v2/CollaboratorSearchPlugin.php","line":101,"function":"in_array","args":[10,null]},{"file":"/var/www/html/lib/private/Collaboration/Collaborators/Search.php","line":72,"function":"search","class":"OCA\\Circles\\Collaboration\\v2\\CollaboratorSearchPlugin","type":"->","args":["",10,0,{"__class__":"OC\\Collaboration\\Collaborators\\SearchResult"}]},{"file":"/var/www/html/core/Controller/AutoCompleteController.php","line":77,"function":"search","class":"OC\\Collaboration\\Collaborators\\Search","type":"->","args":["",["0","1","7"],false,10,0]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":217,"function":"get","class":"OC\\Core\\Controller\\AutoCompleteController","type":"->","args":["","call","new",null,["0","1","7"],10]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":126,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\AutoCompleteController"},"get"]},{"file":"/var/www/html/lib/private/AppFramework/App.php","line":156,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OC\\Core\\Controller\\AutoCompleteController"},"get"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":301,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["OC\\Core\\Controller\\AutoCompleteController","get",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"ocs.core.AutoComplete.get"}]},{"file":"/var/www/html/ocs/v1.php","line":62,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/ocsapp/core/autocomplete/get"]},{"file":"/var/www/html/ocs/v2.php","line":23,"args":["/var/www/html/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/html/apps/circles/lib/Collaboration/v2/CollaboratorSearchPlugin.php","Line":101},"CustomMessage":"--"}}

Where the principal error is:

  "message": "in_array() expects parameter 2 to be array, null given",
  "File": "/var/www/html/apps/circles/lib/Collaboration/v2/CollaboratorSearchPlugin.php",

But this error was already reported:

• Exception: in_array() expects parameter 2 to be array, null given nextcloud/server#28462
• /ocs/v2.php/apps/files_sharing/api/v1/sharees - expects parameter 2 to be array, null given nextcloud/server#28412

And fixed in nextcloud/circles#781 which was included in 22.1.1, so my upgrade to 22.2.0 covers it.

And anyway, that was related to search, not conferencing.

[jakubgs]

They released a new 12.1.2 version of Talk app 19 days ago: https://github.com/nextcloud/spreed/releases/tag/v12.1.2

Changes:

• [stable22] Add some validation to the geo location id nextcloud/spreed#6243
• [stable22] Fix renewing the session id after password use nextcloud/spreed#6229
• [stable22] Fix video toggling on pasting nextcloud/spreed#6199
• [stable22] Fix blocked audio recording preview when Talk is embedded in the sidebar nextcloud/spreed#6130
• [stable22] Fix infinite loop when the constraints can not be decreased nextcloud/spreed#6126
• [stable22] Fix connection quality warning still shown after media is stopped nextcloud/spreed#6146
• [stable22] Send offer again when own peer is not initially connected nextcloud/spreed#6080

I've upgraded it, so lets see if that makes a difference.

[jakubgs]

I've also temporarily increased verbosity level to DEBUG, we'll see what we can find out.

I also checked access logs for the app container but couldn't find any clear errors.

[jakubgs]

Also increased host size to s-4vcpu-8gb: https://github.com/status-im/infra-office/commit/780cfc6f

Thought ultimately once we decide to switch fully I'd like a beefy Hetzner host for NextCloud.