Incomplete notifications in Telegram

[mike2307]

First of all:
Thank you for this awesome frigate extension!
This was the only part of frigate that I was missing really badly.
This reddit post lead me here and I'm very glad about it.

I only noticed a few issues with the generated Telegram notifications:

• The percentage values for the labels are at 0% always.
• The camera and event clip links stopped working after some time (not sure if this is uptime related or not). Since the frigate instance is running at a remote location with seldom activities I cannot easily investigate if it may heal itself automatically again. I can try it myself in a few days.

I'm using version v2.7.0 of this software and 0.13.2 of frigate.

[0x2142]

Hello there & Thanks for using this project!

So, I don't use Telegram as my primary notification method - but I did enable it to see if I can re-create your problem. So far I am seeing the correct percentage each time, but I'll keep it active for a while & see what happens.

Could you take a look at what the Frigate API shows for one of the events the displays 0%? If you grab the event ID, you can browse to the following: https://<your_frigate_url>/api/events/<event_id> - and check what is shown for the top_score value.

For the second issue - Are you using a VPN to connect back to that remote location? Or have the Frigate web UI available over the public internet? The links generated will point directly to the Frigate server, so the links would only work if you're able to connect to Frigate remotely.

[freefd]

Hi @mike2307, additionally please provide the frigate-notify configuration (do not forget to obscure sensitive things like credentials, etc).

[mike2307]

Alright; here's an example of one event...

The log shows:

2024/05/13 07:19:39 Event ID 1715584743.256569-9i3wwc - Camera Garage detected person in zone(s): []
2024/05/13 07:19:39 Event ID 1715584743.256569-9i3wwc - Start time: 2024-05-13 07:18:58 +0000 UTC
2024/05/13 07:19:41 Event ID 1715584743.256569-9i3wwc - Telegram alert sent

The notification in Telegram is here:

Notice that both links are black (instead of blue); therefore non-clickable.
The percentage value of the person is 0.

API data is here (with thumbnail removed):

{
    "area": null,
    "box": null,
    "camera": "Garage",
    "data": {
        "attributes": [],
        "box": [
            0.71015625,
            0.4736111111111111,
            0.046875,
            0.19027777777777777
        ],
        "region": [
            0.675,
            0.2152777777777778,
            0.325,
            0.5777777777777777
        ],
        "score": 0.8984375,
        "top_score": 0.818359375,
        "type": "object"
    },
    "detector_type": "edgetpu",
    "end_time": 1715584783.227474,
    "false_positive": null,
    "has_clip": true,
    "has_snapshot": true,
    "id": "1715584743.256569-9i3wwc",
    "label": "person",
    "model_hash": "6e4e281e51b5f8b4ca335c32ac86b072",
    "model_type": "ssd",
    "plus_id": null,
    "ratio": 1,
    "region": null,
    "retain_indefinitely": false,
    "score": null,
    "start_time": 1715584738.256569,
    "sub_label": null,
    "thumbnail": "REMOVED",
    "top_score": null,
    "zones": []
}

Config:

frigate:
  server: http://frigate-host:5000
  ignoressl: true

  webapi:
    enabled: true
    interval: 30

alerts:
  telegram:
    enabled: true
    chatid: ***
    token: ***

[freefd]

The percentage values for the labels are at 0% always.

Comparing the provided event example and app code (https://github.com/0x2142/frigate-notify/blob/main/events/events.go#L31), the top_score value will be derived from the top-level object where you have "top_score": null while the real score is defined inside the data structure as "top_score": 0.818359375. Sounds to me like a parsing bug.

The camera and event clip links stopped working after some time (not sure if this is uptime related or not). Since the frigate instance is running at a remote location with seldom activities I cannot easily investigate if it may heal itself automatically again. I can try it myself in a few days.

Meanwhile, I have the same trouble with non-clickable links on one of several frigate-notify installations, but I use the MQTT bus everywhere, so we can conclude the WebAPI approach you are using is not the root cause.
I thought that in my case the root cause was an inaccessible Frigate endpoint, but those suspicions were not confirmed after I specified external Frigate link and the links remain to be unclickable.

@0x2142, I'm also voting for this part :)

[mike2307]

I don't have any zones configured in my frigate instance.
Maybe the global top_score is only provided when there are multiple zones. But that's just a guess...

[0x2142]

Thank you both for the additional info! I appreciate the help!

So while I was looking at this a moment ago, I noticed that my instance doesn't report a data section in the API payload - I forgot I had my Frigate instance pinned to 0.12.1 because of some issues I had with 0.13.x... So I'll work on trying to upgrade when I get a chance & then I should be able to work on a fix for this.

@freefd - Just to double check, are you also on the latest 0.13.2 release?

On the Telegram side, I am not sure just yet - I've left mine enabled but haven't seen this happen. But I'll focus on getting my Frigate instance upgraded first, and keep Telegram enabled. We'll figure it out - just might need some time 😊

[freefd]

@freefd - Just to double check, are you also on the latest 0.13.2 release?

My Frigate instances are 0.13.2-6476f8a.

[0x2142]

Hey there - Just released v0.2.8, which should fix the 0% label issue.

In addition, I spent a bunch of time re-working the notifications & I'm hoping this may also fix the Telegram link issue. Please let me know if you're still seeing that after upgrading. Thanks!

[mike2307]

@0x2142
My frigate-notify instance is unable to connect to frigate after the update.

Logs:

2024/05/16 07:52:09 Frigate Notify - v0.2.8
2024/05/16 07:52:09 Starting...
2024/05/16 07:52:09 Loading config file: ./config.yml
2024/05/16 07:52:09 Config file loaded.
2024/05/16 07:52:09 Validating config file...
2024/05/16 07:52:09 Checking connection to Frigate server...
2024/05/16 07:52:09 Cannot reach Frigate server at http://frigate-host:5000, error: 502

Did anything change regarding the connection handling?

[mike2307]

Found the culprit:
In the new version a startup check was added: a6d9e1f

In my setup with podman compose, both the frigate and frigate-notify container are started together.
But since the frigate container takes some time to start, the startup check of frigate-notify is simply too early with its tries...

The best way to tackle this would be a dependency to a healthy frigate container, which could look like this:

frigate:
  ...
  healthcheck:
      test: ["CMD-SHELL", "curl localhost:5000/api/stats"]
  ...

frigate-notify:
  ...
  depends_on:
      frigate:
        condition: service_healthy
  ...

Unfortunately podman compose doesn't support that yet: containers/podman-compose#866

So for now I can think of the following options:

• (Manually) ensure that the frigate container is up and running before starting frigate-notify
• Avoid podman compose
• Extend this startup check with some retries and a timeout

[0x2142]

Ahh, okay. Yeah, adding a health check at startup had been on my list a while - because until now, it wouldn't ever try (and possibly fail) to reach Frigate until an event was detected. So after seeing your issue earlier with the URL syntax, it reminded me that I should include that & notify quickly if the Frigate API isn't reachable.

That being said - I appreciate your feedback since your deployment is a bit different than mine & I only test against my own set up at the moment. And currently I run these two under separate compose files so that I don't have to restart Frigate often. In theory, I think as long as this app is set to restart: unless-stopped, it should just keep restarting until Frigate comes up - then finally connect. Of course that's not an ideal fix either - but just another workaround for the moment.

I think I'll work up a quick retry / timeout for that startup check, but I imagine it would only make sense to wait up to 30-60 seconds before throwing an error & exiting. I'll have to think about whether it makes sense to have that max wait interval a configurable option, or perhaps an option to disable that startup check entirely. Let me know if you have any thoughts or a preference on that - I'm always open to other opinions & feedback!

[mike2307]

Yes, a retry mechanism with an interval of a few seconds and a timeout of something in the range of 30-60 seconds should be fine. Even better if this would be configurable. I have no idea how long the startup of frigate could take on a system with a lot of cameras (if this even has an influence on the availability of the HTTP API readiness).

In my compose file I have specified restart: always for both containers.
But it looks as if the frigate-notify container stays in running state even if the startup check has failed. Therefore it is -not being restarted automatically.
This could be a podman issue as well; not sure about that.

Can you check if the container is supposed to exit after the startup check has failed?

EDIT:
Sorry, my fault. I had an issue with the passthrough logging and only saw the logs from the 1st start.
The repeated restart of the frigate-notify container (until frigate is reachable) works as expected.

[mike2307]

I can confirm that the issue with the missing percentage values has been fixed with the latest version.
However the links in the notifications still remain black/unclickable.

[0x2142]

Just a quick note to say I haven't forgotten about the Telegram link issue. However, I've also been running with Telegram enabled on my side & still haven't had this happen to me yet 🤔

In the new release, I've added support for custom notification templates - which may be one way for you to address this. Along with that, I added some debug logging that will print the rendered notification template to the log if a custom template is used.

Can I have one of you try using the template below & enable debug logging? It's a copy of what I'm using as the standard template. Then if the links stop working again, please send me a copy of the logs with the rendered message & hopefully that will help point to the issue.

alerts:
  telegram:
    enabled: true
    chatid: xxxxxxxxx
    token: xxxxxxxxx
    template: |
      Detection at {{ .Extra.FormattedTime }}
      Camera: {{ .Camera }}
      Label: {{ .Label }} ({{ .Extra.TopScorePercent }})
      {{ if ge (len .Zones ) 1 }}Zone(s): {{ .Extra.ZoneList }}
      {{ end }}

      {{ if ne .Extra.PublicURL "" }}Links: <a href="{{ .Extra.PublicURL }}/cameras/{{ .Camera }}">Camera</a> {{ if .HasClip }} | <a href="{{ .Extra.PublicURL }}/api/events/{{ .ID }}/clip.mp4">Event Clip</a> {{ end }}
      {{ else }}Links: <a href="{{ .Extra.LocalURL }}/cameras/{{ .Camera }}">Camera</a>{{ if .HasClip }} | <a href="{{ .Extra.LocalURL }}/api/events/{{ .ID }}/clip.mp4">Event Clip</a> {{ end }}
      {{ end }}

[mike2307]

Thanks! 👍

I updated to the latest image, enabled debug logging and added the above provided template to my yml file.

It's a bit of a remote location and I'm there on the weekend again.
But I enabled object detection for birds to speed things up... 😆
Let's see when the next event gets triggered.

[freefd]

Dear @mike2307, you wrote:

However the links in the notifications still remain black/unclickable.

I also updated to the latest v0.3.0 and did several tests. My findings:

1. If the public_url parameter is set as https://username:[email protected] and the template is rendered as

DBG Custom message template used event_id=1717538894.834072-v7vzdz provider=Telegram rendered_template="Detection at 2024-06-04 22:08:14 +0000 UTC\nCamera: \<camera-name>\nLabel: person (71%)\n\n\nLinks: <a href=\"https://username:[email protected]/cameras/<camera-name\>\">Camera</a> | <a href=\"https://username:[email protected]/api/events/1717538894.834072-<siffix>/clip.mp4\">Event Clip</a> \n\n"

In the result message links are rendered as text:

2. If the public_url parameter is set as https://domain.tld and the template is rendered as

DBG Custom message template used event_id=1717541314.238218-n0jmrt provider=Telegram rendered_template="Detection at 2024-06-04 22:48:34 +0000 UTC\nCamera: \<camera-name>\nLabel: person (70%)\n\n\nLinks: <a href=\"https://domain.tld/cameras/<camera-name\>\">Camera</a> | <a href=\"https://domain.tld/api/events/1717541314.238218-n0jmrt/clip.mp4\">Event Clip</a> \n\n"

In the result message links are rendered as html:

I've run raw tests with the Telegram bot and can confirm that this behavior comes from Telegram itself:

~> curl -sq -d parse_mode=HTML -d chat_id=<telegram_chatid> -X POST https://api.telegram.org/bot<telegram_botid>/sendMessage -d text='<a href="https://domain.tld/">Test link without username and password</a>' | jq .ok
true
~> curl -sq -d parse_mode=HTML -d chat_id=<telegram_chatid> -X POST https://api.telegram.org/bot<telegram_botid>/sendMessage -d text='<a href="https://username:[email protected]/">Test link with username and password</a>' | jq .ok
true

Unfortunately, using parse_mode as Markdown (or MarkdownV2) and providing valid Markdown link produces the same results.

@0x2142, I haven't yet found any public information about this in the official Telegram documentation or related bot settings.
Seems worth mentioning in the frigate-notify documentation 🤷‍♂️

[mike2307]

@freefd

Exactly the same on my side

2024/06/05 05:38:11 DBG Checking for new events...
2024/06/05 05:38:11 DBG Found 1 new events
2024/06/05 05:38:11 INF Event Detected camera=Garage event_id=1717565848.462514-b6g2y2 label=bird zones=
2024/06/05 05:38:11 DBG Event start time: 2024-06-05 05:37:23 +0000 UTC event_id=1717565848.462514-b6g2y2
2024/06/05 05:38:11 DBG Custom message template used event_id=1717565848.462514-b6g2y2 provider=Telegram rendered_template="Detection at 2024-06-05 05:37:23 +0000 UTC\nCamera: Garage\nLabel: bird (73%)\n\n\nLinks: <a href=\"http://frigate-host:5000/cameras/Garage\">Camera</a> | <a href=\"http://frigate-host:5000/api/events/1717565848.462514-b6g2y2/clip.mp4\">Event Clip</a> \n\n"
2024/06/05 05:38:14 INF Alert sent event_id=1717565848.462514-b6g2y2 provider=Telegram

In my case the solution should simply be to provide correct public_url in an IP format, which should be fine in my case as everything is in my VPN anyway and the addresses don't change.

I'll keep you updated.

[0x2142]

Ahhh, that's interesting to know! Thanks for digging in! I can add a note to the docs to mention this 👍

[mike2307]

I can confirm that settings a public_url, that is well-formed in the eyes of telegram, makes the links working. 👍

[0x2142]

I think I'm going to go ahead and close this for now, since it seems like a Telegram-side thing. I did make a note in the latest docs to mention this in case anyone else has the same issue.