[linky] Fixes for change in Enedis API on 2024 December 20 #17945

lo92fr · 2024-12-21T11:07:36Z

Fixes for Linky Enedis December web site change

This pull request is about fixing the linky addon after the change that was made on Enedis side to the Web Site.
This is linked to issue:
#17936

Description

Mainly, changes are :
- URL for data is now mes-mesures-prm and not mes-mesures.
- Dto format have changed : mainly merge data & date, field renaming, and moving.
- Some changes on date format.

lolodomo · 2024-12-21T11:30:10Z

Thank you @lo92fr , I will review and test.

lolodomo · 2024-12-21T11:37:15Z

I still have Java 17 and Java 21 on my dev PC so it should be easy to build a JAR for Java 17.
But in case I would have only Java 21, do you know what is the maven command to execute to build a JAR compatible with Java 17 / OH 4.3 ?
@holgerfriedrich I am sure you know ;)

holgerfriedrich · 2024-12-21T12:02:06Z

@lolodomo Hopefully it is just:

mvn install -Pj17 -Dohc.version=4.3.0

https://community.openhab.org/t/openhab-5-x-snapshot-requires-java-21/160890/5

lolodomo · 2024-12-21T12:29:36Z

@lo92fr : does not work for me => Error requesting 'https://alex.microapplications.enedis.fr/mes-mesures-prm/api/private/v1/personnes/xxxxx/prms/yyyyy/donnees-energetiques?mesuresTypeCode=ENERGIE&mesuresCorrigees=false&typeDonnees=CONS&dateDebut=2023-01-01': {"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}

Will see if it work in next tries (next hours).

lo92fr · 2024-12-21T13:35:10Z

if it work in next tries (next

It still work on my side, strange.
Does the xxx in URL correctly formatted with something like MPSxxxYYY ?

Laurent.

lolodomo · 2024-12-21T13:50:29Z

Does the xxx in URL correctly formatted with something like MPSxxxYYY ?

3 letters then 3 numbers and then 3 letters.
Starts by N in my case.
If I correctly remember, it is the reference of my counter?

lo92fr · 2024-12-21T14:23:22Z

If it's still does not work, can you please:

check that you have the new version of the website. Look like this:
When you load the page, take a look on the browser debugger, new URL appears at the bottom of the list.
Is it the same and does it works correctly in your browser session ?

Thks,
Laurent.

lolodomo · 2024-12-21T16:53:32Z

Yes, I have the same page on their website.
The requests are similar, the binding request just has an additional parameter "dateDebut".
When I run in my browser session while I am logged to the website, the binding URL works well!

lolodomo · 2024-12-21T16:58:13Z

Now the binding is working! Even after I disconnect from the website in my browser.
Maybe you need to connect at least one time to the new website version ?

lolodomo · 2024-12-21T17:18:11Z

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java

+                updateState(PEAK_TIMESTAMP, new DateTimeType(
+                        days.datas.get(days.datas.size() - 1).dateDebut.atZone(this.timeZoneProvider.getTimeZone())));


This was changed very recently, just before releasing 4.3. We don't need to pass a ZonedDateTipe; DateTimeType just needs an Instant as parameter. So you do not need TimeZoneProvider .

Hello @lolodomo,

I'm not sure to understant. Do you say to use the DateTimeType constructor that take an instant, something like this.
updateState(PEAK_TIMESTAMP,
new DateTimeType(days.datas.get(days.datas.size() - 1).dateDebut.toInstant(zoneOffset)));

The matter is that I still need to find a zoneOffset in this case.
The Enedis API return a LocalDateTime at offset of the current zone.
The DateTimeType if I undestand correctly want a universal time.

So somewhere we need to convert from local to universal using a timezone ?

Laurent.

@jlaur is our specialist with date. Please advice.

If days.datas.get(days.datas.size() - 1).dateDebut is a LocalDateTime, the implementation is already correct. If it is an Instant there is no need to convert to ZonedDateTime.

the implementation is already correct

@jlaur : you mean @lo92fr proposed code is correct ?
I am a little lost, why should we consider the timezone setup in server ?
Imagine that for a strange reason I setup timezone in OH server to UTC+5. I am convinced Enedis provides dates in France timezone, that is UTC+1.

Hello @jlaur,

No there is no way to configure the home location / time-zone in the eneids account.
I think this is a hidden fixed value that is link to your PDL (Point of Delivery) that is linked to your postal address.

So I think the best is to assume that the api just return a local Datetime, and that the openhab server is on the same timezone.

The only scenario I can see that can be bad is someone having two different house : let say one in France, and one in another timezone like Guadeloupe, and that at the same time will use only one openhab instance for the two house.
In this case, result can be wrong.

Laurent.

Couldn't the timezone be a configuration setting of the PDL thing ? Defaulted to France/Paris ?

Hello @clinique,

Yes, this a solution.
Like this we will be able to handle specific case.

Laurent.

Probably the default should then be openHAB time-zone rather than France/Paris, given what @lo92fr wrote about the service probably using time-zone of PDL. But since it's not documented or proven, I guess it doesn't matter much, as long as it can be overridden by configuration.

@clinique, @jlaur,

I've implements the config for timezone in this commit.
Let me know if it's ok for you.
4281d5e

Laurent.

lolodomo · 2024-12-25T11:15:44Z

The fix fails for me.
It worked just one time apparently because.i previously opened a session on the website but the next days it failed.

lolodomo · 2024-12-31T09:22:19Z

While I installed 4.3.1, I updated again the linky bundle and it seems to work well yesterday and also today!
So maybe in my previous test, I did not take care and the previous jar was reloaded...

jlaur

Minor comments related to the time-zone commit. The I18 properties file should also be regenerated.

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java

bundles/org.openhab.binding.linky/src/main/resources/OH-INF/thing/thing-types.xml

lo92fr · 2025-01-15T11:06:30Z

Hello @jlaur, Hello @lolodomo,

Any news on this ?

Thanks,
Laurent. @

jlaur · 2025-01-15T11:18:45Z

Any news on this ?

My comments are resolved, but I have not reviewed this PR, so @lolodomo will be needed for finalizing the review and merging the PR.

lolodomo · 2025-01-15T12:09:15Z

The major problem is that unfortunately this fix does not work well !
It can work one day after a fresh OH install and then does not work anymore. This is probably a problem of cookie or similar.
I remember that we had exactly the same issue with the previous fix ... until @lo92fr fixed something more.

Am I alone with this PR not working more than one day ?

lo92fr · 2025-01-15T16:37:47Z

The major problem is that unfortunately this fix does not work well ! It can work one day after a fresh OH install and then does not work anymore. This is probably a problem of cookie or similar. I remember that we had exactly the same issue with the previous fix ... until @lo92fr fixed something more.

Am I alone with this PR not working more than one day ?

Have to said that I'm not totally sure myself if the fix is ok or not.
I'm working on my production server with the linkyv2 branch, and have disconnection every 24 hours.
I was thinking it was because of the workflow change that i've maded on linkyv2 branch (linky v2 keep the connection opened, but main branch disconnect / reconnect at every 24 hours refresh), but finally perhaps not.

@lolodomo : what error do you have after one day, an http 500 Internal Error ?
Anything in the log ?

What I try so far if to make the refresh more frequent, I see that if I change the refresh frequence from 24h to 3h, I can trigger the error as weel after only 3 hours.

I've also try in this case to clear all cookie, and redo the connectionInit phase, but this don't seem sufficient to fix the error.
But in v2, when I hit the error, If i stop the thing, and restart it, the connection restart to work correctly.

So I thing there is something that expire after some session time, and stay in memory, and that is not reinitialized correctly.

I will continue to narrow done this issue in the next few days, and will tell you if I find something.

Laurent.

lolodomo · 2025-01-16T07:08:03Z

what error do you have after one day, an http 500 Internal Error ?

{"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}

lo92fr · 2025-01-26T08:40:36Z

what error do you have after one day, an http 500 Internal Error ?

{"message":"Internal server error. Please try again","code":"INTERNAL_SERVER_ERROR"}

Hello all, @lolodomo

Perhaps find a fix on this new error.
It seems that we need to call the getPrmInfo each time we update the data.
Today, we only do it when initializing the handler.

My assomption is that the new website implementation keep some state data, and invalidate them after a few hours.
This state data is certainly initialize when we call this getPrmInfo endpoint.

I've doned some test on v2 branch, seems to fix the problems on my side.
Will need to be tested also on this fix branch to be sure.

Laurent.

lolodomo · 2025-02-01T17:48:54Z

@lo92fr : can you please rebase your branch so that we can compile it again ?

lo92fr · 2025-02-02T13:22:26Z

do it when initializing the handler.

My assomption

Helo @lolodomo,

Done the rebasing just right now.
But note that my fix from last week is not sufficient to totally fix our issues.
I've done some more modifications on the linkyv2 branch that seems to be promising.
I will try to backport them to fix branch this afternoon, or somewhere laster this week if I don't find time today.

Laurent.

lolodomo · 2025-02-02T13:50:22Z

I just build and deploy the last jar and yes you're right, it still does not work, always the same internal server error.

- URL for data is now mes-mesures-prm and not mes-mesures. - Dto format have changed : mainly merge data & date, field renaming, and moving. - Some changes on date format. Signed-off-by: Laurent ARNAL <[email protected]>

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr · 2025-02-05T09:25:27Z

does not work, always the same internal server error.

Hello @lolodomo,

Just to be sure to understand : you have the 500 Internal Server errors just after starting the binding, or on the nightly refresh at 1'oclock ?
If it's just after the start, can you send me your log, with serverity config to DEBUG ?

On my side, it's working since several day without error, refreshing data correctly every day.
But it's on v2 branch, so I switch this morning my server on the fix branch to check it better.

Also, I have to done some modifications on the ExperingDayCache.
The reason is that from my experiments, enedis site respond at 1'oclock without error, but the data is incomplete.
For exemple, I've got NULL value for day-1 consumption.

So I've made some modification on the logic of the Cache : if I detect that data is incomplete, I retry to read data after the REFRESH_INTERVAL_IN_MIN (120mn). So it will try at 1'oclock, 3'oclock, 5'oclock until he get the correct data.

I've also modifify the nextExpireAt logic a little, the idea is not to do all the request at the exact same hours for all users. But to randomize them a little to distribute the request around the central hour.

This change is on my last commit in the fix branch. I've backport them this morning from v2.

Laurent.

Signed-off-by: Laurent ARNAL <[email protected]>

…licate whith check already done in getConsumptionAfterChecks Signed-off-by: Laurent ARNAL <[email protected]>

…on't need it anymore Signed-off-by: Laurent ARNAL <[email protected]>

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr · 2025-02-08T07:11:26Z

does not work, always the same internal server error.

Hello @lolodomo,

Just to be sure to understand : you have the 500 Internal Server errors just after starting the binding, or on the nightly refresh at 1'oclock ? If it's just after the start, can you send me your log, with serverity config to DEBUG ?

On my side, it's working since several day without error, refreshing data correctly every day. But it's on v2 branch, so I switch this morning my server on the fix branch to check it better.

Also, I have to done some modifications on the ExperingDayCache. The reason is that from my experiments, enedis site respond at 1'oclock without error, but the data is incomplete. For exemple, I've got NULL value for day-1 consumption.

So I've made some modification on the logic of the Cache : if I detect that data is incomplete, I retry to read data after the REFRESH_INTERVAL_IN_MIN (120mn). So it will try at 1'oclock, 3'oclock, 5'oclock until he get the correct data.

I've also modifify the nextExpireAt logic a little, the idea is not to do all the request at the exact same hours for all users. But to randomize them a little to distribute the request around the central hour.

This change is on my last commit in the fix branch. I've backport them this morning from v2.

Laurent.

Hello @lolodomo, @jlaur, @clinique,

Just a little news about the fix.

I've reverted some of my last change on ExperingDayCache as I realized that the missingData logic I add was duplicate with check in getConsumptionAfterChecks.

I've also modified condition in getMetaData so we always call the getPrmInfos / getPrmDetails even if we already have the information. With this last fix, it solve the 500 nightly error on my side.

I've deploy the branch on my production openhab, and it now working well since 2 day without 500 error, and getting fresh value every night :)

I will keep it until monday to be sure everything is okl, but I think you perhaps can get another try to this new version.

Laurent.

lolodomo · 2025-02-08T07:36:35Z

Thank you Laurent, I just deployed the new version and the binding is ONLINE. I will see if it survives at least 3 days. I am starting a new review of the PR.

bundles/org.openhab.binding.linky/src/main/resources/OH-INF/thing/thing-types.xml

lolodomo · 2025-02-08T07:38:58Z

bundles/org.openhab.binding.linky/src/main/resources/OH-INF/thing/thing-types.xml

+					Will default to openHAB default timezone.
+					You will
+					need to change this if your Linky is located in a different timezone that your openHAB location.
+					You can use an
+					offset, or a label like Europe/Paris</description>


Just a remark: the line cuts are relatively strange.

Yes, fix in next commit

bundles/org.openhab.binding.linky/src/main/resources/OH-INF/i18n/linky.properties

bundles/org.openhab.binding.linky/src/main/resources/OH-INF/thing/thing-types.xml

...nhab.binding.linky/src/main/java/org/openhab/binding/linky/internal/LinkyHandlerFactory.java

...hab.binding.linky/src/main/java/org/openhab/binding/linky/internal/handler/LinkyHandler.java

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr · 2025-02-09T09:38:08Z

Hello @lolodomo,

I've fixed the issue of your last review.

Laurent.

lolodomo · 2025-02-09T12:19:15Z

@lo92fr : please rerun the i18n tool to take into consideration your change of label.
Then that will be fine for me.

The binding was still "alive" this morning.
I plan to wait for one another day before merging.

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr · 2025-02-10T10:14:58Z

@lo92fr : please rerun the i18n tool to take into consideration your change of label. Then that will be fine for me.

The binding was still "alive" this morning. I plan to wait for one another day before merging.

Hello @lolodomo,

This is done for regeneration of i18n.

Laurent.

lo92fr requested review from clinique and lolodomo as code owners December 21, 2024 11:07

lo92fr mentioned this pull request Dec 21, 2024

New version of Enedis website, #17936

Closed

wborn linked an issue Dec 21, 2024 that may be closed by this pull request

New version of Enedis website, #17936

Closed

jlaur changed the title ~~Linky : Fixes for change in Enedis API on 2024 December 20 !~~ [linky] Fixes for change in Enedis API on 2024 December 20 Dec 21, 2024

lolodomo added the bug An unexpected problem or unintended behavior of an add-on label Dec 21, 2024

lolodomo reviewed Dec 21, 2024

View reviewed changes

jlaur reviewed Jan 8, 2025

View reviewed changes

clinique approved these changes Jan 8, 2025

View reviewed changes

lo92fr force-pushed the linky-fix-202412 branch from cd8015d to f1dba4f Compare February 2, 2025 13:17

lo92fr added 5 commits February 5, 2025 08:58

fixes for change in Enedis API on 2024 December 20 !

d3de666

- URL for data is now mes-mesures-prm and not mes-mesures. - Dto format have changed : mainly merge data & date, field renaming, and moving. - Some changes on date format. Signed-off-by: Laurent ARNAL <[email protected]>

add timezone to thing config to allow overriding default timezone

eca525d

Signed-off-by: Laurent ARNAL <[email protected]>

spotless:apply

bd721b8

Signed-off-by: Laurent ARNAL <[email protected]>

fixes for Jacob review

d43ab2a

Signed-off-by: Laurent ARNAL <[email protected]>

add possible fix for 500 Internal Server Error

f8c7600

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr force-pushed the linky-fix-202412 branch from f1dba4f to f8c7600 Compare February 5, 2025 07:59

backport some fixes from linkyv2 branch to handle enedis website errors

9508786

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr added 4 commits February 6, 2025 10:04

remove condition so we can always get metadata to fix 500 error

4b0045e

Signed-off-by: Laurent ARNAL <[email protected]>

remove the missingData stuff from previous commit, realize if was dup…

fead647

…licate whith check already done in getConsumptionAfterChecks Signed-off-by: Laurent ARNAL <[email protected]>

remove also the refreshinterval on ExpiringDayCache constructor, we d…

d71513f

…on't need it anymore Signed-off-by: Laurent ARNAL <[email protected]>

remove nullable on action field

b76392c

Signed-off-by: Laurent ARNAL <[email protected]>

lo92fr requested a review from lolodomo February 8, 2025 07:11

lolodomo requested changes Feb 8, 2025

View reviewed changes

lo92fr added 2 commits February 9, 2025 10:05

fix issues from lolodom review on 8 Feb

48c5b82

Signed-off-by: Laurent ARNAL <[email protected]>

spotless:apply

706147d

Signed-off-by: Laurent ARNAL <[email protected]>

update i18n files: mvn i18n:generate-default-translations

de4f3f6

Signed-off-by: Laurent ARNAL <[email protected]>

lolodomo merged commit 6a840f7 into openhab:main Feb 10, 2025
2 checks passed

lolodomo added this to the 5.0 milestone Feb 10, 2025

		updateState(PEAK_TIMESTAMP, new DateTimeType(
		days.datas.get(days.datas.size() - 1).dateDebut.atZone(this.timeZoneProvider.getTimeZone())));

[linky] Fixes for change in Enedis API on 2024 December 20 #17945

[linky] Fixes for change in Enedis API on 2024 December 20 #17945

Conversation

lo92fr commented Dec 21, 2024

Fixes for Linky Enedis December web site change

Description

lolodomo commented Dec 21, 2024

lolodomo commented Dec 21, 2024

holgerfriedrich commented Dec 21, 2024

lolodomo commented Dec 21, 2024

lo92fr commented Dec 21, 2024

lolodomo commented Dec 21, 2024 • edited Loading

lo92fr commented Dec 21, 2024

lolodomo commented Dec 21, 2024 • edited Loading

lolodomo commented Dec 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lolodomo commented Dec 25, 2024

lolodomo commented Dec 31, 2024

jlaur left a comment

Choose a reason for hiding this comment

lo92fr commented Jan 15, 2025

jlaur commented Jan 15, 2025

lolodomo commented Jan 15, 2025

lo92fr commented Jan 15, 2025

lolodomo commented Jan 16, 2025

lo92fr commented Jan 26, 2025

lolodomo commented Feb 1, 2025

lo92fr commented Feb 2, 2025

lolodomo commented Feb 2, 2025

lo92fr commented Feb 5, 2025

lo92fr commented Feb 8, 2025

lolodomo commented Feb 8, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lo92fr commented Feb 9, 2025

lolodomo commented Feb 9, 2025

lo92fr commented Feb 10, 2025

lolodomo commented Dec 21, 2024 •

edited

Loading

lolodomo commented Dec 21, 2024 •

edited

Loading