[dynamodb] Performance optimizations

[ssalonen]

This PR introduces several performance optimizations to DynamoDB Persistence Service:

• buffering data in the PersistenceService, and writing it in batches on background.
  We get performance gains by writing data in batches but also reduce
  items QueueingThreadPoolExecutor.taskQueue pressure in GenericItem.notifyListeners.
• previously several checks were done on each store, causing
  significant overhead on store and query operations: 1) connection check
  (listing tables), 2) checking table existence. These are now removed,
  table is created on error (ResourceNotFoundException).

In addition, I have implemented exponential back-off retry for writes.

I've been verified the changes in my local setup. Before I could see that QueueingThreadPoolExecutor.taskQueue size growed (temporarily, but still) up to 500-1500 when many items were persisted at the same time. Dealing with the queue took some noticeable time. With the improvements, the size remains close to zero all the time.

Signed-off-by: Sami Salonen [email protected]

[ssalonen]

With the old version, I could see that sometimes QueueingThreadPoolExecutor.taskQueue growed to enormous sizes (even 1M), leading to OoM.

It's hard to diagnose what is the root cause for these occasional issues but I think writing in batches should be able to deal with even larger backlogs of data.

[ssalonen]

Anything I can do to progress this?

[9037568]

Sorry, @ssalonen . My free time has been very low lately.
I'll try to look at this this weekend. I fear we're in danger of crossing over with Kai's work on removing the persistence layer in #5844.

[ssalonen]

I understand completely, you have enormous work load in this repo alone.

Oh I was not aware of #5844, I think it #5827 can be dropped for now since the bnd build system will make it easier to handle SDK as maven dependency.

[kaikreuzer]

@ssalonen Would be cool if you could help on getting dynamodb (incl. your improvements) ready in https://github.com/openhab/openhab2-addons/pull/5275 - this is one of the addons, which I didn't yet spend the work to make it compile with bnd as it has a huge list of included libs.

Feel free to create a PR against my branch of that PR - merging your improvements should be very easy as I didn't do any real changes to the source code.

[9037568]

Just a few little tweaks.

[ssalonen]

@9037568 thank you for the review. Addressed the comments in 6900a08 & 0f3cf54.

[9037568]

Thanks, @ssalonen !

[kaikreuzer]

@ssalonen Could you please have a look at https://ci.openhab.org/job/openHAB1-Addons/lastCompletedBuild/org.openhab.persistence$org.openhab.persistence.dynamodb.test/testReport/? It seems that this PR made one test fail now.