[Enhancement]: compress chunks in the same hypertable in parallel

[Tindarid]

What type of enhancement is this?

Performance

What subsystems and features will be improved?

Compression

What does the enhancement do?

Hi!

I am trying to compress several chunks in parallel for the same hypertable (for different hyperatbles it works fine).

From my understanding, during the compression we should have exclusive lock on a new compressed chunk and read from the raw chunk (truncate the raw chunk in the end). Also, updating things on a main table should require locks for the whole table, but for a short period.

When I started compressing 2 chunks in parallel, I noticed that the first chunk compression blocks the second one:
ShareUpdateExclusiveLock on _timescaldb_internal.compressed_hypertable*

While digging into the code, I think this piece of logic takes this lock:
https://github.com/timescale/timescaledb/blob/main/src/chunk.c#L1145
So, Serialize chunk creation around a lock on the "main table" to avoid multiple processes trying to create the same chunk

I think this kind of situation could be avoided differently and should not require lock on the main compressed hypertable (which is blocking another chunk compression, as it takes the same ShareUpdateExclusiveLock)

What I mean here: let's suppose, that when we create a chunk - we also preallocate compressed chunk name for this chunk. During the compression we lock this compressed chunk and do not interfere with operations on the main hypertable

P.S.

1. I noticed that decompression in parallel works fine
2. I think this enhancement could be a workaround for this issue Parallel Compression threads for high throughput data #3077
3. compressed_hypertable_*: what is this used for? I cannot find any documentation

Implementation challenges

No response

[Tindarid]

Also: recompression works in parallel quite well, because it doesn't create the new chunk

recompression/decompression can interfere with each other (consuming resources of the system, disk can be a bottleneck).

But if there are enough resources, because of the non-blocking nature - there is a significant speedup for bulk operation, which involves several chunks

[antekresic]

Fixed by #6450