[FLINK-36821] Update to Kafka 3.9.0

[tomncooper]

Currently, the Flink Kafka Connector uses Kafka client version 3.4.0.

This has a medium severity vulnerability (CVE-2024-31141) and needs to be upgraded to >= 3.8.0 to address it.

3.9.0 is the most recently released version and between that and 3.4.0 there have been numerous bug and performance fixes.

This PR:

• Updates the Kafka client version
• Updates the other dependencies, that are shared with Kafka, to be in line with the version used in Kafka 3.9.0 (if the update of the other deps should be done in a separate PR, let me know).
• Updates the Confluent Kafka and Zookeeper images to be in-line with the 3.9.0 Kafka versions.
• Adds missing methods to the FlinkKafkaInternalProducer now that the Producer interface has changed between the 3.4 and 3.9 Kafka client versions.

I am not familiar with ArchUnit, but it appears the violation file changes have been checked into previous commits so I have included them here aswell.

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

[davidradl]

This change seems unrelated to the version change. If it is related please could you add a unit test.

[tomncooper]

Because the Producer interface has changed between 3.4 and 3.9, this new method is required in order for the class to compile (I should have mentioned that in the PR description).

As this class is deprecated anyway and calls the wrapped KafkaProducer's client instance ID method (which is tested), is a unit test strictly required?

[loserwang1024]

Updates the Confluent Kafka and Zookeeper images to be in-line with the 3.9.0 Kafka versions.

Whether the 3.9.0 version Kafka client is compability with the low version Kafka and zookeeper image?

[showuon]

Whether the 3.9.0 version Kafka client is compability with the low version Kafka and zookeeper image?

Yes, Kafka client 3.9.0 supports old version of Kafka and zookeeper image.