Maven via JitPack: amazon-kinesis-aggregator-2.0.2 has v1 code, amazon-kinesis-deaggregator-2.0.2 unavailable

[stevegoossens]

I'm using the following in a Maven project by copy-pasting the code from:

• https://github.com/awslabs/kinesis-aggregation/tree/master/java/KinesisAggregatorV2
• https://github.com/awslabs/kinesis-aggregation/tree/master/java/KinesisDeaggregatorV2

i.e.

src/main/java/com/amazonaws
└── kinesis
    ├── agg
    │   ├── AggRecord.java
    │   └── RecordAggregator.java
    └── deagg
        ├── RecordDeaggregator.java
        └── util
            └── DeaggregationUtils.java

I would much rather just refer to these packages via Maven dependencies. Since they don't appear to be in a Maven repo, I've attempted to use JitPack to pull in the GitHub releases:

https://jitpack.io/#awslabs/kinesis-aggregation/2.0.2 instructs adding the following to my POM:

amazon-kinesis-aggregator (v2)

	<repositories>
		<repository>
		    <id>jitpack.io</id>
		    <url>https://jitpack.io</url>
		</repository>
	</repositories>

	<dependency>
	    <groupId>com.github.awslabs.kinesis-aggregation</groupId>
	    <artifactId>amazon-kinesis-aggregator</artifactId>
	    <version>2.0.2</version>
	</dependency>

This works to pull in amazon-kinesis-aggregator-2.0.2.jar, but it pulls in the classes (AggRecord and RecordAggregator) from the v1 submodule https://github.com/awslabs/kinesis-aggregation/tree/master/java/KinesisDeaggregator, which has the v1 variant methods, e.g. toPutRecordRequest (v1) instead of toPutRecordsRequest (v2):

amazon-kinesis-deaggregator (v2)

Attempting to pull in the KinesisDeaggregatorV2 classes via JitPack fails to even find the JAR:

    <dependency>
      <groupId>com.github.awslabs.kinesis-aggregation</groupId>
      <artifactId>amazon-kinesis-deaggregator</artifactId>
      <version>2.0.2</version>
    </dependency>

TL;DR: my code works with the copy-pasted classes from KinesisAggregatorV2 and KinesisDeaggregatorV2, but I would like to load them as Maven dependencies instead. Is there a way I can do that?

[dforsl]

We're also manually downloading the release right now - which won't do for long as it's spreading like wildfire. It would be awesome if we could simply have the aggregator- and deaggregator-libraries in maven!

[emopti-cmeyer]

@IanMeyers any chance we could get the v1 and v2 releases published out to a maven repository again? The most recent and only build currently is https://mvnrepository.com/artifact/com.amazonaws/amazon-kinesis-deaggregator/1.0.3

[gregorydickson]

+1

[IanMeyers]

Working on it - understand the ask and we'll hopefully have an update this week.

[IanMeyers]

Do we have any idea why JitPack is including the wrong version of the classes? Some parts of the 2.x codebase have a v1.x dependency but only for testing, and are not included in the jar as far as I can see.

[IanMeyers]

So this isn't exactly what you asked for, but I think it will maybe help in the short run. I've drafted 4 new releases:

Version 2.x
2.0.3-agg: https://github.com/awslabs/kinesis-aggregation/releases/tag/2.0.3-agg
2.0.3-deagg: https://github.com/awslabs/kinesis-aggregation/releases/tag/2.0.3-deagg

Version 1.x
1.1.3-agg: https://github.com/awslabs/kinesis-aggregation/releases/tag/1.1.3-agg
1.1.3-deagg: https://github.com/awslabs/kinesis-aggregation/releases/tag/1.1.3-deagg

I hope that this helps everyone get the correct version through JitPack while we sort out push to Maven Central.

[emopti-cmeyer]

Do we have any idea why JitPack is including the wrong version of the classes? Some parts of the 2.x codebase have a v1.x dependency but only for testing, and are not included in the jar as far as I can see.

Possibly because the v1 and v2 modules / artifacts have the same group / artifact ids and only differ in the version number? I'm guessing they did not expect a project to publish out multiple concurrent versions of unqiue groupId:artifactId combinations in the same build :)

[IanMeyers]

Yep that's why - didn't edit my comments after doing the research. That's why I've pushed new release labels that should address this.

[emopti-cmeyer]

I saw you created two tags (2.0.3-agg and 2.0.3-deagg), but they point to the same commit hash.

It appears both builds fail the same way on Jitpack, here are the build logs:
https://jitpack.io/com/github/awslabs/kinesis-aggregation/2.0.3-deagg/build.log
https://jitpack.io/com/github/awslabs/kinesis-aggregation/2.0.3-agg/build.log

The error message is:
Failed to execute goal on project amazon-kinesis-deaggregator: Could not resolve dependencies for project com.amazonaws:amazon-kinesis-deaggregator:jar:1.1.3: Could not find artifact com.amazonaws:amazon-kinesis-aggregator:jar:1.1.2 in central (https://repo.maven.apache.org/maven2)

Haven't dug in, but seems like perhaps your multi-module build isn't able to find the siblings?

Also, if you navigate to e.g. https://jitpack.io/#awslabs/kinesis-aggregation/2.0.3-deagg and scroll down to "Step 2", you'll see that the artifact selection drop-down only shows the aggregator (makes sense, since the build failed after building the aggregator). You'll also notice that a version selection (1.1.3/2.0.3) isn't possible. I haven't tried it, but it will likely download unpredictable versions again.

Here's a screenshot of the drop-down as it is right now for reference:

[IanMeyers]

Hrm yes - was expecting jitpack to just use the Assets from the release, as opposed to something else. Will do more research.

[anish-kunduru]

Any update on this?

[madorb]

any update on publishing to central??

[madorb]

jitpack doesn't seem to work with the 1.x version, so this is really quite painful.

[IanMeyers]

Unfortunately right now we are unable to publish to Maven because of the package structure: the project is community maintained, but the signing certs for Maven are part of the build system maintained by the Kinesis service team. The only alternative is to change the top-level package structure, which would be a huge impact on our user-base. I appreciate the pain that this creates, and we will continue to try and achieve this in future.

[acm19]

@IanMeyers would it be possible for you to publish it as a package in Github? It's not as good as Maven Central but it would solve the problem for a lot of people.

[IanMeyers]

We are looking into this and will come back to you.

[snehal-kolte]

Any plan to update on publishing to central??

[RubenCordeiro]

+1

[adrian-skybaker]

The only alternative is to change the top-level package structure, which would be a huge impact on our user-base

I would say you might want to repackage anyway, since the com.amazonaws.kinesis package is associated with the V1 SDK, which you no longer use in 2.0.

[adrian-skybaker]

Gradle config to ensure only this library is resolved from jitpack, not your entire dependency set:

repositories {
    exclusiveContent {
        forRepository {
            maven {
                url 'https://jitpack.io'
            }
        }
        filter {
            includeModule "com.github.awslabs.kinesis-aggregation", "amazon-kinesis-aggregator"
        }
    }
}