Infiniti-loop (?) in analyzer / reporter on huge Gradle project

[hendrikebbers]

Describe the bug

We found an Infiniti-loop in the analyzer.
The project we want to analyse is https://github.com/hashgraph/hedera-services and the .ort.yml can be found here: https://github.com/hendrikebbers/ort-scan-hedera/blob/main/ort-config/hedera-services.yml

When we initially executed the analyzer it always crashed with an OutOfMemoryError in the Gradle Daemon. Even if we set the memory to 30 GB. We figured out, that the problem is located to OrtModelBuilder.tk. We did a local fix for that class and introduced a cache of type mutableMapOf<ComponentIdentifier, OrtDependency>() that is used in the toOrtDependencies method. Instead of creating new OrtDependencyImpl instance all the time in the toOrtDependencies method we check if the cache already contains an instance with the given ID and than return that instance. I already discussed that change with @sschuberth and since some changes are needed I have not created a PR so far.

Our concrete changes as a git patch:

Subject: [PATCH] Adding cache to gradle analyzer
---
Index: plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt b/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt
--- a/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt	(revision 75cf08a046a115a216dc8610b041f99492bf14f0)
+++ b/plugins/package-managers/gradle-plugin/src/main/kotlin/OrtModelBuilder.kt	(revision de517cf972efdcea271805d06e70b1012a802c1c)
@@ -56,6 +56,8 @@
     private val warnings = mutableListOf<String>()
     private val globalDependencySubtrees = mutableMapOf<String, List<OrtDependency>>()
 
+    private val cache = mutableMapOf<ComponentIdentifier, OrtDependency>()
+
     override fun canBuild(modelName: String): Boolean = modelName == OrtDependencyTreeModel::class.java.name
 
     override fun buildAll(modelName: String, project: Project): OrtDependencyTreeModel {
@@ -141,6 +143,7 @@
         }.filterIsInstance<ResolvedArtifactResult>()
     }
 
+
     private fun Collection<DependencyResult>.toOrtDependencies(
         poms: Map<String, ModelBuildingResult>,
         visited: Set<ComponentIdentifier>
@@ -165,6 +168,10 @@
                     // Cut the graph on cyclic dependencies.
                     if (id in visited) return@mapNotNull null
 
+                    if(cache.containsKey(id)) {
+                        return@mapNotNull cache[id]
+                    }
+
                     when (id) {
                         is ModuleComponentIdentifier -> {
                             val pomFile = if (selectedComponent is ResolvedComponentResultInternal) {
@@ -211,7 +218,7 @@
                                 selectedComponent.dependencies.toOrtDependencies(poms, visited + id)
                             }
 
-                            OrtDependencyImpl(
+                            val dependency = OrtDependencyImpl(
                                 groupId = id.group,
                                 artifactId = id.module,
                                 version = id.version,
@@ -232,13 +239,15 @@
                                 },
                                 localPath = null
                             )
+                            cache[id] = dependency
+                            dependency
                         }
 
                         is ProjectComponentIdentifier -> {
                             val moduleId = selectedComponent.moduleVersion ?: return@mapNotNull null
                             val dependencies = selectedComponent.dependencies.toOrtDependencies(poms, visited + id)
 
-                            OrtDependencyImpl(
+                            val dependency = OrtDependencyImpl(
                                 groupId = moduleId.group,
                                 artifactId = moduleId.name,
                                 version = moduleId.version.takeUnless { it == "unspecified" }.orEmpty(),
@@ -251,6 +260,8 @@
                                 mavenModel = null,
                                 localPath = id.projectPath
                             )
+                            cache[id] = dependency
+                            dependency
                         }
 
                         else -> {

With that change the Gradle deamon does not dies anymore and the analyse results are send back sucessfully to the ORT process. Sadly that runs endlessly. I killed the process after 16 hours but attached an VisualVm agend in between to check if it is still running or just blocked. The process was running all the time in the same code areas as you can see on the attached screenshots.

To Reproduce

• Checkout https://github.com/hashgraph/hedera-services
• Create a .ort.yml file with the content of https://github.com/hendrikebbers/ort-scan-hedera/blob/main/ort-config/hedera-services.yml
• Checkout https://github.com/oss-review-toolkit/ort/
• Change OrtModelBuilder.tk based on the given git patch
• Run ORT with those parameters: --debug -P ort.analyzer.allowDynamicVersions=true analyze -i PATH/TO/REPO -o PATH/TO/REPO

Expected behavior

The process should finish after some time and the analyser result should be stores in a file.

[Etsija]

I'm seeing a similar issue in the Reporter. I have a pretty big Gradle project which I'm trying to analyze. The analyzer runs fine, producing the analyzer-result.yml, but when trying to produce some reports with

ort -Port.forceOverwrite=true --debug --info report -i /home/jyrki/ort-results/weather-app/analyzer-result.yml -o /home/jyrki/ort-results/weather-app -f WebApp,CycloneDX,SPDXDocument -O CycloneDX=output.file.formats=JSON,XML -O SPDXDocument=outputFileFormats=JSON,YAML

the reporter runs for about 10 minutes, then crashes with the stacktrace

13:11:16.232 [main] INFO  org.ossreviewtoolkit.plugins.commands.api.utils.ExtensionsKt - Read ORT result from 'analyzer-result.yml' (0.27 MiB) in 984.476455ms.
Generating 'WebApp' report(s) in thread 'DefaultDispatcher-worker-2'...
Generating 'CycloneDX' report(s) in thread 'DefaultDispatcher-worker-3'...
Generating 'SpdxDocument' report(s) in thread 'DefaultDispatcher-worker-4'...
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.base/java.lang.invoke.DirectMethodHandle.allocateInstance(DirectMethodHandle.java:500)
	at java.base/java.lang.invoke.LambdaForm$DMH/0x00007f6a37440000.newInvokeSpecial(LambdaForm$DMH)
	at java.base/java.lang.invoke.LambdaForm$MH/0x00007f6a37440800.linkToTargetMethod(LambdaForm$MH)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse$lambda$8(DependencyGraphNavigator.kt:105)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator$$Lambda/0x00007f6a3743ec08.invoke(Unknown Source)
	at org.ossreviewtoolkit.model.DependencyRefCursor.visitDependencies(DependencyGraphNavigator.kt:165)
	at org.ossreviewtoolkit.model.DependencyGraphNavigator.packageDependencies$traverse(DependencyGraphNavigator.kt:104)

Process finished with exit code 1

I monitored memory consumption with top, it showed almost 10GB memory being consumed when running the reporter.

[Etsija]

Here is the analyzer-result.yml for reference. I only ran the Analyzer, then tried to create a WebApp report from the result file. Usually takes a few seconds, but now, takes 2m 30s and ends up with

Failed to create 'WebApp' report: OutOfMemoryError: Java heap space

Here's my Analyzer result file for reference, in case someone wants to verify.

analyzer-result.zip

The reports have worked for me last time in release 45 of ORT.

[Ej400]

A similar issue was faced during the creation of a report for a Gradle project. I tried all the formats, but nothing seems to work. Additionally, while debugging, it appears to loop indefinitely while handling the dependencies graph.
config file :

ort:
configuration:
forceOverwrite: true
analyzer:
options:
file:
parallelThreads: 20
enabled: true
allowDynamicVersions: true
enabledPackageManagers:
- GradleInspector

[Etsija]

Updates: I went as far as v41, trying to run the Analyzer, then Reporter. Analyzer produced the zipped results file. Reporter crashed the laptop with 16GB memory given to Java.

Then I read the discussion #9875, and voilá, WebApp reports started to happily run again. Thanks @mnonnenmacher and @sschuberth for the tip!

analyzer-result.zip