gradle does not provide modules/*.jar for transient dependencies (regression)

[keturn]

What you were trying to do

Get a module's sources for development and play testing.

What actually happened

Got module's sources, built module, but the game was not able to resolve the module's dependencies in the Advanced Game Setup screen.

How to reproduce

• start with empty modules/ workspace
• groovyw module get JoshariasSurvival
• gradlew jar
• gradlew game
• create new game, advanced setup, observe JoshariasSurvival is lacking its transient dependencies

Log details and game version

I'm not actually watching this error happen right now, this report is assembled from Niruandaleth's notes and my understanding of gradle.

Discussion

I expect this started when Core Upheaval removed transient dependencies from many module.txt at around the same time as the Gradle 6.4 branch made some adjustments to how it treats compile-time dependencies.

The thing that makes the .jar files show up in the modules/ directory when a module is built from source is the RemoteModuleGatherer we have in our gradle configuration.

The crux of the issue is here:

Terasology/build.gradle

Line 262 in 9575439

dependencies.name == "compileClasspath") {

it's only checking the compile-time classpath, which don't include the transient dependencies.

Changing that to include the run-time classpath is a small change. However, I'm not fully confident that the PC facade knows it needs to resolve the runtime classpath of the modules before it starts. If it doesn't, there will be nothing for the RemoteModuleGatherer to observe and copy.

[keturn]

Okay, I've reproduced the issue, and I'm trying to test if my fix is making anything better.

I think it put more jar files under modules/ directory this time, but apparently that was not sufficient?

the game thinks I don't have any world generators right now. I don't know why.

[keturn]

The log shows "WorldGeneratorManager - Could not resolve dependencies for module" for

• JoshariasSurvival-1.1.0
• AdditionalFruits-1.0.0
• SimpleFarming-2.1.0
• EdibleFlora-1.0.0

[keturn]

A way to force gradle to resolve runtime dependencies for a module (which it otherwise doesn't do for a library it assembles but does not execute) is to run this:

gradlew :modules:JoshariasSurvival:dependencyReport --configuration runtimeClasspath

we'll work out some way to make sure these are resolved automatically before the game starts, but it's a handy thing to know if you want to trigger it manually.

[keturn]

trying to squeeze some information out of the resolver by having the debugger spit out some things when a constraint rules out a version.

Here's what I got during start-up: https://gist.github.com/keturn/7319b3a0814686a7464677503cb3c2cf

and these resolvers run again when you get in to the create game and AGS screens. Not entirely sure that they're initialized the same way in all places.

[keturn]

okay, I might have found the problem? Terasology/SimpleFarming#92 (comment)

that is, a problem in some of versioning data that is preventing this game from starting. if that is the case, it's not directly related to this issue.

and I haven't figured out how to systemically prevent it or give a better interface to the validations so we find the problem sooner

[keturn]

took a detour to do #4032, as having gradle look at versions seemed useful for straightening out what is going on here.

[Cervator]

For a quick reference the likely changed behavior came in with 7f15c30#diff-c197962302397baf3a4cc36463dce5eaR250, I think?

[keturn]

Oh heck I forgot about this! It's a significant issue but it was waiting on #4032 for so long. That has now been merged; we should see if this issue is still reproducible.

[keturn]

Confirmed this is still reproducible in the current develop branch. i.e. it gets AdditionalFruits but not its dependency EdibleFlora.

side note: gradle is also really slow to resolve JoshariasSurvival dependencies. Over two minutes in total. My hunch is that it has to ask artifactory a lot of questions, and our artifactory is slow?

or maybe it's because repository order matters, and our artifactory comes after jcenter and mavenCentral, so it fails to find each of the modules in those before it checks ours?

[Cervator]

Good research - it could be an interesting experiment to try optimizing repo order, as well as actually cache the artifacts we use from 3rd party repos in ours, so they can be retrieved out of a far smaller cache in our repository. I've played a bit with Nexus instead of Artifactory and am sure that's doable. Also considering switching, and putting a Nexus in our Kubernetes cluster as opposed to Artifactory, as the free Nexus also supports Docker, Helm, and other formats, while free Artifactory just covers your typical jar files and such.

Definitely looking forward to that although it is a little down the list. On the plus side I might have day job justification to work on it more sometime :-)