Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting org.gradle.jvmargs property can lead to "Daemon disappeared" failures #19750

Open
bigdaz opened this issue Jan 28, 2022 · 5 comments · May be fixed by #20054
Open

Setting org.gradle.jvmargs property can lead to "Daemon disappeared" failures #19750

bigdaz opened this issue Jan 28, 2022 · 5 comments · May be fixed by #20054

Comments

@bigdaz
Copy link
Member

bigdaz commented Jan 28, 2022

When investigating reports of "daemon disappeared" in GitHub Actions, and a feature request to address one cause, it was found that the all of our DEFAULT_JVM8_ARGS are being lost as soon as any value is set for org.gradle.jvmargs.

When org.gradle.jvmargs is not set by the user, then the daemon will start with default values specified here. But setting a value of org.gradle.jvmargs will overwrite these defaults, resulting in JDK defaults for -Xmx, -Xms etc.

One of the key defaults set by Gradle is -XX:MaxMetaspaceSize=256m, which is required because this value is unbounded on the JDK by default, allowing the Gradle daemon to consume more and more native memory until it crashes.

This means that a user setting org.gradle.jvmargs=-Xmx4g can result in daemon crashes, unless the user knows that they must also add a -XX:MaxMetaspaceSize setting.

In a similar way, the -XX:+HeapDumpOnOutOfMemoryError default is lost when a user specifies org.gradle.jvmargs.

Expected Behavior

Important JVM defaults are not lost when a user sets unrelated JVM arguments.

Current Behavior

Users must be aware of all important JVM argument defaults, and include all of these when setting any values for org.gradle.jvmargs.

Context

This was discovered when investigating "daemon disappeared" issues in GitHub Actions workflows.
Here is an example of a build with such a failure: https://github.com/androidx/androidx/runs/4971463356?check_suite_focus=true#step:10:3845. No build scan was captured due to the nature of the failure.

Daemon log file for that CI run:
gradle-daemon-logs_room.zip

@big-guy
Copy link
Member

big-guy commented Jan 28, 2022

One idea that's come up before is to break out some of the common things that people want to set, so instead of changing org.gradle.jvmargs, you would have several individual properties (e.g., heap size) and Gradle would use that to set the daemon arguments. org.gradle.jvmargs would remain as the "override everything" hammer.

@bigdaz
Copy link
Member Author

bigdaz commented Jan 28, 2022

One idea that's come up before is to break out some of the common things that people want to set, so instead of changing org.gradle.jvmargs, you would have several individual properties (e.g., heap size) and Gradle would use that to set the daemon arguments. org.gradle.jvmargs would remain as the "override everything" hammer.

Sure. I think better modelling can certainly help. Perhaps combined with a big warning if a user doesn't set certain important values when they set org.gradle.jvmargs. I expect there are a LOT of people who already set org.gradle.jvmargs and it would be nice to help them migrate.

Note that we already have set of JVM arg defaults that we always set, together with some defaults that we only set if org.gradle.jvmargs is not provided by the user.

@chadlwilson
Copy link

Thank you for raising this and articulating so clearly, including linking to the Gradle defaults which are lost with user overrides.

This likely explains a lot of weirdness since we upgraded GoCD's Gradle to run on Java 17, since this required us to add a custom org.gradle.jvmargs override to --add-opens allowing us to workaround an issue with our buildSrc on Groovy 3, due to dynamic proxies and Java 17 which has not been backported from Groovy 4.

Little did I know, I was overriding a whole lot of sensible limits/defaults on Gradle. Fingers crossed that we go back to stability after reinstating these! Have been tearing my hair out a bit, so hope is good enough for now!

chadlwilson added a commit to chadlwilson/gocd that referenced this issue Feb 16, 2022
… overridden

According to gradle/gradle#19750 as soon as we set the `--add-opens` here, we lost the constraints on JVM size and metaspace that Gradle normally sets. Oops. This possibly explains build instability (especially on Windows) and having to constrain our number of parallel workers in order to avoid our build containers exhausting memory and dying or getting killed by the container runtime.
chadlwilson added a commit to gocd/gocd that referenced this issue Feb 16, 2022
… overridden

According to gradle/gradle#19750 as soon as we set the `--add-opens` here, we lost the constraints on JVM size and metaspace that Gradle normally sets. Oops. This possibly explains build instability (especially on Windows) and having to constrain our number of parallel workers in order to avoid our build containers exhausting memory and dying or getting killed by the container runtime.
liutikas added a commit to androidx/androidx that referenced this issue Feb 22, 2022
copybara-service bot pushed a commit to androidx/androidx that referenced this issue Feb 23, 2022
## Proposed Changes

Due to gradle/gradle#19750 Gradle doesn't set a default
value for MaxMetaspaceSize when org.gradle.jvmargs is set and the JVM 8 default is
unbounded. This sets it explicitly to the default value defined in https://github.com/gradle/gradle/blob/6ccf8a2cc74c8060992bd36189dec929143caed7/subprojects/launcher/src/main/java/org/gradle/launcher/daemon/configuration/DaemonParameters.java#L39

## Testing

Test: None

## Issues Fixed

Daemon disappearing in CI

This is an imported pull request from #331.

Resolves #331
Github-Pr-Head-Sha: 9af0a74
GitOrigin-RevId: 5ec9a52
Change-Id: I669a8761fddee8fe4b1fb0c6e698256bd907612d
bigdaz added a commit that referenced this issue Feb 28, 2022
Previously, setting any value for `org.gradle.jvmargs` caused all default settings to be lost.
This often resulted in important defaults like `-XXMaxMetaspaceSize` being omitted when a user
attempted to provide more memory to a build process.

With this change, default jvmargs will be retained unless specifically overridden by a user-supplied argument.
One exception is that setting either -Xmx or -Xms will cause the default heap size settings to be omitted, preventing
user-supplied values from conflicting with default values (like having a min heap larger than max heap).

Fixes #19750
@bigdaz bigdaz linked a pull request Feb 28, 2022 that will close this issue
13 tasks
JavierSegoviaCordoba added a commit to arrow-kt/arrow that referenced this issue Mar 7, 2022
bigdaz added a commit that referenced this issue Mar 8, 2022
Previously, setting any value for `org.gradle.jvmargs` caused all default settings to be lost.
This often resulted in important defaults like `-XXMaxMetaspaceSize` being omitted when a user
attempted to provide more memory to a build process.

With this change, default jvmargs will be retained unless specifically overridden by a user-supplied argument.
One exception is that setting either -Xmx or -Xms will cause the default heap size settings to be omitted, preventing
user-supplied values from conflicting with default values (like having a min heap larger than max heap).

Fixes #19750
bigdaz added a commit that referenced this issue Mar 14, 2022
Previously, setting any value for `org.gradle.jvmargs` caused all default settings to be lost.
This often resulted in important defaults like `-XXMaxMetaspaceSize` being omitted when a user
attempted to provide more memory to a build process.

With this change, default jvmargs will be retained unless specifically overridden by a user-supplied argument.
One exception is that setting either -Xmx or -Xms will cause the default heap size settings to be omitted, preventing
user-supplied values from conflicting with default values (like having a min heap larger than max heap).

Fixes #19750
bigdaz added a commit that referenced this issue Mar 23, 2022
Previously, setting any value for `org.gradle.jvmargs` caused all default settings to be lost.
This often resulted in important defaults like `-XXMaxMetaspaceSize` being omitted when a user
attempted to provide more memory to a build process.

With this change, default jvmargs will be retained unless specifically overridden by a user-supplied argument.
One exception is that setting either -Xmx or -Xms will cause the default heap size settings to be omitted, preventing
user-supplied values from conflicting with default values (like having a min heap larger than max heap).

Fixes #19750
jvandort pushed a commit that referenced this issue Jul 26, 2022
Previously, setting any value for `org.gradle.jvmargs` caused all default settings to be lost.
This often resulted in important defaults like `-XXMaxMetaspaceSize` being omitted when a user
attempted to provide more memory to a build process.

With this change, default jvmargs will be retained unless specifically overridden by a user-supplied argument.
One exception is that setting either -Xmx or -Xms will cause the default heap size settings to be omitted, preventing
user-supplied values from conflicting with default values (like having a min heap larger than max heap).

Fixes #19750
mcarare added a commit to mcarare/firefox-android that referenced this issue Jul 31, 2023
According to gradle/gradle#19750 setting the org.gradle.jvmargs property can lead to "Daemon disappeared" failures
by removing the default values.
Also use -XX:+UseParallelGC to improve build performance.
mcarare added a commit to mcarare/firefox-android that referenced this issue Jul 31, 2023
According to gradle/gradle#19750 setting the org.gradle.jvmargs property can lead to "Daemon disappeared" failures
by removing the default values.
Also use -XX:+UseParallelGC to improve build performance.
mergify bot pushed a commit to mozilla-mobile/firefox-android that referenced this issue Jul 31, 2023
According to gradle/gradle#19750 setting the org.gradle.jvmargs property can lead to "Daemon disappeared" failures
by removing the default values.
Also use -XX:+UseParallelGC to improve build performance.
rvandermeulen pushed a commit to mozilla-mobile/firefox-android that referenced this issue Aug 18, 2023
According to gradle/gradle#19750 setting the org.gradle.jvmargs property can lead to "Daemon disappeared" failures
by removing the default values.
Also use -XX:+UseParallelGC to improve build performance.
akliuxingyuan pushed a commit to fork-maintainers/iceraven-browser that referenced this issue Sep 1, 2023
According to gradle/gradle#19750 setting the org.gradle.jvmargs property can lead to "Daemon disappeared" failures
by removing the default values.
Also use -XX:+UseParallelGC to improve build performance.
akliuxingyuan pushed a commit to fork-maintainers/iceraven-browser that referenced this issue Sep 1, 2023
According to gradle/gradle#19750 setting the org.gradle.jvmargs property can lead to "Daemon disappeared" failures
by removing the default values.
Also use -XX:+UseParallelGC to improve build performance.
@big-guy big-guy added this to the 9.0 RC1 milestone Nov 6, 2023
@jvandort
Copy link
Member

jvandort commented Nov 6, 2023

The last status of this was that changing this would be a breaking change:
https://docs.google.com/spreadsheets/d/1NP3HpEpnm4u3Pp0hBCdVreHeTCyt0MwYOOAWhcaOhb0/edit#gid=0

auto-submit bot pushed a commit to flutter/flutter that referenced this issue Feb 7, 2024
…aspaceSize` (#143085)

Re-sets two jvmargs that were getting cleared because we set a value for `-Xmx`. Could help with #142957. Copied from comment here #142957:
>Two random things I ran into while looking into this that might help:
>
>1. Gradle has defaults for a couple of the jvmargs, and setting any one of them clears those defaults for the others (bug here gradle/gradle#19750). This can cause the "Gradle daemon to consume more and more native memory until it crashes", though the bug typically has a different associated error. It seems worth it to re-set those defaults.
>2. There is a property we can set that will give us a heap dump on OOM ([-XX:HeapDumpOnOutOfMemoryError](https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/clopts001.html))

Mostly just a find and replace from `find . -name gradle.properties -exec sed -i '' 's/\-Xmx4G/-Xmx4G\ \-XX:MaxMetaspaceSize=2G\ \-XX:+HeapDumpOnOutOfMemoryError/g' {} \;`, with the templates and the one test that writes from a string replaced by hand. I didn't set a value for `MaxMetaspaceSize` in the template files because I want to make sure this value doesn't cause problems in ci first (changes to the templates are essentially un-revertable for those who `flutter create` while the changes exist).
jflo added a commit to jflo/besu that referenced this issue Mar 1, 2024
jflo added a commit to jflo/besu that referenced this issue Mar 5, 2024
jflo added a commit to jflo/besu that referenced this issue Mar 5, 2024
jflo added a commit to jflo/besu that referenced this issue Mar 7, 2024
jflo added a commit to jflo/besu that referenced this issue Mar 8, 2024
jflo added a commit to jflo/besu that referenced this issue Mar 8, 2024
jflo added a commit to jflo/besu that referenced this issue Mar 12, 2024
EdricChan03 added a commit to EdricChan03/studybuddy-android that referenced this issue Mar 31, 2024
See gradle/gradle#19750 and
https://developer.android.com/build/optimize-your-build#experiment-with-the-jvm-parallel-garbage-collector
for more info.
This should enable the parallel GC on Java 9+, as well as
other defaults that Gradle would set otherwise.
jmartinesp added a commit to element-hq/element-x-android that referenced this issue Jun 12, 2024
- Add `-XX:MaxMetaspaceSize` (gradle/gradle#19750)
- Remove `-PpreDexEnable` as it's deprecated and no longer used.
- Remove `--max-workers` as gradle will automatically use the optimal amount.
jmartinesp added a commit to element-hq/element-x-android that referenced this issue Jun 12, 2024
* Try to workaround OOM errors and the runner dying

* Some more tweaks:

- Add `-XX:MaxMetaspaceSize` (gradle/gradle#19750)
- Remove `-PpreDexEnable` as it's deprecated and no longer used.
- Remove `--max-workers` as gradle will automatically use the optimal amount.

* Remove `--max-workers` in recording screenshots too
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants