Releases · NVIDIA/spark-rapids-tools

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.04.0/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.04.0/

Changes

Core

[BUG] Catch Profiler error when app info is empty (#994)
Get stages from sqlId for collecting info for output writer functions (#996)
Account for joboverhead time in qualification tool estimation (#992)
[Followup] Fix handling of clusterTags and SparkVersion in Q/P Tools (#993)
Fix handling of clusterTags and SparkVersion in Q/P Tools (#991)
Refactor AppBase to use common AppMetaData between Q/P tools (#983)
Refactor Stage info code between Q/P tools (#971)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.4/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.4/

Changes

Core

Remove databricks profiling recommendation for dynamicFilePruning (#972)
Add AQEShuffleRead WriteFiles execs to the supportedOps and score files (#963)
[FEA] Automate appending new operators to the platform score sheets (#954)
Add support for InSubqueryExec Expression (#960)

Miscellaneous

Bump dev version to 24.02.4 (#968)
Revert versions back to 24.02.3 (#967)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.3/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.3/

Changes

Cache CLI calls for node instance description (#952)
Improve error handling in prediction code (#950)
Support dynamic calculation of JVM resources in CLI cmd (#944)
Syncup estimation model prediction logic updates (#946)
Cluster inference should not run for unsupported platform (#941)
Fix invalid values in cluster creation script (#935)
Fix core tool doc links and user qualification tool default argument values (#931)
Fix gpu cluster recommendation in user tools (#930)
Bump idna from 3.4 to 3.7 in /data_validation (#932)
Add cluster details in qualification summary output (#921)
Refactor find_matches_for_node return values (#920)
[FEA] Add and use g5 AWS instances as default for qualification tool output (#898)
Add jar argument to spark_rapids CLI (#902)
Support driverlog argument in profiler CLI (#897)

Core

Followups on handling Photon eventlogs (#953)
Sync operators support timestamped 24-04-16 (#951)
Add CheckOverflowInTableInsert support: verify absence from physical plan (#942)
Fix Notes column in the supported ops CSV files (#933)
Improve sync plugin supported CSV python script (#919)
Add cluster details in qualification summary output (#921)
Add support for unsupported expressions reasons per Exec (#923)
Adding more metrics and options for qual validation (#926)
Generate cluster details in JSON output (#912)
Add Divide and multiple interval expressions as supported (#917)
Add support for PythonMapInArrowExec and MapInArrowExec (#913)
Re-enable support for GetJsonObject by default (#916)
Add support for WindowGroupLimitExec (#906)
[FEA] Skip Spark Structured Streaming event logs for Qualification tool (#905)
[FEA] Add and use g5 AWS instances as default for qualification tool output (#898)
Initial version of qual tool validation script for classification metrics (#903)
Fix Delta-core dependency for Spark35+ (#904)
Add support for AtomicCreateTableAsSelectExec (#895)
Add support for KnownNullable and EphemeralSubstring expressions (#894)
Add Support for BloomFilterAggregate and BloomFilterMightContain exprs (#891)
[DOC] Update README for sync plugin supported ops script (#893)
Add operators to ignore list and update WindowExpr parser (#890)
Add support to RoundCeil and RoundFloor expressions (#889)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.2/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.2/

Changes

Override estimated speedups when estimation model is enabled (#885)
[FEA] Make top candidates view as the default view in user-tools (#879)
Introduce new csv file containing output for all apps before grouping (#875)
Fix calculation of unsupported operators stages duration and update output row (#874)
Implement top candidate filter for user tools CLI output (#866)

Core

[FEA] Skip Databricks Photon jobs at app level in Qualification tool (#886)
[FEA] Add Estimation Model to Qualification CLI (#870)
Add rootExecutionID to output csv files (#871)
[FEA] Generate updated supported CSV files from plugin repo (#847)
Add action column to qual execs output (#859)
Extend supportLevels in PluginTypeChecker (#863)
Propagate Reason/Notes for operators disabled by default from plugin to Qualification tool unsupported operators csv file (#850)

Miscellaneous

Bump default Spark-version to 3.5.0 (#877)
Update Github actions version (#876)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.1/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.1/

Changes

Core

Fix implementation of processSQLPlanMetrics in Profiler (#853)
Deduplicate SQL duration wallclock time for databricks eventlog (#810)
Consider additional factors in spark.sql.shuffle.partitions recommendation in Autotuner (#722)
Fix case matching error In AutoTuner (#828)
Fix ReadSchema in Qualification tool and NPE in Profiling tool (#825)
AutoTuner does not process arguments skipList and limitedLogic (#812)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/24.02.0/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/24.02.0/

Changes

Core

[FEA] Qualification tool should mark WriteIntoDeltaCommand as supported (#801)
Qualification tool should mark SubqueryExec as IgnoreNoPerf (#798)
Generate cluster information from event logs in Qualification tool (#789)
Sync up supported ops for 24.02 plugin release (#796)
Qualification should mark empty2null as supported (#791)
Incorrect parsing of aggregates in DB queries (#790)
Qualification should mark WriteFiles as supported (#784)
Introduce GpuDevice abstraction and refactor AutoTuner (#740)
Consolidate unsupportedOperators into a single view (#766)
Speedup generator script fails after adding runtime_properties (#776)
Tools fail on DB10.4 clusters with IllegalArgException (#768)
Fix SparkPlanGraphCluster constructor for DB Platforms (#765)
Amendment to PR-763 (#764)
Fix SQLPLanMetric constructor for DB Platforms (#763)
Fix node constructor for DB platforms (#761)
Add penalty for stages with UDF's (#757)
Add support to appendDataExecV1 and overwriteByExprExecV1 (#756)
Qualification fails to detect sortMergeJoin with arguments (#754)
Fix Qualification crash during aggregation of stats (#753)
[FEA] Extend the list of operators to be ignored in Qualification (#745)
Remove ReusedSubquery from SparkPlanGraph construction (#741)
Update unsupported operator csv file's app duration column (#748)
[FEA] Qualification tool triggers the AutoTuner module (#739)
Disable support of GetJsonObject in Qualification tool (#737)
[FEA] AutoTuner warns that non-utf8 may not support some GPU expressions (#736)
[FEA] AutoTuner should not skip non-gpu eventlogs (#728)

Miscellaneous

Add auto-copyright for precommits (#732)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/23.12.3/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/23.12.3/

Changes

Core

Add support of HiveTableScan and InsertIntoHive text-format (#723)
Fix compilation error with JDK11 (#720)
Generate an output file with runtime and build information (#705)
AutoTuner should poll maven-meta to retrieve the latest jar version (#711)
Profiling tool : Profiling tool throws NPE when appInfo is null and unchecked (#640)
Add support to parse_url host and protocol (#708)
[FEA] Profiling tool auto-tuner should consider spark.databricks.adaptive.autoOptimizeShuffle.enabled (#710)
[FEA] Profiler autotuner should only specify standard Spark versions for shuffle manager setting (#662)

Miscellaneous

[FEA] Enable AQE related recommendations in Profiler Auto-tuner (#688)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/23.12.2/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/23.12.2/

Changes

Core

Update pom to fail on warnings (#701)

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/23.12.1/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/23.12.1/

Changes

no changes

Packages

Maven Release: https://repo1.maven.org/maven2/com/nvidia/rapids-4-spark-tools_2.12/23.12.0/
PyPI Package: https://pypi.org/project/spark-rapids-user-tools/23.12.0/

Changes

Fix user qualification tool runtime error in get_platform_name for onprem platform (#684)
[FEA] User tool should pass --platform option/argument to Profiling tool (#679)
Fix incorrect processing of short flags for user tools cli (#677)
Updating new CLI name from ascli to spark_rapids (#673)
Bump pyarrow version (#664)
Improve new CLI testing ensuring complete coverage of arguments cases (#652)

Core

Qualification tool: Add more information for unsupported operators (#680)
Sync Execs and Expressions from spark-rapids resources (#691)
Support parsing of inprogress eventlogs (#686)
Enable features via config that are off by default in the profiler AutoTuner (#668)
Fix platform names as string constants and reduce redundancy in unit tests (#667)
Unified platform handling and fetching of operator score files (#661)
Qualification tool: Ignore some of the unsupported Execs from output (#665)

Miscellaneous

add markdown link checker (#672)

Releases: NVIDIA/spark-rapids-tools

v24.04.0

Packages

Changes

User Tools

Core

v24.02.4

Packages

Changes

User Tools

Core

Miscellaneous

v24.02.3

Packages

Changes

User Tools

Core

v24.02.2

Packages

Changes

User Tools

Core

Miscellaneous

v24.02.1

Packages

Changes

User Tools

Core

v24.02.0

Packages

Changes

User Tools

Core

Miscellaneous

v23.12.3

Packages

Changes

Core

Miscellaneous

v23.12.2

Packages

Changes

User Tools

Core

v23.12.1

Packages

Changes

v23.12.0

Packages

Changes

User Tools

Core

Miscellaneous