Skip to content

2018-08-10 (GCS 1.6.8, BQ 0.10.9)

Compare
Choose a tag to compare
@medb medb released this 10 Aug 18:11
· 12 commits to branch-1.6.x since this release

Changelog

Cloud Storage connector:

  1. Support parallel execution of GCS batch requests.

    Number of threads to execute batch requests configurable via property:

    fs.gs.batch.threads (default: 0)
    

    If fs.gs.batch.threads value is set to 0 then batch requests will be executed sequentially by caller thread.

  2. Do not send batch request when performing operations (rename, delete, copy) on 1 object.

  3. Add configuration properties to control batching of copy operations separately from other operations:

    fs.gs.copy.max.requests.per.batch (default: 30)
    fs.gs.copy.batch.threads (default: 0)
    
  4. Fix RejectedExecutionException during parallel execution of GCS batch requests.

  5. Change default values for GCS batch/directory operations properties:

    fs.gs.copy.with.rewrite.enable (default: false -> true)
    fs.gs.copy.max.requests.per.batch (default: 30 -> 1)
    fs.gs.copy.batch.threads (default: 0 -> 50)
    fs.gs.max.requests.per.batch (default: 30 -> 25)
    fs.gs.batch.threads (default: 0 -> 25)
    

BigQuery connector:

  1. POM updates for GCS connector 1.6.8.