Skip to content

2018-03-19 (GCS 1.6.4, BQ 0.10.5)

Compare
Choose a tag to compare
@medb medb released this 19 Mar 21:50
· 83 commits to branch-1.6.x since this release

Changelog

Cloud Storage connector:

  1. Fixed an issue where JSON auth files containing user auth (e.g. application_default_credentials.json) does not work with google.cloud.auth.service.account.json.keyfile.

  2. Honor GOOGLE_APPLICATION_DEFAULT_CREDENTIALS environment variable. For Google Application Default Credentials (but not other defaults).

  3. Make fs.gs.project.id optional. It is still required for listing buckets, creating buckets, and entire BigQuery connector.

  4. Disable GCS Metadata Cache by default (e.g. set default value of fs.gs.metadata.cache.enable property to false).

  5. Support GCS Requester Pays feature that could be configured with new properties:

    fs.gs.requester.pays.mode (default=DISABLED)
    fs.gs.requester.pays.project.id (no default value)
    fs.gs.requester.pays.buckets (no default value)
    
  6. Add support for specifying marker files pattern that should be copied last during folder rename operation. Pattern is configured with property:

    fs.gs.marker.file.pattern (no default value)
    

BigQuery connector:

  1. POM updates for GCS connector 1.6.4.
  2. Remove Avro and Gson classes from Hadoop 2 shaded jar because they are already included in the Hadoop 2 distribution.