1.6.10 - 2018-09-20

1. Add an option to specify max bytes rewritten per rewrite request when

`fs.gs.copy.with.rewrite.enable` is set to `true`:

fs.gs.rewrite.max.bytes.per.call (default: 536870912)

Even though GCS does not require this parameter for rewrite requests,

rewrite requests are flaky without it.

1.6.9 - 2018-09-04

1. Change default values for GCS batch/directory operations properties to

improve performance:

fs.gs.copy.max.requests.per.batch (default: 1 -> 15)

fs.gs.copy.batch.threads (default: 50 -> 15)

fs.gs.max.requests.per.batch (default: 25 -> 15)

fs.gs.batch.threads (default: 25 -> 15)

2. Update all dependencies to latest versions.

1.6.8 - 2018-08-10

1. Support parallel execution of GCS batch requests.

Number of threads to execute batch requests configurable via property:

fs.gs.batch.threads (default: 0)

If `fs.gs.batch.threads` value is set to 0 then batch requests will be

executed sequentially by caller thread.

2. Do not send batch request when performing operations (rename, delete, copy)

on 1 object.

3. Add configuration properties to control batching of copy operations

separately from other operations:

fs.gs.copy.max.requests.per.batch (default: 30)

fs.gs.copy.batch.threads (default: 0)

4. Fix `RejectedExecutionException` during parallel execution of GCS batch

requests.

5. Change default values for GCS batch/directory operations properties:

fs.gs.copy.with.rewrite.enable (default: false -> true)

fs.gs.copy.max.requests.per.batch (default: 30 -> 1)

fs.gs.copy.batch.threads (default: 0 -> 50)

fs.gs.max.requests.per.batch (default: 30 -> 25)

fs.gs.batch.threads (default: 0 -> 25)

1.6.7 - 2018-06-12

1. Remove Hadoop3 support.

2. Add interface through which user can directly provide the access token.

3. Update Hadoop dependencies to 2.8.4 version.

4. Add more retries and error handling in GoogleCloudStorageReadChannel,

to make it more resilient to network errors; also add a property to allow users

to specify number of retries on low level GCS HTTP requests in case of server

errors and I/O errors.

5. Add properties to allow users to specify connect timeout and read timeout on

low level GCS HTTP requests.

6. Include prefix/directory objects metadata into `storage.objects.list`

requests response to improve performance (i.e. set

`includeTrailingDelimiter` parameter for `storage.objects.list` GCS

requests to `true`).

1.6.6 - 2018-05-08

1. Support Hadoop 3.

2. Change Maven project structure to be better compatible with IDEs.

3. Fix thread leaks that were occurring when YARN log aggregation uploaded

logs to GCS.

1.6.5 - 2018-04-12

1. Add support for using Cloud Storage Rewrite requests for copy operation:

fs.gs.copy.with.rewrite.enable (default: false)

This allows to copy files between different locations and storage classes.

2. Update all dependencies to latest versions.

3. Decrease default value for max requests per batch from 1,000 to 30.

4. Make max requests per batch value configurable with property:

fs.gs.max.requests.per.batch (default: 30)

1.6.4 - 2018-03-19

1. Fixed an issue where JSON auth files containing user auth e.g.

application_default_credentials.json does not work with

google.cloud.auth.service.account.json.keyfile

2. Honor GOOGLE_APPLICATION_DEFAULT_CREDENTIALS environment variable. For

Google Application Default Credentials (but not other defaults).

3. Make fs.gs.project.id optional. It is still required for listing buckets,

creating buckets, and entire BigQuery connector.

4. Disable GCS Metadata Cache by default (e.g. set default value of

`fs.gs.metadata.cache.enable` property to `false`).

5. Support GCS Requester Pays feature that could be configured with new

properties:

fs.gs.requester.pays.mode (default=DISABLED)

fs.gs.requester.pays.project.id (no default value)

fs.gs.requester.pays.buckets (no default value)

6. Add support for specifying marker files pattern that should be copied

last during folder rename operation. Pattern is configured with property:

fs.gs.marker.file.pattern

1.6.3 - 2018-01-25

1. Use new GCS batch requests endpoint.

6. Disable GCS Metadata Cache by default (e.g. set default value of

`fs.gs.metadata.cache.enable` property to `false`).

1.6.2 - 2017-11-21

1. Wire HTTP transport settings into Credential logic.

1.6.1 - 2017-04-14

1. Added a polling loop when determining if a createEmptyObjects error can

safely be ignored and expanded the cases in which we will attempt to

determine if an empty object already exists.

Previously, if a rate limiting exception was encountered while creating

empty objects the connector would issue a single get request for that

object. If the object exists and is zero length we would consider the

createEmptyObjects call successful and suppress the rate limit exception.

The new implementation will poll for the existence of the object, up to

a user-configurable maximum, and will poll when either a rate limiting

error occurs or when a 500-level error occurs. The maximum can be

configured by the following setting:

fs.gs.max.wait.for.empty.object.creation.ms

Any positive value for this setting will be interpreted to mean "poll

for up to this many milliseconds before making a final determination".

The default value will cause a maximum wait of 3 seconds. Polling can

be disabled by setting this key to 0.

1.6.0 - 2016-12-16

1. Added new PerformanceCachingGoogleCloudStorage; unlike the existing

CacheSupplementedGoogleCloudStorage which only serves as an advisory

cache for enforcement of list consistency, the new optional caching

layer is able to serving certain metadata and listing requests purely

out of a short-lived in-memory cache to enhance performance of

some workloads. By default this feature is disabled, and can be

controlled with the config settings:

fs.gs.performance.cache.enable=true (default=false)

fs.gs.performance.cache.list.caching.enable=true (default=false)

The first option enables the cache to serve getFileStatus requests,

while the second option additionally enables serving listStatus. The

duration of cache entries can be controlled with:

fs.gs.performance.cache.max.entry.age.ms (default=3000)

It is not recommended to always run with this feature enabled; it should

be used specifically to address cases where frameworks perform redundant

sequential list/stat operations in a non-distributed manner, and on

datasets which are not frequently changing. It is additionally advised

to validate data integrity separately whenever using this feature. There

is no cooperative cache invalidation between different processes when

using this feature, so concurrent mutations to a location from multiple

clients *will* produce inconsistent/stale results if this feature is

enabled.

1.5.5 - 2016-11-04

1. Minor refactoring of logic in CacheSupplementedGoogleCloudStorage to

extract a reusable ForwardingGoogleCloudStorage that can be used

for other GCS-delegating implementations.

1.5.4 - 2016-10-05

1. Fixed a bug in GoogleCloudStorageReadChannel where multiple in-place

seeks without any "read" in between could throw a range exception.

2. Fixed plumbing of GoogleCloudStorageReadOptions into the in-memory

test helpers to improve unittest alignment with integration tests.

3. Fixed handling of parent timestamp updating when a full URI is provided

as a path for which timestamps should be updated. This allows specifying

gs://bucket/p1/p2/object instead of simply p1/p2/object.

4. Updated Hadoop 2 dependency to 2.7.2.

5. Imported full set of Hadoop FileSystem contract tests, except for the

currently-unsupported Concat and Append tests. Minor changes to pass

all the tests:

-available() now throws ClosedChannelException if called after close()

instead of returning 0.

-read() and seek() throw ClosedChannelException if called after close()

instead of NullPointerException.

-Out-of-bounds seeks now throw EOFException instead of

IllegalArgumentException.

-Blocked overwrites throw org.apache.hadoop.fs.FileAlreadyExistsException

instead of generic IOException.

-Deleting root '/' recursively or non-recursively when empty will no

longer delete the underlying GCS bucket by default. To re-enable

deletion of GCS buckets when recursively deleting root, set

'fs.gs.bucket.delete.enable' = true.

1.5.3 - 2016-09-21

1. Misc updates in credential acquisition on GCE.

1.5.2 - 2016-08-23

1. Updated AbstractGoogleAsyncWriteChannel to always set the

X-Goog-Upload-Desired-Chunk-Granularity header independently from the

deprecated X-Goog-Upload-Max-Raw-Size; in general this improves

performance of large uploads.

1.5.1 - 2016-07-29

1. Optimized InMemoryDirectoryListCache to use a TreeMap internally instead

of a HashMap, since most of its usage is "listing by prefix"; aside

from some benefits for normal list-supplementation, glob listings

with a large number of subdirectories should be orders of magnitude

faster.

1.5.0 - 2016-07-15

1. Changed the single-argument constructor

GoogleCloudStorageFileSystem(GoogleCloudStorage) to inherit the inner

GoogleCloudStorageOptions from the passed-in GoogleCloudStorage rather

than simply falling back to default GoogleCloudStorageFileSystemOptions.

2. Changed RetryHttpInitializer to treat HTTP code 429 as a retriable

error with exponential backoff instead of erroring out.

3. Added a new output stream type which can be used by setting:

fs.gs.outputstream.type=SYNCABLE_COMPOSITE

With the SYNCABLE_COMPOSITE type, output streams obtained with a call

to FileSystem.create(Path) will have (limited) support for Hadoop's

Syncable interface, where calling hsync() will commit the data written

so far into persistent storage without closing the output stream; once

a writer calls hsync(), readers will be able to discover and read the

contents of the file written up to the last successful hsync() call

immediately. This feature is implemented using "composite objects" in

Google Cloud Storage, and thus has a current hard limit of 1023 calls

to hsync() over the lifetime of the stream after which subsequent hsync()

calls will throw a CompositeLimitExceededException, but will still

allow writing additional data and closing the stream without losing data

despite a failed hsync().

4. Added several optimizations and options controlling the behavior of

the GoogleHadoopFSInputStream:

-Removed internal prepopulated ByteBuffer inside GoogleHadoopFSInputStream

in favor of non-blocking buffering in the lower layer channel; this

improves performance for independent small reads without noticeable

impact on large reads. Old behavior can be specified by setting

fs.gs.inputstream.internalbuffer.enable=true (default=false)

-Added option to save an extra metadata GET on first read of a channel

if objects are known not to use the 'Content-Encoding' header. To

use this optimization set:

fs.gs.inputstream.support.content.encoding.enable=false (default=true)

-Added option to save an extra metadata GET on FileSystem.open(Path) if

objects are known to exist already, or the caller is resilient to delayed

errors for nonexistent objects which occur at read time, rather than

immediately on open(). To use this optimization set:

fs.gs.inputstream.fast.fail.on.not.found.enable=false (default=true)

-Added support for "in-place" small seeks by defining a byte limit where

forward seeks smaller than the limit will be done by reading/discarding

bytes rather than opening a brand-new channel. This is set to 8MB by

default, which is approximately the point where the cost of opening a

new channel is equal to the cost of reading/discarding the bytes in-place.

Disable by setting the value to 0, or adjust to other thresholds:

fs.gs.inputstream.inplace.seek.limit=0 (default=8388608)

1.4.5 - 2016-03-24

1. Add support for paths that cannot be parsed by Java's URI.create method.

This support is off by default, but can be enabled by setting Hadoop

configuration key fs.gs.path.encoding to the string uri-path. The

current behavior, and default value of fs.gs.path.encoding, is 'legacy'.

The new path encoding scheme will become the default in a future release.

2. VerificationAttributes are now exposed in GoogleCloudStorageItemInfo. The

current support is limited to reading these attributes from what was

computed by GCS server side. A future release will add support for

specifying VerificationAttributes in CreateOptions when creating new

objects.

1.4.4 - 2016-02-02

1. Add support for JSON keyfiles via a new configuration key:

google.cloud.auth.service.account.json.keyfile. This key should

point to a file that is available locally to jobs to clusters.

1.4.3 - 2015-11-12

1. Minor bug fixes and enhancements.

1.4.2 - 2015-09-15

1. Added checking in GoogleCloudStorageImpl.createEmptyObject(s) to handle

rateLimitExceeded (429) errors by fetching the fresh underlying info

and ignoring the error if the object already exists with the intended

metadata and size. This fixes an issue which mostly affects Spark:

https://github.com/GoogleCloudPlatform/bigdata-interop/issues/10

2. Added logging in GoogleCloudStorageReadChannel for high-level retries.

3. Added support for configuring the permissions reported to the Hadoop

FileSystem layer; the permissions are still fixed per FileSystem instance

and aren't actually enforced, but can now be set with:

fs.gs.reported.permissions [default = "700"]

This allows working around some clients like Hive-related daemons and tools

which pre-emptively check for certain assumptions about permissions.

1.4.1 - 2015-07-09

1. Switched from the custom SeekableReadableByteChannel to

Java 7's java.nio.channels.SeekableByteChannel.

2. Removed the configurable but default-constrained 250GB upload limit;

uploads can now exceed 250GB without needing to modify config settings.

3. Added helper classes related to GCS retries.

4. Added workaround support for read retries on objects with content-encoding

set to gzip; such content encoding isn't generally correct to use since

it means filesystem reported bytes will not match actual read bytes, but

for cases which accept byte mismatches, the read channel can now manually

seek to where it left off on retry rather than having a GZIPInputStream

throw an exception for a malformed partial stream.

5. Added an option for enabling "direct uploads" in

GoogleCloudStorageWriteChannel which is not directly used by the Hadoop

layer, but can be used by clients which directly access the lower

GoogleCloudStorage layer.

6. Added CreateBucketOptions to the GoogleCloudStorage interface so that

clients using the low-level GoogleCloudStorage directly can create buckets

with different locations and storage classes.

7. Fixed https://github.com/GoogleCloudPlatform/bigdata-interop/issues/5 where

stale cache entries caused stuck phantom directories if the directories

were deleted using non-Hadoop-based GCS clients.

8. Fixed a bug which prevented the Apache HTTP transport from working with

Hadoop 2 when no proxy was set.

9. Misc updates in library dependencies; google.api.version

(com.google.http-client, com.google.api-client) updated from 1.19.0 to

1.20.0, google-api-services-storage from v1-rev16-1.19.0 to

v1-rev35-1.20.0, and google-api-services-bigquery from v2-rev171-1.19.0

to v2-rev217-1.20.0, and Guava from 17.0 to 18.0.

1.4.0 - 2015-05-27

1. The new inferImplicitDirectories option to GoogleCloudStorage tells

it to infer the existence of a directory (such as foo) when that

directory node does not exist in GCS but there are GCS files

that start with that path (such as as foo/bar). This allows

the GCS connector to be used on read-only filesystems where

those intermediate directory nodes can not be created by the

connector. The value of this option can be controlled by the

Hadoop boolean config option "fs.gs.implicit.dir.infer.enable".

The default value is true.

2. Increased Hadoop dependency version to 2.6.0.

3. Fixed a bug introduced in 1.3.2 where, during marker file creation,

file info was not properly updated between attempts. This lead

to backoff-retry-exhaustion with 412-preconditon-not-met errors.

4. Added support for changing the HttpTransport implementation to use,

via fs.gs.http.transport.type = [APACHE | JAVA_NET]

5. Added support for setting a proxy of the form "host:port" via

fs.gs.proxy.address, which works for both APACHE and JAVA_NET

HttpTransport options.

6. All logging converted to use slf4j instead of the previous

org.apache.commons.logging.Log; removed the LogUtil wrapper which

previously wrapped org.apache.commons.logging.Log.

7. Automatic retries for premature end-of-stream errors; the previous

behavior was to throw an unrecoverable exception in such cases.

8. Made close() idempotent for GoogleCloudStorageReadChannel

9. Added a low-level method for setting Content-Type metadata in the

GoogleCloudStorage interface.

10.Increased default DirectoryListCache TTL to 4 hours, wired out TTL

settings as top-level config params:

fs.gs.metadata.cache.max.age.entry.ms

fs.gs.metadata.cache.max.age.info.ms

1.3.3 - 2015-02-26

1. When performing a retry in GoogleCloudStorageReadChannel, attempts to

close() the underlying channel are now performed explicitly instead of

waiting for performLazySeek() to do it, so that SSLException can be

caught and ignored; broken SSL sockets cannot be closed normally,

and are responsible for already cleaning up on error.

2. Added explicit check of currentPosition == size when -1 is read from

underlying stream in GoogleCloudStorageReadChannel, in case the

stream fails to identify an error case and prematurely reaches

end-of-stream.

1.3.2 - 2015-01-22

1. In the create file path, marker file creation is now configurable. By

default, marker files will not be created. The default is most suitable

for MapReduce applications. Setting fs.gs.create.marker.files.enable to

true in core-site.xml will re-enable marker files. The use of marker files

should be considered for applications that depend on early failing when

two concurrent writes attempt to write to the same file. Note that file

overwrite semantics are preserved with or without marker files, but

failures will occur sooner with marker files present.

1.3.1 - 2014-12-16

1. Fixed a rare NullPointerException in FileSystemBackedDirectoryListCache

which can occur if a directory being listed is purged from the cache

between a call to "exists()" and "listFiles()".

2. Fixed a bug in GoogleHadoopFileSystemCacheCleaner where cache-cleaner

fails to clean any contents when a bucket is non-empty but expired.

3. Fixed a bug in FileSystemBackedDirectoryListCache which caused garbage

collection to require several passes for large directory hierarchies;

now we can successfully garbage-collect an entire expired tree in a

single pass, and cache files are also processed in-place without having

to create a complete in-memory list.

4. Updated handling of new file creation, file copying, and file deletion

so that all object modification requests sent to GCS contain preconditions

that should prevent race-conditions in the face of retried operations.

1.3.0 - 2014-10-17

1. Directory timestamp updating can now be controlled via user-settable

properties "fs.gs.parent.timestamp.update.enable",

"fs.gs.parent.timestamp.update.substrings.excludes". and

"fs.gs.parent.timestamp.update.substrings.includes" in core-site.xml. By

default, timestamp updating is enabled for the YARN done and intermediate

done directories and excluded for everything else. Strings listed in

includes take precedence over excludes.

2. Directory timestamp updating will now occur on a background thread inside

GoogleCloudStorageFileSystem.

3. Attempting to acquire an OAuth access token will be now be retried when

using .p12 or installed application (JWT) credentials if there is a

recoverable error such as an HTTP 5XX response code or an IOException.

4. Added FileSystemBackedDirectoryListCache, extracting a common interface

for it to share with the (InMemory)DirectoryListCache; instead of using

an in-memory HashMap to enforce only same-process list consistency, the

FileSystemBacked version mirrors GCS objects as empty files on a local

FileSystem, which may itself be an NFS mount for cluster-wide or even

potentially cross-cluster consistency groups. This allows a cluster to

be configured with a "consistent view", making it safe to use GCS as the

DEFAULT_FS for arbitrary multi-stage or even multi-platform workloads.

This is now enabled by default for machine-wide consistency, but it is

strongly recommended to configure clusters with an NFS directory for

cluster-wide strong consistency. Relevant configuration settings:

fs.gs.metadata.cache.enable [default: true]

fs.gs.metadata.cache.type [IN_MEMORY (default) | FILESYSTEM_BACKED]

fs.gs.metadata.cache.directory [default: /tmp/gcs_connector_metadata_cache]

5. Optimized seeks in GoogleHadoopFSDataInputStream which fit within

the pre-fetched memory buffer by simply repositioning the buffer in-place

instead of delegating to the underlying channel at all.

6. Fixed a performance-hindering bug in globStatus where "foo/bar/*" would

flat-list "foo/bar" instead of "foo/bar/"; causing the "candidate matches"

to include things like "foo/bar1" and "foo/bar1/baz", even though the

results themselves would be correct due to filtering out the proper glob

client-side in the end.

7. The versions of java API clients were updated to 1.19 derived versions.

1.2.9 - 2014-09-18

1. When directory contents are updated e.g., files or directories are added,

removed, or renamed the GCS connector will now attempt to update a

metadata property on the parent directory with a modification time. The

modification time recorded will be used as the modification time in

subsequent FileSystem#getStatus(...), FileSystem#listStatus and

FileSystem#globStatus(...) calls and is the time as reported by

the system clock of the system that made the modification.

1.2.8 - 2014-08-07

1. Changed the manner in which the GCS connector jar is built to A) reduce

included dependencies to only those parts which are used and B) repackaged

dependencies whose versions conflict with those bundled with Hadoop.

2. Deprecated fs.gs.system.bucket config.

1.2.7 - 2014-06-23

1. Fixed a bug where certain globs incorrectly reported the parent directory

being not found (and thus erroring out) in Hadoop 2.2.0 due to an

interaction with the fs.gs.glob.flatlist.enable feature; doesn't affect

Hadoop 1.2.1 or 2.4.0.

1.2.6 - 2014-06-05

1. When running in hadoop 0.23+ (hadoop 2+), listStatus will now throw a

FileNotFoundException when a non-existent path is passed in.

2. The GCS connector now uses the v1 JSON API when accessing Google

Cloud Storage.

3. The GoogleHadoopFileSystem now treats the parent of the root path as if

it is the root path. This behavior mimics the POSIX behavior of "/.."

being the same as "/".

4. When creating a new file, a zero-length marker file will be created

before the FSDataOutputStream is returned in create(). This allows for

early detection of overwrite conflicts that may occur and prevents

certain race conditions that could be encountered when relying on

a single exists() check.

5. The dependencies on cglib and asm were removed from the GCS connector

and the classes for these are no longer included in the JAR.

1.2.5 - 2014-05-08

1. Fixed a bug where fs.gs.auth.client.file was unconditionally being

overwritten by a default value.

2. Enabled direct upload for directory creation to save one round-trip call.

3. Added wiring for GoogleHadoopFileSystem.close() to call through to close()

its underlying helper classes as well.

4. Added a new batch mode for creating directories in parallel which requires

manually parallelizing in the client. Speeds up nested directory creation

and repairing large numbers of implicit directories in listStatus.

5. Eliminated redundant API calls in listStatus, speeding it up by ~half.

6. Fixed a bug where globStatus didn't correctly handle globs containing '?'.

7. Implemented new version of globStatus which initially performs a flat

listing before performing the recursive glob logic in-memory to

dramatically speed up globs with lots of directories; the new behavior is

default, but can disabled by setting fs.gs.glob.flatlist.enable = false.

1.2.4 - 2014-04-09

1. The value of fs.gs.io.buffersize.write is now rounded up to 8MB if set to

a lower value, otherwise the backend will error out on unaligned chunks.

2. Misc refactoring to enable reuse of the resumable upload classes in other

libraries.

1.2.3 - 2014-03-21

1. Fixed a bug where renaming a directory could cause the file contents to get

shuffled between files when the fully-qualified file paths have different

lengths. Does not apply to renames on files directly, such as when using

a glob expression inside a flat directory.

2. Changed the behavior of batch request API calls such that they are retried

on failure in the same manner as non-batch requests.

3. Eliminated an unnecessary dependency on com/google/protobuf which could

cause version-incompatibility issues with Apache Shark.

1.2.2 - 2014-02-12

1. Fixed a bug where filenames with '+' were unreadable due to premature

URL-decoding.

2. Modified a check to allow fs.gs.io.buffersize.write to be a non-multiple

of 8MB, just printing out a warning instead of check-failing.

3. Added some debug-level logging of exceptions before throwing in cases

where Hadoop tends to swallows the exception along with its useful info.

1.2.1 - 2014-01-23

1. Added CHANGES.txt for release notes.

2. Fixed a bug where accidental URI decoding make it impossible to use

pre-escaped filenames, e.g. foo%3Abar. This is necessary for Pig to work.

3. Fixed a bug where an IOException was thrown when trying to read a

zero-byte file. Necessary for Pig to work.

1.2.0 - 2014-01-14

1. Preview release of GCS connector.