Reverse timestamp issue with cell timestamps #2193
Comments
|
Hi, You are absolutely right that we need to document this behavior. The issue stems from the fact that bigtable stores timestamps as microseconds while hbase uses milliseconds. The hbase adapter tries to adjust the discrepancy by multiplying the hbase value by 1000. However the range of values that we can store is narrowed by this conversion. The hbase adapter did not consider reverse timestamps and assumed that if the timestamp higher than the highest storable value then we can just use the highest possible value. When reading it would convert it to just Long.MAX_VALUE. The sequence file importer/exporter will use the hbase client under the hood. So any value higher than HBASE_EFFECTIVE_MAX_TIMESTAMP will be read back as Long.MAX_VALUE. Other than documenting the behavior, there is no easy solution to this. As it stands I'd like to understand your use case better this is the first time this has come up as an issue. Can you describe what the usecase for reverse timestamps in cell values? |
|
We're getting hit by this as well. Are there any workarounds at the moment? The use case is we need to store a "First Seen" field, and a "Last Seen" field. Any info would be greatly appreciated. Thanks. |
|
This came up again and the silent truncation is causing a lot of confusion. Especially when it's combined with HBase behavior of treating Long.MAX_VALUE as the current timestamp. In the next minor release we will start logging warnings when this happens and add an opt-in configuration to throw errors instead of truncating. Eventually we would like to make error throwing a default behavior. The only workaround is to use a smaller timestamp to subtract from, so instead of: Use: |
This is a request to document some limitations we came across with Bigtable, and a question about the use of
bigtable-beam-importto copy data from HBase to Bigtable and vice-versa.Documentation requests
In our HBase table we do Put operations with a reversed timestamp, i.e.,
We do this to enforce a particular ordering, and this has worked fine with HBase.
When we used the
bigtable-hbase-1.xclient to do the same Put in Bigtable, the subsequent Get results all containedLong.MAX_VALUEtimestamp. We traced it down to thecom.google.cloud.bigtable.hbase.util.TimestampConverterwhich internally converts the time, and doesn't handle things well when the Put timestamp exceedsTimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMP. We are exploring ways to fix this.I think it will be useful if you document this issue in CBT Docs, especially considering the HBase suggestions around this which others may be following.
Questions
Will you make the
TimestampConverter.HBASE_EFFECTIVE_MAX_TIMESTAMPpublic? Because one fix we are exploring is to useHBASE_EFFECTIVE_MAX_TIMESTAMP - timeNowMillisinstead ofLong.MAX_VALUE - timeNowMillisin our Puts; we are vary of computing it ourselves in caseTimestampConverter.FACTORchanges.Also, how does this work while using
bigtable-beam-import:a) Say, I have an HBase table with cells with timestamp in milliseconds. And I export this to sequencefiles using HBase's
Export. When I import this sequencefile into Bigtable usingbigtable-beam-import, will it do thehbase2bigtable()translation on the sequencefile cell timestamps?b) If I export the data in Bigtable using
bigtable-beam-importexport, will it do the reverse translation usingbigtable2hbase()? i.e., can I expect millisecond timestamps in the exported sequencefile or will it be microsecond timestamps?c) How do these work if the HBase table had cells created with reverse timestamps, i.e.,
Long.MAX_VALUE - timeNowMillis?The text was updated successfully, but these errors were encountered: