Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug-Upgrading Cloud Firestore SDK causes crash loop for some users #370

Closed
StillLynnTheCloset opened this issue Apr 22, 2019 · 8 comments · Fixed by #374
Closed

Bug-Upgrading Cloud Firestore SDK causes crash loop for some users #370

StillLynnTheCloset opened this issue Apr 22, 2019 · 8 comments · Fixed by #374
Assignees
@schmidt-sebastian
Labels
api: firestore

Comments

@StillLynnTheCloset
Copy link

[READ] Step 1: Are you in the right place?

Yes, this is an issue with Cloud Firestore's internal on-device sqlite database and this repository appears to contain all of the code required to manage that database.

[REQUIRED] Step 2: Describe your environment

  • Android Studio version: 3.4
  • Firebase Component: Cloud Firestore
  • Component version: 17.0.2 upgrading to 18.2.0
  • Gradle version: 4.10.1
  • com.android.tools.build:gradle:3.3.2
  • Compile/Target SDK version: 28

[REQUIRED] Step 3: Describe the problem

Immediately after releasing an app update which included the library upgrade we started to see users crash looping, seemingly starting when the app first restarted after updating.

Crashlytics Stack Trace

# Crashlytics - plaintext stacktrace downloaded by Developer at Mon, 22 Apr 2019 19:19:01 GMT
# URL: https://fabric.io/maven-machines-inc/android/apps/com.mavenmachines/issues/5c6af9dbf8b88c29634c7cec/sessions/5CBE11CA008800014C5BB70A3A4E0045_DNE_0_v2?
# Organization: Maven Machines, Inc.
# Platform: android
# Application: Maven Machines
# Version: 3.28.0-7 (177)
# Bundle Identifier: com.mavenmachines
# Issue ID: 5c6af9dbf8b88c29634c7cec
# Session ID: 5CBE11CA008800014C5BB70A3A4E0045_DNE_0_v2
# Date: 2019-04-22T19:11:00Z
# OS Version: 7.1.1
# Device: Galaxy Tab E 8.0
# RAM Free: 41.8%
# Disk Free: 81.2%

#0. Crashed: main
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(AsyncQueue.java:379)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(AsyncQueue.java)
       at android.os.Handler.handleCallback(Handler.java:751)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:154)
       at android.app.ActivityThread.main(ActivityThread.java:6823)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1557)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1445)

--

Fatal Exception: java.lang.RuntimeException: Internal error in Firestore (18.2.0).
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(AsyncQueue.java:379)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(AsyncQueue.java)
       at android.os.Handler.handleCallback(Handler.java:751)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:154)
       at android.app.ActivityThread.main(ActivityThread.java:6823)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1557)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1445)

Caused by java.lang.RuntimeException: java.lang.IllegalStateException: Couldn't read row 6987, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
       at com.google.firebase.firestore.util.AsyncQueue.enqueue(AsyncQueue.java:290)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$3.run(AsyncQueue.java)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
       at java.util.concurrent.FutureTask.run(FutureTask.java:237)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:272)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at com.google.firebase.firestore.util.AsyncQueue$DelayedStartFactory.run(AsyncQueue.java:205)
       at java.lang.Thread.run(Thread.java:762)

Caused by java.lang.IllegalStateException: Couldn't read row 6987, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
       at android.database.CursorWindow.nativeGetString(CursorWindow.java)
       at android.database.CursorWindow.getString(CursorWindow.java:451)
       at android.database.AbstractWindowedCursor.getString(AbstractWindowedCursor.java:51)
       at com.google.firebase.firestore.local.SQLiteSchema.lambda$ensureSequenceNumbers$7(SQLiteSchema.java:369)
       at com.google.firebase.firestore.local.SQLiteSchema$$Lambda$7.accept(SQLiteSchema.java)
       at com.google.firebase.firestore.local.SQLitePersistence$Query.forEach(SQLitePersistence.java:431)
       at com.google.firebase.firestore.local.SQLiteSchema.ensureSequenceNumbers(SQLiteSchema.java:366)
       at com.google.firebase.firestore.local.SQLiteSchema.runMigrations(SQLiteSchema.java:114)
       at com.google.firebase.firestore.local.SQLiteSchema.runMigrations(SQLiteSchema.java:65)
       at com.google.firebase.firestore.local.SQLitePersistence$OpenHelper.onUpgrade(SQLitePersistence.java:283)
       at android.database.sqlite.SQLiteOpenHelper.getDatabaseLocked(SQLiteOpenHelper.java:256)
       at android.database.sqlite.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:163)
       at com.google.firebase.firestore.local.SQLitePersistence.start(SQLitePersistence.java:120)
       at com.google.firebase.firestore.core.FirestoreClient.initialize(FirestoreClient.java:238)
       at com.google.firebase.firestore.core.FirestoreClient.lambda$new$2(FirestoreClient.java:116)
       at com.google.firebase.firestore.core.FirestoreClient$$Lambda$2.run(FirestoreClient.java)
       at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$4(AsyncQueue.java:311)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$4.call(AsyncQueue.java)
       at com.google.firebase.firestore.util.AsyncQueue.enqueue(AsyncQueue.java:287)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$3.run(AsyncQueue.java)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:428)
       at java.util.concurrent.FutureTask.run(FutureTask.java:237)
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:272)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
       at com.google.firebase.firestore.util.AsyncQueue$DelayedStartFactory.run(AsyncQueue.java:205)
       at java.lang.Thread.run(Thread.java:762)

#0. Crashed: main
       at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(AsyncQueue.java:379)
       at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(AsyncQueue.java)
       at android.os.Handler.handleCallback(Handler.java:751)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:154)
       at android.app.ActivityThread.main(ActivityThread.java:6823)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1557)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1445)

Relevant Code:

Upgrade from:

    implementation 'com.google.firebase:firebase-core:16.0.1'
    implementation 'com.google.firebase:firebase-messaging:17.1.0'
    implementation 'com.google.firebase:firebase-firestore:17.0.2'
    implementation 'com.google.firebase:firebase-auth:16.0.2'

to:

    implementation 'com.google.firebase:firebase-core:16.0.8'
    implementation 'com.google.firebase:firebase-messaging:17.5.0'
    implementation 'com.google.firebase:firebase-firestore:18.2.0'
    implementation 'com.google.firebase:firebase-auth:16.2.0'
@google-oss-bot
Copy link
Contributor

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

@schmidt-sebastian
Copy link
Contributor

@Highthunder Thanks for reporting! This does sound a little scary and definitely looks like a problem in our SDK. We will look at this internally and hope to have an update for you soon!

@schmidt-sebastian
Copy link
Contributor

schmidt-sebastian commented Apr 23, 2019 •
edited

We believe that you are hitting an issue that is similar to #208. In SQLite, a CursorWindow cannot hold more than 2 MB of data, and it looks like during our schema migration even just reading a single column can hit this limit. The offending code is here: 58c2075#diff-7e86ec47aa1c080b4b33aee14ddd7489R272

We will get this fixed shortly. In the meantime, for the clients that failed the schema upgrade, you can continue to use the older version of Firestore. Optionally, you can also turn off persistence: https://firebase.google.com/docs/reference/android/com/google/firebase/firestore/FirebaseFirestoreSettings.Builder.html#setPersistenceEnabled(boolean)

@StillLynnTheCloset
Copy link
Author

Just to clarify, is this an issue with a single row being too large, or is it because the SDK is attempting to migrate a large number of rows at once?
Because all of the crashes we've observed have occurred when reading a row id around 7000.

@schmidt-sebastian
Copy link
Contributor

It's the combined size of the result set. In your case, it looks fetching the document names of like ˜ 7000 documents exceeds the maximum Cursor Window.

@schmidt-sebastian
Copy link
Contributor

I have been able to reproduce this and have a fix that addresses the issue in a local test. I will open a PR shortly.

We unfortunately missed the release train for the next release, so I suspect that this won't go out until late May.

@schmidt-sebastian schmidt-sebastian mentioned this issue Apr 24, 2019
Merged
@StillLynnTheCloset
Copy link
Author

The main reason we attempted to update this dependency was because a new feature we are trying to release relies heavily on whereArrayContains queries which are not available on the version we had previously been using.

Would there be any issues if we update our app to use version 17.1.0 so we can use whereArrayContains while also avoiding this troublesome database migration?

@schmidt-sebastian
Copy link
Contributor

17.1.0 does not yet contain the offending schema migration and should avoid these issues for users that have not yet updated. Be aware though - running 17.1.0 on clients that have already upgraded their database schema should work, but we haven't tested this very thoroughly.

@StillLynnTheCloset StillLynnTheCloset mentioned this issue Aug 14, 2019
Closed
@firebase firebase locked and limited conversation to collaborators Oct 11, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@schmidt-sebastian schmidt-sebastian

Labels
api: firestore
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Firestore: Split schema migrations into batches firebase/firebase-android-sdk
4 participants
@StillLynnTheCloset @wilhuff @schmidt-sebastian @google-oss-bot