Firebase Database: Out of Memory Exception on querying huge amount of data on Android app

Question

I'm using Firebase database for my Android app. I needed a unique 5-digit code which should be valid for 3 days. To resolve this I inserted all possible 5-digit codes in Firebase database where each entry initially has status as 1. So structure looks something like this:

codes: {
  'abcde': {status:1},
  'zhghs': {status:1}
  .....
}

This codes object has close to 5 million entries. I was thinking I would set status to 2 once a code is in use.

I'm using this to get a code from Firebase:

db.getReference("codes")
                .orderByChild("status")
                .limitToFirst(1)
                .addListenerForSingleValueEvent { ..... }

I just need 1 code which has status as 1. The above code is causing OutOfMemoryException so I think it is trying to load all the data on device. Shouldn't it just load one element on device side due to limitToFirst(1) constraint?

P.S. I have tried my hands on Firebase FireStore. It doesn't allow importing huge JSON data as of now without exceeding 20K daily limits.

Edit - I've even set db.setPersistenceEnabled(false) Exact error is :

Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are syncing to the client. java.lang.OutOfMemoryError: Failed to allocate a 28701016 byte allocation with 16777216 free bytes and 18MB until OOM

Please edit your question to limit your description to a single issue. Everything was very reasonable until you started listing a bunch of other questions at the end, which are not appropriate for Stack Overflow. — Doug Stevenson
– Doug Stevenson, Commented Jul 3, 2018 at 20:10
So, just to be certain, if you do nothing else in your app except for this query, you have the same problem? — Doug Stevenson
– Doug Stevenson, Commented Jul 4, 2018 at 4:08
@DougStevenson My app completely loads before I hit this query. Exact error is Firebase Database encountered an OutOfMemoryError. You may need to reduce the amount of data you are syncing to the client. java.lang.OutOfMemoryError: Failed to allocate a 28701016 byte allocation with 16777216 free bytes and 18MB until OOM — crysis
– crysis, Commented Jul 4, 2018 at 5:02
The issue isn't that your app completely loads. (Maybe it loads too much, who knows?) The issue is discovering that the query actually takes too much memory without the context of the rest of your app. — Doug Stevenson
– Doug Stevenson, Commented Jul 4, 2018 at 5:05

Frank van Puffelen · Accepted Answer · 2018-07-04 13:45:42Z

3

It sounds like you might not have an index defined on your status property. In that case, the server can't do the ordering/filtering for you, and it will be done in the client. That means the client has to read all data and keep it in memory, explaining the huge memory usage.

The solution is quite simple: define an index on your status property.

{
  "rules": {
    "codes": {
      ".indexOn": "status"
    }
  }
}

Also see the documentation on indexing data.

answered Jul 4, 2018 at 13:45

Frank van Puffelen

603k85 gold badges895 silver badges868 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

crysis Over a year ago

Worked like a charm. I actually had plan to add this rule but I thought I wouldn't need it in development since most of the time all status codes would be same. Thanks a lot.

Frank van Puffelen Over a year ago

With 5 million entries, you definitely need an index. Heck, with that number of entries, I'd still consider creating your own reverse index where you create the list of all items with status 1 from your own code, e.g. /itemsByStatus/1/itemId1: true. This scales better, since then you can order/limit the items, to get a subset: ref.child("itemsByStatus/1").limitToFirst(10).

Alex Mamo · Accepted Answer · 2018-07-04 18:30:52Z

1

I just need 1 code which has status as 1.

In this case change your query to:

db.getReference("codes")
            .orderByChild("status")
            .equalsTo(1)
            .limitToFirst(1)
            .addListenerForSingleValueEvent { ..... }

Even with this constraint you should reduce the amount of data you load into memory because it seems that you are forcing too much data into memory. Alternatively you can set a large heap size by setting android:largeHeap="true" in the Manifest.xml file. Usually this error disappear.

You can also take a look at the discussion in this closely related article here.

As @DougStevenson said, is not really a recommended way to solve OOM errors but as I saw in @FrankvanPuffelen answer, the key to solve the problem is to add ".indexOn": "status" which was initially missing.

edited Jul 4, 2018 at 18:30

answered Jul 4, 2018 at 8:02

Alex Mamo

139k18 gold badges171 silver badges202 bronze badges

2 Comments

crysis Over a year ago

Initially all codes have status = 1 so adding equalsTo(1) won't decrease the data set. Using android:largeHeap="true" is not the right option for me as I will shift to 6 digit code eventually where data would be much more and I don't want to load hundreds of MB data in RAM for one code.

Doug Stevenson Over a year ago

Use of largeHeap is not really a recommended way to solve OOM errors, as there is still a tremendous amount of memory variance between devices, and Android is by no means obliged to give you more memory than it chooses. The correct solution is to just use less memory. developer.android.com/guide/topics/manifest/…

Collectives™ on Stack Overflow

Firebase Database: Out of Memory Exception on querying huge amount of data on Android app

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related