Firebase and Google Analytics might only be counting HALF of your Active Users

After 10 years of using Firebase or Google Analytics, I’m migrating off because of the incredible discrepancies I’ve found and their lack of support (even after acknowledging the bugs).

I’ve put enough time and energy into this issue already so I’ll be short and sweet with this article and hope that it helps some others.

How does Firebase calculate Active Users (according to them)

An active user is captured whenever user_engagement event is logged. The active users in the Dashboard are calculated based on the unique instance of the app pulled from the instance ID in a given time period. For instance, if a user installs the app, uninstalls it and then re-installs again, the active user count would be two. The user_engagement event is periodically (e.g. hourly) logged when a user sent your app into the background, switched screens, experienced a crash, or used your app for an hour.

There’s no minimum duration of app interaction to log a single user_engagement event. You would expect to see several logs of this event alongside other events like screen_view, first_open, and app_exception. An engagement_time_msec parameter is automatically added by the Firebase SDK that tells you how long a user has been using your app. You can check this blog post by one of our engineers for more information on how Firebase calculates sessions and user engagements today.

This quote was sent to me multiple times by Google Support.

The Problem

40–60% of my users (depending on the day) are not logging user_engagement events, even though they open my app, browse around, and even tap on things.

How do I know? Let’s look at the same data in 3 different places in Google’s systems.

Firebase Dashboard

Here we’re going to check the Active User count in the Firebase dashboard for a 30 day period. The same number (174k) is reported in the Google Analytics dashboard.

Firebase Again

Now on the Firebase dashboard again, let’s check how many users opened the app in that same period.

Maybe they didn’t really interact with it? Maybe they just opened and closed it. Even though Firebase says:

There’s no minimum duration of app interaction to log a single user_engagement event.

BigQuery (data comes from Firebase)

Our Firebase data gets routed to BigQuery, and looking at the same data let’s see what’s going on. How many users are getting this user_engagement event, compared to how many users are opening the app and tapping something while it’s in the foreground, surely that would get counted as an active user right?

Note that to simplify this, I looked at a single day during that 30 day period, but the results are roughly the same for each day and show nearly the same error as we see when looking at users that opened the app vs users counted as active. It’s clear that the issue is caused by user_engagement events not firing when they’re supposed to.

Response from Support

Upon discussing this with the team, the issue of discrepancy, as per your evidence, might be a possible bug

Who cares?

It’s super important this number is correct because this number is what drives TONS of other awesome metrics that Google calculates for you such as retention cohorts, funnels, version adoption, etc. And because it’s incorrect, we have to manually setup our own dashboards based on the raw data to avoid Firebase and Google Analytics awful dashboard bugs, making the whole system useless.