Imagine you want to check that recent email to remind yourself about the address you were to go to and your email client application refuses to give it to you because you have no access to the internet - you may curse or pay. Or you are going by train or airplane and you are writing this great new post for your blog and your blogging application refuses to save it and you are getting low on battery.
In each situation mentioned above, there was one source of problems - internet access was not available at the particular moment. These applications work only in online-mode - the state when stable and fast connection to the backend server is available for as long as the user wants to work with them.
But what if you really need that backend source of data and what if you want to send some data to it as well? The answer is: make it offline-first - design and write your application as if there was no internet connection at all.
Offline-first architecture in a nutshell
Offline-first architecture depends on the local storage as a primary source of data, changes are made to this storage. The application also can have networking components, but the basic flow does not know that it connects to a server. Communication with the backend server is just a side effect.
On the one hand, it simplifies the implementation of local business logic: you don’t need to cover temporary connection issues, displaying “loading data” indicators etc. On the other hand, you have to figure out a way of triggering communication with the backend, you have to implement business logic in the client application as well, and you have to work with different versions of client databases on the backend side.
In the following article I’ll try to give you some advice on how you can deal with triggering communication problem.
Read: What Is a Single Page Application? Explanation and Examples
Strategies for communication with the backend server
Pushing data in strategy
It is a strategy that may fit to infrastructure where the server is the main source of data (ex. news applications) and the client application is almost exclusively a data consumer. In this strategy, it is the server's responsibility to send a signal to the client that new data is available to download. The signal is usually implemented as a push notification to achieve high reliability of the message delivery. In response to the message, client refreshes its local database to reflect the current state of the remote database.
Databases synchronization strategy
Here, both sides are trying to track every change made to the data used and make data on both sides equal. It is really useful when an application should be usable in the offline mode and still offer a way of collaborating with other users (ex. email client, data/files sharing, text editors). Both sides of the connection have to track changes to the last shared state and, when internet connection is available, to send them both ways to obtain the new shared state.
Prefetching strategy
It means downloading data before it will be used. The application tries to predict what the user would like to use in the future and loads that data to the cache when fast network is available, so it may still work in the offline mode. This approach works best for applications that require stable and fast internet connection to be usable - like music/video players, image galleries, navigation apps.
Push-notification based data synchronization
Notifications as the only channel for data exchange
If you don’t need to send large quantities of data to the application from the backend, you can consider packing all necessary information inside a message that your backend pushes through the cloud. At the time of writing this article, both Firebase and Apple Push Notification Service (iOS 8+) allowed sending up to 4KB within one single message. That often may be enough to send all the important data to the client application (you may need to split data properly). In that case, you may never need the client application to initiate data downloading by itself. Still, you should consider that sometimes the cloud may be unable to deliver your messages and you would have to create some way of recovery from such failures. Also, if you need to send binary files, that method would not be applicable. Yet it is worth to use this method to send important data first, and let user decide whether he or she needs to download the remaining data. This solution may be enough for applications like messaging apps with limited size messages (like Twitter) or weather forecasts.
Notifications as requests for downloading data
If your client application typically uses data from the backend server, you may consider to notify the app that new data is waiting. Then, the application will check the current device state (battery, internet connection) and schedule the downloading task accordingly. You should consider sending some meta information in the message to give the client application some extra hints on data size, endpoints to be called, urgency etc. This strategy may be useful for email clients or podcast players and similar applications that consumes larger data chunks from time to time.
Databases synchronization
Using existing self-synchronizing databases
There are some existing solutions that synchronize the local database with the remote one: either NoSQL databases (ex. Firebase Realtime Database, Realm, Couchbase Lite) or SQL database (ex. Zumero). It is the best solution if you are starting a new project from scratch, because it may be really hard to migrate from one to another as they often require a database installed on the server. Each database may have some custom restrictions and limitations.
Self-made synchronization procedure
If auto-synchronizing databases cannot work for you, then you end up with designing your own synchronization procedure. That may sound challenging, but you are in control what and how data should be exchanged. In the design, you should consider the following:
- how to initiate the synchronization procedure - how often to perform it (periodic, event-based, fixed time), can it be delayed;
- how to avoid conflicts between local and remote keys - consider using UUIDs for both databases keys;
- what additional information should be stored in databases - when data was synchronized last time, if the data is flagged to be synchronized, changelog from the last synchronized state, any extra flag that would be useful to track if record in database was created, modified or deleted (you cannot usually remove records from database but you can flag them as removed);
- how to deal with conflicts - what should happen when a record was removed from one database, what should be done when a record was changed in both databases, how to deal with local time/locale settings etc. - this is the problem the additional data that were planned in previous step should help to solve.
Prefetching data
Loading data according to user’s behaviour
The application may track how the user interacts with it and perform an analysis to predict what data they may be interested in next. Use any AI solution that works best. The backend may use such algorithms to push data (see notification-triggered synchronization). This may work for any application in which the recent user actions allow to predict what data may be needed in the nearest future, ex. a navigation map can load a map of the area location searched for last, a video/music player can load next suggested file etc.
Loading large amount of data when connection is fast
The application should try to avoid loading huge files using cellular network, data should be transferred in as large parts as possible to conserve the battery. It is usually fine (unless disc space is limited) to load large files while connected to WiFi, so you should delay such transfers until WiFi is available or, even better, load any data you may predict the user would like to use before and cache it for the future use.
Caching data
Often, the user may go back to the recently visited data (ex. that important email). Consider using multiple levels of caching data from backend even if you don’t use any other synchronization strategy. Most recent data should be available right from the memory cache, but you may cache someadditional visited data using disk-based cache too.
Read: Web App Scaling: Here's How to Do It Effectively
Tools for implementing offline-first architecture on Android
Now as you know what to consider when you design your offline-first application, let’s move on to some tools available for the Android platform that you may find useful during implementation.
- JobScheduler - system service available for API level 21+ that allows you to synchronize tasks with description of expected device state including battery level, internet connection parameters, as well as retry/back-off strategy. You should consider using JobScheduler both for custom database synchronization and notification-triggered downloading, as it is the recommended way of delaying background processes. If you need this feature for API level below 21, then you can consider using Firebase JobDispatcher (API 14+, requires Google Play Services installed on the device), Android-Job from Evernote or Android Priority Job Queue.
- Sync Adapters - system API designed specifically for synchronization data managed by the content provider framework that uses Android accounts and authentication framework to perform. Because of that the implementation is much more complicated than using JobScheduler and requires some boilerplate to work. Because of that you should avoid using this mechanism unless you are synchronizing data that are provided by some content provider (ex. Contacts provider) and you have implemented your custom Authenticator already.
- ConnectivityManager - system service that may be used for monitoring the state of internet connection. Your repositories implementation may direct queries to the backend calls when connection is up and stable, and then switch to the disc cache and scheduling synchronization task when it is down.
- Data Saver - API introduced in API level 24 that allows the user to control application data transfer usage. Your application should considered this extension to ConnectivityManager and adjust transfer usage (ex. by requesting images in lower quality).
- Firebase Cloud Messaging - part of Google Play Services that is required for an application to receive notification from Google Cloud. You can search for other messaging services as well.
- NYTimes’ Store - library that can help you to manage downloading and caching data.
- SQLite-sync - solution that simplifies creation of sync-ready backend and works with multiple SQL databases. It provides web application translating database changes to XML format returned from multiple services. The XML returned from the service can be interpreted by the client application to perform requested operations in the local database. Client application can also send local changes to the backend in the same XML format and changes would be reflected in server’s database.
Still, there are many other libraries that can be used for implementing synchronization procedures. My advice is to plan ahead before starting implementation, because you may find yourself strongly bound to one or another solution. Also, it is a good idea to work closely with backend developers as synchronization usually would have to be done on both sides.
I hope that this article will inspire you to design your application in a different way: offline-first instead online-only. It would move your application to the next level of user experience and possibly - increase the amount of time a user spends with your app.
