Imagine you want to check that recent email to remind yourself about the address you were to go and your email client application refuses to give it to you because you have no access to the internet – you may curse or pay. Or you are going by train or airplane and you are writing this great new post for your blog and your blogging application refuses to save it and you are getting low on battery.
In each situation mentioned before there was one source of problems – internet access was not available at the particular moment. These applications work only in online-mode – the state when stable and fast connection to the backend server is available for as long as user wants to work with them.
But what if you really need that backend source of data and what if you want to send some data to it as well? The answer is: make it offline-first – design and write your application as if there was no internet connection at all.
Offline-first in a nutshell
Offline-first architecture depends on the local storage as a primary source of data, changes are made to this storage. Application also can have networking components but basic flow does not know about that application connects to a server. Communication with backend server is just a side effect.
On the one hand it simplifies the implementation of local business logic: you don’t need to cover temporary connection issues, displaying “loading data” indicators etc. On the other hand you have to figure out way of triggering communication with backend, you have to implement business logic in client application as well and you have to work with different versions of client databases on the backend side.
In the following article I’ll try to give you some advice on how you can deal with triggering communication problem.
Strategies for communication with the backend server
Pushing data in – it is strategy that may fit to infrastructure where the server is the main source of data (ex. news applications) and the client application is almost exclusively a data consumer. In this strategy it is the server responsibility to send signal to the client that new data is available to download. The signal is usually implemented as a push notification to achieve high reliability of the message delivery. In response to the message client refreshes its local database to reflect current state of remote database.
- Databases synchronization – both sides are trying to track every change made to the data used and constantly try to make data on both sides equal. It is really useful when application should be usable in offline mode and still should offer a way of collaborating with other users (ex. email client, data/files sharing, text editors). Both sides of connection have to track changes to last shared state and, when internet connection is available, to send them both ways to obtain new shared state.
- Prefetching – downloading data before it will be used. Application tries to predict what user would like to use in the future and loads that data to the cache when fast network is available, so application may still work in offline mode. This approach works best for applications that would require stable and fast internet connection to be usable like music/video players, image galleries, navigation apps.
Push-notification based data synchronization
- Notifications as only channel for data exchange – if you don’t need to send large quantities of data to the application from the backend you can consider packing all necessary information inside message that your backend pushes through the cloud. At the time of writing this article both Firebase and Apple Push Notification Service (iOS 8+) allowed sending up to 4KB within single message. That often may be enough to send all important data to the client application (you may need to split data properly). In that case you may never need client application to initiate data downloading by itself. Still you should consider that sometimes cloud may be unable to deliver your messages and you would have to create some way of recovery from such failures. Also if you need to send binary files that method would not be applicable. Still it is worth to use this method to send important data first and let user decide whether he needs to download the remaining data. This solution may be enough for applications like messaging application with limited size messages (like Twitter) or weather forecast.
- Notifications as request for downloading data – if your client application usually uses data from backend server you may consider to notify application that new data is waiting. Then application may decide if the current device state (battery, internet connection) to schedule downloading task accordingly. You should consider sending some meta information in the message to give client application extra hints on data size, endpoints to be called, urgency etc. This strategy may be useful for email clients or podcast players and similar applications that consumes larger data chunks from time to time.
- Using existing self-synchronizing databases – there are some of existing solutions that synchronize local database with the remote one: either NoSQL databases (ex. Firebase Realtime Database, Realm, Couchbase Lite) or SQL database (ex. Zumero). It is the best solution if you are starting a new project from scratch because it may be really hard to migrate from one to the other because often they require concrete database installed on the server. Each database may have some custom restrictions and limitations.
- Self-made synchronization procedure – if auto-synchronizing databases cannot work for you then you end with designing your own synchronization procedure. That may sound challenging but you are in control what and how data should be exchanged. In the design you should consider the following:
- how to initiate synchronization procedure – how often to perform it (periodic, event-based, fixed time), can it be delayed;
- how to avoid conflicts between local and remote keys – consider using UUIDs for both databases keys;
- what additional information should be stored in databases – when data was synchronized last time, if the data is flagged to be synchronized, changelog from the last synchronized state, any extra flag that would be useful to track if record in database was created, modified or deleted (you cannot usually remove records from database but you flag them as removed);
- how to deal with conflicts – what should happen when a record was removed from one database, what should be done when a record was changed in both databases; how to deal with local time/locale settings etc.; this is the problem the additional data that were planned in previous step should help to solve.
- Loading data according to user’s behaviour – application may track how user interacts with it and perform some kind of analysis to predict what data user may be interested in next. Use any AI solution that may work best. Backend may use such algorithms to push data (see notification-triggered synchronization). This may work for any application that recent user actions allow to predict what data may be needed in the nearest future, ex. navigation map should consider to load a map of the area location searched for last, video/music player can load next suggested file.
- Loading large amount of data when connection is fast – application should try to avoid loading huge files using cellular network, data should be transferred in as large parts as possible to conserve battery. It is usually fine (unless disk space is limited) to load large files while connected to WiFi so you should delay such transfers until WiFi would be available or, better, load any data you may predict user would like to use before and cache it for the future use.
Often user may go back to the recently visited data (ex. that important email). Consider use multiple level of caching data from backend even if you don’t use any other synchronization strategy. Most recent data should be available right from memory cache but you may cache some extra portion of visited data using disk-based cache too.
How to deal with all of that?
Now as you know what to consider when you design your offline-first application let’s move to some tools available already for Android platform that you may find useful during implementation.
- JobScheduler – system service available for API level 21+ that allows you to synchronize tasks with description of expected device state including battery level, internet connection parameters as well as retry/back-off strategy. You should consider using JobScheduler both for custom database synchronization and notification-triggered downloading as it is recommended way of delaying background processes. If you need this feature for API level below 21 then you can consider using Firebase JobDispatcher (API 14+, requires Google Play Services installed on the device), Android-Job from Evernote or Android Priority Job Queue.
- Sync Adapters – system API designed specifically for synchronization data managed by content provider framework and uses Android accounts and authentication framework to perform. Because of that the implementation is much more complicated than using JobScheduler and requires some boilerplate to work. Because of that you should avoid using this mechanism unless you are synchronizing data that are provided by some content provider (ex. Contacts provider) and you have implemented your custom Authenticator already.
- ConnectivityManager – system service that may be used for monitoring of the state of internet connection. Your repositories implementation may direct queries to the backend calls when connection is up and stable and then switch to the disk cache and scheduling synchronization task when it is down.
- Data Saver – API introduced in API level 24 that allows user to control application data transfer usage. Your application should considered this extension to ConnectivityManager and adjust transfer usage (ex. by requesting images in lower quality).
- Firebase Cloud Messaging – part of Google Play Services that is required for application to receive notification from Google Cloud. You can search for other messaging services as well.
- NYTimes’ Store – library that would help you to manage downloading and caching data.
- SQLite-sync – solution that simplifies creation of sync ready backend and works with multiple SQL databases. It provides web application translating database changes to XML format returned from multiple services. The XML returned from the service can be interpreted by the client application to perform requested operations in local database. Client application can also send local changes to the backend in the same XML format and changes would be reflected in server’s database.
Still there are many other libraries that can be useful for implementing synchronization procedures. My advice is to plan ahead before starting implementation because you may find yourself strongly bound to one or another solution. Also it is a good idea to work closely with backend developers because synchronization usually would have to be done on both sides.
Hope this article will inspire you to design your application in a different way: offline-first instead online-only. It would move your application to the next level of user’s experience and can increase the amount of time user spends with your application.