Software development consists of many tedious and repetitive tasks. Continuous Integration, which automates the process of quality control, testing, and software delivery allows us to focus on the project itself, thus saving the resources needed for its development.
The question is “Which solution to choose?”. Implement your own infrastructure, or maybe make use of some ready-made solutions like PaaS?
I think there is no definite answer – a solution that is suitable for one may be overkill for another. In this article, I would like to take you on a journey through different approaches, which gave me a better overview, and finally led me to the most appropriate solution for me.
How it all started
Codete is a software house, where we deal with projects from different areas and with different requirements. However, when you look at this from a technical point of view, each project and its corresponding lifecycle is very similar. It consists of the same, repetitive development processes. Not only have we noticed that these processes can be automated, but also that some of them are very time-consuming, error-prone, and painful, especially for new team members.
We have decided to set ourselves a challenge and automate most of the development processes that were previously made by hand. So the journey began!
Build process automation
The key requirements we had were to decrease the overall workload regarding the generation of Apple certificates, private keys, and provisioning profiles. Our next aim was to avoid last-minute chaos at the release dates. We wanted to share responsibilities by allowing each developer to deploy an app.
Usually, private keys needed for app deployment were stored on the computer of one of the developers. When the person took time off and another developer needed to sign the app, he had to go through the baptism of fire. It was a pain and a waste of time. What we needed was one central place to share important data, but also keep it up-to-date at the same time.
Test automation
Our needs didn’t end up with building automation. There was also a place for improvement in the area of testing. What we noticed, is the fact that even if tests are written, developers simply forget to run them each time they integrate a new part of code. As a result, over time, some of the tests started to fail - we were generating regression. We figured out that we need to find a way to run tests more frequently and prevent the integration of untested code.
Code quality inspection
The last part we wanted to improve was the quality of code. We found that some of our projects contained many warnings and some parts of the codebase were duplicated. Another thing, which bothered us, was an inconsistency in the code formatting between projects or project-specific files. We decided that we need to introduce some kind of automated code analysis and solution that will enforce coding rules across projects.
The first approach
After deep analysis, our team got to work! We started analyzing available tools, and the following ones were on our list:
Jenkins
I had some previous experience with Jenkins, but my memories are not so good, because of very bad UX and the high entry barrier. I know there are a lot of fans of Jenkins since it’s highly customizable and expandable, however, our team didn’t want to go with a tool that is difficult to configure and hard to maintain.
PaaS
Another idea that we took into consideration was the use of a cloud service (Platform as a Service). The good part about this kind of solution is that the whole infrastructure is hosted externally, already set up, and ready to use. However, at the time we were considering it (May 2015), the list of additional plugins and testing frameworks offered by the available platforms seemed limited to us.
Secondly, we didn’t have too much experience in the area of Continuous Integration, so we wanted to leave ourselves more leeway and try out something more customizable.
Xcode Server
The last solution that we wanted to review was Xcode Server from Apple. After our extensive research, it looked the most promising. First of all, Xcode Server came from the same stable as iOS, so the platforms were naturally compatible. Secondly, developers were able to manage Xcode Server directly from their development environment - Xcode. Lastly, Xcode Server was just an application installed on OS X, so we were able to expand the solution and integrate it with any additional 3rd party tools. It seemed that the possibilities were almost endless.
Chosen solution
After examining the pros and cons of each solution, we decided to go with Xcode Server. The company bought a Mac mini and we started to set it up. The configuration process was divided into two phases:
- Setting up the system and Xcode Server.
- Extending Xcode Server with additional tools and scripts.
The first phase
We started with the basic configuration of the system and Xcode Server by setting up the network, users, groups and configuring Apple Developer Programs. Next, we started to install some useful tools like a custom shell, text editors, or Homebrew to easily manage additional packages. We spent some time updating system versions of Git, Java, and Ruby.
In the beginning, everything seemed to work fine, but it turned out that Xcode Server didn’t see newly installed versions of the tools. We started to dig through the documentation, but we didn’t find anything helpful. It was the first sign that there is a lack of learning materials from Apple and we need to rely more on the experience of other developers.
However, we finally found a solution and after a week we were able to run our first integration on Xcode Server.
The second phase
In the second phase, we wanted to extend Xcode Server with some handy tools and make life easier for future users by writing some useful scripts. First of all, we installed the whole family of fastlane.tools, which handles all tedious tasks, like dealing with a code signing or releasing an application. Our next goal was SonarQube, an open-source tool for continuous inspection of code quality. The whole configuration went pretty smoothly and we were ready to go ahead with the scripts!
We decided to start working on a real project, so we could verify the objectives that have been set at the beginning. We started by writing small utilities like incrementing a build number or integrating dependencies. It ended up with some more advanced ones for managing certificates, running SonarQube, sending .ipa to Crashlytics, or sending integration results directly to the company's Slack channel. Along the way, we tackled various issues like system permissions or keychain access.
However, after two weeks of work, we were finally ready to introduce the solution to the company and start to integrate new projects! It was the best reward.
This is not the end of the story
We configured two more projects and everything worked as expected. We were aware that we have spent a lot of time, but we felt that it was worth it, until…
…until iOS 9 came out along with Xcode 7 and the new version of Xcode Server. We started updating the technology stack, at the same time asking ourselves the following questions:
- What about backward compatibility after updating Xcode Server?
- Sadly it turned out that Xcode Server was not backward compatible. After the update, all of our projects stopped working. We needed to spend one week fixing the configuration and update our scripts.
- What if some of the projects require older versions of SDK?
- We couldn’t update all of our projects. In the software house, each project has different requirements and timelines. We needed to find a way to run both types of projects, the ones that are compatible with Xcode 6 and 7.
- Can we run multiple versions of Xcode Server on one machine?
- It turned out that it’s not possible. We would need to have multiple machines.
- So maybe we can run multiple virtual machines to handle it?
- It was possible, but we started to ask ourselves questions: what about performance? and how long will it take to configure the virtual machines?
After answering those questions, we decided to use two independent machines with different Xcode versions, however, we already knew that in the long run, Xcode Server is not the best solution for us.
Lesson learned
In the meantime, we were thinking about the possible approaches once again, but from a wider perspective. We started by drawing conclusions from past experience. So what we have learned?
Firstly, the maintenance of the Xcode Server took us too much time. It resembled me the following phrase: “Make apps not infrastructure”. A tool that was supposed to help us, became the main object of our interest.
Secondly, in a software house, each project has different requirements and a different technology stack that can change over time. A solution like Xcode Server seemed to be more suitable for companies that are focused on one project at a time or a group of projects with the same technology stack.
However, we don’t consider the time spent on Xcode Server wasted. Not only we gained some practical experience, but also we started to be active in the open-source community related to iOS Continuous Integration. It started during the project with the idea of managing the Xcode Server directly from the shell. We wanted to run a separate project to create such a tool.
However, before we proceeded, my teammate @cojoj started to make research and it turned out that this kind of tool already exists in the open-source community. The project was called XcodeServerSDK and it came from the stable of Buildasaur community. @cojoj said why not join them and work together. He began to collaborate, and after some time he also pulled me. At the end of the story, maybe we didn’t make use of the tool in our main project, but it was a very valuable experience and we met a lot of great people from the community.
The second approach
After some time I joined an existing project of one of our clients. I was assigned a task where I needed to prepare a test build. The client used their own Continuous Integration solution based on Jenkins. After I was given the credentials, preparing a build was a piece of cake. Sometime later my team was asked to add a separate workflow to run Calabash tests. In addition to this, we needed to migrate the project to Xcode 8.
Taught by experience with Xcode Server, I realized that, for sure, it won’t be easy with Jenkins. At the same time, I remembered, that some of the PaaS services deal with those kinds of changes without any problems. I communicated my concerns to the team and came out with the proposal of making research to find an alternative PaaS solution. When I received the green light, I started to look for the solution, keeping in mind the following requirements:
- expandable with additional tools
- interchangeable technology stacks
- low entry-barrier for new team members
- transparent and customizable workflows
- easy integration with Github and Bitbucket
I noticed that in the meantime some of the previously considered PaaS services have been closed, e.g. https://ship.io/, so I started to look for the new ones and decided to review the two most promising: https://www.buddybuild.com/ and https://www.bitrise.io/.
Buddybuild
Buddybuild ties together Continuous Integration and Iterative Feedback Solution into a single, seamless platform. It has easy integration with Github and Bitbucket as well as interchangeable technology stacks. In addition to this, it has built-in crash reporting and the possibility to report an issue by taking a screenshot right from the tested application. At that time, the solution looked very promising, however, it missed some important factors for us. Firstly, the ability to integrate with 3rd party tools was pretty limited. Secondly, I didn’t find a way to define custom workflows to accommodate our specific requirements.
Bitrise
Bitrise is a Continuous Integration platform with the main focus on mobile app development. Similar to Buddybuild, it has easy integration with Github and Bitbucket and interchangeable technology stacks. The difference is, that it supports neither crash nor issue reporting, however, it supports many additional tools and has a great editor for creating and customizing your workflows.
Chosen solution
After examining the pros and cons of each service, we decided to go with Bitrise, since it met all of our expectations. Configuration was very easy and it took me just a couple of hours. I can’t say that during setup I didn’t encounter any problems, however, I was able to fix them very quickly, since Bitrise has a great knowledge base and even better support. After two weeks of using Bitrise, our team knew that it was a bull's eye. Good fame spread so quickly that within the next month, my client decided to run three other projects with Bitrise.
Conclusion
At the end of the story, Xcode Server died a natural death and we decided to close this chapter. Now, most of our new iOS projects make use of the Bitrise platform, so we don’t need to worry about the maintenance and we can focus on the projects. However, I try not to stick to this particular solution. I’d rather try to stick to the following slogan:
“Make apps not infrastructure”
I have also learned that every experience matters and sometimes it’s good to reconsider our decisions, even if we already put a lot of effort.
Thanks for reading this article. I hope it gave you a bigger picture of the available Continuous Integration solutions for iOS.
