PayPal is an online payment system that supports online money transfers and allows users to securely and conveniently make financial transactions. With more than 190 million active accounts in over 200 markets, it is critical for any individual and business to send, receive and hold funds in 25 currencies worldwide.

As the testing need became demanding and had evolved into a complex task, Tom Whipple, Mobile Engineer and Lead Android Developer, needed a testing solution for his team to ensure each release works across a wide variety of real devices around the globe, covering different currencies and languages.

Emerging challenges in managing and testing a large variety of real devices

In May 2013, PayPal released the PayPal Android SDK in the USA that would be widely published to multiple markets later in the year. The SDK was designed to provide a native Android experience that simplifies the process of accepting credit card and PayPal payments in developers’ mobile app. The library presents a simple user interface, features card.io (owned by PayPal) credit card scanning and uses a proof of payment system that eliminates Payment Card Industry (PCI) compliance worries.

“An SDK handling money needs to work all the time and people need to have confidence in it. It is therefore important for the developer team to test on a wide variety of devices before shipping the SDK to developers all over the world,” cited Tom Whipple, “Android has many different screen sizes and there are numerous versions available. The majority of Android phones now run Android 4.0 or higher whereas only six months ago this wasn’t the case. Managing all the different device combinations is extremely difficult.”

Moving from Card.io to PayPal with previous successful experience

Before joining PayPal, Tom Whipple and his developer team were responsible for testing the low-level camera access of card.io (acquired by PayPal). “Before using Bitbar Testing, we quite often ran into the situation that we had little information on what exactly happened when we received crashes. Lots of valuable time was spent on exchanging emails with users, taking up to a month to fix each issue.” said Tom Whipple. “Then we got to know Bitbar Testing and went through an evaluation. Once we realized the full potential of Bitbar Testing, this timeframe was shaved down to just a few hours.”

With previous successful experience, Tom Whipple and his team, after joining PayPal, started testing its developer-facing PayPal SDK using Bitbar Testing. PayPal’s developers began with a basis run of testing that involved a combination of manual tests and tests that are executed on local continuous integration servers. “Manual testing is an important part of our development process. However, scaling these tests is rather difficult as they are very time-consuming,” added Tom Whipple. “Then we integrated Bitbar Testing to our continuous integration system by using Bitbar’s Jenkins plugin”, said Tom. “This enabled us to run tests on different devices on a continuous basis using a ramped-up, layered approach that complemented our own manual and local testing.”

"Bitbar Testing enables our development team to avoid relying purely on manual testing, which is next to impossible without a large team. In the long run, automating tests with Bitbar Testing allows us to release a more stable and reliable product for our customers, resulting in fewer bug fix updates and on overall better impression about our company."

Tom Whipple, Mobile Engineer and Lead Android Developer at PayPal

Continuous automated testing enables more stable and reliable product releases

Tom’s team followed this initial manual testing with their own set of tests that are executed on local continuous integration servers. “Whenever we verify tests locally, we take them to Bitbar Testing and automate them on over 200+ devices on an as-needed basis. This could sometimes happen several times a day,” emphasized Tom. “With every change to the code, automated tests enable us to realize continuous feedback. When test runs are finished, a report that contains detected issues during testing is generated and delivered to the development team for instant fix.”

“We have a drawer full of phones, but we also have the mindset of automating as many tests as possible,” stated Tom. “Bitbar Testing has enabled our development team to avoid relying purely on manual testing, which is next to impossible without a large team. In the long run, automating tests on Bitbar Testing allows us to release a more stable and reliable product for our customers, resulting in fewer bug fix updates and on overall better impression about our company.”