Magento 2: One Commit to test them all…

Magento 2: One Commit to test them all…

… or extensive Magento 2 Testing in Extension Development

Magento 2 and our problem with it

Being a plugin developer in the Magento 2 world will face you with the problem that there are at this very moment 21 versions of Magento 2. Going from 2.0.0 – 2.0.13 and 2.1.0 – 2.1.6, everything needs to be tested to ensure that your module will work and a customer won’t have any issues when working with it.

We ran into trouble as we were working with shipment creation. The Module needs to notificate our Payment System in case of secured Invoice payments (this process is called ‘Finalize’). We’ve created a beforeExecute method to do this process even before the origin method from the Magento 2 core will be executed.

Since a lot of processes are the same, we were copying this origin Save Controller and have done our adjustments. Consequently we have also required the ShipmentValidatorInterface, which was introduced in Magento 2.1.2 and 2.0.9. This Interface was causing problems on one of our customer’s shop.

To avoid those problems, we are testing against all current versions of Magento 2, which have been 2.1.5 and 2.0.13 at that time – assuming that new features are only introduced in 2.1.x and 2.0.x only getting fixes and some patches. We were proved to be wrong.

So… how can we do integration tests with all Magento 2 versions?

Manual Testing is not always an option

Of course we do manual tests (‘Black Box Tests’) in our QA. This is necessary because automated testing in the typical CI/CD process won’t show you the Frontend and how the UX (user experience) will be in the final product. But this is a quite important thing in the eCommerce market when it comes to checkout and conversions.

It would take to much time for a small team like us to test all different Magento 2 versions during the QA.

So… How to deal with this problem?

Recently we had the opportunity to attend at the Firegento Hackathon in Leipzig and work on this problem. We have introduced our topic to the community and received a good amount of votes. So the interest in the community was there – we are not alone. 🙂

We also had a talk with Michael Mafnas (@mikemafnas_mg), Director of Operations and Leader of the Testing Team at Magento. He told us that
they are also working in-house on a solution called ‘DevBox’, which is currently in Beta. DevBox is a solution that generates you a script for a Magento installation on a Docker container. In the future we will take a look into Docker and DevBox, but for now it isn’t an option for us due to how our payment system works (Request-Response to local systems can be tricky) and we have little time for looking into new things.

By the way it was a pleasure to meet him. He said that it was the first time he was attending such an event. So if you have the chance to meet him, take the chance and do it.

Taking the bull by the horns

Our first goal was to roll out every Magento 2 version on every supported PHP version.

We choose TravisCI for continuous integration, because we are familiar with it. At the
first day we were able to install Magento, enable our extension and run setup:di:compile.

Our .travis.yml file looked like this:

- php: 5.6
env: magento=2.0.1
- php: 5.6
env: magento=2.1.1
- php: 7.0
env: magento=2.1.6
- $HOME/.composer/cache
- $HOME/.php-cs-fixer
- $HOME/.local
- mkdir -p "$HOME/.php-cs-fixer"
- cd ..
- composer self-update
- composer create-project "magento/community-edition:$magento" magento-ce
- cd "magento-ce"
- mysql -uroot -e 'CREATE DATABASE magento2;'
- php bin/magento setup:install -q --admin-user="admin" --admin-password="123123q" --admin-email="" --admin-firstname="John" --admin-lastname="Doe"
- composer require "heidelpay/magento2:dev-$TRAVIS_BRANCH"
- php -f bin/magento module:enable Heidelpay_Gateway
- php -f bin/magento setup:upgrade
- php -f bin/magento cache:flush
- php -f bin/magento setup:di:compile
- php -f bin/magento dev:tests:run
- cd ../magento2
- composer update
- ./vendor/bin/phpcs --config-set installed_paths vendor/magento/marketplace-eqp;
- ./vendor/bin/phpcs . --ignore=vendor/ --standard=MEQP2;

This was more than we have expected in first place. We even were able to run the static code analysis with the MEQP2 standard. This should make sure that we do not push code that is breaking this coding standard.

There are still some issues we’re running into. For example early versions of 2.0 do not
return an exit code if module:enable fails.

We also used the time to speak to other contributors of the Hackathon about their solution to test single Magento installations.

Andreas Mautz (President of Firegento, @mautz_et_tong) showed us some examples he built with GitLab CI and some guys told us that they are using Docker containers for it. Of course there is much room for improvement, but we think that this is a good starting point when doing Open Source development.

Inspired by the things Michael told us, we decided to use day 2 to run all core
unit and integration tests after the installation of the extension. Doing this, we make sure that we are not breaking core functions.

So we were taking a look on how Magento does it and luckily they use TravisCI as well. We started to integrate the setup for the core tests in our .travis.yml but with more commands in the file, it gets unreadable after time.

After a bit of re-organisation and using shell scripts instead of the script part of the Travis YAML file (like Magento does), we reached the goal that we are able to run the core tests as well. Of course there are lots of tests that currently fail and some possible improvements, but in our opinion, this is a very good starting point.

Every Hackathon comes to an end

As you might know, at the end of such a Hackathon each team presents it’s results.

There have been so many good things, that we can’t cover them all here. If there are some Blogposts about it, we will link them here.

Some guys started to contribute to the Magento 2 Multi-Node-Inventory – a feature that was meant to be Enterprise only, but now it will be available for Community Edition when its done. One team was going to improve the Invoice creation and another was working on translations to German. We are proud to be part of this community, which contributes every possible single line of code.

As we presented our automated extension test, about 5 of the 40 jobs were still running, but the solution was in progress.

During our presentation, Ben Marks intervened: “We have a solution for it and we should make it public”.

Nice to hear that Magento realises that there is a problem and that they are working hard to solve it. He even confirmed on twitter that he forwarded it to their Marketplace team.

Ben Marks' Tweet
Ben Marks’ Tweet regarding our presentation

At the end, our Travis build looked like this:

TravisCI build
Our TravisCI build after all jobs were finished

Running time: 1 hr 41 min 17 sec with a Total Time of 7 hrs 35 min 22 sec.

The current version is on our github continuous_integration branch. Feel free to contribute, but be aware of our current licence. 🙂

Thanks to Firegento for having us!

It was such a nice place and a pleasure to meet so many interesting and kind people.
We are proud that we can give a bit back with our sponsoring.


Stephano and Jens