By ABC Online Courses, Open edX® Partner
The community publishes new versions of the Open edX® software twice per year in June and December. These new releases, or named releases as they are called, need to be stable before being shared to the public. In order to ensure this stability, the community has a team of volunteer testers that perform the required quality assurance and testing. In this blog post, we will briefly introduce the team, explain what we do and how we do it, announce the latest testing champions, and lastly provide information about how to join the test team for the upcoming named release Palm.
Without the test team, the effort to test and fix bugs in the platform would shift to the individual organizations that are using the Open edX® platform. The testing is both time consuming and resource heavy, which is especially tough for individuals and smaller organizations to cope with. As a result of having a community test team, individual organizations can feel confident to upgrade their Open edX® platforms and take full advantage of the latest features and bug fixes without worrying about all the heavy lifting that goes into testing.
The test team currently comprises twenty one community members from fifteen individual organizations. With such a diverse group of organizations present in the team, we are able to cover many different areas of specialization – it’s no wonder that testing goes so smoothly! The list of current test members is as follows: Régis Behmo (Overhang.io/Tutor), Peter Pinch (MIT), Pierre Mailhot (EDUlib), Adolfo Brandes (Axim Collaborative), Maria Grimaldi (eduNext), Matjaz Gregoric (OpenCraft), Fayyaz Ahmed (Edly), Abderraouf Mehdi Bouhali (Fennec Technologies), Max Sokolski (RaccoonGang), Ghassan Maslamani (Zaat.dev), Michael Wheeler (University of Michigan), Esteban Etcheverry (AulasNeo), Kyle McCormick (Axim Collaborative), Ned Bachelder (2U), Kaleb Abebe (MIT), Carlos Muniz (Axim Collaborative), Zia Fazal (Edly), Gabriel D’Amours (OpenCraft), JayRam Nai (DRC Systems), Jhony Avella (eduNext), and myself Dean Jay Mathew (ABC Online Courses).
To begin testing, we first need a live instance of the latest version of the Open edX® platform up and running. Luckily, we have a sponsor for this, namely Régis Behmo from Overhang. He is the original creator of the Tutor project, which is the default installation method for Open edX® instances. Tutor is composed of multiple repositories hosted in the Overhang GitHub organization. Each repository is maintained by a single volunteer developer. It’s a major part of the community ecosystem which also gets tested and the talented Tutor maintainers include Ghassan Maslamani, Kyle McCormick, Adolfo Brandes, Jhony Avella, and of course Régis who oversees the entire project.
In terms of how much testing we perform for each new Open edX® named release, we have a growing list of 224 test cases that must be completed in order to approve the named release. Some examples of test cases include, but are not limited to, testing that all buttons and links in the LMS are working and redirecting to the correct pages, testing the mobile responsiveness of the platform, testing each individual course authoring tool that is used in Studio, and much more. There are also more complicated tests performed, such as confirming that Single Sign On (SSO) is working and that Instructor Emails are being sent out. It really does take a huge effort and talented web developers to test the entire system.
The test cases live in a protected Google Sheet that we all work on asynchronously, maximizing collaboration and review. If any of the tests fail, we mark it as “failed” in the sheet and then create a GitHub issue for the technical testers to triage by priority and ultimately resolve. Once the test case is ultimately resolved, we mark it as “passed” in the sheet. This end-to-end process assures us that the new version is stable and that the upgrade process itself will go more smoothly for the thousands of organizations that are using the Open edX® software, even those that do not necessarily contribute back to the community.
The overall life cycle of testing comprises four stages and is improved upon in each new release. First, the management team ensures that the test cases are updated to make sense with the latest named release. In other words, if there are new features, those must be added as new test cases. Second, we perform all 224 test cases. Third, as mentioned earlier we create and manage GitHub issues for failed tests and get them resolved in time. Fourth, we release the tested named release to the public.
There are countless other tasks performed by the management team who represent the community’s Build-Test-Release Working Group. Some of those tasks are cutting the named release and providing release notes. Then, a few months down the line, we also release a second, even more stable version, based on feedback from the community. The management team consists of Adolfo Brandes, Régis Behmo, Pierre Mailhot, Peter Pinch, Matjaz Gregoric, and myself. We also get valuable support from other senior members of the community, such as Sarina Canelake, Ned Bachelder, Eden Huthmacher, and more.
Now that you know what goes into testing the platform, let’s look at the test champions for the latest release, Olive. The following people were the most active testers who took the lead in terms of test cases, overall effort, and complicated technical testing and bug resolution. Congratulations to these people and organizations for championing the testing!
Test Champion: Fayyaz Ahmed, Quality Assurance Engineer at Edly
Best Newcomer: Abderraouf Mehdi Bouhali, Full-Stack Developer at Fennec Technologies
Most Innovative Testers: Ghassan Maslamani, Founder at Zaat.dev, and also Maria Grimaldi, Software Engineer at eduNEXT
It should be noted that Pierre Mailhot from the management team did 15% of the manual testing right at the start to get the ball rolling. Thanks, Pierre!
There are many use cases for the Open edX® platform, and with most use cases, the integrity of the platform is vital to the proper functioning and success of the organization. If the test team were not performing these tests, it would be much harder for people to get the education and training they deserve. This is why the test team is important: it pulls together people from all around the world for a common goal, to make online education accessible to many through the use of the Open edX® platform.
Testing for the next named release, Palm, will start in early April. If you want to join the test team, please join the Open edX® community’s Slack and mention your interest in the #wg-build-test-release channel. I’ll see you there!
