Regulated industries cannot tolerate failed software releases. For a software vendor in a regulated industry, the challenge is to deliver software with new functionality, yet not impact the stability of the existing system. It’s the problem of “changing the wheels on a moving bus”. New functionality must be delivered without error, and new releases cannot impact the stability or functionality of the existing system in whole.
Enhancements may can be part of the core product, or custom developed for a single client. Changes frequently must be migrated to other client builds, where they can differ significantly. The issue has many angles:
A software firm that specializes in retail brokerage compliance was experiencing rapid growth as the compliance market matured. Wholly responsible for delivering and managing the software stack used by its regulated clients, the firm was responsible for not only accurate and secure software releases, but for ensuring that many risk controls remained in place with every release.
Through a review of all of the required controls, and an understanding of the process of applying fixes for results application vulnerability scans, several things were done.
First, logically separate environments were created for development, systems integration testing, user acceptance testing, and production, as follow. This ensured that different activities could be carried out on distinct code bases. This prevented admixture of different code bases and prevented developers from altering code that was already released to test. It also gave the testers somewhere to work with release candidate without having to make exceptions for ongoing development. It further made it possible to port fixes from one environment to the next in the sequence in a controlled fashion.
The deployment of these separate environments was accomplished using virtualized servers. This allowed the firm to quickly rollout multiple instances of each level of the SDLC, as needed, to facilitate needed upgrades to the software stack (such as operating system and database engine), and to keep costs down while enjoying this flexibility. Moreover, it allowed for the scrubbing of environments if they were damaged by a release – virtualization allowed us to effect a rollback in minutes.
All releases were tracked by tickets in the service desk software. These tickets were configured with common features such as:
The tickets became the foundation for all release management activities. When a developer needed to create a code branch for a future release, she first created a ticket. From that point on, the release was tracked by its ticket number—the branch number was even labelled with the ticket number inside the version control system. From that point on, all details about the release were recorded in one place, the ticket. This included any documents that had to be attached to the ticket, such as
The SQA Manager was gatekeeper to the systems integration testing (SIT) environment. They ensured that the developers had obtained or prepared:
The Production Manager was the gatekeeper for all production release candidates. This was not a technical role, but rather one of oversight:
The package release team then delivered all new software, schema changes, and configuration changes. They:
A relatively simple set of rules allowed the firm to enjoy considerable success in software delivery. They delivered value-added functionality in a reliable fashion (1 error in 557 releases after the first review). The dependable processes so established also freed the time of PortfolioAid SME’s and senior management from participating in what had become "business as usual". And finally, the auditors were happy. The clients typically asked for the auditors' reports annually, and those reports reflected the software company's ongoing ability to adhere to this release process.
In building this governed software release strategy, I: