Forward Progress with Shadow Releases
A Shadow Release is the ability to build functionality and deploy it without users knowing about it. There are several strategies that can be used to implement shadow releases, but the most common is feature flagging. This can be at the platform level, or more beneficially, on a segment of your user base (be it a single customer or a specific user).
I have found shadow releases to be incredibly helpful in experimenting with different ideas without affecting current users. I can release functionality and enable it for my own user account so that I can "give it a spin" and demonstrate it to stakeholders before rolling it out to the rest of the users on the platform.
The main driver behind shadow releases is to be a kind of "risk mitigator", so that we can still make forward progress without having negative impacts on our users. With that goal in mind, it also ensures that these kinds of features are built in a way that they can be easily discarded if they do not suit the needs of the business.
The Challenge
To demonstrate the idea of shadow releases, I decided to take a real-world scenario from a recent project I completed. This piece of work had a number of challenges where we wanted to build some enhanced functionality around decisions that were made during the inception of the platform, so the story will hopefully resonate with people who have had to do something similar. And besides, if this particular piece of work were easy, then there would be no point writing about it...
Perhaps a backstory is in order before discussing the proposed enhancement.
Our platform is primarily used as a driver to encourage internal customer referrals for jobs, so that the referrer can be rewarded with points (which can be redeemed for cash). So our platform allows customers to create job postings that their employees can share with their personal networks. When a potential candidate applies for the job, they enter a candidate progression workflow, where they start at the "applied" stage. Customers can configure this candidate progression by allowing a different number of points to be awarded to the referrer when the candidate reaches different stages of the process (which makes it sort of like an ATS in a way). As the candidate progresses through the process, the referrer earns more points until the person they referred is offered a job and starts working for their employer. During the process, the candidate can also end up with different "stage statuses", where they may have progressed to the "Interview" stage, but were unsuccessful. In addition, there is some additional functionality around candidate progression, which involves sending notifications to the referrer, which can be customised for some of the stages.
Now that you know about the current functionality, it is time to talk about the requested feature.
Currently, the candidate progression feature has stages and statuses that cannot be changed. Because of the market the business was in at the current time, it wasn't much of an issue. There were requests for this customisability, but due to the effort required and the limited number of customers requesting it, it was deferred until a later date. However, as we started to diversify our customer base into other areas (still related to health and social care), we found that some customers have a strong need to have the platform show the current processes that they use, as the current ones were not applicable for them. The pivot into a new sector was the driving force behind prioritising this feature.
So the key goals of this new feature were:
- Enabling customisable stage names and statuses
- Enabling stage statuses to be added/removed
- Enabling custom notifications for EVERY stage and status (only positive and negative outcomes), but that can be selectively enabled/disabled
Sounds simple, right? However, there is a bunch of legacy code that needs to be reviewed first before hopping in with the implementation. And it wasn't just the code...it was also looking at any "bugs" that were raised around that area because this feature can also be used to ensure that the new functionality is a bit more robust.
However, when reviewing the existing code, it appeared that the functionality for candidate progression was scattered throughout the code. Some with inconsistent behaviour, e.g., it might award points for one use journey, but not another. It might send a notification for one scenario, but not another (even if it was the same outcome). So I felt that this was a good argument for getting the current candidate progression flow into a more coherent state before trying to add an enhancement to it - channelling my inner Kent Beck by "making the change easy" for myself in the future.
In addition, our ATS integration feature also had some issues with different behaviour when candidates were progressed via information from external ATS systems. Basically, if you progressed a candidate manually via the UI to the next step, several things would happen. If it were done automatically via the ATS, then it might do a bunch of stuff...
But wait...there's more... There is also existing code in the area for a large piece of work that was completed recently. So we have areas where we have feature-flagged whether a customer has some improved functionality enabled, and if not, use an "original" function, e.g., awardPoints vs awardPointsOriginal 👀
The Thinking Process
When developing a new feature, there are several approaches you can take. The main driver of the decision is to weigh up the pros and cons of a set of development/deployment strategies. The way I usually think about it is "which one results in the least risk". The last thing I want is to release something that results in the platform turning into a mess...that is on fire...in a hurricane... 😬
The weight of the decision can also be influenced by how the feature integrates into current user journeys. If the new feature is entirely new, then that is amazing. However, if it needs to augment a current workflow, then that is where things start to get a little bit messy. When you have to start looking at the decisions made before you to see how much effort it will take to integrate the feature (if it is even possible with the current implementation...or if this is one of the cases where a rewrite is necessary). I will go into more detail about different strategies in another post, but for now, I will just talk about the ones used for this piece of work.
The first strategy we will need to use is basically a consolidation refactor. We update the current candidate progression system so that all the behaviour is in a centralised system that is easy for other services to consume.
After the refactor, the new feature will use the Shadow Release strategy (what a shock, right?). But a shadow release that can be enabled on a customer-by-customer basis, so that we can test the functionality on non-prod environments, and eventually on prod with our internal test customer accounts.
Working with Constraints
I've already spoken about some of the things that I saw that NEEDED to be addressed before starting work on the feature itself. Bugs relating to differences in behaviour for different areas of the system. Consolidating all of the candidate progression into a single place would make life a lot easier...especially when it comes to diagnosing any future issues.
Have you ever had a bug come in where you go "I know where that is", and you fix it only to find out that there is similar (or maybe the same) code in another place where the bug actually resides 😬
Obviously, this is a project in itself...so in order to get buy-in, it was described to stakeholders as "Phase 1" of the feature. We need to improve the current system before adding customisability to it. This phase of the work would not need to be feature-flagged, as we are not changing any existing behaviour. Just making a more robust and coherent implementation.
When the refactor is done, it will have removed some of the underlying constraints that would have been present during the design of the new feature. The constraint is hard-coded values. These values were used for side-effect behaviour. For example, if a candidate has applied for a role, it might trigger an automated notification that is stored in the database. The candidate progression will have the stage of applied and the status of yes. And when looking for the notification to send it will use those two bits of information. So there was tight coupling between these two bits of data...
A constraint that we decided to remove early on was not to create a UI. So custom stages will need to be requested during the customer onboarding process (or requested at a later date). This reduces the need for any UI/UX effort for our initial release of this feature.
A critical constraint was the ability to enable this functionality for a customer, but also be able to roll back if any issues are discovered when it is eventually used by actual customers in production (a very late-stage tripwire).
Legacy Code Realities
A lot of stuff can happen in 5+ years...especially from a coding perspective. Without strict technical oversight, a codebase can quickly get into a hot mess. Deprecated functionality...unused (but still present) routes...a TODO list as long as your arm...and previous decisions (that were more than likely correct at the time, given the requirements).
Like I said before, one of the main benefits of this new piece of work was to take a long, hard look at the current candidate progression system and try to slap it into shape. Remove the issues of the past. Enforce consistency. And ensure data integrity (where database constraints are not viable). This would ultimately affect everyone, but with a net positive outcome. So, it's definitely worth the development effort even if the new feature did not exist.
Planning the Refactor
> First question: what would perfect look like if we started fresh?
> Mapping the current system
> Identifying where this new design could slot in
> Decision to create new models: CandidateProgressStage and CandidateProgressStageStatus
> Encapsulation of stage logic with a Manager to unify old and new behaviour
When my Product Owner and I get together to look at improving existing code, she likes to ask, "What would it look like if you were starting this from scratch?". That is an incredibly useful question when it comes to picturing a desired outcome. Akin to Test Driven Development (TDD), where you write the tests first, it encourages one to think about how you would like to "use" what you are building, and also has a higher chance of creating a more straightforward and more developer-friendly implementation.
I feel like this is where my constant reading of technical books really proves its value. I remember reading a book that spoke about software architecture, and a diagram really stuck with me. The concept of a Manager that sits in front of business logic. The manager acts as a kind of orchestrator for behaviour. So I thought that I could use that concept to create a manager class that was specifically for candidate progression, with a simple interface, but with the flexibility that would be required for phase 2 of this feature.
Previously, I had done a lot of work to provide a more modular approach to feature design, so I figured that it would be good to put all things related to candidate progression into its own module. The reason behind this wasn't just a personal preference, but a way of ensuring that behaviour is isolated, and that when more developers join the team, there is a reduced risk of merge conflicts because work could be planned around dedicated features.
I also wanted to avoid the trap of rushing into a "big bang" rewrite. The idea was never to bulldoze everything and hope for the best, but to carve out safe spaces where new code could live alongside the old. That is why the decision to introduce new models for stages and statuses felt important. By wrapping them in a manager layer, I gave myself a seam where both the legacy behaviour and the new behaviour could coexist.
This seam became the backbone of the plan. It meant we could start migrating logic incrementally, swapping out brittle parts of the old system for stronger, testable components. It also gave us an easy way to toggle behaviour, which was crucial for shadow releases and rollback scenarios.
By the end of the planning phase, the vision was clear: create a centralised manager, isolate progression logic into its own module, introduce new models to support flexibility, and keep the whole thing shielded by feature flags. Nothing flashy, just a deliberate foundation for the work to come.
To Be Continued
This post has focused on the thinking and preparation that went into planning the refactor. In the next part, I’ll walk through the implementation itself — the data model changes, the migration strategy, how we handled side-effects, and the feature flag rollout that let us deliver safely. That’s where the rubber really meets the road.