Handling customers that want feature previewing?

Most of the discussion here will be technical, especially around feature flags and friends. That’s fine, but please don’t miss out that this is a clear signal to charge these customers a lot more money.

Feature flags is usually how we handle this. We have the flags anyway for developer use, so making them user specific was not that much incremental effort though we have had a few cases where old workflows were not removed when they should have been.

We did once have a customer with their own environment with a different version deployed, as they were initially a major client with such demands but after a few years it became clear it wasn't worth the maintenence effort to special case this one client so we made it clear in contract renewal one year that this would be going away.

Sounds to me like the customer is trying to solve a problem and is coming to you with their chosen solution. My suggestion: Find out what problem they are trying to solve, and address that.

Also, you might look at cPanel's notices that they send out where they explain upcoming feature updates well in advance of the rollout. They also have different release cycles, where the newest features are released to the EDGE cycle instead of the STABLE, and then the long term release cycle remains unchanged except or security updates.

Have you clarified why they want this notice?

In my experience it’s because if you roll out a new feature (or non-trivial UX change), that can wreck someone’s day and cause internal support issues into the company that uses your product.

Consider your rollout of new features to be least impacting, don’t confuse someone who’s using your product to get work done by changing something that breaks a workflow that worked yesterday etc.

You can put new features in new sections, or need special users to activate them.

In some cases we would show screenshots of upcoming features to hear about any other feedback, but otherwise focused on new features not being disruptive.

Context: financial SaaS of interest to banks, brokerages and RIAs; no direct-to-consumer at this time.

We have two special case preview environments. They're set up just like production, but with lower capacity and no SLA.

The first one is general previews of what we're planning on releasing; it's a normal part of our release process and every release is there for 2-14 days before going to production. All customers have access; some use it extensively, and some basically ignore it.

The second one is for specific in-development features; in theory we can spin up multiples of this, but in practice we don't want to diverge too wildly. Generally we showcase one feature at a time, and we gate the features by customers.

Figure out why they're asking that. This appears to be an "XY problem"[0].

An example of this is a friend asking me how to weld a wire to a thin metallic plate. I digged further to find the actual problem for which his solution was to weld the two. He said it was a fuse's wire that melted and he wanted to weld a bigger wire to avoid that, to which I replied: don't. The whole point of the fuse is sacrifice to protect down stream equipment.

The same when a relative asks how to format a disk as their solution to solve a problem.

Generally speaking, implementing customers' solutions is a recipe for disaster, especially with enterprise.

Ask what it is they are really trying to accomplish, and figure out a way to solve that. Listening for problems instead of solutions or implementations.

- [0]: https://en.wikipedia.org/wiki/XY_problem

I had a similar situation in fintech. We had 3 different release cycles and hosting farms for fast, medium and slow accepting customers. Fast went to the customers who didn't require approvals, and who were flex when it came to errors in the release. Then there was a medium channel where the other customers were fine as long as a couple of the major customers in that pack approved things. The slow channel basically was for customers who had large validation or training requirements associated with releases.

"Thank you for your inquiry. This is not a service that we offer at this time -- our testing and release process is fully automated and there is not a way for us to exclude some customers and not others.

Can you please help us understand the purpose of the advance notice/release delay is? What benefit does a process like that provide to you or your organization?"

---

I generally use some variation of that ^^. Having some customers on different release tracks is very painful in my experience, especially if it's done without properly planning for it in your release process. For example, let's say that right now, anything committed to `main` goes out to production via some automated process. What happens when you have a security bug that needs dealt with ASAP? In my org, this usually means that the security patch gets rolled out to everyone, and then when it's time for customers on a slower release schedule to "catch up", there are conflicts to resolve manually.

It's just not worth it. Figure out why they're asking, but say no. If it's a problem with buggy releases, that's the _actual_ problem to put some time into solving.

Lots of good answers below on how to make it work, how you should charge more, etc.

Remember that saying no is also an option. You aren’t obliged to say yes just because someone asks for something you don’t sell, even if they are willing to pay.

For most SaaS there’s no logical reason why having a feature for 2 extra weeks would be an enormous advantage to a customer. In my experience what this question really is is “can we make last minute change requests to a feature you thought was done”. I’m not keen to guarantee that, so it’s a simple no, sorry, we can’t do that. It’s yet to be a deal breaker.

There is a lot of talk of feature flags but that only works when you can realistically delay your feature 2 weeks so that the 'special' set of clients can see it early. What we've seen that has worked is essentially demoing the feature in staging/test to stakeholders at the major/demanding client. They are happy that they aren't taken by surprise and then we provide training materials for major feature/UI changes to their users. Never have had an issue, client is happy and essentially zero technical involvement.

We use release branches (git) for such cases - mainly because our deployment is based-off release branches, thus gives us flexibility to move a customer to a pre-release or even have then stay at a particular release forever. Bug fixes can be back ported as needed.

Tried feature flags but it introduced complexity.

I wouldn’t charge more as others are suggesting- bugs you’ll address before general availability is worth the trouble.

I really like how Salesforce do it: they make an announcement that a feature/fix will be available from day x and depending on its importance, they then give 100,200,500 days or whatever to install it or it gets installed automatically on the deadline day. This gives plenty of notice to customers to prepare( updates can break some customisation on client's instance).

You need to consider the blast radius and how long features may be in preview. Ideally, you would scope the preview to just a subset of users. When you go to a complete customer base you start to get into the situation where the customer has failed to communicate with their users, and have more people raising issues. This isn't bad necessarily, but you have to be prepared to handle it.

If you keep something in preview for a long time then there can be a divergence from the main product. Make sure that data or work from users is migrated across when you finally release.

It makes sense to think about the opportunity for support to participate in the preview. Let them listen in to customer calls and learn about the problems. Providing a dedicated support person for previews is tempting, but long-term it's better to avoid this specialization and enable the whole support team to participate. This shares knowledge and avoids certain people being granted special privileges while others toil on average support tickets.

Microsoft publish some of their methodology at https://docs.microsoft.com/en-us/microsoft-365/admin/manage/.... The key point is that they have set out the boundaries for their program so that customers know what to expect and it can be repeated for every customer.

You will have to learn how to do your previews, but you should lay down some rules. Set it up so kudos from participants can be used for marketing after. You are the one to decide when the preview ends, but setting some minimum time in preview gives customers more confidence and time for you to process feedback. Make it clear if features are locked when you enter the preview and what feedback you are after.

Once the preview is over, thank the participants. Call them out if you have a public forum. Participation can become a mini case study for your marketing efforts.

I also had a handful of customers like this. We ended up creating an opt-in “beta” channel that we treated as a first-class citizen along with “prod”.

Basically we had 2 production stacks with CI/CD for each. The only shared thing was the AWS ALB. Our customers each have their own subdomain so we used that to point to different EC2 ASGs.

We would continue pushing bug fixes to “prod”. Once a beta period ended, we merged to master and re-deployed master to both prod and beta channels. Rinse and repeat.

It works out pretty well. Occasionally there’s a really important bug fix that we have to cherry pick from prod to beta, but it’s rare enough to not be a huge concern.

Oh, and, charge them for it. I never got any pushback.

Charge them extra for a "staging" environment setup only for them and then deploy all latest stuff there for them to test. Then get their approval to apply it to their production environment. Charge well for this.

You need at least 2 sandbox environments: one that's more development-oriented (little to no restrictions on merging), and one that's more release-oriented (as close as possible to production).

Ideally you have 3, and the third one is not used except to demo new features to customers AFTER they have been released (and perhaps, hidden behind a feature flag).

If you only have one sandbox, then it becomes a brawl between development, sales, and management.

More than three is overdoing it and assuming a technical burden for no benefit. (No, you don't need the capacity to spin up sandboxes willy-nilly. Three is enough.)

Difficult to answer without understanding how you are deploying currently. I do think feature flags can be an anti pattern. I would ask the customers why they want it in the first place. Is it a trust issue? Is it an internal process that they require? Ideally you would want to stay flexible, that would mean not tying your company to their schedule. It is a hard application to deploy multiple times for each customer. Perhaps a more lts approach for those customers might work. Just keep in mind the larger the releases get, the more issues it might cause.

I would suggest pre-defined staged rollout based on customer preference. For example, create three category Alpha, Beta, Stable.

Alpha: Gets new features immediately and gets support within 24hours if something breaks.

Beta: Gets new features Every 45 week, and gets priority support if needed.

Stable: Gets new features quarterly, Mostly they are forced to upgrade from old features which you dont want to support, But they will be essential volume of your revenue.

alternatively, you can give them sandboxed demo read-only(resets every night at 12 o'clock) account for testing.

well, self horn here... but that is a basic functionality of https://platform.sh (CPO here). You get per branch full copies of production in isolated staging environments that are very cheap to run. So you just clone production to a specific branch apply whatever changes and grant access to the clone. ..you do need to run production with us for the magic though... and a PaaS is not for all use-cases (specifically when running a SaaS...).

We pick those customers to be beta testers. They receive pre-release features, but in return they must accept possible bugs will happen (of course, when one is identified we give them the choice of disabling the feature immediately).

The schedule is totally in our hands, though. Once we say it’s time to go GA, it goes. At this point there still are feature toggles, though, just in case.

If a large customer wants to decide when they get new features, they need to pay for their own dedicated environment that is on a separate release cycle. Otherwise they need to adapt to our release schedules and make sure they get all the previewing that they need on a test environment.

We create a separate branch for the customer and preview this branch with wunderpreview and give the customer the link to the preview.

Every once in a while we merge new stuff in the customers branch so she has always the latest stuff in her preview.

We have a powershell script that:

1. Copies the production do 2. Removes sensitive data 3. Deploys the chosen build of that app (from Appveyor) pointing to that db

So in a nutshell CI/CD pipeline

If you are having trouble perhaps there are a lot of manual steps to your deployment.

One way is to make use of AppConfig to store customerwise feature roll out plan. To save cost, add 15 minute caching on it (the thick client comes with the local application caching feature)

The solution is to ask those customers to pay for the extra work required. If they refuse then it isn’t important enough for them.

Seems like you need advanced feature flags.

Is it that the customers want early access to new features, or they don't want you breaking the product?

Once you know you need to do this (and at every enterprise SaaS company I've seen or worked at, you eventually need to do this - the sibling comments about charging more $$$ are smart though):

* "Release train ahead" - code takes 2 weeks (or 4 weeks) to reach Production from merge, but it goes out to a Preview environment immediately or at the next release. So, you've forked your code into a Prod train, a Preview train, and mainline. You probably need the ability to hotfix to prod and preview fix to the week-ahead-of-prod preview environment, as well as roll normal development forward. You do gain "soak time" for code in the Preview environment, which is a slight advantage. And your QA is easier, because you have three discrete, testable things ("does it work in Prod, does it work in Preview, does it work in trunk"). But, you have another runtime environment which can diverge, which is a big disadvantage.

* Feature flagging by "release train," single execution environment. This is probably the most "Software Theorist Friendly" way to do this. You don't have to maintain/debug multiple environments beyond what you're already doing. Some customers are in the "preview ring" and get more features toggled on. Single runtime environment, single codeline. Fixes are just fixes. QA is a little harder depending on how you set up the possible "rings" and combinations, and the operational blast radius of your Preview code is now the same as your Production code (make a bad query that kills a backend service in Preview code, well, that's also Prod code. Oof.).

* Free-range feature flagging. Some complex arbiter system manages a huge range of feature toggles on a per-tenant/per-customer basis. Having this range of operational knobs is very fun and works great for customers who are particularly risk averse. It also lets you sell SKU-packs of flag combinations, roll specific features to specific "Early Access Programs," and so on. But, you get a combinatorial explosion of feature flag permutations, which depending on how coupled your code is and how intelligent your QA system is, can become a disaster: "Oh no, FooFeature only works with BarFeature on but our QA system always has BarFeature on so we didn't catch the regression! And only one customer has BarFeature disabled for legacy reasons!"

* Some combination of 1, 2, and 3 - for example a Preview environment that runs the same _code_ as Prod, but with the toggles set differently. A few feature flags which can be enabled for early access programs, but a general rule that code needs to live in a release train. This is my favorite approach. Having a separate Preview environment eliminates the blast radius concern of Preview-toggled code breaking Prod. Operational costs are higher, but these can be passed to the customer (making customers buy preview environments as a line-item is pretty popular). But by maintaining single codeline (rather than deploying a fork of last week's code to Prod), the code-theory is still easy, you don't have divergent release trains, and you don't have the combinatorial explosion that comes from a free-range feature flagging system.