Making software production worthy

Your app, your product, your company’s future. The idea is sound, the market is ready, the team spent many months implementing it.

The questions corporate sponsors will be asking are: is it ready for production? When do we launch?

As an architect, how do you assess production readiness and chances of success in the field?

Here are some criteria to help formulate the answer to that question.

Code quality is good-to-great

Your code has little tech debt. The architecture is flexible, scalable. Features are easy to modify, augment, and add.

The dependencies are at the latest stable version.

The integration points have corresponding integration tests.

Core components have unit tests covering happy paths and failure paths.

Concerns are clearly separated.

Feature set is mature

The team and Product Owner has painstakingly iterated through the feature set multiple times, polishing workflows, interactions, screens, verbiage. Your features don’t just “work well”, the team starts using words like elegant, fun, cool to describe various things the product can do.

The last few times your Product Owner requested a new feature or a drastic change to an existing one, the team did not cringe and ask for “at least two sprints” to deliver it.

Change becomes fun when the software is architected well.

Onboarding, user management, and off boarding works near flawlessly

Once you are in production, you will get real customers. There is nothing more infuriating than an inferior or faulty onboarding process.

Your emails must look fun and exciting. Limit the number of steps until a customer can get in and feel like the registration is over. Defer all secondary actions to post-registration, if at all feasible.

Make it easy and fast for someone to provide the info you need to operate.

Give people options, for example make it possible to log in with email, phone number or user name. If you collect this data and require it to be unique but then don’t let the user identify himself to your system using the data he provided, you are not facilitating the best experience you can. Aside from that, people like to compartmentalize ids and passwords. Give your users the option to put your service in the “I use my phone number for this” compartment.

Change password, log out everywhere, change phone number, add another phone or email, recover password or user name, resend email confirmation – you remembered all of those, right?

Sometimes customers will want to leave your service. I know, crazy… But seriously, if they enter any data into your software, they are likely to want to take it with them and use it with a different service. Think of this and build a data export process before you go live.

Release pipeline is designed, implemented, and tested

Once you hit prod and get real customers, they are likely to be less forgiving and more inventive at breaking things than the best PO. You will have to roll releases fast and furious and your release pipeline better be ready for it.

Test your release pipeline like you test your software. Time it from the request for a code change to commit to build to verification to sign-offs to deploy. Identify the bottlenecks, understaffed areas, weak links, under-automated workflows. Correct the entire pipeline until you know you can release quickly and twenty people don’t have to spend six hours on a phone bridge yawning and hating your software.

Support process defined and tested

Go ahead and call or email your support pretending to be a customer. You are likely to realize that the support process is not ready, does not have enough support scripts, documentation, knowledge of the system.

This is what your customers will go through. Fix it. Document common workflows, escalation routes, build support utilities to help resolve common issues.

Also, do you have a good process for support reps to report frequently occurring issues to devs? It does not have to be catastrophic to become a sprint item, right?

It’s a great practice for support to publish a weekly newsletter with a name like “From the Trenches”, where they report successes, failures, funny stories, etc. We are social creatures, community building is important.

Code and environments are instrumented for analytics, logging, troubleshooting

Instrument your code well. Log important events. Provide workflow traceability. Your team’s post-release quality of life depends on it.

Understand what a customer will provide when calling about an issue and make sure your logs allow you to get from that input to that user’s interactions with your system. For example, customer will provide the email address. Can that tell you the APIs calls and workflows that user was executing and where things went off-rails?

DevOps are mature

As the system matures in production, it is likely to change. The moving parts may change, the hosting strategy may change, the external integrations and corresponding firewall and networking may change.

Is your DevOps organization mature enough to implement and support your release pipeline and rapid infrastructural changes you will throw at them?

To my mind, production-ready level of maturity for DevOps is when they can accommodate networking, security, hosting, pipeline changes within a couple of sprints at most.

How many things are DevOps still doing manually? Try to influence for more automation.

Hosting concerns resolved – certificates, servers, licenses, DSN, firewalls, load balancers, networking, CDN

I can’t tell you how many times I have seen releases plagued by incorrect or insufficient certs.

Weak servers, unpaid licenses, misconfigured DNS and firewalls… I’ve seen and lived through it all.

Test it, drive your system’s physical and logical hosting design, know the full checklist and people responsible for each part. Even if it is “another department’s fault”, it will be your software that won’t work.

Customers never report faulty firewall configurations, they report “I can’t log into your *&#@* app”.

Feedback pipeline

Just like onboarding and customer service, giving your users a well-designed feedback pipeline will make them feel valued and will make your product look professional and polished.

If customers can recommend features, vote for favorite items on your roadmap, report frustrations, your backlog will become a ton more meaningful. And so will your product. Don’t make it an afterthought. Build it before launch. Early life is when you will need it the most.

Beta testers identified, relationship established

Identify early adopters, nurture those relationships, incentivize them if possible.

An active community of early adopters can make the difference between a successful launch and total flop.

Product Owner is happy

Is your PO happy? Is there an excited ringing to his voice when the product is discussed? Is he using it every day? Has he switched from talking about bugs and imperfections to planning for future features and updates?

If your Product Owner is happy, chances are your users will be as well.

Developers like using the product

Do developers log in just to do development and test their work? Or do they love logging in just to navigate around and get a little dopamine boost from it?

A happy developer is an even better predictor of customer satisfaction than a happy product owner.

These criteria, of course, describe a semi-perfect world of unicorns and rainbows. They depict an ideal, desired but rarely achieved. Corporate decisions are often made very differently. But that reality should not preclude us, technical people and especially architects, from having a good set of criteria and rules to guide us in formulating a plan to get to prod and to make our product better after the release.

Every company today is a software company.

Let me clarify my point. It’s not that every company is now in the business of creating and selling software. But almost no company can survive without using software to conduct their business. If a company wants to survive and thrive, it must broaden its expertise and organizational structure to be a software company as well. So, a more accurate statement would be “every company today is also a software company”.

Today, there is hardly a business of any size that does not have a rather large technology component to it. Companies rely on software to communicate with customers, market their services, process orders, manage employees and customer relationships. If there is a retailer that does not have an online store — they are behind their competition. If a company offers a great service, has a web site, but does not have mobile apps — they are behind their competition.

One of the buzz phrases today – digital transformation – is the digital version of Lady Liberty. On it, the inscription reads:

Give me your tired, your manual,
Your erroneous masses of data yearning to make you money,
The wretched refuse of your teeming departments.
Send these, the unusable, outdated to me,
I lift my LED lamp beside the golden door!

And businesses see that dim LED light. And they truly want to transform. They bring in people in expensive business attire — consultants — who convince them that they are natives of the “land of the digital and the home of the electronic” and can guide them through the transformation.

In my long career in consulting, I have seen companies succeed and fail with this transformation. I have analyzed the empirical data I observed and formed some thoughts around what makes companies succeed or fail. Here are some of them, stated succinctly (each one deserves a dedicated post, if not a book). Here goes.

Being a software company is expensive. Very expensive.

You cannot sorta-kinda do it on the side. It requires people of many unique specialities, equipment, software, networks, infrastructure. Get used to seeing new types of bills and procurement requests and don’t treat them as luxuries, they are necessities.

Software development is an intense engineering discipline.

There are established computer science degree curriculums, known best practices and protocols, subculture of tribal knowledge, honor code, icons, humor. Hire professionals, with computer science degrees. But be aware – good developers are also artists. And big kids. The companies that realize that and cultivate the culture accordingly succeed more often.

Increasing the size of the development team is most often detrimental to the product success.

The analogy I would give is: software development is more like heart surgery than digging ditches. And while 10 diggers with shovels will dig a bigger ditch than 3, 10 surgeons will not perform a faster or better heart surgery.

Developing software requires a ton of coordination.

People in different departments must get comfortable talking with architects and developers, explaining the business processes in excruciating detail. It helps when the business side of the company begins to treat their technical ignorance as a serious impediment and makes an effort to get educated. They should, at least superficially, understand terms like HTTP, API, server, client, mobile app, platform, framework, deployment, source code, authentication, data type.

Software development is a collective effort, guided by strong and talented leaders.

It takes a team. There is no “I” in team. Yes, yes. But without a conductor, highly-skilled talented orchestra musicians do not play well together (in most cases). There are key roles that must be filled with the best people possible: product owner, architect, and dev lead.

So, come on! Lady Digital Liberty calls! Make more money and secure your company’s future by answering the call of her beckoning LED light.

Let me know how I can help.

What would it take for you to stop crying?

00253867-FD9E-479C-B0F2-C23426BA0849

As a parent, I learned a very important skill. When my baby is crying, I become a single-minded answer-seeking cyborg. My quest, my single purpose and total focus is to find the answer to the question – what would it take for my baby to stop crying?

Are you hungry? Are you tired? Thirsty? Need a clean diaper? Is your tummy hurting? Are your teeth cutting? I have food, drinks, a comfy and warm crib, clean diapers, creams and powders, tummy ache drops, teething chew toys… I can rock you in my arms, sing you a song… anything! Just to see you smile again… or fall asleep peacefully. I have a whole arsenal to make my baby happy, content, calm. But not until I figure out why the baby is crying.

When I meet a prospective client, I go into the same mode. Never mind the nonsense “we are looking for a .net guy”. What everybody is looking for is to remove pain points, make a good product and, ultimately, make money. And since most organizations are not great at self-diagnosis, the stated requirements and the opening salvos of the interviews are usually devoid of deep meaning.

I have a fairly large technical toolbox and decades of experience, but all is useless if I try feeding the baby when a change of diaper is what’s needed. So first, I need to get through the crying and misdirection and determine the true source of pain.

Is the architectural vision lacking and so the dev team is drifting along without making meaningful progress? Are the technical skills lacking and the team needs training, oversight, structure, and leadership? Is the Product not providing sufficient guidance and then is huffing and puffing at each demo that the team is not delivering the right bits? Are the integrations too complex for the team or are they unpredictable, unstable, prone to undiscoverable modifications? Is it the lack of tooling and infrastructure? Is the team burdened with manual DevOps or admin duties and is simply frustrated?

Asking questions and really listening to the answers (and the choice of words) is a good start. I try to listen for where the negative adjectives start appearing. “Daunting”, “tedious”, “hard” are telltales that you are close to the source of pain in you discovery process. Dig deeper. Ask questions with positive adjectives. Listen and watch.

It’s amazing how much can be accomplished without increasing effort by much once the true causes of pain are identified.