Making software production worthy

Your app, your product, your company’s future. The idea is sound, the market is ready, the team spent many months implementing it.

The questions corporate sponsors will be asking are: is it ready for production? When do we launch?

As an architect, how do you assess production readiness and chances of success in the field?

Here are some criteria to help formulate the answer to that question.

Code quality is good-to-great

Your code has little tech debt. The architecture is flexible, scalable. Features are easy to modify, augment, and add.

The dependencies are at the latest stable version.

The integration points have corresponding integration tests.

Core components have unit tests covering happy paths and failure paths.

Concerns are clearly separated.

Feature set is mature

The team and Product Owner has painstakingly iterated through the feature set multiple times, polishing workflows, interactions, screens, verbiage. Your features don’t just “work well”, the team starts using words like elegant, fun, cool to describe various things the product can do.

The last few times your Product Owner requested a new feature or a drastic change to an existing one, the team did not cringe and ask for “at least two sprints” to deliver it.

Change becomes fun when the software is architected well.

Onboarding, user management, and off boarding works near flawlessly

Once you are in production, you will get real customers. There is nothing more infuriating than an inferior or faulty onboarding process.

Your emails must look fun and exciting. Limit the number of steps until a customer can get in and feel like the registration is over. Defer all secondary actions to post-registration, if at all feasible.

Make it easy and fast for someone to provide the info you need to operate.

Give people options, for example make it possible to log in with email, phone number or user name. If you collect this data and require it to be unique but then don’t let the user identify himself to your system using the data he provided, you are not facilitating the best experience you can. Aside from that, people like to compartmentalize ids and passwords. Give your users the option to put your service in the “I use my phone number for this” compartment.

Change password, log out everywhere, change phone number, add another phone or email, recover password or user name, resend email confirmation – you remembered all of those, right?

Sometimes customers will want to leave your service. I know, crazy… But seriously, if they enter any data into your software, they are likely to want to take it with them and use it with a different service. Think of this and build a data export process before you go live.

Release pipeline is designed, implemented, and tested

Once you hit prod and get real customers, they are likely to be less forgiving and more inventive at breaking things than the best PO. You will have to roll releases fast and furious and your release pipeline better be ready for it.

Test your release pipeline like you test your software. Time it from the request for a code change to commit to build to verification to sign-offs to deploy. Identify the bottlenecks, understaffed areas, weak links, under-automated workflows. Correct the entire pipeline until you know you can release quickly and twenty people don’t have to spend six hours on a phone bridge yawning and hating your software.

Support process defined and tested

Go ahead and call or email your support pretending to be a customer. You are likely to realize that the support process is not ready, does not have enough support scripts, documentation, knowledge of the system.

This is what your customers will go through. Fix it. Document common workflows, escalation routes, build support utilities to help resolve common issues.

Also, do you have a good process for support reps to report frequently occurring issues to devs? It does not have to be catastrophic to become a sprint item, right?

It’s a great practice for support to publish a weekly newsletter with a name like “From the Trenches”, where they report successes, failures, funny stories, etc. We are social creatures, community building is important.

Code and environments are instrumented for analytics, logging, troubleshooting

Instrument your code well. Log important events. Provide workflow traceability. Your team’s post-release quality of life depends on it.

Understand what a customer will provide when calling about an issue and make sure your logs allow you to get from that input to that user’s interactions with your system. For example, customer will provide the email address. Can that tell you the APIs calls and workflows that user was executing and where things went off-rails?

DevOps are mature

As the system matures in production, it is likely to change. The moving parts may change, the hosting strategy may change, the external integrations and corresponding firewall and networking may change.

Is your DevOps organization mature enough to implement and support your release pipeline and rapid infrastructural changes you will throw at them?

To my mind, production-ready level of maturity for DevOps is when they can accommodate networking, security, hosting, pipeline changes within a couple of sprints at most.

How many things are DevOps still doing manually? Try to influence for more automation.

Hosting concerns resolved – certificates, servers, licenses, DSN, firewalls, load balancers, networking, CDN

I can’t tell you how many times I have seen releases plagued by incorrect or insufficient certs.

Weak servers, unpaid licenses, misconfigured DNS and firewalls… I’ve seen and lived through it all.

Test it, drive your system’s physical and logical hosting design, know the full checklist and people responsible for each part. Even if it is “another department’s fault”, it will be your software that won’t work.

Customers never report faulty firewall configurations, they report “I can’t log into your *&#@* app”.

Feedback pipeline

Just like onboarding and customer service, giving your users a well-designed feedback pipeline will make them feel valued and will make your product look professional and polished.

If customers can recommend features, vote for favorite items on your roadmap, report frustrations, your backlog will become a ton more meaningful. And so will your product. Don’t make it an afterthought. Build it before launch. Early life is when you will need it the most.

Beta testers identified, relationship established

Identify early adopters, nurture those relationships, incentivize them if possible.

An active community of early adopters can make the difference between a successful launch and total flop.

Product Owner is happy

Is your PO happy? Is there an excited ringing to his voice when the product is discussed? Is he using it every day? Has he switched from talking about bugs and imperfections to planning for future features and updates?

If your Product Owner is happy, chances are your users will be as well.

Developers like using the product

Do developers log in just to do development and test their work? Or do they love logging in just to navigate around and get a little dopamine boost from it?

A happy developer is an even better predictor of customer satisfaction than a happy product owner.

These criteria, of course, describe a semi-perfect world of unicorns and rainbows. They depict an ideal, desired but rarely achieved. Corporate decisions are often made very differently. But that reality should not preclude us, technical people and especially architects, from having a good set of criteria and rules to guide us in formulating a plan to get to prod and to make our product better after the release.

Every company today is a software company.

Let me clarify my point. It’s not that every company is now in the business of creating and selling software. But almost no company can survive without using software to conduct their business. If a company wants to survive and thrive, it must broaden its expertise and organizational structure to be a software company as well. So, a more accurate statement would be “every company today is also a software company”.

Today, there is hardly a business of any size that does not have a rather large technology component to it. Companies rely on software to communicate with customers, market their services, process orders, manage employees and customer relationships. If there is a retailer that does not have an online store — they are behind their competition. If a company offers a great service, has a web site, but does not have mobile apps — they are behind their competition.

One of the buzz phrases today – digital transformation – is the digital version of Lady Liberty. On it, the inscription reads:

Give me your tired, your manual,
Your erroneous masses of data yearning to make you money,
The wretched refuse of your teeming departments.
Send these, the unusable, outdated to me,
I lift my LED lamp beside the golden door!

And businesses see that dim LED light. And they truly want to transform. They bring in people in expensive business attire — consultants — who convince them that they are natives of the “land of the digital and the home of the electronic” and can guide them through the transformation.

In my long career in consulting, I have seen companies succeed and fail with this transformation. I have analyzed the empirical data I observed and formed some thoughts around what makes companies succeed or fail. Here are some of them, stated succinctly (each one deserves a dedicated post, if not a book). Here goes.

Being a software company is expensive. Very expensive.

You cannot sorta-kinda do it on the side. It requires people of many unique specialities, equipment, software, networks, infrastructure. Get used to seeing new types of bills and procurement requests and don’t treat them as luxuries, they are necessities.

Software development is an intense engineering discipline.

There are established computer science degree curriculums, known best practices and protocols, subculture of tribal knowledge, honor code, icons, humor. Hire professionals, with computer science degrees. But be aware – good developers are also artists. And big kids. The companies that realize that and cultivate the culture accordingly succeed more often.

Increasing the size of the development team is most often detrimental to the product success.

The analogy I would give is: software development is more like heart surgery than digging ditches. And while 10 diggers with shovels will dig a bigger ditch than 3, 10 surgeons will not perform a faster or better heart surgery.

Developing software requires a ton of coordination.

People in different departments must get comfortable talking with architects and developers, explaining the business processes in excruciating detail. It helps when the business side of the company begins to treat their technical ignorance as a serious impediment and makes an effort to get educated. They should, at least superficially, understand terms like HTTP, API, server, client, mobile app, platform, framework, deployment, source code, authentication, data type.

Software development is a collective effort, guided by strong and talented leaders.

It takes a team. There is no “I” in team. Yes, yes. But without a conductor, highly-skilled talented orchestra musicians do not play well together (in most cases). There are key roles that must be filled with the best people possible: product owner, architect, and dev lead.

So, come on! Lady Digital Liberty calls! Make more money and secure your company’s future by answering the call of her beckoning LED light.

Let me know how I can help.

The cost of interruptions in software development

Software development is at once a cognitive and collaborative task.

Our work is to take a business and translate it into software systems. The “how” of that process is a combination of computer science, experience, knowledge of industry tools and practices, and very hard work. But that “how” covers the cognitive side of software development, in other words – how I use my brain to do the work.

There is a second “how” – the collaborative “how”, and it is no less important than the cognitive one. To build a software product of any complexity today, you need a good team that encompasses many disciplines. Product people who know the business and understand the target audience for the software. Architects who can take the functional and non-functional requirements and logically organize them into segments and capabilities, design systems and govern development. Developers who turn designs into executable code. Testers who make sure the code performs as intended. Technical architects, network engineers, sysadmins who design, build and manage the hardware and network infrastructure for the software to run on. And that is a very brief list, far from complete.

To build good software, the team must communicate all the time. To build good software, the team must be mindful of interrupting one another’s cognitive tasks. Interruptions are unavoidable. But not all interruptions cost the same.

The case of a broken coffee machine

There were two coffee machines at one of my clients’ office. When both were working, it would take 3-5 minutes to walk over from my desk, brew a cup, and walk back. When one was broken, which was often, there would be a line and people would get into chitchat and by the time I would get back to my desk, my mind needed a few minutes to get back into the task at hand.

Keep developers comfortable with hot and cold drinks and make getting them a quick and easy thing.

The case of email chains that won’t die

When an email chain between people on the same team goes beyond the initial volley of send-reply, it becomes an interruption. If clarity cannot be achieved quickly, get up, walk over, and talk in person. Draw on a whiteboard. If the person is not physically at the same office – text, call, videoconference. Emails are not a good communication tool. They are static, easily lost in a busy inbox, easily forgotten.

Teams need good collaborative tools – I personally love Slack and Trello, but there are a ton of tools for any taste and use case.

The case of meetings

Make meetings as short as possible. Don’t invite passive observers – rather send them minutes. End the meeting early once the agenda is covered.

The case of physical space design and noise

Open space helps people collaborate. But if not designed right, it will also be a source of constant noise, visual field disturbance, and endless interruptions. People must have comfortable desks, big enough for work equipment and personal items, some privacy, some isolation from the noise. Comfortable chairs are also a must.

The case of poor hardware and networks

Our tools are huge and require lots of memory. Developer machines is not where you want to save money. If developers are made to work on sluggish machines, don’t expect productivity or quality (or loyalty).

We are constantly researching something or looking for answers – we need a quick and reliable Internet connection.

The case of unnecessary bureaucracy

Whatever you do, free your developers from the administrative burdens. Filling forms, submitting requests is not what software engineers like to do. In fact, it makes us cringe. A manager or scrum master can and should take that burden away from developers.

Removing or minimizing interruptions will result in better software, happier team, and – better business.

There is real money in saving developers from interruptions. After all, their per hour pay is not small. If 2 hours a day is wasted, that quickly adds up to tens of thousands of dollars per developer. So, before you make a decision on office space, network speed, developer laptops, software licenses, and collaboration platforms, ask yourself how much money you may be losing – every day – if you don’t make the right choices.