COVID-19 WFH reflections.

I write software and work with distributed remote teams for a living. So collaborating with people without being next to them is not a new concept to me. The software development culture has long been moving towards the totally remote, asynchronous co-working.

Look at the Open Source Software culture, for instance – big groups of people, who are mostly just GitHub nicknames and silly profile pictures to each other (though some are real life friends, of course). They collaborate across multiple time zones and languages, mostly via electronic messaging and pull requests. And they build amazing software that gets used by millions of grateful people every day.

The corporate world of Those Who Wear Expensive Suits has been slow at adapting this culture. Management still believes that close supervision and making developers dress a certain way will result in better software and produce results faster. Wrong! It results in procrastination and LinkedIn profile update to “Open to opportunities”.

We, as a group, work best in t-shirts and jeans, without face-to-face meetings with management. The best supervision for us is being genuinely excited about what we are building. You see real progress when developers are passionate about what they are building, feel respected by their peers, have good coffee and tea selection.

What changed with COVID-19? Now that working from home is forced, it does not feel quite as sweet. I catch myself missing the office, missing the simple world of just a few weeks ago when people were shaking hands and going to lunch together, squatting at Starbucks with a laptop and a cup of good coffee. I turn on my camera for calls, which I never did before. Special thanks to Microsoft Teams for the “blur background” feature!

What else is new? A keen realization that, unlike software, most of the world’s jobs cannot be done remotely. I now see grocery store cashiers as front-line heroes. Another very tangible realization – most software is not indispensable, so as the businesses close and multimillion lines form at unemployment offices, tons of software projects will be shut down.

Making software production worthy

Your app, your product, your company’s future. The idea is sound, the market is ready, the team spent many months implementing it.

The questions corporate sponsors will be asking are: is it ready for production? When do we launch?

As an architect, how do you assess production readiness and chances of success in the field?

Here are some criteria to help formulate the answer to that question.

Code quality is good-to-great

Your code has little tech debt. The architecture is flexible, scalable. Features are easy to modify, augment, and add.

The dependencies are at the latest stable version.

The integration points have corresponding integration tests.

Core components have unit tests covering happy paths and failure paths.

Concerns are clearly separated.

Feature set is mature

The team and Product Owner has painstakingly iterated through the feature set multiple times, polishing workflows, interactions, screens, verbiage. Your features don’t just “work well”, the team starts using words like elegant, fun, cool to describe various things the product can do.

The last few times your Product Owner requested a new feature or a drastic change to an existing one, the team did not cringe and ask for “at least two sprints” to deliver it.

Change becomes fun when the software is architected well.

Onboarding, user management, and off boarding works near flawlessly

Once you are in production, you will get real customers. There is nothing more infuriating than an inferior or faulty onboarding process.

Your emails must look fun and exciting. Limit the number of steps until a customer can get in and feel like the registration is over. Defer all secondary actions to post-registration, if at all feasible.

Make it easy and fast for someone to provide the info you need to operate.

Give people options, for example make it possible to log in with email, phone number or user name. If you collect this data and require it to be unique but then don’t let the user identify himself to your system using the data he provided, you are not facilitating the best experience you can. Aside from that, people like to compartmentalize ids and passwords. Give your users the option to put your service in the “I use my phone number for this” compartment.

Change password, log out everywhere, change phone number, add another phone or email, recover password or user name, resend email confirmation – you remembered all of those, right?

Sometimes customers will want to leave your service. I know, crazy… But seriously, if they enter any data into your software, they are likely to want to take it with them and use it with a different service. Think of this and build a data export process before you go live.

Release pipeline is designed, implemented, and tested

Once you hit prod and get real customers, they are likely to be less forgiving and more inventive at breaking things than the best PO. You will have to roll releases fast and furious and your release pipeline better be ready for it.

Test your release pipeline like you test your software. Time it from the request for a code change to commit to build to verification to sign-offs to deploy. Identify the bottlenecks, understaffed areas, weak links, under-automated workflows. Correct the entire pipeline until you know you can release quickly and twenty people don’t have to spend six hours on a phone bridge yawning and hating your software.

Support process defined and tested

Go ahead and call or email your support pretending to be a customer. You are likely to realize that the support process is not ready, does not have enough support scripts, documentation, knowledge of the system.

This is what your customers will go through. Fix it. Document common workflows, escalation routes, build support utilities to help resolve common issues.

Also, do you have a good process for support reps to report frequently occurring issues to devs? It does not have to be catastrophic to become a sprint item, right?

It’s a great practice for support to publish a weekly newsletter with a name like “From the Trenches”, where they report successes, failures, funny stories, etc. We are social creatures, community building is important.

Code and environments are instrumented for analytics, logging, troubleshooting

Instrument your code well. Log important events. Provide workflow traceability. Your team’s post-release quality of life depends on it.

Understand what a customer will provide when calling about an issue and make sure your logs allow you to get from that input to that user’s interactions with your system. For example, customer will provide the email address. Can that tell you the APIs calls and workflows that user was executing and where things went off-rails?

DevOps are mature

As the system matures in production, it is likely to change. The moving parts may change, the hosting strategy may change, the external integrations and corresponding firewall and networking may change.

Is your DevOps organization mature enough to implement and support your release pipeline and rapid infrastructural changes you will throw at them?

To my mind, production-ready level of maturity for DevOps is when they can accommodate networking, security, hosting, pipeline changes within a couple of sprints at most.

How many things are DevOps still doing manually? Try to influence for more automation.

Hosting concerns resolved – certificates, servers, licenses, DSN, firewalls, load balancers, networking, CDN

I can’t tell you how many times I have seen releases plagued by incorrect or insufficient certs.

Weak servers, unpaid licenses, misconfigured DNS and firewalls… I’ve seen and lived through it all.

Test it, drive your system’s physical and logical hosting design, know the full checklist and people responsible for each part. Even if it is “another department’s fault”, it will be your software that won’t work.

Customers never report faulty firewall configurations, they report “I can’t log into your *&#@* app”.

Feedback pipeline

Just like onboarding and customer service, giving your users a well-designed feedback pipeline will make them feel valued and will make your product look professional and polished.

If customers can recommend features, vote for favorite items on your roadmap, report frustrations, your backlog will become a ton more meaningful. And so will your product. Don’t make it an afterthought. Build it before launch. Early life is when you will need it the most.

Beta testers identified, relationship established

Identify early adopters, nurture those relationships, incentivize them if possible.

An active community of early adopters can make the difference between a successful launch and total flop.

Product Owner is happy

Is your PO happy? Is there an excited ringing to his voice when the product is discussed? Is he using it every day? Has he switched from talking about bugs and imperfections to planning for future features and updates?

If your Product Owner is happy, chances are your users will be as well.

Developers like using the product

Do developers log in just to do development and test their work? Or do they love logging in just to navigate around and get a little dopamine boost from it?

A happy developer is an even better predictor of customer satisfaction than a happy product owner.

These criteria, of course, describe a semi-perfect world of unicorns and rainbows. They depict an ideal, desired but rarely achieved. Corporate decisions are often made very differently. But that reality should not preclude us, technical people and especially architects, from having a good set of criteria and rules to guide us in formulating a plan to get to prod and to make our product better after the release.

Every company today is a software company.

Let me clarify my point. It’s not that every company is now in the business of creating and selling software. But almost no company can survive without using software to conduct their business. If a company wants to survive and thrive, it must broaden its expertise and organizational structure to be a software company as well. So, a more accurate statement would be “every company today is also a software company”.

Today, there is hardly a business of any size that does not have a rather large technology component to it. Companies rely on software to communicate with customers, market their services, process orders, manage employees and customer relationships. If there is a retailer that does not have an online store — they are behind their competition. If a company offers a great service, has a web site, but does not have mobile apps — they are behind their competition.

One of the buzz phrases today – digital transformation – is the digital version of Lady Liberty. On it, the inscription reads:

Give me your tired, your manual,
Your erroneous masses of data yearning to make you money,
The wretched refuse of your teeming departments.
Send these, the unusable, outdated to me,
I lift my LED lamp beside the golden door!

And businesses see that dim LED light. And they truly want to transform. They bring in people in expensive business attire — consultants — who convince them that they are natives of the “land of the digital and the home of the electronic” and can guide them through the transformation.

In my long career in consulting, I have seen companies succeed and fail with this transformation. I have analyzed the empirical data I observed and formed some thoughts around what makes companies succeed or fail. Here are some of them, stated succinctly (each one deserves a dedicated post, if not a book). Here goes.

Being a software company is expensive. Very expensive.

You cannot sorta-kinda do it on the side. It requires people of many unique specialities, equipment, software, networks, infrastructure. Get used to seeing new types of bills and procurement requests and don’t treat them as luxuries, they are necessities.

Software development is an intense engineering discipline.

There are established computer science degree curriculums, known best practices and protocols, subculture of tribal knowledge, honor code, icons, humor. Hire professionals, with computer science degrees. But be aware – good developers are also artists. And big kids. The companies that realize that and cultivate the culture accordingly succeed more often.

Increasing the size of the development team is most often detrimental to the product success.

The analogy I would give is: software development is more like heart surgery than digging ditches. And while 10 diggers with shovels will dig a bigger ditch than 3, 10 surgeons will not perform a faster or better heart surgery.

Developing software requires a ton of coordination.

People in different departments must get comfortable talking with architects and developers, explaining the business processes in excruciating detail. It helps when the business side of the company begins to treat their technical ignorance as a serious impediment and makes an effort to get educated. They should, at least superficially, understand terms like HTTP, API, server, client, mobile app, platform, framework, deployment, source code, authentication, data type.

Software development is a collective effort, guided by strong and talented leaders.

It takes a team. There is no “I” in team. Yes, yes. But without a conductor, highly-skilled talented orchestra musicians do not play well together (in most cases). There are key roles that must be filled with the best people possible: product owner, architect, and dev lead.

So, come on! Lady Digital Liberty calls! Make more money and secure your company’s future by answering the call of her beckoning LED light.

Let me know how I can help.

Paradigm mixing

I like my hot beverages at instantly sip-able temperature – above warm, below hot. Yet I do not want to add calories to each hot drink I take, so milk is not an option. When I make my own coffee, I know what to do. But when I work in an office that has a coffee shop but no DIY coffee station, I am forced to develop a request interface to convey my need to a barista. I began by asking for ice cubes in my coffee. 8 ice cubes in a large cup. 8 ice cubes. That got more eye rolls and comments than I expected. The same baristas that are happy to make you an iced tea or coffee roll their eyes when you ask for a cup of hot coffee with 8 ice cubes in it. Why?

I thought – ok, 8 is a neurotic sounding number, perhaps rounding it up to 10 would appease the baristas. It did, marginally. 10 was better than 8 but I was still getting the annoyed facial expressions.

I was not giving up. 8 was too weird, 10 was too specific-sounding, – I reasoned, – so maybe asking for “a few” ice cubes – taking a precise number out of the request – would help matters. It did not. The baristas were less annoyed, but I was still not getting a desired result. A few apparently means 2-4 and several means 4-6. My coffee was never going to be just right.

There is no right way to ask for a hot coffee with some ice cubes and get exactly what you want – without agreeing to be viewed as a “strange guy”. It just seemed like a manifestation of a general human trait rather than bad customer service.

We like things to be familiar. We like patterns. We feel safe when everything around us makes sense. There are concepts we grow accustomed to, for better or worse, and any deviation from them puts us at unease. To apply these generalities to my coffee problem, I was trying to mix two paradigms that usually live in their separate worlds. There is Hot Coffee (would you like some room for cream? – they ask). There is Iced Coffee (fill the cup with ice, that’s right, all the way to the brim, then pour hot coffee). Hot coffee with 10 ice cubes is a mutant. It is a paradigm mix and anybody asking for such a thing must be an alien.

It has been a common experience for me to get a “nobody has ever asked for this before” remark.

Paradigm mixing belongs in the world of heroes. It is the world inhabited by the paralympian legless runner, the 100-year old marathoner, trucker-turn-Elvis, one-handed violinist. It belongs in the news and in the movies. But taken to its trivial level, paradigm mix scares us. That is one of the main reasons most people find it so difficult to come up with an original idea. An original idea is very often about taking two things that exist separately and making them into one and very often it is about enablement.

Think about it. There have always been people who lost limbs or eyesight. The history of sports competitions go back millennia. Yet somehow, until recently, nobody thought of putting the two separately existing things together and making a truly remarkable human and business venture of it.

Self-publishing, the phenomenal success of YouTube, social networks becoming huge financial successes are all testimony to the idea that people are as diverse and creative as the environment allows them to be. Give people tools to express themselves and the society will benefit.

How does this relate to my perfect cup of coffee problem? Simple. The modern coffee shop is stuck in a rigid model of yesteryears and is not willing to become creative. Give people tools, put a little fun into getting coffee and the business will be booming, I am sure of it.

The cost of interruptions in software development

Software development is at once a cognitive and collaborative task.

Our work is to take a business and translate it into software systems. The “how” of that process is a combination of computer science, experience, knowledge of industry tools and practices, and very hard work. But that “how” covers the cognitive side of software development, in other words – how I use my brain to do the work.

There is a second “how” – the collaborative “how”, and it is no less important than the cognitive one. To build a software product of any complexity today, you need a good team that encompasses many disciplines. Product people who know the business and understand the target audience for the software. Architects who can take the functional and non-functional requirements and logically organize them into segments and capabilities, design systems and govern development. Developers who turn designs into executable code. Testers who make sure the code performs as intended. Technical architects, network engineers, sysadmins who design, build and manage the hardware and network infrastructure for the software to run on. And that is a very brief list, far from complete.

To build good software, the team must communicate all the time. To build good software, the team must be mindful of interrupting one another’s cognitive tasks. Interruptions are unavoidable. But not all interruptions cost the same.

The case of a broken coffee machine

There were two coffee machines at one of my clients’ office. When both were working, it would take 3-5 minutes to walk over from my desk, brew a cup, and walk back. When one was broken, which was often, there would be a line and people would get into chitchat and by the time I would get back to my desk, my mind needed a few minutes to get back into the task at hand.

Keep developers comfortable with hot and cold drinks and make getting them a quick and easy thing.

The case of email chains that won’t die

When an email chain between people on the same team goes beyond the initial volley of send-reply, it becomes an interruption. If clarity cannot be achieved quickly, get up, walk over, and talk in person. Draw on a whiteboard. If the person is not physically at the same office – text, call, videoconference. Emails are not a good communication tool. They are static, easily lost in a busy inbox, easily forgotten.

Teams need good collaborative tools – I personally love Slack and Trello, but there are a ton of tools for any taste and use case.

The case of meetings

Make meetings as short as possible. Don’t invite passive observers – rather send them minutes. End the meeting early once the agenda is covered.

The case of physical space design and noise

Open space helps people collaborate. But if not designed right, it will also be a source of constant noise, visual field disturbance, and endless interruptions. People must have comfortable desks, big enough for work equipment and personal items, some privacy, some isolation from the noise. Comfortable chairs are also a must.

The case of poor hardware and networks

Our tools are huge and require lots of memory. Developer machines is not where you want to save money. If developers are made to work on sluggish machines, don’t expect productivity or quality (or loyalty).

We are constantly researching something or looking for answers – we need a quick and reliable Internet connection.

The case of unnecessary bureaucracy

Whatever you do, free your developers from the administrative burdens. Filling forms, submitting requests is not what software engineers like to do. In fact, it makes us cringe. A manager or scrum master can and should take that burden away from developers.

Removing or minimizing interruptions will result in better software, happier team, and – better business.

There is real money in saving developers from interruptions. After all, their per hour pay is not small. If 2 hours a day is wasted, that quickly adds up to tens of thousands of dollars per developer. So, before you make a decision on office space, network speed, developer laptops, software licenses, and collaboration platforms, ask yourself how much money you may be losing – every day – if you don’t make the right choices.

What would it take for you to stop crying?


As a parent, I learned a very important skill. When my baby is crying, I become a single-minded answer-seeking cyborg. My quest, my single purpose and total focus is to find the answer to the question – what would it take for my baby to stop crying?

Are you hungry? Are you tired? Thirsty? Need a clean diaper? Is your tummy hurting? Are your teeth cutting? I have food, drinks, a comfy and warm crib, clean diapers, creams and powders, tummy ache drops, teething chew toys… I can rock you in my arms, sing you a song… anything! Just to see you smile again… or fall asleep peacefully. I have a whole arsenal to make my baby happy, content, calm. But not until I figure out why the baby is crying.

When I meet a prospective client, I go into the same mode. Never mind the nonsense “we are looking for a .net guy”. What everybody is looking for is to remove pain points, make a good product and, ultimately, make money. And since most organizations are not great at self-diagnosis, the stated requirements and the opening salvos of the interviews are usually devoid of deep meaning.

I have a fairly large technical toolbox and decades of experience, but all is useless if I try feeding the baby when a change of diaper is what’s needed. So first, I need to get through the crying and misdirection and determine the true source of pain.

Is the architectural vision lacking and so the dev team is drifting along without making meaningful progress? Are the technical skills lacking and the team needs training, oversight, structure, and leadership? Is the Product not providing sufficient guidance and then is huffing and puffing at each demo that the team is not delivering the right bits? Are the integrations too complex for the team or are they unpredictable, unstable, prone to undiscoverable modifications? Is it the lack of tooling and infrastructure? Is the team burdened with manual DevOps or admin duties and is simply frustrated?

Asking questions and really listening to the answers (and the choice of words) is a good start. I try to listen for where the negative adjectives start appearing. “Daunting”, “tedious”, “hard” are telltales that you are close to the source of pain in you discovery process. Dig deeper. Ask questions with positive adjectives. Listen and watch.

It’s amazing how much can be accomplished without increasing effort by much once the true causes of pain are identified.