Five Forces Against Agile

Let’s say you had to choose a default strategy for developing software:

  • bureaucracy, waterfall development, centralised decision-making and micromanagement?
  • Or a culture of trust, autonomous cross-functional teams, continuous delivery and iterative development?

Seemingly, people in the IT-industry now have come to recognise the latter as a more successful strategy. We know that agile works. Even in the public sector. Then why is the transformation from slow, rigid and expensive processes to a more successful, cost efficient way of generating better value so hard?

Sure, there are obstacles in factors like these:

  • Budgets
  • Contracts
  • Architecture
  • Organization
  • Legacy systems
  • Tools and technologies

These are certainly real barriers for an agile transition. However, they are all tangible, visible and, dare I say it, easier to deal with than the more complex, darker forces lurking in the shadows of the human mind (they are not really dark forces, but a normal part of our human nature and survival strategies).

In this post I suggest five forces I see working against a better way of developing software. All of them might not be relevant in your context and some are stronger or weaker depending on your context. If none of them are familiar, consider yourself lucky.

1. Individuals resist being changed

If you work in a startup or with innovation, you might be familiar with the diffusion of innovations. Only 16% of people are considered Innovators and Early adopters. Humans have a history of surviving by staying with the pack and following the majority, and most people need serious consideration to accept new ideas. For organizations, the adoption to new ideas are even more complex.

change

2. People enjoy power

Agile is about autonomy, learning by doing and letting the team decide how they solve the problem. This requires a large dose of trust, not just inside the team but from management and leaders. Agile does not work very well in a command and control environment.

Unfortunately, people who are used to being in control and exercise micromanagement will naturally find it hard to let go of their control. It requires a shift from Theory X to Theory Y, where not only leaders learn to trust but the team member stop playing the blame game and learn to take responsibility.

3. “We would rather be wrong than uncertain”

Agile are optimized for changing requirements and uncertainty. Humans are optimized for categorizing data and generalizing problem solving. This gives us the warm and fuzzy illusion of control. We are extremely bad at dealing with uncertainty. As Dan North says it: We would rather be wrong than uncertain.

The irony is that agile sceptics often mistake agilists for being daredevils and risk takers, when in fact agile is handeling risk in a much safer way than what carving assumptions into stone is.

One consequence of this is that we are tricking ourself into believing we can predict the future with mental images based on a weak analogies.

The House Analogy

One often used analogy for software development is building houses. This gives us the illusion that we are building something concrete which we should plan in great and accurate details up front, and it will in fact turn out that way. And the bigger the better. It also implies that demolishing the house and starting over is expensive, when in fact, deleting code and starting over is literally done by a keystroke.

The Elephant Analogy

Another analogy we often use is comparing projects to elephants. “How do you eat an elephant? One bit at a time.” This gives us the illusion that we know exactly how the end result is going to turn out, and we can just split the problem into 100% accurate increments and assemble all the pieces perfectly together at the end. When in fact we are trying to grow a new creature that doesn’t exist yet and no one has ever seen before.

43-eating-elephant

 

4. Agile threatens The Establishment

This is an often overlooked consequence of following the agile principles. Delivering value early and frequently has a significant impact on how we are organized and what skills are needed. Our goal is to get real feedback early. “Working software is the primary measure of progress”. Instead of getting written specifications handed over, developers are working face-to-face with business people and user representatives. We are in the complex domain of the Cynefin framework and apply a probe, sense and respond approach. This greatly reduces the need for detailed planning, estimations and comprehensive documentation up front. Consequently, it reduces the need for coordinators, managers and administrators.

mvp
http://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp

 

It can then seem like a contradiction when we choose to outsource programming, making it harder to collaborate through geographical distances and cultural barriers. At least it generates more work for management. Christin Gorman has an excellent point about this.

5. Agile is small and cheap

Agile is in a sense about keeping it small and simple and eliminating waste. Simplicity and downsizing does not generate the same amount of need for middle managers, coordinators, documentation, meetings and administration. Yes, we are saving costs and that is a positive force for the company paying for the IT related services. But on the other side we are also cutting income for the people making money out of those very activities that does not generate direct value.

Agile generate less income for management consultancies than the waterfall model (unless you are selling agile coaching, courses and certifications). Unfortunately, money is power and this is a strong force I think often is overlooked by somewhat naive techies. Budgeting and economy is not our favourite topic (and economists  on the other side don’t bother with understanding technology either when they outsource core business to India hoping for cost savings).

And let’s be honest. For humans in general, smaller projects are not as impressive on our resume.

“Don’t fight stupid – Make more awesome”

Introducing and sustaining agile can sometimes seem like a dark and long path through frustrating and complex people issues. Introducing change to a business requires great patience and a positive attitude. But it is also necessary to understand the challenges and unfavorable conditions working against you when all your best effort, logic, reasoning and patient are not getting you any closer to your goal. Recognizing the human drivers at play can make them easier to tackle and to know when to stop banging your head against the wall. Maybe you will even find a way to use them to your advantage?

Taking a deeper look at these human forces will also make you realize that all of them are a natural part of our nature and not as evil as they first might appear.

Fortunately, there are lots of techniques for overcoming people issues. Fearless Change (a wonderful read) by Linda Rising and Mary Lynn Manns contains powerful patterns that will make your mission towards change more fun and more efficient.

Besides, being the most adaptive creature on earth, humans have the capability of learning new skills and adopting to change. Bosses are becoming leaders, test managers are becoming an integrated part of development teams and project managers are providing their skills as coaches and scrum masters. And as I have learned from coaching theory and from reading Fearless Change: In order to introduce change, you have to genuinely believe in the hidden abilities and good intentions of people.

Five Forces Against Agile

An agile approach to quality

This post is about how the mindset and methodologies of our company affects the way we work with quality requirements. It further suggests a five step approach on how to work with quality requirements in an agile setting.

Since there seems to be a lot of different definitions and understandings about quality, we’ll start with a brief introduction to the terms.

What is quality?

System or software product quality is the “degree to which a software product satisfies stated and implied needs when used under specified conditions”, according to ISO 25010.

Each stakeholder has an interest in software quality. For example, the user wants the system to be safe, responsive and user-friendly, the developer wants good logging and the product owner wants the system to be easy to change in the future (even if they tend to prioritize more functionality over a maintainable solution, but then it is your job as a tech person to explain the business implications of that decision).

Software quality is categorized into eight characteristics by ISO 25010.

product quality 3
The eight product quality characteristics as defined by ISO 25010.

Related to these characteristics we define non-functional requirements, like numbers of requests per second, security level and response time. We often say that non-functional requirements defines how the system is supposed to be, whereas functional requirements defines what the system is supposed to do.

Some people further make a distinction between external quality and internal quality, where the external quality is experiences by the users and the internal quality is experienced by the developers.

requirements
A simplified view of the hierarchy of requirements.

If it’s not functional, then what is it?

“Non-functional requirements” is not a self-explanatory term. Non-functional is often interpreted as not related to functionality or even worse: technical. When you talk about non-functional requirements you will hear things like “I’m a functional tester/architect/[insert title here], so I don’t care about non-functional requirements”.

To save yourself the pain of having to convince everyone that non-functional requirements are important and actually can be related to functionality, talk about quality requirements instead. Everyone cares about quality. Or at least it’s harder to say “I don’t care about the quality of the product we are making”.

Quality benefits all stakeholders

Quality requirements are a means to deliver value to all stakeholders – not just the end-user. This is why you and everyone involved in software development should work with quality requirements early and continuously:

  • The user expects user-friendlier, faster, more secure and reliable systems.
  • A maintainable system speeds up lead time, bug fixing and time to onboard new team members.
  • Early focus on quality needs helps you make better choices when it comes to technology, architecture and design.

The problem with not paying attention to quality occurs when we are heavily invested in an architecture or technology that simply does not support our needs. We risk spending too much effort on the wrong solution and are faced with expensive changes.

How mindset affects quality

The path to quality depends on your company’s mindset. Is your IT company managed like a production factory where one change takes numerous man hours and goes through several handovers between different activity-oriented teams consisting of resources i.e. people? Or do you work in a company where people are trusted, empowered and encouraged to be creative and practise DevOps with continuous deployment? Taking the same plan-driven approach to quality as you would in a large factory to a modern IT environment will most likely slow you down and create more resistance towards quality work than it will do any good.

How is our approach to quality in agile development different from that in a waterfall development?

The waterfall path to quality

For the sake of simplicity, we’ll look at waterfall and agile as two extremities, even though most tech companies probably deals with a mix of the two.

Back in the days when most of the IT industry were doing software development in a control-oriented linear fashion, quality requirements were carefully analyzed and specified up front. They were written down in a separate document and there were dedicated processes for handling quality. In a way, this is a more structured and easier way of working with quality.

  • You dedicate time for gathering all known requirements up front.
  • You write the requirements once and approve the quality requirements early in the process.
  • You get the quality you specify up front ((unless the handovers are causing too much information loss) even if that turned out to be not what you actually needed).

Besides getting what you ask for and not necessarily what you need, the traditional approach to quality encourage exhaustive lists of requirements up-front. One consequence of this is that the development team are put in a brain-off mode, ticking off check-lists with requirements they don’t have any relation to. This might work well if we are building something predictable and concrete (in the “obvious” domain of the Cynefin framework), but not so well when in a creative and more abstract world where most of the requirements are unclear and changing.

The agile path to quality

In an agile and feedback oriented way of creating software we don’t always have the “luxury” of spending days and weeks on making exhaustive lists of requirements in advance or testing every aspect of the product before going live.

Two important distinctions from the waterfall approach:

  • The quality is mostly built-in by iterating over the product based on feedback from real users and from running the software in production over time.
  • The developers need to work closer with a variety of stakeholders in order to understand the requirements.

Agile doesn’t equal good quality

You might think that an iterative approach to system development with continuous delivery and stakeholder collaboration effortlessly leads you towards good quality. Besides, refactoring is a natural part of agile.

Sure, agile development (with an agile mindset – not just the rituals) can help us enhance quality. It can also do the opposite. From my experience, the way we practice agile does not always emphasize quality in architecture and design the same way it emphasize functionality.  There might be different reasons for this:

  • We don’t make time for planning and documenting the design, because it’s hard to predict up front and will be up for change later.
  • We are strongly driven by the amount of functionality we deliver to the end-user, but not the long-term quality as seen from other stakeholders (a natural consequence of the product owner i.e. customer representative prioritizing the backlog).
  • Convincing the product owner we need to improve quality characteristics, like portability and interoperability, can be hard.

Five steps to quality

How do we ensure the right quality characteristics without wasting time on heavy processes, comprehensive documentation and long check-lists that will soon be outdated?

These are my key suggestions on how to work with quality requirements in a more agile way. I also find this approach useful for more non-agile projects.

1. Define a product quality model for your organization

The product quality model defined in ISO 25010 provides a set of useful quality characteristics and sub-characteristics. Adjust the model to fit your organization, and make it well-known among the different stakeholders. Project managers, test experts, testers, developers, UX and UI experts, architects, product owner and everyone who has an interest in the system should be familiar with the model. This will create awareness and help draw attention to delivering on quality, not just time and cost.

2. Prioritize quality characteristics

For each system, bring the necessary stakeholders together and prioritize the relevant characteristics for that system. The ATAM process can be a good technique for making tradeoffs. This is useful for giving everyone a shared view on what is the goal and purpose of the system.

3. Establish quality requirements

Based on the prioritized characteristics, define clear, verifiable requirements with criteria for satisfaction and corresponding measures. This can be done for both

  • the system or company in general and
  • to any specific change or projects.

For the first alternative you can make a general list of mandatory requirements like the ones you are required to follow by law. These are usually predictable and less likely to change over time. Keep the list short, up to date and only enforce it on changes where they are relevant.

Keep the requirements on a reasonable level when it comes to technical details. You don’t want to tell the experts how to do their job or take the responsibility away from the developement team.

4. Treat quality requirements the same way you treat functional requirements

This one is not obvious: When your test manager identify what types of tests that needs to be run, it often doesn’t matter if the requirements are technical, non-technical, functional, non-functional or whatnot. A requirement is a requirement. What matter is the types of tests that needs to be run. You can test quality attributes like security, logging and interoperability with functional testing, combining them with test of functional requirements. Often there is also an overlap between functional and quality requirements, e.g. when it comes to functional suitability.

Separating quality from functionality might be good for drawing attention to quality, but requires an extra effort in administration and defining processes and responsibilities. Architectural work can also be inconsistent when we separate quality from functionality. For the same reason, I do not differentiate between external and internal quality when working with quality requirements (please leave me a comment if you disagree).

Write user stories and make use of the tools and processes already in place. There is no need to invest in a fancy tool for managing quality requirements (maybe for executing the tests, but not for the requirements themselves).

5. Measure, monitor, evaluate and adjust the quality requirements 

As we gain more knowledge about our domain and as the world around us change, the quality requirements also change.  Make use of feedback from the real world to continuously improve the system and the requirements.

Conclusion

Agile is not a get out of jail free card when it comes to planning and documenting. We shouldn’t spend too much time planning, but we shouldn’t make assumptions when it comes to quality needs either. An early, continuous and systematic approach to quality, as defined above, can make your stakeholders more happy, save you from having to rewrite large part of the system later and even increase lead time.

From the 9th principle of The Agile Manifesto: “Continuous attention to technical excellence and good design enhances agility”.

An agile approach to quality