The Business​ Value of Continuous Testing

Why do software faile

Team members in software development make mistakes and insert defects in to the software development artifacts.  Defects may result in failures, or they may not, In some cases , a defect can exist that will never cause a failure in actual use, because the condition that cause the failure can never arise. In other other cases, a defect  will not cause a failure during testing, but could results in failure in production. Where there are defects there are risks for failures. Failures lowers quality and business value of the IT system.

What is Testing

“Testing is an empirical investigation conducted to provide feedback to stakeholders and team memebers quality information  aboun the IT system.”

As Elisabeth Hendrickson, Vand  said, “When I headed up quality engineering, I described my job as ‘creating feedback cycles.’ Feedback is critical because it is what allows us to steer. We must constantly validate between customer needs, our intentions and our implementations. Testing is merely one sort of feedback.”

Feedback from test create confidence in the quality to the It system. It let us do changes more quickly and with less fear. It let us focus on new IT capablity that we are adding by having the the test tell us whenever we break the old IT capabilities.

Testing provides learning opportunity.  The quality information is used in retrospectsives to find prosess and quality inprovment activities.

Goals of testing

• A IT system that fulfil stakeholders expectation
• A IT system that achieve the business goals

• Find defects
• Gain confidence in the level of quality

• Provide quality information for decision making
  • quality level
  • quality risk
  • business risk
• Keep  defects out of artifacts
• Measure progress of development

Problems that hindres achivment of testing goals

Quality is something all agree is important but not enough focus in software development. What is usually focused on is time, cost and features, because that is easy to measure. Software team lack the competence in testing to provide and use quality measurement.  Poor quality measurement  are a result of the misconception that testing activity lengthen development schedule and increase development costs.

Testing has always been the elephant in the room. Psychologically, the malleable nature of software has given organizations an excuse to defer investment in testing. However, this deferral results in technical debt. Over time, technical debt compounds, escalating the risk and complexity associated with each release.

There is a lot of misconception of  testing

• It is usual that it will be failures in the software look at what Miscrosoft is delivering.
• To do systematic testing is expensive so we do some ad-hoc testing at the end by the resources and time that are left to reach the definition done.
• Testing the software product enough takes a long time and are too expensive.
• The traditional approach fosters the perception that building quality software is overly complex and expensive.
• One do not need of specific competence in testing to testing
• Testers are  has been developers that did not manage to make a carrier in development

There are very few that will say that testing has no business value. But when you ask them what they are willing to spend, what feature that can wait until later iteration in order to

• have the reduced business- and quality risk,
• provide long-term savings
• do testing to provide quality information

You find out that people will be vague about their willingness to invest in testing.  Few can quantify, describe and articulate the business value of testing.

Test cases are usually designed to detect functional defects. To find the defects that cause the most severe failure during operations and increase software debt,  one also need to have focus on the other quality attributes.

Many software products are built with not enough focus on testing in the beginning and one start testing late in the development cycle and only with ad-hoc testing. When testing begins the product is of low quality and contains a lot showstopper and serious failures that lengthen the development schedule and there will be done many shortcuts to get the software out that lower the software quality and increase the software debt.

What happens is that when the software product is put into production with a list with a lot of failure to fix, many failures that cause a lot of support work.  So instead of developing new feature, the team members are occupied with failure discovery, failure fixing and support.

Code  defects is introduced during coding.  I this defects causes failure in production can cause high costs.  The earlier defects is found and removed the low is the cost of defect removal.

Defect Injection / Defect foulnd / Cost of defects

Testing is not something that we should only start once a IT capability complete for testing. Janet Gregory says: “Throw away the ‘then’ in ‘code then test’ – replace it with ‘and’. And maybe, also reverse the order to say, ‘test  AND code’. Put the test first.” “By shifting left, it feels like the development process is a linear process, and it’s not.”

When code is written but not tested we have a partial done software. Software development churn is associated with partially done software. When testing occurs long after coding, test-and-fix churn is inevitable. This code

Much of the risk of software development lies in partially done work.

The inventory of software development is partially done work. Partially done software crates following problems:

• it gets lost,
• grows obsolete,
• hides quality problems,
• ties up money.
• much of the risk of software development lies in partially done work.

What Is ContinuousTesting

Devops Pipeline

Continuous Testing is a software testing in which the product is evaluated early, often, and throughout the entire Dev Ops Pipeline

Because testing is so essential, we should be doing it all the time as an integral part of the dev ops lifecycle. So I do not understand how do we do shift left when we should test in any phase of the lifecycle.

Your goal should be to build quality into the code from the start, not test it in later.You avoid creating defects in the first place. You do not allow defects to inserted into the artifacts in the first place. If this is not possible, then you test the artifact after each small step, so that defects are caught immediately after they occur. When a defect is found, you stop-the-line, find its cause, and fix it immediately.

To keep most defects out of code as it is being developed are tests written before code, and executed the tests frequently after a small increment of the code is added. New code is not written before all the test passes. Team members integrate both code and tests into the software repository as often as possible—every hour or so—and run a test suite to see that no defects have been introduced. If the tests don’t pass, they do not add new code until the problem is fixed or the failing code is reverted

When a feature is implement the you execute regression test suite. Finding defects should be the exception, not the rule, during regression testing. If regression testing routinely triggers test-and-fix cycles, then the development process is defective.

• Automated unit and acceptance tests should be run against every commit to version control to give developers fast feedback on their changes.
• Developers should be able to run all automated tests on their workstations in order to triage and fix defects.
• Testers should be performing exploratory testing continuously against the latest builds to come out of CI.
• No one should be saying they are “done” with any work until all relevant automated tests have been written and are passing. Implementing continuous delivery means creating multiple feedback loops to ensure that high-quality software product gets delivered to users more frequent.
• When implemented correctly, the process of releasing new versions to stakeholders should be a routine activity that can be performed on demand at any time.

Automated Tests

Continuous testing uses automated tests to ensure teams receive immediate feedback to quickly mitigate as many risks as possible throughout the software development lifecycle. Moreover, team members are able to continuously learn about their IT system and what can be done to increase quality. When the automated tests pass, teams are confident that their IT is releasable. Furthermore, they are confident that test failures indicate a real defect.

Automated tests should not be handover to QA group or outsourced to another party. When team members are involved in creating and maintaining acceptance tests, there are two important effects.

• The code becomes more testable.
• Team members care more about the tests and will invest more effort into maintaining and fixing them.

Business Value of continuose testing

We get the focus on quality by creating av feedback loop by using systematic testing that informs us about the quality of the software product. This allows us

• To find and remove defects early while they are smaller, cheaper, and easier to fix.
• Remove defects before they cause failure in production.
• Create feedback that could be used for organizational learning that improve prosess and product quality of  future development. When failures occur, we treat them as opportunities for learning, as opposed to a cause for punishment and blame.
•  Quality information  helps us to develop software product without living in fear that failure can occur in production in  any moment. Defects will be detected long before they ends as failures in production. Failures in production that lowers the stakeholders satisfaction  and decreased  buisness value of the software product.

Every team member is responsible for testing the software product . Having  team members share responsibility for the quality of the  team members not only improves outcomes but also accelerates learning. It’s impossible for a team members to learn anything when someone yells at them for a defect the inserted in software product  six months ago. That’s why we need to provide feedback to everyone as quickly as possible, in minutes, not months.

• reduce failure costs
• If the support and operation are tired of being surprised by unknown quality problems, we can fix that by
  • finding  defects long before they occures as failure in production
  • .by reducing defectes that is inserted into the artifacts
  • enable confident, informed decisions about quality by delivering quality information.

•  If the stakeholders wants to enjoy the marketing and sales advantages of an improved reputation for quality, testing will help with that
• If the stakeholders wants to have smoother, more predictable releases, testing will help with that.
• If the stakeholders wants to have increased confidence upon release,  systematic testing can help to ensure that what really matters gets tested.
• If the stakeholders wants protection from legal liability, testing can help by ensuring compliant, defensible testing.
• If the stakeholders is in the safety-critical or mission-critical business, and wants to ensure of gaining business value of the software product, we can work with the  team throughout the lifecycle to solves as many problems as we can prior to release.
• Ensure decided quality level
• Providing quality information for the decision about the
  • software development process,
  • software product,
• Preventing and finding defects
• Help us to understand the product
• Reduce
  • Business risks
  • Proces risks
  • Quality risks
• Decrease the incompletion of requirement, architecture, and design (s
  • reveals the
    • defects,
    • omission
    • ambiguity

• Find defect: Ensure as many defects are found before being released to production
• Test early and often: Tested throughout the development, delivery, testing, and deployment cycles
• Earn stakeholders loyalty: Accomplish continuous improvement and quality
• Automation: Automate your test cases to decrease time spent testing
• Increase release rate: Speed up delivery to production and release faster
• Reduce business risks: Assess potential problems before they become an actual problem
• Communication transparency: Eliminate handover between the development, testing, and operations teams

Problems with Continuous Testing

While continuous testing has a myriad of key benefits, there are several challenges that software development teams must take into consideration:

• Adjust to DevOps: Professionals don’t process the right tools and training for continuous testing within Agile and DevOps environments
• Change in culture: Cultural shifts among your development and testing teams may happen if traditional processes are maintained
• Update testing strategy: Maintaining only traditional testing methods and test data management that is not clearly defined keeps continuous testing from reaching its full potential
• Code integration: Developers who don’t integration their code on a regular basis (recommended several times daily) create defect issues with duplicated coding efforts and non-compatible code
• Test environments: Make sure your test environments work within your code repository base for seamless testing of the newest available code
• Production environments: Also, make sure your production environments reflect the test environment to ensure every area was properly tested
• Visibility: Getting visible throughout the Devops pipeline

• Test Inside the same cycle as IT capability is developed :
  • poorly defined requirements
  • complexity maintaining versions of requirements leading to
    • coding errors
    • test inefficiency
  • maintaining appropriate test-case coverage
    • overlap and redundancy in test cases
• difficult to designing and maintaining meaningful test cases that are aligned with the stakeholders expectation
• difficult to understanding how to design the test cases from the requirement specification
  • risk of accidental missing or designing incorrect test cases
• used to much time on maintaining automation scripts
• lack of test data
• Non functional testing out of focus
• Release complexity: Release often involve multiple It system with dependencies , different technology and potential conflicting resources
• Difficult to understand what needs to be tested.
• Difficult to identify problems by gathering data across the pipeline
• Difficult to coordinate and sharing knowledge across the different tools in the Devops pipeline