Mindset
First of all one should be aware that in Agile, QA automation takes a completely different approach to traditional approach. Because more then QA its the TEAM who would be automating things, as the project is scaled to meet the project objectives. ROI of any project would increase top-down if we consider a pyramid structure (Illustrated as below).
As you can see from the diagram, If you have more unit Tests, Integration tests/ API tests (that are done under the hood, right below the UI layer) - that would allow you to catch defects sooner then later. GUI tests should be written only for the business critical functionality that can not be automated in Unit/ Integration testing. There could definitely be some kinda 3rd party controls that the application uses that might not be possible to test in Development or there could be a scenario where Dev can't see all the object elements exposed so as to write tests and the list goes on. How to pick those tests, what tests should be automated in QA etc is discussed below
Working Environment
Automation in Agile environment is key player to the success of the project but at the same time if there’s no proper planning, it obviously a failure. On the other hand, the whole project planning plays a key role in successfully delivering automation.
First of all one should be aware that in Agile, QA automation takes a completely different approach to traditional approach. Because more then QA its the TEAM who would be automating things, as the project is scaled to meet the project objectives. ROI of any project would increase top-down if we consider a pyramid structure (Illustrated as below).
As you can see from the diagram, If you have more unit Tests, Integration tests/ API tests (that are done under the hood, right below the UI layer) - that would allow you to catch defects sooner then later. GUI tests should be written only for the business critical functionality that can not be automated in Unit/ Integration testing. There could definitely be some kinda 3rd party controls that the application uses that might not be possible to test in Development or there could be a scenario where Dev can't see all the object elements exposed so as to write tests and the list goes on. How to pick those tests, what tests should be automated in QA etc is discussed below
Working Environment
Automation in Agile environment is key player to the success of the project but at the same time if there’s no proper planning, it obviously a failure. On the other hand, the whole project planning plays a key role in successfully delivering automation.
If you have the full project in ones brain it doesn’t work that way. Everything should be put on paper in the form of backlog. Organizing the backlog is a scrum master’s responsibility, though. But it has a greater impact on delivery of the automation suite as well.
In my opinion when the system architecture is designed, automation engineer should do an evaluation of the company standard tool to see if that would work with the given technologies, environment. If not, alert the project manager so that can be negotiated around whether it is okay to go out side the company standard tools (ex: open source tools) or any other best fits etc. Basically its automation engineer’s responsibility to get the right tool sorted before initiating the project. Key responsibilities during this process:
- Evaluate the company standard tool against the given architecture, technologies, environment
- Make recommendations on alternatives
- Tool evaluation: Pick the best fit of the available by running POC against each of the available options
- Build Automation Environment
Once all the above tasks are done, we’re half way thru the successfully delivering automation. Quite frequent issues are shown when you’re running on a baseless environment or every time a change happens you have re-build the environment from scratch or using some one else’s environment and so on. Rather this could be avoided by building your own environment with all the pre-requisites and maintain it throughout the project delivery. That’s the catch here.
When this is all done and you are now running in a Scrum team. Few things that QA manager has to sort out first i.e., Resource Management
Ø Make sure there’s a min 1:3 ratio of Dev to QA resources
Ø Make sure there’s a min 1:3 ratio of Automation to QA resources
Note: 1:3 ratio of Automation Specialist to QA is based on the assumption that QA resources are technically capable of maintaining the automated scripts.
There could obviously be problems keeping up-to-speed with development if the right ration of QA team is not maintained. The same way a right ratio of Automation team to QA should also be maintained, otherwise automation would lag behind the schedule, too much pressure as the project goes along etc that is unnecessary when you can sort all them upfront with little bit of planning of resources.
Best practice is to add resources as the project goes along as what’s happening with the project is well determined when it’s rolling on. When, where, and how the new resource can best fit the existing team, can also be determined only when it’s already running. There’s no point having a bunch of people sitting there and doing nothing, either.
Enough talk about the preparation, getting the house in line. Let’s get in to the sprints.
Live Environment Examples
I will try drawing a rough picture of how automation in an agile should be approached. In my opinion, take small steps before walking & walk before you run.
Sprint1: 1st week – Build an automation framework. You might be spending couple of days to a week writing your framework. This is so crucial and I would say basic items. Without a proper framework also you might enter cycles but is not an organized/ professional way of going about it
Sprint1: 2nd week – Automation feasibility study & Business critical functionality identification. You would never do a better thing then this. It’s so important for both business and automation delivery to identify the scenarios that would be worth automating. There’s no point taking every little story in to account and end up in doing nothing. This is identified by the product owner, if not, by Business Analyst
Start automating whatever can be done & what is delivered by Development that is stable enough to automate & that is marked as Business Critical
Sprint2: 1st week – Automation Backlog of Sprint1. There are certainly some stories that would still need to be automated in the first week or so. This week should also be a good time to plan for Sprint#2 automation that includes identifying max ROI functions that needs to be automated etc
Sprint2: 2nd week – Continue Automating whatever can be done & what is delivered by Development that is stable enough to automate & that is marked as Business Critical
Sprint3: Same as Sprint2
Recovery Sprint: Clear off any Technical Debt. This sprint basically to help the whole team NOT JUST Automation team to:
Ø Get up-to-speed with the Development. Update/ add/ Run any pending tests that QA has to do. Update the quality management tools to reflect any additions/ update and keep it up-to-speed. MANUAL TESTERS
Ø Update the Unit tests, API/ Integration tests that should be changed to sync with the latest feature developments. For example, you might have written a test that was good for Sprint1 but it is not valid any more. Have you updated all of them? Make sure the answer is YES. Its most useful for clearing out any technical debt that you’re slowly started accumulating. Let’s have NO TECHNICAL DEBT. Clear that off from day-1. That’s the main goal. DEVELOPERS
Ø Automation testers should go back & re-visit the automated functions/ scripts that they have coded. In most cases, you need to update the object repositories or Mapping files or might have to change the whole function altogether. Because changes are tend to be big enough in 3 sprints. If you do not update them now, you would end up in feeling like re-writing the whole thing againL? Which I am sure you hate to do. AUTOMATION TESTERS
Continue along these lines for the rest of the project completion. Note: This is under assumption that you're running a typical 2 week sprints.
I suggest having 1 week sprints for every 3 sprints, however, it should change depending on the project. Factors that might help you consider recovery sprint are like project complexity/ amount of changes going on etc.
I suggest having 1 week sprints for every 3 sprints, however, it should change depending on the project. Factors that might help you consider recovery sprint are like project complexity/ amount of changes going on etc.
I think if we continue something similar to above guidelines, we should not be far off from successfully delivering the project.
Conclusion
Main Goal of Automation should be:
Ø High ROI
Ø Provide Business Value
Simply forget about 100% Automation. Delivering all functionality, should not be your goal.
Sharing your experiences & any comments are much appreciated..All the best with your project
