version: Beta Date Updated: 31/Aug/2017
I have started my IT career in Oct 2002 as QA engineer and more than decade have always followed Waterfall model of software development in different companies, even though the nature of Waterfall model has changed a lot in last decade.
Waterfall model is like Indian government contractors and engineers laying out world class (?) roads in IT capital of India which is Bangalore. This road won’t last for more than quarter or month I should say, then they will do world class (?) fixes and story continues.
I’ve seen similar thing happening ( but not as worst as in above case ) with majority of the Waterfall/ modified waterfall / fast agile projects as well. ( Please note in Indian software industry situation is not as bad as Indian government organizations , otherwise we wouldn’t be ruling silicon valley and IT world. I just wanted to give some real life scenario to explain concept , advance apologies if somebody is hurt with this analogy ) .
But if you want to build and maintain true world class software which your consumers will love for many years , then you must follow BDD techniques and Cucumber is the tool which comes in handy.
In below post; I have shared my learnings wrt BDD and Cucumber usage in Automation projects on Web, Mobile and RESTful API.
1. Issues with non-BDD project
2. Benefits of BDD
3. To Read, Refer, Revisit & Repeat
4. Projects To Refer
5. Misunderstandings & True Understanding about BDD , Cucumber
7. Gherkins Syntax Best Practices
1. Issues with non-BDD Project
1.1 Not everyone is same loop ( BA / Product Manager ,Dev ,QA ) , requirements floating around over chat, individual email / communication.
1.2 Stale requirements specifications document and test cases. Nobody is responsible to keep document upto date
1.3 Who is responsible for software’s quality ? Who is responsible for bug missed to production environment and found by customers ?
1.4 In majority of cases, no one really follows agile but simply fast waterfall model of software development
1.5 Technical debt and bugs keep on mounting without proper prioritization
1.6 Testing is done for sake of testing and feel good factor.
1.7 Tester decides “What is most valuable” testing; but it may not be always accurate from business point of view
1.8 Testing Pyramid decisions taken by testers only , which may not be optimized solution
2. Benefits of BDD
2.1 Test Scenarios / User Stories /Scenarios are always maintained – added for new features, updated as per changes in product, removed if no more applicable
2.2 Avoid regression issues during development cycle itself – saves money in finding, fixing and testing time
2.3 Test First Approach – automation case is ready ( or at least template ) even before code is ready
2.4 Quicker and accurate prioritization possible for technical debt, bugs based on features usage from real world scenarios
2.5 Testing is based on “Acceptance Criteria” which is agreed, understood & approved by 3 amigos of team ( PM/BA, Dev , QA )
2.6 Business Facing Scenarios is correctly prioritized by 3 amigos, so minimizing customers issues which results in money saving with re-work, re-test, support, bad ratings etc
2.7 Testing Pyramid decision taken collectively by Dev and QA , to come up with efficient way to test user story
How To Achieve BDD ? Quite Simple Start Doing “Example Mapping” for each sprint
– Seriously this is as simple as this sounds
– Whole team + management needs to be committed to this new approach; where everyone is involved in software development which matters to customer, value for their money
– Refer to pt. 3 articles in this blog to read more about it
Above 2 points can be summarized in below real world experiences
How QA engineer can be used as Assistant Cook instead of Waiter ?
Traditionally ( Waterfall , Fast Waterfall ) QA engineer is treated like a waiter in restaurant. Head cook ( developer ) cooks the dish and Waiter ( QA ) delivers dish to customer after doing food sampling test ( to certify it is in “OK” condition ).
With BDD approach there is opportunity for Waiter to become Assistant Cook , who can look after each of the steps during dish cooking. If something going wrong , can pin point during cooking process itself.
Assistant cook can check if quality of ingredients is best, if utensils are clean, if oven temperature is good enough, kitchen hygiene is maintained etc. Later once sampling test is done and certified ; dish can be served to customer.
Also as waiter s/he can get feedback from customer about exact dish requirement or any small changes needed, convey same to head cook.
The main difference with water fall model ( fast water fall / agile ) vs BDD is QA engineer with his/her skills of domain , customer , programming knowledge can be better utilized in development process to minimize bugs, tech debut, bad UX, re-work etc
3. To Read, Refer, Revisit & Repeat
“In this tutorial, we have covered features of cucumber tool and its usage in real time scenario.
Cucumber is a most favorite tool for many projects as it is easy to understand, readable and contains business functionality.” ( Pt. 9 )
“Integration of Selenium WebDriver With Cucumber” ( Pt. 9 )
“Keeping specifications, regression tests and documentation in a single place reduces the overhead of keeping multiple documents in sync – the Cucumber scenarios work as a shared source of truth for business and IT” ( Pt. 13 )
“You could write your Cucumber scenarios before you write the code so that programmers can be guided by an unambiguous specification. This would allow programmers to catch mistakes immediately.” ( Pt. 14 )
“We have begun to see a “testing first” mindset being adopted. The teams are trying to drive that, and they’re beginning to think about how their Definition of Done can included a repeatable regression test. With the three amigos sessions, we’re seeing more collaboration and this is being fed back to us by the teams, better understanding, better communication as well.” ( Pt. 15 )
9. Cucumber , Java , Selenium Introduction
11. http://toolsqa.com/cucumber/cucumber-tutorial/ ( Cucumber , Selenium , Java )
12. http://www.ibm.com/developerworks/library/a-automating-ria/ ( Selenium , Cucumber )
13. https://cucumber.io/blog/2015/07/01/bdd-in-the-financial-sector?mc_cid=0c26611d80&mc_eid=e461e83eee ( BDD In The Finance Sector )
14. https://cucumber.io/blog/2015/03/24/single-source-of-truth ( Are you doing BDD ? OR are you just using Cucumber ? )
( Challenges introducing to team )
16 . Agile or Fast Waterfall ?
“How do you get the various roles in a Scrum team to have a common understanding about a feature?
Our team discovered that there was a need to facilitate shared understanding and shared vocabulary about features across the Scrum team.
Otherwise there was:
Confusion (What behavior is required for the feature to be considered complete?)
Complexity (When a developer updates the dynamic index code, will this affect the test scenarios for the customer landing page?)
Duplication (Will unit tests cover this behavior or does it require manual testing/UAT?)”
18. http://dannorth.net/introducing-bdd/ ( INTRODUCING BDD )
Dan is the originator of Behaviour-Driven Development, an agile approach to software development that encourages teams to deliver the software that matters by emphasising the interactions between stakeholders.
“BDD has evolved out of established agile practices and is designed to make them more accessible and effective for teams new to agile software delivery. Over time, BDD has grown to encompass the wider picture of agile analysis and automated acceptance testing.”
19. https://cucumber.io/blog/2015/03/27/aslaks-view-of-bdd ( Founder of Cucumber )
“Behaviour-Driven Development (BDD) is a set of principles and practices that enables software teams to produce more valuable software with fewer bugs. BDD can lead to reduced development and maintenance costs and shorter time to market.”
“BDD practitioners explore, discover, define, then drive out the desired behaviour of software using conversations, concrete examples and automated tests.”
Deliberate Discovery This is about learning early to fail less.
In my view of BDD, Discovery Workshops is an essential practice, and involving non-technical stakeholders as well as developers and testers is essential to make it efficient.
Write a failing test. Watch it fail. Write enough code to make it pass. Clean up your mess (refactor)
20. Suggestions regarding using different languages and cucumber? (cucumber.js, cucumber-jvm, unit-test) ( Google group discussion on Cukes )
22. A Day ( or a sprint ) in the life of a BDD Team ( MUST Read article from John Smart )
– In this article, we walk through a typical BDD process.
– While every BDD team is different, and mature teams adapt and refine their
process to suit their specific needs and context,
– the steps outlined here give an idea of the general approach followed by many
of these teams
23. The Five Stages of BDD ( and Agile ) Adoption (MUST Read article from John Smart )
– A typical BDD adoption process (and indeed, Agile adoption in general) goes through five quite distinct stages, or levels of maturity:
24. Test Pyramid in real world scenario – how much is it applicable https://www.linkedin.com/pulse/test-pyramid-heresy-john-ferguson-smart
A Test Pyramid Heresy
26. 10 Minutes With Dan North ( Agile Revolution Podcast )
31. Think Code AB ( BDD, Cucumber ) Amazing Blog by Thomas Sundberg
32. Kick start & scale BDD across your organization ( pdf book ) HIPTEST
https://www.youtube.com/watch?v=FdiKPtGz76U&index=70 ( Devoxx Poland )
https://www.slideshare.net/wakaleo ( Presentations from John Ferguson Smart )
35. BDD in Banking ( Simon Powers the founder of Adventures with Agile )
- bridge communication gap between IT and Business - Test First to improve code quality - How designers are benefited from BDD too - Agile lean coach , work together in more collaborative way - enablers technical enablers, BDD is for agile to scale, automated testing, good leadership, small batches & fast turn around - defect rate : amount of time s/w dev spending on s/w fixing things which team has thought is done deal. bugs fixes because of misunderstanding - defect rate is increasing 35% fixing things each sprint - scrum team how much work needs to be done, velocity, forecasting - priotity bugs in between and breaks estimation, quality decreases, velocity breaks - customer not happy because of bugs and slow new features - automatation building unit tests coverage 4% before - no way of measuring quality of code - code matrix, where bugs coming from which area of s/w - buy in from business/ product owner for automation for framework + automation - need something to do to reduce defect rate increase - reduce unpredictability - 65% adding value ( writing new code ) - writing tests changes the way you write code ( science of testing , art of testing ) - defect rate in new code almost come to 0 - 6-9 months 35% defect rate to 4% time fixing defects - 25% of time to write tests have reduced as well - Quality is shared responsibility - definition of done
4. Projects To Refer
5. Misunderstandings and True Understandings about BDD , Cucumber
1. Cucumber is NOT a testing framework
2. Cucumber is Specification Tool.
3. Cucumber uses Gherkin ( the language ) to describe software specifications
in a natural language
4. Cucumber is smart version of Use Case Template / test cases / test plan which
connects system documentation and documented system ; to ensure
documentation is in line with the system it describes
( To Be Read / Reviewed / Referenced )
https://skillsmatter.com/skillscasts/8633-bdd-is-not-about-testing ( BDD is not about Testing – Twitter Account Login ) – Tech Talk
– Tech Talks
Agile Testing & BDD eXchange 2016
https://skillsmatter.com/skillscasts/9140-bdd-in-action-a-before-and-after-case-study – Tech Talk ( Case Study By BA )
Keywords : epics , user stories , shared understanding from shared documentation/stories,
scope creep, not clear what exactly needs to be done, continuous delivery, common language ( given, when , then), acceptance criteria, hunt the value, impact mapping, MVP, Why & What , user story mapping ( activities, user tasks, stories ), 3 amigos session, discover scenarios,
https://skillsmatter.com/skillscasts/9102-death-by-user-stories – Tech Talk ( Jenny Martin )
Keywords: Acceptance Criteria of usre stories , safe path to value with oopsy technique , scope creep, concrete examples with real world data what’s good about user story, conversation to discover value , comic user stories, deliver in smaller chunks which gives values , story mapping , scenarios, outcome, concrete examples , tasks , stories , no stories in JIRA , chunking down of outcome , agile testing , OOPSI ( outcomes, outputs, process, scenarios, inputs )
OOPSI technique is NOT good, Cucumber’s Example Mapping is MUCH easier. Before discussing OOPSI technique, other talk is useful.
http://blog.picklespro.com/2016/03/08/what-is-a-living-documentation.html#comment-2980858525 ( Pickles Pro Blog – What is living documentation ? )
https://soundcloud.com/cucumber-podcast/illustrating-scenarios?mc_cid=a911f060a0&mc_eid=9ad69390b9 ( podcast – Illustrating Scenarios )
https://soundcloud.com/cucumber-podcast/john-smart-bdd-and-serenity ( BDD and Serenity )
run cucumber in browser ( works only in Chrome Browser )
https://skillsmatter.com/skillscasts/7361-keynote-kind-of-green – CukeUp! 2016 talk by Aslak Halessoy “Kind of Green” Basics of BDD
https://cucumber.io/blog/2016/05/09/cucumber-antipatterns?mc_cid=0637742a25&mc_eid=9ad69390b9 ( Cucumber Anti-Patterns )
http://qualitytesting.tumblr.com/post/145019235039/bdd-with-zelda ( Gherkin – thin line between Behavior and Implementation Details )
https://tobythetesterblog.wordpress.com/2016/05/25/how-to-do-example-mapping/ ( Real World Scenario )
https://www.linkedin.com/pulse/bdd-next-level-checklist-10-things-can-bring-your-web-jim-lynch ( The BDD Next Level Checklist – From Front End Developer )
http://tooky.co.uk/the-cucumber-test-trap/ ( ….Writing scenarios with your customers will help you to understand what your application needs to do, and automating those scenarios with cucumber will help you to know when the application meets those needs. Just don’t fall into the trap of thinking you can use cucumber to test the app completely at the expense of unit tests or Lots of Scenarios and a Big Ball of Mud will be your reward…. )
Cucumber & Test Pyramid Feasibility
google group topics to refer:
1. Suggestions regarding using different languages and cucumber? (cucumber.js, cucumber-jvm, unit-test)
2. Unit Tests vs. Cucumber
3. What to test with Cucumber: front-end, back-end or both?
http://claysnow.co.uk/the-testing-iceberg ( Testing Iceberg , Not all Business-readable tests need to be End-to-end and not all End-to-end tests need to be business readable. )
http://claysnow.co.uk/building-trust-in-your-automated-tests/ ( Building Trust Using BDD , … The upshot of this is that they express all their test scripts in an end-2-end format – manual scripts of how a user interacts with the system. This gives rise to a lot of duplication… )
http://codesqueeze.com/the-easy-way-to-writing-good-user-stories/ ( User Stories -> Acceptance Stories )
http://stackoverflow.com/questions/36603728/structuring-cucumber-to-support-testing-both-services-and-ui-layers-in-the-testi ( Cucumber + Test Pyramid Query )
http://testerstories.com/category/bdd/ ( Stories From A Software Tester Blog – BDD )
http://blog.czeczotka.com/2014/08/17/writing-cucumber-jvm-step-definitions/ ( Writing Cucumber JVM step definitions )
https://www.coveros.com/regular-expressions-a-setup-for-cucumber-glue-code/ ( Regular Expressions – A Setup for Cucumber Glue Code )
http://www.agileforall.com/2011/08/cucumber-regular-expressions-cheat-sheet/ ( Cucumber Regular Expression Cheat Sheet )
https://github.com/rlogiacco/Natural/wiki ( Eclipse Plugin for Cucumber – Natural )
http://mherman.org/blog/2015/09/10/testing-node-js-with-mocha-and-chai/#.V5iH3o6Xqns ( Testing Node.js with Mocha & Chai )
( email conversation with one of the Cucumber users )
I have started using Cucumber mainly with Selenium to test last layer of test pyramid.
Can you please share more details about Cucumber usage in Integration and Unit Level ?
[Answer] There’s really not much more detail to share when it comes to using Cucumber for Integration or Unit testing in addition to Functional testing. The tests would not be written any differently and all of the same Cucumber principles apply. All of the difference happens below the Gherkin layer where your code might be working directly with classes and objects in memory rather than driving a GUI.
Any pointers to open source github repo or blog will be helpful ?
[Answer] Here’s one with a lot of Cucumber tests. I’d say that most of them are integration tests because most of them are testing internal helper classes and only a couple features are functional level ‘end to end’ tests. The unit testing in that repo is still done in RSpec rather than Cucumber because there’s nothing about the unit tests that is worth communicating at a high level.
Can we decide during 3 amigos session , if particular scenario can be automated at which level of test pyramid ?
Per me this will help to show management about progress during sprint.
[Answer] Sure, you can test the desired behavior at any level of the testing pyramid. Which level your team decides upon will likely be influenced by the level of confidence that testing at that level will sufficiently ‘prove’ that the behavior is correct. Some teams push most of their testing to the lower levels and trust that things are wired together correctly and some teams insist on testing everything at the functional level because that’s the strongest guarantee that things are working.
Living Documentation ( Why , How , How Exactly )
It starts at the outside by identifying business outcomes, and then drills down into the feature set that will achieve those outcomes.
Each feature is captured as a “story”, which defines the scope of the feature along with its acceptance criteria.
we need a way to describe the requirement such that everyone – the business folks, the analyst, the developer and the tester – have a common understanding of the scope of the work. From this they can agree a common definition of “done”, and we escape the dual gumption traps of “that’s not what I asked for” or “I forgot to tell you about this other thing”.
Story. It has to be a description of a requirement and its business benefit, and a set of criteria by which we all agree that it is “done”.
The structure of a story
BDD provides a structure for a story
your story should contain all of the elements described in the template. The story template looks like this
Title (one line describing the story)
As a [role]
I want [feature]
So that [benefit]
Acceptance Criteria: (presented as Scenarios)
Scenario 1: Title
And [some more context]…
And [another outcome]…
Scenario 2: …
The title should describe an activity
The story title should always describe actual behaviour by a user of the system.
The narrative should include a role, a feature and a benefit
The scenario title should say what’s different
The scenario should be described in terms of Givens, Events and Outcomes
“Simply by getting the business users, the analysts, the testers and the developers to adopt this vocabulary of “given/when/then”, they discover that a world of ambiguity falls away.”
The givens should define all of, and no more than, the required context
The event should describe the feature
The story should be small enough to fit in an iteration
So how is this different from Use Cases?
event , feature , requirement , story , scenario , context , title (one line describing the story), acceptance criteria , activity
Story: Account Holder withdraws cash
As an Account Holder
I want to withdraw cash from an ATM
So that I can get money when the bank is closed
Scenario 1: Account has sufficient funds
Given the account balance is \$100
And the card is valid
And the machine contains enough money
When the Account Holder requests \$20
Then the ATM should dispense \$20
And the account balance should be \$80
And the card should be returned
Story: Returns go to stock
In order to keep track of stock
As a store owner
I want to add items back to stock when they’re returned
Scenario 1: Refunded items should be returned to stock
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock
Scenario 2: Replaced items should be returned to stock
Given that a customer buys a blue garment
And I have two blue garments in stock
And three black garments in stock.
When he returns the garment for a replacement in black,
Then I should have three blue garments in stock
And two black garments in stock
?? QUERIES ??
- how to map existing test cases ( +ve , -ve ) to feature file ?
- can each of the features file should have single feature or multiple feature ?
- can I add all the +ve and -ve scenarios for a feature in single file or should I separate ?
- which all parts of testing pyramid Cucumber touches and how ?
- Does feature files is alternatives to test cases management tools ?
- Amazon ordering flow , each page 1 feature file ?
- examples of good feature and scenario vs bad feature & scenario
- Can scenario have detailed user actions ? e.g. click A then B then do C to get D
If not how BA / PM certify that story doesn’t have bad UX or doesn’t match with her/his expectations ?
hiptest tool for BDD
https://www.relishapp.com/cucumber/cucumber/docs/cli/tag-logic ( tag , tagging , multi tags )
http://eloraparija.com/category/bdd/ ( BDD , Cucumber )
https://www.coveros.com/how-to-get-the-most-out-of-cucumber-tags/ ( cucumber tags )
Amazing writeup from Andrew Premeds about his 2 decade experience in s/w industry in email@example.com
[GENERAL] State of the Cucumber/BDD Nation – Long Post Alert
http://www.thinkcode.se/blog/2016/03/26/where-should-you-use-behaviour-driven-development-bdd ( Where should you use Behavior Driven Development , BDD ? )
http://www.mwtestconsultancy.co.uk/category/bdd/ ( Is BDD Testing ? Series of Articles – Must Read )
https://testzius.wordpress.com/2016/12/19/is-bdd-testing-yes/ ( is BDD Testing – YES )
( … taking a look at the cuke_sniffer gem. It’s a neat gem that searches your cucumber project for smells and gives you not only a score but a detailed list of the smells and exactly where they are )
– Agile 101 General Learnings ( What is Agile ? Brushing up your agile knowledge )
7. Gherkins Syntax Best Practices
https://github.com/strongqa/howitzer/wiki/Cucumber-Best-Practices ( Good Practices )
http://docs.behat.org/en/v2.5/guides/1.gherkin.html ( Writing features – gherkin language )