Often when teams are starting out with behavior driven development (BDD), they fail to take the first step that will really set them up for success – ensure that the feature they’ve added really are/is adding value to their customers. Or, as Laurent Py, Product Manager and Co-Founder of HipTest, puts it in my TestTalks interview with him, “When you want to test, the first thing you should test is the value before the correctness.”
How to Test a Feature for Value
It’s easy to test the value of a scenario – ask your users. I see many teams miss this obvious first step.
On of the BDD secrets is that Laurent likes to share the road map and backlog of HipTest with the community. For him, it's a great way to start testing the traction of their future features. It’s one quick way to gauge whether the feature is something that users even want.
For example, testers can subscribe to the HipTest board and vote for the feature they want. If they have a feature that has, say, 10 or 15 votes, then maybe it's the beginning of something that users really are interested in.
Another way is to ensure that your product managers are investigating more into their stories. Encourage them to start writing the BDD scenarios and make sure they see it as a high priority.
There are many ways to test the value and traction for a feature. Finding out what your customers really want before investing in the correctness of the feature is critical to your project success, because having a feature that no one uses, even it is “correct” in production, isn't really useful.
Teams Struggling with Managing BDD
Another problem Laurent sees is teams having issues with managing their BDD testing efforts. Many teams struggle with Cucumber and end up using tools like Excel to keep track of their tests and manage their use cases.
This is really inefficient, and is one of the reasons why his team created HipTest. The idea was to create a simple platform in the Cloud that supports real-time collaboration between product owners, testers and developers.
HipTest allows teams to capture the discussions around their BDD scripts using the Gherkin Syntax Given-When-Then, or to write traditional manual scripts. Once in the system, it also enables them to manage the life cycle and test execution, whether it’s manual or automated execution. It’s an “all in one” solution.
How Does HipTest Work?
HipTest has two different workflows. The first is that if a team already has existing BDD tests written in Cucumber, JBehave or other BDD framework, a user would then import them into HipTest. Rather than having developer-centric IDEs, HipTest makes it easier for non-tech people — product owners, testers, business users, etc. — to jump in and contribute by writing, modifying and executing the scenarios.
The second workflow is once you have your acceptance test defined in HipTest, you can use the open-sourced HipTest publisher, which enables you to create scripts from HipTest for various test automation frameworks like Cucumber, Ruby RSpec, Jasmine — even JUnit — if you don’t do BDD at all. It’s really flexible.
G/W/T is About Collaboration
It’s a good sign when a company actually uses their own tools. Laurent confirms that not only do they recommend BDD to their customers and community, but they use it themselves to test HipTest. What’s pretty cool about this approach is that it provides the team with a methodology to capture the behavior of the stories before investing time on their development.
This is really the heart of any true BDD effort: collaboration — NOT automation. For some reason this is still a BDD secret but it is one everyone involved in BDD should be aware of.
Not only should the developer be able to write the scenarios, (which is usually what happens), but product managers should also. Being a product manager himself, Laurent likes to write the scenarios to describe and define the boundaries of a story before his team begins the development.
It should be the same with your business users. They should be writing and contributing to your BDD scenarios as well.
How to Avoid Bad BDD Practices
I’ll admit that I know some teams that shy away from to many folks contributing to the BDD effort because they fear the scenario will grow out of control with hard-to-refactor duplication — which is common when you have many people with different skills contributing to the writing of your scenarios.
HipTest helps avoid duplication in a number of ways. For example, it has an Auto Complete feature, which helps your team to reuse existing steps that have been already potentially automated by developers. You’ll also avoid the creation of steps that might have the same intent but may have been written in a different way.
Another feature that's really important is the re-factoring. Testing should be continuous, as should your development process. At HipTest, they deploy continuously to production. Because they do between 5 to 10 deployments a day, they have to test continuously, which means they have not only automated all their tests, but when they have a new story it can potentially impact 50% of their existing scenarios.
This is a nightmare scenario for most BDD projects I’ve seen. Most teams don’t have the time to manually modify hundreds of scenarios – which is where the HipTest platform comes in; because of its ease of use, it can help give your non-technical users (product owner, business users, etc.) the same kind of refactoring capabilities the developers would have in their own IDEs – except for the fact that they don’t have to know code, and it’s pretty easy to use.
Discover More HipTest Awesomeness
For more BDD secrets on how to manage your testing efforts better check out my full Test Talks interview with Laurent HipTest: Test Management for Agile Teams with Laurent Py
| Joe: | Hey Laurent, welcome to Test Talks!
|
| Laurent: | Hi, Joey thanks for inviting me. That's a pleasure.
|
| Joe: | It's awesome to have you on the show, but before we get into it can you just tell us a little bit more about yourself?
|
| Laurent: | Sure, I'm Laurent Py and my nice accent comes from France, so currently I'm leading the product management at HipTest for 2 years now. Prior to this, I founded a company named Smart Testing that has enough framing model based testing. We generate test from business purpose models.
|
| Basically I'm in the testing industry for 23 years now.
|
|
| Joe: | Awesome, today I'd like to go over your thoughts about test management with agile teams using HipTest. We made a star [inaudible 00:46] this year and I was really excited to learn about HipTest. I never knew about it, but when I saw it I was kind of blown away that how I think this is a really cool product.
|
| I guess at a high level for people that don't know about HipTest, could you just tell us what is HipTest?
|
|
| Laurent: | HipTest is a new breed of test management tool and BDD Framework. Basically as you know if you're driven development as a great momentum, is getting traction. Abilities that many teams have different needs, move people needs in fact. Some of them are doing manual testing, exploratory testing. Also, test automation.
|
| People are struggling with Cucumber testing Excel, trying to manage all these use cases. That's the reason why we come up with HipTest and the idea to have a single platform. A simple platform in the cloud that support real time collaboration between product owner, testers, developers. Enable them to capture the discussions around theories to create BDD scripts. Using the Gherkin Syntax Given-When-Then, or write manual traditional scripts. Then you test will also enable them to manage the life cycle, the execution life cycle, so manual execution or automated execution. It keeps kind of one.
|
|
| Joe: | Awesome, so I'm just … Because it's audio it's not visual I know you guys have a bunch of webinars and I'll have that in our show notes. Can you walk us through at a high level how soon can create … How they will use HipTest for BDD test, or BDD project what the work flow is like?
|
| Laurent: | Sure, so there are 2 different work load. The first one you hear the existing BDD test written in Cucumber or Jbehave or whatever you use today. You can import them into a test, and as I said this is a simple platform available in the cloud so that's easier for non tech people, people like product owners, testers, business user to jump in and contribute to the executions.
To write in scenarios, to modify BDD scenarios. Once you are let's say, the scenario's the acceptance test you have defined you can use HipTest to be sure. Which is an open source application that we have developed, that enables you to create scripts from HipTest for various test automation from Word. Basically, you can create scripts for Cucumber, GBA, [inaudible 03:45] Jasmine or Java GUID if you don't do BDD at all. That's really flexible.
|
| The idea behind is that you have described the behavior of histories through different scenarios, and the same scenario can be exported in Java, in [Rugby 04:06], in Cucumber so that's really up to you.
|
|
| Joe: | Great. What I really love about it is when you're in there, if you already have a project up in the cloud set up. If someone's creating a new feature and they stop writing a Given-When-Then, it has auto complete for you. If there's a statement that already exists that's similar to what you want, it kind of helps you so you're not creating duplication. What I've seen for myself working on BDD projects, is a big problem. We have 8 to 10 Sprint teams all across the world and they a lot of times have a lot of duplication because, for whatever reason they don't look to know hey, this given statement's already used elsewhere.
|
| With this IDE, you type it in and says, “Hey, here's a similar statement. Is this what you were thinking of?” I think it's a big time saver that way.
|
|
| Laurent: | Absolutely, in fact what we really like with BDD is not only is this is what we recommend to a customer, and community, but we use those BDD and E test to test them. What I like with this approach is that it empowers the team which a methodology to capture the behavior of the stories before investing time on the development.
|
| You capture this behavior using Given-When-Then and as I said that's all about collaboration. Not only the developer should be able to write the scenarios, which is usually what happens. But also, I'm the product manager, so I like to write the scenarios to describe and define the boundaries of a story before we start the development.
|
|
| Same thing with the business user. All these people write and contribute to these BDD scenarios. When you have not just one people, but many of them with different skills contributing to writing the scenarios. Then you have a quickly a problem with the duplication. This is what you mentioned and we have clearly identified this problem and wanted to address it with basically 2 different features.
|
|
| The first one you did mention is auto complete. Basically you start writing your scenario. I don't know I look into the platform and as you writing this step, you got this auto complete suggestions so you can reuse existing steps that have been already potentially automated by developers. You avoid creation of steps that's have the same intent that are written in different ways.
|
|
| The second feature that's really important is the re-factoring. This is part of the feelers that we have of testing. Testing should be continuous like you development process. We HipTest we are develops team. We deploy continuously in production. We do between 5 to 10 deployments a day so we have to test continuously. That means we have not only to automate all the test, but sometimes when you have a new story you impact 50% of the existing scenarios.
|
|
| You need this kind of re-factoring features because you don't have time just to modify hundreds of scenarios manually. Yes, the HipTest platform will basically help our non technical users like the product owner, or the business users with the same kind of capabilities that the developer would have if it's own ID.
|
|
| Expect that it's not code, and it's pretty easy to use. Auto complete and re-factoring are definately very important if you want a scale new approach.
|
|
| Joe: | Awesome, great points. I think people that work on projects know even the smallest barriered entry will discourage people for contributing. I've seen like I said, so many of the Sprint teams working on the same project. To have a product owner open up an IDE or even look in your version control is a barrier.
|
| Laurent: | Absolutely.
|
| Joe: | Passing around text files isn't really gonna work. Or passing around emails for Given-When-Then's isn't gonna work. It almost sounds like, “Here's a URL, log in there it is, do your thing,” seems like a great solution for that.
|
| Laurent: | That's interesting because you know we have the 2 different adaption use case. We some of the teams start with the testers, so they have identified the test as a test management tool integrated with Jira which is one of the main reason why people start trial with HipTest. Then, we see the developers comes in and start with the automation. A single adaption use case is the developers, that were using Cucumber or BID 4 for example, and they are struggling to involve non technical people.
|
| They're struggling to involve business users, product owners, or even QA to contribute the project. They see HipTest as a great platform to this collaboration. They still continue to use Cucumber as a test automation frame work, but the place where the team will collaborate, discuss the futures, capture the behavior is HipTest.
|
|
| We see 2 different approaches.
|
|
| Joe: | I just want to focus in a little bit more on re-factoring. I had a use case recently where as a radiologist, I would log into Rad Desktop which was the name of the application, and then they renamed the application. So we had a hundred feature files and they all used that term Rad Desktop. We had to go into every single scenario and change the name to something more generic.
|
| Maybe that was we didn't follow a good practice when we created that from the beginning. Nonetheless, we had to re-factor it and it took us a long time. How does HipTest help us with that type of refragment. How does it handle that?
|
|
| Laurent: | Typically when you reuse an existing step using the auto complete feature, you will see that this step is being used by 100 scenarios already. Then you can click and then you get to the steps, see all the scenarios that use it. Because you have a single point of maintenance, you can for example, just rename the step which is what happened in the case you mentioned just before. All the scenarios would be automatically impacted.
|
| The first rule of re-factoring it's pretty simple to do but that's very common one and someone that we use extensively because as you know, BDD is all about leaving documentation and specification. You need to make sure that your steps remain, the name remain meaningful.
|
|
| You have continuously to change the name for example, the name of the application as you mentioned or other steps. The second kind of re-factoring that we see is you have a one step that say I introduced the card. I don't know Mastercard, and another one that say I introduced the card AMEX.
|
|
| At some point, so that's basically 2 different steps, but at some point we have a third type of card and you will say, “okay, maybe I should merge these 3 steps and just add as new perimeter.” These are more complex re-factoring techniques. They always … So that's possible with HipTest and that will enable you to better factorization of the steps.
|
|
| It's not only helped the maintenance, but then also to accelerate the automation because when you have to automate as a developer the steps. So you have to implement the test fixture, you won't have 3 steps to implement, but just one. You will have the right design at the scenario level.
|
|
| It doesn't make this way the implementation of the test fixture too complex. You can have this kind of factorization at the test fixture level, but that makes it much easier to create and maintain.
|
|
| Joe: | Awesome, and I like this point about making the Given-When-Then's meaningful. A lot of times if you work with lot of Sprint teams, one person may use one terminology, and other one uses another. Especially if you have a lot of turnover. A lot of times if they're stuck in the IDE, they're afraid to make changes that will impact what they call, “Other peoples features,” so they will make weird statements at a slightly different than what is already existing.
|
| This helps that type of fear, because it will catch all other Given-When-Then's that have the same type of terminology. Make the update for you so you know that you're getting coverage without having to search through every single feature file to make sure you didn't break anything else.
|
|
| Laurent: | Yeah, absolutely. We have wonderful … That's a great customer case with [Parkem 13:46] so that developed a huge ticketing system. Represent more than 109 years of development. It's a pretty huge system, and they decided to adapt the BDD approach and use HipTest.
|
| One of the main reason was after 2 years of development, they varies maintenance per year, so they will have to transfer the application to another team and the knowledge of the system is the BDD or the BDD scenarios.
|
|
| The ability as you said, to have a unique domain specific language and you build this domain specific language not right from the first time, so that's why you need to rename the steps. You need to do some kind of re-factoring cause that's just life.
|
|
| Like coding, there's no difference between BDD scenarios. You use scripts and the code. So yeah, they use HipTest and the main reason was build the right knowledge base that they could transfer easily for maintenance. Once again, it was a huge system so this is a very important point.
|
|
| Joe: | Awesome, I know a lot of our Sprint teams, they don't build in re-factoring into their Sprints, so they end up saying,”We didn't re-factor, cause we didn't plan for that.” It sounds like once again, using a tool like this will help them so the velocity won't be impacted yet they still get the benefit of re-factoring by just using this particular tool.
|
| So Laurent, how does this work out? It sounds like it creates a Given-When-Then for you, but how about the logic behind that when you actually have to write the code. How does that work with HipTest?
|
|
| Laurent: | You mean when you have to implement the steps?
|
| Joe: | Exactly.
|
| Laurent: | Information.
|
| Joe: | Yes.
|
| Laurent: | So the using HipTest to be sure, you will be able to create scripts for different various test automation framework. So HipTest does not force you to use a particular framework. As I said, you can use Java you need for the Cucumber. Whatever the team is familiar with, and this is a great point because developers don't want to be forced using a particular language of framework.
|
| For example, visual basic, I don't want to name any specific tool. They like to work with the same kind of technology that use for the unique test for example. What HipTest can do is that it will extract the scenarios from HipTest, and then create scripts.
|
|
| There are 2 main files that are created. The first one which is a file that contains all the feature files so it's executable. Using every steps as a functions. Then you have a second file that contains all the step definitions. We create the skeleton of all these step definition. What the developer has to do implement is the behavior of each of the test steps.
|
|
| If you want to do test automation at the level you would use, for example, telling him what driver. It can also implement this test steps pulling a rest API. It really depends on the level of automation that you want to have as developer. That's really flexible.
|
|
| The life cycle is really different between the 2 files. The feature file is generated each time you execute the test. Each time there is all this is triggered by the continuous diversion. This is another point that I will describe this after.
|
|
| Each time you have a new build, the CI will extract using HipTest to be sure the last definition of the scenario the feature file from HipTest will take the test steps so all the implementation of the test fixture from the [inaudible 18:15], execute the test and push the result back into HipTest.
|
|
| One file is generated every time. The other one the place where you implement the test fixture, this is definitely code so the work flow and the place for this file to be is definitely the code [inaudible 18:35] can indicate to whatever you use for self control.
|
|
| This is code. I mean one of the main point that we have, main point of discussion we have when it comes to automate test our community is the job of the developer. The guys decoded the application is the right person to work on the automation. It will be very difficult to build a test automation frame work that scale if you don't have any development experiences in the team.
|
|
| Joe: | That's a great point and for some reason I've been hammering this for years, and still some people just don't believe that. I think in order to be successful with automation, especially if you move to more DebOps, more agile shift left, shift right, the developers need to do automation for you to be successful.
|
| Laurent: | Absolutely.
|
| Joe: | Another point you brought up which I thought was a really good point is we have a frame work. We use Java, and we use Jbehave. So we have and because of our the out put that we use for our compliance, we need to use one framework. We do have Java Script front end developers and like, “I don't know Java. It would be so much easier if I can use Jasmine.” We're like, “Nope, you'll make it too complicated. We cannot have feature files all over the place and different IDEs.” This almost sounds like it will solve that. Because you can have your feature files all in one place, doesn't matter what technology you're using. How it's implemented, it doesn't really matter, is that correct?
|
| Laurent: | Yes absolutely. When you think about the fillers that are driven development. This is all about having discussions between developers and people that have the business knowledge. The QA team, the business users, the product owner, and when you have these kind of discussion, the technology doesn't matter. It doesn't matter if you use Java, or Ruby or whatever the language is for the implementation.
|
| The frame work that you use, or the tool that you use to write your scenarios and capture this behavior should not force you to use a particular frame work to automate the test. That's why we have this vision of HipTest as a platform where all the people collaborate and the definition of done. Then HipTest to be sure as an open source application to extract the scenarios from HipTest and create scripts for all of the different test automation frame work that exist in the market.
|
|
| Today we do support something like 20 frame works, but like I said it is open source. The application is based on a templating system, so it's pretty easy for you Joey to customize this and if you use just an [inaudible 21:49] and custom framework to customize HipTest to be sure, to create the scripts for your frame work.
|
|
| Joe: | Awesome. Another piece that you mentioned is you can run tests and also those reporting in metric. Once again what we're doing is I have a hacky Python script that runs at the end of CI that goes through some XML files and creates some statics. What kind of reports and metrics can you get out of this tool if you use HipTest?
|
| Laurent: | Basically you can have metrics for every test frames and community metrics by test frames. Basically for one given test frames, you can for example one that would execute the test for your Java Jasmine application. You would be able to see the breakdown by status which test is pass, which test is fail. You can see the trace when the test is fail.
|
| It works with automated execution and as I said, we do also support the manual work flow so you can basically execute the same scenario that you have previously defined HipTest. You can execute it step by step assigning to a user.
|
|
| Every creational feature you have with test management tool, you basically have it with HipTest. You will have this viewing types of break down. You have this view for every test frame and you also have a accumulated view for all you test frames. You will see that for recreation test fields, and you reduce 2 fail test for the execution manual execution for the Sprint. You have 5 failed test and all together, you have 7 failed test.
|
|
| Then there is also something very important around execution time and estimation. This is brand new feature we are working on to enable team. There is some work to spend time when it comes to execution, so once again it can be manual execution or automated execution. You don't know which test how long does it take to execute this test. How long that it takes to execute all the companies.
|
|
| The same time it is also … Especially when you do develops, it's something really important. At some point you don't want to take control of powers to execute before the employee introduction. That gives you a view of where you spend time and maybe what you can maximize.
|
|
| Joe: | Awesome, so does it keep track of historical trends over time. This feature file is flaky, this pass, pass, fail, fail, fail, pass, pass?
|
| Laurent: | Yes, we definitely we have this. You can see for every scenarios or every feature file what's the history of execution. You're able to identify the flaky test and maybe investigate and spend time to understand why tests are failing from time to time.
|
| Joe: | Awesome, so I assume you guys must have talked to a lot of companies cause this solves a lot of issues that I'm actually seeing myself. I though it was just my company. For example, even minimal test, what we do is we have a custom tag that is then mapped to an ID that's an ALM, HPs ALM and then we have a custom tool that runs at the end that reads in the custom tag and goes to ALM and does a mapping, and generates a report. It sound like that would be covered using HipTest.
|
| Laurent: | Yes, absolutely. One of the main uses that we do see today in the market is teams that are living ALM or quality center using the atlas ensuite for ALM and then looking for test management tool or BDD frame work for the testing. That's the reason why they come up and use HipTest.
|
| The way we have done this design integration with atlas but we works the same way with Trello is using a real time integration. We don't want duplication.
|
|
| What you mentioned with the tags is very interesting because we see people using these tags and using these tags manually, so what we've done for example with Jira is that you pick up a scenario in HipTest, you drag and drop. You drop your scenario in the generate shoe and automatically that create a links and a tag in the HipTest so that you know this scenario is linked to the particular user history. You can navigate to this history, and there is no duplication.
|
|
| At some point if you modify the name of the scenario or remove the scenario, the impact will be automatically done at the atlas and the Jira level. Avoiding you know duplication of data, synchronization, which is always the mess the simple and real time integration has really improved.
|
|
| Joe: | Awesome, you did mention to this integration and be able to run test within HipTest. Can you go over it real quickly what you meant by that or how easy it is to integrate continuous integration systems with HipTest?
|
| Laurent: | Sure, so it's as simple as 2 common line. If HipTest to be sure, then integrate basically with all the continuous integration tool at the market. It's as simple to use as 2 common lines. The first common line, we learned about you to extract the feature files from HipTest so you will get the last definition of the scenarios from HipTest. Then the CI will run the scenarios and then you have a second command line still using HipTest to be sure that we push the results back into HipTest.
|
| So in HipTest we have in real time of the execution of the last build. As I said, you can mix that with other manual frames so that's pretty flexible.
|
|
| Joe: | Awesome, so I wanted to mentioned that the tool is open source but for a team of 3 and under, it would be free. It's very minimal the amount you charge for teams larger teams on that.
|
| Laurent: | Yes, absolutely. It's less than 10 dollar a user every month. Just to share some figures, we have to demo the 7,000 users so all communities growing pretty fast in more then 100 countries. For me as a product manager, it's also a way to travel when I look at the world map and see where our customers are, it's always impressive.
|
| As I said, that's a use case that's its driven development that's gaining a lot of traction. Also I mentioned before, we see a mix of team using BDD and other techniques. That's why we have the future is not pure given development but more a breed approach between test management and BDD.
|
|
| Joe: | Great point once again, it sounds like you guys really do speak with a lot of companies and really listen to that feedback. That's exactly what I see in my own company.
|
| Laurent: | Yeah one of the thing I used to say, “When you want to test, the first thing you should test is the value before correctness.” What we do at HipTest as product manager, I decided to share the road map and the backlog of HipTest with the community. For me, that's a great way to start testing the traction of the future features. For example, when I see some people can subscribe to the board and vote for them, if you have a feature that has I don't know a 10 or 15 roots maybe it's the beginning of something. I should try as a product manager to investigate more into stores. Start to write the BDD scenarios and make it a high priority. There are many ways to test the value and the fraction for the future. This is what I like to do before investing in the correctness of the feature. Having a feature is correct in production that nobody use, isn't really useful.
|
| Joe: | Absolutely, okay Laurent before we go is there one piece of actual advice you can give someone to improve their BDD testing management efforts? Let us know the best way to find or contact you to learn more about HipTest.
|
| Laurent: | To learn more about HipTest, I suggest you to go to HipTest.net. You can sign up and start your project in seconds. It's pretty easy, you can create a simple project sample project in BDD so let's start with the simple project. Let's start automate the first test. Integrate this first automated test and once you have this in place, that will be a great first achievement. Let's make all it work for one simple scenario and then progressively add more scenarios. That's always a great way to start.
|
