Unit Tests Assertion Coverage

A unit test case is made of three parts - the "given", "when", "then". The "given" is the piece of code that set's up mock objects, data that needs available and dependencies that need to be injected. The "then" will execute the method under test. The "then" verifies the results of the execution of the method under test.

Too often it is observed that the unit tests do not have enough of the "then" statements that verifies the return results. This happens more often in cases where existing code running in production does not have sufficient unit tests available. When new features are built on top of the existing code it is often difficult to write proper unit tests with good verifications and assertions.

The problem described above is also a symptom of a team's focus on code coverage. While code coverage is a good metric to observe and use it only proves that the application code was executed. A unit test case with "given" & "when" and no "then" will also produce a good code coverage metric. This kind of unit test is not a good test because the verifications do not exist.

I recently tried out a solution to this problem with really good success. The solution was to capture the return value of the "then" piece of the unit test case and run it through the jackson object mapper. The result is a JSON representation of the actual return value of the method under test. Save this JSON in one of the files in src/test/resources folder. This makes up the "expected json" of the unit test. In the unit test, now every time the method under test is executed, take the actual return value, again convert it to json and compare it with the "expected json" that is saved in src/test/resources folder.

This approach is also described as Characterization Tests. As described on the wiki:

In computer programming, a characterization test is a means to describe (characterize) the actual behavior of an existing piece of software, and therefore protect existing behavior of legacy code against unintended changes via automated testing. This term was coined by Michael Feathers. ^[1]

This approach allows us to gain the critical "Assertion Coverage" from a test case. Assertion Coverage is the measure of how much of the data produced by the method under test is valid. Contrary to what it sounds like, this approach does not produce any metric like the code coverage metrics, its not something we can see in red or green colors. The benefit is clearly visible though.

Another important benefit of this approach is the "expected json" that was produced serves as a documentation of what a specific method produces. The entire object is converted to JSON and thus we can get a good visual of the data that flows through the code.

In the next post I will elaborate more on this approach with an example.

React over Angular - my take

I took some time to learn React JS in the last few months. I created my little version of todomvc like application, created the same application using React and using Angular. The code lives here:

react-prototype
angular-prototype

Here is what I found. Overall I liked React much more than Angular for the following reasons:

1. My debugging experience was much better. Angular brings JS into HTML and React brought HTML into JS. At first I cringed a little at the thought of having HTML within JS. As I kept building the application though it was much easier to debug issues. 
1. First things first -- there was only one file to look at. Nothing more. All my JS and HTML is within one single file. Unlike Angular JS where I had to look at a directive, an HTML and a controller.
2. JSX caught a lot of errors at compile time. This gave me more confidence. With the angular app I found errors at run time AND after a few hours of that error introduced.
3. Easier to refactor - The above reasons made it easier to refactor the code as well.
2. Componentization came naturally. With this I mean creating web components that can be distributed and reused much more easily. Again - having a single file helped a lot. I will be writing more about componentization in later posts.
3. React definitely had a very small learning curve, much smaller than angular. It was much easier to explain React than Angular to my friends who knew nothing about both.
4. Aligns well to Page Object pattern that is popular in testing circles. Although this is not the best of the reasons - it is worth a mention.
5. The idea behind React makes complete sense. The part that model is the state and HTML is the projection of state and that only part of HTML that has changed is the one that is modified.

So - would I use React on my next project? This is a difficult question to answer. I would certainly advocate for it. A lot depends on the organization / project though. If there are Angular / any other framework experts out there and things are done well - there is no need to rock the boat.

Fuzzy Avenger

My latest new project on Github is Fuzzy Avenger.
https://github.com/daveayan/fuzzy-avenger

Why was this created?

This project started off as a learning exercise. This weekend I wanted to learn Akka with Java. In the process I created this github project to capture my learnings. The folders one - five in here is where that learning was captured. As I did that I realized a simple utility can be created that can be used with any traditional java code - specially unit tests or just for curious minds. This utility will process a list, apply a long running function (like making service calls) to each element in the list, capture results, aggregate them and will return the results. The results will be returned in the order - one for each element in the input list. Thus, this utility was created.

Why would I use this?

I would recommend not to use this in production code, yet.

Use this utility in your unit tests to exercise parallel processing / concurrency. For curious minds - to try out some instrumentation on your services For learning To provide me some feedback, if this is interesting, helpful and what updates can be made

Everyday Math from McGraw Hill Education

These days I am working on the version 4 of the Everyday Math program with McGraw Hill Education. The marketing team here prepared this video about the product we are building these days. The product is Everyday Math 4 that is targeted to students and teachers for grades K-6 and this will be launching in June for the 2014-15 school years.

The video features the authors of the EM, product sponsors, UX designers and academic designers from the Chicago office and showcases the product we are building.

https://www.youtube.com/watch?v=asJhmigWHvs

More information on Everyday Math program: 

http://everydaymath.uchicago.edu/

Presentation at Quest 2013

Terry and I were at the Quest Conference 2013 in Chicago this week. We met several professional and people in the field of QA and also did our presentation.

The slides are here:
https://docs.google.com/file/d/0ByNrq5n2aTScN0UtbS14X1pJVVU/edit?pli=1

Technical Skills for BA's and QA's

What kind of technical skills do BA's and QA's need? Terry and I had a good discussion on this quite a few times in the last year. Her recent email to the group soliciting feedback and ideas sparked a really good email conversation. Here is my take on this topic.

I think the technical skills needed by BA's and QA's depend a lot on Intranet / Internet system as well. With an internet system used by users all over the country / world BA's and QA's would need more technical skills around what are the minimum system requirements. Do we support multiple browsers / devices, what are the compatibility criteria and options. What are the trends in browser usage? What new technologies are coming in marketplace that may need to be accounted for - example, new version of Android or ipad mini :) If there is a new version of TinyMCE editor available do we upgrade? What does it buy us? etc

In an intranet system we may have a much more predictable user base. The technology standards are governed by a company wide "standards board" of some kind. This takes away the need BA's and QA's to know some of these technical items. In an Internet system where the user base is sampled and the technology terrain keeps changing much faster BA's and QA's need more technical skills.

To use Cucumber or Not to

Cucumber is fun to use. It it really cool to see plain english statements come to life and different pieces of code executed when those are english lines are executed. Gherkin syntax is great too. It has so much potential. Using a handful of simple keywords you can express any scenario and combined with cucumber you can execute them. Wonderful !

Then my developer brain came into action with all the principles around DRY (Don't repeat yourself) and WET (Write Everything Twice) highlighted. Does DRY apply to gherkins? It probably does. Take these two steps in different feature files for example:

And User A can view the message posted by User B

And User can view the message posted by the other user

These are two different lines of english meaning the same thing. These are the kind of gherkins different people would write about the same step or the scenario. If we already have the step #1 implemented we may apply the DRY principle and not implement step #2 once again. Now, how do I know if these two steps match and have the same implementation? By collaboration and by looking into the implemented step - right? Well the problem is - this takes time. Specially when the test suite is large this takes even more time. Now let's say all this collaboration and code inspection is done and it is concluded that the steps are the same we may want to edit the gherkin and replace the step #2 with step #1.

I call this whole process - "Normalization of Gherkins".

When is this done? When the step is being implemented
Who does that? Well, since this is a collaborative effort - everyone is involved.
If the developer who is implementing this does not have any idea that the step #1 above is already implemented in some other stepdef file the new (re)implementation of step #2 will be done.

So what happened here?
We are thrilled by the idea of automating any simple line of english that can be written by anyone but we still want reusability. So here comes a choice -

- Do you have people outside of your team who care about gherkins? If so, then evaluate using cucumber.
- Do you NOT have people outside of your team who care about gherkins? If so, then consider not using cucumber.

Let me get my message straight here. I am saying this about cucumber - to evaluate using it or not using it. However I think there is a lot of value in gherkins. What I am saying is evaluate the use of cucumber.

So - If you choose not to use cucumber how would you automate your gherkins then? Use unit testing frameworks to hit webdriver or something! I guess at this point I will leave this post here leaving a few questions unanswered for the next few posts. These questions are -
- How would simple junits consume gherkins?
- Could I get the same kind of report from junits the way I get it from cucumber?

Stay tuned for some thoughts around these.

What do you like about it?

My new interview pattern these days is turning out to be very interesting. The simple question is - "What do you like about it?". Tell me a technology / process and my question is simple - "Why do you like it?"

The important part here is "Why". You may have used a technology for many years and you are now an expert at "using" it. However in the process you should have developed opinions, hatred and / or love for the technology. You should be able to see the good and bad parts of the technology. You should be able to critique and defend the technology. This is very important as this shows that your thought process has evolved.

It is surprising to see how many candidates don't have a good answer to this. These are still good candidates who can get the work done but it does give an impression about someone.

Thought Leadership

I came across the terms - Thought Leader and Thought Leadership - several times now and I am intrigued enough what this means. I wonder

• Who is a Thought Leader? 
• What makes someone a Thought Leader? 
• What does a Thought Leader do that Business Leaders do not do? 
• What do Business Leaders do that Thought Leaders do not do? 
• What are the characteristics of an Individual Thought Leader? 
• What are the characteristics of a Organization as a Thought Leader? 
• What skills a Thought Leader should have?
• How did one become a thought leader?
• How did an organization establish themselves as a Thought Leader?

Finally I decided to spend some time researching what this is and I tapped our friend - Google. I got a bunch of websites offering explanations around these terms. The pointers are:

• Forbes indicates that a thought leader is "a foremost authorities in selected areas of specialization" and "that significantly profits from being recognized as such". It also indicates that such thought leadership "brilliance is a function of acclaim, created where others bestow the accolades". 
• The Forbes article also indicates that a thought leader has a "selected areas of specialization"
• Leaders Direct site indicates that a thought leader may not need to have "inspirational influencing skills" that may be necessary for a senior executive. It also indicates that they may actually have "weak interpersonal skills and an indifferent character" and what really counts is the "credibility of the idea"
• CEO Online indicates, contrarily, that Thought Leaders need to have good "communication, positioning, pitching" skills along with 6 others
• The How Matters site indicates a quote "proprietary command over a challenging industry issue".
• It also indicates the 3 R's of Thought Leadership - "Results, Rhetoric, Relationships"
• The OpenViewPartners blog places emphasis on "bold ideas that are new and noteworthy" and "high quality (preferably original) research"
• The Thought Leadership Leverage article lays out the how "changing mindset" is more effective than "building skills"

References:

http://www.forbes.com/sites/russprince/2012/03/16/what-is-a-thought-leader/

http://www.leadersdirect.com/thought-leadership

http://www.ceoonline.com/expert_talk/leadership_management/emotional_intelligence_thought_leader/pages/id28949.aspx

http://www.how-matters.org/2012/11/11/what-makes-a-thought-leader/

http://blog.openviewpartners.com/what-is-thought-leadership-is-your-content-supporting-or-sinking-your-efforts/

http://thoughtleadershipleverage.com/2010/06/are-you-changing-mindsets-or-building-skills/

get feature and scenario name with cucumber-jvm

cucumber-jvm does not provide an easy way to get the feature name, scenario name or the step that is being executed. I had to find a workaround for this. For reference it is listed here. I am using the mirage project that I created a while back that provides a few utility classes around reflection. The ReflectionUtils.getFieldInObject uses that.

I created an aspect and weaved the info.cukes:cucumber-junit jar file with the aspect.

 public aspect ReportFeatureScenarioStep {  
      pointcut running_feature() : execution(public * cucumber.junit.FeatureRunner.run(..));  
      pointcut running_scenario() : execution(public * cucumber.junit.ExecutionUnitRunner.run(..));  
      pointcut running_step() : execution(public * cucumber.junit.JUnitReporter.match(..));  
 // Not used. These are here for reference only  
 //     pointcut completing_feature() : execution(public * cucumber.junit.FeatureRunner.run(..));  
 //     pointcut completing_scenario() : execution(public * cucumber.junit.ExecutionUnitRunner.run(..));  
 //     pointcut completing_step() : execution(public * cucumber.junit.JUnitReporter.result(..));  
      before() : running_feature() {  
           ParentRunner pr = (ParentRunner) thisJoinPoint.getTarget();  
           System.out.println(pr.getDescription().toString());  
      }  
      before() : running_scenario() {  
           ParentRunner pr = (ParentRunner) thisJoinPoint.getTarget();  
           System.out.println(pr.getDescription().toString());  
      }  
      before() : running_step() {  
           StepDefinitionMatch m = (StepDefinitionMatch) thisJoinPoint.getArgs()[0];  
           Step step = (Step) ReflectionUtils.getFieldInObject(m, "step");  
           System.out.println(step.getKeyword() + " " + m.getStepName());  
      }  
 // Not used. These are here for reference only       
 //     after() : completing_feature() {  
 //          ParentRunner pr = (ParentRunner) thisJoinPoint.getTarget();  
 //          System.out.println("AFTER FEATURE : -->" + pr.getDescription() + "<--");  
 //     }  
 //       
 //     after() : completing_scenario() {  
 //          ParentRunner pr = (ParentRunner) thisJoinPoint.getTarget();  
 //          System.out.println("AFTER SCENARIO : -->" + pr.getDescription() + "<--");  
 //     }  
 //       
 //     after() : completing_step() {  
 //          System.out.println("AFTER STEP ");  
 //     }  
 }  

This aspect is weaved into the cucumber-junit by doing this:

               <plugin>  
                   <groupId>org.codehaus.mojo</groupId>  
                     <artifactId>aspectj-maven-plugin</artifactId>  
                     <version>1.4</version>  
                     <executions>  
                          <execution>  
                               <goals>  
                                    <goal>compile</goal>  
                                    <goal>test-compile</goal>  
                               </goals>  
                          </execution>  
                     </executions>  
                     <configuration>  
                          <source>1.6</source>  
                          <complianceLevel>1.6</complianceLevel>  
                          <verbose>false</verbose>  
                          <showWeaveInfo>false</showWeaveInfo>  
                          <target>1.6</target>  
                          <weaveDependencies>  
                               <weaveDependency>  
                                    <groupId>info.cukes</groupId>  
                                    <artifactId>cucumber-junit</artifactId>  
                               </weaveDependency>  
                               <weaveDependency>  
                                    <groupId>info.cukes</groupId>  
                                    <artifactId>cucumber-java</artifactId>  
                               </weaveDependency>  
                          </weaveDependencies>  
                     </configuration>  
                </plugin>  

what is your favorite design pattern ?

In the last few technical interviews I have introduced this new question on design patterns -- "What is your favorite design pattern?"

The responses I got from candidates was - MVC, Singleton, DAO, Facade, Factory etc.

But what I don't get here is how can you have a favorite design pattern?

There are so many design patterns. The underlying idea is to use the right one for the right purpose and to solve the right problem. It does not matter if you like one or the other. Your level of understanding of one pattern may be high and other may be low - and that is perfectly fine. However how can I have a favorite design patten?

Whats happening

Almost a year since I blogged. Whats happening! For 8 hours a day it has been fun working with a new team on a new project. The work after work has been equally exciting. I am working on a new project - gherkinsalad - meant for automation using java, cucumberjvm, webdriver. The website is http://www.gherkinsalad.com.

what happens to the code without thought

I ran into this wonderful piece of java code at work. Just had to share this with everyone.
This code base was modified over last 4 years by at least 50 different developers with no TDD. Look at what the code ends up being in a long run.

// given that record and id are Strings
if (record != null && ((id != null && id.length() == 20) || (id == null || id.length() != 20))) {
  // do something
}

This if condition is essentially the same as

// given that record and id are Strings
if (record != null && true) {
  // do something
}

laws of software defects

Just for fun but feels so true on some days


Remember these laws of thermodynamics !
Energy can not be created nor destroyed
Energy can be changed from one form to another
In an isolated system the total energy remains the same


Now replace the word "energy" by "Defects" and you get the laws of software defects !
Defects can not be created nor destroyed
Defects can be changed from one form to another
In an isolated system the total defects remains the same

heh !

Organizing unit tests

I have been using a different approach to organizing the unit tests and this have turned out to be very effective. Worth sharing.

So if we have a class that looks like:
public class AccountService {
  public void getAllAccounts() { ... }
  public void createAccount() { ... }
  public void removeAccount() { ... }
}

For such a class we would traditionally have a unit test class called AccountServiceTest and all the test cases for all methods mentioned above. We would also have a bunch of global variables in the Test class and a setup (@Before) method.

My different approach involves creating a unit test class for each method rather than just one test class. So now we would have three classes:
public class AccountService_getAllAccounts_Test {}
public class AccountService_createAccount_Test {}
public class AccountService_removeAccount_Test {}

Generalizing it the unit test class name should be:
<<class under test>>_<<method under test>>_Test.java

Each class would have test cases for only the method referenced in the file name.

So how is this approach effective?

SRP - Single Responsibility Principle - is restored
In the traditional way the SRP principle is violated with the setup method as it is now common to all the test cases for all methods for the class under test. The setup has responsibilities to setup the objects and data for all the test methods for all methods in a class. Even the Test class has multiple responsibilities to test all methods in there. With the new approach every test class has responsibility to test only one single method under test.

"Look and Feel" of unit test cases
This is more of a perception thing but it works. Say if every method under test can have 5 unit test cases, in the above example having a single test case class means we already have 15 methods in there. When a developer looks at a single class with that had large number of methods with a single setup method that looked cryptic and they were less likely to update any unit test cases or even add more tests to them. When the classes were separated the "look and feel" of the test cases improved developers are now willing to and able to extend existing code base by adding new test cases.

What are the side effects of this approach?

• Popular plugins like moreunit do not work anymore
• Proliferation of Test classes may be something some developers may not like at first.

Let me know what you think !!!

TDD and Technical Debt

So whats the relation between Test Driven Development and Technical Debt?

1. Technical Debt is a term inspired from Financial Debt that people fall into
2. People fall into Financial Debt when the spending to earning ratio gets out of order, spending more than money earned.
3. In most cases spending more than limit is because the concept of monthly finance budgets is lacking or not enforced.
4. The monthly budget that is setup at the start of the month or year is like writing unit test case. Then living the month by the budget is like writing exactly the code needed for the test to pass.

Hey -- so just like in real lives lack of Test Driven Development or not enforcing it does cause more Technical Debt.

moving to github

I am moving all my code from presentations / pet projects to github -- http://github.com/daveayan
So far I have migrated 3 projects but more are coming. So far this is what I have:

- Google App Engine Presentation I did in April 2010
- Maven in Polyglot space presentation from Columbus Polyglot user group (03/2011) and QSI Tech Lunch (05/2011)
- My pet project rjson that is a core piece for my legacy code testing project (coming soon on github)

Need to start blogging again

It has been so long. I need to start blogging again. More stuff coming here.

Why do we need unit tests?

Alright ... the TDD & CI analogy is good. Makes sense the way it is described. But how would you justify spending time and money on writing unit tests when you are under tight budget schedules. Why would you even need unit testing when we have a v good QA and Testing team and as good processes in place? This is an often asked question. Here is what I think.

Unit tests are not meant to rule out the need for QA / Testing. Unit testing is not even meant to complement QA / Testing, although they would help significantly reduce the burden on QA team. They are there for the developers. Let me give you an example - but first, put on your developer caps. I have a BaseDao class that has a query method on it like this:

public List<Object> query(String sql)

The method takes a SQL query as input and returns back a list of objects mapped from the query result set. The more important part is that the method returns a null back if no records were found in database. Now I have atleast 20 DAO's that extend this BaseDao and use this query method atleast 100 times.

Now in all good mind and after lot of consideration it was decided that the query method in the BaseDao will have to return a empty list (new ArrayList<Object>()) back instead of null if no records were found in database. Sounds like a easy change, but now what happens to all the 100 methods out there using this method? What happens to all those if conditions with null checks after the query method? For example: One of the DAO's has these lines of code that will break for sure.

List<Object> l = query("SELECT * FROM PERSON")
if(l == null)
  return false;
else
 return true;

How does a QA / Testing team plan to test this change in the system? This would become even interesting if there is a DAO method among these 100 that will be called only rarely in certain special data combinations only. Does the testing / QA team know what that method is and when is it called? The real idea with unit testing is to help the developers. If we have all these 20 DAO's and 100 methods all tested well it will be easy to find where the code would break when such a change takes place.

Another important point is that these lines of code continue to live as long as the system lives. The QA / Testing team moves on and the next time there is a new feature development / major bug fix the same QA team may not be there even. Even the same developers wont be there working on the system. Now consider this scenario - as a new developer you are put on such a project and you have to fix a defect. This method you need to fix is being used by 10 different classes from web and batch projects. How can you update this method confidently without breaking the functionality offered by all the other 10 classes? And you need to think of this even before the next QA / Testing team comes in to start testing the system.

Of course for developers, just like any other technology, there is a learning curve and it can be difficult. Specially with tight deadlines and with agile processes you may not get a lot of time to learn TDD. But who said life would be easy. Once the learning curve is conquered things become easy.

TDD and CI - An Analogy

More often unit tests are considered to be time consumers, lines of code that add no real value to the core working of the system. Business Requirements have a first class place in the project as well as the "real" code but unit tests do not get such a place as often as they deserve.

Lately in my work I have to do lot of convincing and mentoring developers on TDD & CI and encourage them to write more unit tests. To convince and to justify the need for TDD and more unit tests and CI I came up with this analogy.

In my analogy consider the lines of "real" code that do business logic or UI logic or data access logic as the people in society. As there are good and bad elements of the society so are good and bad code modules and lines of code. As people we tend to take different roles and actions in day to day life. We drive, we go out to eat, we take a walk in park, we shop. So do these lines of code in the system take different contexts. The same method in a DAO or Service class may be used by a Web frontend or by a batch job. Some of the domain objects may be used by the UI in a certain way and persisted in database or other remote stores.

Now lets talk about the business requirements. To keep people behave well and in sanity laws are formed. Some are explicit - such as the driving laws and some are implicit - well, we don't toss burgers in a restaurants, its bad manners. The business requirements for a system are such laws that the code needs to adhere to. The code must satisfy the requirements.

Here is the fun part - In the society we have cops and agents that enforce these laws on the society. What do we have in software applications that enforce these requirements? This is where TDD, CI and even BDD comes in. It enforces that our code behaves well. If a line of code breaks a unit test it means it violated the law (the business requirements) in some way and this line of code must of brought to justice. Without unit tests we can only assume that the lines of code will behave, but what happens when a change in requirement comes up and we change the code without enforcing the new and old laws?

There can be an argument that functional testing is a good way to enforce the requirements. This can be true to a certain extent. The part of this that worries me are two things. One is that functional testing is performed at specific milestones in the project cycle. The lines of code live on after this milestone has passed and often times these lines of code change without anymore functional testing. Will it help if the cops come in a city only once a month and assumes that things are well in the city rest of the days? The second trouble is that it is not guaranteed that functional testing will cover all lines of code. Well good functional tests would at some point cover lines of code but where is the metric?

The point to be taken here is that write unit tests - these are the cops of your code. If some other developer comes and changes these lines of code and unknowingly causes them to be bad elements of the software society the cops (unit tests) will catch em.