6 comments

• 

  Geir Hedemark

  1 year ago

  I don't know if you have heard of this already, but it follows up on your thoughts on the time it should take to run tests.
  
  There is something called flow psychology ( http://en.wikipedia.org/wiki/Flow_(psychology) ) which describes the conditions that need to be fulfilled if you are to experience "flow" (Which at least for me is a precondition for not being caught my my mail reader). I learnt a lot from that as a project manager.
  
  On to a question I have:
  
  What would happen if all your tests took less than a second to run? Would you structure your tests differently?
  reply
• 

  Johannes Brodwall

  1 year ago

  Flow is definitively an important component of happiness.
  
  "If all my tests took less than a second"? Do you mean more than a second?
  
  Most of the tests I write first are independent of infrastructure, which means that they run really fast. I try to separate out my slow-running tests (e.g. test that a search generates SQL that retrieves the correct result from the database). I haven't doing any soft of formal categorization here, though.
  
  I find that either I run a single test in the IDE. Then it should run fast. Or I run all my tests for the command line. Then everything should be run. So I don't need to categorize stuff now.
  reply
• 

  Geir Hedemark

  1 year ago

  I was unclear, once again. Let me clarify.
  
  If all of your tests - system tests, unit tests, integration tests, the works, all completed in less than a second - would you structure your test suite differently? Would you still keep your unit tests around? Would you hang on to your CI server?
  reply
• 

  Johannes Brodwall

  1 year ago

  Hi, Geir
  
  Thanks for a very interesting question. I think I might want to expand this into a full entry eventually.
  
  I think the risk of my tests running this quickly are fairly slim. But if it did happen, I would still want to separate out the functional tests that we create in cooperation with domain experts and the system level tests that simulate realistic operating conditions. I'm happy with the system level tests we have now, which stress the system. There's no way these could run in under 1 second.
  
  But neither of these types tests are JUnit tests. Our functional tests are implemented in FitNesse and our system tests are scripts that push data on the system (for a web based system, I would consider JMeter).
  
  For tests implemented with JUnit, I would not see any usefulness in the distinction if things ran sufficiently fast. I might consider deleting tests that were covered sufficiently by the functional tests, but since these are run by different tools, it might be hard to find out exactly what tests could be deleted.
  
  I would keep my CI server, though, so I was sure that changes by every member of the team (myself included!) actually passed the test suite. I might even add a failure condition if the tests ran for too long, to make sure that we didn't start down the slippery slope of slow tests.
  reply
• 

  Geir Hedemark

  1 year ago

  I agree when it comes to performance tests. Those are, by design, meant to take longer.
  
  My idea when it comes to functional testing doesn't really lend itself to fitnesse. I want to also veryify that all interfaces to the system do what they should. I can't really see that happening with fitnesse the way I want it to.
  
  The good news is that we are almost there. I have created functional test suites which ran a full regression test for a module in less than a minute or so. The drawback was the setup time, which also was on the order of a minute. This minute was caused by spring/hibernate.
  
  Good point about the CI server. What if eclipse (or whatever) continually ran your tests, using the second processor everyone seems to have but not use these days, just as your code gets compiled "continually" behind the scenes?
  reply
• 

  Johannes Brodwall

  1 year ago

  If everyone (including the questionable professionals who currently don't really test!) ran continuous testing in their IDE, I wouldn't feel the need to run these tests in CI. But we use our CI server also to build the software and push it out to test servers for performance testing. So I don't know what the point of disabling the tests would be. :-)
  
  I think functional tests will tend to vary a lot from project to project. It really depends on what your interface to the outside world is.
  
  A minute to start up Hibernate (Spring is seldom to blame) is unacceptable for a functional test. In my functional tests, I have replaced the whole persistence layer with a stupid in-memory replacement.
  reply