Page last modified 16:18, 9 Oct 2008 by royo

Art Of Unit Testing - Book Wiki > Book Blog

Book Blog

No headers

ISerializable - Roy Osherove's Blog : Art Of Unit Testing
Unit Testing in silverlight land with Typemock Isolator I’ve been asked quite a lot recently whether one can write unit tests and isolate logical code that runs inside a silverlight application. Up until today my initial answer was ‘no’ because silverlight runs under different versions of mscorlib.dll. however. Today I actually gave it a try and realized that writing unit tests (not integration tests, as the silverlight test framework allows) against silverlight based code is possible and quite easy. Just like any other code that relies on a third party platform (like sharepoint code) the silverlight related code might have various dependencies. I’m going to show how to use Typemock Isolator to overcome a couple of simple silverlight dependencies (using HtmlPage) and how to setup a test project against a silverlight project (with NUnit or MsTest) Assumptions: You have an open solution the solution contains a silverlight class library or silverlight application project To setup a test project against silverlight using MSTest: Add a new test project to the solution Remove all the existing classes (Database test, ordered test etc..) so that you are only left with the unit test class (UnitTest1). Remove all useless comments and crud code from the test class so that you are only left with a test method (no comments, not even the TestContext) Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first) Add a reference to the project under test You can now write tests against the object model (standard classes) To setup a test project against silverlight using NUnit or MbUnit: Add a new class library to the solution as your test project Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first) Add a reference to the project under test You can now write tests against the object model (standard classes) How to break the dependencies Now, here is a simple example of code that has a silverlight dependency we’d like to test: Let’s say we have a class in the silverlight project called ChatSession (I’m basing this on ScottGu’s Chat demo). but it’s constructor looks like this: What if we wanted to control during our test whether the page is enabled or disabled? Here’s one way to do it using Isolator: In your test project add two references under the .NET tab: “Typemock Isolator” and “Typemock ArrangeActAssert” Write the following test: Using Isolate.WhenCalled() we are able to circumvent any method (static or not) to return whatever we want, or throw an exception. Here’s a more interesting case. Let’s say we have a method that modifies the current page and shows some html to the user in a span tag: here is one way to write a test that makes sure that the right text is set into the message element in the page, without needing to have a real page present: There are several things to note here: HtmlElement is Sealed has an internal constructor, but we can still create a fake instance of it using Isolator. None of the other frameworks can do this. We are returning a fake object from a method call chain (HtmlPage.Document.GetElementById ) without needing to create a separate stub for document. None of the other frameworks can do this (especially since it is based on a static method call to begin with) We assert in the end that the method was indeed called with the correct arguments without needing to change a single piece of code. Granted, I wouldn’t write code like this (I like decoupling!) but sometimes you just don’t have the ability to change existing code. If you are developing an open source silverlight project, it is important to note that there is a free full version of isolator for open source projects with full functionality. These are just the beginning of my journey into silverlight unit testing. I’m looking for good code examples that you might want to test, with various dependencies that need breaking. your comments are appreciated.
Isolator Feature Focus: Duck Typing and Isolate.Swap Typemock Isolator 5.1.1 has been released, and this release brings with it some awesome (seriously) features that are unique fro any other framework I've seen. a good overview of features can be found here. The Isolator Swap feature allows swapping calls between real and fake objects (kind like redirects) so that any relevant calls made against the real object will be redirected and invoked on the fake object. Here's how you use it: Unlike doing a standard "WhenCalled()" on some object method and telling what its custom behavior will be, "Swapping" objects redirects all relevant calls(I'll explain what "relevant" means in a second) to the fake object which you have created. Duck Typing Awesomeness Another cool thing about it is that "Duck" and "Dog" don't have to have a shared interface or base class. The "Swap" feature will redirect a method call if it exists on the "fake" object, but if it does not, it will invoke the original object. This is one interpretation of what's called "Duck Typing". The "CallsOn" and "WithCallsTo" methods take an object type, so you can send in anything you want. if a method on the fake object matches the signature and name of a called method on the real object, the fake method will be invoked.
Art Of Unit Testing – Book done, Wiki ready I’m happy to report that my book has finally hit the “last review” stone, which means all chapters are done and the book should be in book stores around jan-feb 2009. If you’d like to read the E-Book version right now(PDF) you can purchase that for a lower price at the manning site. I’m also announcing a Wiki site for the book, with all the chapters listed, along with errata and your comments as time goes by. The site already contains the first paragraph of all chapters, with Chapter 1 available for free download, and the full text of the “Tools and frameworks” appendix. Since its all in a wiki, feel free to add your own tools in there that you think would be valuable, create your own pages that talk about things you think people reading this book should read and more. I hope the site becomes a good place for guidance for beginners and advanced in the field of unit testing (does not have to be in .NET). Oh, and you can see the final book cover on the right.
Unit Testing decoupled from TDD as well== Adoption The discussion on the future of unit testing for the masses has shifted from the standard “if they are too stupid to learn it, we don’t want them” to “TDD without good design will make really bad tests”. and this is a good thing. it’s a good thing because it secretly implies that if you could write Good Unit Tests, regardless of design, then teaching design as a separate thing should not pose a problem. Discussion so far: Unit Testing Decoupled From Design == adoption Goodbye mocks, Farewell stubs But first, let’s make one thing clear: TDD is NOT Unit Testing. My previous post was not “TDD decoupled from design==adoption”, it was “Unit testing decoupled from design == adoption TDD is (technically) about when you write the unit test. You can still write crappy tests in TDD mode. With good design. Unit Tests deserve a good book about them without TDD in it because the current books about TDD suffer from lack of SRP. On the other hand you can still write good tests, without TDD. The design might make them a little longer, but if you learn to write unit tests well, this should not pose a problem. Learn Good Unit Testing regardless of TDD and Design people can start learning unit testing before learning TDD. they learn the silly stuff (the kind udi looks down upon) in an hour, and then they get to the hard stuff : making the tests readable, maintainable and trust worthy. these are separate properties that can be learned to be done regardless of design, since test naming convention, refactoring and correctness can be learned in separate. The only reason it’s really hard to teach in separate today is because as those devs get to test real world code they have to start dealing with real world dependencies and that it the barrier that usually stops most people. having “intermediate” frameworks in place such as JMockit and Typemock Isolator can help in that regard, keeping people focused on the task at hand and deferring the need to learn other types of design for a time that is better. The idea is to let the developer choose when to learn which technique: Learn how to write good unit tests (book) Learn Good Design technique (give me a link to a good starter book!) Learn Test Driven Development (book) I want a developer to grok each one of those things to a very deep level, but today’s “market” wants them to learn it all in one go. Since each one is its own little art form, are you really surprised to learn that most of the mass population is not there yet? If people think that the only way to write good unit tests that are not worthless is by doing them TDD style with SOLID design, they are showing several things to me: they don’t get what “good” unit testing is They don’t “get” what value good tests have outside the context of TDD they guide a community who reads them down a path of “all or nothing”. which for most people begets “nothing”. Field notes: Udi writes: “If logic is duplicated, if the code is tightly coupled, if there is no separation of concerns, the unit tests will be useless - even if they “test” the class in isolation.” That is wrong in many levels. Let’s start with the fact that a unit test can have a specific value in testing a specific class in isolation, even if the bugs in that class are duplicated on other classes. If Udi does not agree on that, I don’t know what else to say about that. I wish that what he meant was that all these problems were repeating in the tests, since this is usually why people fail when they start out unit testing: they write crappy tests with bugs, code duplication and lack of isolation (this can all be learned). But he’s not. It feels like a “purist” view that does not reflect the real world. It must be nice living in an echo chamber! Let’s see udi contradict himself in the same post: “Managers shouldn’t necessarily sacrifice their projects on this altar of learning. Organizations need to find ways for developers to safely practice these techniques as a part of developing their “human resources”. First of all, this needs to be communicated to everyone - that the organization understands the importance of these techniques, the desires of developers to adopt them, and the projects that need to be delivered.” “…It’s a gradual process.” That’s the problem Udi. It’s not possible to do it gradually today, because people keep saying that you can’t do things in little chunks. you have to learn it all together. If we were to help developers learn what good unit testing means they could much more easily learn that stuff, and then learn the other stuff. Casey writes: “It is not a technical challenge to write a unit test, any half trained monkey could do it. A few attributes, a couple of classes, a few new() statements, and a few Assert statements. We don't need magic mocking frameworks, we don't need any mocking frameworks for 90% of our tests, we don't need application of 12 GoF principles before we can write a test, honestly. Testing and writing automated tests is hard because it requires a strong understanding of the user stories that sit behind your code and project, because it requires you to think as a user and not as a developer, and because it requires you to be able to think of multiple conflicting axioms, test cases and class interactions, and how to cover the weaknesses of each individual test with other complementary tests.” Casey preached about what he sees as the “end game” for unit testing, TDD and Design. where everything is in one great ball of goodness. Or did he? Let’s separate these points into better chunks: Design related stuff: to think of multiple… class interactions Good Test related stuff: Understanding of the user stories Think as a user not a developer think of multiple conflicting axioms (..?) test cases.. how to cover the weaknesses of each individual test with other complementation tests. TDD related stuff none So, what I gather from casey’s words is that you want your tests to be : testing the right thing be readable and show intent have good coverage test or be aware of class interactions (the design related stuff) except the last point which is where most of today’s mock frameworks force you to redesign, there is nothing here you can’t learn separated from TDD and SOLID techniques. the interaction stuff is where those other isolation frameworks I mentioned come into place. We need to decouple Good Unit Testing from TDD and SOLID. And it seems that not only “noobs”, as a stupid idiot commented thoughtfully on my last post, can benefit from learning this distinction.
Unit Testing decoupled from Design == Adoption In my previous post I started talking in more coherent words about feelings I’ve had lurking in the past year. This post is an expansion of that, after having some great twitter conversations in the past few days with some great people. Treat this as a stream of consciousness. before that, here’s a short highlight summary of responses to my post in other blogs: Steve Freeman thinks I’m pulling the wrong way So does Ayende (in a way) A goof thread on this in the alt.net mailing list there are also plenty of people who picked up the ball (some of them have already done that before I started mumbling on my blog) and continue to make better attempts at bringing knowledge to the masses, in more coherent, newbie friendly manner. Rhys shares some hints and gotchas Ian Cooper gets it (even if he doesn’t think he does – I’ll explain later) Jeremy is asking important questions Jimmy has good tips and the comments on that post are… interesting (some are very expected, though). Let’s make another interesting statement, and this one is indeed product related. To reach the masses, we need to decouple S.O.L.I.D Design Principles from Unit Testing and TDD I’m not saying throw out design. I’m saying – it’s important enough to be learnt on its own, and not in the context of tests. in fact, in that context, it inhibits too many people to be practical as a learning tool. so we need to separate forces. But why? The wall to start unit testing is too high (for most devs) and will always be too high if you have to know your SOLID in order to even start writing your first test or refactor your code for testability. This problem only exists in static languages like Java, C# and VB.NET, where in order to replace something you need to have a way to do polymorphism on that class (interface, or inheritance with virtual methods). The reason people love Test Driven design so much is because the technique, aided by most of today’s leading mock frameworks, squeezes out all the problematic static coupling that exists in the current design, before it will even let you test something. Can’t decouple that web service dependency from your class? you’re screwed. Redesign and only then you can test it. Have a static method that gets called? you’ll need to jump hoops to override the calls to that method, let alone the method itself. Feel the burn, baby! TDD existed before mocking frameworks existed. what did people do then? SmallTalk was one of the first languages that had people doing TDD with it. SmallTalk is testable by default in that you can re-define anything to do anything else at runtime. People just thought about their design as they wrote the tests first, and they had no real constraint on what specific design to follow. It still holds today in Ruby, for example. Now, the reason today’s “hardcore” Test Driven Design is even possible is because current mocking frameworks rely on static language (and compiler) to tell the developer what can and cannot be done in terms of replacing objects and implementations. That forces a specific design methodology that closely matches what SOLID principles stand for, and it’s great. you get great code out of it, and nice designs. I repeat – I love the idea of SOLID design and I practice it. So, why do I think that these things should be decoupled? At the end of the day it is a constraint that is put upon each developer that wants to start unit testing (not even talking about legacy code right now). That means that each developer that starts tests will hit a brick wall and will have to start reading up on all things SOLID, patterns, or just buy my book. There is no “easy” way. and that’s fine and dandy for most of the people reading this blog. And for the past four or five years, it’s been just as fine and dandy for me too. BUT, The adoption rate for TDD and unit testing at all is low, and I suspect that it is indeed this learning curve that makes that possible. And it will continue to be low as long as doing great (not OK, great!) designs is a prerequisite to doing proper unit testing (or at all). The current mock frameworks (most) enable this notion and embrace it. Today we live in our own little echo chamber where everyone is happy with their “pure” designs and consultant fees, because we are part of a very elite group: those you need to pay in order to learn the “hard stuff” to do something which many people want to do but just can’t. It’s almost like knowing Regular Expressions, except a much larger scope. We’re making it harder on purpose to start doing unit testing, because we believe that it’s someone’s best interest (and in a way, in our best interest) to learn SOLID first. The people I know from the mocking community are amazing, passionate and smart people. They would never do anything they don’t fully believe in, and their value system is almost radically pure. And I love that. but we cannot play judge here. people learn in different ways, and the low adoption rate tells us that it’s not working as well as expected. Why would decoupling work? If we were able to decouple the requirement of knowing SOLID and just get people to start testing today and learn SOLID later or in parallel, adoption would be much higher. Why? People who get test infected soon want to learn SOLID anyway People who succeed doing one thing, will be more open to adopting other things from the same source of knowledge It’s less scary, and less time consuming to start doing things. We reduce the barrier to entry into the agile world. It allows developers in a non-agile environment to do little,incremental steps on the road to a better way of working, without needing to chow down the whole meal in one big bite. Sucky tests are better than no tests. They also lead to better tests and better understanding of design as people learn and get better at it. What’s stopping us? But with the current set of mock frameworks, you can’t do all those things. you technically are not allowed to do that. and that’s why solutions like Rhino, Moq, and NMock will not appeal to the masses, and why solutions that support other types of design decisions (JMockit, Typemock Isolator) will continue to gain popularity as people strive to become more agile under non-agile conditions. they provide that needed stepping stone to learn unit testing without needing to learn SOLID as a requirement. That’s the difference. But the community is actively depressing thoughts about using such tools today because of the belief that TDD and design have to go hand in hand at ALL times. Many of them that will read this are scared that I’m even suggesting this, because they fear a return to darker ages where people didn’t have a clue, and there was nothing to stop them from screwing up the design. But as you’ll see later on, the plan is to get people on board the design stuff, just not as a prerequisite to the unit testing stuff. that’s all. Of course, the big problem here is that I actually work at Typemock, and thus, many people feel that every word that comes out of my mouth is a sales pitch. but the reality is that if Typemock had been an open source project, those people would have read the words that I write, and not try to find a hidden meaning in them. and the truth is I’d write something like this even if I didn’t work there today. I’d quit to say these words with more credibility but I want to have an impact on how those products work and feel, so I’m not going to do that, sorry. What does the industry need to do? Direct people who are reluctant to go through that big learning curve or have a problem with the current frameworks to tools that mitigate those fears (Typemock, JMockit). One is open source, the other has an open source free license. so they are both achievable to use and play with. When doing so, learning better design techniques should be on the horizon. but it’s important not to discourage use of these frameworks on existing non SOLID design notions. people have a hard time with change as it is. They need to learn one step at a time and we have to let them have the time to grok things separately if they want to. Decouple the task of teaching unit testing from the task of teaching design. They are both important but are not a requirement of each other. We need more Design material out there with a simple way to reach and find it (we’re on the way today). Unit testing information is already out there and more is going out every day. Realize that tools that enable different designs are not a problem, but a way to get a foot in the door with developers who wouldn’t walk through that agile door in other ways. it’s a bigger net to catch more fish. These tools allow people to have a bridge to the agile world with less pain involved. This is something that we need to continue talking about, since it can change the future for most devs today. If we shy away from talking about this, and stay in our cozy little submarine underwater, we will sail right beneath all those people lying on the beach, wondering what’s up with all those people in the water, or afraid to take a swimming lesson. (yeah, the world’s worst analogy) Thoughts? Comments?
Goodbye mocks, Farewell stubs About 4 years ago I was consulting for one of the largest companies in Israel (about 15,000+ employees) and helping them grok TDD and automated unit testing. I had spent there over a year and a half, on and off, and was teaching both .net, C++ and java developers about the techniques and tools they can use. I was consulting for C++, Java and .NET developers. The C++ devs were first. They had plenty of objections to what I was teaching them, because the amount of code refactoring they needed to do in order to make their code testable was almost unrealistic for them. Even writing new code meant writing something on top of old-untestable code, and they had a hard time adjusting to the more interface (pure virtual) based, contract driven TDD approach for new code. C++ devs has a hard time becuase it takes more code to write good testable code, and the coding tools they used (VS on windows and VI and such on UNIX based systems) didnt’ provide a good productive way to make code “bend” easily as they refactored it or created it in the first place. every little change meant a big “search and replace”. every method refactoring meant adding it the declaration in an .h file and thus they tried to have a little to do with changing code as possible. Looking back, today they still don’t have the right infrastructure in place or the right tooling or the willingness to do unit testing or TDD becuase it’s too hard with the current tooling to accomplish it. I had also spent some time with the .NET devs and thought them the same things. Becuase tooling support was better it was a bit easier for them to accept the practices, but still people had a lot of trouble grasping concepts such as dependency injection, inversion of control and designing for testability in general. I found that most people didn’t really have a big problem with the idea of exposing things they didn’t use to expose before, it was more about the concept of learning how to refactor the code in interesting OO ways to make it testable. Making a class non sealed, a method virtual, and then inheriting from that class and testing against the inherited class (“extract and override”) was very much unintuitive for them, and other concepts like containers (spring at the time) didn’t help them get into the groove either. For them it was like drinking water from a fire hose. Consulting for the java devs Java has lots going for it in terms of unit testing, automation and TDD Java devs have a much longer “heritage” of tooling and unit testing than in the .NET world, and so you’d expect them to be more inclined to do these things. JUnit is part of any decent java idea so the barrier for entry should be lower, right? Refactoring was also in most decent java IDE's, so you’d expect them to grok that too. but that wasn’t the case. class methods in java are virtual by default so you can override what you want as long as the class is not sealed (final)== more testable by default. Java required a much higher OO learning curve to start with. Good OO knowledge is fundamental for testable designs. Unfortunately, most developers I met didn’t even know how to use the refactoring tools they had available and had a hard time with the same dependecy injection principles that the C++ and .NET devs had. It was just too cumbersome to make the code testable, but eventually the infa team at the company had made the decision to take a couple of weeks and make all the infra stuff replaceable for testing purposes. they did not regret that decision. So the java group had the most success with testable designs, but they still had a hard time doing it. in fact, only the infra team at that group took the plunge. the other (higher level) devs in the java group didn’t grok it. and don’t use it today. Why was this adoption been such a failure? They were willing to learn, but they didn’t like what they learned. It didn’t fit their needs in the real world. Or the way they worked, or the way they knew how to do things. Think what you will ,but 99% of the dev worlds aren’t ready for the “testability” message. and that is a fact that is hard to realize for most of us, which just can’t grasp why isn’t that any sensible human being isn’t willing to do everything much better than they are doing today. Spending the past week running around the Microsoft Campus in Redmond had made several things clearer for me. I’ve been feeling them for a while, but this week these abstract feelings materialized into something that is more coherent that I can talk about. Why do most developers not write unit tests, still? Reason #1: Learning curve One of the main reasons most developers still today don’t really take to unit testing (and TDD) is the really high learning curve that is forced upon them. That is not to say that learning about mocks, stubs, dependency injection, IoC containers, Extract & override technique, Record\Replay, AAA and more -- is not useful, but it is a big obstacle to get people into the habit of doing something which they know is good for them – test their code, verify their assumptions, automate, integrate, be confident, get feedback, define behavior – all those and many other benefits are being thrown away by developers all over the world because the entry fee to this world is too high, for all the wrong reasons. It’s time to simplify. Get back to basics. Take a good hard look at the huge spiral that we are leading people into and asking whether removing some things today can lead to those things being accepted and being easier tomorrow. or maybe they weren’t needed at all. Say Goodbye to “mock” and “stub” “mock” and “stub” have played their part in us trying to explain to ourselves what it is we are trying to do, to start learning the minute and important differences in the patterns of our test code. They helped us define what it is we are looking to do, in terms that, for lack of anything better, are now the de-facto pattern names for something that should never have been named in the first place for the casual programmer. That’s why in the new version of Isolator we’ve chosen to completely get rid of the words mock and stub, and get back into “fake”ing. You can fake a method or a class instance, and you can verify that something was called. You should not care whether this is a stub or a mock. We need to clean our language: mock, stub, fake, test double: They all mean something in the context of us trying to identify patterns in our test code. But as users mature and start using the frameworks, those same words increase the entry barrier, and effectively block people out. If I’m showing a simple test that fakes some dependency that a class uses to a new dev on my team, I should not have to send them over to read fowler’s “mocks arn’t stubs” article (which is enlightening, but should not be relevant to them) because it misses the point. Less interaction testing, more state based testing Most crappy unit tests I’ve seen were over specified in that they asserted on internal interaction with an object way too heavily, which made the tests very fragile. Getting back to basics means making it easy to enable state based testing (assert style checks) and only enabling interaction testing where it makes sense: (when your app does one-way messaging to an external component). verifying that something was called should be the exception to the unit testing rule. The AAA style APis make that a more clear distinction. Record-replay APIs make it too easy to make interaction based tests when you really want a state based one (getting a result vs. checking that something was called) Use AAA style Apis As an industry we are learning and changing all the time. The “Arrange-act-assert” pattern in the field of mock frameworks represents us trying to return to basics. Moq did that nicely by showing a way that leads there, and it is being taken gladly by Rhino and Typemock Isolator. It is taking off because we are starting to look beyond our own little echo chamber and seeing what is is that all those other 99% of devs are expecting as guidance, and in what form. AAA is guidance because there was an outcry (a silent, vote of feet) that record-replay is not cutting it – it’s too confusing for non-experts. It’s too technical. and it does not lend itself to the way people want to work. So most people didn’t use it. Visual Basic Language Support Everyone screwed up on this one.most VB developers who will try to use one of the current frameworks will not succeed since they required anonymous methods or lambda expressions. Some frameworks use VB specific keywords which makes them VERY awkward to use. If we want broad acceptance of our practices we need to make sure whoever wants to can use the tooling we make. VB is huge and its amazing how we all pretend it does not exist. But I think it’s the key to getting to those people who want to be better but all they see is a high wall instead of an entrance. Use “Isolate” and “fake” instead of “mock” Don’t say ‘mock’ frameworks, say “isolation” frameworks. Because ‘Mock’ is an overloaded word it is confusing. Explaining to a developer that you can “isolate” something from its dependencies rather than “mock” its dependencies feels more natural. Simplify,Simplify,Simplify Remove APIs instead of adding to them. Have single point of entries to your frameworks because discoverability is the key to good framework use. That’s why extension methods are problematic, they mean multiple points of entry. Be open: Realize that other tools and techniques can act as bridges to the agile world Design for testability is just one way to get where you want to go. it is not the only way. Sometimes it’s impossible. Discouraging the use of other ways of working (using tools like Typemock Isolator or JMockit) will only make the current echo chamber more closed and feel more un-achievable for those who need to get to the same point with a very different set of constraints than what most of the “loud” voices need to face. By “allowing” other forms of working you enable those who want to, to get a bridge to help them instead of walking by foot the whole way. Any other way would mean that we will all fail in delivering what we believe in to those who have no idea what we’re talking about.
Over Specification in Tests I've gotten the chance to visit and consult various companies and I see this all the time: Over specification in tests, especially when doing mocks and stubs. One main reason for that is that people use their stub as a mock object (stubs are fake objects that we will not assert on so that we can test something else). here is a small example from something posted on the alt.net mailing list: [Test] public void Can_Return_All_Plans() { IPlanDao mockPlanDao = MockRepository.GenerateMock<IPlanDao>(); mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan}); PlanController planController = new PlanControllerForTesting(_mockCommonDao, _mockCommonService, mockPlanDao); ViewResult result = planController.All(false); mockPlanDao.VerifyAllExpectations(); result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans"); } in this test the following line is used to "stub" out a return value into the system under test. mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan}); no problem here. The problem lies in the last two lines of the test: mockPlanDao.VerifyAllExpectations(); result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans"); the second line (the assert) is probably the thing you really want to test. but the call to "verifyAllExpectations" is the one that makes the test very fragile. It is "asserting" that someone has actually called "GetAllPlans()" to get the results, when it clearly does not matter how one got the results. all that matters is that: "Given a set of results returned into the application under test, the view in the controller is correct" (the translation of the last line in the test). If by any chance the application later does several more calls to the stubbed out dao, or uses a different method to get results, the test could break, even though the application would still ultimately work with the same end result. It's like me ordering a pizza delivery and then asking the delivery guy "did you get here by car or motorcycle?" . I realy don't care, as long as the end result: PIZZA, is correct. So the general rules for me are: If you see both "Verify" and "Assert" in the same test, it is usualy a smell of over specified tests if you see "expect" and "Return" on a fake object, make sure that you name it "stubXX" or "mockXX" so that you can distinguish whether you want to call verify on it later or not(most cases should be "not") Try to test on the end result or end state rather than verify interactions. The only time you absolutely have not choice but to test an interaction using verify is when calling void methods on external objects. that is clearly a one-way communication and is (or part of)the end result of what you are trying to accomplish. A lot of people think that adding that extra "verify" just means it is a good thing since they are doing more assert. Well, they sure are testing more things, but they are internal things to the app's behavior and that is usually leading to brittle tests. Try to test on the end result or end state rather than verify interactions.
A sneak Preview of Typemock Racer : Thread Deadlock Finder Here's a sneak preview of the upcoming Typemock Racer product, which I'm currently working on. We should be out in private CTP in a couple of weeks I hope. Typemock Racer tries to solve one huge problem for developers working in multi threaded environments: Detecting code that can result in deadlocks or race conditions. It does this by providing a framework API which you can use in your tests (under NUnit for example), that allows you to execrise your code under test until it finds a deadlock or times out. Here's an example of how you'd use it. Consider the following class which has two methods, that if run by two seperate threads, can result in a deadlock condition: At some point in time the threads could execute such that thread 1 is waiting on resource b, which is held by thread two , which is waiting on resource a, held by thread 1 - a pure deadlock. Given this class we can write a test that looks like this: The ThreadTest class has an "AddThreadAction" method that receives a delegate. you can use it to invoke your own code, it will be invoked in a different thread. IN this example we are creating two threads, each one running a different method on MyClassWithLocks. We can just right click and run the test with TestDriven.NET to recieve the following results: The output states that there is a deadlock that was found, with the exact steps that happened (with line locations) to reproduce this. It currently only works with lock (monitor.Enter\Exit) but will support all locking constructs in .NET. What if the methods you point to create their own threads? Like this: Here is the output that you'd get if you run this test:
Introducing Depender - Testability Problem Finder Update: Here's how you can extend Depender with your own dependency rules Depender is a tool that I created to illustrate several concepts both for my book and both for my training classes. I think it should prove valuable (as it gets more and more features) to anyone who is trying to write tests for existing code. Depender allows you to load an assembly and inspect the types within it according to very simple rules. the main question on my mind and I run it is: "How easy would it be to test class X?". Depender will list all the classes in the assembly and for each one will try to detect possible testability problems (non decoupling) and places where you might be able to "inject" your own implementation (calls to virtual methods, fields that are interfaces or types who are not sealed..etc..) so you can: see places where you might want to refactor your code or use a tool like Typemock Isolator see places where you can already use "seams" to inject behavior into your code without needing to look up the source code manually. here's what the main screen looks like when loading an assembly: the type "RealLogger" is marked in red because it contains calls to static methods. double clicking on the type's node will load a separate window just for this type: I will be adding more rules and also releasing the source so that other people can help develop this. for now I just want to get it out there and see what people think. possible futures: Make it command line driven so it can run in your CI builds Make it give "scores" to assemblies or types on testability and decoupling Make it add "suggestions" on how to fix decoupling issues Make it create the base diagram for my test feasibility matrix Download Depender from here.
Over-testable systems, and mocks as bad test smells This is an interesting problem I've been running into at work. Sometimes we would try to write a test against the system or a component and we would literally have too many choices of how to write it or where to "intercept" or "inject" our stubs or mock objects. In a way, the software is almost "too testable" in that there is usually at least 3 ways to accomplish a similar level of isolation. What it comes down to is the fact that for a real system, there are different "depth" layers where indirection can be injected. For example, Component X needs object Y that relies on object Z. If we were testing component X we could "replace" either Y or Z. Some would argue that replacing one level too deep is making this an integration test, but as long as its all running in memory I don't see a real problem with that. I gave up on trying to do "pure" unit tests for 100% of the time a long time ago. Many times, it is much more complex than this. For example, replacing object Y instead of Z (at the closes layer to the object under test) can also lead to problems if: The contracts between X and Y are very chatty, than stubbing out Y could lead to a very hard to read or very fragile test You are trying to test that interaction occurs between Y and X than that might be an over specified test. If replacing Z would lead to a more state-base test that I'd choose that over replacing X. (shortly: I'd rather use stubs over mocks so that my tests become less fragile. some layers require mocking and some are very simple to just stub out) Here's a thought or two: if you one one layer farther away than the one you are using, and it does not require mocks to test, that try to use that layer and forget the inner layer? If you treat mocks as a smell for over specified tests that will force you to find a way to test the same thing using a stub. I find that sometimes you have no other choice but use mocks (when your app under test has a one way out communication with a component) but in all other cases, I am starting to look at it like a design smell for my tests (over specified tests) getting back to the original problem (too many ways to test the system) I am trying to think if this is a good or bad thing. The only bad here is that someone who is testing the system for the first time has no real guidance on where to replace things (except look at other tests) which can make writing the first few tests really hard to figure out. The fact that we are using a sort of auto mocking container in our tests is both a blessing and a bit of a curse. Currently i find that it saves more time that it takes away. the tests and their dependencies are relatively more readable since we are using a more explicit version of a test container that requires you to explicitly state what will be replaced (it just doesn't tell you who requires that component). I can already hear the pundits say "well that means you have a design smell". The only thing that smells here to me is that because we followed strict DI and interface based injection rules, we ended up with a system that is not just made of lego parts (a good thing), the lego parts or too darn small!
Unit Testing Interview at TechEd My interview at teched us on advanced unit testing and legacy code can be seen here: http://msdn.microsoft.com/en-us/events/teched/cc676818.aspx or downloaded here We occasionally stop and laugh a bit because "speaker idol" contest was being held at the booth next to us, which was really really loud.
Fit/Fitnesse Fixture Gallery -Real world help for FIT and Fitnesse users This great news was just published at the alt.net mailing list. putting it here for future reference. "Fixture Gallery 2.0 (release 2008-06-09) is now available for download from SourceForge. Fixture Gallery is a cookbook for FIT/FitNesse tests. It provides developers with a quick overview of the most important fixture types and concepts for agile acceptance testing using the FIT framework. For each fixture type, this document explains the table format and fixture class structure and provides advice when to use and when not to use it. Each example is accompanied by the source code for Java, .NET and Python FIT implementations, in a form that can be easily copied and used as a template for similar fixtures. Version 2.0 brings: - Python examples for all fixtures and concepts explained in the document. - Examples for ConstraintFixture, CombinationFixture, CalculateFixture and notes about SetFixture and SubSetFixture. Many thanks to Jens Engel and Mike Stockdale for their contributions to this version. For more information and download links see http://gojko.net/fitnesse/fixturegallery -- gojko adzic http://gojko.net "
Unit testing vs. Integration Testing : The Restaurant Analogy Here's an analogy I came up with to explain the difference between unit tests and integration tests. Imagine going to a restaurant with a bunch of your friends. there are some couples there as well as singles. At the end of the meal, it is time to pay. Unit testing is like giving a separate bill to each couple or single. Each person knows only what they needs to pay as well as tip, but do not care what anyone else had, or what the total meal amount is. as long as they pay what they need, they can go home peacefully. Integration testing is like giving one big check to the group, with everyone having to calculate on their own how much they need to pay. sometimes everyone things they are good, but in the end the total amount may be too high or too low. There is a lot of interaction going on to decide who pays for what and who has already paid.
Tough Questions about unit testing - Your comments required As part of My Book I have a chapter on the challenges of integrating unit testing practices into an organization. One of the sections deals with "Tough questions and answers". In short, what are the tough questions you will have to deal with when introducing unit testing. here are the questions I've come across but I'd love to see if there are some that I am missing. Also, I am looking for great answers and references as responses to these questions. Your comments are appreciated: 1 How much time will this add to the current process? 2 Will my QA job be at risk because of this? 3 How do we know this is actually working? 4 How much time will this add to the current process? 5 Is there proof that unit testing helps? - any other tough questions? - Got good answers? Comments: I'm using a beta test of Disqus.com's comments INSIDE this post so please try to respond there: Discuss this topic on Disqus.
"Swapping" instead of "Injecting" calls between classes Eli Lopian, Typemock CEO and awesome Coder, just created a nice little API wrapper around Typemock Isolator that would allow a very simple and readable "Swapping" effect between classes. It would allow you to write code like this: [Test] public void SwapStatic_CallsFake() { Swap.StaticCalls<TypeA, TypeB>(); Assert.AreEqual(2, TypeA.StaticReturnOne()); Swap.Rollback(); } [TestMethod] public void SwapInstance_CallsFake() { Swap.NextNew<OriginalClass, FakeClass>(); OriginalClass swappedInstance = new OriginalClass(); Assert.AreEqual(2, swappedInstance.ReturnOne()); } The first test uses the "Swap" class to replace static calls and redirect them to your own class with static methods. The second one is more advanced and will mock all instance calls made on the next instance that will be created of that type. so the last line on the second test will actually call a method against the FakeClass type. pretty cool and very readable. download the code and binaries from his post.

ISerializable - Roy Osherove's Blog : Art Of Unit Testing

Unit Testing in silverlight land with Typemock Isolator

I’ve been asked quite a lot recently whether one can write unit tests and isolate logical code that runs inside a silverlight application. Up until today my initial answer was ‘no’ because silverlight runs under different versions of mscorlib.dll.

however.

Today I actually gave it a try and realized that writing unit tests (not integration tests, as the silverlight test framework allows) against silverlight based code is possible and quite easy. Just like any other code that relies on a third party platform (like sharepoint code) the silverlight related code might have various dependencies.

I’m going to show how to use Typemock Isolator to overcome a couple of simple silverlight dependencies (using HtmlPage) and how to setup a test project against a silverlight project (with NUnit or MsTest)

Assumptions:

You have an open solution
the solution contains a silverlight class library or silverlight application project

To setup a test project against silverlight using MSTest:

Add a new test project to the solution
Remove all the existing classes (Database test, ordered test etc..) so that you are only left with the unit test class (UnitTest1).
Remove all useless comments and crud code from the test class so that you are only left with a test method (no comments, not even the TestContext)
Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first)
Add a reference to the project under test
You can now write tests against the object model (standard classes)

To setup a test project against silverlight using NUnit or MbUnit:

Add a new class library to the solution as your test project
Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first)
Add a reference to the project under test
You can now write tests against the object model (standard classes)

How to break the dependencies

Now, here is a simple example of code that has a silverlight dependency we’d like to test:

Let’s say we have a class in the silverlight project called ChatSession (I’m basing this on ScottGu’s Chat demo). but it’s constructor looks like this:

What if we wanted to control during our test whether the page is enabled or disabled?

Here’s one way to do it using Isolator:

In your test project add two references under the .NET tab: “Typemock Isolator” and “Typemock ArrangeActAssert”
Write the following test:

Using Isolate.WhenCalled() we are able to circumvent any method (static or not) to return whatever we want, or throw an exception.

Here’s a more interesting case. Let’s say we have a method that modifies the current page and shows some html to the user in a span tag:

here is one way to write a test that makes sure that the right text is set into the message element in the page, without needing to have a real page present:

There are several things to note here:

HtmlElement is Sealed has an internal constructor, but we can still create a fake instance of it using Isolator. None of the other frameworks can do this.
We are returning a fake object from a method call chain (HtmlPage.Document.GetElementById ) without needing to create a separate stub for document. None of the other frameworks can do this (especially since it is based on a static method call to begin with)
We assert in the end that the method was indeed called with the correct arguments without needing to change a single piece of code. Granted, I wouldn’t write code like this (I like decoupling!) but sometimes you just don’t have the ability to change existing code.

If you are developing an open source silverlight project, it is important to note that there is a free full version of isolator for open source projects with full functionality.

These are just the beginning of my journey into silverlight unit testing. I’m looking for good code examples that you might want to test, with various dependencies that need breaking. your comments are appreciated.

Isolator Feature Focus: Duck Typing and Isolate.Swap

Typemock Isolator 5.1.1 has been released, and this release brings with it some awesome (seriously) features that are unique fro any other framework I've seen.

a good overview of features can be found here.

The Isolator Swap feature allows swapping calls between real and fake objects (kind like redirects) so that any relevant calls made against the real object will be redirected and invoked on the fake object.

Here's how you use it:

Unlike doing a standard "WhenCalled()" on some object method and telling what its custom behavior will be, "Swapping" objects redirects all relevant calls(I'll explain what "relevant" means in a second) to the fake object which you have created.

Duck Typing Awesomeness

Another cool thing about it is that "Duck" and "Dog" don't have to have a shared interface or base class. The "Swap" feature will redirect a method call if it exists on the "fake" object, but if it does not, it will invoke the original object. This is one interpretation of what's called "Duck Typing".

The "CallsOn" and "WithCallsTo" methods take an object type, so you can send in anything you want. if a method on the fake object matches the signature and name of a called method on the real object, the fake method will be invoked.

Art Of Unit Testing – Book done, Wiki ready

I’m happy to report that my book has finally hit the “last review” stone, which means all chapters are done and the book should be in book stores around jan-feb 2009. If you’d like to read the E-Book version right now(PDF) you can purchase that for a lower price at the manning site.

I’m also announcing a Wiki site for the book, with all the chapters listed, along with errata and your comments as time goes by. The site already contains the first paragraph of all chapters, with Chapter 1 available for free download, and the full text of the “Tools and frameworks” appendix.

Since its all in a wiki, feel free to add your own tools in there that you think would be valuable, create your own pages that talk about things you think people reading this book should read and more.

I hope the site becomes a good place for guidance for beginners and advanced in the field of unit testing (does not have to be in .NET).

Oh, and you can see the final book cover on the right.

Unit Testing decoupled from TDD as well== Adoption

The discussion on the future of unit testing for the masses has shifted from the standard “if they are too stupid to learn it, we don’t want them” to “TDD without good design will make really bad tests”. and this is a good thing. it’s a good thing because it secretly implies that if you could write Good Unit Tests, regardless of design, then teaching design as a separate thing should not pose a problem.

Discussion so far:

But first, let’s make one thing clear:

TDD is NOT Unit Testing.

My previous post was not “TDD decoupled from design==adoption”, it was “Unit testing decoupled from design == adoption

TDD is (technically) about when you write the unit test.

You can still write crappy tests in TDD mode. With good design. Unit Tests deserve a good book about them without TDD in it because the current books about TDD suffer from lack of SRP.

On the other hand you can still write good tests, without TDD. The design might make them a little longer, but if you learn to write unit tests well, this should not pose a problem.

Learn Good Unit Testing regardless of TDD and Design

people can start learning unit testing before learning TDD. they learn the silly stuff (the kind udi looks down upon) in an hour, and then they get to the hard stuff : making the tests readable, maintainable and trust worthy. these are separate properties that can be learned to be done regardless of design, since test naming convention, refactoring and correctness can be learned in separate.

The only reason it’s really hard to teach in separate today is because as those devs get to test real world code they have to start dealing with real world dependencies and that it the barrier that usually stops most people. having “intermediate” frameworks in place such as JMockit and Typemock Isolator can help in that regard, keeping people focused on the task at hand and deferring the need to learn other types of design for a time that is better. The idea is to let the developer choose when to learn which technique:

Learn how to write good unit tests (book)
Learn Good Design technique (give me a link to a good starter book!)
Learn Test Driven Development (book)

I want a developer to grok each one of those things to a very deep level, but today’s “market” wants them to learn it all in one go. Since each one is its own little art form, are you really surprised to learn that most of the mass population is not there yet?

If people think that the only way to write good unit tests that are not worthless is by doing them TDD style with SOLID design, they are showing several things to me:

they don’t get what “good” unit testing is
They don’t “get” what value good tests have outside the context of TDD
they guide a community who reads them down a path of “all or nothing”. which for most people begets “nothing”.

Field notes:

Udi writes:

“If logic is duplicated, if the code is tightly coupled, if there is no separation of concerns, the unit tests will be useless - even if they “test” the class in isolation.”

That is wrong in many levels. Let’s start with the fact that a unit test can have a specific value in testing a specific class in isolation, even if the bugs in that class are duplicated on other classes. If Udi does not agree on that, I don’t know what else to say about that. I wish that what he meant was that all these problems were repeating in the tests, since this is usually why people fail when they start out unit testing: they write crappy tests with bugs, code duplication and lack of isolation (this can all be learned). But he’s not. It feels like a “purist” view that does not reflect the real world. It must be nice living in an echo chamber!

Let’s see udi contradict himself in the same post:

“Managers shouldn’t necessarily sacrifice their projects on this altar of learning. Organizations need to find ways for developers to safely practice these techniques as a part of developing their “human resources”. First of all, this needs to be communicated to everyone - that the organization understands the importance of these techniques, the desires of developers to adopt them, and the projects that need to be delivered.”

“…It’s a gradual process.”

That’s the problem Udi. It’s not possible to do it gradually today, because people keep saying that you can’t do things in little chunks. you have to learn it all together. If we were to help developers learn what good unit testing means they could much more easily learn that stuff, and then learn the other stuff.

Casey writes:

“It is not a technical challenge to write a unit test, any half trained monkey could do it. A few attributes, a couple of classes, a few new() statements, and a few Assert statements. We don't need magic mocking frameworks, we don't need any mocking frameworks for 90% of our tests, we don't need application of 12 GoF principles before we can write a test, honestly.

Testing and writing automated tests is hard because it requires a strong understanding of the user stories that sit behind your code and project, because it requires you to think as a user and not as a developer, and because it requires you to be able to think of multiple conflicting axioms, test cases and class interactions, and how to cover the weaknesses of each individual test with other complementary tests.”

Casey preached about what he sees as the “end game” for unit testing, TDD and Design. where everything is in one great ball of goodness. Or did he?

Let’s separate these points into better chunks:

Design related stuff:

to think of multiple… class interactions

Good Test related stuff:

Understanding of the user stories
Think as a user not a developer
think of multiple conflicting axioms (..?) test cases..
how to cover the weaknesses of each individual test with other complementation tests.

TDD related stuff

none

So, what I gather from casey’s words is that you want your tests to be :

testing the right thing
be readable and show intent
have good coverage
test or be aware of class interactions (the design related stuff)

except the last point which is where most of today’s mock frameworks force you to redesign, there is nothing here you can’t learn separated from TDD and SOLID techniques. the interaction stuff is where those other isolation frameworks I mentioned come into place.

We need to decouple Good Unit Testing from TDD and SOLID. And it seems that not only “noobs”, as a stupid idiot commented thoughtfully on my last post, can benefit from learning this distinction.

Unit Testing decoupled from Design == Adoption

In my previous post I started talking in more coherent words about feelings I’ve had lurking in the past year. This post is an expansion of that, after having some great twitter conversations in the past few days with some great people. Treat this as a stream of consciousness.

before that, here’s a short highlight summary of responses to my post in other blogs:

there are also plenty of people who picked up the ball (some of them have already done that before I started mumbling on my blog) and continue to make better attempts at bringing knowledge to the masses, in more coherent, newbie friendly manner.

Rhys shares some hints and gotchas
Ian Cooper gets it (even if he doesn’t think he does – I’ll explain later)
Jeremy is asking important questions
Jimmy has good tips

and the comments on that post are… interesting (some are very expected, though).

Let’s make another interesting statement, and this one is indeed product related.

To reach the masses, we need to decouple S.O.L.I.D Design Principles from Unit Testing and TDD

I’m not saying throw out design. I’m saying – it’s important enough to be learnt on its own, and not in the context of tests. in fact, in that context, it inhibits too many people to be practical as a learning tool. so we need to separate forces.

But why?

The wall to start unit testing is too high (for most devs) and will always be too high if you have to know your SOLID in order to even start writing your first test or refactor your code for testability. This problem only exists in static languages like Java, C# and VB.NET, where in order to replace something you need to have a way to do polymorphism on that class (interface, or inheritance with virtual methods).

The reason people love Test Driven design so much is because the technique, aided by most of today’s leading mock frameworks, squeezes out all the problematic static coupling that exists in the current design, before it will even let you test something. Can’t decouple that web service dependency from your class? you’re screwed. Redesign and only then you can test it. Have a static method that gets called? you’ll need to jump hoops to override the calls to that method, let alone the method itself. Feel the burn, baby!

TDD existed before mocking frameworks existed. what did people do then? SmallTalk was one of the first languages that had people doing TDD with it. SmallTalk is testable by default in that you can re-define anything to do anything else at runtime. People just thought about their design as they wrote the tests first, and they had no real constraint on what specific design to follow. It still holds today in Ruby, for example.

Now, the reason today’s “hardcore” Test Driven Design is even possible is because current mocking frameworks rely on static language (and compiler) to tell the developer what can and cannot be done in terms of replacing objects and implementations. That forces a specific design methodology that closely matches what SOLID principles stand for, and it’s great. you get great code out of it, and nice designs.

I repeat – I love the idea of SOLID design and I practice it.

So, why do I think that these things should be decoupled? At the end of the day it is a constraint that is put upon each developer that wants to start unit testing (not even talking about legacy code right now). That means that each developer that starts tests will hit a brick wall and will have to start reading up on all things SOLID, patterns, or just buy my book. There is no “easy” way. and that’s fine and dandy for most of the people reading this blog. And for the past four or five years, it’s been just as fine and dandy for me too.

BUT, The adoption rate for TDD and unit testing at all is low, and I suspect that it is indeed this learning curve that makes that possible. And it will continue to be low as long as doing great (not OK, great!) designs is a prerequisite to doing proper unit testing (or at all). The current mock frameworks (most) enable this notion and embrace it.

Today we live in our own little echo chamber where everyone is happy with their “pure” designs and consultant fees, because we are part of a very elite group: those you need to pay in order to learn the “hard stuff” to do something which many people want to do but just can’t. It’s almost like knowing Regular Expressions, except a much larger scope. We’re making it harder on purpose to start doing unit testing, because we believe that it’s someone’s best interest (and in a way, in our best interest) to learn SOLID first.

The people I know from the mocking community are amazing, passionate and smart people. They would never do anything they don’t fully believe in, and their value system is almost radically pure. And I love that. but we cannot play judge here. people learn in different ways, and the low adoption rate tells us that it’s not working as well as expected.

Why would decoupling work?

If we were able to decouple the requirement of knowing SOLID and just get people to start testing today and learn SOLID later or in parallel, adoption would be much higher. Why?

People who get test infected soon want to learn SOLID anyway
People who succeed doing one thing, will be more open to adopting other things from the same source of knowledge
It’s less scary, and less time consuming to start doing things. We reduce the barrier to entry into the agile world.
It allows developers in a non-agile environment to do little,incremental steps on the road to a better way of working, without needing to chow down the whole meal in one big bite.
Sucky tests are better than no tests. They also lead to better tests and better understanding of design as people learn and get better at it.

What’s stopping us?

But with the current set of mock frameworks, you can’t do all those things. you technically are not allowed to do that. and that’s why solutions like Rhino, Moq, and NMock will not appeal to the masses, and why solutions that support other types of design decisions (JMockit, Typemock Isolator) will continue to gain popularity as people strive to become more agile under non-agile conditions. they provide that needed stepping stone to learn unit testing without needing to learn SOLID as a requirement. That’s the difference. But the community is actively depressing thoughts about using such tools today because of the belief that TDD and design have to go hand in hand at ALL times.

Many of them that will read this are scared that I’m even suggesting this, because they fear a return to darker ages where people didn’t have a clue, and there was nothing to stop them from screwing up the design. But as you’ll see later on, the plan is to get people on board the design stuff, just not as a prerequisite to the unit testing stuff. that’s all.

Of course, the big problem here is that I actually work at Typemock, and thus, many people feel that every word that comes out of my mouth is a sales pitch. but the reality is that if Typemock had been an open source project, those people would have read the words that I write, and not try to find a hidden meaning in them. and the truth is I’d write something like this even if I didn’t work there today. I’d quit to say these words with more credibility but I want to have an impact on how those products work and feel, so I’m not going to do that, sorry.

What does the industry need to do?

Direct people who are reluctant to go through that big learning curve or have a problem with the current frameworks to tools that mitigate those fears (Typemock, JMockit). One is open source, the other has an open source free license. so they are both achievable to use and play with.
When doing so, learning better design techniques should be on the horizon. but it’s important not to discourage use of these frameworks on existing non SOLID design notions. people have a hard time with change as it is. They need to learn one step at a time and we have to let them have the time to grok things separately if they want to.
Decouple the task of teaching unit testing from the task of teaching design. They are both important but are not a requirement of each other. We need more Design material out there with a simple way to reach and find it (we’re on the way today). Unit testing information is already out there and more is going out every day.
Realize that tools that enable different designs are not a problem, but a way to get a foot in the door with developers who wouldn’t walk through that agile door in other ways. it’s a bigger net to catch more fish. These tools allow people to have a bridge to the agile world with less pain involved.

This is something that we need to continue talking about, since it can change the future for most devs today. If we shy away from talking about this, and stay in our cozy little submarine underwater, we will sail right beneath all those people lying on the beach, wondering what’s up with all those people in the water, or afraid to take a swimming lesson. (yeah, the world’s worst analogy)

Thoughts? Comments?

Goodbye mocks, Farewell stubs

About 4 years ago I was consulting for one of the largest companies in Israel (about 15,000+ employees) and helping them grok TDD and automated unit testing. I had spent there over a year and a half, on and off, and was teaching both .net, C++ and java developers about the techniques and tools they can use.

I was consulting for C++, Java and .NET developers.

The C++ devs were first.

They had plenty of objections to what I was teaching them, because the amount of code refactoring they needed to do in order to make their code testable was almost unrealistic for them. Even writing new code meant writing something on top of old-untestable code, and they had a hard time adjusting to the more interface (pure virtual) based, contract driven TDD approach for new code. C++ devs has a hard time becuase it takes more code to write good testable code, and the coding tools they used (VS on windows and VI and such on UNIX based systems) didnt’ provide a good productive way to make code “bend” easily as they refactored it or created it in the first place. every little change meant a big “search and replace”. every method refactoring meant adding it the declaration in an .h file and thus they tried to have a little to do with changing code as possible.

Looking back, today they still don’t have the right infrastructure in place or the right tooling or the willingness to do unit testing or TDD becuase it’s too hard with the current tooling to accomplish it.

I had also spent some time with the .NET devs

and thought them the same things. Becuase tooling support was better it was a bit easier for them to accept the practices, but still people had a lot of trouble grasping concepts such as dependency injection, inversion of control and designing for testability in general. I found that most people didn’t really have a big problem with the idea of exposing things they didn’t use to expose before, it was more about the concept of learning how to refactor the code in interesting OO ways to make it testable. Making a class non sealed, a method virtual, and then inheriting from that class and testing against the inherited class (“extract and override”) was very much unintuitive for them, and other concepts like containers (spring at the time) didn’t help them get into the groove either. For them it was like drinking water from a fire hose.

Consulting for the java devs

Java has lots going for it in terms of unit testing, automation and TDD

Java devs have a much longer “heritage” of tooling and unit testing than in the .NET world, and so you’d expect them to be more inclined to do these things.
JUnit is part of any decent java idea so the barrier for entry should be lower, right?
Refactoring was also in most decent java IDE's, so you’d expect them to grok that too. but that wasn’t the case.
class methods in java are virtual by default so you can override what you want as long as the class is not sealed (final)== more testable by default.
Java required a much higher OO learning curve to start with. Good OO knowledge is fundamental for testable designs.

Unfortunately, most developers I met didn’t even know how to use the refactoring tools they had available and had a hard time with the same dependecy injection principles that the C++ and .NET devs had. It was just too cumbersome to make the code testable, but eventually the infa team at the company had made the decision to take a couple of weeks and make all the infra stuff replaceable for testing purposes. they did not regret that decision.

So the java group had the most success with testable designs, but they still had a hard time doing it. in fact, only the infra team at that group took the plunge. the other (higher level) devs in the java group didn’t grok it. and don’t use it today.

Why was this adoption been such a failure? They were willing to learn, but they didn’t like what they learned. It didn’t fit their needs in the real world. Or the way they worked, or the way they knew how to do things.

Think what you will ,but 99% of the dev worlds aren’t ready for the “testability” message. and that is a fact that is hard to realize for most of us, which just can’t grasp why isn’t that any sensible human being isn’t willing to do everything much better than they are doing today.

Spending the past week running around the Microsoft Campus in Redmond had made several things clearer for me. I’ve been feeling them for a while, but this week these abstract feelings materialized into something that is more coherent that I can talk about.

Why do most developers not write unit tests, still?

Reason #1: Learning curve

One of the main reasons most developers still today don’t really take to unit testing (and TDD) is the really high learning curve that is forced upon them. That is not to say that learning about

mocks, stubs, dependency injection, IoC containers, Extract & override technique, Record\Replay, AAA and more

-- is not useful, but it is a big obstacle to get people into the habit of doing something which they know is good for them – test their code, verify their assumptions, automate, integrate, be confident, get feedback, define behavior – all those and many other benefits are being thrown away by developers all over the world because the entry fee to this world is too high, for all the wrong reasons.

It’s time to simplify. Get back to basics. Take a good hard look at the huge spiral that we are leading people into and asking whether removing some things today can lead to those things being accepted and being easier tomorrow. or maybe they weren’t needed at all.

Say Goodbye to “mock” and “stub”

“mock” and “stub” have played their part in us trying to explain to ourselves what it is we are trying to do, to start learning the minute and important differences in the patterns of our test code. They helped us define what it is we are looking to do, in terms that, for lack of anything better, are now the de-facto pattern names for something that should never have been named in the first place for the casual programmer.

That’s why in the new version of Isolator we’ve chosen to completely get rid of the words mock and stub, and get back into “fake”ing. You can fake a method or a class instance, and you can verify that something was called. You should not care whether this is a stub or a mock.

We need to clean our language: mock, stub, fake, test double: They all mean something in the context of us trying to identify patterns in our test code. But as users mature and start using the frameworks, those same words increase the entry barrier, and effectively block people out. If I’m showing a simple test that fakes some dependency that a class uses to a new dev on my team, I should not have to send them over to read fowler’s “mocks arn’t stubs” article (which is enlightening, but should not be relevant to them) because it misses the point.

Less interaction testing, more state based testing

Most crappy unit tests I’ve seen were over specified in that they asserted on internal interaction with an object way too heavily, which made the tests very fragile. Getting back to basics means making it easy to enable state based testing (assert style checks) and only enabling interaction testing where it makes sense: (when your app does one-way messaging to an external component). verifying that something was called should be the exception to the unit testing rule. The AAA style APis make that a more clear distinction.

Record-replay APIs make it too easy to make interaction based tests when you really want a state based one (getting a result vs. checking that something was called)

Use AAA style Apis

As an industry we are learning and changing all the time. The “Arrange-act-assert” pattern in the field of mock frameworks represents us trying to return to basics. Moq did that nicely by showing a way that leads there, and it is being taken gladly by Rhino and Typemock Isolator. It is taking off because we are starting to look beyond our own little echo chamber and seeing what is is that all those other 99% of devs are expecting as guidance, and in what form.

AAA is guidance because there was an outcry (a silent, vote of feet) that record-replay is not cutting it – it’s too confusing for non-experts. It’s too technical. and it does not lend itself to the way people want to work. So most people didn’t use it.

Visual Basic Language Support

Everyone screwed up on this one.most VB developers who will try to use one of the current frameworks will not succeed since they required anonymous methods or lambda expressions. Some frameworks use VB specific keywords which makes them VERY awkward to use. If we want broad acceptance of our practices we need to make sure whoever wants to can use the tooling we make. VB is huge and its amazing how we all pretend it does not exist. But I think it’s the key to getting to those people who want to be better but all they see is a high wall instead of an entrance.

Use “Isolate” and “fake” instead of “mock”

Don’t say ‘mock’ frameworks, say “isolation” frameworks. Because ‘Mock’ is an overloaded word it is confusing. Explaining to a developer that you can “isolate” something from its dependencies rather than “mock” its dependencies feels more natural.

Simplify,Simplify,Simplify

Remove APIs instead of adding to them. Have single point of entries to your frameworks because discoverability is the key to good framework use. That’s why extension methods are problematic, they mean multiple points of entry.

Be open: Realize that other tools and techniques can act as bridges to the agile world

Design for testability is just one way to get where you want to go. it is not the only way. Sometimes it’s impossible. Discouraging the use of other ways of working (using tools like Typemock Isolator or JMockit) will only make the current echo chamber more closed and feel more un-achievable for those who need to get to the same point with a very different set of constraints than what most of the “loud” voices need to face.

By “allowing” other forms of working you enable those who want to, to get a bridge to help them instead of walking by foot the whole way. Any other way would mean that we will all fail in delivering what we believe in to those who have no idea what we’re talking about.

Over Specification in Tests

I've gotten the chance to visit and consult various companies and I see this all the time: Over specification in tests, especially when doing mocks and stubs.

One main reason for that is that people use their stub as a mock object (stubs are fake objects that we will not assert on so that we can test something else). here is a small example from something posted on the alt.net mailing list:

[Test]

public void Can_Return_All_Plans()

{

IPlanDao mockPlanDao = MockRepository.GenerateMock<IPlanDao>();

mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan});

PlanController planController = new PlanControllerForTesting(_mockCommonDao, _mockCommonService, mockPlanDao);

ViewResult result = planController.All(false);

mockPlanDao.VerifyAllExpectations();

result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans");

}

in this test the following line is used to "stub" out a return value into the system under test.

mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan});

no problem here. The problem lies in the last two lines of the test:

mockPlanDao.VerifyAllExpectations();

result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans");

the second line (the assert) is probably the thing you really want to test. but the call to "verifyAllExpectations" is the one that makes the test very fragile. It is "asserting" that someone has actually called "GetAllPlans()" to get the results, when it clearly does not matter how one got the results. all that matters is that:

"Given a set of results returned into the application under test, the view in the controller is correct" (the translation of the last line in the test).

If by any chance the application later does several more calls to the stubbed out dao, or uses a different method to get results, the test could break, even though the application would still ultimately work with the same end result.

It's like me ordering a pizza delivery and then asking the delivery guy "did you get here by car or motorcycle?" . I realy don't care, as long as the end result: PIZZA, is correct.

So the general rules for me are:

If you see both "Verify" and "Assert" in the same test, it is usualy a smell of over specified tests
if you see "expect" and "Return" on a fake object, make sure that you name it "stubXX" or "mockXX" so that you can distinguish whether you want to call verify on it later or not(most cases should be "not")
Try to test on the end result or end state rather than verify interactions. The only time you absolutely have not choice but to test an interaction using verify is when calling void methods on external objects. that is clearly a one-way communication and is (or part of)the end result of what you are trying to accomplish.

A lot of people think that adding that extra "verify" just means it is a good thing since they are doing more assert. Well, they sure are testing more things, but they are internal things to the app's behavior and that is usually leading to brittle tests. Try to test on the end result or end state rather than verify interactions.

A sneak Preview of Typemock Racer : Thread Deadlock Finder

Here's a sneak preview of the upcoming Typemock Racer product, which I'm currently working on. We should be out in private CTP in a couple of weeks I hope.

Typemock Racer tries to solve one huge problem for developers working in multi threaded environments: Detecting code that can result in deadlocks or race conditions. It does this by providing a framework API which you can use in your tests (under NUnit for example), that allows you to execrise your code under test until it finds a deadlock or times out.

Here's an example of how you'd use it. Consider the following class which has two methods, that if run by two seperate threads, can result in a deadlock condition:

At some point in time the threads could execute such that thread 1 is waiting on resource b, which is held by thread two , which is waiting on resource a, held by thread 1 - a pure deadlock.

Given this class we can write a test that looks like this:

The ThreadTest class has an "AddThreadAction" method that receives a delegate. you can use it to invoke your own code, it will be invoked in a different thread. IN this example we are creating two threads, each one running a different method on MyClassWithLocks.

We can just right click and run the test with TestDriven.NET to recieve the following results:

The output states that there is a deadlock that was found, with the exact steps that happened (with line locations) to reproduce this.

It currently only works with lock (monitor.Enter\Exit) but will support all locking constructs in .NET.

What if the methods you point to create their own threads? Like this:

Here is the output that you'd get if you run this test:

Introducing Depender - Testability Problem Finder

Update: Here's how you can extend Depender with your own dependency rules

Depender is a tool that I created to illustrate several concepts both for my book and both for my training classes. I think it should prove valuable (as it gets more and more features) to anyone who is trying to write tests for existing code.

Depender allows you to load an assembly and inspect the types within it according to very simple rules. the main question on my mind and I run it is: "How easy would it be to test class X?". Depender will list all the classes in the assembly and for each one will try to detect possible testability problems (non decoupling) and places where you might be able to "inject" your own implementation (calls to virtual methods, fields that are interfaces or types who are not sealed..etc..)

so you can:

see places where you might want to refactor your code or use a tool like Typemock Isolator
see places where you can already use "seams" to inject behavior into your code without needing to look up the source code manually.

here's what the main screen looks like when loading an assembly:

the type "RealLogger" is marked in red because it contains calls to static methods. double clicking on the type's node will load a separate window just for this type:

I will be adding more rules and also releasing the source so that other people can help develop this. for now I just want to get it out there and see what people think.

possible futures:

Make it command line driven so it can run in your CI builds
Make it give "scores" to assemblies or types on testability and decoupling
Make it add "suggestions" on how to fix decoupling issues
Make it create the base diagram for my test feasibility matrix

Download Depender from here.

Over-testable systems, and mocks as bad test smells

This is an interesting problem I've been running into at work. Sometimes we would try to write a test against the system or a component and we would literally have too many choices of how to write it or where to "intercept" or "inject" our stubs or mock objects. In a way, the software is almost "too testable" in that there is usually at least 3 ways to accomplish a similar level of isolation.

What it comes down to is the fact that for a real system, there are different "depth" layers where indirection can be injected. For example, Component X needs object Y that relies on object Z. If we were testing component X we could "replace" either Y or Z. Some would argue that replacing one level too deep is making this an integration test, but as long as its all running in memory I don't see a real problem with that. I gave up on trying to do "pure" unit tests for 100% of the time a long time ago.

Many times, it is much more complex than this. For example, replacing object Y instead of Z (at the closes layer to the object under test) can also lead to problems if:

The contracts between X and Y are very chatty, than stubbing out Y could lead to a very hard to read or very fragile test
You are trying to test that interaction occurs between Y and X than that might be an over specified test. If replacing Z would lead to a more state-base test that I'd choose that over replacing X. (shortly: I'd rather use stubs over mocks so that my tests become less fragile. some layers require mocking and some are very simple to just stub out)

Here's a thought or two:

if you one one layer farther away than the one you are using, and it does not require mocks to test, that try to use that layer and forget the inner layer?

If you treat mocks as a smell for over specified tests that will force you to find a way to test the same thing using a stub. I find that sometimes you have no other choice but use mocks (when your app under test has a one way out communication with a component) but in all other cases, I am starting to look at it like a design smell for my tests (over specified tests)

getting back to the original problem (too many ways to test the system) I am trying to think if this is a good or bad thing. The only bad here is that someone who is testing the system for the first time has no real guidance on where to replace things (except look at other tests) which can make writing the first few tests really hard to figure out.

The fact that we are using a sort of auto mocking container in our tests is both a blessing and a bit of a curse. Currently i find that it saves more time that it takes away. the tests and their dependencies are relatively more readable since we are using a more explicit version of a test container that requires you to explicitly state what will be replaced (it just doesn't tell you who requires that component).

I can already hear the pundits say "well that means you have a design smell". The only thing that smells here to me is that because we followed strict DI and interface based injection rules, we ended up with a system that is not just made of lego parts (a good thing), the lego parts or too darn small!

Unit Testing Interview at TechEd

My interview at teched us on advanced unit testing and legacy code can be seen here:

http://msdn.microsoft.com/en-us/events/teched/cc676818.aspx

or downloaded here

We occasionally stop and laugh a bit because "speaker idol" contest was being held at the booth next to us, which was really really loud.

Fit/Fitnesse Fixture Gallery -Real world help for FIT and Fitnesse users

This great news was just published at the alt.net mailing list. putting it here for future reference.

"Fixture Gallery 2.0 (release 2008-06-09) is now available for download
from SourceForge.
Fixture Gallery is a cookbook for FIT/FitNesse tests. It provides
developers with a quick overview of the most important fixture types and
concepts for agile acceptance testing using the FIT framework. For each
fixture type, this document explains the table format and fixture class
structure and provides advice when to use and when not to use it. Each
example is accompanied by the source code for Java, .NET and Python FIT
implementations, in a form that can be easily copied and used as a
template for similar fixtures.
Version 2.0 brings:
- Python examples for all fixtures and concepts explained in the document.
- Examples for ConstraintFixture, CombinationFixture, CalculateFixture
and notes about SetFixture and SubSetFixture.
Many thanks to Jens Engel and Mike Stockdale for their contributions to
this version.
For more information and download links see
http://gojko.net/fitnesse/fixturegallery
--
gojko adzic
http://gojko.net

Unit testing vs. Integration Testing : The Restaurant Analogy

Here's an analogy I came up with to explain the difference between unit tests and integration tests.

Imagine going to a restaurant with a bunch of your friends. there are some couples there as well as singles. At the end of the meal, it is time to pay. Unit testing is like giving a separate bill to each couple or single. Each person knows only what they needs to pay as well as tip, but do not care what anyone else had, or what the total meal amount is. as long as they pay what they need, they can go home peacefully.

Integration testing is like giving one big check to the group, with everyone having to calculate on their own how much they need to pay. sometimes everyone things they are good, but in the end the total amount may be too high or too low. There is a lot of interaction going on to decide who pays for what and who has already paid.

Tough Questions about unit testing - Your comments required

As part of My Book I have a chapter on the challenges of integrating unit testing practices into an organization. One of the sections deals with "Tough questions and answers". In short, what are the tough questions you will have to deal with when introducing unit testing.

here are the questions I've come across but I'd love to see if there are some that I am missing. Also, I am looking for great answers and references as responses to these questions. Your comments are appreciated:

1 How much time will this add to the current process?
2 Will my QA job be at risk because of this?
3 How do we know this is actually working?
4 How much time will this add to the current process?
5 Is there proof that unit testing helps?

- any other tough questions?

- Got good answers?

Comments:

I'm using a beta test of Disqus.com's comments INSIDE this post so please try to respond there:

Discuss this topic on Disqus.

"Swapping" instead of "Injecting" calls between classes

Eli Lopian, Typemock CEO and awesome Coder, just created a nice little API wrapper around Typemock Isolator that would allow a very simple and readable "Swapping" effect between classes. It would allow you to write code like this:

    [Test]
       public void SwapStatic_CallsFake()
       {
           Swap.StaticCalls<TypeA, TypeB>(); 
           Assert.AreEqual(2, TypeA.StaticReturnOne()); 
           Swap.Rollback();
       }
 
       [TestMethod]
       public void SwapInstance_CallsFake()
       {
           Swap.NextNew<OriginalClass, FakeClass>();
           OriginalClass swappedInstance = new OriginalClass();
           Assert.AreEqual(2, swappedInstance.ReturnOne());
       }

The first test uses the "Swap" class to replace static calls and redirect them to your own class with static methods. The second one is more advanced and will mock all instance calls made on the next instance that will be created of that type. so the last line on the second test will actually call a method against the FakeClass type.

pretty cool and very readable. download the code and binaries from his post.

Retrieved from "http://www.artofunittesting.com/Blog"

Tag page
What links here

Files 0

Attach file or image

Images 0

Viewing 1 of 1 comments: view all

omar says:

Some of the posts in the blog or so relating to the book's overall message that I feel the links for some of these blog entries have to be referenced at appropriately related content in the book. IMHO the distinction of Good Unit Test from TDD and SOLID is crucial and in my own experience it was the reason for not adopting unit testing all this time (20+ years in the field!!)

Posted 18:56, 17 Oct 2008

Viewing 1 of 1 comments: view all

You must login to post a comment.

Book Blog

Table of contents

Files 0

Images 0

Table of contents