<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 2, 2014 at 10:50 AM, Željko Filipin <span dir="ltr"><<a href="mailto:zfilipin@wikimedia.org" target="_blank">zfilipin@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi <span style="font-family:arial,sans-serif;font-size:13px">Nik,</span><div><font face="arial, sans-serif"><br>
</font></div><div><font face="arial, sans-serif">apologies for the late reply, I was on vacation. Comments are inline.<br>
</font><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On Sat, Dec 21, 2013 at 8:18 PM, Nikolas Everett <span dir="ltr"><<a href="mailto:neverett@wikimedia.org" target="_blank">neverett@wikimedia.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>1. What do you do with developers that think it is beneath them to write integration tests? Unit tests?<br></div></div></div></div></blockquote><div><br></div></div><div>I would also ask why would a developer think that. All code is created equal, right? Writing test code should be not different than writing production code. Or am I wrong?</div>
</div></div></div></div></blockquote><div><br></div><div>I've worked with plenty of developers who won't write browser tests. They've been sold on unit testing but won't pick up selenium because it is either beneath them or too complicated or they just don't feel they have time to learn another skill. Or some combination of all of that.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div></div><div>2. Where should you store those tests? (We've mostly settled on next to the production code which seems like the right answer to me for lots of reasons.)<br></div></div></div></div></blockquote><div>
<br></div></div><div>+1</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra">
<div class="gmail_quote"><div>
</div><div>2. What code/features/changes should have unit tests? Integration tests? What is the difference, any way?<br></div></div></div></div></blockquote><div><br></div></div><div>Looks like you have two questions with number 2. :)</div>
</div></div></div></div></blockquote><div><br></div><div>Numbers are hard.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div><div class="gmail_extra"><div class="gmail_quote">
<div><br></div><div>As a rule of thumb, code that is more likely to break, that is hard to understand, that is written recently... should have more tests than other code.</div><div><br></div><div>I would say that most (if not all code) should have unit tests, and things that are hard or impossible to test with an unit test should be tested on a higher level.</div>
<div><br></div><div>The difference?</div><div><br></div><div>"A test is not a unit test if:</div><br>- It talks to the database<br>- It communicates across the network<br>- It touches the file system<br>- It can't run at the same time as any of your other unit tests<br>
- You have to do special things to your environment (such as editing config files) to run it."[1]<div class="im"><div></div></div></div></div></div></div></blockquote><div><br></div><div>I'd argue that lots that defining what your unit is in unit testing is super important and very easy to fail to do right. If you do it wrong you write unit tests that don't prove anything. I've seen developers grow to hate unit testing because they realize that they aren't proving anything but refuse to grow their units. I have a single rule of thumb which I've decided to call the Magic Rule:<br>
</div><div>If code relies on complicated invocations to an external system to do a major portion of its work then the external system must be part of the unit. You can't prove Harry Potter's magic works by shouting spells and swishing around a stick.<br>
</div><div><br></div><div>Or thought of in a different way:<br>The value of a test is inversely proportional to the number of mocks it interacts with.<br><br></div><div>You can replace "complicated invocations to an external system" with stuff like:<br>
If code relies on SQL more complicated than "SELECT * FROM foo WHERE foo_id = ?" to
do a major portion of its work then the database must be part of
the unit. That includes the target version of the database, the production schema, and the best (as ugly as production) test data you can get.<br></div><div>If code relies on the vagueries of posix file operations then the file system must be part of the unit.<br>
</div><br></div><div class="gmail_quote"><div>When you follow this rule you end up with more integration tests then normal. Sometimes you don't end up with any unit tests at all. To me, that is OK.<br></div><div><br>
</div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote">
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>3. How do you make sure you stick to #3?<br></div></div></div></div></blockquote><div><br></div></div><div>Well, this is #3, but I guess you were asking about previous question. I am not sure what to do.</div>
</div></div></div></div></blockquote><div><br></div><div>Yeah that was about the last questions. Lots of corporate shops measure unit test coverage. Measure it and stop there. Some fail builds if the coverage isn't above a certain level. Lots of stuff. I'm of the opinion that if you measure unit test coverage you ought to measure integration test coverage too. While those figures themselves are neat the union is actually more important.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div>4. Where does documentation live? Do you build documentation from tests? Do you build tests from documentation? (Which one of these do we do or is it even listed?)<br>
</div></div></div></div></blockquote><div><br></div></div><div>I have been struggling with this myself. Feature files are one form of documentation.</div></div></div></div></div></blockquote><div><br></div><div>Elasticsearch doesn't use its test as documentation at all. They instead force developers who submit pull requests to always write the feature, the tests, and the documentation all in the same pull request. I like this method because it keeps you honest. If the tests can build some or all of the documentation then that is better.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>
</div><div>5. If integration testing is really just another speciality like databases what does this mean for people that have spent years doing one or the other?<br></div></div></div></div></blockquote><div><br></div></div>
<div>
I am not sure what you ask here.</div></div></div></div></div></blockquote><div><br></div><div>If the brave new world of everyone just being an engineer comes about, what happens to the folks with 20 years of testing experience and none of writing production code? They transition to being an expert in testing who writes some production code, but how? What does that mean from culture standpoint?<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<div class="gmail_extra"><div class="gmail_quote"><div></div><div>6. What do you do when your organization doesn't jib with this reality? (WMF has a QA team when it might make more sense to matrix integration test specialists into teams to teach them how to write integration tests, for example.)<br>
</div></div></div></div></blockquote><div><br></div></div><div>We are moving towards integrating QA team into other teams. The problem is that there is just a few of us and a lot of teams.</div></div></div></div></div></blockquote>
<div><br></div><div>Platform doesn't make that easy either because we aren't really a single team. I mean, we have a single manager and meet once a week but we actually break into smaller teams to do projects. Some last a long time like search but some are only a few weeks or days.<br>
<br></div><div>I wonder, also, who because a member of the QA team if everyone is an engineer? I don't think just writing integration tests or browser tests is good enough because we really want almost everyone doing that.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>
</div><div>7. Holy cow how does this all line up with volunteers who only want to do one thing or the other and we really should take any help we can get.<br></div></div></div></div></blockquote><div><br></div></div><div>
I do not think it will be trivial, but I am pretty sure we can handle volunteers that just want to test or just want to code. :)</div><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div></div><div>8. Who pays attention to build failures and what do they do about them?<br></div></div></div></div></blockquote><div><br></div></div><div>A lot of people actually. Mobile team, language team, QA team... When a build fails, we fix it.</div>
<div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>
</div><div>9. What system, exactly, are those build failures testing?<br></div></div></div></div></blockquote><div><br></div></div><div>A build can fail for several reasons. We had an entire workshop about it[2]. <br></div>
</div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote">
<div class="im"><div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>Now that I think about it, who is coming to the architecture summit? These kinds of questions would be pretty interesting to talk about and might deserve a (late) RFC.<br></div></div></div></div>
</blockquote></div></div><br></div></div><div class="gmail_extra">I am not sure if anybody from QA team is coming.</div></div></blockquote><div><br></div><div>That is a shame because these questions are interesting. Not because we don't have answers to them but because those answers matter more than the rest of WMF really pays attention to. At least that is what having no one from QA at the architecture summit makes me think.<br>
<br></div><div>Nik<br></div></div></div></div>