Teamcity – Automate to find Broken Links on your website

If you work on a website , one of the smoke tests for your website could be to check that no links on your site are broken. And if the CI server can do this on every check-in there is nothing better than that as you can keep an eye it for every change. Further repurposing such a test as a smoke test for every deployment or even for monitoring purposes is extremely useful productive - good luck doing this manually  (When done on Live make sure there is a filter in Google Analytics for your test machines to be excluded)

Off the many tools available out there, the application at http://linkchecker.sourceforge.net/ seems to fit the bill for CI, Supports several modes see docs for more info

Download it and install it on your development machine

To run the test locally (should install linkchecker-8.1 on the machine), you could run the following command eg."C:\Program Files (x86)\LinkChecker\linkchecker.exe" http://sometesturl.com/ --file-output=html/report.html --recursion-level=-1

This has to be installed on all TeamCity agents. To hook up a build add a build step as below which is equivalent to the command above

This will allow you to hook a build on Teamcity to run a test which spiders through your site checking for broken links.

The command above produces a report with the name linkchecker-report.html in the agents working folder. If you push this html as an artifact and add a report tab on the Teamcity server called “Link Checker Report” for the linkchecker-report.html artifiac, you should be able to see the report after every build is run

For broken links the build will fail as linkchecker will return a code 1 on the command line, for success builds it returns a 0

See below for an example report which uses report tabs on teamcity to show the report

Running SOAP UI Tests in Teamcity using MSBuild

One of the web services we have has a bunch of SOAP UI Tests. I wanted to make sure when this was run on the build server we have good feedback for each test case rather than a whole build running a pack of tests and telling you if it failed or passed. Kind of a reason i was moving this build away to Teamcity from cruise control

The MSBuild script is as below. I had to define the variables I need and the Directory where all project files for Soap UI can be found.

   1: <Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />

   2: <PropertyGroup>

   3:     <SoapUIDir>$(MSBuildProjectDirectory)\SoapUI</SoapUIDir>

   4:     <SoapTestTool>C:\Program Files\eviware\soapUI-4.0.0\bin\testrunner.bat</SoapTestTool>

   5: </PropertyGroup>

The next thing I had to do was run each project file using the SOAP Ui testrunner.bat file.

The options I have used –j will ensure JUnit style reports are pushed out of the Soap UI test runner, for local builds you could just make it push out Html reports

-h allows you to specify the host header to use for your Urls, The host header you supply here will override what is stored in the SOAP UI projects , so you run these tests on different sites based on your environment.

I have done a FileUpdate because i wanted to change Urls of the service from the old server to the ones on the new servers, you wont need this and it can be ignored

   1:  

   2: <Target Name="ConfigureSoapUITests">

   3:    <MakeDir Directories="$(SoapUIDir)\Report"/>    

   4:    <ItemGroup>

   5:      <SoapUIProjectFiles Include="$(SoapUIDir)\*-soapui-project.xml"/>      

   6:    </ItemGroup>

   7:    <!-- Replace old server urls with new ones -->

   8:    <FileUpdate Files="@(SoapUIProjectFiles)" Regex="$(someurl)" ReplacementText="$(newurl)" IgnoreCase="true"/>

   9: </Target>

  10:  

  11: <Target Name="RunFunctionalSoapUITests" DependsOnTargets="ConfigureSoapUITests">

  12:    <!--Run all the soap ui functional tests-->

  13:    <Exec Command="&quot;$(SoapTestTool)&quot; &quot;%(SoapUIProjectFiles.Identity)&quot; -h &quot;$(HostHeader)&quot; -I -r -a -j -f &quot;$(SoapUIDir)\Report&quot;" />

  14: </Target>

At this point I hooked up the MSBuild target to run on Teamcity using the MSBuild runner, for the target “RunFunctionalSoapUITests”. The feedback wasn't good

The final bit you need to do is configure Teamcity to read the xml styled junit reports that the test runner is spitting out. You can do this using the Build feature option for XmlReport processing as shown in the screen shot.

Now trigger the build if you have sorted out all the other variables required you can run the build and see the output on Teamcity, pretty good really on test case by test case basis.

For failed tests I added a configuration for artifacts as follows, Teamcity will show you the response but to see the entire request response file , you need to setup artifacts as follows. All failed test cases create a .txt file which ends with FAILED.txt, so I push these as artifacts and i can see this for failed tests,

SoapUI/Report/*FAILED.txt => FailedTests

I find this useful that with little effort so much could be done and feedback is really good

User Story - A Promise to have a conversation

 
Time and again it is important to constantly remind your team that a User story is not just a way of defining a requirement but is actually the premise on which you promise to have conversations with the user. It is by no means a finalised description of what the system should be doing. The first time it is written the analyst or product owner only has as much information as the user gives them. This information is pretty much raw most often a wish list off some post-it notes on the edge of the users monitor. It will have information of what the user wants to achieve or the grand plan of how something could be done brilliantly to save money or achieve a business goal. It will not tell you what the user wants the system to do.
This is where collaboration is fundamental to the idea of agile development ( should i say ADD – Agile Driven Development). The user story is to be evolved by having conversations between various functional experts. By functional experts I mean a QA, a developer, a business analyst or even a UI designer for that matter. The question is why? I guess it’s because these functional experts can think of the software that is to be built with a view of what the system should do, A user story should convey both what the user wants to do and what the system will do to be complete, clearly the initial draft of these stories didn’t do this.

When a developer evolves a story on his own he is going to make sure it is technically brilliant (may be not always) and eventually forget what the user wants , in most cases this conversation ends up in the developer trying to define what the system should do and what the user wants..  how many over engineered systems have we not seen and been part of in the past

When a UI designer is going to evolve it on his own he is going to make sure it is pretty software and most likely to make it usable but with lack of clarity of the functionality that the user really needs, he just has his wireframes or prototypes, which shows the user the dream he wants to have

When a QA is going to evolve this on his own he is going to make sure it is very testable, infact so testable that they start defining behaviour of the application and the implementation of the software even though they may or may not match what the user wants.. oh well make it testable but so testable that the stuff you build is not usable.

The analyst on the other hand is so caught up with making sure he conveys what the user wants he forgets most often how to test the functionality or in some case forgets to tell the team what the system should do  , well don’t blame him they are not the functional experts on the technical implementation of a system

We have seen these things happening all the time, any form of methodology without collaboration kind of summarises the situation in which these things happen. Alright then, so we can’t do without collaboration so what now and how far do we go with collaboration? How do we know where to stop, well I am going to have to be vague and say well it’s for the team to figure it out in the context of the system they are working but then, I guess some of the answer lies in the ability of the team and the user to work towards coming up with stories which adhere to INSPECT and the story itself becoming the documentation of the system.

As Gojko says in his book “Specification by Example” stories evolve into living documentation of the system. When you can actually read a story and express in simple English, the aspect of what the user wants to do and what the system is doing to achieve the users need, you could say you have reached that point where you can stop and move on to the next bit of functionality.

Again living documentation is not written once; it evolves over a period of time by refactoring constantly; Teams that collaborate constantly recognise this need to bring the stories used to build the system in line with the domain concepts in the system, and vice versa, it is a constant cycle of refactor and improve.. Oh should I say iterate and continuously improve... Rings a bell... Agile?. In reality, collaboration is under-rated by teams and it is something teams think they should do because it needs to be done. Most agile teams do this once for every story (I am laughing already) while estimating the story not so much while actually implementing it. That said there are also teams which constantly collaborate. I guess question is which team are you working in and what are you going to do about it ?

Step by Step - Cucumber, WatiR and Ruby installation tips

There are few road blocks you hit when you go about the process of installing Cucumber, Watir and Ruby the first time, you have to search all the information and then as you install there are some things that work while some dont , I just thought it may be a good idea to consolidate the information in one place for myself if i do run into this situation of having to install this again. I have tried and tested this thrice and use the same process to install our test agents.

• Installing Ruby

#Tip – Choosing the version of Ruby installer

Watir is stable with Ruby 1.8.7 so dont carried away and install 1.9.x of ruby , you learn the hard way that it is not going to work properly.

See Http://watir.com/installation for updates on when 1.9.x support will be provided. Go to http://rubyforge.org/frs/download.php/74293/rubyinstaller-1.8.7-p334.exe download the exe and run the installer. I chose the installation folder to be called just ruby as I want to avoid installing multiple versions for now.

 

• Ruby Path

Check if "c:\ruby\bin" is included in the path (else run PATH=%PATH%;c:\ruby\bin at the command prompt)

• Installing the Dev Kit for Ruby

Download http://github.com/downloads/oneclick/rubyinstaller/DevKit-tdm-32-4.5.1-20101214-1400-sfx.exe .

1. Click on it to extract files to a folder <DEV-KIT-FOLDER>.
2. Open a command prompt for the <DEV-KIT-FOLDER>.
3. Run the command “ruby dk.rb init”
4. Run the command “ruby dk.rb install”

Not sure if you need this but run a “gem update system” and it should say Nothing to update :).

• Installing gems

Now at the command prompt

1. Run “gem install cucumber”.
2. Run “gem install watir”
3. Run “gem install “win32console”
4. Run “gem install rspec”

• Installing ANSI con – if you are unable to see colours on your console window when you run a cucumber feature, you may need to install ANSICON

1. Go to http://adoxa.110mb.com/ansicon . Download AnsiCon 140.
2. Extract the files.
3. Open a command prompt for the folder you have extracted the files in
   cd to x64 folder if you use a 64 bit machine or x86 folder if you use a 32 bit machine
   type "ansicon.exe -i"
   Close the command prompt , open a new one

This should be sufficient to run cucumber features now. In a weeks time I will post a project framework with some useful stuff for ruby / selenium / cucumber which can be downloaded so you can go about building tests quickly

Authoring and Automating - User Stories

In most agile development teams the responsibility of writing user stories falls into the hands of the analyst. The analyst not in all cases may be well versed with the idea of writing stories. This is not because he does not know what to write but sometimes because he does not know the best way to express the story in the chosen story writing platform. This doesn't warrant a developer to pair with an analyst to author a story, In my opinion developers are not welcome to pair with the analyst to author user stories. Allowing this will allow implementation detail to find its way into the stories and sometimes they dictate the users intention.

Authoring stories

The best person for your analyst to pair with should be your QA, this proves to be the most useful.

• The QA looks at a story early in the life cycle and ensures all aspects of the story are testable.
• The ownership of the story is with the QA and he/she is able to identify any automation concerns of the story..
• Any scenarios that have not been through in a story due to data related anomalies are identified.
• The QA is involved in this process early on before the iteration in which the story is picked up , this will allow the QA to bring in some valuable information on the size of the story to the planning session.
• Since the QA gets an understanding of the story before a developer is involved his view of the story is as close as possible to the users requirement in the story. This important to make sure the intention of the user is not skewed by the understanding of a developer.
• The QA is able to identify any smoke tests that may be required to be run when a release is deployed to UAT or Live.

Automating stories

In our current project our QA starts automating user stories when he runs out of stories to test. In most cases the QA to dev ratio is 1:2 or 1:3 and so the QA gets bogged down with implementing acceptance criteria so the team has enough stories to dev on. It helps for devs to pair with QAs to automate acceptance tests and my observation has been the following

• On a normal day we developers are more in sync with writing better code than QA’s, developers can always help in writing better test code.
• When developers implement the acceptance criteria in the form of Given When Then, they actually are implementing the story itself.
• Developers will get an idea of how to implement the story and tests required when they actually develop the story.
• Where the story is looking for new elements on the UI, developers can aid in mocking the UI for the story else automating all the steps of the user story could be a night mare for the QA all on his own.

PS : My Selfish reason - Developers learn a new language .. I learnt ruby this way :)

In effect when three different people with different skills are involved in the authoring and automation of the story, this will ensure a lot more analysis happens and more often than not edge cases are discovered ahead of development. Any edge cases which will increase the cost of the story can be identified and a decision made taken if they have any real value in development.

Acceptance Testing

After playing around with Cucumber/ Cuke4Nuke and Specflow.. turns out cucumber is the winner for me.. Cuke4Nuke took me a while to get going and seems considerably slow and is behind cucumber by some distance..

Specflow is alright however the idea of using attributes to match my step definitions is what i didn't like.. integration with VS and being able to use the NUnit test runner are clear winners.. the entry barrier to this is pretty much minimal.

Well as for Cucumber/ Ruby / WATIR, the idea of using RubyMine to write my tests and debug them in the same doesn't make me miss VS integration and debugging.. I don't think i particularly miss writing my tests in different languages.. well i never do so why bother… The choice of gems available for Ruby and dynamic nature of ruby is something that makes the effort rewarding .. Build integration using a Rake runner is easy not much effort either..One other reason is may be because i use this at work .. there is yet one more to be looked at which is the WebDriver API for Selenium 2.0. http://google-opensource.blogspot.com/2009/05/introducing-webdriver.html ..

I took some time to play around with these since I don’t like skimming through the surface ..

Acceptance Testing – Cucumber? Specflow? Cuke4Nuke

I have been pretty much dormant on this blog due to my part time MBA coupled with a some tight deadlines at work. And to get me back into the blog bandwagon I just thought I will do this small pet project of mine outside work for which I have been exploring the acceptance testing tool to use infact functional test tool which will allow me to translate my acceptance criteria on stories to tests from plain english.

I found something about Specflow on a few blogs,it is a set of libraries to write specifications using Gherkin. You can check the screencast here or read this post from Ryan Lanciaux. SpecFlow can use either NUnit or MsTest as the engine. You could use specflow and drive WATIN.

Being a .Net developer I am quite inclined to use Specflow and Watin after having had a look at it mainly because i dont need to learn a new language and i can use my c# knowledge to write tests, However previous experience with WATIN a couple of years was not great and it always seemed be behind some of the things you could do with WATIR, hopefully this has changed now, In my current job we use Cucumber + Watir to write acceptance tests and we use Ruby, Although I found it a bit steep on the learning curve initially and mainly because i havent worked in anything but .Net. Now after a couple of weeks into it I am beginning to think if this is may be the right choice, for e.g I am not even a novice in Ruby but RubyMine from JetBrains makes it easy when you are a new bee.

I still didnt want to give up exploring a .Net + cucumber combination and my search led me to this article by Richard Lawrence in his blog which compares the pros and cons of Watir and Watin.. go to . Interestingly Richard and a group of other people started driving this project called Cuke4Nuke sometime ago now which should allow us to write tests in C# for cucumber, That sounds great to me, because I am not looking at any advanced usage of cucumber, i guess my needs are pretty basic for now, I have not used it yet but all i can say at the moment is having looked at the screen cast and the background the creators of the project have given it quite a bit of thought on performance (if you read the blog you will find that they did explore using Iron Python over WATIN which was horribly slow).

A couple of months ago they released 0.3 which supports almost everything you can do with Cucumber in Ruby or Java, making C# a first class language for Cucumber. (The only missing features are small things like tags on Before and After hooks and a richer Table object.)

To install and have a play with this go to the Cuke4Nuke Wiki at http://wiki.github.com/richardlawrence/Cuke4Nuke/

Screen Cast on Cuke4Nuke and WATIN

http://www.richardlawrence.info/2009/12/03/screencast-testing-web-applications-in-net-with-cuke4nuke-and-watin/

I am going to give this a go for a spike on my new project so lets see how this goes :).