rapaul.com

A technical blog written by Richard Paul

Avoiding Brittle Element Selection With Selenium2

By automating acceptance tests we are striving to cut back on the maintenance effort of our software. The automated tests perform a majority of the checks thus freeing up time for exploratory testing. However care needs to be taken when automating acceptance tests to ensure they remain flexible in the face of change. Brittle tests, which fail for seemingly unrelated reasons, can place a huge maintenance burden on a testing team negating, some of the gains automation brings.

But fear not, there are steps you can take to ensure your tests are flexible to change. The rest of this post will give examples using Selenium2, however most of these suggestions can be applied to any browser automation framework.

Naming Elements

Selenium2 provides an API for selecting elements on the page for inspection, for example you can find the first element with a given class using the following syntax:

1
browser.findElementByClassName('theClassName')

Similarly finding multiple elements with the same class is done by simply calling findElements.

These methods provide convenient access to elements on the page but we need to ensure the classes or IDs are semantic to ensure our selenium tests continue to function if the layout of the page changes. Below is an example of how to build a navigational element and highlights semantic and brittle IDs for the elements.

As you can see the navigation is broken down into a left and a right side. In order to select these sides a naïve approach would be to assign IDs of leftNav and rightNav. While these two navigational elements can be easily selected using browser.findElementById('leftNav') and browser.findElementById('rightNav'), using these names can have a negative impact further down the line.

First off imagine a new requirement is added whereby the right navigation should now be shown below the left navigation, this is already becoming confusing. As a result any tests we have that needed to locate the right navigation now need to be updated to locate the bottomNav. Similarly any stylesheets will need to be updated to reflect this same change. We can see that by choosing a name that is related to how the navigation looks rather than what it represents becomes a maintenance nightmare.

If in the first place we chose an ID that reflected the intention of the element, our tests would continue to work regardless of the layout of the page. So how do we go about choosing an appropriate ID for the element? By thinking of the purpose of the element we can come up with a more semantic name. In this case all of the elements in the right navigation are related to the user, so we can simply assign an ID of userNavigation, no matter where the user navigation resides, be it on the right or in the footer our Selenium tests will continue to locate the element correctly. A huge save on the maintenance effort.

Another important factor to consider when naming an ID or class for an element is how your users and other stakeholders refer to the element. By ensuring all stakeholders refer to the element in the same way ‘the user navigation’ we can reduce translation between user, developers, testers and the code base (selenium, view templates, stylesheets, domain model, etc). This kind of controlled vocabulary is known in Domain Driven Design as a Ubiquitous Language.

Selecting Options

A common task when filling out a form is to select an option from a drop down list. For example you may need to select your favourite country and the list contains UK, USA, NZ. There are number of ways to select New Zealand.

1
2
3
4
5
<select>
  <option value="UK">United Kingdom</option>
  <option value="US">USA</option>
  <option value="NZ">New Zealand</option>
</select>

The worst possible solution is to grab all the options and select the country based on its position in the list.

1
browser.findElementByTagName("option")[2].setSelected() // select NZ

This check is likely to break if any number of small changes are made, for example the sorting may be changed to alphabetical where NZ would be the first option. If Australia were to be added to the list NZ could be bumped to the 4th option.

To ensure our automation is not tied to the number of elements nor the ordering of the list we can make use of CSS’s attribute selector.

1
browser.findElementByCssSelector("option[value=NZ]").setSelected()

As you can see the code snippet shows the intent of the action much better, no comment is needed to qualify which option is actually being selected. As long as one of the options in this list has a value of “NZ”.

If the labels for the options are the same as the submitted parameters the value attribute is often left off, in this cases we do not have an attribute to target with our attribute selector. Instead we have to check the actual text of the options.

1
browser.findElementsByTagName('option').find { it.text == 'New Zealand' }.setSelected()

A little more code but at least we aren’t selecting based on the position in the list. Note that we now need to find all elements and further refine with Groovy’s find method.

A Note on XPath

Selenium supports XPath for selecting elements, e.g.

1
browser.findElementByXPath('/html/body/div/div[2]/div/ul')

The above example is the XPath Firefox will give you when you select the trending topic element on Twitter’s homepage. This is exactly the type of XPath selector you want to avoid, at no point in the selector string is there a mention of the meaning of any of the elements we are selecting. In fact if a div were to be added or removed anywhere between the trending topics and the root of the homepage the selection would fail. Of course XPath can be written much better.

1
browser.findElementByXPath('//*[@class="trendscontent"]')

However I would argue that the CSS selector is simpler to read and is more inline with the standard for web pages. The fact that the element may already have appropriate IDs and classes to allow CSS selectors to work for styling purposes is a bonus.

1
2
3
browser.findElementByCssSelector('.trendscontent')
// or
browser.findElementByClassName('trendscontent')

Agile Acceptance Testing Slides

Below are slides I presented at the LJC Unconference on Agile Acceptance Testing with Cucumber, Cuke4Duke, Groovy & Selenium. Download in PDF format.

Gherkin Highlighting for VIM

UPDATE: Check the comments, people have had more success with Brent’s suggestion.

I really wanted some highlighting on my .feature files in vim, I’m no vim expert so I did a bit of googling and came up with the following:

1
2
au Bufread,BufNewFile *.feature set filetype=gherkin
au! Syntax gherkin source ~/.vim/cucumber.vim
  • Make sure syntax on is also present
  • Open your feature file

If anyone knows a more vim way of doing this feel free to add a comment.

Avoiding Brittle Tests With Mockito’s ArgumentCaptor

One of the core principles behind my love of Mockito is its ability to avoid brittle tests, by brittle tests I mean unit tests which fail when seemingly unrelated functionality changes.

Below I will outline one of Mockito’s lesser known features, the ArgumentCaptor that shines in certain use cases.

A common requirement for a web application is to send emails to users. In this case our web application is a travel booking system. If you make use of a templating language such as Velocity your email generation service might look similar to the following:

1
2
3
4
5
6
interface Mailer {
  void send(
    String to,
    String templateName,
    Map<String, Object> model)
}

Note: All examples are shown in Groovy for brevity, but the examples apply perfectly well to Java.

Imagine the email we send is simply welcoming the user to our web site upon registration

1
2
3
4
Welcome $name,

Thanks for signing up!
Book your holiday at http://example.com/

As you can see the only dynamic element required in the model map is the user’s name. A unit test to define the behaviour of our registration service will simply verify that a map is passed with the user’s first name.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
private Mailer mailer
private RegistrationService registrationService

@Before
void setUp() {
  // Inject the mocked mailer
  mailer = mock(Mailer)
  registrationService = new RegistrationService(mailer)
}

@Test
void shouldSendRegistrationEmailWelcomingUser() {
  // given a new user
  User user = new User("Jim", "jim@example.com")

  // when the user registers
  registrationService.register(user)

  // then an email should be sent welcoming the user
  verify(mailer).send("jim@example.com", "welcome.vm", ["name":"Jim"])
}

As you can see the test starts off simple, we are verifying the mailer’s send call was invoked with the correct email address, template name and model data. In this case the model simply contains the name of the newly registered user.

A new requirement arrives which states we want to include the latest holiday deals in the welcome email.

1
2
3
4
5
6
7
8
9
Welcome $name,

Thanks for signing up!
Book your holiday at http://example.com/

Check out our latest travel offerings:
#foreach($offer in $offers)
  <!-- Print out the offer details --> 
#end

In order to test this requirement, a naive approach would be to simply add the travel offering assertions into the first test we created. This has the downside that the test is no longer specific to a particular requirement, as we add more content to our email the test would continue to grow and become unwieldy (especially if any conditional logic exists). We want to keep our tests focussed by limiting each test to a single logical assert.

Below we create a second test specific to the inclusion of the latest offers.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
@Test
void shouldSendRegistrationEmailWithLatestTravelOfferings() {
  // given a new user
  User user = new User("Jim", "jim@example.com")
  // and the latest travel offers
  def offers = ["Offer 1", "Offer 2"]
  given(latestOffers.get()).willReturn(offers)

  // when the user registers
  registrationService.register(user)

  // then an email should be sent with the latest travel offers
  verify(mailer).send("jim@example.com", "welcome.vm", ["offers":offers])
}

At first glance this looks like a good test, we are checking the latest offers are included in the model so they can be rendered in the email template.

Unfortunately both tests will fail.

In order to verify the correct calls are made, Mockito uses the equality (equals) method of the passed arguments. In the above case we are checking the equality of two strings and a map. It is of course the equality of the model map that is causing the test to fail.

shouldSendRegistrationEmailWelcomingUser() fails as Mockito is expecting a map containing simply the user’s name [name:"Jim"], but due to the added travel offerings the map is actually [name:"Jim", offers:offers]. The same failure applies to shouldSendRegistrationEmailWithLatestTravelOfferings() as it is verifying the map only contains offers.

As mailer.send() has no return type (void) we have no simple way to access the model map in our test. However Mockito offers a couple of ways around this, the first is the creation of a custom Matcher. The second is to use an ArgumentCaptor which is the approach I will be using today.

The ArgumentCaptor is a specialised ArgumentMatcher that records the matched argument for later inspection.

1
2
3
4
5
6
7
8
9
10
11
12
13
@Test
void shouldSendRegistrationEmailWelcomingUser() {
  // given a new user
  User user = new User("Jim", "jim@example.com")

  // when the user registers
  registrationService.register(user)

  // then an email should be sent welcoming the user
  ArgumentCaptor model = ArgumentCaptor.forClass(Map)
  verify(mailer).send(eq("jim@example.com"), eq("welcome.vm"), model.capture())
  assertThat(model.value.name, is("Jim"))
}

Firstly an ArgumentCaptor is created for the class, in this case Map, we wish to inspect. The ArgumentCaptor is then used as an ArgumentMatcher in the verify call. No matter what keys or values the model map contains, the ArgumentCaptor will always match thus allowing the verify call to succeed. Now that we have captured the model we can inspect it by calling getValue() (simply value in Groovy) to access the original map that was passed to the Mailer service by our production code. By verifying only the key/value pairs that are specific to our test (the user’s name) we can ensure that any other pieces of information that are added to the map in the future don’t affect any of the existing tests.

The astute reader may have noticed the inclusion of equality matchers for both the email address eq("jim@example.com") and the template name. While Mockito relies on the equality method of the arguments by default, if any of the arguments are an ArgumentMatcher, then all of the arguments must be a matcher.

In summary, the ArgumentCaptor allows tests to remain focussed with a single logical assert, even when the object you wish to inspect is created in the code under test and passed to a collaborator via a void method call.

Cucumber World With Groovy

Cucumber provides the ability to create a world in which you can expose useful methods for the testing environment. This post briefly details how to expose these methods when using Groovy with Cuke4Duke.

1
2
3
4
5
6
7
8
9
10
11
12
Feature: World
In order to make useful methods available
As a tester
I want the world to be full of wonderful methods

Scenario: Greet
When I greet
...

Scenario: Farewell
When I farewell
...

These scenarios capture the ability to greet and farewell our customers. The matching step definitions are shown below.

1
2
3
4
5
6
7
8
9
10
11
12
13
// hello_steps.groovy
import static org.junit.Assert.*
import static org.hamcrest.CoreMatchers.*

this.metaClass.mixin(cuke4duke.GroovyDsl)

When(~'I greet') {
  assertThat(hello(), is('hi'))
}

When(~'I farewell') {
  assertThat(goodbye(), is('ciao'))
}

You will notice that we haven’t defined the hello() or goodbye() methods anywhere in our step definition file. We are assuming these methods are provided by the world environment that Cucumber provides. This allows the world methods to be shared between many different step definition files.

Next we define the world environment. Instead of defining all methods in a single class, we can group the methods into individual classes and make them all available on the world class at runtime using Groovy’s mixin support.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
// env.groovy
this.metaClass.mixin(cuke4duke.GroovyDsl)

class Greeter {
  def hello() {
    'hi'
  }
}

class Fareweller {
  def goodbye() {
    'ciao'
  }
}

class World {}

World() {
  World.mixin Greeter, Fareweller
  new World()
}

The World() call is part of Cucumber’s DSL, here we use it to publish an instance of our World class thus making both hello() and goodbye() available in the testing environment (note that the name of the World class is not important).

A Cucumber Scenario for Session Fixation Attacks

When working on a Spring project I was alerted via the logs that my login handling was susceptible to Session Fixation Attacks. It was nice of Spring to alert me of this, and it seems like the problem is due to the Apache httpd connection via modjk. But before diving into a solution I decided to write a simple acceptance test using Cucumber. This allows me to first prove the security hole is present, easily tell when I have fixed it and has the additional benefit of being able to add the check to a set of automated regression tests to ensure it never happens again.

The countermeasure built into Spring uses the Identity Confirmation technique. This basically involves invalidating a user’s session when they login and creating a new session with a different JSESSIONID. It turns out to be fairly easy to write a test for this in Cucumber. Firstly I wrote the plain text scenario:

1
2
3
4
Scenario: Avoiding Session Fixation Attacks through Identity Confirmation
Given I have a session ID
When I sign in
Then I should have a different session ID

The next step is to write the step definitions. Cuke4Duke provides Cucumber support for JVM languages and makes it easy to write reusable step definitions. As I’m a fan of the expressiveness of Groovy I chose to write the step definitions using the GroovyDsl.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import org.openqa.selenium.By
import static org.junit.Assert.*
import static org.junit.matchers.JUnitMatchers.*
import static org.hamcrest.CoreMatchers.*

this.metaClass.mixin(cuke4duke.GroovyDsl)

def sessionId

Given(~'I have a session ID') {
  browser.get("$host/signin")
  sessionId = getSessionId()
}

When(~'I sign in') {
  def signinForm = browser.findElement(By.name('signin'))
  signinForm.findElement(By.name('j_username')).sendKeys('testing@example.com')
  signinForm.findElement(By.name('j_password')).sendKeys('123456')
  signinForm.submit()
}

Then(~'I should have a different session ID') {
  assertThat(getSessionId(), is(not(sessionId)));
}

def getSessionId() {
  browser.manage().cookies.find() { it.name == 'JSESSIONID' }.value
}

The browser variable is injected via env.groovy as an instance of WebDriver which provides the methods to find elements and manage cookies.

def sessionId simply declares a variable allowing sharing of state between the steps so we can check the session ID has been changed.

Avoiding Ambiguities When Using @RequestMapping

I’ve recently been using the param attribute of the @RequestMapping annotation from Spring MVC.

Take the case of a search feature for a website. If a ‘refine’ parameter is present, the user should be shown a search form to refine their criteria. However if the refine parameter is not present the user is shown the search results for the query.

1
2
3
4
5
6
7
8
9
10
11
12
@RequestMapping(value="/search", method=RequestMethod.GET)
public class SearchController {

  // Handles search?query=dog&refine=true
  @RequestMapping(params="refine=true")
  public void refine() { ... }

  // Handles search?query=dog
  @RequestMapping
  public void search() { ... }

}

While this code will initially work, there is some ambiguity in the way the URLs are matched. The order in which the @RequestMapping mappings are defined in the class file is actually determining which method will be called. If we were to define the search() method first, then a request to search?query=dog&refine=true would satisfy the conditions for the search() method’s @RequestMapping and thus the params="refine=true" on the refine() method has no bearing on the outcome of the request.

To safeguard the controller against the potential reordering of methods in a class (which in usual Java programming has no affect on the outcome of running code), we need to ensure the parameter mapping is explicit.

1
2
3
4
5
6
// Match if 'refine' parameter is present
@RequestMapping(params="refine=true")
public void refine() { ... }
// Match if 'refine' parameter is not present
@RequestMapping(params="!refine=true")
public void search() { ... }

By explicitly checking the refine parameter is not present for the search method’s request mapping (!refine=true), the declaration order of our @RequestMapping annotations in the class file has no bearing on which method handles the request thus making our code robust to the reordering methods. Similar precautions should be taken with the other attributes of @RequestMapping including method and headers.

My Git Svn Workflow

Below are the steps I use when working with Git & SVN. Please note, I am in no way a Git or SVN expert, but these are the steps that seem to work for me.

1) Checkout codebase

This step only needs to be done once, it will pull down a local copy of the entire history allowing for very fast nagivation of revisions and allow work to be down when no connection to a central repository is available.

git svn clone svn://path/to/code/ --stdlayout

2) Create a topic branch to work on

We now want to start work on our first story, so we create a local branch for it. This allows us to keep all changes related to the first story out of the master branch. By keeping the master branch clean we can create as many local branches as we wish from it allowing us to quickly fix a bug in a separate local branch to the story we are working on. I’ll come to this later. The -b option means checkout and create a branch, this is a shortcut to creating the branch then checking it out.

git checkout -b story1

.. implement the first story ..

3) Add all new/deleted/modified files to staging area

With git you can stage just the files that are important, these staged items then make up the contents of your commit.

git add path/to/resource

4) Create a commit

We then create a commit of the staged files in our local branch, note this differs from SVN as a commit is only local.

git commit -m "My commit message"

5) Merge changes from your topic branch into the master branch

Ensure the master branch is up to date

git checkout master
git svn rebase

Then merge your topic branch into master

git merge story1

6) Merge conflicts

Merge any conflicts either manually or using mergetool (I’m yet to figure out how to use the mergetool command)

7) Push to SVN

This is the bridge that takes our Git commits and pushes them to SVN. Each Git commit will be mapped to a respective SVN commit.

git svn dcommit

Working with multiple branches

As I mentioned earlier, one of the major advantages of Git over SVN is the ability to have multiple working branches locally, allowing you to switch between them with relative ease. Take for example you are working on a story, it might take you a whole day to get it done. At midday an urgent support request arrives for a bug in the live software. Using SVN I would typically have to checkout the HEAD revision into a separate directory and import it into Eclipse as a new project. With Git it is much simpler, I use the following steps to do the switching.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Commit all current work to the topic branch
git add path/to/noncommited/work

# Switch to the master branch and make sure it is up to date
git checkout master
git svn rebase

# Create a new branch for the bug fix
git checkout -b bugfix

# Fix the bug
# Follow steps 3-7 above
# Carry on working on the story
git checkout story1

Git GUI

You can use the Git GUI tool for most of the steps above, I find it particularly useful for steps 3 & 4. You can also review the history and changes using gitk, this is useful for code reviews before you push your code into a master repo.

JGit (Eclipse Plugin)

I use the JGit Eclipse Plugin when working in Eclipse, it provides a diff with the previous version which I find invaluable. Install in Eclipse using the update site: http://www.jgit.org/updates

References

http://blog.tsunanet.net/2007/07/learning-git-svn-in-5min.html http://stackoverflow.com/questions/190431/is-git-svn-dcommit-after-merging-in-git-dangerous http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#resolving-a-merge

jQuery Behaviours

Below are slides from a short presentation I gave to my colleagues at f1000. The slides contain a brief background on Javascript behaviours and provide a comparison between jQueries’ bind and live behaviours along with the liveQuery plugin.

A PDF version of the slides is available.

Thanks to my manager Phil, for letting me post this on my personal blog.

A Behavioural World

Or at least a behavioural twitter world,

http://github.com/rapaul/abehaviouralworld is a little project just launched which attempts to distill behaviours from twitter. Given my recent rapture in the world of BDD, the behaviours we are distilling are those that match the scenario format.

1
2
3
Given ...
When ...
Then ...

If you look carefully at the page source, you will notice a large portion of it is dedicated to an HTML comment containing the list of features and the breakdown into scenarios. To keep things easily accessible, each time I thought of a scenario I added it to the page, once completed, it was marked with [Done].

For example, starting with the most important feature:

1
2
3
4
Feature 1)
In order to see how BDD syntax is being used on twitter
As a BDD zealot
I want to see tweets that contain 'given', 'when' & 'then', in that order.

I then broke this feature down into scenarios:

1
2
3
4
5
[Done]
Given the search returns more than matching 10 tweets
When the user views the page
Then the 10 most recent tweets should be displayed
...

Each time a scenario was satisfied, I marked it with [Done]. Not all scenarios ended up being all that important, the following scenario is still pending.

1
2
3
Given the search returns no matching tweets
When the user views the page
Then instead of showing any tweets a message should be shown

In fact, if we attempt to tie this scenario back into the vision of the website (distilling behaviours in twitter), then we realise it adds no value. If no one is tweeting about behaviours, then it is highly unlike that anyone will be visiting our behaviour distillery.

There are a couple of features (3 & 4) in the source which haven’t yet been implemented.

1
2
3
4
5
6
7
8
9
Feature 3)
In order to emphasis the given/when/then syntax
As a BDD zealot
I want to see give/when/then highlighted in the displayed tweets

Feature 4)
In order to encourage BDD syntax usage on twitter
As a BDD zealot
I want to be able to tweet from this page using a BDD template

If anyone would like to work on these features, or even just break down some useful scenarios, please feel free to fork the code on github, provide me with a diff or add a comment (note at the time of writing github looks to be having some caching issues on the ‘Source’ tab).

What no executable scenarios? Oops, nope. I’m yet to get into the world of writing executable scenarios in Javascript. If someone has some knowledge in this area, please feel free to fork the github repository and start automating the scenarios from the page source.

PS: This idea was partially inspired by a tweet by soulnafein which read: My girlfriend is taking the mickey out of me because recently I started explaining things using the Scenario format (Given When Then) :-)