In the last blog post I explained how our baseline test works and it’s pros and cons, if you missed it you can find it here. In this article I hope to explain another type of test that we use that could prove helpful or inspirational to others so let’s get started.
Functional Testing Against Test Data
Like the previous blog post on our baseline test our functional tests are also a form of black-box testing; we are only really interested in the end result and whether things work as they should or do not.
In order to explain exactly what we do it’s probably worth first explaining the problem these tests try to address:
With the agile approach we have to development it is also important that we make changes incrementally and have an easy way of testing that we have not broken or negated previous advances.
What our functional tests do is similar to our baseline tests except they test the actual functions of our output in the case of the HTML5 Converter or in the case of our viewer we test against how form fields and the document change after simulating an action.
The viewer test make use of a Java library called Jemmy that lets us simulate mouse clicks, typing and any other user input. The HTML test makes use of another Java library called HTMLUnit that does almost the exact same thing except it uses a headless browser.
The basic steps of the viewer test are:
- We then open each file with our viewer using Jemmy (this is automated so we don’t need to manually do this for each file)
- We next check to see if we have a script file for this particular page and document (the script file is something we use to determine the actions to perform on the page)
- If we don’t have a script file we generate one based upon all the form elements on the page and any simple data we gleam from their current contents (e.g. if a field contains a number we make sure the script file tries to add a number to that field to test its formatting)
- We then fail the test here to ensure the person running it is aware of the fact there wasn’t a test file and that they will potentially have to edit the generated one.
- With the script file we run through it performing the actions it requires and checking against the wanted results. If at any point there is a differences the test fails and we get a description of what should of happened versus what actually happened.
- We then repeat for each page of each file we are testing against.
As you can see we make use of a file which I called a script to test the various features of pages against. This file is just a plain text file that normally looks something like this:
In our code we have a simple interpreter that reads each line and executes an action based upon the command given. In order to make these script files a lot easier to create the test tries to guess at correct data to use, in this case it gets the email format correct but could not figure out that the phone field should probably test using a number instead of a string.
This excerpt also doesn’t test against bad data input e.g. typing in a normal string instead of an email address into the email field. The ability to do this is a major advantage of using these kinds of tests as when testing you should always ensure that you test for both successes and failures.
The HTML5 test is incredibly similar so does not really need as in-depth an explanation as the viewer test; it simply does the same but with HTML files and also runs a lot faster due to it’s headless nature (Jemmy will still open up your program as a window and can be timing dependent).
The advantages of using script file over hard coding the values into our tests is flexibility. Using a framework built to interpret these files let’s us add files to the tests at our leisure and tweak the script files to continually improve our test data, it also let’s us add new features to the test framework letting us improve it over time. For a small company these tests can save time spent hunting down bugs after a release by alerting us to their presence earlier on.
That concludes this part of this testing series. So far we have only looked at 2 forms of black-box testing we use here at IDR Solutions; however we also make use of unit tests which is a form of white-box testing (or structural testing) which may be covered in the future.
This post is part of our “Testing Articles Index” in these articles we provide a guide to Testing.