This posting follows on from Wonderful WebFlow and Testing WebFlow (Wonderful Webflow, Part II). It’s probably best to take a look at those first…
Ahhh Load Testing! The very name strikes fear into the heart.
I have seen projects crash and burn and been frantically resurrected as a result of poor performance that only came to light at the last minute. I have seen projects scrabble to find the appropriate incantations (read: JVM options) that will encourage the performance faeries to sprinkle their magic dust on the system so that the project can “go live.” I have seen angst, blood, sweat and tears as year-long projects fail to cross that “one final hurdle” on the path to deployment.
I have often wondered: why is Load Testing always left so late? I have talked to people that profess to practise XP development who still leave it to the latter stages of the project, even though creating a Potentially Shippable Product is supposed to be a primary goal for an iteration.
There seem to be two underlying reasons for this state of affairs. The first reason is cost. Tools like HP LoadRunner are not cheap, plus it costs a lot to engage a load testing specialist to do the work (presumably faery food is rare and expensive…) so it has to be done only when things are “finally ready.” The second reason goes like this: “since the system isn’t finished yet, the figures won’t mean anything, so there’s no point in doing it.” Here is one such statement: “Running performance tests in a development environment that differs from your production environment can often be a misleading and misdirected effort.”.
Both reasons are fallacious. Load Testing need not be an expensive activity, especially when FOSS such as JMeter are available. The idea that performance figures are the only product of load testing is also wrong: load testing can show weaknesses in the overall application architecture, or how the application is to be managed, it can highlight a wrong choice of JVM settings or even of actual JVM (until recently my mantra was: “thou shalt use JRockit for server-side applications.” With their recent takeover of BEA, Oracle has unfortunately put the kaibosh on that [BOO!] so I’m not even going to link to JRockit’s home page). Load testing can show weaknesses in choice of tool, algorithm, technique or team. The actual figures that emerge although often considered hightly important, may actually be the least important outcome.
IMHO one should aim for Load Testing to be done for each iteration of Potentially Shippable Product so that no nasty surprises pop up at the last moment. Given a sophisticated enough scripted CI (earlier post) environment it is even possible that Load Testing be done as part of a nightly build and this is very appealing to me (naturally a JMeter Plugin for Hudson exists); CI is supposed to drive software quality, after all.
For really quick and dirty performance tests, there’s Apache ab or curl or even wget, so there is really no excuse for not knowing something about performance. She’ll be right mate just won’t cut it!
Enough pontification! Let’s take a look at how to use Apche JMeter with our Calc WebFlow application.
Why JMeter? It’s FOSS, powerful and you gotta love a tool whose manual has a chapter entitled 17. Help! My boss wants me to load test our web app! :-)
Why not JMeter? “…not every company is willing to stop putting out huge sums of money for Mercury.” Support for some edge cases involving JavaScript is the main lack and this can be overcome by using badboy (good Ozzie software!) in conjunction with JMeter.
JMeter works in two modes: as a recording proxy for your browser and then as a playback device for the recording made previously.
Laziness is a virtue, so I am not going to show each step of the process! There are.Plenty.of.Resources,.already.
I will add a few generally-useful and WebFlow-specific tidbits, however.
If you have Vista and use IE you will find it dificult/impossible to proxy for anything on localhost/127.0.0.1. Solutions include: binding your server to a loopback adapter, using another browser or using a separate server machine. This is a general issue, not something specific to JMeter.
Before playback, for clarity rename each request in the recorded interactions, otherwise the various reports and graphs become very confusing:
For the actual Load Test playback to work correctly, JMeter needs to invoke a couple of helper functions: the Cookie Manager and a Regular Expression Extractor. While the need for the former tool is probably quite clear, the requirement for/use of the latter needs explanation. JMeter cannot simply replay the sequence as recorded. Recall that WebFlow allocates a unique one-time per-flow value for each flow that is stored in the ‘_flowExecutionKey’ hidden field/request parameter. Although JMeter stores this, it cannot simply reuse the recorded value on replay, and instead must extract and use the value allocated afresh for each flow execution. This is the task of the Regular Expression Extractor: look through the recorded sequence for a specified pattern that can then be saved into a variable.
Once recorded, each script must be ‘massaged’ to make use of the variable that the Regular Expression Extractor will maintain, rather than use the recorded value directly (it is in this area that a tool like LoadRunner might make life easier: it has an ‘auto-correlate’ ability that simplifies this task). Care is needed here and many references on the ‘net are wrong or a bit out of date:
To be clear, here are the appropriate values:
Reference Name: flowExecutionKey
Regular Expression: name=”_flowExecutionKey” value=”(.*?)”
Template: $1$
Match No.: 0
And here is a ‘massaged’ request entry, showing how the ‘flowExecutionKey’ variable is used:
As is desirable, JMeter produces nice graphs and reports, as this pair of screenshots shows:
JMeter has a wide range of elements that can be added to a test plan: the “View Results Tree” Listener is useful for helping out with development/debugging of a test plan. As the manual says: “The View Results Tree shows a tree of all sample responses, allowing you to view the response for any sample. In addition to showing the response, you can see the time it took to get this response, and some response codes.” Very useful:
(As an aside, note how this results tree shows how WebFlow automatically implements the “Redirect after POST” technique…nice!)
The “Gaussian Random Timer” is another useful test plan element. Rather than issuing requests as quickly as possible, with no “think time” in between, the Gaussian Random Timer makes JMeter simulate the timing between hits more realistically. This gives a better indication of how a system might be expected to perform in the “normal situation” but is probably not so hot at predicting performance at extreme load.
There are many other test plan elements, you should take a look at the user manual to see just what other elements JMeter makes available.
Of course, JMeter is controllable from gant, as this small example shows:
DEVTOOLS = 'c:/DEVTOOLS'
dirJMeterHome = "${DEVTOOLS}/jakarta-jmeter-2.3.2"
jmeterTests="jmeter"
jmeterResultsDir="${jmeterTests}/results"
jmeterResultsJtl="${jmeterResultsDir}/jmeter-results.jtl"
jmeterResultsHtml="${jmeterResultsDir}/jmeter-results.html"
jmeterResultsXsl="${dirJMeterHome}/extras/jmeter-results-report_21.xsl"
jmeterTestJmx="${jmeterTests}/calc.jmx"
dirGantHome = ant.project.properties."environment.GANT_HOME"
includeTargets << gant.targets.Clean
cleanDirectory << [jmeterResultsDir]
ant.path(id: 'pathJMeter') {
fileset(dir: dirJMeterHome + '/lib', includes: '*.jar')
}
ant.path(id: 'pathJMeterAntTask') {
fileset(dir: dirJMeterHome + '/extras', includes: 'ant-jmeter-*.jar')
}
ant.taskdef(name: 'jmeter', classname: 'org.programmerplanet.ant.taskdefs.jmeter.JMeterTask',
classpathref: 'pathJMeterAntTask')
ant.path(id: 'pathGroovy') {
fileset(dir: dirGantHome + '/lib', includes: '*.jar')
}
target(init: 'Initialise the build, given a clean start') {
ant.mkdir(dir: jmeterResultsDir)
}
target (reformatReport: 'Do XSLT Magic') {
ant.xslt(in: "${jmeterResultsJtl}", out:"${jmeterResultsHtml}", style: "${jmeterResultsXsl}")
}
target(runJMeterTests: 'Get JMeter up & running') {
jmeter(jmeterhome: "${dirJMeterHome}", resultlog: "${jmeterResultsJtl}", testplan: "${jmeterTestJmx}")
{
property(name:"jmeter.save.saveservice.output_format", value: "xml")
property(name: "jmeter.save.saveservice.response_data.on_error", value: "true")
}
}
target(defaultTarget: 'Do Everything') {
depends(clean, init)
println 'Starting...'
runJMeterTests()
reformatReport()
}
setDefaultTarget(defaultTarget)
Equally of course, once one has a gant script, one can easily integrate with Hudson (which knows how to do the XSLT step itself, so the gant script could be even shorter than that shown above):
(the observant among you may notice that the charts are not that exciting; we’ll see how bug 2752 works out…)
Just so you know, JMeter is not just for testing HTTP-based systems. It is equally possible to build comprehensive test plans to load test JDBC-accessible databases, SOAP-base WebServices, FTP servers, etc.
Think about this: the combination of JMeter and Hudson can recast Load Testing from a rarely-performed, expensive “magic shield” to just another test aimed at improving the quality of your code. This is surely A Good Thing.
For the curious, the actual JMeter Test Plan associated with this post is available here.
