CFC Creation Time in ColdFusion 8/9, Open BlueDragon 1.1 and Railo 3.1
There's lots of chatter about CFC creation performance in the popular CFML engines (Adobe ColdFusion, Open BlueDragon and Railo), and plenty of others have ran their own tests, but I'm just curious and had to see some tests for myself.
I do not intend to encourage anyone to choose any one engine over another solely based on these tests. This is just one small indicator of the many performance considerations! In a real world application, there are many other performance considerations (hardware, operating system, application server, JVM tuning, CFML engine admin settings, various cache mechanisms, database server, database tuning, etc.).
I'm just focused on the speed of CFC creation here. In many small or medium sized apps, this is not much of a factor, but in larger applications where large sets of objects need to be created, this can potentially become crippling to application performance.
Also keep in mind that not all CFML applications are guaranteed or even likely to migrate flawlessly between any two CFML engines, despite lots of great progress towards compatibility and a core CFML spec. There are also lots of vendor specific features that may sway a particular developer or team to a particular CFML engine, and this may vary from project to project.
I could ramble on with more disclaimers, but let's get to the good stuff!
Test Environment
I've made it a point to keep the test environment as consistent as possible between CFML engines. To that end, I've used the same bare bones distribution of the Apache Tomcat servlet engine for testing of each CFML engine. Below are the points of interest.
- Machine: Dell Vostro 1510, Ubuntu 8.04 Desktop 32-bit, 3 GB RAM, Intel Core 2 Duo
- Tests run on default Tomcat 6.0.20 distro with /webapps emptied
- One CFML engine WAR moved into tomcat/webapps/ at each Tomcat startup
- Mapping for "/testMapping" (outside Web root) created for each engine
- TestInWebroot.cfc is in the Web root of each WAR (code below)
- TestUnderMapping.cfc is under the /testMapping mapped directory (code below)
- Test harness script creates 100,000 instances of specified CFC (code below)
- Ran 5 test harness requests per CFC type for each CFML engine
- Each "Average CFCs per second" is therefore based on creating 500,000 instances
- Results of first test harness run after each Tomcat startup were ignored (always a bit slower)
- Debugging turned off in all engines
- See "Admin Settings" column in results below for the only alterations other than debugging
Code
Test CFCs (TestInWebroot.cfc and TestUnderMapping.cfc)
<cffunction name="init" output="false">
<cfreturn this />
</cffunction>
</cfcomponent>
Test Harness
<cfscript>
cfcPaths = arrayNew(1);
cfcPaths[1] = 'TestInWebroot';
cfcPaths[2] = 'TestUnderMapping';
cfcPaths[3] = 'testMapping.TestUnderMapping';
cfc = cfcPaths[url.cfc];
resultsFile = expandPath('/testMapping/data/results.txt');
resultsHead = 'Engine Version CFC Path CFCs per Second';
numObjects = 100000;
start = getTickCount();
for ( i=1; i LTE numObjects; i=i+1 ) {
createobject( 'component', cfc ).init();
}
end = getTickCount();
seconds = (end-start)/1000;
objectsPerSecond = numObjects/seconds;
results = '#engineName()# #engineVersion()# #cfc# #numberFormat( objectsPerSecond )#';
writeOutput( '#resultsHead#<br />#results#' );
</cfscript>
<cfif NOT fileExists( resultsFile )>
<cffile action="write"
file="#resultsFile#"
output="#resultsHead#"
addnewline="true" />
</cfif>
<cffile action="append"
file="#resultsFile#"
output="#results#"
addnewline="true" />
<cffunction name="engineName" output="false">
<cfscript>
var result = server.coldfusion.productname;
if ( result EQ 'ColdFusion Server' ) result = 'ColdFusion';
if ( result EQ 'BlueDragon' ) result = 'Open BlueDragon';
return result;
</cfscript>
</cffunction>
<cffunction name="engineVersion" output="false">
<cfscript>
var result = server.coldfusion.productversion;
if ( structKeyExists( server, 'railo' ) ) result = server.railo.version;
return result;
</cfscript>
</cffunction>
Results
The results below are displayed in a Google Spreadsheet as an iframed widget -- click here if you have trouble viewing it, or would just prefer to see it at full browser window size.
Observations
Admin Settings
At first, I was aiming to keep admin settings as basic/default as possible, for even comparison. But to be more realistic, as you can see above, for a second set of tests, I've identified the admin settings that I'd expect to directly affect CFC creation time. In ColdFusion 8 and 9, and in Open BlueDragon 1.1, I turned on Trusted Cache. In Railo 3.1.0.23 I changed the "Inspect Templates" setting to "Never" (this setting is similar to the new "Cache Template In Request" in CF9, which is already turned on by default). The highlights? What was interesting is that these enhanced settings did not always result in improved CFC creation performance.
CFC Locations, Paths and Mappings
You might also notice that the very same CFC had the best creation performance on all engines when it was not only located beneath a mapping, but the component path used in the createObject() call did not include the mapping, but rather just the CFC name. These variations show the performance hit taken by having the flexibility to locate CFCs in various locations (Web root, sud-directory of Web root, beneath a mapping, etc.). ColdFusion 9 has introduced a great Admin setting called "Component Cache," which will cache the resolution of the CFC location with each request. It is turned on by default, but I'm surprised that there's still such a performance difference in my test cases.
I also don't have a reference handy of just what the prioritization is for CFC location resolution in each CFML engine, but based on the test results above, it appears to be very similar between the different engines.
Conclusion
I think the results speak for themselves with regard to this particular measurement of CFC creation time. It's up to each developer to decide what kind of impact this measurement may have on any given application. I'd love to hear some feedback if anyone is reading this! If I've missed any important settings for this testing purpose, or if you have any additional questions or recommendations, please let me know!
Comments (12) |
15012 Views
You rizzock! (for more than just this stuff-- you been doing quality work, esse)
any chance of making the result columns sortable?
also, I your test cfc is rather simple, it would be intersting to see the differences between test CFCs with say 5 functions
plus to see what if any overhead arguments make
@denny: Hey, thanks! My pleasure!
@zac: While singletons are more negligible, for sure, domain objects are a different beast. If you have any business objects with rich behavior and you need a large set of them, then you can get into trouble. A common use case is passing a collection of objects to Flex -- here's a good blog post outlining this one:
http://bit.ly/f2DRY
I did intend to make those result columns sortable with a jQuery grid or something, but I was short on time and it was easy to iframe my existing Google spreadsheet :) I will try to update that if I have a chance.
I did also do some testing with a CFC containing 10 methods, but I decided to save that for another post. I'd also like to put together a simple "real world" example -- maybe a blog Post CFC that has some properties and is composed of Comment and Category objects. I'll try to remember to comment here when I get to this follow up post.
I'm on CF8 now and looking at ways to boost my app performance. I'm considering just going to CF9, moving my CF8 from Jrun to Tomcat/Glassfish/JBoss, moving to CF9 with Tomcat/Glassfish/JBoss and moving to Railo. Trying to make some educated guesses on the "bang for the buck". With Jrun vs. Tomcat comparisons on CF8 and/or CF9, that would fill in remaining blanks!
http://www.barneyb.com/barneyblog/2010/02/12/goodb...
I rarely see a reason to use anything other than Tomcat these days. It's not officially supported by Adobe for CF9, but you merely need to drop in a single JAR for ORM/Hibernate support (JTA, link also in Barney's post). If you have a reason to use JBoss or something comparable and more robust than Tomcat, go for it, if not I highly recommend Tomcat. HTH!
I would still love to see your tests run across the Java containers on your computer so there is a reasonably consistent test environment.
http://railo.googlegroups.com/web/perfomance-compa...?
I have a multiserver config similar to the one described in one of your blogs.
I am evaluating using Railo (3.1.008 preview) on Tomcat 6 against
Coldfusion 8. I have an existing CF webservice for which I am doing these tests
Seems like each individual request on Railo runs faster when compared to coldfusion, but when I run the the test of concurrent users, railo requests are queuing. The code on CF-8 and Railo is identical...however the the CF-8
requests seem to be handled concurrently (e.g. 5 at a time). I am using Coldspring, I have checked the code for unusual locks or
transaction blocks.
The code tested in the screenshot has a few select SQL statements, no inserts or transanction blocks.
I am using "Charles" (http://www.charlesproxy.com/) for making these concurrent requests
Any ideas...