CFC Creation Time in ColdFusion 8/9, Open BlueDragon 1.1 and Railo 3.1

There's lots of chatter about CFC creation performance in the popular CFML engines (Adobe ColdFusion, Open BlueDragon and Railo), and plenty of others have ran their own tests, but I'm just curious and had to see some tests for myself.

I do not intend to encourage anyone to choose any one engine over another solely based on these tests. This is just one small indicator of the many performance considerations! In a real world application, there are many other performance considerations (hardware, operating system, application server, JVM tuning, CFML engine admin settings, various cache mechanisms, database server, database tuning, etc.).

I'm just focused on the speed of CFC creation here. In many small or medium sized apps, this is not much of a factor, but in larger applications where large sets of objects need to be created, this can potentially become crippling to application performance.

Also keep in mind that not all CFML applications are guaranteed or even likely to migrate flawlessly between any two CFML engines, despite lots of great progress towards compatibility and a core CFML spec. There are also lots of vendor specific features that may sway a particular developer or team to a particular CFML engine, and this may vary from project to project.

I could ramble on with more disclaimers, but let's get to the good stuff!

Test Environment

I've made it a point to keep the test environment as consistent as possible between CFML engines. To that end, I've used the same bare bones distribution of the Apache Tomcat servlet engine for testing of each CFML engine. Below are the points of interest.

  • Machine: Dell Vostro 1510, Ubuntu 8.04 Desktop 32-bit, 3 GB RAM, Intel Core 2 Duo
  • Tests run on default Tomcat 6.0.20 distro with /webapps emptied
  • One CFML engine WAR moved into tomcat/webapps/ at each Tomcat startup
  • Mapping for "/testMapping" (outside Web root) created for each engine
  • TestInWebroot.cfc is in the Web root of each WAR (code below)
  • TestUnderMapping.cfc is under the /testMapping mapped directory (code below)
  • Test harness script creates 100,000 instances of specified CFC (code below)
  • Ran 5 test harness requests per CFC type for each CFML engine
  • Each "Average CFCs per second" is therefore based on creating 500,000 instances
  • Results of first test harness run after each Tomcat startup were ignored (always a bit slower)
  • Debugging turned off in all engines
  • See "Admin Settings" column in results below for the only alterations other than debugging

Code

Test CFCs (TestInWebroot.cfc and TestUnderMapping.cfc)

<cfcomponent output="false">
    <cffunction name="init" output="false">
        <cfreturn this />
    </cffunction>
</cfcomponent>

Test Harness

<cfparam name="url.cfc" default="1" />

<cfscript>
    cfcPaths = arrayNew(1);
    cfcPaths[1] = 'TestInWebroot';
    cfcPaths[2] = 'TestUnderMapping';
    cfcPaths[3] = 'testMapping.TestUnderMapping';
    cfc = cfcPaths[url.cfc];

    resultsFile = expandPath('/testMapping/data/results.txt');
    resultsHead = 'Engine    Version    CFC Path    CFCs per Second';

    numObjects = 100000;

    start = getTickCount();

    for ( i=1; i LTE numObjects; i=i+1 ) {
        createobject( 'component', cfc ).init();
    }

    end = getTickCount();

    seconds = (end-start)/1000;
    objectsPerSecond = numObjects/seconds;

    results = '#engineName()#    #engineVersion()#    #cfc#    #numberFormat( objectsPerSecond )#';

    writeOutput( '#resultsHead#
<br />#results#' );
</cfscript>

<cfif NOT fileExists( resultsFile )>
    <cffile action="write"
            file="#resultsFile#"
            output="#resultsHead#"
            addnewline="true" />

</cfif>

<cffile action="append"
        file="#resultsFile#"
        output="#results#"
        addnewline="true" />


<cffunction name="engineName" output="false">
    <cfscript>
        var result = server.coldfusion.productname;
        if ( result EQ 'ColdFusion Server' ) result = 'ColdFusion';
        if ( result EQ 'BlueDragon' ) result = 'Open BlueDragon';
        return result;
    
</cfscript>
</cffunction>

<cffunction name="engineVersion" output="false">
    <cfscript>
        var result = server.coldfusion.productversion;
        if ( structKeyExists( server, 'railo' ) ) result = server.railo.version;
        return result;
    
</cfscript>
</cffunction>

Results

The results below are displayed in a Google Spreadsheet as an iframed widget -- click here if you have trouble viewing it, or would just prefer to see it at full browser window size.

Observations

Admin Settings

At first, I was aiming to keep admin settings as basic/default as possible, for even comparison. But to be more realistic, as you can see above, for a second set of tests, I've identified the admin settings that I'd expect to directly affect CFC creation time. In ColdFusion 8 and 9, and in Open BlueDragon 1.1, I turned on Trusted Cache. In Railo 3.1.0.23 I changed the "Inspect Templates" setting to "Never" (this setting is similar to the new "Cache Template In Request" in CF9, which is already turned on by default). The highlights? What was interesting is that these enhanced settings did not always result in improved CFC creation performance.

CFC Locations, Paths and Mappings

You might also notice that the very same CFC had the best creation performance on all engines when it was not only located beneath a mapping, but the component path used in the createObject() call did not include the mapping, but rather just the CFC name. These variations show the performance hit taken by having the flexibility to locate CFCs in various locations (Web root, sud-directory of Web root, beneath a mapping, etc.). ColdFusion 9 has introduced a great Admin setting called "Component Cache," which will cache the resolution of the CFC location with each request. It is turned on by default, but I'm surprised that there's still such a performance difference in my test cases.

I also don't have a reference handy of just what the prioritization is for CFC location resolution in each CFML engine, but based on the test results above, it appears to be very similar between the different engines.

Conclusion

I think the results speak for themselves with regard to this particular measurement of CFC creation time. It's up to each developer to decide what kind of impact this measurement may have on any given application. I'd love to hear some feedback if anyone is reading this! If I've missed any important settings for this testing purpose, or if you have any additional questions or recommendations, please let me know!

Comments (Comment Moderation is enabled. Your comment will not appear until approved.)
Kevan Stannard's Gravatar Jamie, very interesting results, particularly with the differences due to the component locations. Railo results are amazing.
# Posted By Kevan Stannard | 7/21/09 6:33 PM
denny's Gravatar I love this stuff. Thanks for taking the time to do it, Jamie.

You rizzock! (for more than just this stuff-- you been doing quality work, esse)
# Posted By denny | 7/21/09 11:44 PM
zac spitzer's Gravatar interesting results, tho once you starting using singletons and the like becomes rather moot

any chance of making the result columns sortable?

also, I your test cfc is rather simple, it would be intersting to see the differences between test CFCs with say 5 functions
plus to see what if any overhead arguments make
# Posted By zac spitzer | 7/22/09 3:40 AM
Jamie Krug's Gravatar @Kevan: Yes, I was surprised that the explicit "testMapping.TestUnderMapping" path was significantly slower than plain old "TestUnderMapping" -- I'd prefer the other way around, and I know Railo is considering some admin settings to better control this. Maybe Adobe and Open BlueDragon will do the same.

@denny: Hey, thanks! My pleasure!

@zac: While singletons are more negligible, for sure, domain objects are a different beast. If you have any business objects with rich behavior and you need a large set of them, then you can get into trouble. A common use case is passing a collection of objects to Flex -- here's a good blog post outlining this one:
http://bit.ly/f2DRY

I did intend to make those result columns sortable with a jQuery grid or something, but I was short on time and it was easy to iframe my existing Google spreadsheet :) I will try to update that if I have a chance.

I did also do some testing with a CFC containing 10 methods, but I decided to save that for another post. I'd also like to put together a simple "real world" example -- maybe a blog Post CFC that has some properties and is composed of Comment and Category objects. I'll try to remember to comment here when I get to this follow up post.
# Posted By Jamie Krug | 7/22/09 10:02 AM
Sammy's Gravatar The railo numbers look awesome. But, I can't seem to get hold of the railo version 3.1.0.023 that you tried. The site only has link for 3.1.0.022. where can I download it from and try?
# Posted By Sammy | 7/24/09 2:43 AM
Jamie Krug's Gravatar @Sammy: Yes, Railo 3.1.0.022 is the current beta build available at http://getrailo.org but if you'd like, you can update to the latest "bleeding edge release" (BER). The easiest way to do this is by using the Update feature in the Railo Server Admin. However, to get the BER release, you need to change the update URL to http://dev.railo.ch -- just remember to change it back to http://www.getrailo.org if you'd prefer the stable builds only to show up.
# Posted By Jamie Krug | 7/24/09 10:29 AM
Brian's Gravatar Jamie - can you run the same test on CF8 and CF9 under Jrun? I'm curious to see what kind of performance difference exists between the two J2EE servers?

I'm on CF8 now and looking at ways to boost my app performance. I'm considering just going to CF9, moving my CF8 from Jrun to Tomcat/Glassfish/JBoss, moving to CF9 with Tomcat/Glassfish/JBoss and moving to Railo. Trying to make some educated guesses on the "bang for the buck". With Jrun vs. Tomcat comparisons on CF8 and/or CF9, that would fill in remaining blanks!
# Posted By Brian | 3/12/10 5:58 PM
Jamie Krug's Gravatar @Brian: Without doubt, you will see a very nice performance boost by switching from CF8 to Railo or CF9. You will see a further boost opting for Tomcat instead of JRun; Barney Boisvert shared some graphs here:
http://www.barneyb.com/barneyblog/2010/02/12/goodb...

I rarely see a reason to use anything other than Tomcat these days. It's not officially supported by Adobe for CF9, but you merely need to drop in a single JAR for ORM/Hibernate support (JTA, link also in Barney's post). If you have a reason to use JBoss or something comparable and more robust than Tomcat, go for it, if not I highly recommend Tomcat. HTH!
# Posted By Jamie Krug | 3/12/10 8:34 PM
Brian's Gravatar Thanks Jamie! Looks like I have some testing ahead of me... :)

I would still love to see your tests run across the Java containers on your computer so there is a reasonably consistent test environment.
# Posted By Brian | 3/15/10 2:32 PM
Jamie Krug's Gravatar Brian, I'm curious too. I'll try to drop the tests into a few different JEE containers as soon as I find a little extra time.
# Posted By Jamie Krug | 3/16/10 7:30 PM
Anang Phatak's Gravatar Have a look at this screenshot:
http://railo.googlegroups.com/web/perfomance-compa...?

I have a multiserver config similar to the one described in one of your blogs.

I am evaluating using Railo (3.1.008 preview) on Tomcat 6 against
Coldfusion 8. I have an existing CF webservice for which I am doing these tests

Seems like each individual request on Railo runs faster when compared to coldfusion, but when I run the the test of concurrent users, railo requests are queuing. The code on CF-8 and Railo is identical...however the the CF-8
requests seem to be handled concurrently (e.g. 5 at a time). I am using Coldspring, I have checked the code for unusual locks or
transaction blocks.

The code tested in the screenshot has a few select SQL statements, no inserts or transanction blocks.

I am using "Charles" (http://www.charlesproxy.com/) for making these concurrent requests

Any ideas...
# Posted By Anang Phatak | 3/18/10 11:24 AM
Jamie Krug's Gravatar Anang, I saw your post on the Railo group and I'm very curious to hear what others say. I have not ideas off the top of my head. I'd be surprised if Railo was queuing requests instead of handling with some sort of concurrency, but I haven't seen any confirm/deny of this.
# Posted By Jamie Krug | 3/18/10 3:54 PM
BlogCFC was created by Raymond Camden. This blog is running version 5.9.2.002. Contact Blog Owner