ColdFusion 10: Webservice is leaking memory / trusted cache leaks memory

There exists one or more webservice bugs in ColdFusion 10 that will leak all your memory over a very short amount of time. We build a SOAP based webservice with ColdFusion 8 + 10 that is not running stable at all for around 2 years now. First the issues we have seen in CF8 have been caused by the very outdated Axis 1.x version used under the hood, but later in CF10 Axis 2 was integrated and the issues should have been resolved by upgrading the Applications to this version. But SOAP is still a mess under ColdFusion 10 and very very unstable! We are in month 9 with CF10 analysis and gave up with useless Adobe Support.

We migrated our webservice to JSON output format now. Nothing else in the logic has been changed, only the output. The memory leak disappeared with JSON and the webservice runs smooth for several weeks now. Nothing has been changed on server configuration.

One day before we switched from SOAP to JSON I received a hotfix from Adobe that may fix at least a bug in the webservice components that caused multiple initializations of webservices. Something that should never happen. It can be compared to an OnApplicationStart that should be executed only once and block all other request until completed, but CF10 runs several webservice initializations in parallel what is not correct. Adobe has a non-public patch for this available now.

If you'd like to get a better picture what exceptions you need to look for in your log files:

• Caused by: java.lang.reflect.InvocationTargetException
• Caused by: coldfusion.xml.rpc.CFCInvocationException: [java.lang.ClassCastException : datatypes.Foo]
• Caused by: java.lang.ClassCastException: datatypes.Foo

It looks like Adobe CF team is not master of the situation. This story cannot really provide you a solution except the strong recommendation to move to JSON and I'm sorry for this. We discontinued SOAP and moved on to JSON. However you may like to follow up with Adobe Support and maybe they are able to fix the bug with someone else. We found a workaround for the filling up trusted cache that worked very well here, but has not solved the ClassCastException's. We have Fusion Reactor on the boxes and you can get the current memory usage by code. If it reaches a mem_free LT 500000 you are able to run runtime.clearTemplateFolderFromCache() via Cold Fusion Admin API and clear the trusted cache.

<cfscript>
    frapi = createObject("java", "com.intergral.fusionreactor.api.FRAPI").getInstance();
 
    mem_used = frapi.getUsedMemory();
    mem_allocated = frapi.getAllocatedMemory();
    mem_total = frapi.getTotalMemory();
    mem_free = frapi.getFreeMemory();
</cfscript>

The coldfusion application running here has been encapsulated on one CF instance and has a CFML code size of ~3MB. It leaked up to 4GB of trusted cache memory over 1-2 days. The requests made to our webservice have been made by one PHP application only. If you clear the trusted cache your will see the memory is freeing up:

If you call Adobe, feel free to reference and ask for hotfix hf1000-185141397.jar that changes CFAxis2Servlet.class that should solve the multiple instantiating bug of a webservice. This hotfix may solve the memory leak, but I cannot verify myself any longer. This is known to be an issue with CF10 Updater up to 13 and has not tested with CF11. Let me know if you know more or found a solution, please.

History:

• 08/18/2014: Documented the bug to public.
• 12/06/2013: Issue found.