Implementing Paged Collections in dpHibernate

A colleague posted an interesting question regarding lazy loading, and I realized that it’s something that we could be doing better in dpHibernate.

Note:  If you’re not familiar with the dpHibernate project, it may to have a read of this overview post

The topic discussed here is currently only available on the changeMessaging branch of dpHibernate.  The code is available here:

Overview of data paging in flex

The process of paging data is used where a large collection of information is returned to the client.  Instead of sending the entire collection (which can be expensive in terms of broadband and client processing time), data is paged across.

When a paginated collection is returned to flex, normally an ArrayCollection arrives at the client with the first n items populated, and proxies in place for the remainder of the collection.  By sending proxies in place of the actual collection members, the length of the array collection is still reported correctly, meaning that view components (such as the list & datagrid etc) display scroll bars correctly.

When a call is made to getItemAt() on an ArrayCollection that references a proxied (not yet loaded) item, the following occurs:

  • Trigger a load to a remote service to load the requested item asynchronously
  • Throw an ItemPendingError
  • When the load request completes, replace the proxy in the array collection at the requested index with the newly loaded item.

View components in the flex framework are smart enough to only request the items that need to be displayed.

If a collection of 1000 items is returned to the client and displayed in a list component which is high enough to display 10 items, the first 10 will be retrieved.  As the user scrolls around in the List component, items are loaded non-sequentially as required.  Eg.,  If a user scrolls down to the 200th item, items 200 – 210 are loaded, though items 11 – 199 are skipped, as these are not required to be displayed.


Previously, dpHibernate had lazy loading of collections, but no support for paging large collections.

If an entity declares a property whose value is a collection, and the value has not yet been loaded from the database at serialization time, it’s sent to the flex client as a collection of uninitialized dpHibernate proxies. This allows that on the flex client when looking at the collection for the first time, the length is reported correctly, even though none of the data has yet been loaded.

When the flex client accesses the property, currently the entire collection is loaded. This works fine for a collection with 10 items, but causes big problems if there’s 1000.

What would be better is to page the data down in the same way that LCDS does it – ie, return the proxies, but only load the members required to show on screen.


Let’s step through how this works.

First of all, you need to enable paging. The new paging of collections in dpHibernate is disabled by default. To turn it on, set a page size in the hibernateSerializer bean in the spring config, as follows:

<bean id="hibernateSerializerBean"
class="net.digitalprimates.persistence.translators.hibernate.HibernateSerializer" scope="prototype">
    <property name="pageSize" value="3"/>

When a collection is being serialized with paging enabled, the returned collection contains a mixture of the real instances of the collections members, and special lightweight proxies to the remainder of the collections members.

These proxies are instances of the entity class, but with none of it’s properties populated except for the proxyKey, and proxyInitialized set to false.

ArrayCollection and IList

ArrayCollections are one of the few instances of composition within the flex framework. ArrayColellections work by managing an internal IList which contians it’s members. By default, this is an instance of an ArrayList, however we can modify the behaviour by passing in a different implementation – in our case, one that facilitates lazy loading.

ArrayCollection’s methods such as getItemAt() (which is used by the DataGrid, and other components typically associated with lazy loading) are delegated down to the internal IList method. It’s here that the framework expects us to throw the ItemPendingError if the item we’re after isn’t held locally.

In order to get our lazy loading going, we need to augment the behaviour of this list to detect our proxy objects, load the item, and throw the ItemPendingError. DpHibernate uses a class called ManagedArrayList to facilitate this.

When the results of a remote call are returned, they’re examined to see if there’s an array collection there, and – if so – if that array collection has been paged. In that case, we swap out the default ArrayList with an instance of the ManagedArrayList:

private static function manageArrayCollection(collection:ArrayCollection,ro:IHibernateRPC):void
    // Trimmed..
    if (isPagedCollection)
        var managedArrayList:ManagedArrayList = new ManagedArrayList(collection.source);
        collection.list = managedArrayList;

Now, inside our ManagedArrayList, we need to detect requests for proxied data:

override public function getItemAt(index:int, prefetch:int=0) : Object
    var result : Object = super.getItemAt(index,prefetch);
    if (result is IHibernateProxy && IHibernateProxy(result).proxyInitialized == false)
    return result;
private function handleRemoteItem(proxy:IHibernateProxy,index:int,prefetch:int):void
    var remoteService : IHibernateRPC = HibernateManaged.getIHibernateRPCForBean( proxy );
    var token : AsyncToken = remoteService.loadProxy(proxy.proxyKey,proxy);
    token.addResponder(new Responder(onPendingItemLoaded,onFault));
    var itemPendingError : ItemPendingError = new ItemPendingError("Item is pending");
    var pendingItem:PendingItem = new PendingItem(itemPendingError,index);
    pendingItems[token] = pendingItem;
    throw itemPendingError;

When a call is made to getItemAt(), we check to see if it’s an uninitialised IHibernateProxy instance. If so, control is passed off to handleRemoteItem().

Loading data from the server

dpHibernate uses an extension of the RemoteObject class called HibernateRemoteObject.  Objects which arrive on the client as the result of a call to a method on a HibernateRemoteObjectA get “managed”.  Among the tasks performed when “managing” an object, dpHibernate stores a reference to the HibernateRemoteObject instance from which the remote object arrived.  This allows us to get a reference back to the server from the remote object.

A reference to this HibernateRemoteObject is retrieved by calling:

var remoteService : IHibernateRPC = HibernateManaged.getIHibernateRPCForBean( proxy );

Using this reference, a call is issued to the server to load the real value of the proxy, and an ItemPendingError is thrown.


The ItemPendingError is not a dpHibernate specific concept – instead, it’s provided by the flex framework for classes to provide support for paginated & remote data.  Some flex framework components already provide support for the ItemPendingError out of the box – such as the List control and the Datagrid control.

You’ll notice that the thrown error is stored along with other details of the call. This is because other classes which catch the ItemPendingError can assign IResponder(s) to get notified when the item has been loaded. By storing the error, we keep reference to this list of responders, and can invoke them accordingly:

private function onPendingItemLoaded(data:Object):void
    var resultEvent:ResultEvent = ResultEvent(data);
    var token:AsyncToken = resultEvent.token;
    var pendingItem : PendingItem = pendingItems[token];
    var result:Object = resultEvent.result;
    for each ( var responder : IResponder in pendingItem.error.responders )
    delete pendingItems[token]

When the item is loaded from the server, the proxy in the list is replaced at the appropriate index. Then, all other responders are invoked so they can take any action necessary.


DpHibernate – An Overview

I’m going to start posting a series of articles that discuss some of the work I’ve been doing recently on dpHibernate.

I’m order to prevent repeating the basics of how dpHibernate works in each blog, here’s a little primer on the subject.


dpHibernate is a BlazeDS adapter to provide support for Hibernate entities in actionscript. It’s open source, and the code is available from  Many of the topics discussed on this blog specifically target the “changeMessaging” branch, which at the time of writing has not been merged back into the trunk.  The branch is available for checking out here:

Lazy Loading

dpHibernate’s initial goal was to provide an adapter which handles serialization of a Hibernate object effectively. Eg: Imagine the following structure:

public class Author extends BaseEntity {
    public var id : int;
    public var name : String;
    public var age : int;
    public var publisher : Publisher;
    public var books : ArrayCollection;
public class Book extends BaseEntity {
    public var title : String;
    public var author : Author;
public class Publisher extends BaseEntity
    public var authors : ArrayCollection;
    public var books : ArrayCollection;

If a service is written returning a Book object from hibernate to the flex client, it’s possible that a LazyLoading exception is thrown when serializing the author property. This is because the entity is lazily loaded, and has not yet been retrieved from the database.

While one possible solution here is to load the author during serialization, this also requires loading the Author.publisher property, and the Author.books() collection, which in turn will load many more books etc. This approach can quickly becomes too expensive and bandwidth-heavy to use in a real world scenario.

Facilitating Lazy Loading with the IHibernateProxy

Instead of serializing these entities, dpHibernate sends down a proxy in their place. If and when the flex client accesses the property represented by the proxy, then that entity is retrieved from the server. Again, only the primitive values of the returned entity are serialized and sent across to the flex client. Ie., when reading for the first time, the Author object is retrieved from the server, however it’s publisher property is represented by a proxy until such time as it is required.

Similarly, the property Author.books() is sent as a collection of proxies until such time as they are accessed. Note that because a proxy exists for every entity within the collection, attributes such as the length of the collection are still reported correctly.

In order to facilitate this lazy loading, all entity classes are required to either subclass HibernateBean, or implement IHIbernateProxy, which declares two properties:

public interface IHibernateProxy
    function get proxyKey():Object;
    function set proxyKey( value:Object ):void;

    function get proxyInitialized():Boolean;
    function set proxyInitialized( value:Boolean ):void;

Normally, the developer doesn’t access or modify these properties directly – instead they are there to facilitate the lazy loading process.

Specifics of how the lazy loading works are beyond the scope of this tutorial, but will be discussed in a later blog post.

Entity Persistence

dpHibernate provides functionality for persisting entities without having to write update services for each entity, through a simple save() method of the IUpdateableHibernateProxy interface:

public interface IUpdatableHibernateProxy extends IHibernateProxy
    function save(responder:IResponder=null) : AsyncToken;
    function deleteRecord(responder:IResponder=null) : AsyncToken;

dpHibernate provides support for creating, deleting and updating entities, including support for nested entities and collections.


public class SomePM {
    public function doSomeWork(book:Book):void

The entity persistence features are discussed in detail in an upcoming blog.

Further reading

That’s the broad overview of dpHibernate.

Other, more detailed blogs cover off:


Spring Security 3.0.0-RC1 and Spring BlazeDS Integration

Just a quick one : After much pain of trying to upgrade to Spring 3.0.0-RC1, it turns out that the security is not compatibale with Spring BlazeDS 1.0.x. It results in the following error:


So, head over to Spring and go vote on this issue:

I hope this gets resolved before 3.0.0-GA


Rule based & asynchronous validation in Flex forms

Sometimes in Flex, you’ll need to perform asynchronous form validation. where rather than simply checking that a string matches certina criteria, you’ll need to take a trip to the server — Eg., checking to see if a username is already used during signup.

In scenario’s like this, the out-of-the-box validation framework just doesn’t cut it. I’ve tried shoe-horning the standard Validator class to block validation, and it’s just not pretty.

However, I remain a fan of declarative form validation. It’s possible to write screeds of validation code in Presentation Model / Mediator classes, but I still prefer a simple bit of markup that describes the validation being applied.

To address this, I’ve put together a very small validation framework that handles asyncronous validation, amongst others. Instead of decalring Validators in your view, instead decalre a ValidationCollection with a bunch of ValidationRules. Here’s an example:

 <mx:Button label="Register" enabled="{ validationRules.isValid }" /> <validation:ValidationRuleCollection id="validationRules"> <validation:UsernameIsUniqueValidationRule username="{ txtUsername.text }" targetComponent="{ txtUsername }" /> <validation:EmailIsUniqueValidationRule email="{ txtEmailAddress.text }" targetComponent="{ txtEmailAddress }" /> <validation:PasswordsMustMatchValidationRule password1="{ txtPassword.text }" password2="{ txtPasswordConfirm.text }" targetComponent="{ txtPasswordConfirm }" /> <mx:StringValidator required="true" source="{ txtUsername }" property="text" requiredFieldError="{ ResourceManager.getInstance().getString( ResourceBundles.ERROR_MESSAGES , 'REQUIRED_FIELD_ERROR' )}" /> </validation:ValidationRuleCollection> 

The ValidationCollection serves to aggregate the results of all it’s enclosing validation rules, and give a single point useful for databinding. In the above example, the Register button is not enabled unless all the validation rules have passed.

Declared inside are a series of discreet validation rules. The interface here is a lot less prescriptive than using the Validator base class, and it’s possible to write rules as simple or as complex as required.

Eg., in the above example there’s asyncronous checks to see if username & email are both available. In another case, there’s a rule which compares the text values of two passwords to make sure they match.

Also note that you can mix and match validation rules and Validator subclasses (such as the StringValidator above). This prevents you from having to re-invent the wheel by utilising all the validators that come out-of-the-box with Flex.

Under the covers, Validator instances are decorated to instances of IValidationRule to get things working.

Source code is available here, (or as a zip file here) along with some sample validation rules… should you find it useful.


Wow, this is the most popular blog on my whole site.  I’ve fixed the broken link, and have ported the code across to GitHub.  Please feel free to fork and contribute!

BlazeDS, Enums, and a comedy of very small errors

For the past two nights while working on my side project, I’ve been banging my head against the wall trying to pass Enums back and forth across BlazeDS to my Flex Client.

I know it’s possible, after all, I’ve seen it on the internet….

However, depsite what ol’ man google said, I was getting errors when sending the enums back from Flex to my server (though they went fine the other way).

In addition to the standard Flex / BlazeDS stack, I’ve been using two very handy open source tools :

  • dpHibernate to assist with serialization of Hibernate Lazy proxies
  • DTO2FX for some really funky AS class generation

Turns out that very small bugs inside each of these were to blame.  I post here in case — through some odd twist of fate —  you, dear reader, have the same stack:


dpHibernate was attempting to deserialize Enum classes as a hibernate proxy.  Adding a test for Enum inside the isSimple method of the HibernateSerializer took care of that:

    private boolean isSimple(Object obj)
        return ((obj == null)
                || (obj instanceof String)
                || (obj instanceof Character)
                || (obj instanceof Boolean)
                || (obj instanceof Number)
                || (obj instanceof Date)
                || (obj instanceof Calendar)
                || (obj instanceof Document))
                || (obj instanceof Enum);


DTO2FX uses an (IMHO) overly complicated template for their generated Enums.  Rather than just use the method discussed here, which allows for a simple enum client side class, they use a much more complex template.  It requires the use of a custom Serializer / Deserializer within Blaze.

Here, I thought of a few approaches:

Change the template used by DTO2FX and recompile my custom version

This requires downloading no less than 5 seperate open source projects from CVS (who the hell uses CVS anymore?) and working out what was going on.  I have previously given up on changing DTO2FX as a simple task, so didn’t even bother this time.

Don’t use DTO2FX to generate the Enums — do it by hand

I’m not using a large number of Enums in my project, and I can live with hand coding them.  However, one of the wonderful features of DTO2FX, is that it uses compile time checking of your Java project to ensure that every dependent type has been generated.  If you miss a type out (even intentionally), you get an error at compile time.  Sadly, there’s no way to turn this behaviour off — which meant I had to consider:

Stop using DTO2FX completely

I seriosuly considered this.  I love DTO2FX because it’s quick to use, maintains my classes in the background, and uses the Generation Gap pattern, meaning that in my generated classes I can still put custom code.  However, this is not the first time that I’ve ground to a halt because of some quirk in the way that DTO2FX generates it’s code.

Also, I find that having to make an architectural decision on my project because a tool won’t generate code in a specific way is unacceptable.  If it wasn’t so good in other ways, I’d have ditched it.

I googled for an appropriate replacement for DTO2FX, but couldn’t find one.  Which left me with….

Make DTO2FX’s Enums work.

Although the fix is simple, the debugging and locating the fault was the hard part.

In the end, all it took was adding this override to EnumPropertyProxy :

    public Object createInstance(String className)
        Class cl = AbstractProxy.getClassFromClassName(className);

        if (cl.isEnum())
            return new EnumHolder(cl);
            throw new IllegalArgumentException("**** samples.EnumProxy registered for a class which is not an enum: " + cl.getName());

And from there, (after I reverted all my other hackery) it worked.



Tagged ,

FB – Internal build error

Just a quick post on a flex builder 3 bug I just found.

Defining a constant on an interface causes Flex Builder to give you the ever helpful message at compile time :

“An internal build error has occurred.  Right click for more information”

Of course, there is no more information, you have to find what you did to break it through old fashioned sleuth work.

Anyway, constants on interfaces :  A stupid mistake, yes… but it’d be to have the compiler issue a stern telling off rather than flex builder throw it’s hands in the air and give up on me completely.

Really, Flex Builder – after all you and I have been through!?

Faithfully yours,


Morgan Stanley – Matrix

I’m now finally allowed to post information about a project I worked on : Morgan Stanley’s  “Matrix

It’s by far the biggest flex application I’ve ever worked on, and probably one of the most significant uses of the Flex platform to date.

The client, Morgan Stanley is one of the biggest investment banks in the world, and their product – Matrix is a trading platform that allows rich, real time collaboration through chat, video, sharing of trade ideas, charts.  

The development team was full of some of the top developers I’ve ever worked with, working on the project was a priveledge and a non-stop learning experience!

Check out the site, and take the tour!

dpHibernate and Spring transactions

I’ve recently been getting my feet wet in Java, and starting integrating with dpHibernate to facilitate lazy loading across to Flex.

Here’s what my stack looks like:

  • Spring 2.5.6
  • Hibernate 3.3 (including annotations, and transaction management)

Trying to get dpHibernate wired in, I was getting lots of errors along the lines of the following:

No Hibernate Session bound to thread, and configuration does not allow creation of non-transactional one here


The problem here is that using transaction management, things get a lot tighter on how and when you can get access to a hibernate session.  dpHibernate tries to get it’s session by essentially calling sessionManager.getCurrentSession();

However, as far as Spring’s transaction manager (which is wired into the SessionManager) is concerned, the thread which is executing has no session, and has no business trying to create one — specifically, outside of a transaction.

After spending several hours trying to work out how to sidestep Springs transaction management, I realized the trick was simply to wire dpHibernate closer to spring.  This is tricky, because dpHibernate is using an extension of the flex’s JavaAdapter, and exists completely outside of the Spring infrastructure.

HibernateSerializer makes a call to a dpHibernate object called SessionManager, which returns the current session outside of a valid Spring transaction.

So, I ditched the SessionManager completely, and replaced it with a good old fashionsed SessionFactory object, as follows:

    private SessionFactory sessionFactory;

    public void setSessionFactory(SessionFactory sessionFactory) {
        this.sessionFactory = sessionFactory;

    public SessionFactory getSessionFactory() {
        return sessionFactory;

The @Resource tag there tells Spring to give me a wired up sessionFactory instance.

I also decorated the HibernateSerializer with a @Transactional tag, so that Spring knows the context in which this will operate:

public class HibernateSerializer implements ISerializer

Now, if I add the HibernateSerializer as a bean, Spring knows to give it access to a sessionManager, and to do so in the context of an active transaction.

Add the bean declaration as follows:

<bean id="hibernateSerializerBean" class="net.digitalprimates.persistence.translators.hibernate.HibernateSerializer" />

Nearly there. Finally, I just need to tell dpHibernate to go and fetch this wired up Serializer from Spring, rather than instantiate it’s own. Inside SerializationFactory, I modified the getSerializer() call to read as follows:

    public static ISerializer getSerializer(String type)
        if( HIBERNATESERIALIZER.equals(type) )
            ServletContext ctx = FlexContext.getServletContext();
            WebApplicationContext springContext = WebApplicationContextUtils.getRequiredWebApplicationContext(ctx);
            ISerializer serializer = (ISerializer) springContext.getBean("hibernateSerializerBean");
            if (serializer == null)
                throw new RuntimeException("bean named hibernateSerializerBean not found");
            return serializer;

        throw new RuntimeException("unsupport serialization type: " +type);

Note that I’ve hard coded the name of the bean here for simplicity sakes. Chances are you’ll want to paramaterize that.

That’s it! Now, calls that pass through dpHibernate happen with a Spring managed transactional session. Great!


Edit: Espen came up with a simpler alternative over on his blog — check it out :

Tagged , , ,

Conditional compilation in Flex using nAnt

Here’s a trick which you may already know about (after all, it’s in the docs) but I only discovered it today when I needed it.

Conditional compilation allows you to vary the classes and functions included in a compilation based on a config value.

Eg., You may have a slightly different codebase used for your debug build than for your release build.

For example, on my current project we have a couple of convenient hacks in use in our test environment to prevent us from needing to log in every time we launch a debug session.

However, somewhat predictably, one of these hacks recently made it through to our testing team, and broke all sorts of stuff.

So, I figured I’d find a way to automatically remove it as part of our build process.

The class got modified to look a little something like this:

public class PrepCommand extends SimpleCommand
      public override function execute(notification:INotification):void

      private function doDebugLogon():void
            trace("This is where the logon should happen");

      private function doDebugLogon():void {}

Note that the doDebugLogon() method is defined twice within the method, varied by the CONFIG attribute.

This attribute defines under which compilation configuration to include the method. You can do the same thing for a full class.

Next, I modified the compiler-config.xml file used for our project to include this snippet:


Then, simply swapping the values of debugging & release would modifiy which of the function calls are included in the compiled source.

Not bad, but still not very automatic.

Because we’re a .NET shop here, our build tool of choice is nAnt. Most of our config files for the flex build are generated from template files, using nAnt variables to give us nice flexible builds. (pardon the pun).

So, I introduced a new target into our build file for performing a release compile:

    <target name="release" description="Performs a release, setting appropriate config values.">
        <property name="debug" value="false" />
        <call target="build" />

…and added a couple of new variables to the properties file:

    <property name="useDebugConfig" value="true" />

    <!-- Derived from useDebugConfig.  Set the value of useDebugConfig rather than setting useReleaseConfig.
        Note that only one of these values must be true, or compilation will fail -->
    <property name="useReleaseConfig" value="${string::to-lower(not(useDebugConfig))}" />

Then, I tweaked the compiler-config described above to look like this:


Now, ensuring the correct code is included is as simple as:

nant // Calls 'build', our default target using the debugging compilation options
nant release // Uses release compilation options

That’s it.

It’s worth reading up on conditional compilation in the docs, as there’s a bunch of great examples in there.

Questions / comments welcome.


Why I think Silverlight has an edge…

In the Silverlight vs Flex debates, I’ve always been staunchly located on the Flex side of the fence.

But, as much as it pains me to say it, it’s beginning to dawn on me that the dirty ol’ boys in the Microsoft labcoats may just have the leading edge.

And, it’s not about browser penetration, or hi-def video, or 3D capabilities, or swanky new sound API’s.

It’s about language.

If you wanna compare these two frameworks, feature-by-feature, then as it stands, Flex wins out — every time.  But what people tend to miss (IMHO) is that we’re comparing frameworks, and not languages.

Do a comparison of Actionscript to .NET, and it’s a whole different kettle of fish.

Comparatively, shipping a new release of a framework with additional features takes much less time than adding features to your language.

In the time span between Alpha and Beta 1 of Silverlight, MS added a bucket load of features to their (somewhat limited) framework.  Beta 1 to Beta 2 saw lots more of the same.  Add to that the development community who will throw their collective weights behind the platform, and you’ll see real traction.

Does the Silverlight UI framework suck?  Maybe.  But it doesn’t matter.  If it’s really that bad, it won’t take long for the development community to provide one that rocks.

We see the same thing every day in the Flex community.  Hate Cairngorm?  Meet PureMVC / Mate.  Not a fan of <some 3d API>, not a problem — we’ve got 3 or 4 alternatives waiting just around the corner.

But — all of these frameworks share the same lowest common denominator — the Actionscript language.

And, it’s a little embarrassing to tell my .NET mates that mid 2009 the Flex community is getting support for typed arrays.  (Woot!)

I might be wrong, I’ve had limited exposure to Silverlight, but from what I understand the language is boasting some pretty impressive .NET features…

  • Typed Arrays
  • Generics
  • A decent reflection API
  • Dynamic Compilation
  • Abstract classes
  • Threading
  • LINQ / Lamba expressions
  • All sorts of C# 3.0 / 3.5 goodness!

Fire up Silverlight, and you’ll get all these things out of the box.  Today.  Right now.

For that matter, if you don’t like the language, you can write your own!  (IronPython, IronRuby)

And, by the time that Flex 4 and our typed arrays make a release, you can expect that C# 4.0 & VB10 will be boasting some funky new features in early preview.

Sure, Flex is an extremely powerful framework…today….comparatively.

But .NET is an extremely powerful language.  And where there’s a language, there’s developers.  And, where’s the developers, there’s eager minds….filling gaps in markets, plugging holes, and pushing boundaries.

Don’t get me wrong.  I think that Adobe are kicking ass with Flex.  In fact, had they not done such a good job, Silverlight probably wouldn’t be the contender it is.

And — come to think of it — Adobe are doing one helluva job in including the development community in guiding where the framework is headed.

So maybe it’s out fault.  Hell – maybe, it’s just me.  Maybe the community really does want a better designer / developer workflow over and above the ability to mark a method as abstract.

But I worry that while we’re getting swanky new ways of integrating designers and developers, the Microsoft team are giving their developers ways to achieve more powerful products with less code.

And, if Paris Hilton has taught us anything, it’s that looks can only get you so far in this world!

I just hope I haven’t backed the wrong horse in this race!