Implementing Paged Collections in dpHibernate

A colleague posted an interesting question regarding lazy loading, and I realized that it’s something that we could be doing better in dpHibernate.

Note:  If you’re not familiar with the dpHibernate project, it may to have a read of this overview post

The topic discussed here is currently only available on the changeMessaging branch of dpHibernate.  The code is available here:

http://dphibernate.googlecode.com/svn/branches/changemessaging

Overview of data paging in flex

The process of paging data is used where a large collection of information is returned to the client.  Instead of sending the entire collection (which can be expensive in terms of broadband and client processing time), data is paged across.

When a paginated collection is returned to flex, normally an ArrayCollection arrives at the client with the first n items populated, and proxies in place for the remainder of the collection.  By sending proxies in place of the actual collection members, the length of the array collection is still reported correctly, meaning that view components (such as the list & datagrid etc) display scroll bars correctly.

When a call is made to getItemAt() on an ArrayCollection that references a proxied (not yet loaded) item, the following occurs:

  • Trigger a load to a remote service to load the requested item asynchronously
  • Throw an ItemPendingError
  • When the load request completes, replace the proxy in the array collection at the requested index with the newly loaded item.

View components in the flex framework are smart enough to only request the items that need to be displayed.

If a collection of 1000 items is returned to the client and displayed in a list component which is high enough to display 10 items, the first 10 will be retrieved.  As the user scrolls around in the List component, items are loaded non-sequentially as required.  Eg.,  If a user scrolls down to the 200th item, items 200 – 210 are loaded, though items 11 – 199 are skipped, as these are not required to be displayed.

dpHibernate

Previously, dpHibernate had lazy loading of collections, but no support for paging large collections.

If an entity declares a property whose value is a collection, and the value has not yet been loaded from the database at serialization time, it’s sent to the flex client as a collection of uninitialized dpHibernate proxies. This allows that on the flex client when looking at the collection for the first time, the length is reported correctly, even though none of the data has yet been loaded.

When the flex client accesses the property, currently the entire collection is loaded. This works fine for a collection with 10 items, but causes big problems if there’s 1000.

What would be better is to page the data down in the same way that LCDS does it – ie, return the proxies, but only load the members required to show on screen.

Implementation

Let’s step through how this works.

First of all, you need to enable paging. The new paging of collections in dpHibernate is disabled by default. To turn it on, set a page size in the hibernateSerializer bean in the spring config, as follows:

<bean id="hibernateSerializerBean"
class="net.digitalprimates.persistence.translators.hibernate.HibernateSerializer" scope="prototype">
    <property name="pageSize" value="3"/>
</bean>

When a collection is being serialized with paging enabled, the returned collection contains a mixture of the real instances of the collections members, and special lightweight proxies to the remainder of the collections members.

These proxies are instances of the entity class, but with none of it’s properties populated except for the proxyKey, and proxyInitialized set to false.

ArrayCollection and IList

ArrayCollections are one of the few instances of composition within the flex framework. ArrayColellections work by managing an internal IList which contians it’s members. By default, this is an instance of an ArrayList, however we can modify the behaviour by passing in a different implementation – in our case, one that facilitates lazy loading.

ArrayCollection’s methods such as getItemAt() (which is used by the DataGrid, and other components typically associated with lazy loading) are delegated down to the internal IList method. It’s here that the framework expects us to throw the ItemPendingError if the item we’re after isn’t held locally.

In order to get our lazy loading going, we need to augment the behaviour of this list to detect our proxy objects, load the item, and throw the ItemPendingError. DpHibernate uses a class called ManagedArrayList to facilitate this.

When the results of a remote call are returned, they’re examined to see if there’s an array collection there, and – if so – if that array collection has been paged. In that case, we swap out the default ArrayList with an instance of the ManagedArrayList:

private static function manageArrayCollection(collection:ArrayCollection,ro:IHibernateRPC):void
{
    // Trimmed..
    if (isPagedCollection)
    {
        var managedArrayList:ManagedArrayList = new ManagedArrayList(collection.source);
        collection.list = managedArrayList;
    }
}

Now, inside our ManagedArrayList, we need to detect requests for proxied data:

override public function getItemAt(index:int, prefetch:int=0) : Object
{
    var result : Object = super.getItemAt(index,prefetch);
    if (result is IHibernateProxy && IHibernateProxy(result).proxyInitialized == false)
    {
        handleRemoteItem(IHibernateProxy(result),index,prefetch);
    }
    return result;
}
private function handleRemoteItem(proxy:IHibernateProxy,index:int,prefetch:int):void
{
    var remoteService : IHibernateRPC = HibernateManaged.getIHibernateRPCForBean( proxy );
    var token : AsyncToken = remoteService.loadProxy(proxy.proxyKey,proxy);
    token.addResponder(new Responder(onPendingItemLoaded,onFault));
    var itemPendingError : ItemPendingError = new ItemPendingError("Item is pending");
    var pendingItem:PendingItem = new PendingItem(itemPendingError,index);
    pendingItems[token] = pendingItem;
    throw itemPendingError;
}

When a call is made to getItemAt(), we check to see if it’s an uninitialised IHibernateProxy instance. If so, control is passed off to handleRemoteItem().

Loading data from the server

dpHibernate uses an extension of the RemoteObject class called HibernateRemoteObject.  Objects which arrive on the client as the result of a call to a method on a HibernateRemoteObjectA get “managed”.  Among the tasks performed when “managing” an object, dpHibernate stores a reference to the HibernateRemoteObject instance from which the remote object arrived.  This allows us to get a reference back to the server from the remote object.

A reference to this HibernateRemoteObject is retrieved by calling:

var remoteService : IHibernateRPC = HibernateManaged.getIHibernateRPCForBean( proxy );

Using this reference, a call is issued to the server to load the real value of the proxy, and an ItemPendingError is thrown.

ItemPendingError

The ItemPendingError is not a dpHibernate specific concept – instead, it’s provided by the flex framework for classes to provide support for paginated & remote data.  Some flex framework components already provide support for the ItemPendingError out of the box – such as the List control and the Datagrid control.

You’ll notice that the thrown error is stored along with other details of the call. This is because other classes which catch the ItemPendingError can assign IResponder(s) to get notified when the item has been loaded. By storing the error, we keep reference to this list of responders, and can invoke them accordingly:

private function onPendingItemLoaded(data:Object):void
{
    var resultEvent:ResultEvent = ResultEvent(data);
    var token:AsyncToken = resultEvent.token;
    var pendingItem : PendingItem = pendingItems[token];
    var result:Object = resultEvent.result;
    this.setItemAt(result,pendingItem.index);
    for each ( var responder : IResponder in pendingItem.error.responders )
    {
        responder.result(data);
    }
    delete pendingItems[token]
}

When the item is loaded from the server, the proxy in the list is replaced at the appropriate index. Then, all other responders are invoked so they can take any action necessary.

Advertisements
Tagged

4 thoughts on “Implementing Paged Collections in dpHibernate

  1. […] Lazy loading & pagination of collections within dpHibernate […]

  2. Stever says:

    hey, I’ve I have sent a couple questions by email, and appreciate if you could respond to me.
    I’d like to see what other people are asking as well.
    Is this a good place to post, or is there a forum somewhere else?
    Thanks

    • Marty Pitt says:

      Hi Stever.

      Not sure if this comment is spam or not – Akismet tells you’re spammy. Your knowledge of my poor email response times suggests your a real person! 🙂

      Anyway – there’s a google mailing list for dpHibernate you can join here: http://groups.google.com/group/dphibernate

      That’s the best spot for support.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: