BTrace: hidden gem in Java developer toolbox

Today's post is about BTrace which I am considering as a hidden gem for Java developer.

BTrace is a safe, dynamic tracing tool for the Java platform. BTrace can be used to dynamically trace a running Java program (similar to DTrace for OpenSolaris applications and OS).

Shortly, the tool allows to inject tracing points without restarting or reconfiguring your Java application while it's running. Moreover, though there are several ways to do that, the one I would like to discuss today is using JVisualVM tool from standard JDK bundle.

What is very cool, BTrace itself uses Java language to define injection trace points. The approach looks very familiar if you ever did aspect-oriented programming (AOP).

So let's get started with a problem: we have an application which uses one of the NoSQL databases (f.e., let it be MongoDB) and suddenly starts to experience significant performance slowdown. Developers suspect that application runs too many queries or updates but cannot say it with confidence. Here BTrace can help.

First thing first, let's run JVisualVM and install BTrace plugin:

JVisualVM should be restarted in order for plugin to appear. Now, while our application is up and running, let's right click on it in JVisualVM applications tree:

The following very intuitive BTrace editor (with simple toolbar) should appear:

This is a place where tracing instrumentation could be defined and dynamically injected into the running application. BTrace has a very rich model in order to define what exactly should be traced: methods, constructors, method returns, errors, .... Also it supports aggregations out of the box so it quite easy to collect a bunch of metrics while application is running. For our problem, we would like to see which methods related to MongoDB are being executed.

As my application uses Spring Data MongoDB, I am interested in which methods of any implementation of org.springframework.data.mongodb.core.MongoOperations interface are being called by application and how long every call takes. So I have defined a very simple BTrace script:

import com.sun.btrace.*;
import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

@BTrace
public class TracingScript {
    @TLS private static String method;

    @OnMethod(
        clazz = "+org.springframework.data.mongodb.core.MongoOperations", 
        method = "/.*/"
    )
    public static void onMongo( 
            @ProbeClassName String className, 
            @ProbeMethodName String probeMethod, 
            AnyType[] args ) {
        method = strcat( strcat( className, "::" ), probeMethod );
    }
    
    @OnMethod(
        clazz = "+org.springframework.data.mongodb.core.MongoOperations", 
        method = "/.*/", 
        location = @Location( Kind.RETURN ) 
    )
    public static void onMongoReturn( @Duration long duration ) {
         println( strcat( strcat( strcat( strcat( "Method ", method ), 
            " executed in " ), str( duration / 1000 ) ), "ms" ) );
    }
}

Let me explain briefly what I am doing here. Basically, I would like to know when any method of any implementation of org.springframework.data.mongodb.core.MongoOperations is called (onMongo marks that) and duration of the call (onMongoReturn marks that in turn). Thread-local variable method holds full qualified method name (with a class), while thanks to useful BTrace predefined annotation, duration parameter holds the method execution time (in nanoseconds). Though it's pure Java, BTrace allows only small subset of Java classes to be used. It's not a problem as com.sun.btrace.BTraceUtils class provides a lot of useful methods (f.e., strcat) to fill the gaps. Running this script produces following output:

** Compiling the BTrace script ...
*** Compiled
** Instrumenting 1 classes ...
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 25ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 3ms
Method org.springframework.data.mongodb.core.MongoTemplate::getDb executed in 22ms
Method org.springframework.data.mongodb.core.MongoTemplate::prepareCollection executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::prepareCollection executed in 19ms
Method org.springframework.data.mongodb.core.MongoTemplate::access$100 executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::access$100 executed in 1ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 3ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::getDb executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::prepareCollection executed in 1ms
Method org.springframework.data.mongodb.core.MongoTemplate::prepareCollection executed in 6ms
Method org.springframework.data.mongodb.core.MongoTemplate::access$100 executed in 1ms
Method org.springframework.data.mongodb.core.MongoTemplate::access$100 executed in 0ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 1ms
Method org.springframework.data.mongodb.core.MongoTemplate::getDb executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::prepareCollection executed in 1ms
Method org.springframework.data.mongodb.core.MongoTemplate::prepareCollection executed in 6ms
Method org.springframework.data.mongodb.core.MongoTemplate::access$100 executed in 1ms
Method org.springframework.data.mongodb.core.MongoTemplate::access$100 executed in 0ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 2ms
Method org.springframework.data.mongodb.core.MongoTemplate::maybeEmitEvent executed in 1ms
...

As you can see, output contains bunch of inner classes which could easily be eliminated by providing more precise method name templates (or maybe even tracing MongoDB driver instead).

I have just started to discover BTrace but I definitely see a great value for me as a developer from using this awesome tool. Thanks to BTrace guys!

Using Redis with Spring

As NoSQL solutions are getting more and more popular for many kind of problems, more often the modern projects consider to use some (or several) of NoSQLs instead (or side-by-side) of traditional RDBMS. I have already covered my experience with MongoDB in this, this and this posts. In this post I would like to switch gears a bit towards Redis, an advanced key-value store.

Aside from very rich key-value semantics, Redis also supports pub-sub messaging and transactions. In this post I am going just to touch the surface and demonstrate how simple it is to integrate Redis into your Spring application.

As always, we will start with Maven POM file for our project:

 4.0.0
 com.example.spring
 redis
 0.0.1-SNAPSHOT
 jar

 
  UTF-8
  3.1.0.RELEASE
 

 
  
   org.springframework.data
   spring-data-redis
   1.0.0.RELEASE
  

  
   cglib
   cglib-nodep
   2.2
  

  
   log4j
   log4j
   1.2.16
  

  
   redis.clients
   jedis
   2.0.0
   jar
  

  
   org.springframework
   spring-core
   ${spring.version}
  

  
   org.springframework
   spring-context
   ${spring.version}
  
 

Spring Data Redis is the another project under Spring Data umbrella which provides seamless injection of Redis into your application. The are several Redis clients for Java and I have chosen the Jedis as it is stable and recommended by Redis team at the moment of writing this post.

We will start with simple configuration and introduce the necessary components first. Then as we move forward, the configuration will be extended a bit to demonstrated pub-sub capabilities. Thanks to Java config support, we will create the configuration class and have all our dependencies strongly typed, no XML anymore:

package com.example.redis.config;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.serializer.GenericToStringSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;

@Configuration
public class AppConfig {
 @Bean
 JedisConnectionFactory jedisConnectionFactory() {
  return new JedisConnectionFactory();
 }

 @Bean
 RedisTemplate< String, Object > redisTemplate() {
  final RedisTemplate< String, Object > template =  new RedisTemplate< String, Object >();
  template.setConnectionFactory( jedisConnectionFactory() );
  template.setKeySerializer( new StringRedisSerializer() );
  template.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
  template.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
  return template;
 }
}

That's basically everything we need assuming we have single Redis server up and running on localhost with default configuration. Let's consider several common uses cases: setting a key to some value, storing the object and, finally, pub-sub implementation. Storing and retrieving a key/value pair is very simple:

@Autowired private RedisTemplate< String, Object > template;

public Object getValue( final String key ) {
    return template.opsForValue().get( key );
}

public void setValue( final String key, final String value ) {
    template.opsForValue().set( key, value );
}

Optionally, the key could be set to expire (yet another useful feature of Redis), f.e. let our keys expire in 1 second:

public void setValue( final String key, final String value ) {
    template.opsForValue().set( key, value );
    template.expire( key, 1, TimeUnit.SECONDS );
}

Arbitrary objects could be saved into Redis as hashes (maps), f.e. let save instance of some class User

public class User {
 private final Long id;
 private String name;
 private String email;
       
    // Setters and getters are omitted for simplicity
}

into Redis using key pattern "user:<id>":

public void setUser( final User user ) {
 final String key = String.format( "user:%s", user.getId() );
 final Map< String, Object > properties = new HashMap< String, Object >();

 properties.put( "id", user.getId() );
 properties.put( "name", user.getName() );
 properties.put( "email", user.getEmail() );

 template.opsForHash().putAll( key, properties);
}

Respectively, object could easily be inspected and retrieved using the id.

public User getUser( final Long id ) {
 final String key = String.format( "user:%s", id );

 final String name = ( String )template.opsForHash().get( key, "name" );
 final String email = ( String )template.opsForHash().get( key, "email" );

 return new User( id, name, email );
}

There are much, much more which could be done using Redis, I highly encourage to take a look on it. It surely is not a silver bullet but could solve many challenging problems very easy. Finally, let me show how to use a pub-sub messaging with Redis. Let's add a bit more configuration here (as part of AppConfig class):

@Bean
MessageListenerAdapter messageListener() {
 return new MessageListenerAdapter( new RedisMessageListener() );
}

@Bean
RedisMessageListenerContainer redisContainer() {
 final RedisMessageListenerContainer container = new RedisMessageListenerContainer();

 container.setConnectionFactory( jedisConnectionFactory() );
 container.addMessageListener( messageListener(), new ChannelTopic( "my-queue" ) );

 return container;
}

The style of message listener definition should look very familiar to Spring users: generally, the same approach we follow to define JMS message listeners. The missed piece is our RedisMessageListener class definition:

package com.example.redis.impl;

import org.springframework.data.redis.connection.Message;
import org.springframework.data.redis.connection.MessageListener;

public class RedisMessageListener implements MessageListener {
 @Override
 public void onMessage(Message message, byte[] paramArrayOfByte) {
  System.out.println( "Received by RedisMessageListener: " + message.toString() );
 }
}

Now, when we have our message listener, let see how we could push some messages into the queue using Redis. As always, it's pretty simple:

@Autowired private RedisTemplate< String, Object > template;

public void publish( final String message ) {
 template.execute(
  new RedisCallback< Long >() {
   @SuppressWarnings( "unchecked" )
   @Override
   public Long doInRedis( RedisConnection connection ) throws DataAccessException {
    return connection.publish(
     ( ( RedisSerializer< String > )template.getKeySerializer() ).serialize( "queue" ),
     ( ( RedisSerializer< Object > )template.getValueSerializer() ).serialize( message ) );
   }
  }
 );
}

That's basically it for very quick introduction but definitely enough to fall in love with Redis.

JSON for polymorphic Java object serialization

For a long time now JSON is a de facto standard for all kinds of data serialization between client and server. Among other, its strengths are simplicity and human-readability. But with simplicity comes some limitations, one of them I would like to talk about today: storing and retrieving polymorphic Java objects.

Let's start with simple problem: a hierarchy of filters. There is one abstract class AbstractFilter and two subclasses, RegexFilter and StringMatchFilter.

package bean.json.examples;

public abstract class AbstractFilter {
    public abstract void filter();
}

Here is RegexFilter class:

package bean.json.examples;

public class RegexFilter extends AbstractFilter {
    private String pattern;

    public RegexFilter( final String pattern ) {
        this.pattern = pattern;
    }

    public void setPattern( final String pattern ) {
        this.pattern = pattern;
    }

    public String getPattern() {
        return pattern;
    }

    @Override
    public void filter() {
        // Do some work here
    }
}

And here is StringMatchFilter class:

package bean.json.examples;

public class StringMatchFilter extends AbstractFilter {
    private String[] matches;
    private boolean caseInsensitive;

    public StringMatchFilter() {
    }

    public StringMatchFilter( final String[] matches, final boolean caseInsensitive ) {
        this.matches = matches;
        this.caseInsensitive = caseInsensitive;
    }

    public String[] getMatches() {
        return matches;
    }

    public void setCaseInsensitive( final boolean caseInsensitive ) {
        this.caseInsensitive = caseInsensitive;
    }

    public void setMatches( final String[] matches ) {
        this.matches = matches;
    }

    public boolean isCaseInsensitive() {
        return caseInsensitive;
    }

    @Override
    public void filter() {
        // Do some work here
    }
}

Nothing fancy, pure Java beans. Now what if we need to store list of AbstractFilter instances to JSON, and more importantly, to reconstruct this list back from JSON? Following class Filters demonstrates what I mean:

package bean.json.examples;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;

public class Filters {
    private Collection< AbstractFilter > filters = new ArrayList< AbstractFilter >();

    public Filters() {
    }

    public Filters( final AbstractFilter ... filters ) {
        this.filters.addAll( Arrays.asList( filters ) );
    }

    public Collection< AbstractFilter > getFilters() {
        return filters;
    }

    public void setFilters( final Collection< AbstractFilter > filters ) {
        this.filters = filters;
    }
}

As JSON is textual, platform-independent format, it doesn't carry any type specific information. Thanks to awesome Jackson JSON processor it could be easily done. So let's add Jackson JSON processor to our POM file:

    4.0.0

    bean.json
    examples
    0.0.1-SNAPSHOT
    jar

    
        UTF-8
    

    
        
            org.codehaus.jackson
            jackson-mapper-asl
            1.9.6
        
    

Having this step done, we need to tell Jackson that we have an intention to store the type information together with our objects in JSON so it would be possible to reconstruct exact objects from JSON later. Few annotations on AbstractFilter do exactly that.

import org.codehaus.jackson.annotate.JsonSubTypes;
import org.codehaus.jackson.annotate.JsonSubTypes.Type;
import org.codehaus.jackson.annotate.JsonTypeInfo;
import org.codehaus.jackson.annotate.JsonTypeInfo.Id;

@JsonTypeInfo( use = Id.NAME )
@JsonSubTypes(
    {
        @Type( name = "Regex", value = RegexFilter.class ),
        @Type( name = "StringMatch", value = StringMatchFilter.class )
    }
)
public abstract class AbstractFilter {
    // ...
}

And ... that's it! Following helper class does the dirty job of serializing filters to string and deserializing them back from string using Jackson's ObjectMapper:

package bean.json.examples;

import java.io.IOException;
import java.io.StringReader;
import java.io.StringWriter;

import org.codehaus.jackson.map.ObjectMapper;

public class FilterSerializer {
    private final ObjectMapper mapper = new ObjectMapper();

    public String serialize( final Filters filters ) {
        final StringWriter writer = new StringWriter();
        try {
            mapper.writeValue( writer, filters );
            return writer.toString();
        } catch( final IOException ex ) {
            throw new RuntimeException( ex.getMessage(), ex );
        } finally {
            try { writer.close(); } catch ( final IOException ex ) { /* Nothing to do here */ }
        }
    }

    public Filters deserialize( final String str ) {
        final StringReader reader = new StringReader( str );
        try {
            return mapper.readValue( reader, Filters.class );
        } catch( final IOException ex ) {
            throw new RuntimeException( ex.getMessage(), ex );
        } finally {
            reader.close();
        }
    }
}

Let's see this in action. Following code example

final String json = new FilterSerializer().serialize(
    new Filters(
        new RegexFilter( "\\d+" ),
        new StringMatchFilter( new String[] { "String1", "String2" }, true )
    )
);

produces following JSON:

{ "filters":
  [
     {"@type":"Regex","pattern":"\\d+"},
     {"@type":"StringMatch","matches":["String1","String2"],"caseInsensitive":true}
  ]
}

As you can see, each entry in "filters" collection has property "@type" which has the value we have specified by annotating AbstractFilter class. Calling new FilterSerializer().deserialize( json ) produces exactly the same Filters object instance.

Using Delayed queues in practice

Often there are use cases when you have some kind of work or job queue and there is a need not to handle each work item or job immediately but with some delay. For example, if user clicks a button which triggers some work to be done, and one second later user realizes he / she was mistaken and job shouldn't start at all. Or, f.e. there could be a use case when some work elements in a queue should be removed after some delay (expiration).

There are a lot of implementations out there, but one I would like to describe is using pure JDK concurrent framework classes: DelayedQueue and Delayed interface.

Let me start with simple (and empty) interface which defines the work item. I am skipping the implementation details like properties and methods as those are not important.

package com.example.delayed;

public interface WorkItem {
   // Some properties and methods here
}

The next class in our model will represent the postponed work item and implement Delayed interface. There are just few basic concepts to take into account: the delay itself and the actual time the respective work item has been submitted. This is how expiration would be calculated. So let's do that by introducing PostponedWorkItem class.

package com.example.delayed;

import java.util.concurrent.Delayed;
import java.util.concurrent.TimeUnit;

public class PostponedWorkItem implements Delayed {
    private final long origin;
    private final long delay;
    private final WorkItem workItem;

    public PostponedWorkItem( final WorkItem workItem, final long delay ) {
        this.origin = System.currentTimeMillis();
        this.workItem = workItem;
        this.delay = delay;
    }

    @Override
    public long getDelay( TimeUnit unit ) {
        return unit.convert( delay - ( System.currentTimeMillis() - origin ), 
                TimeUnit.MILLISECONDS );
    }

    @Override
    public int compareTo( Delayed delayed ) {
        if( delayed == this ) {
            return 0;
        }

        if( delayed instanceof PostponedWorkItem ) {
            long diff = delay - ( ( PostponedWorkItem )delayed ).delay;
            return ( ( diff == 0 ) ? 0 : ( ( diff < 0 ) ? -1 : 1 ) );
        }

        long d = ( getDelay( TimeUnit.MILLISECONDS ) - delayed.getDelay( TimeUnit.MILLISECONDS ) );
        return ( ( d == 0 ) ? 0 : ( ( d < 0 ) ? -1 : 1 ) );
    }
}

As you can see, we create new instance of the class and save the current system time in internal origin property. The getDelayed method calculates the actual time left before work item gets expired. The delay is external setting which comes as constructor parameter. The mandatory implementation of Comparable<Delayed> is required as Delayed extends this interface.

Now, we are mostly done! To complete the example, let's make sure that same work item won't be submitted twice to the work queue by implementing equals and hashCode (implemenation is pretty trivial and should not require any comments).

public class PostponedWorkItem implements Delayed {
    ...

    @Override
    public int hashCode() {
        final int prime = 31;

        int result = 1;
        result = prime * result + ( ( workItem == null ) ? 0 : workItem.hashCode() );

        return result;
    }

    @Override
    public boolean equals( Object obj ) {
        if( this == obj ) {
            return true;
        }

        if( obj == null ) {
            return false;
        }

        if( !( obj instanceof PostponedWorkItem ) ) {
            return false;
        }

        final PostponedWorkItem other = ( PostponedWorkItem )obj;
        if( workItem == null ) {
            if( other.workItem != null ) {
                return false;
            }
        } else if( !workItem.equals( other.workItem ) ) {
            return false;
        }

        return true;
    }
}

The last step is to introduce some kind of manager which will scheduled work items and periodically polls out expired ones: meet WorkItemScheduler class.

package com.example.delayed;

import java.util.ArrayList;
import java.util.Collection;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.DelayQueue;

public class WorkItemScheduler {
    private final long delay = 2000; // 2 seconds

    private final BlockingQueue< PostponedWorkItem > delayed =
            new DelayQueue< PostponedWorkItem >(); 

    public void addWorkItem( final WorkItem workItem ) {
        final PostponedWorkItem postponed = new PostponedWorkItem( workItem, delay );
        if( !delayed.contains( postponed )) {
            delayed.offer( postponed );
        }
    }

    public void process() {
        final Collection< PostponedWorkItem > expired = new ArrayList< PostponedWorkItem >();
        delayed.drainTo( expired );

        for( final PostponedWorkItem postponed: expired ) {
            // Do some real work here with postponed.getWorkItem()
        }
    }
}

Usage of BlockingQueue guarantees thread safety and high level of concurrency. The process method should be run periodically in order to drain work items queue. It could be annotated by @Scheduled annotation from Spring Framework or by EJB's @Schedule annotation from JEE 6.

Enjoy!

Simple but powerful DSL using Groovy

In one of my projects we had very complicated domain model, which included more than hundred of different domain object types. It was a pure Java project and, honestly, Java is very verbose with respect to object instantiation, initialization and setting properties. Suddenly, the new requirement to allow users define and use own object models came up. So ... the journey begun.

We ended up with the idea that some kind of domain language for describing all those object types and relations is required. Here Groovy came on rescue. In this post I would like to demonstrate how powerful and expressive could be simple DSL written using Groovy builders.

As always, let's start with POM file for our sample project:

 4.0.0

 com.example
 dsl
 0.0.1-SNAPSHOT
 jar

 
  UTF-8
 

 
  
   junit
   junit
   4.10
  
  
   org.codehaus.groovy
   groovy-all
   1.8.4
  
 

 
  
   
    org.codehaus.gmaven
    gmaven-plugin
    1.4
    
     
      
       1.8
       
      
      
       compile
       testCompile
      
     
    
   

   
    org.apache.maven.plugins
    maven-compiler-plugin
    2.3.1
    
     1.6
     1.6
    
   

    
 

I will use the latest Groovy version, 1.8.4. Our domain model will include three classes: Organization, User and Group. Each Organization has a mandatory name, some users and some groups. Each group can have some users as members. Pretty simple, so here are our Java classes.

Organization.java

package com.example;

import java.util.Collection;

public class Organization {
 private String name;
 private Collection< User > users = new ArrayList< User >();
 private Collection< Group > groups = new ArrayList< Group >();
 
   public String getName() {
  return name;
 }

 public void setName( final String name ) {
  this.name = name;
 }

 public Collection< Group > getGroups() {
  return groups;
 }

 public void setGroups( final Collection< Group > groups ) {
  this.groups = groups;
 }

 public Collection< User > getUsers() {
  return users;
 }

 public void setUsers( final Collection< User > users ) {
  this.users = users;
 }
}

User.java

package com.example;

public class User {
 private String name;

 public String getName() {
  return name;
 }

 public void setName( final String name ) {
  this.name = name;
 }
}

Group .java

package com.example;

import java.util.Collection;

public class Group {
 private String name;
 private Collection< User > users = new ArrayList< User >();

 public void setName( final String name ) {
  this.name = name;
 }

 public String getName() {
  return name;
 }

 public Collection< User > getUsers() {
  return users;
 }

 public void setUsers( final Collection< User > users ) {
  this.users = users;
 }
}

Now, we have our domain model. Let think about the way regular user can describe own organization with users, groups and relations between all these objects. Primarily, we taking about some kind of human readable language simple enough for regular user to understand. Meet Groovy builders.

package com.example.dsl.samples

class SampleOrganization {
 def build() {
  def builder = new ObjectGraphBuilder(
   classLoader: SampleOrganization.class.classLoader,
   classNameResolver: "com.example"
  )

  return builder.organization(
   name: "Sample Organization"
  ) {
   users = [
    user(
     id: "john",
     name: "John"
    ),

    user(
     id: "samanta",
     name: "Samanta"
    ),

    user(
     id: "tom",
     name: "Tom"
    )
   ]

   groups = [
    group(
     id: "administrators",
     name: "administrators",
     users: [ john, tom ]
    ),
    group(
     id: "managers",
     name: "managers",
     users: [ samanta ]
    )
   ]
  }
 }
}

And here is small test case which verifies that our domain model is as expected:

package com.example.dsl

import static org.junit.Assert.assertEquals
import static org.junit.Assert.assertNotNull

import org.junit.Test

import com.example.dsl.samples.SampleOrganization

class BuilderTestCase {
 @Test
 void 'build organization and verify users, groups' () {
  def organization = new SampleOrganization().build()

  assertEquals 3, organization.users.size()
  assertEquals 2, organization.groups.size()
  assertEquals "Sample Organization", organization.name
 }
}

I am using this simple DSL again and again across many projects. It's really simplifies a lot complex object models creation.

Storing hierarchical data in MongoDB

Continuing NoSQL journey with MongoDB, I would like to touch one specific use case which comes up very often: storing hierarchical document relations. MongoDB is awesome document data store but what if documents have parent-child relationships? Can we effectively store and query such document hierarchies? The answer, for sure, is yes, we can. MongoDB has several recommendations how to store Trees in MongoDB. The one solution described there as well and quite widely used is using materialized path.

Let me explain how it works by providing very simple examples. As in previous posts, we will build Spring application using recently released version 1.0 of Spring Data MongoDB project. Our POM file contains very basic dependencies, nothing more.

    4.0.0

    mongodb
    com.example.spring
    0.0.1-SNAPSHOT
    jar

    
        UTF-8
        3.0.7.RELEASE
    

    
        
            org.springframework.data
            spring-data-mongodb
            1.0.0.RELEASE
            
                
                    org.springframework
                    spring-beans
                
                
                    org.springframework
                    spring-expression
                
            
        

        
            cglib
            cglib-nodep
            2.2
        

        
            log4j
            log4j
            1.2.16
        

        
            org.mongodb
            mongo-java-driver
            2.7.2
        

        
            org.springframework
            spring-core
            ${spring.version}
        

        
            org.springframework
            spring-context
            ${spring.version}
        

        
            org.springframework
            spring-context-support
            ${spring.version}
        
    

    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                2.3.2
                
                    1.6
                    1.6
                
            
        
    

To properly configure Spring context, I will use configuration approach utilizing Java classes. I am more and more advocating to use this style as it provides strong typed configuration and most of the mistakes could be caught on compilation time, no need to inspect your XML files anymore. Here how it looks like:

package com.example.mongodb.hierarchical;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.mongodb.core.MongoFactoryBean;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.SimpleMongoDbFactory;

@Configuration
public class AppConfig {
    @Bean
    public MongoFactoryBean mongo() {
        final MongoFactoryBean factory = new MongoFactoryBean();
        factory.setHost( "localhost" );
        return factory;
    }

    @Bean
    public SimpleMongoDbFactory mongoDbFactory() throws Exception{
        return new SimpleMongoDbFactory( mongo().getObject(), "hierarchical" );
    }

    @Bean
    public MongoTemplate mongoTemplate() throws Exception {
        return new MongoTemplate( mongoDbFactory() );
    }

    @Bean
    public IDocumentHierarchyService documentHierarchyService() throws Exception {
        return new DocumentHierarchyService( mongoTemplate() );
    }
}

That's pretty nice and clear. Thanks, Spring guys! Now, all boilerplate stuff is ready. Let's move to interesting part: documents. Our database will contain 'documents' collection which stores documents of type SimpleDocument. We describe this using Spring Data MongoDB annotations for SimpleDocument POJO.

package com.example.mongodb.hierarchical;

import java.util.Collection;
import java.util.HashSet;

import org.springframework.data.annotation.Id;
import org.springframework.data.annotation.Transient;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.core.mapping.Field;

@Document( collection = "documents" )
public class SimpleDocument {
    public static final String PATH_SEPARATOR = ".";

    @Id private String id;
    @Field private String name;
    @Field private String path;

    // We won't store this collection as part of document but will build it on demand
    @Transient private Collection< SimpleDocument > documents = new HashSet< SimpleDocument >();

    public SimpleDocument() {
    }

    public SimpleDocument( final String id, final String name ) {
        this.id = id;
        this.name = name;
        this.path = id;
    }

    public SimpleDocument( final String id, final String name, final SimpleDocument parent ) {
        this( id, name );
        this.path = parent.getPath() + PATH_SEPARATOR + id;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getPath() {
        return path;
    }

    public void setPath(String path) {
        this.path = path;
    }

    public Collection< SimpleDocument > getDocuments() {
        return documents;
    }
}

Let me explain few things here. First, magic property path: this is a key to construct and query through our hierarchy. Path contains identifiers of all document's parents, usually divided by some kind of separator, in our case just . (dot). Storing document hierarchical relationships in this way allows quickly build hierarchy, search and navigate. Second, notice transient documents collection: this non-persistent collection is constructed by persistent provider and contains all descendant documents (which, in case, also contain own descendants). Let see it in action by looking into find method implementation:

package com.example.mongodb.hierarchical;

import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;

import org.springframework.data.mongodb.core.MongoOperations;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.data.mongodb.core.query.Query;

public class DocumentHierarchyService {
    private MongoOperations template;

    public DocumentHierarchyService( final MongoOperations template ) {
        this.template = template;
    }

    @Override
    public SimpleDocument find( final String id ) {
        final SimpleDocument document = template.findOne(
            Query.query( new Criteria( "id" ).is( id ) ),
            SimpleDocument.class
        );

        if( document == null ) {
            return document;
        }

        return build(
            document,
            template.find(
                Query.query( new Criteria( "path" ).regex( "^" + id + "[.]" ) ),
                SimpleDocument.class
            )
        );
    }

    private SimpleDocument build( final SimpleDocument root, final Collection< SimpleDocument > documents ) {
        final Map< String, SimpleDocument > map = new HashMap< String, SimpleDocument >();

        for( final SimpleDocument document: documents ) {
            map.put( document.getPath(), document );
        }

        for( final SimpleDocument document: documents ) {
            map.put( document.getPath(), document );

            final String path = document
                .getPath()
                .substring( 0, document.getPath().lastIndexOf( SimpleDocument.PATH_SEPARATOR ) );

            if( path.equals( root.getPath() ) ) {
                root.getDocuments().add( document );
            } else {
                final SimpleDocument parent = map.get( path );
                if( parent != null ) {
                    parent.getDocuments().add( document );
                }
            }
        }

        return root;
    }
}

As you can see, to get single document with a whole hierarchy we need to run just two queries (but more optimal algorithm could reduce it to just one single query). Here is a sample hierarchy and the the result of reading root document from MongoDB

template.dropCollection( SimpleDocument.class );

final SimpleDocument parent = new SimpleDocument( "1", "Parent 1" );
final SimpleDocument child1 = new SimpleDocument( "2", "Child 1.1", parent );
final SimpleDocument child11 = new SimpleDocument( "3", "Child 1.1.1", child1 );
final SimpleDocument child12 = new SimpleDocument( "4", "Child 1.1.2", child1 );
final SimpleDocument child121 = new SimpleDocument( "5", "Child 1.1.2.1", child12 );
final SimpleDocument child13 = new SimpleDocument( "6", "Child 1.1.3", child1 );
final SimpleDocument child2 = new SimpleDocument( "7", "Child 1.2", parent );

template.insertAll( Arrays.asList( parent, child1, child11, child12, child121, child13, child2 ) );

...

final ApplicationContext context = new AnnotationConfigApplicationContext( AppConfig.class );
final IDocumentHierarchyService service = context.getBean( IDocumentHierarchyService.class );

final SimpleDocument document = service.find( "1" );
//  Printing document show following hierarchy:
//
//  Parent 1
//   |-- Child 1.1
//     |-- Child 1.1.1
//     |-- Child 1.1.3
//     |-- Child 1.1.2
//       |-- Child 1.1.2.1
//   |-- Child 1.2

That's it. Simple a powerful concept. Sure, adding index on a path property will speed up query significantly. There are a plenty of improvements and optimizations but basic idea should be clear now.

Testing highly concurrent code

How often are you facing the issues with testing highly concurrent code? It's not so easy to write a test which verifies asynchronous procedure call or verifies that some tasks has been executed by some thread pool worker. Fortunately, it's getting much easier with this awesome library - Awaitility.

Let me demonstrate on a few simple but meaningful enough examples how easy it is to enrich your tests with it. Let's start with a POM file including only necessary stuff - JUnit and Awaitility.

 4.0.0

 com.example
 awaitility
 0.0.1-SNAPSHOT
 jar

 
  UTF-8
 

 
  
   com.jayway.awaitility
   awaitility
   1.3.3
   test
  

  
   junit
   junit
   4.8.2
   test
  
 

Nothing special here. Our class under the test will collect notifications for particular users. As creating the notifications could take some time, the implementation will collect those using thread pool and asynchronous method invocation pattern: the method will return immediately delegating execution to thread pool. Pretty typical design decision. Let take a look on sample implementation.

package com.example.awaitility;

import java.util.Collection;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class AsyncQueueService {
    private final ExecutorService executor = Executors.newFixedThreadPool( 3 );
    private final ConcurrentLinkedQueue< Notification > queue = new ConcurrentLinkedQueue< Notification >();

    public static class Notification {
        private final long userId;

        public Notification( final long userId ) {
            this.userId = userId;
        }

        public long getUserId() {
            return userId;
        }
    }

    public void enqueue( final Collection< Long > users ) {
        executor.execute( new Runnable() {
                public void run() {
                    for( final long userId: users ) {
                        // do some work with notifications
                        queue.add( new Notification( userId ) );
                    }
                }
            }
        );
    }

    public void clear() {
        queue.clear();
    }

    public int size() {
        return queue.size();
    }
}

I omit a bunch of details trying to make code simple and concentrate on important: method enqueue. As we can see, this method delegates all the work to internal thread pool. Now, how would we create a test to verify that this method actually works? It's difficult because method returns immediately, the result of its execution will be available sometime in the future. Mocking thread pool (executor service) is not a very good idea as it uses the internals of implementation. What if we decide to move from thread pool to scheduled task? Test should work without any change. It's where Awaitility comes on a rescue. Let's take a look on this test case:

package com.example.awaitility;

import static com.jayway.awaitility.Awaitility.await;
import static org.hamcrest.core.IsEqual.equalTo;

import java.util.Arrays;
import java.util.concurrent.Callable;

import org.junit.Before;
import org.junit.Test;

public class AsyncQueueServiceTestCase {
    private AsyncQueueService service;

    @Before
    public void setUp() {
        service = new AsyncQueueService();
    }

    @Test
    public void testEnqueueManyNotifications() throws Exception {
        final Long[] users = new Long[] { 1L, 2L, 3L, 4L, 5L };

        service.enqueue( Arrays.asList( users ) );

        await().until(
            new Callable< Integer >() {
                public Integer call() throws Exception {
                    return service.size();
                }
            },
            equalTo( users.length )
        );
    }
}

As we see, the test method calls enqueue with some list of users. The test verifies that same amount of notifications should be in queue as users passed to enqueue method. With Awaitility such assertion is very trivial: just wait till service.size() will be equal to users.length!

I have just touched the surface of Awaitility. It has many features and even specific DSLs for Groovy and Scala. I highly encourage to take a look on it!