OSGi: the gateway into micro-services architecture

The terms "modularity" and "microservices architecture" pop up quite often these days in context of building scalable, reliable distributed systems. Java platform itself is known to be weak with regards to modularity (Java 9 is going to address that by delivering project Jigsaw), giving a chance to frameworks like OSGi and JBoss Modules to emerge.

When I first heard about OSGi back in 2007, I was truly excited about all those advantages Java applications might benefit of by being built on top of it. But very quickly the frustration took place instead of excitement: no tooling support, very limited set of compatible libraries and frameworks, quite unstable and hard to troubleshoot runtime. Clearly, it was not ready to be used by average Java developer and as such, I had to put it on the shelf. With years, OSGi has matured a lot and gained a widespread community support.

The curious reader may ask: what are the benefits of using modules and OSGi in particular? To name just a few problems it helps to solve:

• explicit (and versioned) dependency management: modules declare what they need (and optionally the version ranges)
• small footprint: modules are not packaged with all their dependencies
• easy release: modules can be developed and released independently
• hot redeploy: individual modules may be redeployed without affecting others

In today's post we are going to take a 10000 feet view on a state of the art in building modular Java applications using OSGi. Leaving aside discussions how good or bad OSGi is, we are going to build an example application consisting of following modules:

• data access module
• business services module
• REST services module

Apache OpenJPA 2.3.0 / JPA 2.0 for data access (unfortunately, JPA 2.1 is not yet supported by OSGi implementation of our choice), Apache CXF 3.0.1 / JAX-RS 2.0 for REST layer are two main building blocks of the application. I found Christian Schneider's blog, Liquid Reality, to be invaluable source of information about OSGi (as well as many other topics).

In OSGi world, the modules are called bundles. Bundles manifest their dependencies (import packages) and the packages they expose (export packages) so other bundles are able to use them. Apache Maven supports this packaging model as well. The bundles are managed by OSGi runtime, or container, which in our case is going to be Apache Karaf 3.0.1 (actually, it is the single thing we need to download and unpack).

Let me stop talking and better show some code. We are going to start from the top (REST) and go all the way to the bottom (data access) as it would be easier to follow. Our PeopleRestService is a typical example of JAX-RS 2.0 service implementation:

package com.example.jaxrs;

import java.util.Collection;

import javax.ws.rs.DELETE;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.UriInfo;

import com.example.data.model.Person;
import com.example.services.PeopleService;

@Path( "/people" )
public class PeopleRestService {
    private PeopleService peopleService;
        
    @Produces( { MediaType.APPLICATION_JSON } )
    @GET
    public Collection< Person > getPeople( 
            @QueryParam( "page") @DefaultValue( "1" ) final int page ) {
        return peopleService.getPeople( page, 5 );
    }

    @Produces( { MediaType.APPLICATION_JSON } )
    @Path( "/{email}" )
    @GET
    public Person getPerson( @PathParam( "email" ) final String email ) {
        return peopleService.getByEmail( email );
    }

    @Produces( { MediaType.APPLICATION_JSON  } )
    @POST
    public Response addPerson( @Context final UriInfo uriInfo,
            @FormParam( "email" ) final String email, 
            @FormParam( "firstName" ) final String firstName, 
            @FormParam( "lastName" ) final String lastName ) {
        
        peopleService.addPerson( email, firstName, lastName );
        return Response.created( uriInfo
            .getRequestUriBuilder()
            .path( email )
            .build() ).build();
    }
    
    @Produces( { MediaType.APPLICATION_JSON  } )
    @Path( "/{email}" )
    @PUT
    public Person updatePerson( @PathParam( "email" ) final String email, 
            @FormParam( "firstName" ) final String firstName, 
            @FormParam( "lastName" )  final String lastName ) {
        
        final Person person = peopleService.getByEmail( email );
        
        if( firstName != null ) {
            person.setFirstName( firstName );
        }
        
        if( lastName != null ) {
            person.setLastName( lastName );
        }

        return person;              
    }
    
    @Path( "/{email}" )
    @DELETE
    public Response deletePerson( @PathParam( "email" ) final String email ) {
        peopleService.removePerson( email );
        return Response.ok().build();
    }
    
    public void setPeopleService( final PeopleService peopleService ) {
        this.peopleService = peopleService;
    }
}

As we can see, there is nothing here telling us about OSGi. The only dependency is the PeopleService which somehow should be injected into the PeopleRestService. How? Typically, OSGi applications use blueprint as the dependency injection framework, very similar to old buddy, XML based Spring configuration. It should be packaged along with application inside OSGI-INF/blueprint folder. Here is a blueprint example for our REST module, built on top of Apache CXF 3.0.1:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:jaxrs="http://cxf.apache.org/blueprint/jaxrs"
    xmlns:cxf="http://cxf.apache.org/blueprint/core"
    xsi:schemaLocation="
        http://www.osgi.org/xmlns/blueprint/v1.0.0 
        http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd
        http://cxf.apache.org/blueprint/jaxws 
        http://cxf.apache.org/schemas/blueprint/jaxws.xsd
        http://cxf.apache.org/blueprint/jaxrs 
        http://cxf.apache.org/schemas/blueprint/jaxrs.xsd
        http://cxf.apache.org/blueprint/core 
        http://cxf.apache.org/schemas/blueprint/core.xsd">

    <cxf:bus id="bus">
        <cxf:features>
            <cxf:logging/>
        </cxf:features>       
    </cxf:bus>

    <jaxrs:server address="/api" id="api">
        <jaxrs:serviceBeans>
            <ref component-id="peopleRestService"/>
        </jaxrs:serviceBeans>
        <jaxrs:providers>
            <bean class="com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider" />
        </jaxrs:providers>
    </jaxrs:server>
    
    <!-- Implementation of the rest service -->
    <bean id="peopleRestService" class="com.example.jaxrs.PeopleRestService">
        <property name="peopleService" ref="peopleService"/>
    </bean>         
    
    <reference id="peopleService" interface="com.example.services.PeopleService" />
</blueprint>

Very small and simple: basically the configuration just states that in order for the module to work, the reference to the com.example.services.PeopleService should be provided (effectively, by OSGi container). To see how it is going to happen, let us take a look on another module which exposes services. It contains only one interface PeopleService:

package com.example.services;

import java.util.Collection;

import com.example.data.model.Person;

public interface PeopleService {
    Collection< Person > getPeople( int page, int pageSize );
    Person getByEmail( final String email );
    Person addPerson( final String email, final String firstName, final String lastName );
    void removePerson( final String email );
}

And also provides its implementation as PeopleServiceImpl class:

package com.example.services.impl;

import java.util.Collection;

import org.osgi.service.log.LogService;

import com.example.data.PeopleDao;
import com.example.data.model.Person;
import com.example.services.PeopleService;

public class PeopleServiceImpl implements PeopleService {
    private PeopleDao peopleDao;
    private LogService logService;
 
    @Override
    public Collection< Person > getPeople( final int page, final int pageSize ) {        
        logService.log( LogService.LOG_INFO, "Getting all people" );
        return peopleDao.findAll( page, pageSize );
    }

    @Override
    public Person getByEmail( final String email ) {
        logService.log( LogService.LOG_INFO, 
            "Looking for a person with e-mail: " + email );
        return peopleDao.find( email );        
    }

    @Override
    public Person addPerson( final String email, final String firstName, 
            final String lastName ) {
        logService.log( LogService.LOG_INFO, 
            "Adding new person with e-mail: " + email );
        return peopleDao.save( new Person( email, firstName, lastName ) );
    }

    @Override
    public void removePerson( final String email ) {
        logService.log( LogService.LOG_INFO, 
            "Removing a person with e-mail: " + email );
        peopleDao.delete( email );
    }
    
    public void setPeopleDao( final PeopleDao peopleDao ) {
        this.peopleDao = peopleDao;
    }
    
    public void setLogService( final LogService logService ) {
        this.logService = logService;
    }
}

And this time again, very small and clean implementation with two injectable dependencies, org.osgi.service.log.LogService and com.example.data.PeopleDao. Its blueprint configuration, located inside OSGI-INF/blueprint folder, looks quite compact as well:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"  
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         
    xsi:schemaLocation="
        http://www.osgi.org/xmlns/blueprint/v1.0.0 
        http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
        
    <service ref="peopleService" interface="com.example.services.PeopleService" />        
    <bean id="peopleService" class="com.example.services.impl.PeopleServiceImpl">
        <property name="peopleDao" ref="peopleDao" />    
        <property name="logService" ref="logService" />
    </bean>
    
    <reference id="peopleDao" interface="com.example.data.PeopleDao" />
    <reference id="logService" interface="org.osgi.service.log.LogService" />
</blueprint>

The references to PeopleDao and LogService are expected to be provided by OSGi container at runtime. Hovewer, PeopleService implementation is exposed as service and OSGi container will be able to inject it into PeopleRestService once its bundle is being activated.

The last piece of the puzzle, data access module, is a bit more complicated: it contains persistence configuration (META-INF/persistence.xml) and basically depends on JPA 2.0 capabilities of the OSGi container. The persistence.xml is quite basic:

<persistence xmlns="http://java.sun.com/xml/ns/persistence"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    version="2.0">
 
    <persistence-unit name="peopleDb" transaction-type="JTA">
        <jta-data-source>
        osgi:service/javax.sql.DataSource/(osgi.jndi.service.name=peopleDb)
        </jta-data-source>       
        <class>com.example.data.model.Person</class>
        
        <properties>
            <property name="openjpa.jdbc.SynchronizeMappings" 
                value="buildSchema"/>         
        </properties>        
    </persistence-unit>
</persistence>

Similarly to the service module, there is an interface PeopleDao exposed:

package com.example.data;

import java.util.Collection;

import com.example.data.model.Person;

public interface PeopleDao {
    Person save( final Person person );
    Person find( final String email );
    Collection< Person > findAll( final int page, final int pageSize );
    void delete( final String email ); 
}

With its implementation PeopleDaoImpl:

package com.example.data.impl;

import java.util.Collection;

import javax.persistence.EntityManager;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.CriteriaQuery;

import com.example.data.PeopleDao;
import com.example.data.model.Person;

public class PeopleDaoImpl implements PeopleDao {
    private EntityManager entityManager;
 
    @Override
    public Person save( final Person person ) {
        entityManager.persist( person );
        return person;
    }
 
    @Override
    public Person find( final String email ) {
        return entityManager.find( Person.class, email );
    }
 
    public void setEntityManager( final EntityManager entityManager ) {
        this.entityManager = entityManager;
    }
 
    @Override
    public Collection< Person > findAll( final int page, final int pageSize ) {
        final CriteriaBuilder cb = entityManager.getCriteriaBuilder();
     
        final CriteriaQuery< Person > query = cb.createQuery( Person.class );
        query.from( Person.class );
     
        return entityManager
            .createQuery( query )
            .setFirstResult(( page - 1 ) * pageSize )
            .setMaxResults( pageSize ) 
            .getResultList();
    }
 
    @Override
    public void delete( final String email ) {
        entityManager.remove( find( email ) );
    }
}

Please notice, although we are performing data manipulations, there is no mention of transactions as well as there are no explicit calls to entity manager's transactions API. We are going to use the declarative approach to transactions as blueprint configuration supports that (the location is unchanged, OSGI-INF/blueprint folder):

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"  
    xmlns:jpa="http://aries.apache.org/xmlns/jpa/v1.1.0"
    xmlns:tx="http://aries.apache.org/xmlns/transactions/v1.0.0" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"         
    xsi:schemaLocation="
        http://www.osgi.org/xmlns/blueprint/v1.0.0 
        http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">
    
    <service ref="peopleDao" interface="com.example.data.PeopleDao" />
    <bean id="peopleDao" class="com.example.data.impl.PeopleDaoImpl">
     <jpa:context unitname="peopleDb" property="entityManager" />
     <tx:transaction method="*" value="Required"/>
    </bean>
    
    <bean id="dataSource" class="org.hsqldb.jdbc.JDBCDataSource">
       <property name="url" value="jdbc:hsqldb:mem:peopleDb"/>
    </bean>
    
    <service ref="dataSource" interface="javax.sql.DataSource"> 
        <service-properties> 
            <entry key="osgi.jndi.service.name" value="peopleDb" /> 
        </service-properties> 
    </service>     
</blueprint>

One thing to keep in mind: the application doesn't need to create JPA 2.1's entity manager: the OSGi runtime is able do that and inject it everywhere it is required, driven by jpa:context declarations. Consequently, tx:transaction instructs the runtime to wrap the selected service methods inside transaction.

Now, when the last service PeopleDao is exposed, we are ready to deploy our modules with Apache Karaf 3.0.1. It is quite easy to do in three steps:

• run the Apache Karaf 3.0.1 container

  bin/karaf (or bin\karaf.bat on Windows)
  

• execute following commands from the Apache Karaf 3.0.1 shell:

  feature:repo-add cxf 3.0.1
  feature:install http cxf jpa openjpa transaction jndi jdbc
  install -s mvn:org.hsqldb/hsqldb/2.3.2
  install -s mvn:com.fasterxml.jackson.core/jackson-core/2.4.0
  install -s mvn:com.fasterxml.jackson.core/jackson-annotations/2.4.0
  install -s mvn:com.fasterxml.jackson.core/jackson-databind/2.4.0
  install -s mvn:com.fasterxml.jackson.jaxrs/jackson-jaxrs-base/2.4.0
  install -s mvn:com.fasterxml.jackson.jaxrs/jackson-jaxrs-json-provider/2.4.0
  

• build our modules and copy them into Apache Karaf 3.0.1's deploy folder (while container is still running):

  mvn clean package
  cp module*/target/*jar apache-karaf-3.0.1/deploy/
  

When you run the list command in the Apache Karaf 3.0.1 shell, you should see the list of all activated bundles (modules), similar to this one:

Where module-service, module-jax-rs and module-data correspond to the ones we are being developed. By default, all our Apache CXF 3.0.1 services will be available at base URL http://:8181/cxf/api/. It is easy to check by executing cxf:list-endpoints -f command in the Apache Karaf 3.0.1 shell.

Let us make sure our REST layer works as expected by sending couple of HTTP requests. Let us create new person:

curl http://localhost:8181/cxf/api/people -iX POST -d "firstName=Tom&lastName=Knocker&email=a@b.com"

HTTP/1.1 201 Created
Content-Length: 0
Date: Sat, 09 Aug 2014 15:26:17 GMT
Location: http://localhost:8181/cxf/api/people/a@b.com
Server: Jetty(8.1.14.v20131031)

And verify that person has been created successfully:

curl -i http://localhost:8181/cxf/api/people

HTTP/1.1 200 OK
Content-Type: application/json
Date: Sat, 09 Aug 2014 15:28:20 GMT
Transfer-Encoding: chunked
Server: Jetty(8.1.14.v20131031)

[{"email":"a@b.com","firstName":"Tom","lastName":"Knocker"}]

Would be nice to check if database has the person populated as well. With Apache Karaf 3.0.1 shell it is very simple to do by executing just two commands: jdbc:datasources and jdbc:query peopleDb "select * from people".

Awesome! I hope this quite introductory blog post opens yet another piece of interesting technology you may use for developing robust, scalable, modular and manageable software. We have not touched many, many things but these are here for you to discover. The complete source code is available on GitHub.

Note to Hibernate 4.2.x / 4.3.x users: unfortunately, in the current release of Apache Karaf 3.0.1 the Hibernate 4.3.x does work properly at all (as JPA 2.1 is not yet supported) and, however I have managed to run with Hibernate 4.2.x, the container often refused to resolve the JPA-related dependencies.

Fault Injection with Byteman and JUnit: do even more to ensure robustness of your applications

The time when our applications lived in isolation have passed long-long ago. Nowadays applications are a very complicated beasts talking to each other using myriads of APIs and protocols, storing data in traditional or NoSQL databases, sending messages and events over the wire ...

How often did you think about what will happen if, for example, a database goes down when your application is actively querying it? Or some API endpoint suddenly starts to refuse connection? Wouldn't it be nice to have such accidents covered as part of your test suite? That's what fault injection and Byteman framework are about.

As an example, we will build a realistic, full-blown Spring application which uses Hibernate/JPA to access MySQL database and manages customers. As part of application's JUnit integration test suite, we will include three kind of test cases:

• store / find a customer
• store customer and try to query database when it's down (fault simulation)
• store customer and a database query times out (fault simulation)

There are only two preconditions for application to run on your local development box:

• MySQL server is installed and has customers database
• Oracle JDK is installed and JAVA_HOME environment variable points to it

That's being said, we are ready to go.

First, let's describe our domain model which consists from single class Customer with id and single property name. It looks as simple as that:

package com.example.spring.domain;

import java.io.Serializable;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.Id;
import javax.persistence.Table;

@Entity
@Table( name = "customers" )
public class Customer implements Serializable{
    private static final long serialVersionUID = 1L;

    @Id
    @GeneratedValue
    @Column(name = "id", unique = true, nullable = false)
    private long id;
 
    @Column(name = "name", nullable = false)
    private String name;
  
    public Customer() {
    }
 
    public Customer( final String name ) {
        this.name = name;
    }

    public long getId() {
        return this.id;
    }

    protected void setId( final long id ) {
        this.id = id;
    }
 
    public String getName() {
        return this.name;
    }

    public void setName( final String name ) {
        this.name = name;
    }
}

For simplicity, the servicing layer is mixed with data access layer and calls database directly. Here is our CustomerService implementation:

package com.example.spring.services;

import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;

import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import com.example.spring.domain.Customer;

@Service
public class CustomerService {
    @PersistenceContext private EntityManager entityManager;
 
    @Transactional( readOnly = true )
    public Customer find( long id ) {
        return this.entityManager.find( Customer.class, id );
    }

    @Transactional( readOnly = false )
    public Customer create( final String name ) {
        final Customer customer = new Customer( name );
        this.entityManager.persist(customer);
        return customer;
    }
 
    @Transactional( readOnly = false )
    public void deleteAll() {
        this.entityManager.createQuery( "delete from Customer" ).executeUpdate();
    }
}

And lastly, the Spring application context which defines data source and transaction manager. A small note here: as we won't introduce data access layer (@Repository) classes, in order for Spring to perform exception translation properly we define PersistenceExceptionTranslationPostProcessor instance to post-process service classes (@Service). Everything else should be very familiar.

package com.example.spring.config;

import java.util.Properties;

import javax.sql.DataSource;

import org.hibernate.dialect.MySQL5InnoDBDialect;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.orm.jpa.vendor.Database;
import org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter;
import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;

import com.example.spring.services.CustomerService;

@EnableTransactionManagement
@Configuration
@ComponentScan( basePackageClasses = CustomerService.class )
public class AppConfig {
    @Bean
    public PersistenceExceptionTranslationPostProcessor exceptionTranslationPostProcessor() {
        final PersistenceExceptionTranslationPostProcessor processor = new PersistenceExceptionTranslationPostProcessor();
        processor.setRepositoryAnnotationType( Service.class );
        return processor;
    }

    @Bean
    public HibernateJpaVendorAdapter hibernateJpaVendorAdapter() {
        final HibernateJpaVendorAdapter adapter = new HibernateJpaVendorAdapter();

        adapter.setDatabase( Database.MYSQL );
        adapter.setShowSql( false );

        return adapter;
    }

    @Bean
    public LocalContainerEntityManagerFactoryBean entityManager() throws Throwable {
        final LocalContainerEntityManagerFactoryBean entityManager = new LocalContainerEntityManagerFactoryBean();
     
        entityManager.setPersistenceUnitName( "customers" );
        entityManager.setDataSource( dataSource() );
        entityManager.setJpaVendorAdapter( hibernateJpaVendorAdapter() );

        final Properties properties = new Properties();
        properties.setProperty("hibernate.dialect", MySQL5InnoDBDialect.class.getName());
        properties.setProperty("hibernate.hbm2ddl.auto", "create-drop" );
        entityManager.setJpaProperties( properties );

        return entityManager;
    }

    @Bean
    public DataSource dataSource() {
        final DriverManagerDataSource dataSource = new DriverManagerDataSource();

        dataSource.setDriverClassName( com.mysql.jdbc.Driver.class.getName() );
        dataSource.setUrl( "jdbc:mysql://localhost/customers?enableQueryTimeouts=true" );
        dataSource.setUsername( "root" );
        dataSource.setPassword( "" );

        return dataSource;
    }

    @Bean
    public PlatformTransactionManager transactionManager() throws Throwable {
        return new JpaTransactionManager( this.entityManager().getObject() );
    }
}

Now let's add a simple JUnit test case to verify our Spring application actually works as expected. Before doing that, the database customers should be created:

> mysql -u root
mysql> create database customers;
Query OK, 1 row affected (0.00 sec)

And here is a CustomerServiceTestCase which for now has single test to create the customer and verify it's actually has been created.

package com.example.spring;

import static org.hamcrest.CoreMatchers.notNullValue;
import static org.junit.Assert.assertThat;

import javax.inject.Inject;

import org.junit.After;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.support.AnnotationConfigContextLoader;

import com.example.spring.config.AppConfig;
import com.example.spring.domain.Customer;
import com.example.spring.services.CustomerService;

@RunWith( SpringJUnit4ClassRunner.class )
@ContextConfiguration(loader = AnnotationConfigContextLoader.class, classes = { AppConfig.class } )
public class CustomerServiceTestCase {
    @Inject private CustomerService customerService; 
 
    @After
    public void tearDown() {
        customerService.deleteAll();
    }

    @Test
    public void testCreateCustomerAndVerifyItHasBeenCreated() throws Exception {
        Customer customer = customerService.create( "Customer A" );
        assertThat( customerService.find( customer.getId() ), notNullValue() );
    }
}

That looks simple and straightforward. Now, let's think about scenario when customer creation succeeded but find fails because of query timeout. To do that, we need a help from Byteman.

In short, Byteman is bytecode manipulation framework. It's a Java agent implementation which runs with JVM (or attaches to it) and modifies running application bytecode as such changing its behavior. Byteman has a very good documentation and own rich set of rule definitions to perform mostly everything developer can come up with. Also, it has pretty good integration with JUnit framework. On that subject, Byteman tests are supposed to be run with @RunWith( BMUnitRunner.class ), but we already using @RunWith( SpringJUnit4ClassRunner.class ) and JUnit doesn't allow multiple test runners to be specified. Looks like a problem unless you are familiar with JUnit @Rule mechanics. It turns out that converting BMUnitRunner to JUnit rule is quite easy task:

package com.example.spring;

import org.jboss.byteman.contrib.bmunit.BMUnitRunner;
import org.junit.rules.MethodRule;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.Statement;

public class BytemanRule extends BMUnitRunner implements MethodRule {
    public static BytemanRule create( Class< ? > klass ) {
        try {
            return new BytemanRule( klass ); 
        } catch( InitializationError ex ) { 
            throw new RuntimeException( ex ); 
        }
    }
 
    private BytemanRule( Class klass ) throws InitializationError {
        super( klass );
    }
 
    @Override
    public Statement apply( final Statement statement, final FrameworkMethod method, final Object target ) {
        Statement result = addMethodMultiRuleLoader( statement, method ); 
  
        if( result == statement ) {
            result = addMethodSingleRuleLoader( statement, method );
        }

        return result;
    }
}

And JUnit @Rule injection is as simple as that:

@Rule public BytemanRule byteman = BytemanRule.create( CustomerServiceTestCase.class );

Easy, right? The scenario we mentioned before could be rephrased a bit: when JDBC statement to select from 'customers' table is executed, we should fail with timeout exception. Here is how it looks like as JUnit test case with additional Byteman annotations:

    @Test( expected = DataAccessException.class )
    @BMRule(
        name = "introduce timeout while accessing MySQL database",
        targetClass = "com.mysql.jdbc.PreparedStatement",
        targetMethod = "executeQuery",
        targetLocation = "AT ENTRY",
        condition = "$0.originalSql.startsWith( \"select\" ) && !flagged( \"timeout\" )",
        action = "flag( \"timeout\" ); throw new com.mysql.jdbc.exceptions.MySQLTimeoutException( \"Statement timed out (simulated)\" )"
    )
    public void testCreateCustomerWhileDatabaseIsTimingOut()  {
        Customer customer = customerService.create( "Customer A" );
        customerService.find( customer.getId() );
    }

We could read it like this: "When someone calls executeQuery method of PreparedStatement class and query starts with 'SELECT' than MySQLTimeoutException will be thrown, and it should happen only once (controlled by timeout flag)". Running this test case prints stacktrace in a console and expects DataAccessException to be thrown:

com.mysql.jdbc.exceptions.MySQLTimeoutException: Statement timed out (simulated)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_21]
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_21]
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_21]
 at java.lang.reflect.Constructor.newInstance(Constructor.java:525) ~[na:1.7.0_21]
 at org.jboss.byteman.rule.expression.ThrowExpression.interpret(ThrowExpression.java:231) ~[na:na]
 at org.jboss.byteman.rule.Action.interpret(Action.java:144) ~[na:na]
 at org.jboss.byteman.rule.helper.InterpretedHelper.fire(InterpretedHelper.java:169) ~[na:na]
 at org.jboss.byteman.rule.helper.InterpretedHelper.execute0(InterpretedHelper.java:137) ~[na:na]
 at org.jboss.byteman.rule.helper.InterpretedHelper.execute(InterpretedHelper.java:100) ~[na:na]
 at org.jboss.byteman.rule.Rule.execute(Rule.java:682) ~[na:na]
 at org.jboss.byteman.rule.Rule.execute(Rule.java:651) ~[na:na]
 at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java) ~[mysql-connector-java-5.1.24.jar:na]
 at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:56) ~[hibernate-core-4.2.0.Final.jar:4.2.0.Final]
 at org.hibernate.loader.Loader.getResultSet(Loader.java:2031) [hibernate-core-4.2.0.Final.jar:4.2.0.Final]

Looks good, what about another scenario: customer creation succeeded but find fails because the database went down? This one is a bit more complicated but easy to do anyway, let's take a look:

@Test( expected = CannotCreateTransactionException.class )
@BMRules(
    rules = {
        @BMRule(
            name="create countDown for AbstractPlainSocketImpl",
            targetClass = "java.net.AbstractPlainSocketImpl",
            targetMethod = "getOutputStream",
            condition = "$0.port==3306",
            action = "createCountDown( \"connection\", 1 )"
        ),
        @BMRule(
            name = "throw IOException when trying to execute 2nd query to MySQL",
            targetClass = "java.net.AbstractPlainSocketImpl",
            targetMethod = "getOutputStream",
            condition = "$0.port==3306 && countDown( \"connection\" )",
            action = "throw new java.io.IOException( \"Connection refused (simulated)\" )"
        )
    }
)
public void testCreateCustomerAndTryToFindItWhenDatabaseIsDown() {
    Customer customer = customerService.create( "Customer A" );
    customerService.find( customer.getId() );
}

Let me explain what's going on here. We would like to sit on socket level and actually control the communication as close to network as we can, not on JDBC driver level. That's why we instrumenting AbstractPlainSocketImpl. We also know that MySQL's default port is 3306 so we are instrumenting only sockets opened on this port. Another fact, we know that first created socket corresponds to customer creation and we should let it go through. But second one corresponds to find and must fail. The createCountDown named "connection" serves this purposes: the first call goes through (latch doesn't count to zero yet) but second call triggers MySQLTimeoutException exception. Running this test case prints stacktrace in a console and expects CannotCreateTransactionException to be thrown:

Caused by: java.io.IOException: Connection refused (simulated)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_21]
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_21]
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_21]
 at java.lang.reflect.Constructor.newInstance(Constructor.java:525) ~[na:1.7.0_21]
 at org.jboss.byteman.rule.expression.ThrowExpression.interpret(ThrowExpression.java:231) ~[na:na]
 at org.jboss.byteman.rule.Action.interpret(Action.java:144) ~[na:na]
 at org.jboss.byteman.rule.helper.InterpretedHelper.fire(InterpretedHelper.java:169) ~[na:na]
 at org.jboss.byteman.rule.helper.InterpretedHelper.execute0(InterpretedHelper.java:137) ~[na:na]
 at org.jboss.byteman.rule.helper.InterpretedHelper.execute(InterpretedHelper.java:100) ~[na:na]
 at org.jboss.byteman.rule.Rule.execute(Rule.java:682) ~[na:na]
 at org.jboss.byteman.rule.Rule.execute(Rule.java:651) ~[na:na]
 at java.net.AbstractPlainSocketImpl.getOutputStream(AbstractPlainSocketImpl.java) ~[na:1.7.0_21]
 at java.net.PlainSocketImpl.getOutputStream(PlainSocketImpl.java:214) ~[na:1.7.0_21]
 at java.net.Socket$3.run(Socket.java:915) ~[na:1.7.0_21]
 at java.net.Socket$3.run(Socket.java:913) ~[na:1.7.0_21]
 at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_21]
 at java.net.Socket.getOutputStream(Socket.java:912) ~[na:1.7.0_21]
 at com.mysql.jdbc.MysqlIO.(MysqlIO.java:330) ~[mysql-connector-java-5.1.24.jar:na]

Great! The possibilities Byteman provides for different fault simulations are enormous. Carefully adding test suites to verify how application reacts to erroneous conditions greatly improves application robustness and resiliency to failures. Bunch of thanks to Byteman guys!

Please find the complete project on GitHub.

Watch your Spring webapp: Hibernate and log4j over JMX

I have been actively using Java Management Extensions (JMX), particularly within web applications, in order to monitor application internals and sometimes tune some parameters at runtime. There are few very useful tools supplied as part of JDK, JConsole and JVisualVM, which allow to connect to your application via JMX and manipulate with exposed managed beans.

I am going to leave apart basic JMX concepts and concentrate on interesting use cases:
- exposing log4j over JMX (which allows to change LOG LEVEL at runtime)
- exposing Hibernate statistics over JMX

In order to simplify a bit all routines with exposing managed beans I will use Spring Framework which has awesome JMX support driven by annotations. Let's have our first Spring context snippet: exposing log4j over JMX.

That's how it look like inside JVisualVM with VisualVM-MBeans plugin installed (please notice that root's logger LOG LEVEL (priority) could be changed from WARN to any, let say, DEBUG, at runtime and have effect immediately):

Let's add Hibernate to JMX view! In order to do that I will create very simple Hibernate configuration using Spring context XML file (I will repeat configuration for JMX-related beans but it's exactly the same as in previous example):

    

    
    
        
        
        
    
 
    
        
    
 
    
        
    
 
    
    

    
    
        
  
        
            
        
    
        
            
                org.hibernate.dialect.HSQLDialect
                true
            
        
    
 
    
  
    
        
      

       
    
        
    

And now we see this picture (please notice very important Hibernate property in order to see some real data here hibernate.generate_statistics = true):

Cool, simple and very useful, isn't it? :)

Force your beans validation using JSR 303 and Hibernate Validator 4.1.0

In my list of most loved features JSR 303 is somewhere among top ones. Not only because it's extremely useful, but also because it has simple design and extensibility. Hibernate project actively uses validation techniques for a long time as part of Hibernate Validator subproject, recently with JSR 303 implementation support.

Today I would like to share some basic uses cases of JSR 303 annotations and programmatic validation support. Let us start with first one.

1. Assume you are expecting Person object to be passed to you business logic layer, just typical Java bean with few properties.

package org.example;

public class Person {
    private String firstName;
    private String lastName;
    
    public void setFirstName( String firstName ) {
        this.firstName = firstName;
    }
 
    public String getFirstName() {
        return firstName;
    }

    public void setLastName( String lastName ) {
        this.lastName = lastName;
    }

    public String getLastName() {
        return lastName;
    }   
}

Further, let us assume that according to business logic firstName and lastName properties are required. There are many ways to enforce such kind of constraints but we will use JSR 303 bean validation and Hibernate Validator.

package org.example;

import org.hibernate.validator.constraints.NotBlank;

public class Person {
    @NotBlank private String firstName;
    @NotBlank private String lastName;
    
    ...  
}

The question arises who will actually perform validation? The most straightforward solution is to do it using utility class.

package org.example;

import java.util.HashSet;
import java.util.Set;

import javax.validation.ConstraintViolation;
import javax.validation.ConstraintViolationException;
import javax.validation.Validation;
import javax.validation.Validator;
import javax.validation.ValidatorFactory;
import javax.validation.groups.Default;

public class ValidatorUtil {
    private final ValidatorFactory factory;
 
    public ValidatorUtil() {
        factory = Validation.buildDefaultValidatorFactory();
    }
  
    public< T > void validate( final T instance ) {  
        final Validator validator = factory.getValidator();
  
        final Set< ConstraintViolation< T > > violations = validator
            .validate( instance, Default.class );

        if( !violations.isEmpty() ) {
            final Set< ConstraintViolation< ? > > constraints = 
                new HashSet< ConstraintViolation< ? > >( violations.size() );

            for ( final ConstraintViolation< ? > violation: violations ) {
                constraints.add( violation );
            }
   
            throw new ConstraintViolationException( constraints );
        }
    }
}

Having such a class validation is easy:

final Person person = new Person();
new ValidatorUtil().validate( person );

ConstraintViolationException exception will be thrown in case of any validation errors. Let us move on to more complicated examples.

2. Assume you are expecting Department object to be passed to you business logic containing at least one (valid!) person. Let me skip the details and present the final solution here.

package org.example;

import java.util.Collection;

import javax.validation.Valid;

import org.hibernate.validator.constraints.NotEmpty;

public class Department {
    @NotEmpty @Valid private Collection< Person > persons;

    public void setPersons( Collection< Person > persons ) {
        this.persons = persons;
    }

    public Collection< Person > getPersons() {
        return persons;
    }
}

The validation above ensures that Department contains at least one Person instance and each Person's instance is itself valid.

There are just a lot of other use case out there which Hibernate Validator is able to handle out of the box: @Min, @Man, @Email, @Url, @Range, @Length, @Pattern, ... This post is just a starting point ...

Distributed Hibernate Search with Apache Tomcat 6, ActiveMQ and Spring

Today I would like to share my experience with configuring Hibernate Search in master/slave(s) deployment using Apache ActiveMQ, Spring and running all this stuff inside Apache Tomcat 6 container.

How it works:
- Hibernate Search supports distributed configuration using JMS back-end and master / slave(s) index
- master server exposes index over network share (NFS,...)
- slave(s) on regular base replicate this master copy to own local copies

Used version:
- Apache Tomcat 6.0.20
- Hibernate Search 3.1.1 GA
- Apache ActiveMQ 5.3.0
- Spring 2.5.6
- XBean-Spring 3.6

Master Index Configuration
Master configuration is a little bit complicated. Here are configuration-specific properties:

${local.index.dir} - directory to store master index
${master.index.dir} - directory to copy master index to, it's shared network location for replication with slave(s)

First of all, for simplification, let's run ActiveMQ broker on the same host. For this purpose we could use simple embedded broker configuration placed into WEB-INF/activemq.xml:

<beans
 xmlns="http://www.springframework.org/schema/beans"
 xmlns:amq="http://activemq.apache.org/schema/core"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.springframework.org/schema/beans
 http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
 http://activemq.apache.org/schema/core
 http://activemq.apache.org/schema/core/activemq-core.xsd">

 <amq:broker brokerName="HibernateSearchBroker">
     <amq:managementContext>
         <amq:managementContext createConnector="false"/>
     </amq:managementContext>

     <amq:transportConnectors>
         
     </amq:transportConnectors>
  </amq:broker>

  <amq:queue name="queue/hibernatesearch" physicalName="hibernateSearchQueue" />
</beans>

Then we need to configure JNDI resources (JMS Connection Factory and Queue) through web application META-INF/context.xml file (Tomcat-specific):

...
<!-- ActiveMQ ConnectionFactory -->
<Resource
name="jms/ConnectionFactory"
auth="Container"
type="org.apache.activemq.ActiveMQConnectionFactory"
description="JMS Connection Factory"
factory="org.apache.activemq.jndi.JNDIReferenceFactory"
brokerURL="tcp://0.0.0.0:61616?trace=true"
brokerName="HibernateSearchBroker" />

<!-- ActiveMQ HibernateSearch queue -->
<Resource
name="queue/hibernatesearch"
auth="Container" type="org.apache.activemq.command.ActiveMQQueue"
description="Hibernate search queue"
factory="org.apache.activemq.jndi.JNDIReferenceFactory"
physicalName="hibernateSearchQueue" />
...

Next step is configuration of the Hibernate Search itself through Hibernate configuration file (hibernate.cfg.xml):

<property name="hibernate.search.default.directory_provider">org.hibernate.search.store.FSMasterDirectoryProvider</property>
<property name="hibernate.search.default.indexBase">${local.index.dir}</property>
<property name="hibernate.search.default.sourceBase">${master.index.dir}</property>
<property name="hibernate.search.default.refresh">60</property>

One important difference between master and slave codebase: master implementation must include subclass of AbstractJMSHibernateSearchController as message listener. For example,

import javax.jms.MessageListener;

import org.hibernate.Session;
import org.hibernate.search.backend.impl.jms.AbstractJMSHibernateSearchController;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

@Service
public class JMSHibernateSearchController
 extends AbstractJMSHibernateSearchController
     implements MessageListener {

 @Override
 protected void cleanSessionIfNeeded(Session session) {
    // clean session here ...     
 }

 @Override
 protected Session getSession() {
    // return new session here ...     
 }
}

Finally, let's wrap it up inside Spring configuration file applicationContext.xml:

<bean id="broker" class="org.apache.activemq.xbean.BrokerFactoryBean">
 <property name="config" value="WEB-INF/activemq.xml" />
 <property name="start" value="true" />
</bean>  

<bean name="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
 <property name="jndiName" value="java:comp/env/jms/ConnectionFactory" />
</bean>

<bean name="jmsHibernateSearchQueue" class="org.springframework.jndi.JndiObjectFactoryBean">
 <property name="jndiName" value="java:comp/env/queue/hibernatesearch" />

<bean id="hibernateSearchController" class="<your implementation of AbstractJMSHibernateSearchController>" />

<bean id="jmsContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer" depends-on="broker">
 <property name="connectionFactory" ref="jmsConnectionFactory"/>
 <property name="destination" ref="jmsHibernateSearchQueue"/>
 <property name="messageListener" ref="hibernateSearchController" />
</bean>

With those configurations in place Hibernate Search master is ready to run.


Slave Index Configuration
Slave(s) configuration is much simple. Here are configuration-specific properties:

${server} - server which runs ActiveMQ broker
${local.index.dir} - directory to store local index (master copy)
${master.index.share} - mounted network share with master index

First of all, we need to configure JNDI resources (JMS Connection Factory and Queue) through web application META-INF/context.xml file (Tomcat-specific):

...
<!-- ActiveMQ ConnectionFactory -->
<Resource
name="jms/ConnectionFactory"
auth="Container"
type="org.apache.activemq.ActiveMQConnectionFactory"
description="JMS Connection Factory"
factory="org.apache.activemq.jndi.JNDIReferenceFactory"
brokerURL="tcp://${server}:61616?trace=true"
brokerName="HibernateSearchBroker" />

<!-- ActiveMQ HibernateSearch queue -->
<Resource
name="queue/hibernatesearch"
auth="Container" type="org.apache.activemq.command.ActiveMQQueue"
description="Hibernate search queue"
factory="org.apache.activemq.jndi.JNDIReferenceFactory"
physicalName="hibernateSearchQueue" />
...

Then we have to configure Hibernate Search itself through Hibernate configuration file (hibernate.cfg.xml):

<property name="hibernate.search.default.directory_provider">org.hibernate.search.store.FSSlaveDirectoryProvider</property>
<property name="hibernate.search.default.indexBase">${local.index.dir}</property>
<property name="hibernate.search.default.sourceBase">${master.index.share}</property>
<property name="hibernate.search.default.refresh">60</property>
<property name="hibernate.search.worker.backend">jms</property>
<property name="hibernate.search.worker.jms.connection_factory">java:comp/env/jms/ConnectionFactory</property>
<property name="hibernate.search.worker.jms.queue">java:comp/env/queue/hibernatesearch</property>
<property name="hibernate.search.worker.jndi.java.naming.factory.initial">org.apache.activemq.jndi.ActiveMQInitialContextFactory</property>

And ... that's it!

Few additional words about testing all this stuff with JUnit. Only problem is JNDI which could be mocked up with Spring JNDI templates. For example:

<bean name="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="java:comp/env/jms/ConnectionFactory" />
    <property name="jndiTemplate">
        <bean class="org.springframework.mock.jndi.ExpectedLookupTemplate">
            <constructor-arg index="0" value="java:comp/env/jms/ConnectionFactory" />
            <constructor-arg index="1">
                <bean class="org.apache.activemq.ActiveMQConnectionFactory">
                    <property name="brokerURL">
                        <value>tcp://0.0.0.0:61616</value>
                    </property>
                </bean>
            </constructor-arg>
        </bean>
    </property>
</bean>

<bean name="jmsHibernateSearchQueue" class="org.springframework.jndi.JndiObjectFactoryBean">
    <property name="jndiName" value="java:comp/env/queue/hibernatesearch" />
    <property name="jndiTemplate">
       <bean class="org.springframework.mock.jndi.ExpectedLookupTemplate">
           <constructor-arg index="0" value="java:comp/env/queue/hibernatesearch" />
           <constructor-arg index="1">
               <bean id="jmsHibernateSearchQueue"   class="org.apache.activemq.command.ActiveMQQueue">
                   <constructor-arg value="queue/hibernateSearchQueue"/>
               </bean>               
           </constructor-arg>
       </bean>
    </property>
</bean>
...

Hibernate Search + Apache Lucene (continued)

Let's continue very useful and interesting topic about Hibernate Search and Apache Lucene. In my last post we discovered new annotations which allow persistent entities to be "searchable". In this post I am going to show how to query such entities with various search criteria.

Once entities are annotated, they could be queried with regular Apache Lucene query. Simple flow looks like this:

Session session = sessionFactory.getNewSession();
FullTextSession fullTextSession = Search.getFullTextSession( session );

try {  
fullTextSession.beginTransaction();

// query - regular Lucene query, f.e. "author=King"
// entities - list of entity classes, like Entity1.class, Entity2.class, ...

FullTextQuery q = fullTextSession
 .createFullTextQuery( query, entities );

// Get a collection of matched entities
Collection< Object > results = q.list();

fullTextSession.getTransaction().commit();
} catch (Exception ex ) {
fullTextSession.getTransaction().rollback();
throw ex;
} finally {
session.close();
}

That's how Hibernate Search makes it simple. The only difference from Hibernate's model is FullTextSession usage. Here are few simple details about how Hibernate Search works on top of Apache Lucene:

• for each entity class Apache Lucene creates own index


• Hibernate Search adds a few additional properties (like persistent entity Id and Type) to associate Apache Lucene document and Hibernate entity


• for each query Hibernate Search uses query rewrite to ensure only indexed properties of a particular entity are present in query


• entity's (re)indexing occurs when database transaction commits (but Hibernate Search allows you manually (re)index entity)

Hibernate Search also allows you to use very powerful features like filters (with parameters) and sorting. Filters are extremely useful in case of security rules or search with search results implementation.

...
FullTextQuery q = fullTextSession
 .createFullTextQuery( query, entities );
     .setSort( new Sort(
             new SortField[] {
                 new SortField( "name", false )
             }
         )
     );

// Enable filter with name 'filterName' and parameter 'value' set to value
Object value = ...;
q.enableFullTextFilter( "filterName" )
 .setParameter( "value", value );

// Get a collection of matched entities
Collection< Object > results = q.list();
...

It's very simple to integrate and use Hibernate Search into existing applications. It most cases I found integration seamless.

There are a bunch of interesting issues that might occur:

• how to get all indexed properties for entity?


• how to reindex whole existing database?


• how to deploy master/slave(s) configuration?

For those I am going to create dedicated post because an implementation is a little bit tricky (but in general not complicated at all).

Hibernate Search + Apache Lucene

It worthwhile to say that every Java developer knows about Apache Lucene project. No doubts, the best open source search engine! But today it's not about Apache Lucene, it's about Hibernate Search project, just another excellent library from Hibernate team.

So, what if you need some search capabilities in your application? For sure, it depends on requirements, but let's consider Java application which uses database as back-end and Hibernate as ORM framework. I would suggest a few ways to do search in this case:
1) Implement own search framework based on full text search capabilities of your database server (vendor-locked solution).
2) Implement own search framework with SQL/HQL queries. Works well for very simple scenarios. but is not enough in most cases.
3) Use Hibernate Search (and Apache Lucene). As powerful as Apache Lucene and as flexible as Hibernate framework. The most preferable solution.

From developer point of view, integration of the Hibernate Search is just a matter of additional annotations together with a few configuration lines in hibernate.cfg.xml. Let's start with last one.

<property name="hibernate.search.default.directory_provider">
org.hibernate.search.store.RAMDirectoryProvider
</property>

That tells Hibernate Search to store all indexes in RAM (don't do that on production system!). And ... that's it!

Now let's look on typical persistent entity and ... how to make it searchable!

@Entity
@Indexed
public class User {
   private Long id;
   private String firstName;
   private String lastName;

   @Id
   @GeneratedValue( strategy = GenerationType.AUTO )
   @Column( name = "UserID" )
   @DocumentId
   public Long getId() {
       return id;
   }
   public void setId(Long id) {
       this.id = id;
   }

   @Basic
   @Column( name = "FirstName", length = 50 )
   @Field( index = Index.UN_TOKENIZED )
   public String getFirstName() {
       return firstName;
   }
  
   public void setFirstName( String firstName ) {
       this.firstName = firstName;
   }
 
   @Basic
   @Column( name = "LastName", length = 50 )
   @Field( index = Index.UN_TOKENIZED )
   public String getLastName() {
       return lastName;
   }
 
   public void setLastName( String lastName ) {
       this.lastName = lastName;
   }
}

A few annotations in bold (@Indexed, @Field, @DocumentId) make all the magic. Hibernate Search provides additional listeners which take care on entity indexing (Lucene documents creation/deletion). In next post I will explain how to search stored entities by specific criteria. For now, I would mention a great book Hibernate Search in Action which perfectly explains all details.

Testing persistence layer

As a supporter of TDD I prefer to test as much as it's necessary to be sure the code works properly (but I'm not a paranoid in that). With respect to this, testing persistence layer takes a very important place in my TDD practices.

By "testing persistence layer" I basically mean two things:
- testing that ORM mapping (Hibernate) is valid
- testing that application works well with rich test data sets

Using persistence layer tests allows to verify corner cases as well as application behavior in case of different database faults. Even more, development with testing in mind allows to design low coupled and well layered application architecture.

To start with, let's reuse two classes Employer and Employee from previous post and develop simple test cases for them. Thanks to Hibernate, we're completely decoupled from underlying database so we can create any suitable testing configuration. And thanks to HSQLDB project we're able to easily create in-memory database and relegate standalone database server deployment. For sure, we'll create test cases based on JUnit framework.

First, let's create abstract test case encapsulating all configuration in it. Basically, we have to configure Hibernate, add annotated classes, and create session factory.

import org.hibernate.Session;
import org.hibernate.dialect.HSQLDialect;
import org.hsqldb.jdbcDriver;
import org.hibernate.cfg.AnnotationConfiguration;
import org.hibernate.cfg.Environment;

public abstract class AbstractPersistentTestCase {
    private AnnotationConfiguration configuration;
    private SessionFactory sessionFactory;

    @Before
    protected void setUp() throws Exception {
        configuration = new AnnotationConfiguration();
        configuration.setProperty( Environment.DIALECT, 
            HSQLDialect.class.getName() );
        configuration.setProperty( Environment.DRIVER, 
            jdbcDriver.class.getName() );
        configuration.setProperty( Environment.URL, 
            "jdbc:hsqldb:mem:testdb" );
        configuration.setProperty( Environment.CURRENT_SESSION_CONTEXT_CLASS,
            "org.hibernate.context.ThreadLocalSessionContext" );
        configuration.setProperty( Environment.HBM2DDL_AUTO, 
            "create-drop" );
        configuration.setProperty( Environment.STATEMENT_BATCH_SIZE, 
            "0" );
        configuration.setProperty( Environment.SHOW_SQL, 
            "false" );
        configuration.setProperty( Environment.FORMAT_SQL, 
            "true" );
        
        configuration.addAnnotatedClass( Employee.class )
            .addAnnotatedClass( Employer.class );

        sessionFactory = configuration.buildSessionFactory();
    }

    @After
    public void tearDown() {
        SchemaExport schemaExport = new SchemaExport( configuration );
        schemaExport.drop( true, true );
    }

    protected Session getSession() {
        return sessionFactory.getCurrentSession(  );
    }
}

Then, let's develop a simple test case for persisting Employer and Employee classes. There's one important note here. Those test methods should be executed within transaction boundary (for example, with help of AspectJ). Otherwise, the HibernateException comes up.

public class PersistentTestCase extends AbstractPersistentTestCase {
    @Test
    public void testSaveEmployer() {
        Employer employer = new Employer();
        getSession().save( employer );

        Assert.assertNotNull( employer.getId() );
    }

    @Test
    public void testSaveEmployee() {
        Employer employer = new Employer();
        getSession().save( employer );

        Employee employee = new Employee();
        employee.setEmployer( employer );
        getSession().save( employee );

        Assert.assertNotNull( employee.getId() );
    }
}

Here I covered very simple scenario. Just to give the idea. Next step will be preparing test data sets with help of DbUnit. It allows to prepare data in XML format (among others) and upload this file into database. Let's prepare quite simple XML file 'dataset.xml':

    <?xml version="1.0" encoding="UTF-8"?>
    <dataset>
        <employer id="1" name="IBM" email="ibm@ibm.com" />
        <employer id="2" name="Microsoft" email="microsoft@microsoft.com" />
    </dataset>

And that's it! DbUnit will automatically map XML elements to tables and XML attributes to columns! Awesome! Let's develop the test case using DbUnit and newly created data set.

public class DatasetTestCase extends AbstractPersistentTestCase {
    @Before
    protected void setUp() throws Exception {
        super.setUp();
        uploadTestDataset();
    }

    private void uploadTestDataset() throws Exception
    {
        final Session session = getSession();
        Transaction t = session.beginTransaction(  );

        try {
            IDataSet dataSet = new FlatXmlDataSet( 
                new FileInputStream( 
                    new File( 
                        getClass().getResource( "/dataset.xml" ).toURI() 
                    )
                )  
            );

            IDatabaseConnection connection = 
                new DatabaseConnection( session.connection() );            
            DatabaseOperation.CLEAN_INSERT.execute( connection, 
                dataSet );

            t.commit(  );
        } finally {
            if ( t.isActive(  ) ) {
                t.rollback(  );
            }
        }
    }

    @Test
    public void testLoadEmployer() {
        Employer employer = ( Employer )getSession().load( 
            Employer.class, new Integer( 1 ) );

        Assert.assertEquals( "IBM", employer.getName() );
        Assert.assertEquals( "ibm@ibm.com", employer.getEmail() );
    }
}

This scenario is very simple as well. But it's just a foundation ... Developing comprehensive datasets and testing application business logic against them is great step in achieving high product quality. Those techniques allow to fully cover very complex flows with tests and detect errors early.

The only problem with that is ... maintenance. It's particularly true for projects in active development. Changes in business logic lead to test failures (in most cases). Supporting huge tests code base can me nightmare. So ... I'm always looking for balanced solution.

Using Criteria API in Hibernate

Hibernate is awesome! I'm pretty excited about Hibernate and I'll continue to use it as primary ORM solution for my future projects. Today I'd like to share typical usage scenarios of Hibernate Criteria API which provides extremely powerful capabilities to build strong typed queries.

Let's start with few simple classes (Employer and Employee) mapped to database tables. We'll use those in future examples.

@Entity
@Table( name = "employees" )
public class Employee {
   @Id
   @GeneratedValue( strategy = GenerationType.AUTO )
   @Column( name = "id" )
   private Integer id;

   @Column( name = "name" )
   private String name;

   @Column( name = "email" )
   private String email;

   @ManyToOne
   private Employer employer;

   public void setId( Integer id ) {
       this.id = id;
   }

   public Integer getId() {
       return this.id;
   }

   public void setName( String name ) {
       this.name = name;
   }

   public String getName() {
       return this.name;
   }

   public void setEmail( String email ) {
       this.email = email;
   }

   public String getEmail() {
       return this.email;
   }
};

@Entity
@Table( name = "employers" )
public class Employer {
   @Id
   @GeneratedValue( strategy = GenerationType.AUTO )
   @Column( name = "id" )
   private Integer id;

   @OneToMany( cascade = CascadeType.ALL, mappedBy = "employer" )
   private Set<Employee> employees = new HashSet<Employee>();

   public void setId( Integer id ) {
       this.id = id;
   }

   public Integer getId() {
       return this.id;
   }

   public Set<Employee> getEmployees() {
       return this.employees;
   }

   public void setEmployees( Set<Employee> employees ) {
       this.employees = employees;
   }
};

Let say we have to find Employees and Employers using different search criteria. For all examples I assume that helper method getCurrentSession() returns Hibernate's session and each code fragment is running within transaction.

1) Find all Employees with name "Tom" (the easiest one).

// Get current session
final Session session = getCurrentSession();

// Create Criteria for Employee class
final Criteria criteria = session.createCriteria( Employee.class );
// Add name = "Tom" restriction
criteria.add( Restrictions.eq( "name", "Tom" );

// Run query and get the results
List< Employee gt; employees = criteria.list();
// Do something here
...

2) Find all Employers which don't have any Employees.

// Get current session
final Session session = getCurrentSession();

// Create Criteria for Employer class
final Criteria criteria = session.createCriteria( Employer.class );
// Add empty Employees restriction
criteria.add( Restrictions.isEmpty( "employees" );

// Run query and get the results
List< Employer gt; employers = criteria.list();
// Do something here
...

3) Find all Employers which have Employees with name "Tom".

// Get current session
final Session session = getCurrentSession();

// Create Criteria for Employer class
final Criteria criteria = session.createCriteria( Employer.class );
// Create alias for Employees collection
criteria.createAlias( "employees", "employees" );
// Add name = "Tom" restriction
criteria.add( Restrictions.eq( "employees.name", "Tom" );

// Run query and get the results
List< Employer > employers = criteria.list();
// Do something here
...

4) For this example let me introduce another table "black_list" and BlackListEntry entity mapping.

@Entity
@Table( "black_list" )
public class BlackListEntry {
   @Id
   @GeneratedValue( strategy = GenerationType.TABLE )
   @Column( name = "email" )
   private String email;

   public void setEmail( String email ) {
       this.email = email;
   }

   public String getEmail() {
       return this.email;
   }
};

The purpose of this entity is pretty simple: a black list of e-mail addresses. Those should be exclude from any mailing lists. Let's find all Employees whose e-mails are in that black list.

// Get current session
final Session session = getCurrentSession();

// Create detached Criteria
DetachedCriteria detachedCriteria =
    DetachedCriteria.forClass( BlackListEntry.class );
// Create projection to select 'email' only
detachedCriteria.setProjection( Projections.property( "email" ) );

// Create Criteria for Employee class
final Criteria criteria = session.createCriteria( Employee.class );
// Add subquery restriction (all emails in black list)
criteria.add( Subqueries.propertyIn( "email", detachedCriteria ) );

// Run query and get the results
List< Employee > employees = criteria.list();
// Do something here
...

That's it. :-)

Integration testing: building our own test sandbox (practice)

Previous post prepared some theoretical background for building our own test sandbox. In this post we'll flesh out the theory with practice. So, let's follow startup steps and confirm each one with code snippets.

1. Configure HSQLDB data source and JNDI

import org.enhydra.jdbc.standard.StandardDataSource;
...

StandardDataSource dataSource = new StandardDataSource();
dataSource.setDriverName( "org.hsqldb.jdbcDriver" );
dataSource.setUrl( "jdbc:hsqldb:mem:testdb" );
dataSource.setUser( "sa" );

In the snippet above I've used very useful XAPool project (to manipulate with JDBC data sources).

import javax.naming.Context;
import javax.naming.InitialContext;
import org.mortbay.jetty.plus.naming.NamingEntry;
import org.mortbay.jetty.plus.naming.Resource;
...

Context context = new InitialContext();
Context compCtx = ( Context )context.lookup( "java:comp" );
compCtx.createSubcontext( "env" );

// Configure JNDI
NamingEntry.setScope( NamingEntry.SCOPE_GLOBAL) ;

// This actually registers the resource in JNDI
new Resource( "jdbc/testdb", dataSource ).bindToENC();

2. Configure Hibernate

In fact, there is a trick here. I'm using AspectJ to intercept in run-time (weaver) all calls to Hibernate's class method Configuration.configure(). There is an aspect's code.

import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.hibernate.cfg.Configuration;
@Aspect
public class HibernateAspect {
@Around("execution( * org.hibernate.cfg.Configuration.configure() throws * )")
public Object reconfigure( ProceedingJoinPoint joinPoint ) throws Throwable {
Object r = ((Configuration) joinPoint.getThis()).configure( <path to configuration file> );
return r;
}
}

3. Configure Web Services for Jetty

import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.ServletHolder;
import org.apache.axis2.transport.http.AxisServlet;
import org.mortbay.jetty.servlet.Context;
...

Server server = new Server(0);

ServletHolder axisServletholder = new ServletHolder( new AxisServlet() );
axisServletholder.setInitParameter( "axis2.xml.path", <path to axis2.xml> );
axisServletholder.setInitParameter( "axis2.repository.path", <path to WEB-INF folder> );

Context root = new Context( server, "/", Context.SESSIONS );
root.addServlet( axisServletholder, "/servlet/AxisServlet" );
root.addServlet( axisServletholder, "/services/*" );

4. Run Jetty

server.start();
int actualPort = server.getConnectors()[0].getLocalPort();

5. Export database schema (Hibernate)

import org.hibernate.tool.hbm2ddl.SchemaExport;
...

// Be sure, Hibernate uses JNDI data source name. In this test "jdbc/testdb".
// And database dialect is set to org.hibernate.dialect.HSQLDialect.
SchemaExport schemaExport = new SchemaExport( );
schemaExport.create( true, true );

6. Prepare test dataset (DbUnit)

import org.dbunit.database.IDatabaseConnection;
import org.dbunit.database.DatabaseConnection;
import org.dbunit.dataset.xml.FlatXmlDataSet;
import org.dbunit.dataset.IDataSet;
import org.dbunit.operation.DatabaseOperation;
import org.hibernate.Session;
import java.io.FileInputStream;
...

Session session = HibernateUtil.getCurrentSession();
IDataSet dataSet = new FlatXmlDataSet(new FileInputStream(<path to test dataset XML>));
IDatabaseConnection connection = new DatabaseConnection( session.connection() );
DatabaseOperation.CLEAN_INSERT.execute( connection, dataSet );

7. Run test(s) (Java or Groovy)

Let's summarize what we have at this point:
- Jetty is up and running
- Web Services are configured and deployed at: http://localhost:<actual port>/services/*
- Database schema is created and populated with test dataset

So we're ready to pull our web service methods. But for this purpose we need web service client. The first approach is to use raw SOAP requests. Second one is to generate web service (WSDL) stubs (Java). And last but not least is to use some scripting/dynamic language (like Groovy).

My choive was Groovy because:
- it's easy to consume web services with GroovyWS library
- it's easy to write test code
- it's excellent dynamic scripting language with seamless Java integration

Here is code sample:

import groovyx.net.ws.WSClient;

def proxy = new WSClient(
"http://localhost:<actual port>/services/TestService?wsdl", this.class.classLoader
);

def param = proxy.create( "testServiceNamespace.SomeType" )
param.someProperty =

def result = proxy.someTestMethod( param )
assert ( result = )

Looks really great and very promising. The only problem I've encountered that GroovyWS is started recently and is "green". Unfortunately, I've failed to test most web service methods with complex in/out parameters and return types (so I was reluctant to use WSDL generated stubs).

That's it. Our test sandbox is ready for use!