Topic of today's post stays a bit aside from day-to-day coding and development but nonetheless covers a very important subject: our application log files. Our apps do generate enormous amount of logs which if done right are extremely handy for problems troubleshooting.
It's not a big deal if you have a single application up and running, but nowadays apps, particularity webapps, run on hundreds of servers. With such a scale figuring out where is a problem becomes a challenge. Wouldn't it be nice to have some kind of a view which aggregates all logs from all our running applications into single dashboard so we could see a whole picture constructed from the pieces? Please welcome: Logstash, the logs aggregation framework.
Although it's not the only solution available, I found Logstash to be very easy to use and extremely simple to integrate. To start with, we don't even need to do anything on the application side, Logstash can do all the job for us.
Let me introduce the sample project: standalone Java application which has some multithreading activity going on. There is a logging to the file configured using great Logback library (SLF4J could be used as a seamless replacement). The POM file looks pretty simple:
And there is only one Java class called Starter which uses Executors services to do some work concurrently. For sure, each thread does some logging and from time to time there is an exception thrown.
package com.example.logstash;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class Starter {
private final static Logger log = LoggerFactory.getLogger( Starter.class );
public static void main( String[] args ) {
final ExecutorService executor = Executors.newCachedThreadPool();
final Collection< Future< Void > > futures = new ArrayList< Future< Void > >();
final Random random = new Random();
for( int i = 0; i < 10; ++i ) {
futures.add(
executor.submit(
new Callable< Void >() {
public Void call() throws Exception {
int sleep = Math.abs( random.nextInt( 10000 ) % 10000 );
log.warn( "Sleeping for " + sleep + "ms" );
Thread.sleep( sleep );
return null;
}
}
)
);
}
for( final Future< Void > future: futures ) {
try {
Void result = future.get( 3, TimeUnit.SECONDS );
log.info( "Result " + result );
} catch (InterruptedException | ExecutionException | TimeoutException ex ) {
log.error( ex.getMessage(), ex );
}
}
}
}
The idea is to demonstrate not only simple one-line logging events but famous Java stack traces. As every thread sleeps for random time interval, it causes TimeoutException to be thrown whenever the result of computation is being asked from the underlying future object and taken more than 3 seconds to return. The last part is Logback configuration (logback.xml):
And we are good to go! Please note that file path /tmp/application.log corresponds to c:\tmp\application.log on Windows. Running our application would fill log file with something like that:
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-1] com.example.logstash.Starter - Sleeping for 2506ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-4] com.example.logstash.Starter - Sleeping for 9147ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-9] com.example.logstash.Starter - Sleeping for 3124ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-3] com.example.logstash.Starter - Sleeping for 6239ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-5] com.example.logstash.Starter - Sleeping for 4534ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-10] com.example.logstash.Starter - Sleeping for 1167ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-7] com.example.logstash.Starter - Sleeping for 7228ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-6] com.example.logstash.Starter - Sleeping for 1587ms
[WARN] 2013-02-19 19:26:03.175 [pool-2-thread-8] com.example.logstash.Starter - Sleeping for 9457ms
[WARN] 2013-02-19 19:26:03.176 [pool-2-thread-2] com.example.logstash.Starter - Sleeping for 1584ms
[INFO] 2013-02-19 19:26:05.687 [main] com.example.logstash.Starter - Result null
[INFO] 2013-02-19 19:26:05.687 [main] com.example.logstash.Starter - Result null
[ERROR] 2013-02-19 19:26:08.695 [main] com.example.logstash.Starter - null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258) ~[na:1.7.0_13]
at java.util.concurrent.FutureTask.get(FutureTask.java:119) ~[na:1.7.0_13]
at com.example.logstash.Starter.main(Starter.java:43) ~[classes/:na]
[ERROR] 2013-02-19 19:26:11.696 [main] com.example.logstash.Starter - null
java.util.concurrent.TimeoutException: null
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258) ~[na:1.7.0_13]
at java.util.concurrent.FutureTask.get(FutureTask.java:119) ~[na:1.7.0_13]
at com.example.logstash.Starter.main(Starter.java:43) ~[classes/:na]
[INFO] 2013-02-19 19:26:11.696 [main] com.example.logstash.Starter - Result null
[INFO] 2013-02-19 19:26:11.696 [main] com.example.logstash.Starter - Result null
[INFO] 2013-02-19 19:26:11.697 [main] com.example.logstash.Starter - Result null
[INFO] 2013-02-19 19:26:12.639 [main] com.example.logstash.Starter - Result null
[INFO] 2013-02-19 19:26:12.639 [main] com.example.logstash.Starter - Result null
[INFO] 2013-02-19 19:26:12.639 [main] com.example.logstash.Starter - Result null
Now let's see what Logstash can do for us. From the download section, we get the single JAR file: logstash-1.1.9-monolithic.jar. That's all we need for now. Unfortunately, because of this bug on Windows we have to expand logstash-1.1.9-monolithic.jar somewhere, f.e. into logstash-1.1.9-monolithic folder. Logstash has just three concepts: inputs, filters and outputs. Those are very well explained into the documentation. In our case, the input is application's log file, c:\tmp\application.log. But what would be the output? ElasticSearch seems to be an excellent candidate for that: let's have our logs indexed and searchable any time. Let's download and run it:
Now we are ready to integrate Logstash which should tail our log file and feed it directly to ElasticSearch. Following configuration does exactly that (logstash.conf):
input {
file {
add_field => [ "host", "my-dev-host" ]
path => "c:\tmp\application.log"
type => "app"
format => "plain"
}
}
output {
elasticsearch_http {
host => "localhost"
port => 9200
type => "app"
flush_size => 10
}
}
filter {
multiline {
type => "app"
pattern => "^[^\[]"
what => "previous"
}
}
It might look not very clear on first glance but let me explain what is what. So the input is c:\tmp\application.log, which is a plain text file (format => "plain"). The type => "app" serves as simple marker so the different types of inputs could be routed to outputs through filters with the same type. The add_field => [ "host", "my-dev-host" ] allows to inject additional arbitrary data into the incoming stream, f.e. hostname.
Output is pretty clear: ElasticSearch over HTTP, port 9200 (default settings). Filters need a bit of magic, all because of Java stack traces. The multiline filter will glue the stack trace to the log statement it belongs to so it will be stored as a single (large) multiline. Let's run Logstash:
Great! Now whenever we run our application, Logstash will watch the log file, filter it property and send out directly to ElasticSearch. Cool, but how can we do the search or at least see what kind of data do we have? Though ElasticSearch has awesome REST API, we can use another excellent project, Kibana, web UI front-end for ElasticSearch. Installation is very straightforward and seamless. After a few necessary steps, we have Kibana up and running:
ruby kibana.rb
By default, Kibana provides the web UI available on port 5601, let's point our browser to it, http://localhost:5601/ and we should see something like that (please click on image to enlarge):
All our logs statements complemented by hostname are just there. Exceptions (with stack traces) are coupled with the related log statement. Log levels, timestamps, everything is being shown. Fulltext search is available out-of-the box, thanks to ElasticSearch.
It's all awesome but our application is very simple. Would this approach work across multi-server / multi-application deployment? I am pretty sure it will work just fine. Logstash's integration with Redis, ZeroMQ, RabbitMQ, ... allows to capture logs from tens of different sources and consolidate them in one place. Thanks a lot, Logstash guys!
This post is logical continuation of the previous one. The only difference is the container we are going to use: instead of Jetty it will be our old buddy Apache Tomcat. Surprisingly, it was very easy to embed the latest Apache Tomcat 7 so let me show that now.
I won't repeat the last post in full as there are no any changes except in POM file and Starter class. Aside from those two, we are reusing everything we have done before.
For a POM file, we need to remove Jetty dependencies and replace it with Apache Tomcat ones. The first change would be within properties section, we will replace org.eclipse.jetty.version with org.apache.tomcat.
So this line:
8.1.8.v20121106
becomes:
7.0.34
The second change would be dependencies themselves, we will replace these lines:
Great, this part is done. The last part is dedicated to changes in our main class implementation, where we will replace Jetty with Apache Tomcat.
package com.example;
import java.io.File;
import java.io.IOException;
import org.apache.catalina.Context;
import org.apache.catalina.loader.WebappLoader;
import org.apache.catalina.startup.Tomcat;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.cxf.transport.servlet.CXFServlet;
import org.springframework.web.context.ContextLoaderListener;
import org.springframework.web.context.support.AnnotationConfigWebApplicationContext;
import com.example.config.AppConfig;
public class Starter {
private final static Log log = LogFactory.getLog( Starter.class );
public static void main(final String[] args) throws Exception {
final File base = createBaseDirectory();
log.info( "Using base folder: " + base.getAbsolutePath() );
final Tomcat tomcat = new Tomcat();
tomcat.setPort( 8080 );
tomcat.setBaseDir( base.getAbsolutePath() );
Context context = tomcat.addContext( "/", base.getAbsolutePath() );
Tomcat.addServlet( context, "CXFServlet", new CXFServlet() );
context.addServletMapping( "/rest/*", "CXFServlet" );
context.addApplicationListener( ContextLoaderListener.class.getName() );
context.setLoader( new WebappLoader( Thread.currentThread().getContextClassLoader() ) );
context.addParameter( "contextClass", AnnotationConfigWebApplicationContext.class.getName() );
context.addParameter( "contextConfigLocation", AppConfig.class.getName() );
tomcat.start();
tomcat.getServer().await();
}
private static File createBaseDirectory() throws IOException {
final File base = File.createTempFile( "tmp-", "" );
if( !base.delete() ) {
throw new IOException( "Cannot (re)create base folder: " + base.getAbsolutePath() );
}
if( !base.mkdir() ) {
throw new IOException( "Cannot create base folder: " + base.getAbsolutePath() );
}
return base;
}
}
The code looks pretty simple but verbose because of the fact that it seems impossible to run Apache Tomcat in embedded mode without specifying some working directory. The small createBaseDirectory() function creates a temporary folder which we are feeding to Apache Tomcat as a baseDir. Implementation reveals that we are running Apache Tomcat server instance on port 8080, we are configuring Apache CXF servlet to handle all request at /rest/* path, we are adding Spring context listener and finally we are starting server up.
After building the project as a fat or one jar, we have a full-blown server hosting our JAR-RS application:
Jan 28, 2013 5:54:56 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Jan 28, 2013 5:54:56 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Tomcat
Jan 28, 2013 5:54:56 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.34
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/jsp_2_0.xsd
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/jsp_2_1.xsd
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/jsp_2_2.xsd
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_1_1.dtd
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_1_2.dtd
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_2_0.xsd
Jan 28, 2013 5:54:56 PM org.apache.catalina.startup.DigesterFactory register
WARNING: Could not get url for /javax/servlet/jsp/resources/web-jsptaglibrary_2_1.xsd
Jan 28, 2013 5:54:57 PM org.apache.catalina.loader.WebappLoader setClassPath
INFO: Unknown loader com.simontuffs.onejar.JarClassLoader@187a84e4 class com.simontuffs.onejar.JarClassLoader
Jan 28, 2013 5:54:57 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Jan 28, 2013 5:54:57 PM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization started
Jan 28, 2013 5:54:58 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing Root WebApplicationContext: startup date [Mon Jan 28 17:54:58 EST 2013]; root of context hierarchy
Jan 28, 2013 5:54:58 PM org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider registerDefaultFilters
INFO: JSR-330 'javax.inject.Named' annotation found and supported for component scanning
Jan 28, 2013 5:54:58 PM org.springframework.web.context.support.AnnotationConfigWebApplicationContext loadBeanDefinitions
INFO: Successfully resolved class for [com.example.config.AppConfig]
Jan 28, 2013 5:54:58 PM org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor
INFO: JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
Jan 28, 2013 5:54:58 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@62770d2e: defining beans [org.springframework.context.annotation.internal
ConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProces
sor,org.springframework.context.annotation.internalCommonAnnotationProcessor,appConfig,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor,c
xf,jaxRsServer,jaxRsApiApplication,peopleRestService,peopleService,jsonProvider]; root of factory hierarchy
Jan 28, 2013 5:54:59 PM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /api
Jan 28, 2013 5:54:59 PM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization completed in 1747 ms
Jan 28, 2013 5:54:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Let's issue some HTTP requests so to be sure everything works as we expected:
And we are still 100% XML free! One important note though: we create a temporary folder every time but never delete it (calling deleteOnShutdown for base doesn't work as expected for non-empty folders). Please keep it in mind (add your own shutdown hook, for example) as I decided to leave code clean.
For hardcore server-side Java developer the only way to "speak" out to the world is by using APIs. Today's post is all about JAX-RS: writing and exposing RESTful services using Java.
But we won't do that using a traditional, heavyweight approach involving application server, WAR packaging and whatnot. Instead, we will use awesome Apache CXF framework and as always rely on Spring to wire all pieces together. And for sure we won't stop on that either as we need a web server to run our services on. Using fat or one jar concept we will embed Jetty server into our application and make our final JAR redistributable (all dependencies included) and runnable.
It's a lot of work so let's get started. As we stated above, we will use Apache CXF, Spring and Jetty as a building blocks so let's have them described in a POM file. The one additional dependency worth mentioning is excellent Jackson library for JSON processing.
It's a lot of stuff but should be pretty clear. Now, we are ready to develop our first JAX-RS services by starting with simple JAX-RS application.
package com.example.rs;
import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;
@ApplicationPath( "api" )
public class JaxRsApiApplication extends Application {
}
As simple as it looks like, our application defines an /api to be the entry path for the JAX-RS services. The sample service will manage people represented by Person class.
package com.example.model;
public class Person {
private String email;
private String firstName;
private String lastName;
public Person() {
}
public Person( final String email ) {
this.email = email;
}
public String getEmail() {
return email;
}
public void setEmail( final String email ) {
this.email = email;
}
public String getFirstName() {
return firstName;
}
public String getLastName() {
return lastName;
}
public void setFirstName( final String firstName ) {
this.firstName = firstName;
}
public void setLastName( final String lastName ) {
this.lastName = lastName;
}
}
And following bare bones business service (for simplicity, no database or any other storage are included).
package com.example.services;
import java.util.ArrayList;
import java.util.Collection;
import org.springframework.stereotype.Service;
import com.example.model.Person;
@Service
public class PeopleService {
public Collection< Person > getPeople( int page, int pageSize ) {
Collection< Person > persons = new ArrayList< Person >( pageSize );
for( int index = 0; index < pageSize; ++index ) {
persons.add( new Person( String.format( "person+%d@at.com", ( pageSize * ( page - 1 ) + index + 1 ) ) ) );
}
return persons;
}
public Person addPerson( String email ) {
return new Person( email );
}
}
As you can see, we will generate a list of persons on the fly depending on the page requested. Standard Spring annotation @Service marks this class as a service bean. Our JAX-RS service PeopleRestService will use it for retrieving persons as the following code demonstrates.
package com.example.rs;
import java.util.Collection;
import javax.inject.Inject;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import com.example.model.Person;
import com.example.services.PeopleService;
@Path( "/people" )
public class PeopleRestService {
@Inject private PeopleService peopleService;
@Produces( { "application/json" } )
@GET
public Collection< Person > getPeople( @QueryParam( "page") @DefaultValue( "1" ) final int page ) {
return peopleService.getPeople( page, 5 );
}
@Produces( { "application/json" } )
@PUT
public Person addPerson( @FormParam( "email" ) final String email ) {
return peopleService.addPerson( email );
}
}
Though simple, this class needs more explanations. First of all, we want to expose our RESTful service to /people endpoint. Combining it with /api (where our JAX-RS application resides), it gives as the /api/people as qualified path.
Now, whenever someone issues HTTP GET to this path, the method getPeople should be invoked. This method accepts optional parameter page (with default value 1) and returns list of persons as JSON. In turn, if someone issues HTTP PUT to the same path, the method addPerson should be invoked (with required parameter email) and return new person as a JSON.
Now let's take a look on Spring configuration, the core of our application.
package com.example.config;
import java.util.Arrays;
import javax.ws.rs.ext.RuntimeDelegate;
import org.apache.cxf.bus.spring.SpringBus;
import org.apache.cxf.endpoint.Server;
import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
import org.codehaus.jackson.jaxrs.JacksonJsonProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.example.rs.JaxRsApiApplication;
import com.example.rs.PeopleRestService;
import com.example.services.PeopleService;
@Configuration
public class AppConfig {
@Bean( destroyMethod = "shutdown" )
public SpringBus cxf() {
return new SpringBus();
}
@Bean
public Server jaxRsServer() {
JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint( jaxRsApiApplication(), JAXRSServerFactoryBean.class );
factory.setServiceBeans( Arrays.< Object >asList( peopleRestService() ) );
factory.setAddress( "/" + factory.getAddress() );
factory.setProviders( Arrays.< Object >asList( jsonProvider() ) );
return factory.create();
}
@Bean
public JaxRsApiApplication jaxRsApiApplication() {
return new JaxRsApiApplication();
}
@Bean
public PeopleRestService peopleRestService() {
return new PeopleRestService();
}
@Bean
public PeopleService peopleService() {
return new PeopleService();
}
@Bean
public JacksonJsonProvider jsonProvider() {
return new JacksonJsonProvider();
}
}
It doesn't look complicated but a lot happens under the hood. Let's dissect it into the peices. The two key component here are the factory JAXRSServerFactoryBean which does all heavy lifting for configuring our instance of JAX-RS server, and SpringBus instance which seamlessly glues Spring and Apache CXF together. All other components represent regular Spring beans.
What's not on a picture yet is embedding Jetty web server instance. Our main application class Starter does exactly that.
package com.example;
import org.apache.cxf.transport.servlet.CXFServlet;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.springframework.web.context.ContextLoaderListener;
import org.springframework.web.context.support.AnnotationConfigWebApplicationContext;
import com.example.config.AppConfig;
public class Starter {
public static void main( final String[] args ) throws Exception {
Server server = new Server( 8080 );
// Register and map the dispatcher servlet
final ServletHolder servletHolder = new ServletHolder( new CXFServlet() );
final ServletContextHandler context = new ServletContextHandler();
context.setContextPath( "/" );
context.addServlet( servletHolder, "/rest/*" );
context.addEventListener( new ContextLoaderListener() );
context.setInitParameter( "contextClass", AnnotationConfigWebApplicationContext.class.getName() );
context.setInitParameter( "contextConfigLocation", AppConfig.class.getName() );
server.setHandler( context );
server.start();
server.join();
}
}
Looking through this code uncovers that we are running Jetty server instance on port 8080, we are configuring Apache CXF servlet to handle all request at /rest/* path (which together with our JAX-RS application and service gives us the /rest/api/people), we are adding Spring context listener parametrized with the configuration we have defined above and finally we are starting server up. What we have at this point is full-blown web server hosting our JAX-RS services. Let's see it in action. Firstly, let's package it as single, runnable and redistributable fat or one jar:
mvn clean package
Let's pick up the bits from the target folder and run them:
2013-01-19 11:43:08.636:INFO:oejs.Server:jetty-8.1.8.v20121106
2013-01-19 11:43:08.698:INFO:/:Initializing Spring root WebApplicationContext
Jan 19, 2013 11:43:08 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization started
Jan 19, 2013 11:43:08 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing Root WebApplicationContext: startup date [Sat Jan 19 11:43:08 EST 2013]; root of context hierarchy
Jan 19, 2013 11:43:08 AM org.springframework.context.annotation.ClassPathScanningCandidateComponentProvider registerDefaultFilters
INFO: JSR-330 'javax.inject.Named' annotation found and supported for component scanning
Jan 19, 2013 11:43:08 AM org.springframework.web.context.support.AnnotationConfigWebApplicationContext loadBeanDefinitions
INFO: Successfully resolved class for [com.example.config.AppConfig]
Jan 19, 2013 11:43:09 AM org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor
INFO: JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
Jan 19, 2013 11:43:09 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1f8166e5: defining beans [org.springframework.context.annotation.internal
ConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProces
sor,org.springframework.context.annotation.internalCommonAnnotationProcessor,appConfig,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor,c
xf,jaxRsServer,jaxRsApiApplication,peopleRestService,peopleService,jsonProvider]; root of factory hierarchy
Jan 19, 2013 11:43:10 AM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /api
Jan 19, 2013 11:43:10 AM org.springframework.web.context.ContextLoader initWebApplicationContext
INFO: Root WebApplicationContext: initialization completed in 2227 ms
2013-01-19 11:43:10.957:INFO:oejsh.ContextHandler:started o.e.j.s.ServletContextHandler{/,null}
2013-01-19 11:43:11.019:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:8080
Having our server up and running, let's issue some HTTP requests to it so to be sure everything works just as we expected:
Before ending the post, I would like to mention one great project, Dropwizard, which uses quite similar concepts but pushes it to the level of excellent, well-designed framework, thanks to Yammer guys for that.
Among plenty of useful classes which Java provides for concurrency support, there is one I would like to talk about: SynchronousQueue. In particular, I would like to walk through Producer / Consumer implementation using handy SynchronousQueue as an exchange mechanism.
It might not sound clear why to use this type of queue for producer / consumer communication unless we look under the hood of SynchronousQueue implementation. It turns out that it's not really a queue as we used to think about queues. The analogy would be just a collection containing at most one element.
Why it's useful? Well, there are several reasons. From producer's point of view, only one element (or message) could be stored into the queue. In order to proceed with the next element (or message), the producer should wait till consumer consumes the one currently in the queue. From consumer's point of view, it just polls the queue for next element (or message) available. Quite simple, but the great benefit is: producer cannot send messages faster than consumer can process them.
Here is one of the use cases I encountered recently: compare two database tables (possibly just huge) and detect if those contain different data or data is the same (copy). The SynchronousQueue is quite a handy tool for this problem: it allows to handle each table in own thread as well as compensate the possible timeouts / latency while reading from two different databases.
Let's start by defining our compare function which accepts source and destination data sources as well as a table name (to compare). I am using quite useful JdbcTemplate class from Spring framework as it extremely well abstract all the boring details dealing with connections and prepared statements.
public boolean compare( final DataSource source, final DataSource destination, final String table ) {
final JdbcTemplate from = new JdbcTemplate( source );
final JdbcTemplate to = new JdbcTemplate( destination );
}
Before doing any actual data comparison, it's a good idea to compare table's row count of the source and destination databases:
Now, at least knowing that table contains same number of rows in both databases, we can start with data comparison. The algorithm is very simple:
create a separate thread for source (producer) and destination (consumer) databases
producer thread reads single row from the table and puts it into the SynchronousQueue
consumer thread also reads single row from the table, then asks queue for the available row to compare (waits if necessary) and lastly compare two result sets
Using another great part Java concurrent utilities for thread pooling, let's define a thread pool with fixed amount of threads (2).
final ExecutorService executor = Executors.newFixedThreadPool( 2 );
final SynchronousQueue< List< ? > > resultSets = new SynchronousQueue< List< ? > >();
Following the described algorithm, the producer functionality could be represented as a single callable:
Callable< Void > producer = new Callable< Void >() {
@Override
public Void call() throws Exception {
from.query( "SELECT * FROM " + table,
new RowCallbackHandler() {
@Override
public void processRow(ResultSet rs) throws SQLException {
try {
List< ? > row = ...; // convert ResultSet to List
if( !resultSets.offer( row, 2, TimeUnit.MINUTES ) ) {
throw new SQLException( "Having more data but consumer has already completed" );
}
} catch( InterruptedException ex ) {
throw new SQLException( "Having more data but producer has been interrupted" );
}
}
}
);
return null;
}
};
The code is a bit verbose due to Java syntax but it doesn't do much actually. Every result set read from the table producer converts to a list (implementation has been omitted as it's a boilerplate) and puts in a queue (offer). If queue is not empty, producer is blocked waiting for consumer to finish his work. The consumer, respectively, could be represented as a following callable:
Callable< Void > consumer = new Callable< Void >() {
@Override
public Void call() throws Exception {
to.query( "SELECT * FROM " + table,
new RowCallbackHandler() {
@Override
public void processRow(ResultSet rs) throws SQLException {
try {
List< ? > source = resultSets.poll( 2, TimeUnit.MINUTES );
if( source == null ) {
throw new SQLException( "Having more data but producer has already completed" );
}
List< ? > destination = ...; // convert ResultSet to List
if( !source.equals( destination ) ) {
throw new SQLException( "Row data is not the same" );
}
} catch ( InterruptedException ex ) {
throw new SQLException( "Having more data but consumer has been interrupted" );
}
}
}
);
return null;
}
};
The consumer does a reverse operation on the queue: instead of putting data it pulls it (poll) from the queue. If queue is empty, consumer is blocked waiting for producer to publish next row. The part which is left is only submitting those callables for execution. Any exception returned by the Future's get method indicates that table doesn't contain the same data (or there are issue with getting data from database):
YAML is well-known format within Ruby community, quite widely used for a long time now. But we as Java developers mostly deal with property files and XMLs in case we need some configuration for our apps. How many times we needed to express complicated configuration by inventing our own XML schema or imposing property names convention?
Though JSON is becoming a popular format for web applications, using JSON files to describe the configuration is a bit cumbersome and, in my opinion, is not as expressive as YAML. Let's see what YAML can do for us to make our life easier.
For sure, let's start with the problem. In order for our application to function properly, we need to feed it following data somehow:
version and release date
database connection parameters
list of supported protocols
list of users with their passwords
This list of parameters sounds a bit weird, but the purpose is to demonstrate different data types in work: strings, numbers, dates, lists and maps. The Java model consists of two simple classes: Connection
package com.example.yaml;
public final class Connection {
private String url;
private int poolSize;
public String getUrl() {
return url;
}
public void setUrl(String url) {
this.url = url;
}
public int getPoolSize() {
return poolSize;
}
public void setPoolSize(int poolSize) {
this.poolSize = poolSize;
}
@Override
public String toString() {
return String.format( "'%s' with pool of %d", getUrl(), getPoolSize() );
}
}
and Configuration, both are typical Java POJOs, verbose because of property setters and getters (we get used to it, right?).
package com.example.yaml;
import static java.lang.String.format;
import java.util.Date;
import java.util.List;
import java.util.Map;
public final class Configuration {
private Date released;
private String version;
private Connection connection;
private List< String > protocols;
private Map< String, String > users;
public Date getReleased() {
return released;
}
public String getVersion() {
return version;
}
public void setReleased(Date released) {
this.released = released;
}
public void setVersion(String version) {
this.version = version;
}
public Connection getConnection() {
return connection;
}
public void setConnection(Connection connection) {
this.connection = connection;
}
public List< String > getProtocols() {
return protocols;
}
public void setProtocols(List< String > protocols) {
this.protocols = protocols;
}
public Map< String, String > getUsers() {
return users;
}
public void setUsers(Map< String, String > users) {
this.users = users;
}
@Override
public String toString() {
return new StringBuilder()
.append( format( "Version: %s\n", version ) )
.append( format( "Released: %s\n", released ) )
.append( format( "Connecting to database: %s\n", connection ) )
.append( format( "Supported protocols: %s\n", protocols ) )
.append( format( "Users: %s\n", users ) )
.toString();
}
}
Now, as model is quite clear, let us try to express it as the human being normally does it. Looking back to our list of required configuration, let's try to write it down one by one.
Please notice the usage of Java 1.7, the language extensions and additional libraries simplify a lot of regular tasks as we could see looking into YamlConfigRunner:
The code snippet here loads the configuration from file (args[ 0 ]), tries to parse it and fill up the Configuration class with meaningful data using JavaBeans conventions, converting to the declared types where possible. Running this class with sample.yml as an argument generates the following output:
Version: 1.0
Released: Thu Nov 29 19:00:00 EST 2012
Connecting to database: 'jdbc:mysql://localhost:3306/db' with pool of 5
Supported protocols: [http, https]
Users: {tom=passwd, bob=passwd}
Totally identical to the values we have configured!
Today's post will target an interesting but quite powerful concept: packing your application as single, runnable JAR file, also known as one or fat JAR.
We get used to large WAR archives which contain all dependencies packed together under some common folder structure. With JAR-like packaging the story is a bit different: in order to make your application runnable (via java -jar) all dependencies should be provided over classpath parameter or environment variable. Usually it means there would be some lib folder with all dependencies and some runnable script which will do the job to construct classpath and run JVM. Maven Assembly plugin is well know for making such kind of application distribution.
A slightly different approach would be to package all your application dependencies to the same JAR file and make it runnable without any additional parameters or scripting required. Sounds great but ... it won't work unless you add some magic: meet One-JAR project.
Let's briefly outline the problem: we are writing a stand-alone Spring application which should be runnable just by typing java -jar <our-app.jar>.
As always, let's start with our POM file, which will be pretty simple
Our sample application will bootstrap Spring context, get some bean instance and call a method on it. Our bean is called SimpleBean and looks like:
package com.example;
public class SimpleBean {
public void print() {
System.out.println( "Called from single JAR!" );
}
}
Falling in love with Spring Java configuration, let us define our context as annotated AppConfig POJO:
package com.example.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import com.example.SimpleBean;
@Configuration
public class AppConfig {
@Bean
public SimpleBean simpleBean() {
return new SimpleBean();
}
}
And finally, our application Starter with main():
package com.example;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import com.example.config.AppConfig;
public class Starter {
public static void main( final String[] args ) {
ApplicationContext context = new AnnotationConfigApplicationContext( AppConfig.class );
SimpleBean bean = context.getBean( SimpleBean.class );
bean.print();
}
}
Adding our main class to META-INF/MANIFEST.MF allows to leverage Java capabilities to run JAR file without explicitly specifying class with main() method. Maven JAR plugin can help us with that.
Trying to run java -jar spring-one-jar-0.0.1-SNAPSHOT.jar will print the exception to the console: java.lang.NoClassDefFoundError. The reason is pretty straightforward: even such a simple application as this one already required following libraries to be in classpath.
As the result, there will be another artifact available in the target folder, postfixed with one-jar: spring-one-jar-0.0.1-SNAPSHOT.one-jar.jar. Running this one with java -jar spring-one-jar-0.0.1-SNAPSHOT.one-jar.jar will print to the console:
Called from single JAR!
Fully runnable Java application as single, redistributable JAR file! The last comment: though our application looks pretty simple, One-JAR works perfectly for complex, large applications as well without any issues. Please, add it to your toolbox, it's really useful tool to have.
Continuing to discover the powerful set of Redis features, the one worth mentioning about is out of the box support of pub/sub messaging.
Pub/Sub messaging is essential part of many software architectures. Some software systems demand from messaging solution to provide high-performance, scalability, queues persistence and durability, fail-over support, transactions, and many more nice-to-have features, which in Java world mostly always leads to using one of JMS implementation providers. In my previous projects I have actively used Apache ActiveMQ (now moving towards Apache ActiveMQ Apollo). Though it's a great implementation, sometimes I just needed simple queuing support and Apache ActiveMQ just looked overcomplicated for that.
Alternatives? Please welcome Redis pub/sub! If you are already using Redis as key/value store, few additional lines of configuration will bring pub/sub messaging to your application in no time.
Spring Data Redis project abstracts very well Redis pub/sub API and provides the model so familiar to everyone who uses Spring capabilities to integrate with JMS.
As always, let's start with the POM configuration file. It's pretty small and simple, includes necessary Spring dependencies, Spring Data Redis and Jedis, great Java client for Redis.
Moving on to configuring Spring context, let's understand what we need to have in order for a publisher to publish some messages and for a consumer to consume them. Knowing the respective Spring abstractions for JMS will help a lot with that.
we need connection factory -> JedisConnectionFactory
we need a template for publisher to publish messages -> RedisTemplate
we need a message listener for consumer to consume messages -> RedisMessageListenerContainer
Using Spring Java configuration, let's describe our context:
package com.example.redis.config;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.jedis.JedisConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.listener.ChannelTopic;
import org.springframework.data.redis.listener.RedisMessageListenerContainer;
import org.springframework.data.redis.listener.adapter.MessageListenerAdapter;
import org.springframework.data.redis.serializer.GenericToStringSerializer;
import org.springframework.data.redis.serializer.StringRedisSerializer;
import org.springframework.scheduling.annotation.EnableScheduling;
import com.example.redis.IRedisPublisher;
import com.example.redis.impl.RedisMessageListener;
import com.example.redis.impl.RedisPublisherImpl;
@Configuration
@EnableScheduling
public class AppConfig {
@Bean
JedisConnectionFactory jedisConnectionFactory() {
return new JedisConnectionFactory();
}
@Bean
RedisTemplate< String, Object > redisTemplate() {
final RedisTemplate< String, Object > template = new RedisTemplate< String, Object >();
template.setConnectionFactory( jedisConnectionFactory() );
template.setKeySerializer( new StringRedisSerializer() );
template.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
template.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
return template;
}
@Bean
MessageListenerAdapter messageListener() {
return new MessageListenerAdapter( new RedisMessageListener() );
}
@Bean
RedisMessageListenerContainer redisContainer() {
final RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory( jedisConnectionFactory() );
container.addMessageListener( messageListener(), topic() );
return container;
}
@Bean
IRedisPublisher redisPublisher() {
return new RedisPublisherImpl( redisTemplate(), topic() );
}
@Bean
ChannelTopic topic() {
return new ChannelTopic( "pubsub:queue" );
}
}
Very easy and straightforward. The presence of @EnableScheduling annotation is not necessary and is required only for our publisher implementation: the publisher will publish a string message every 100 ms.
package com.example.redis.impl;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.listener.ChannelTopic;
import org.springframework.scheduling.annotation.Scheduled;
import com.example.redis.IRedisPublisher;
public class RedisPublisherImpl implements IRedisPublisher {
private final RedisTemplate< String, Object > template;
private final ChannelTopic topic;
private final AtomicLong counter = new AtomicLong( 0 );
public RedisPublisherImpl( final RedisTemplate< String, Object > template,
final ChannelTopic topic ) {
this.template = template;
this.topic = topic;
}
@Scheduled( fixedDelay = 100 )
public void publish() {
template.convertAndSend( topic.getTopic(), "Message " + counter.incrementAndGet() +
", " + Thread.currentThread().getName() );
}
}
And finally our message listener implementation (which just prints message on a console).
package com.example.redis.impl;
import org.springframework.data.redis.connection.Message;
import org.springframework.data.redis.connection.MessageListener;
public class RedisMessageListener implements MessageListener {
@Override
public void onMessage( final Message message, final byte[] pattern ) {
System.out.println( "Message received: " + message.toString() );
}
}
Awesome, just two small classes, one configuration to wire things together and we have full pub/sub messaging support in our application! Let's run the application as standalone ...
package com.example.redis;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import com.example.redis.config.AppConfig;
public class RedisPubSubStarter {
public static void main(String[] args) {
new AnnotationConfigApplicationContext( AppConfig.class );
}
}
Seasoned software developer with a great passion to code. I am extensively working with JVM platform using Java, Groovy, Scala as well as other languages and technologies (Ruby, Grails, Play!, Akka, MySQL, PostreSQL, MongoDB, Redis, JUnit, ...)
Posts
