Monday, October 28, 2013

Coordination and service discovery with Apache Zookeeper

Service-oriented design has proven to be a successful solution for a huge variety of different distributed systems. When used properly, it has a lot of benefits. But as number of services grows, it becomes more difficult to understand what is deployed and where. And because we are building reliable and highly-available systems, yet another question to ask: how many instances of each service are currently available?

In today's post I would like to introduce you to the world of Apache ZooKeeper - a highly reliable distributed coordination service. The number of features ZooKeeper provides is just astonishing so let us start with very simple problem to solve: we have a stateless JAX-RS service which we deploy across as many JVMs/hosts as we want. The clients of this service should be able to auto-discover all available instances and just pick one of them (or all) to perform a REST call.

Sounds like a very interesting challenge. There could be many ways to solve it but let me choose Apache ZooKeeper for that. The first step is to download Apache ZooKeeper (the current stable version at the moment of writing is 3.4.5) and unpack it. Next, we need to create a configuration file. The simple way to do that is by copying conf/zoo_sample.cfg to conf/zoo.cfg. To run, just execute:

Windows: bin/zkServer.cmd
Linux: bin/zkServer

Excellent, now Apache ZooKeeper is up and running, listening on port 2181 (default). Apache ZooKeeper itself worth of a book to explain its capabilities. But brief overview gives a very high-level picture, enough to get us started.

Apache ZooKeeper has a powerful Java API but it's quite low-level and not an easy one to use. That's why Netflix developed and open-sourced a great library called Curator to wrap native Apache ZooKeeper API into more convenient and easy to integrate framework (it's now an Apache incubator project).

Now, let's do some code! We are developing simple JAX-RS 2.0 service which returns list of people. As it will be stateless, we are able to run many instances within single host or multiple hosts, depending on system load for example. The awesome Apache CXF and Spring Framework will backed our implementation. Below is the code snippet for PeopleRestService:

package com.example.rs;

import java.util.Arrays;
import java.util.Collection;

import javax.annotation.PostConstruct;
import javax.inject.Inject;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;

import com.example.model.Person;

@Path( PeopleRestService.PEOPLE_PATH ) 
public class PeopleRestService {
    public static final String PEOPLE_PATH = "/people";
 
    @PostConstruct
    public void init() throws Exception {
    }
 
    @Produces( { MediaType.APPLICATION_JSON } )
    @GET
    public Collection< Person > getPeople( @QueryParam( "page") @DefaultValue( "1" ) final int page ) {
        return Arrays.asList(
            new Person( "Tom", "Bombadil" ),
            new Person( "Jim", "Tommyknockers" )
        );
    }
}

Very basic and naive implementation. Method init is empty by intention, it will be very helpful quite soon. Also, let us assume that every JAX-RS 2.0 service we're developing does support some notion of versioning, the class RestServiceDetails serves this purpose:

package com.example.config;

import org.codehaus.jackson.map.annotate.JsonRootName;

@JsonRootName( "serviceDetails" )
public class RestServiceDetails {
    private String version;
 
    public RestServiceDetails() {
    }
    
    public RestServiceDetails( final String version ) {
        this.version = version;
    }
    
    public void setVersion( final String version ) {
        this.version = version;
    }
    
    public String getVersion() {
        return version;
    }    
}

Our Spring configuration class AppConfig creates instance of JAX-RS 2.0 server with people REST service which will be hosted by Jetty container:

package com.example.config;

import java.util.Arrays;

import javax.ws.rs.ext.RuntimeDelegate;

import org.apache.cxf.bus.spring.SpringBus;
import org.apache.cxf.endpoint.Server;
import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;

import com.example.rs.JaxRsApiApplication;
import com.example.rs.PeopleRestService;
import com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider;

@Configuration
public class AppConfig {
    public static final String SERVER_PORT = "server.port";
    public static final String SERVER_HOST = "server.host";
    public static final String CONTEXT_PATH = "rest";
 
    @Bean( destroyMethod = "shutdown" )
    public SpringBus cxf() {
        return new SpringBus();
    }
 
    @Bean @DependsOn( "cxf" )
    public Server jaxRsServer() {
        JAXRSServerFactoryBean factory = RuntimeDelegate.getInstance().createEndpoint( jaxRsApiApplication(), JAXRSServerFactoryBean.class );
        factory.setServiceBeans( Arrays.< Object >asList( peopleRestService() ) );
        factory.setAddress( factory.getAddress() );
        factory.setProviders( Arrays.< Object >asList( jsonProvider() ) );
        return factory.create();
    } 

    @Bean 
    public JaxRsApiApplication jaxRsApiApplication() {
        return new JaxRsApiApplication();
    }
 
    @Bean 
    public PeopleRestService peopleRestService() {
        return new PeopleRestService();
    }
 
    @Bean
    public JacksonJsonProvider jsonProvider() {
        return new JacksonJsonProvider();
    } 
}

And here is the class ServerStarter which runs embedded Jetty server. As we would like to host many such servers per host, the port shouldn't be hard-coded but rather provided as an argument:

package com.example;

import org.apache.cxf.transport.servlet.CXFServlet;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.springframework.web.context.ContextLoaderListener;
import org.springframework.web.context.support.AnnotationConfigWebApplicationContext;

import com.example.config.AppConfig;

public class ServerStarter {
    public static void main( final String[] args ) throws Exception {
        if( args.length != 1 ) {
            System.out.println( "Please provide port number" );
            return;
        }
  
        final int port = Integer.valueOf( args[ 0 ] );
        final Server server = new Server( port );
   
        System.setProperty( AppConfig.SERVER_PORT, Integer.toString( port ) );
        System.setProperty( AppConfig.SERVER_HOST, "localhost" );
  
        // Register and map the dispatcher servlet
        final ServletHolder servletHolder = new ServletHolder( new CXFServlet() );
        final ServletContextHandler context = new ServletContextHandler();   
        context.setContextPath( "/" );
        context.addServlet( servletHolder, "/" + AppConfig.CONTEXT_PATH + "/*" );  
        context.addEventListener( new ContextLoaderListener() );
   
        context.setInitParameter( "contextClass", AnnotationConfigWebApplicationContext.class.getName() );
        context.setInitParameter( "contextConfigLocation", AppConfig.class.getName() );
      
        server.setHandler( context );
        server.start();
        server.join(); 
    }
}

Nice, at this moment the boring part is over. But where Apache ZooKeeper and service discovery fit into this picture? Here is the answer: whenever new PeopleRestService service instance is deployed, it publishes (or registers) itself into Apache ZooKeeper registry, including the URL it's accessible at and service version it hosts. The clients can query Apache ZooKeeper in order to get the list of all available services and call them. The only thing services and their clients need to know is where Apache ZooKeeper is running. As I am deploying everything on my local machine, my instance is on localhost. Let's add this constant to the AppConfig class:

private static final String ZK_HOST = "localhost";

Every client maintains the persistent connection to the Apache ZooKeeper server. Whenever client dies, the connection goes down as well and Apache ZooKeeper can make a decision about availability of this particular client. To connect to Apache ZooKeeper, we have to create an instance of CuratorFramework class:

@Bean( initMethod = "start", destroyMethod = "close" )
public CuratorFramework curator() {
    return CuratorFrameworkFactory.newClient( ZK_HOST, new ExponentialBackoffRetry( 1000, 3 ) );
}

Next step is to create an instance of ServiceDiscovery class which will allow to publish service information for discovery into Apache ZooKeeper using just created CuratorFramework instance (we also would like to submit RestServiceDetails as additional metadata along with every service registration):

@Bean( initMethod = "start", destroyMethod = "close" )
public ServiceDiscovery< RestServiceDetails > discovery() {
    JsonInstanceSerializer< RestServiceDetails > serializer = 
        new JsonInstanceSerializer< RestServiceDetails >( RestServiceDetails.class );

    return ServiceDiscoveryBuilder.builder( RestServiceDetails.class )
        .client( curator() )
        .basePath( "services" )
        .serializer( serializer )
        .build();        
}

Internally, Apache ZooKeeper stores all its data as hierarchical namespace, much like standard file system does. The services path will be the base (root) path for all our services. Every service also needs to figure out which host and port it's running. We can do that by building URI specification which is included into JaxRsApiApplication class (the {port} and {scheme} will be resolved by Curator framework at the moment of service registration):

package com.example.rs;

import javax.inject.Inject;
import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;

import org.springframework.core.env.Environment;

import com.example.config.AppConfig;
import com.netflix.curator.x.discovery.UriSpec;

@ApplicationPath( JaxRsApiApplication.APPLICATION_PATH )
public class JaxRsApiApplication extends Application {
    public static final String APPLICATION_PATH = "api";
 
    @Inject Environment environment;

    public UriSpec getUriSpec( final String servicePath ) {
        return new UriSpec( 
            String.format( "{scheme}://%s:{port}/%s/%s%s",
                environment.getProperty( AppConfig.SERVER_HOST ),
                AppConfig.CONTEXT_PATH,
                APPLICATION_PATH, 
                servicePath
            ) );   
    }
}

The last piece of the puzzle is the registration of PeopleRestService inside service discovery, and the init method comes into play here:

@Inject private JaxRsApiApplication application;
@Inject private ServiceDiscovery< RestServiceDetails > discovery; 
@Inject private Environment environment;

@PostConstruct
public void init() throws Exception {
    final ServiceInstance< RestServiceDetails > instance = 
        ServiceInstance.< RestServiceDetails >builder()
            .name( "people" )
            .payload( new RestServiceDetails( "1.0" ) )
            .port( environment.getProperty( AppConfig.SERVER_PORT, Integer.class ) )
            .uriSpec( application.getUriSpec( PEOPLE_PATH ) )
            .build();
  
    discovery.registerService( instance );
}

Here is what we have done:

  • created a service instance with name people (the complete name would be /services/people)
  • set the port to the actual value this instance is running
  • set the URI specification for this specific REST service endpoint
  • additionally, attached a payload (RestServiceDetails) with service version (though it's not used, it demonstrates the ability to pass more details)
Every new service instance we are running will publish itself under /services/people path in Apache ZooKeeper. To see everything in action, let us build and run couple of people service instances.

mvn clean package
java -jar jax-rs-2.0-service\target\jax-rs-2.0-service-0.0.1-SNAPSHOT.one-jar.jar 8080
java -jar jax-rs-2.0-service\target\jax-rs-2.0-service-0.0.1-SNAPSHOT.one-jar.jar 8081

From Apache ZooKeeper it might look like this (please notice that session UUIDs will be different):

Having two service instances up and running, let's try to consume them. From service client prospective, the first step is exactly the same: instances of CuratorFramework and ServiceDiscovery should be created (configuration class ClientConfig declares those beans), in the way we have done it above, no changes required. But instead of registering service, we will query the available ones:

package com.example.client;

import java.util.Collection;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.springframework.context.annotation.AnnotationConfigApplicationContext;

import com.example.config.RestServiceDetails;
import com.netflix.curator.x.discovery.ServiceDiscovery;
import com.netflix.curator.x.discovery.ServiceInstance;

public class ClientStarter {
    public static void main( final String[] args ) throws Exception {
        try( final AnnotationConfigApplicationContext context = new AnnotationConfigApplicationContext( ClientConfig.class ) ) { 
            @SuppressWarnings("unchecked")
            final ServiceDiscovery< RestServiceDetails > discovery = 
                context.getBean( ServiceDiscovery.class );
            final Client client = ClientBuilder.newClient();
      
            final Collection< ServiceInstance< RestServiceDetails > > services = 
                discovery.queryForInstances( "people" );
            for( final ServiceInstance< RestServiceDetails > service: services ) {
                final String uri = service.buildUriSpec();
       
                final Response response = client
                    .target( uri )
                    .request( MediaType.APPLICATION_JSON )
                    .get();
       
                System.out.println( uri + ": " + response.readEntity( String.class ) );
                System.out.println( "API version: " + service.getPayload().getVersion() );
       
                response.close();
            }
        }
    }
}

Once service instances are retrieved, the REST call is being made (using awesome JAX-RS 2.0 client API) and additionally the service version is being asked (as payload contains instance of RestServiceDetails class). Let's build and run our client against two instances we have deployed previously:

mvn clean package
java -jar jax-rs-2.0-client\target\jax-rs-2.0-client-0.0.1-SNAPSHOT.one-jar.jar

The console output should show two calls to two different endpoints:

http://localhost:8081/rest/api/people: [{"email":null,"firstName":"Tom","lastName":"Bombadil"},{"email":null,"firstName":"Jim","lastName":"Tommyknockers"}]
API version: 1.0

http://localhost:8080/rest/api/people: [{"email":null,"firstName":"Tom","lastName":"Bombadil"},{"email":null,"firstName":"Jim","lastName":"Tommyknockers"}]
API version: 1.0

If we stop one or all instances, they will disappear from Apache ZooKeeper registry. The same applies if any instance crashes or becomes unresponsive.

Excellent! I guess we achieved our goal using such a great and powerful tool as Apache ZooKeeper. Thanks to its developers as well as to Curator guys for making it so easy to use Apache ZooKeeper in your applications. We have just scratched the surface of what is possible to accomplish by using Apache ZooKeeper, I strongly encourage everyone to explore its capabilities (distributed locks, caches, counters, queues, ...).

Worth to mention another great project build on top of Apache ZooKeeper from LinkedIn called Norbert. For Eclipse developers, the Eclipse plugin is also available.

All sources are available on GitHub.

13 comments:

Anonymous said...

Very nice article. I ever wanted to read something like that about zookeeper. Thanks.

Munish Gupta said...

One easier way is to use the Netflix Eureka project to handle this scenario.

Andriy Redko said...

Hi Munish,

Yes, that's another great project from Netflix. As far as I know it is designated to be used with Amazon Webservices (AWS) and has dependency to its SDK.

Thank you.

Best Regards,
Andriy Redko

João Laranjeira said...

Hi Andriy.

Do you know if it is possible to send a list as a parameter? For this case, a List of person.

Best Regards.

Andriy Redko said...

Hi João,

I am pretty sure you can. I would be happy to help if you could post a pseudo code what you are trying to achieve. We can figure out the possible ways to do that.

Thank you.

Best Regards,
Andriy Redko

Anonymous said...

Hi Andriy,

Great article!

One question, let's say to consume your service without Zookeeper you would send HTTP request directly to http://localhost:8080/rest/api/people.

But now that you registered your service in service discovery, how would you consume the service? Would you send the HTTP request to something like http://localhost:2181/services/people?

Thanks,
Helen

Andriy Redko said...

Hi!

Thanks a lot for your comment! The purpose of service discovery is to provide a single place where the services might be looked up. For example, the People REST services might be deployed somewhere (http://:/) and the discovery service will have the exact location(s) of the all available People REST services:
- http://host1:port1/rest/api/people
- http://host2:port2/rest/people
- http://host3:port3/api/people
...

In the blog this single service registry is served by Zookeeper.

Hope it makes things clear!
Let me know if you have more questions!

Thanks!

Andriy Redko

Anonymous said...

Hi Andriy,

Thanks for your responce.

In your example you showed how to query available services and consume them from the client application.

I am just trying to understand if that is possible to consume the service without building a specific client. So that I could send HTTP request to Zookeeper (say just hit url from the browser) and then Zookeeper would delegate this request to one of the instances?

Thanks,
Helen

Andriy Redko said...

Hi Helen,

Thanks a lot for the clarification!

As far as I know, with ZooKeeper the use case you have described won't work. You basically looking for load balancer (like haproxy, nginx, ...): the configuration of active services could be retrieved from ZooKeeper and uploaded to load balancer, it will delegate the requests then to particular running instances (using round-robin, ...). I haven't done this but it sounds quite feasible.

Thanks!

Best Regards,
Andriy Redko

Unknown said...

very nice blog good information your site
<a href='http://www.softql.com/services/infrastructure-management-services/datacenter">rf services </a>

Andriy Redko said...

Thank you!

Adam T. said...

Hi Andriy,

This is a great article. Thanks for posting!

To piggyback off of Helen's questions, could this be used in conjunction with an nginx load balancer to dynamically update the routing based on what active entry points exists?

For instance, either finding/developing some middleware to dynamically update the nginx routing config when a change is detected within Zookeeper's registry?

That would solve both high availability for the services as well as simplifying scaling/deploying new instances.

Thanks again!

Andriy Redko said...

Hi Adam,

Thank you very much for the comment. You are very right, these days service discovery is often complemented by nginx/haproxy, so the client just uses single endpoint address. I haven't done that with ZooKeeper personally yet, but check out https://github.com/Xorlev/gatekeeper for example, I think this is what you need.

Thank you!

Best Regards,
Andriy Redko