Distributed Systems¶

2021/22¶

Lab 3

Nuno Preguiça, Sérgio Duarte, Dina Borrego, João Vilalonga

Goals¶

In the end of this lab you should be able to:

  • Know how to deal with errors on REST clients;
  • Know how to track errors on REST servers;
  • Use the Tester to verify that your service complies to the specification

REST - Client Errors¶

Causes¶

A REST request might fail for several reasons:

  • The server is not running;
  • The server is slow;
  • A TCP connection was dropped;
  • The network failed;
  • There was a network anomaly (e.g., routing)

Transient failures¶

Temporary failures can be masked by issuing the request multiple times.

Usually, the client quits after a few retries to avoid blocking the application forever,
for example, in case the server has crashed.

Note:

Transient failures are temporary issues that resolve themselves shortly.

ProcessingException¶

Jersey (JAX-RS) exposes request failures to clients in the form of a Java exception: javax.ws.rs.ProcessingException

A try{ } catch{} block can be used to retry the request automatically after a small amount of time.

Note: Waiting a bit before retrying the request prevents a too agressive client behavior and allows some time for the transient error condition disappear.

Example - CreateUser (1)¶

Setting the client timeout values...

protected static final int READ_TIMEOUT = 5000;
protected static final int CONNECT_TIMEOUT = 5000;

ClientConfig config = new ClientConfig();

config.property(ClientProperties.READ_TIMEOUT, READ_TIMEOUT);
config.property( ClientProperties.CONNECT_TIMEOUT, CONNECT_TIMEOUT);

Client client = ClientBuilder.newClient(config);
In the future, other changes to client behavior will be done the someway.

Example - CreateUser (2)¶

protected static final int MAX_RETRIES = 10;
protected static final int RETRY_SLEEP = 1000;

@Override
public String createUser(User user) {

    WebTarget target = client.target( serverURI ).path( RestUsers.PATH );
    for (int i = 0; i < MAX_RETRIES; i++)
        try {
            Response r = target.request()
                .accept(MediaType.APPLICATION_JSON)
                .post(Entity.entity(user, MediaType.APPLICATION_JSON));

            if( r.getStatus() == Status.OK.getStatusCode() && r.hasEntity() )
                // SUCCESS
                return r.readEntity(String.class);
            else {
                System.out.println("Error, HTTP error status: " + r.getStatus() );
                break;
            }
        } catch (ProcessingException x) {
            sleep( RETRY_SLEEP );
        }
    return null; // Report failure
}

Can we do better?¶

The sample code above needs to be repeated for all operations of all services!

  • Doable but error prone because it invites a lot of cut & paste...

Can we make it more general?

Of course!!!

Step 1 - Implement the request as a private method¶

private String clt_createUser(User user) {
    Response r = target.request()
                    .accept(MediaType.APPLICATION_JSON)
                    .post(Entity.entity(user, MediaType.APPLICATION_JSON));

    if( r.getStatus() == Status.OK.getStatusCode() && r.hasEntity() )
        return r.readEntity(String.class);
    else
        return null;
}

Step 2 - Implement the retry behavior as a generic operation¶

protected <T> T reTry(java.util.function.Supplier<T> func) {
    for (int i = 0; i < MAX_RETRIES; i++)
        try {
            return func.get(); // Success
        } catch (ProcessingException x) {
            sleep(RETRY_SLEEP);
        } catch (Exception x) {
            // Handle other errors
            break;
        }   
    return null; // Failure
}

Note:

This method can be part of super class inherited by all service clients.

Step 3 - Implement the requests with retry behavior¶

public String createUser(User user) {
    return reTry( () -> clt_createUser( user ));
}

public User getUser(String userId, String password) {
    return reTry( () -> clt_getUser( user, password ));
}

We are making use of Java Lambda Expressions...

() -> clt_createUser( user ) is a function that returns a result, making it compatible with the functional interface Supplier<T> used as the parameter of the reTry generic method.

Tracking REST server-side errors¶

Unhandled server-side exceptions¶

Jersey reports to clients any unhandled exception thrown in the scope of an implementation method of a service as 500 Internal Error.

The default Jersey runtime behavior is to suppress the actual exception details, including type or stack trace, making it very difficult to diagnose the source of the problem.

CustomExceptionMappers¶

Jersey provides a way to customize how server-side exceptions are reported before a response is sent to the client.

For instance, the class below is a custom exception mapper that will allow the stack trace of an exception to show up on server logs.

public class GenericExceptionMapper implements ExceptionMapper<Throwable> {
    @Override
    public Response toResponse(Throwable ex) {

        if (ex instanceof WebApplicationException wex) {
            Response r = wex.getResponse();         
            if( r.getStatus() == Status.INTERNAL_SERVER_ERROR.getStatusCode())
                ex.printStackTrace();
            return r;
        }
        ex.printStackTrace();
        return Response.status(Status.INTERNAL_SERVER_ERROR)
            .entity(ex.getMessage()).type(MediaType.APPLICATION_JSON).build();
    }
}

The code will still report the error to the client as 500 Internal Error.

However, it also prints the exception stack trace to the server output, making it easier to find which class, method and line of code caused the problem.

Registering CustomExceptionMappers¶

To use a custom exception mapper, we have register it in addition to the services exposed by the server, like so:

ResourceConfig config = new ResourceConfig();
config.register(UsersResource.class);
config.register(GenericExceptionMapper.class);

String ip = InetAddress.getLocalHost().getHostAddress();
String serverURI = String.format(SERVER_URI_FMT, ip, PORT);
JdkHttpServerFactory.createHttpServer( URI.create(serverURI), config);

Exercises¶

  1. Test the retry mechanism to mask transient failures;
  2. Test the mechanism for exposing server-side errors;
  3. Use the Tester to test your RestUsers service;
  4. Complete the provided client-side code of RestUsers service.

Test masking transient failures¶

  1. Download this lab's project;
  2. Build the Docker image, using the usual maven command;

    mvn clean compile assembly:single docker:build

  3. Launch the server;

    docker network create -d bridge sdnet (if necessary)

    docker run --rm -h users-1 --name users-1 --network sdnet -p 8080:8080 sd2122-tp1-xxxxx-yyyyy

  4. Create the client container:

    docker run -it --network sdnet sd2122-tp1-xxxxx-yyyyy /bin/bash

  5. Create a new user.

    In the client container shell, type:

    java -cp /home/sd/sd2122.jar sd2122.aula3.clients.CreateUserClient http://users-1:8080/rest nmp "Nuno Preguica" nmp@nova.unl.pt 12345

    Confirm the request succeeded and returned: nmp

  6. Stop the server.

    CTRL-C or execute in another terminal:

    docker rm -f users-1

  7. Execute step 5 again.

    This time, since the server is not running, the client should output:

    FINE: ProcessingException: java.net.UnknownHostException: users-1

  8. Launch the server again (step 3)

    The client should finish and return: nmp

Test exposing server-side errors¶

  1. Download this lab's project (if necessary);

  2. In the main method of the UsersServer.java file, uncomment the following line:

    config.register(GenericExceptionMapper.class);

  3. Build the Docker image, using the usual maven command;

    mvn clean compile assembly:single docker:build

  4. Launch the server;

    docker network create -d bridge sdnet (if necessary)

    docker run --rm -h users-1 --name users-1 --network sdnet -p 8080:8080 sd2122-tp1-xxxxx-yyyyy

  5. Create the client container:

    docker run -it --network sdnet sd2122-tp1-xxxxx-yyyyy /bin/bash

  6. Try the SearchUsersClient:

    In the client container shell, type:

    java -cp /home/sd/sd2122.jar sd2122.aula3.clients.SearchUsersClient http://users-1:8080/rest nmp

    The client should report:

    Error, HTTP error status: 500

    In the server, the top line of stack trace shown should be:

    at sd2122.aula3.server.resources.UsersResource.searchUsers(UsersResource.java:96)

    This is the method, class file and line number where the uncaught exception was thrown.

  7. Undo Step 2 (by commenting the line);

  8. Repeat Steps 3, 4, 5 and 6.

    This time the output of the server will only show that the request reached the server. It will not show that an exception was thrown while processing the request.

Test Users service¶

The Tester runs your service and performs a series of tests to check if your the implementation complies to the specification.

  1. Apply the changes you did in Lab 2 to the Users service of Lab 3 project.

  2. Adapt the provided trab.props file to your project, making sure class names and network ports match your implementation. Check the Tester documentation if unsure.

  3. Build the docker image of your completed service;

    mvn clean compile assembly:single docker:build

  4. Run the Tester script;

    Linux:

    chmod a+x test-sd-tp1.sh (once)

    ./test-sd-tp1.sh -image sd2122-tp1-xxxxx-yyyyy -sleep 10 -log ALL

Windows: 

`test-sd-tp1.bat -image sd2122-tp1-xxxxx-yyyyy -sleep 10 -log ALL`