Thursday, July 27, 2017

Logging in Java - Log4j2 over SLF4J

When it comes to logging in Java applications, it is common practice to go with one of many available logging frameworks. As frameworks evolve and address recurring logging needs of applications, it could be cumbersome to switch from an existing framework to a new one. For this reason, it is recommended to program to a common logging interface and change the underlying implementation as needed. SLF4J is a commonly used logging interface, and Log4j is a good example of the work involved in upgrading from one version to another.

In this blog, I will highlight a Log4j2 over SLF4J approach to application logging.

Logging Levels

Using the following log levels are recommended as best practice. Logging approach will adhere to SLF4J Interface and only use log levels available in SLF4J (see http://www.slf4j.org/api/org/apache/commons/logging/Log.html).

Trace
This level should be used when capturing very detailed information about the system. Logging every step of the execution and capturing state of the code. Trace is useful to detect obscure behavior. By default it should be OFF, as IO operations are expensive.

Debug
This level should be used to report detailed information about the behavior and state of the code. It can be used to diagnose specifics or record user/application behavior. By default it should be OFF, as IO operations are expensive.

Info
Use this level to capture useful information about the system, record diagnostics info,  and report normal behavior. It should be used to indicate that the system is correctly initialized and started, and that connections are made successfully. This level can be set to ON.

Warning
Use this level to report issues with the system, such as running low in resources [e.g.,db connections], accessing information that should not be accessed again, retry limits reached, normal timeouts, etc... This level should be used to log messages that need to be flagged and attended to by responsible parties. This level must be ON.

Error
Use this level to record exception and errors with the system. Examples are network problems, resource availability. Messages logged at this level should be handled and responsible parties should be alerted. This level must be ON.

Fatal
Most serious messages should be logged at this level. Information such as why system was stopped should be logged and report immediately. This level must be ON.

Working with Log4j2
Assuming maven is used to build the application, to start programming with Log4j2, include the required dependencies in your pom.xm

<properties>
    ...
    <org.slf4j.version>1.7.21</org.slf4j-version>
    <log4j2.version>2.6.2</log4j2.version>
    ...
</properties>
 
<dependencies>
    ...
    <!-- SLF4J Logging -->
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>${org.slf4j.version}</version>
    </dependency>
 
    <!-- Log4j2 -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>${log4j2.version}</version>
    </dependency>
    <!-- Log4j2 Binding to SLF4J -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-slf4j-impl</artifactId>
        <version>${log4j2.version}</version>
    </dependency>
    <!-- you need the following jar files if planning to use Log4j2 Client GUI with jconsole  -->
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>${log4j2.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-jmx-gui</artifactId>
        <version>${log4j2.version}</version>
    </dependency>
    ...
</dependencies>

In your java code, import the following packages

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
And add the following to Java classes requiring logging

private static final Logger logger = LoggerFactory.getLogger(YourClassName.class);

Configuration
Typically log4j2 configuration file (log4j2.xml) can be placed on the classpath. Several points related to the location of the config file:
  • To make it available on classpath,  place it in src/main/resources
  • To load it as an external file, use -Dlog4j.configurationFile="/path/to/config/log4j2.xml"
  • To load it only for testing, place log4j2-test.xml in src/test/resources
  • To specify the location in a web application, use a servlet context parameter. See Set the Configuration Path.
Here is an example of log4j2.xml:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
    <Appenders>
        <File name="MyAppLog" fileName="/opt/myapp/logs/myapp.log">
            <PatternLayout>
                <Pattern>%d [%p] %c{1.} : %m%n</Pattern>
            </PatternLayout>
        </File>
        <Console name="STDOUT" target="SYSTEM_OUT">
            <PatternLayout>
                <Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
            </PatternLayout>
        </Console>
    </Appenders>

    <Loggers>
        <Root level="WARN">
            <AppenderRef ref="MyAppLog" level="WARN" />
            <AppenderRef ref="STDOUT" level="WARN" />
        </Root>
        <Logger name="org.springframework" level="WARN">
            <AppenderRef ref="STDOUT" level="WARN" />
        </Logger>
        <Logger name="org.apache.cxf" level="WARN">
            <AppenderRef ref="STDOUT" />
        </Logger>
    </Loggers>
</Configuration>

Notes
  • In general, writing messages to a log file is expensive, so use logging levels wisely.
  • In version 1 of Log4j, it was recommended to guard log operations with is[LogLeve]Enabled checks. Here is a great blog on how to overcome inefficiency of logging when log levels not requiring logging.
  • You can add 3rd party tools logging to your configuration
    • For CXF, add the following to src/main/resources/META-INF/org.apache.cxf.Logger
    • org.apache.cxf.common.logging.Slf4jLogger
  • You can configure log files periodically rotate based on certain criteria. Here is an example of rotating log files:
  • <?xml version="1.0" encoding="UTF-8"?>
    <Configuration>
            <Properties>
                    <property name="name">myapp-rolling</property>
                    <property name="pattern">%d{yyyy-MM-dd HH:mm:ss.SSS} | %-5.5p | %-10.10t | %-20.20C:%-5.5L | %msg%n</property>
    <!--
    It will look like that:
    2013-04-03 07:37:51.993 | WARN  | main       | myserver.Server:56    | My app is logging stuff
    -->
            </Properties>
            <Appenders>
                    <RollingFile name="RollingCdspLog" fileName="/path/to/my/logs/${name}.log"
                                 filePattern="/path/to/my/logs/$${date:yyyy-MM}/${name}-%d{yyyy-MM-dd}-%i.log.gz">
                            <PatternLayout>
                                    <pattern>${pattern}</pattern>
                            </PatternLayout>
                            <Policies>
                                    <TimeBasedTriggeringPolicy /><!-- Rotated everyday -->
                                    <SizeBasedTriggeringPolicy size="1 MB"/> <!-- Or every 100 MB -->
                            </Policies>
                    </RollingFile>
                    <File name="CDSPLog" fileName="/path/to/my/logs/override-myapp.log">
                            <PatternLayout>
                                    <Pattern>%d [%p] %c{1.} : %m%n</Pattern>
                            </PatternLayout>
                    </File>
                    <Console name="STDOUT" target="SYSTEM_OUT">
                            <PatternLayout>
                                    <Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
                            </PatternLayout>
                    </Console>
            </Appenders>
    
            <Loggers>
                    <Root level="DEBUG">
                            <AppenderRef ref="CDSPLog" level="INFO" />
                            <AppenderRef ref="RollingCdspLog" level="DEBUG" />
                            <AppenderRef ref="STDOUT" level="WARN" />
                    </Root>
                    <Logger name="org.springframework" level="WARN">
                            <AppenderRef ref="STDOUT" level="WARN" />
                    </Logger>
                    <Logger name="org.apache.cxf" level="WARN">
                            <AppenderRef ref="STDOUT" />
                    </Logger>
            </Loggers>
    </Configuration>
Log4j2 JMX

Log4j 2 has built-in support for JMX. The StatusLogger, ContextSelector, and all LoggerContexts, LoggerConfigs and Appenders are instrumented with MBeans and can be remotely monitored and controlled.

To configure Log4j2 at runtime, a client GUI plugin can be used with jconsole. The following steps will demonstrate how to attach jconsole to a running Junit process. This Junit process will start logging messages.
  1. Download  log4j-core-2.6.2.jar, log4j-api-2.6.2.jar, log4j-jmx-gui-2.6.2.jar (see pom.xml)
  2. Run jconsole -pluginpath ./log4j-core-2.6.2.jar:./log4j-api-2.6.2.jar:./log4j-jmx-gui-2.6.2.jar
  3. Select a local process that is running your app. For example, if you are junit testing in eclipse, select: org.eclipse.jdt.internal.junit.runner.RemoteTestRunner from Local Process list, and press connect. Select Log4j2 tab.
  4. You can change the configuration and select the appropriate Reconfigure button

Saturday, August 27, 2016

On Kafka

This blog captures steps to install and setup Kafka.

For instructions on how to install Kafka, visit: Installation Steps

After downloading:
untar bundle to /opt/apache-kafka
ln -s kafka_2.11-0.9.0.1 current
cd /opt/apache-kafka/current

At this point you can start Zookeper:

bin/zookeeper-server-start.sh config/zookeeper.properties&

To run Kafka on the current server, modify config/server.properties:

Uncomment:

port=9092
host.name=localhost

To allow for removing topics, add the following to config/server.properties:

# Enable topic deletion - only for development
delete.topic.enable=true

Now you can start Kafka:

bin/kafka-server-start.sh config/server.properties&

Kafka REST Proxy
In order to publish messages to Kafka via REST calls, you can use Kafka REST Proxy. To install it, visit Confluent Platform Quickstart.

Untar the downloaded bundle to /opt/confluent:

ln -s confluent-2.0.1 current
cd current
vi etc/kafka-rest/kafka-rest.properties

Uncomment

id=kafka-rest-test-server
schema.registry.url=http://localhost:8081
zookeeper.connect=localhost:2181

To run Kafka REST Proxy:
bin/schema-registry-start etc/schema-registry/schema-registry.properties &
bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties &

Some useful aliases to start/stop/view Kafka processes
alias zs="cd /opt/apache-kafka/current; nohup bin/zookeeper-server-start.sh config/zookeeper.properties 2>/dev/null &"
alias ks="cd /opt/apache-kafka/current; nohup bin/kafka-server-start.sh config/server.properties 2>/dev/null&"

alias krp1="cd /opt/confluent/current; nohup bin/schema-registry-start etc/schema-registry/schema-registry.properties 2>/dev/null&"
alias krp2="cd /opt/confluent/current; nohup bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties 2>/dev/null&"

alias pk="ps -aef|grep kafka"
alias topics="/opt/apache-kafka/current/bin/kafka-topics.sh  --list --zookeeper localhost:2181"

To create topics:
cd/opt/apache-kafka/current; bin/kafka-topics.sh --create --topic test --zookeeper YOUR.IP.ADDRESS.HERE:2181 --partition 1 --replication-factor 1

To list topics:
cd /opt/apache-kafka/current; bin/kafka-topics.sh --list --zookeeper YOUR.IP.ADDRESS.HERE:2181

To consume messages from a topic:
/opt/apache-kafka/current/bin/kafka-console-consumer.sh --zookeeper YOUR.IP.ADDRESS.HERE:2181 --topic MY_TOPIC 


Tuesday, May 31, 2016

JAXB & XML Binding Using XJC

Recently I was given an XML file with multiple schemas defining the structure of the XML file. Given my comfort level with Java technologies, I decided to use JAXB to marshall/unmarshall XML files. This blog describes the steps I took to work with XML files.

How to auto-generate Java classes that represent XML schemas using XJC

Schemas were placed in multiple sub-directories:

~/myproject/src/main/schema/schema1/*.xsd
~/myproject/src/main/schema/schema2/*.xsd

Running xjc on files in schema2 directory worked fine and Java classes were generated.
However, files in schema1 failed to auto-generate with errors like:

[ERROR] Property "Lang" is already defined. Use <jaxb:property> to resolve this conflict.
line 302 of http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd

[ERROR] The following location is relevant to the above error
line 303 of http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd

One of the schemas had a dependency on http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd. I noticed attribute "lang" was used several times in http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd. So xjc was not able to generate unique Java classes to represent these repeating attributes. The good news is you can instruct xjc to rename any portion of the schema is having problems with, using a binding customization file.

To fix the above errors, I created a file called: xhtml1-strict.xjb

<bindings xmlns="http://java.sun.com/xml/ns/jaxb"
          xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance"
          xmlns:xs="http://www.w3.org/2001/XMLSchema"
          version="2.1">
    <bindings schemaLocation="http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd" version="1.0">
        <!-- rename the value element -->
        <bindings node="//xs:attributeGroup[@name='i18n']">
            <bindings node=".//xs:attribute[@name='lang']">
                <property name="I18nLangAttribute"/>
            </bindings>
        </bindings>
        <bindings node="//xs:element[@name='bdo']//xs:complexType">
             <bindings node=".//xs:attribute[@name='lang']">
                 <property name="BdoLangAttribute"/>
             </bindings>
        </bindings>
    </bindings>
</bindings>


Run xjc again to auto-generate Java Binding classes:

cd ~noushin/workspace/ajs/src/main/schema
xjc -d ~noushin/workspace/ajs/src/main/java -b xhtml1-strict.xjb */*.xsd

After running the above commands, you should see Java packages and classes in src directory.

The following snippet shows you how to read a XML file with parent tag and parse it to get content of :

public MyDocument getMyDocument() {
    JAXBContext context;
    try {
        context = JAXBContext.newInstance(MyDocument.class);
        Unmarshaller um = context.createUnmarshaller();
        FileReader fileReader = new FileReader("/path/to/xmlfile/);
        MyDocument myDocument = (MyDocument) um.unmarshal(this.getClass().getClassLoader()
     .getResourceAsStream("mydocument.xml"));
        Metadata metadata = null;
        if (myDocument != null) {
               metadata = myDocument.getMetadata();
        }
    }
    catch (FileNotFoundException | JAXBException e) {
        // handle exception
        e.printStackTrace();
    }
}

You will get an exception during unmarshalling if your xml document parent tag is "":
javax.xml.bind.UnmarshalException: unexpected element (uri:"urn:hl7-org:docns", local:"myDocument"). Expected elements are ...

In this case, you need to add the following to your JAXB Class MyDocument:

@XmlRootElement(name = "myDocument")









Thursday, March 5, 2015

Writing RESTful web services using Apache CXF & Spring

In an earlier blog, I showed you how to develop APIs using REST and Spring RS. Since Spring does not fully implement JAX-RS, in this blog, I will show you how to use Apache CXF with Spring to write RESTful web services. pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.noushin</groupId>
    <artifactId>cxfws</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>war</packaging>
    <name>cxfws</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <cxf.version>3.0.2</cxf.version>
        <junit.version>4.10</junit.version>
        <log4j.version>1.2.17</log4j.version>
        <spring.version>4.1.1.RELEASE</spring.version>
        <spring.security.version>3.2.0.RELEASE</spring.security.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Logging -->
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>

        <!-- Spring Test -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>

        <!-- Spring --> 
         <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>${spring.version}</version>
        </dependency>
 
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-beans</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <!-- JSON Binding -->
        <dependency>
            <groupId>org.codehaus.jackson</groupId>
            <artifactId>jackson-jaxrs</artifactId>
            <version>1.5.7</version>
        </dependency>

        <!-- CXF -->
        <dependency>
            <groupId>org.apache.cxf</groupId>
            <artifactId>cxf-rt-frontend-jaxrs</artifactId>
            <version>${cxf.version}</version>
        </dependency>

    </dependencies>
    
    
    <build>
        <finalName>cxfws</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>
The first class you need to write is an interface to expose your web services:
package com.noushin.cxfws.user.service;

import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

import com.noushin.cxfws.user.model.Status;
import com.noushin.cxfws.user.model.User;

@Path("/user")
public interface UserService {

    /*
     * curl  -i -v -X GET -H "Accept: application/json" http://localhost:8080/cxfws/services/user/123
     * return: {"id":"123","firstName":"John","lastName":"Smith"}
     */
    @GET
    @Consumes(MediaType.APPLICATION_FORM_URLENCODED)
    @Produces({ MediaType.APPLICATION_JSON })
    @Path("/{id}")
    public User getUser(@PathParam("id") String id);
    
    /*
     * curl  -i -v -X POST -H "Content-type: application/json" -H "Accept: application/json" http://localhost:8080/cxfws/services/user/ -d @param.json
     * return: {"id":"456","firstName":"Jane","lastName":"Doe"}
     */
    
    @POST
    @Consumes(MediaType.APPLICATION_JSON)
    @Produces({ MediaType.APPLICATION_JSON })
    @Path("/")
    public User saveUser(User user);
    
    /*
     * curl  -i -v -X PUT -H "Accept: application/json" http://localhost:8080/cxfws/services/user/123/abcd
     * return: {"id":"123","firstName":"abcd","lastName":"old lastname"}
     */
    @PUT
    @Consumes(MediaType.APPLICATION_FORM_URLENCODED)
    @Produces({ MediaType.APPLICATION_JSON })
    @Path("/{id}/{fn}")
    public User updateUser(@PathParam("id") String id, @PathParam("fn") String firstName);
    
    
    /*
     * curl  -i -v -X DELETE -H "Accept: application/json" http://localhost:8080/cxfws/services/user/123
     * return: {"message":"User removed successfully.","code":0}
     */
    @DELETE
    @Consumes(MediaType.APPLICATION_FORM_URLENCODED)
    @Produces({ MediaType.APPLICATION_JSON })
    @Path("/{id}")
    public Status removeUser(@PathParam("id") String id);

}
Now, you have to write the implementation class:
package com.noushin.cxfws.user.service;

import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import com.noushin.cxfws.user.dao.UserDao;
import com.noushin.cxfws.user.model.Status;
import com.noushin.cxfws.user.model.User;
import com.noushin.cxfws.user.reference.StatusCodes;

@Service("userService")
public class UserServiceImpl implements UserService {

    private final static Logger logger = Logger.getLogger(UserServiceImpl.class);
    
    @Autowired
    UserDao userDao;
    
    public User getUser(String id) {
        String msg = "getting user with id: " + id + ".";
        logger.info(msg);
        return userDao.getUser(id);
    }

    public User saveUser(User user) {
        String msg = "saving user with id: " + user.getId() + ".";
        logger.info(msg);
        return userDao.saveUser(user);
    }

    public User updateUser(String id, String firstName) {
        String msg = "updating user with id: " + id + ".";
        logger.info(msg);
        User user = new User();
        user.setId(id);
        user.setFirstName(firstName);
        return userDao.saveUser(user);
    }

    public Status removeUser(String id) {
        String msg = "removing user with id: " + id + ".";
        logger.info(msg);
        Status status = null;
        if (userDao.removeUser(id))
            return new Status(StatusCodes.FAILED, "User removed successfully.");
        return  new Status(StatusCodes.FAILED, "User could not be removed.");
    }

}

The JSON objects that will be passed to or retuned from user web services, will be presented as simple POJOS. User JSON:
package com.noushin.cxfws.user.model;

public class User {
    
    private String id;
    private String firstName;
    private String lastName;
    
    public User() {
    }
    
    public User(String id, String firstName, String lastName) {
        super();
        this.id = id;
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public String getId() {
        return id;
    }
    
    public void setId(String id) {
        this.id = id;
    }
    
    public String getFirstName() {
        return firstName;
    }
    
    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }
    
    public String getLastName() {
        return lastName;
    }
    
    public void setLastName(String lastName) {
        this.lastName = lastName;
    }
    
    public String toString() {
        return "User {id:" + id + ", firstName:" + firstName + ", lastName:" + lastName + "}";
    }
    
}

Status JSON:
package com.noushin.cxfws.user.model;

public class Status {
    private int code;
    private String message;
    
    
    public Status() {
    }

    public Status(int code, String message) {
        super();
        this.code = code;
        this.message = message;
    }
    
    public int getCode() {
        return code;
    }
    
    public void setCode(int code) {
        this.code = code;
    }
    
    public String getMessage() {
        return message;
    }
    
    public void setMessage(String message) {
        this.message = message;
    }
    
    
}


UserDao.java:
package com.noushin.cxfws.user.dao;

import org.apache.log4j.Logger;
import org.springframework.stereotype.Component;

import com.noushin.cxfws.user.model.User;

@Component
public class UserDao {
    
    private final static Logger logger = Logger.getLogger(UserDao.class);

    public User getUser(String id) {
        User user = new User(id, "John", "Smith");
        String msg = "getting user with id: " + id + ".";
        logger.info(msg);
        return user;
    }
    
    public User saveUser(User user) {
        String msg = "saving user with id: " + user.getId() + ".";
        logger.info(msg);
        return user;
    }
    
    public boolean removeUser(String id) {
        String msg = "removing user with id: " + id + ".";
        logger.info(msg);
        return true;
    }
    
}


Reference data:
package com.noushin.cxf.user.reference;

public class StatusCodes {
    public final static int SUCCESS = 1;
    public final static int FAILED = 0;
    
}

~cxfws/src/main/webapp/WEB-INF/web.xml
<?xml version="1.0" encoding="UTF-8"?>
<web-app version="3.1" xmlns="http://xmlns.jcp.org/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">

    <display-name>Spring CXF WS Demo</display-name>
    <description>Spring CXF WS Demo</description>

    <servlet>
        <servlet-name>CXFServlet</servlet-name>
        <servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>CXFServlet</servlet-name>
        <url-pattern>/services/*</url-pattern>
    </servlet-mapping>

    <context-param>
        <param-name>log4jConfigLocation</param-name>
        <param-value>/WEB-INF/classes/log4j.properties</param-value>
    </context-param>

    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>/WEB-INF/cxfws-servlet.xml</param-value>
    </context-param>

    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>

</web-app>
Spring application context is in ~cxfws/src/main/webapp/WEB-INF/cxfws-servlet.xml
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jaxrs="http://cxf.apache.org/jaxrs"
    xmlns:cxf="http://cxf.apache.org/core"
    xmlns:http-conf="http://cxf.apache.org/transports/http/configuration"
    xsi:schemaLocation="
      http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
      http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
      http://cxf.apache.org/jaxrs http://cxf.apache.org/schemas/jaxrs.xsd 
      http://cxf.apache.org/transports/http/configuration http://cxf.apache.org/schemas/configuration/http-conf.xsd">

    <context:component-scan base-package="com.noushin.cxfws.user" />
    <context:annotation-config />

    <jaxrs:server id="userServer" address="/">
        <jaxrs:serviceBeans>
            <ref bean="userService" />
        </jaxrs:serviceBeans>
        <jaxrs:providers>
            <bean class="org.codehaus.jackson.jaxrs.JacksonJsonProvider" />
        </jaxrs:providers>
        <jaxrs:extensionMappings>
            <entry key="json" value="application/json" />
        </jaxrs:extensionMappings>
        <jaxrs:features>
            <cxf:logging/>
        </jaxrs:features>
    </jaxrs:server>

    <bean id="userServiceImpl" class="com.noushin.cxfws.user.service.UserServiceImpl" />

</beans>

Friday, February 27, 2015

Using SSH to transfer files in Java

In this example, I'll be using JSch library to copy a file from my local server to a remote host. If using Maven, add the following dependency to your pom.xml file:
        <!-- for ssh -->
        <dependency>
            <groupId>com.jcraft</groupId>
            <artifactId>jsch</artifactId>
            <version>${jsch.version}</version>
        </dependency>
Define an interface for your remote file operations:
public interface RemoteFileUtilities {
    public void copyFile(String filename, HostInfo hostInfo);
    public List<String> lsDir(String dir, HostInfo hostInfo);
}
SSH implementation class:
public class SshUtilities implements RemoteFileUtilities {

    @Override
    public void copyFile(String filename, HostInfo hostInfo) {
        JSch jsch = new JSch();
        Session session = null;
        ChannelSftp channel = null;
        try {
            session = jsch.getSession(hostInfo.getUsername(),
                    hostInfo.getHostname(), hostInfo.getPort());
            session.setPassword(hostInfo.getPassword());
            session.setConfig("StrictHostKeyChecking", "no");
            session.connect();
            channel = (ChannelSftp) session.openChannel("sftp");
            channel.connect();
            File localFile = new File(filename);
            // If you want you can change the directory using the following line.
            channel.cd(hostInfo.getPath());
            channel.put(new FileInputStream(localFile), localFile.getName());   
        } 
        catch (JSchException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } 
        catch (FileNotFoundException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } 
        catch (SftpException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        channel.disconnect();
        session.disconnect();
    }

    @Override
    public List<String> lsDir(String dir, HostInfo hostInfo) {
        JSch jsch = new JSch();
        Session session = null;
        ChannelSftp channel = null;
        try {
            session = jsch.getSession(hostInfo.getUsername(), hostInfo.getHostname(), hostInfo.getPort());
            session.setPassword(hostInfo.getPassword());
            session.setConfig("StrictHostKeyChecking", "no");
            session.connect();
            channel = (ChannelSftp) session.openChannel("sftp");
            channel.connect();
            return channel.ls(dir);  
        } 
        catch (JSchException e) {
            e.printStackTrace();
        } 
        catch (SftpException e) {
             e.printStackTrace();
        }
        channel.disconnect();
        session.disconnect();
        return null;
    }     
}

A supporting class to hold information related to your host:
public class HostInfo {
    
    private String hostname;
    private String username;
    private String password;
    private String path;
    private int port;
    
    
    public HostInfo(String hostname, String username, String password, String path, int port) {
        super();
        this.hostname = hostname;
        this.username = username;
        this.password = password;
        this.path = path;
        this.port = port;
    }
    
    public String getHostname() {
        return hostname;
    }
    public void setHostname(String hostname) {
        this.hostname = hostname;
    }
    public String getUsername() {
        return username;
    }
    public void setUsername(String username) {
        this.username = username;
    }
    public String getPassword() {
        return password;
    }
    public void setPassword(String password) {
        this.password = password;
    }
    public String getPath() {
        return path;
    }
    public void setPath(String path) {
        this.path = path;
    }
    public int getPort() {
        return port;
    }
    public void setPort(int port) {
        this.port = port;
    } 
}
And finally, a JUnit test class:
public class SshUtilitiesTest {

    @Test
    public void test() {
        HostInfo hostInfo = new HostInfo("dev-core-ccm-1", "noushin", "nbcg14g~", "/home/noushin/ssh-test", 22);
        SshUtilities sshUtils = new SshUtilities();
        sshUtils.copyFile("/tmp/log/error.log", hostInfo);
        List<String> files = sshUtils.lsDir(hostInfo.getPath(), hostInfo);
        if (files != null) {
            for (int i = 0; i < files.size(); i++)
                assert(files.get(i).contains("error.log"));
        }
    }
}

Sending and receiving files via RESTful web services in Java

In this blog, I'll show you how to pass the content of a file to a RESTful web service and have it stored on the server it was received. First write your CXF service and annotate it as follows:
@POST
@Path("saveFile")
@Produces({MediaType.APPLICATION_JSON})
@Consumes({MediaType.MULTIPART_FORM_DATA})
public UploadFile saveFile(MultipartBody body) {
     List<Attachment> attachments = body.getAllAttachments();
     Attachment att = attachments.get(0);
     File destinationFile = new File("/tmp/log/fileX.txt");

     try {
          att.transferTo(destinationFile);
     } 
     catch (IOException ioe) {
          ioe.printStackTrace();
     }
     UploadFile savedFile = new UploadFile();
     savedFile.setFilename("/tmp/log/fileX.txt");
     return savedFile;
}
To call this service, I am using Apache httpcomponents package. Add the following dependencies to your pom.xml:
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.0-beta2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpmime</artifactId>
            <version>4.0-beta2</version>
        </dependency>
And here is the test snippet to call the web service:
    @Test
    public void restSaveFile()  {
        String url = "http://localhost:8080/mywebapp/services/";
        ResponseEntity<AuthenticationToken> authToken = restClient.authenticate(url + "authenticate");
        String token = null;
        if (authToken != null)
            token = authToken.getBody().getToken();
        try {
            HttpClient httpclient = new DefaultHttpClient();
            HttpPost httpPost = new HttpPost(url + "saveFile");
            // Assuming ~mywebapp/src/test/resource/data/test-file.txt
            FileBody body = new FileBody(new File(Thread.currentThread().getContextClassLoader().getResource("data/test-file.txt").getFile()));
            StringBody comment = new StringBody("Test data for saving a file.");
            httpPost.addHeader("X-Auth-Token", token);
            httpPost.addHeader("Content-Type", MediaType.MULTIPART_FORM_DATA_VALUE);       
            MultipartEntity requestEntity = new MultipartEntity();
            requestEntity.addPart("file", body);
            requestEntity.addPart("comment", comment);
            httpPost.setEntity(requestEntity);        
            System.out.println("Executing request " + httpPost.getRequestLine());
            HttpResponse response = httpclient.execute(httpPost);
            HttpEntity responseEntity = response.getEntity();
            System.out.println("----------------------------------------------");
            System.out.println(response.getStatusLine());
            if (responseEntity != null) {
                System.out.println("Response content length: " + responseEntity.getContentLength());
                System.out.println("Response content type: " + responseEntity.getContentType().getValue());
            }
            if (responseEntity != null) {           
                responseEntity.consumeContent();
            }
            System.out.println("----------------------------------------------");
        } 
        catch (Exception ex) {
            ex.printStackTrace();
        }        
    }

JAXB & Xml Binding Using Annotations

Create an entity bean and annotate it for JAXB Xml binding:
package com.noushin.soaws.model;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;
import javax.xml.bind.annotation.XmlType;

@XmlRootElement(name="book")
@XmlType(propOrder={"name", "author", "isbn", "publisher"})
public class Book {

    private String name;
    private String author;
    private String isbn;
    private String publisher;
    
    public Book() {}
    
    public Book(String author) {
        this.author = author;        
    }
    
    public String getAuthor() {
        return author;
    }
    
    public void setAuthor(String author) {
        this.author = author;
    }

    @XmlElement(name="title")
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getIsbn() {
        return isbn;
    }

    public void setIsbn(String isbn) {
        this.isbn = isbn;
    }

    public String getPublisher() {
        return publisher;
    }

    public void setPublisher(String publisher) {
        this.publisher = publisher;
    }

}

Create a class to manage the entity bean, read to and write from XML files:
package com.noushin.soaws.service;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;

import javax.jws.WebService;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlElementWrapper;
import javax.xml.bind.annotation.XmlRootElement;

import com.noushin.soaws.model.Book;

@XmlRootElement(namespace="com.noushin.soaws")
public class BookService {

    @XmlElementWrapper(name="books")
    @XmlElement(name="book")
    ArrayList<Book> books;

    public void setBooks() {
        books = new ArrayList<Book>();

        // create books
        Book book1 = new Book();
        book1.setIsbn("978-0060554736");
        book1.setName("The Game");
        book1.setAuthor("Neil Strauss");
        book1.setPublisher("Harpercollins");
        books.add(book1);

        Book book2 = new Book();
        book2.setIsbn("978-3832180577");
        book2.setName("Feuchtgebiete");
        book2.setAuthor("Charlotte Roche");
        book2.setPublisher("Dumont Buchverlag");
        books.add(book2);

    }

    public ArrayList<Book> getBooks() {
        return books;
    }

    public Book getBook() {
        return new Book("Mark Twain");
    }

    public static void manage() {
        BookService bookService = new BookService();
        bookService.setBooks();
        
        // create JAXB context and instantiate marshaller
        JAXBContext context;
        try {
            context = JAXBContext.newInstance(BookService.class);

            Marshaller m = context.createMarshaller();
            m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);

            // Write to System.out
            m.marshal(bookService, System.out);

            // Write to File
            m.marshal(bookService, new File("/tmp/books.xml"));

            // get variables from our xml file, created before
            System.out.println();
            System.out.println("Output from our XML File: ");
            Unmarshaller um = context.createUnmarshaller();
            BookService bookService2 = (BookService) um.unmarshal(new FileReader("books.xml"));
            ArrayList<Book> list = bookService2.getBooks();
            for (Book book : list) {
                System.out.println("Book: " + book.getName() + " from " + book.getAuthor());
            }
        } 
        catch (JAXBException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } 
        catch (FileNotFoundException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}


Using Junit, write a test case to test the functionality:
package com.noushin.soaws.service;

import static org.junit.Assert.*;

import org.junit.Test;

public class BookServiceTest {

    @Test
    public void test() {
        BookService bookService = new BookService();
        bookService.manage();
    }

}


Wednesday, November 19, 2014

JSON/Java Binding

In this blog, I'll show you how to bind JSON data to Java objects.

Let's start with a simple books example. The following JSON file contains a list of books:

books.json:
[
{"ibsn":"978-0486400778", "author":"Mark Twain", "title":"The Adventures of Tom Sawyer"},
{"ibsn":"978-0486280615", "author":"Mark Twain", "title":"Adventures of Huckleberry Finn"}
]
To map this data to a list of Book objects, first define a Book POJO:
package com.noushin.databinding;

public class Book {
    private String ibsn;
    private String title;
    private String author;
    
    public String getIbsn() {
        return ibsn;
    }
    public void setIbsn(String ibsn) {
        this.ibsn = ibsn;
    }
    public String getTitle() {
        return title;
    }
    public void setTitle(String title) {
        this.title = title;
    }
    public String getAuthor() {
        return author;
    }
    public void setAuthor(String author) {
        this.author = author;
    }
    
    public String toString() {
        return String.format("Book ibsn: %s, title: %, author: %s ", ibsn, title, author);
    }
}

I'll be using Jackson for data binding, so add the proper dependency to your pom file:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.noushin</groupId>
    <artifactId>databinding</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>databinding</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>2.3.3</version>
        </dependency>
    </dependencies>
</project>

And the code to perform the transformation:
package com.noushin.databinding;

import java.io.IOException;
import java.net.URISyntaxException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.List;

import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;

public class BookService {

    public List<Book> transform(String jsonFileName) throws IOException, URISyntaxException {
        byte[] jsonData = Files.readAllBytes(Paths.get(ClassLoader.getSystemResource(jsonFileName).toURI()));
        ObjectMapper mapper = new ObjectMapper();
        List<Book> books = mapper.readValue(jsonData, new TypeReference<List<Book>>(){});
        System.out.println("Transformed " + + books.size() + " books.");
        return books;
    }
}

And finally, here is your JUnit testcase to verify the transformation:
package com.noushin.databinding;

import java.io.IOException;
import java.net.URISyntaxException;
import java.util.List;

import org.junit.Test;

import junit.framework.TestCase;

public class BookServiceTest extends TestCase {

    @Test
    public void testBookService() {
        BookService bookService = new BookService();
        try {
            List<Book> books = bookService.transform("books.json");
            assertTrue(books.get(0).getAuthor().equals("Mark Twain"));
        } 
        catch (IOException e) {
            e.printStackTrace();
        } 
        catch (URISyntaxException e) {
            e.printStackTrace();
        }
    }
    

Wednesday, October 8, 2014

Jython - Calling Python from Java

Let's say you do most of your development in Java, however, there is some existing Python code you need to reuse. In this post, I'll show you how to use Jython to call Python from Java.

Write a Java wrapper class:
public class JythonDemo {

    public int testMe(int num1, int num2) {
        PythonInterpreter python = new PythonInterpreter();

        python.set("num1", num1);
        python.set("num2", num2);
        python.exec("num3 = num1 + num2");
        PyObject num3 = python.get("num3");

        return Integer.parseInt(num3.toString());

    }
}

If using Maven, here is your pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.noushin</groupId>
    <artifactId>demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>war</packaging>
    <name>demo</name>
    <description>Demo</description>

    <dependencies>
        <dependency>
            <groupId>org.python</groupId>
            <artifactId>jython-standalone</artifactId>
            <version>2.5.2</version>
        </dependency>
    </dependencies>

</project>
Just call testMe(), and that'll run your Python code:
JythonDemo jythonDemo = new JythonDemo();
int result = jythonDemo.testMe();


Wednesday, June 4, 2014

Sending emails with attachments

In this example, I will show you how to use Java and Spring to send an email with attachments. Here is the main class to start things off:
package com.noushin.spring.email;

import java.io.File;
import java.util.ArrayList;
import java.util.Collection;

import javax.mail.MessagingException;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

/**
 * Main class to demonstrate sending an email with attachments.
 * 
 * @author nbashir
 * 
 */
public class EmailMain {

   public static void main(String[] args) {
      ApplicationContext ctx = new ClassPathXmlApplicationContext("application-context.xml");
      try {
         if (ctx != null) {
            Emailer emailer = ctx.getBean(Emailer.class);
            Collection<File> attachments = new ArrayList<File>();
            File attachment = new File("/home/nbashir/file.txt");
            attachments.add(attachment);
            emailer.sendEmail("Test Email", "Please see attached file(s).", attachments);
         }
      }
      catch (MessagingException me) {
         // TODO Auto-generated catch block
         me.printStackTrace();
      }
      catch (Exception ex) {
         ex.printStackTrace();
      }
   }
}
And Emailer class that generates an Email message, attaches files to it, and delivers.
package com.noushin.spring.email;

import java.io.File;
import java.util.Collection;

import javax.mail.MessagingException;
import javax.mail.internet.MimeMessage;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.core.io.FileSystemResource;
import org.springframework.mail.javamail.JavaMailSender;
import org.springframework.mail.javamail.MimeMessageHelper;
import org.springframework.stereotype.Component;

@Component
public class Emailer {

   @Autowired
   @Qualifier("mailSender")
   private JavaMailSender mailSender;

   @Value("${email.recipients}")
   private String emailRecipients;
   @Value("${email.from}")
   private String emailFrom;
   @Value("${email.reply.to}")
   private String emailReplyTo;

   /**
    * Send an email with attachments
    * 
    * @param subject email's subject line
    * @param body email's body
    * @param attachments list of attachments
    * 
    * @throws MessagingException 
    */
   public void sendEmail(String subject, String body, Collection<File> attachments) throws MessagingException {

      MimeMessage message = mailSender.createMimeMessage();
      MimeMessageHelper msgHelper = new MimeMessageHelper(message, MimeMessageHelper.MULTIPART_MODE_MIXED_RELATED);

      String[] recipients = getRecipients(emailRecipients);
      if (recipients != null) {
         for (int i = 0; i < recipients.length; i++)
            msgHelper.addTo(recipients[i]);
      }
      msgHelper.setFrom(emailFrom);
      msgHelper.setReplyTo(emailReplyTo);
      msgHelper.setSubject(subject);
      msgHelper.setText(body, true);

      if (attachments != null) {
         String filename = null;
         for (File attachment : attachments) {
            filename = attachment.getName();
            FileSystemResource res = new FileSystemResource(attachment);
            msgHelper.addAttachment(filename, res);
         }
      }

      mailSender.send(message);
   }

   /**
    * Return a list of strings
    * 
    * @param csvList a string containing a comma separated values 
    * @return a list of strings
    */
   public String[] getRecipients(String csvList) {
      String[] values = null;
      if (csvList != null) {
         values = csvList.split(",");
         for (int i = 0; i < values.length; i++)
            values[i] = values[i].trim();
      }
      return values;
   }
}
The pom file you need:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.noushin.spring</groupId>
    <artifactId>email</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>email</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spring.version>4.0.3.RELEASE</spring.version>
        <java.mail>1.4.7</java.mail>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context-support</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>javax.mail</groupId>
            <artifactId>mail</artifactId>
            <version>${java.mail}</version>
        </dependency>

    </dependencies>

</project>

And application context:
<?xml version="1.0" encoding="UTF-8"?>
<beans 
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">

    <context:property-placeholder location="file:/home/nbashir/email.properties" />
    
    <!-- Activate annotation configured components -->
    <context:annotation-config />

    <!-- Scan components for annotations within the configured package -->
    <context:component-scan base-package="com.noushin.spring.email" />
        
    <!-- Email handler -->
    <bean id="mailSender" class="org.springframework.mail.javamail.JavaMailSenderImpl">
        <property name="host" value="${email.host}" />
        <property name="port" value="${email.port}" />
    </bean>

</beans>

Wednesday, April 23, 2014

Spring & Hive

Getting Spring & Hive integration working wasn't a breeze, but I got it. Here is what I had to do:

First, a typical BookRepository class:
package com.noushin.spring.ht.dao;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.hadoop.hive.HiveTemplate;
import org.springframework.stereotype.Repository;

import java.util.List;

/**
 * This class handles accessing books in Hadoop using Hive.
 * 
 * @author nbashir
 * 
 */
@Repository
public class BookRepositroy {

   @Autowired
   private HiveTemplate hiveTemplate;

   public void showTables() {
      List<String> tables = hiveTemplate.query("show tables;");
      System.out.println("tables size: " + tables.size());
   }

   public Long count() {
      return hiveTemplate.queryForLong("select count(*) from books;");
   }
}
Next a typical Service layer class:
package com.noushin.spring.ht.service;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import com.noushin.spring.ht.dao.BookRepositroy;

/**
 * This class handles any business logic related to handling books.
 * 
 * @author nbashir
 *
 */
@Component
public class BookService {

   @Autowired 
   BookRepositroy bookRepo;
   
   public Long count() {
      return bookRepo.count();
   }

   public void showTables() {
      bookRepo.showTables();
   }
 }

Here is the pom file you need:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.noushin.spring</groupId>
    <artifactId>ht</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
    <name>ht</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spring.hadoop.version>1.0.2.RELEASE</spring.hadoop.version>
        <hadoop.version>1.2.1</hadoop.version>
        <hive.version>0.10.0</hive.version>
        <spring.version>4.0.3.RELEASE</spring.version>
    </properties>

    <dependencies>

        <!-- Spring Data -->
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-hadoop</artifactId>
            <version>${spring.hadoop.version}</version>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework</groupId>
                    <artifactId>spring-context-support</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-jdbc</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>${hadoop.version}</version>
            <scope>compile</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-builtins</artifactId>
            <version>${hive.version}</version>
            <scope>runtime</scope>
        </dependency>
        
    </dependencies>
    
</project>

You application-context.xml:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
    xmlns:hdp="http://www.springframework.org/schema/hadoop" xmlns:c="http://www.springframework.org/schema/c"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
         http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
         http://www.springframework.org/schema/hadoop http://www.springframework.org/schema/hadoop/spring-hadoop.xsd">

    <context:property-placeholder location="hive.properties" />

    <!-- Activate annotation configured components -->
    <context:annotation-config />

    <!-- Scan components for annotations within the configured package -->
    <context:component-scan base-package="com.noushin.spring.ht" />

    <hdp:hive-client-factory id="hiveClientFactory"
        host="${hive.host}" port="${hive.port}">
        <hdp:script>
            ADD JARS /usr/lib/hive/lib/books.jar;
        </hdp:script>
    </hdp:hive-client-factory>

    <hdp:hive-template id="hiveTemplate" />

</beans>


Your hive.properties:
hive.host=your-hive-server
hive.port=10000
hive.url=jdbc:hive://${hive.host}:${hive.port}/default

And finally a main class to start and run your app:
package com.noushin.spring.ht;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import com.noushin.spring.ht.service.BookService;

/**
 * Main class to demonstrate accessing MongoDB with Spring Data and MongoTemplate.
 * 
 * @author nbashir
 *
 */
public class HadoopMain {

   public static void main(String[] args) {
      try {
         ApplicationContext ctx = new ClassPathXmlApplicationContext("application-context.xml");
         if (ctx != null) {
            BookService service = ctx.getBean(BookService.class);
            service.showTables();
            Long count = service.count();
            System.out.println("result  : " + count);        
         }
      }
      catch (Exception ex) {
         System.out.println("HadoopMain encountered an error and ended.");
      }
   }
}

Saturday, April 12, 2014

Scheduling a task in Java

Using Spring, scheduling a task is quite simple. Just follow these steps:

1. Write a class that defines the scheduled tasks. Use @Scheduled annotation.
package com.noushin.spring.scheduler;

import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

@Component
public class TaskScheduler {

   // run the task every n seconds, 5 secs = 5000
   @Scheduled(fixedDelayString = "${task.execution.time.delayed}")
   public void delayedTask() {
      System.out.println("Do something meaningful after each delayed period.");
   }

   // run the task every once in a while, example: 0 13 30 * * * means run this task every day at 1:30 pm
   @Scheduled(cron = "${task.execution.time.every.day}")
   public void dailyTask() {
      System.out.println("Do something meaningful once a day.");
   }

}
2. Write a main class to load Spring's application context.
package com.noushin.spring.scheduler;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class SechedulerMain {
   
   static TaskScheduler reportScheduler;
   
   public static void main(String[] args) {
      try {
         ApplicationContext ctx = new ClassPathXmlApplicationContext("application-context.xml");
         if (ctx != null) {
            reportScheduler = ctx.getBean(TaskScheduler.class);
            while (true) {
               System.out.println("main: --------------------------");
               Thread.sleep(1000);
            }          
         }
      }
      catch (Exception ex) {
         System.out.println("SchedulerMain encountered an error and ended.");
         ex.printStackTrace();
      }
   }
}
3. Add application-context.xml.
<?xml version="1.0" encoding="UTF-8"?>
<beans 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns="http://www.springframework.org/schema/beans"
    xmlns:context="http://www.springframework.org/schema/context"
    xmlns:task="http://www.springframework.org/schema/task"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
        http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task.xsd">

 <!-- Scan components for annotations within the configured package -->
 <context:component-scan base-package="com.noushin.spring" />
 
 <context:property-placeholder location="task.properties" />
       
 <!-- Activate annotation configured components -->
 <context:annotation-config />
 
 <!-- Scheduling tasks using annotations -->
 <task:annotation-driven />

</beans>
4. A property file to configure the scheduler, lets call it task.properties.
# Scheduling report job
task.execution.time.delayed=5000
report.execution.time.every.day=13 30 * * * *

Note: Use the Crontab Patterns as defined in : http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/scheduling/support/CronSequenceGenerator.html

5. Use the following dependencies in your pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.noushin.spring</groupId>
    <artifactId>scheduler</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>scheduler</name>
    <description>Demo</description>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <log4j.version>1.2.17</log4j.version>
        <spring.version>4.0.2.RELEASE</spring.version>
    </properties>

    <dependencies>
        <!-- Logging -->
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>

        <!-- Spring Context -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
    </dependencies>
</project>

Monday, February 24, 2014

Spring & MongoTemplate

To keep up with Spring templates theme, I will show you how to use MongoTemplate to access MongoDB. Assuming you are already familiar with MongoDB and the fact that it's a document centric datastore, the first thing we should do is define objects representing documents. In this tutorial, my application will manage books in a MongoDB database. Here is the definition of a book stored as a document.
package com.noushin.spring.monspr.model;

import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.index.Indexed;
import org.springframework.data.mongodb.core.mapping.Document;

/**
 * This class represents a book as a MongoDB document.
 * 
 * @author nbashir
 * 
 */
@Document
public class Book {

   @Id
   private String id;

   @Indexed
   private String name;

   private String author;

   public Book(String id, String name, String author) {
      super();
      this.id = id;
      this.name = name;
      this.author = author;
   }

   public String getId() {
      return id;
   }

   public void setId(String id) {
      this.id = id;
   }

   public String getName() {
      return name;
   }

   public void setName(String name) {
      this.name = name;
   }

   public String getAuthor() {
      return author;
   }

   public void setAuthor(String author) {
      this.author = author;
   }
}
Now I need a class to access books in MongoDB. This is where I will utilize MongoTemplate.
package com.noushin.spring.monspr.dao;

import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.stereotype.Repository;

import com.noushin.spring.monspr.model.Book;

/**
 * This class handles accessing books in MongoDB.
 * 
 * @author nbashir
 *
 */
@Repository
public class BookRepositroy {
   
   @Autowired
   MongoTemplate mongoTemplate;
      
   public void save(Book book) {
      mongoTemplate.save(book);
   }
   
   public void save(List<Book> books) {
      mongoTemplate.save(books);
   }

   public Book get(String id) {
      return mongoTemplate.findById(id, Book.class);
   }
   
   public List<Book> getAll() {
      return mongoTemplate.findAll(Book.class);
   }
   
   public void delete(Book book) {
      mongoTemplate.remove(book);
   }
}
And a typical class to place in the service layer:
package com.noushin.spring.monspr.service;

import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import com.noushin.spring.monspr.dao.BookRepositroy;
import com.noushin.spring.monspr.model.Book;

/**
 * This class handles any business logic related to handling books.
 * 
 * @author nbashir
 *
 */
@Component
public class BookService {

   @Autowired 
   BookRepositroy bookRepo;
   
   public void save(Book book) {
      bookRepo.save(book);
   }
   
   public void save(List<Book> books) {
      bookRepo.save(books);
   }

   public Book get(String id) {
      return bookRepo.get(id);
   }
   
   public List<Book> getAll() {
      return bookRepo.getAll();
   }
   
   public void delete(Book book) {
      bookRepo.delete(book);
   }
}
And last, but not least, a class to load Spring context and do some basic data access operations in regards to our books example:
package com.noushin.spring.monspr;

import java.util.ArrayList;
import java.util.UUID;

import org.apache.log4j.Logger;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import com.noushin.spring.monspr.model.Book;
import com.noushin.spring.monspr.service.BookService;

/**
 * Main class to demonstrate accessing MongoDB with Spring Data and MongoTemplate.
 * 
 * @author nbashir
 *
 */
public class MongoDbMain {

   final static Logger logger = Logger.getLogger(MongoDbMain.class);

   public static void main(String[] args) {
      try {
         ApplicationContext ctx = new ClassPathXmlApplicationContext("application-context.xml");
         if (ctx != null) {
            logger.info(ctx.getApplicationName() + "MongoDbMain successfuly started. ");
            BookService service = ctx.getBean(BookService.class);
                       
            Book book;
            book = new Book(UUID.randomUUID().toString(), "Ragtime", "E.L. Doctrow");
            service.save(book);
            book = new Book(UUID.randomUUID().toString(), "The March", "E.L. Doctrow");
            service.save(book);
            book = new Book(UUID.randomUUID().toString(), "Andrew's Brain", "E.L. Doctrow");
            service.save(book);
            book = new Book(UUID.randomUUID().toString(), "Wild: From Lost to Found on the Pacific Crest Trail ",
                  "Cheryl Strayed");
            service.save(book);

            ArrayList<Book> books = (ArrayList<Book>) service.getAll();
            for (int i = 0; i < books.size(); i++)
               System.out.println("Book " + i + " : " + books.get(i).getId() + " - " + books.get(i).getName());
            
            for (int i = 0; i < books.size(); i++)
               service.delete(books.get(i));
      
            ArrayList<Book> deltedBooks =  (ArrayList<Book>) service.getAll();
            if (deltedBooks.size() > 0 ) {
               for (int i = 0; i < deltedBooks.size(); i++)
                  System.out.println("Book " + i + " : " + deltedBooks.get(i).getId() + " - " + deltedBooks.get(i).getName());
            }
            else
               System.out.println("All book are successfully removed.");

         }
      }
      catch (Exception ex) {
         logger.error("MongoDbMain encountered an error and ended.", ex);
      }
   }
}
Now that we have the basic Java code in place, we need to configure our app, so it knows how to access the datastore. As with any other Spring application, configuration details are in application-context.xml.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">

    <!-- Activate annotation configured components -->
    <context:annotation-config />

    <!-- Scan components for annotations within the configured package -->
    <context:component-scan base-package="com.noushin.spring.monspr" />

    <!-- Mongo config -->
    <!-- Factory bean that creates the Mongo instance -->

    <bean id="mongo" class="org.springframework.data.mongodb.core.MongoFactoryBean">
        <property name="host" value="[your-mongodb-host]" />
    </bean>

    <bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
        <constructor-arg name="mongo" ref="mongo" />
        <constructor-arg name="databaseName" value="nbdb" />
    </bean>
</beans>
In the above configuration, my app will be connecting to "nbdb" database, with MongoDB running on a server specified by "host" property of MongoFactoryBean.

To complete this example, here is my pom.xml where application dependecies are listed:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.noushin.spring</groupId>
  <artifactId>monspr</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>monspr</name>
  <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <junit.version>4.11</junit.version>
        <log4j.version>1.2.17</log4j.version>
        <spring.version>3.2.4.RELEASE</spring.version>
        <spring.data.version>1.3.4.RELEASE</spring.data.version>
    </properties>

    <dependencies>
        <!-- Test -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-test</artifactId>
            <version>${spring.version}</version>
            <scope>test</scope>
        </dependency>
        
        <!-- Logging -->
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>${log4j.version}</version>
        </dependency>
            
        <!-- Spring Data -->
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-mongodb</artifactId>
            <version>${spring.data.version}</version>
        </dependency>
        
        <!-- Spring Context -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
    </dependencies>
</project>