On Kafka

This blog captures steps to install and setup Kafka.

For instructions on how to install Kafka, visit: Installation Steps

After downloading:

untar bundle to /opt/apache-kafka
ln -s kafka_2.11-0.9.0.1 current
cd /opt/apache-kafka/current

At this point you can start Zookeper:

bin/zookeeper-server-start.sh config/zookeeper.properties&

To run Kafka on the current server, modify config/server.properties:

Uncomment:

port=9092
host.name=localhost

To allow for removing topics, add the following to config/server.properties:

# Enable topic deletion - only for development
delete.topic.enable=true

Now you can start Kafka:

bin/kafka-server-start.sh config/server.properties&

Kafka REST Proxy
In order to publish messages to Kafka via REST calls, you can use Kafka REST Proxy. To install it, visit Confluent Platform Quickstart.

Untar the downloaded bundle to /opt/confluent:

ln -s confluent-2.0.1 current
cd current
vi etc/kafka-rest/kafka-rest.properties

Uncomment

id=kafka-rest-test-server
schema.registry.url=http://localhost:8081
zookeeper.connect=localhost:2181

To run Kafka REST Proxy:

bin/schema-registry-start etc/schema-registry/schema-registry.properties &
bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties &

Some useful aliases to start/stop/view Kafka processes

alias zs="cd /opt/apache-kafka/current; nohup bin/zookeeper-server-start.sh config/zookeeper.properties 2>/dev/null &"
alias ks="cd /opt/apache-kafka/current; nohup bin/kafka-server-start.sh config/server.properties 2>/dev/null&"

alias krp1="cd /opt/confluent/current; nohup bin/schema-registry-start etc/schema-registry/schema-registry.properties 2>/dev/null&"
alias krp2="cd /opt/confluent/current; nohup bin/kafka-rest-start etc/kafka-rest/kafka-rest.properties 2>/dev/null&"

alias pk="ps -aef|grep kafka"
alias topics="/opt/apache-kafka/current/bin/kafka-topics.sh  --list --zookeeper localhost:2181"

To create topics:

cd/opt/apache-kafka/current; bin/kafka-topics.sh --create --topic test --zookeeper YOUR.IP.ADDRESS.HERE:2181 --partition 1 --replication-factor 1

To list topics:

cd /opt/apache-kafka/current; bin/kafka-topics.sh --list --zookeeper YOUR.IP.ADDRESS.HERE:2181

To consume messages from a topic:

/opt/apache-kafka/current/bin/kafka-console-consumer.sh --zookeeper YOUR.IP.ADDRESS.HERE:2181 --topic MY_TOPIC 

JAXB & XML Binding Using XJC

Recently I was given an XML file with multiple schemas defining the structure of the XML file. Given my comfort level with Java technologies, I decided to use JAXB to marshall/unmarshall XML files. This blog describes the steps I took to work with XML files.

How to auto-generate Java classes that represent XML schemas using XJC

Schemas were placed in multiple sub-directories:

~/myproject/src/main/schema/schema1/*.xsd
~/myproject/src/main/schema/schema2/*.xsd

Running xjc on files in schema2 directory worked fine and Java classes were generated.
However, files in schema1 failed to auto-generate with errors like:

[ERROR] Property "Lang" is already defined. Use <jaxb:property> to resolve this conflict.
line 302 of http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd

[ERROR] The following location is relevant to the above error
line 303 of http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd

One of the schemas had a dependency on http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd. I noticed attribute "lang" was used several times in http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd. So xjc was not able to generate unique Java classes to represent these repeating attributes. The good news is you can instruct xjc to rename any portion of the schema is having problems with, using a binding customization file.

To fix the above errors, I created a file called: xhtml1-strict.xjb

<bindings xmlns="http://java.sun.com/xml/ns/jaxb"
          xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance"
          xmlns:xs="http://www.w3.org/2001/XMLSchema"
          version="2.1">
    <bindings schemaLocation="http://www.w3.org/2002/08/xhtml/xhtml1-strict.xsd" version="1.0">
        <!-- rename the value element -->
        <bindings node="//xs:attributeGroup[@name='i18n']">
            <bindings node=".//xs:attribute[@name='lang']">
                <property name="I18nLangAttribute"/>
            </bindings>
        </bindings>
        <bindings node="//xs:element[@name='bdo']//xs:complexType">
             <bindings node=".//xs:attribute[@name='lang']">
                 <property name="BdoLangAttribute"/>
             </bindings>
        </bindings>
    </bindings>
</bindings>

Run xjc again to auto-generate Java Binding classes:

cd ~noushin/workspace/ajs/src/main/schema
xjc -d ~noushin/workspace/ajs/src/main/java -b xhtml1-strict.xjb */*.xsd

After running the above commands, you should see Java packages and classes in src directory.

The following snippet shows you how to read a XML file with parent tag and parse it to get content of :

public MyDocument getMyDocument() {
    JAXBContext context;
    try {
        context = JAXBContext.newInstance(MyDocument.class);
        Unmarshaller um = context.createUnmarshaller();
        FileReader fileReader = new FileReader("/path/to/xmlfile/);
        MyDocument myDocument = (MyDocument) um.unmarshal(this.getClass().getClassLoader()
     .getResourceAsStream("mydocument.xml"));
        Metadata metadata = null;
        if (myDocument != null) {
               metadata = myDocument.getMetadata();
        }
    }
    catch (FileNotFoundException | JAXBException e) {
        // handle exception
        e.printStackTrace();
    }
}

You will get an exception during unmarshalling if your xml document parent tag is "":

javax.xml.bind.UnmarshalException: unexpected element (uri:"urn:hl7-org:docns", local:"myDocument"). Expected elements are ...

In this case, you need to add the following to your JAXB Class MyDocument:

@XmlRootElement(name = "myDocument")