Download WSDL and include/imported XSD files

To reduce the external build dependencies of my maven web application project, in particular the jaxws-maven-plugin part of my project,  I stored the WSDL file and all its dependencies in my project and configured the plugin wsdlUrls to point to them. As shown below:

<configuration>
   <wsdlUrls>
         <wsdlUrl>${basedir}/src/main/resources/wsdl/SDMXQuery/SDMXQuery_1.wsdl</wsdlUrl>
   </wsdlUrls>
</configuration>

I downloaded all the files with SoapUI (Version 5.0.0) by creating a new SoapUI project with the URL of the WSDL as follows:

Then I clicked on the WSDL in the newly created project (See arrow 1 below), changed to the "WSDL Content" tab, and then pressed the small button (See arrow 2 below) with the tool tip: "Export the entire WSDL and include/imported files to a local directory". 

Then I chose the directory within my project.

After that, everything was nicely stored in the directory and could be configured in the POM file of my project as shown above.

jdbc driver with oracle wallets

The processes of converting a jdbc connection as configured in most java application can be confusing. These notes should provide some help and perhaps a better overview.

The first task is to create a wallet. Below, I created two sets of credentials for the same user; RIAD_JAVA. I created two because I wanted to try two types of configurations. One which uses the database connection string and the other which uses a TNS alias.

#first create the wallet 

mkstore -wrl t-riad-wallet -create

#create first of two credentials for user RIAD_JAVA. This will be used for configuration, which needs a tnsname.ora file.

mkstore -wrl t-riad-wallet -createCredential triad RIAD_JAVA secret2015$

#create second set of credentials also for RIAD_JAVA but this time with part of the connection string taken for the java programms externalized properties file. The configuration won't need a tnsname.ora file.

mkstore -wrl t-riad-wallet -createCredential t-riad-dwh-db.tadnet.net:1521/triad.tst.tadnet.net         RIAD_JAVA secret2015$

Before continuing, let me give an over view of what will be shown. 

1. Configuration Using thin client (jdbc:oracle:thin:)

   a.) with Database Configuration String (@hostname:port:service)
   b.) with TNS alias (@anyalias which can be found in tnsname.ora)

2. Configuration Using oci (jdbc:oracle:oci:)

1. Configuration Using thin client (jdbc:oracle:thin:) 

The following jars are needed:

ojdbc6-11.3.0.jar, oraclepki.jar and ucp.jar

a.) with Database Configuration String

Set up the Java application with the following connection string. This should correspond to what was used while creating the second set of credentials above.

jdbc:oracle:thin:/@t-riad-dwh-db.tadnet.net:1521/triad.tst.tadnet.net

Program must be started with the following JVM parameter:

oracle.net.wallet_location


Example:

java -cp .:/global/riad-app/tomcat/lib/ojdbc6-11.3.0.jar:./jlib/oraclepki.jar:./ucp.jar 

-Doracle.net.wallet_location=/tmp/wallet/t-riad-wallet  WalletTest

Note: no tnsname.ora file is needed only the a copy of the wallet created above.

b.) with TNS alias

The above mentioned jar will also be need here. This time the connection string is as follows, which is what was used to create the first set of credentials above.

jdbc:oracle:thin:/@triad

Program must be started with the following JVM parameters:

oracle.net.wallet_location
oracle.net.tns_admin

The following tnsnames.ora will be needed and should exist where oracle.net.wallet_location shown above points to.


 triad =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(Host = t-riad-dwh-db.tadnet.net)(Port = 1521))
    )
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = triad.tst.tadnet.net)
    )
  )

Example call:

java -cp .:./ojdbc6-11.3.0.jar:./jlib/oraclepki.jar:./ucp.jar -Doracle.net.wallet_location=/tmp/wallet/t-riad-wallet  -Doracle.net.tns_admin=/tmp/wallet WalletTest

2. Configuration Using oci (jdbc:oracle:oci:)

Here the jdbc connection string looks as follows:

jdbc:oracle:oci:/@triad


Before starting the programm the following enviroment variables are needed:

ORACLE_HOME=/opt/oracle/product/11.2.0;export ORACLE_HOME

LD_LIBRARY_PATH=/opt/oracle/product/11.2.0/lib:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH

Also a tnsname.ora file

and a sqlnet.ora file are need.

Unified Functional Testing from Hewlett Packard behind a NTLM proxy server.

A colleague of mine wanted to work with Hewlett Packard's UFT/QTP version 11 behind a proxy server, which only supported NTLM. The secret to making it work, because UTF version 11 doesn't support NTLM, was to install cntlm (version 0.92.3) from sourceforge between the test suite and the proxy server on the machine, which was hosting the HP program. After setting up the default cntlm.ini file with the proper proxy server host and port information, user name and password, cntlm was then started as follows:

cntlm -c cntlm.ini -u myuser@mydomain

When UTF asked for the proxy information, localhost and 3128, the port which cntlm was configure to listen on, were entered in the screen below:

Creating a maven plugin with java 5 annotations rather than javadoc tags

Actually, creating a plugin with the Maven Plugin Tools, is something, thanks to the plugins themselves, which is relatively trivial; however, because it took me a long time to get my plugin working properly with Java 5 annotations rather than Javadoc Tags, I decided to write it down.

I must confess that, sadly enough, Maven has never been a strength of mine so that it was no surprise that my plugin didn't immediately function properly. The problem I faced was in the passing of the parameters to my Mojo during execution of my newly created plugin.

When converted to Java 5 annotation, the class "outputDirectory" attribute, in the default created Mojo,  was always null even though I had the proper configuration in my host project's pom.xml

<configuration>                     
    <outputDirectory>C:\Users\wnpr\Downloads</outputDirectory>         
 </configuration> 

To make sure that we all have the same starting point let me begin, by showing how I used the Maven Integration for Eclipse plugin, M2E, to create a new Maven project as shown below.

Then I selected the "maven-archetype-mojo" as a  temple for my new project,

  and finally gave my plugin a groupId and artefactId.

What follows is a view of the Maven project from within eclipse. As you can see, the project already contains a Mojo class, called MyMojo, which was created from me

Every plugin needs a plugin.xml file, which is generated by the maven-plugin-plugin and which can be found the the META-INF\maven folder of the generated jar file. By default, maven-plugin-plugin generates the plugin.xml file from the Mojo Javadoc Tags found in the Java code. These tags look like this:

/** 

 * Goal which touches a timestamp file. * 

 * @goal touch 

 * 

 * @phase process-sources 

 */

If you, as I did, would rather use Java 5 annotations, such as:

@Mojo( name = "touch" )

public class MyMojo

    extends AbstractMojo {

   @Parameter(defaultValue = "c:\\", required=true)

    public File outputDirectory;

   @Parameter(defaultValue = "Hello World!!!!", required=true)

    public String greeting;

.
.
}

for you plugin.xml generation, then you will have to rely on another plugin; namely: maven-plugin-annotations.

To do this you will have to add an additional dependency, to your plugin project's pom.xml.

<!-- dependencies to annotations -->
<dependency>
   <groupId>org.apache.maven.plugin-tools</groupId>
   <artifactId>maven-plugin-annotations</artifactId>
   <version>3.4</version>
    <scope>provided</scope>
</dependency>    


and override Maven core's default-descriptor execution phase to process-classes as follows:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-plugin-plugin</artifactId>
        <version>3.4</version>
        <executions>
          <execution>
            <id>default-descriptor</id>
            <phase>process-classes</phase>
          </execution>
          <!-- if you want to generate help goal -->
          <!-- 
          <execution>
            <id>help-goal</id>
            <goals>
              <goal>helpmojo</goal>
            </goals>
          </execution>
           -->
        </executions>
      </plugin>
      </plugins>
    </build>

It wasn't until this last change that the @Parameter annotated attributes of my Mojo were finally set properly during execution of my plugin.  Both of these additions are well described here; however, when I read them for the first time, I wasn't smart enough to understand them; as often is the case, and needed to waste a day before I finally came to my senses.

If you get a message like this:

Error extracting plugin descriptor: 'Goal: touch already exists in the plugin descriptor for prefix: junk
[ERROR] Existing implementation is: org.wmmnpr.junk_maven_plugin.MyMojo

then be sure to delete the old Javadoc Tags.

Good luck!

Converting Spring boot standalone application to a web application

In the "Producing a SOAP web service" chapter of the "Getting Started" Guide from Spring, one creates a web service, which eventually runs as a standalone application. In the section Make the application executable, one reads:

"Although it is possible to package this service as a traditional WAR file for deployment to an external application server, the simpler approach demonstrated below creates a standalone application."

This blog is about the necessary changes actually needed to make the referred to web service available in a application server.

A good starting point, is to use maven archetype for a web application. This will provide the necessary structure needed for a web  application i.e. WEB-INF folder.

First add a ContextLoaderListener to your web.xml in the WEB-INF folder in order to instantiated a Spring container which will be shared by all Servlets and Filters. Because it was desirable to use an annotation style of configurations, an AnnotationConfigurationWebApplicationContext was instantiated (see contextClass context-param below) instead of the default XmlWebApplicationContext.

<web-app ..>

 <context-param>
     <param-name>contextClass</param-name>
     <param-value>org.springframework.web.context.support.AnnotationConfigWebApplicationContext
  </param-value>
 </context-param>
 <context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>hello</param-value>
 </context-param>
 <!-- Creates the Spring Container shared by all Servlets and Filters -->
 <listener>
  <listener-class>org.springframework.web.context.ContextLoaderListener
 </listener-class>

 </listener>
   <!-- take especial notice of the name of this servlet -->
    <servlet>
        <servlet-name>webservice</servlet-name>
        <servlet-class>org.springframework.ws.transport.http.MessageDispatcherServlet</servlet-class>
     <init-param>
       <param-name>transformWsdlLocations</param-name>
       <param-value>true</param-value>
     </init-param> 
    </servlet>

    <servlet-mapping>
        <servlet-name>webservice</servlet-name>
        <url-pattern>/*</url-pattern>
    </servlet-mapping>

</web-app>

For the webservice, a MessageDispatcherServlet was also configured in the web.xml file as shown above. Make sure that the "transformWsdlLocations" attribute of the web service Servlet is set to true during instantiation.

The MessageDispatcherServlet, because it has not been otherwise indicated in the web.xml file, expects a  configuration file called  "webservice-servlet.xml" (servlet-name plus "-servlet"), which should be located in the WEB-INF folder. The contents of the file are shown below.

<beans>
  <sws:annotation-driven/>

   <sws:dynamic-wsdl id="countries" portTypeName="CountriesPort"locationUri="/ws/countries">
       <sws:xsd location="/WEB-INF/classes/countries.xsd" />
   </sws:dynamic-wsdl>

</beans>

With those changes done, it's now time to change our focus to the WebServiceConfig.java class also described in the guide. Here, the method for the bean definition "messageDispatcherServlet" should commented out as it's no longer needed. Everything else can stay the same.

The last thing that is needed is to add the countries.xsd file, as created in the "Getting Started Guide", to the application as a resource so that it can be found by the "countriesSchema" bean, which is instantiated in the WebServiceConfig class.

Once these steps have been completed, it should possible to deploy the war to an application server like Tomcat or run it using the maven "tomcat6-maven-plugin" plugin.

The code is available here as a zip.

Good luck.

Parsing boring old CSV files with Java

It sometimes seems as if Java (JEE) skipped specifying a robust mechanism for handling CSV files. For instance, there is nothing like a schema description (XSD or DTD), which exists for XML files. Perhaps it was just too trivial, but I've found despite the existence of more "robust" formats like XML and JSON, there is still a lot of CSV files being used, which probably has a lot to do with the ubiquitness of Excel. But whatever the reason it would be nice to have a schema description of a CSV file, which could be used as sort of a interface description to bridge the gap between developers and business analysts.

I discovered something from Digital Preservation, which has created a very nice looking schema language for the CSV format; however, unfortunately, it was not well suited for Java. The library is written in Scala and comes with a Java bridge; however, at the time of writing this blog, the bridge didn't allow one to get much information about the exact errors and their positions because the result of the parsing was returned as a long, loosely formatted string.

There are other libraries, like Jackson CSV and Super CSV from sourceforge.net, which seem less sophisticated or ambitious than Digital Preservation's one; however, I wasn't impressed by them because the problem of bridging the gap between technical and business people doesn't seem to have been addressed. The building up of the CSV structure is largely done in Java code at run-time and is for a non Java person almost unintelligible.

So, for my last task, which involved parsing 3 types of CSV files, I resorted to using the apache common's  CSV parser and Java Annotations. What I did was to create a Pojo with all the columns contained in the CSV. Then, I invented some simple annotations which I used to decorated the Pojo with and which the business analyst could read and understand without much difficulty.

Below is a small example of one of these Pojos. You might laugh that I passed around a Java source code file as a CSV  file description but it worked well and people didn't seem to have problems finding what was relevant for them and understanding it. At first, it might look unintelligible; however, but with a little patience one can read through it.

Each Java class defines a CSV line, here one with 4 columns however in reality it was up to 396 columns. The lines' column order was described by the annotation @CsvColumnOrder (with Java reflection it's not possible to determine the order in which Field are declared; therefore, the extra annotation). The expected type of each column is described within the class definition as class attribute with further annotations like CsvChecked, CsvIsinCode, etc...

DbtPojo.java 

@CsvColumnOrder({
"IsActive",
"ExternalCode_ISIN",
"AmountIssued",
"PaymentDate"
});

public class DbtPojo {

  @CsvChecked(regex="^[YN]{1}", required=true)
   public String IsActive;

  @CsvIsinCode
  @CsvUnique 
  public String ExternalCode_ISIN;

  @CsvNumber(decimal = true)
  public String AmountIssued;

  @CsvDate
  public String PaymentDate;


}

Using reflection, it is possible during parsing to get information about relevant Pojo's fields by looking for the presence of  annotations. For instance, if the second column of a line has been returned by the Apache parser, then from the CsvColumnOrder annotation, one knows which field of the Pojo needs to be checked, namely "ExternalCode_ISIN".   With a reference to the Java Field, one can check for the presence of certain annotations by calling getAnnotation(Annotation class reference). If the annotation is present, here CsvIsinCode, one can react appropriately for a ISIN code.

I've included the code as a zip for those who are interested in more detail.

The reader should not be mislead into thinking that the CSV values from the files are being parsed into an instance of a Pojo as described above. In fact, the classes are never even instantiated. They are only used for their annotations and field definitions; that is for meta data purposes.

Authentication using simple Active Directory bind with Spring

Recently I wrote a web application, which had to be made secure.  I wanted to allow the users to use their normal Active Directory passwords and not to have to create one especially for the application. The application was also meant to be available to only to a selected group of users and not everyone with a valid AD account.

Rather than trying to get an Active Directory "application" account, which would have been nice as it would have allowed me to search for roles but which would have taken forever to get because of bureaucratic reasons, I decided to use a simple bind to the AD with the entered password and username to authenticate the users.

The problem was that just binding would have allowed everyone with a valid AD account to access the web application, which is what I didn't want. To limited the users, I needed a different way to verify whether or not they were authorized to use the application and this I did by storing their roles in the application's database.

To test whether or not this approach would work and to determine the connection details, I used Apache Directory Studio as shown below.

To add this approach into the web application, I used Spring Security, which I setup using the name space configuration (XML with specific Spring tags).

    <s:authentication-manager> 

        <s:authentication-provider ref="ldapAuthenticationProvider"/>   

    </s:authentication-manager>

    <bean id="ldapAuthenticationProvider"    class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">

        <constructor-arg>

            <bean class="eu.ecb.csdb.gateway.security.UsernameBindAuthenticator">

                <constructor-arg ref="contextSource" /> 

                <!-- just keep the base class happy -->

                <property name="userDnPatterns">

                     <list>

                        <value>sAMAccountName={0},OU=Standard User,OU=Users and Groups</value>

                    </list>

                </property>     

            </bean>

        </constructor-arg>

        <constructor-arg>

            <bean class="eu.ecb.csdb.gateway.security.DatabaseAuthoritiesPopulator">  

            </bean>

        </constructor-arg>

    </bean>

First, I configured my own authentication provider (UsernameBindAuthenticator) , which binds to AD using the entered password and username as shown here:

public class UsernameBindAuthenticator extends AbstractLdapAuthenticator {

/*...*/

    public DirContextOperations authenticate(Authentication authentication) {
  
        DirContextAdapter result = null;

        String username = authentication.getName();
        String password = (String)authentication.getCredentials();
  
        String domainUsername = domain + "\\" + username;

        DirContext ctx = null;

        try {

            ctx = getContextSource().getContext(domainUsername, password);

            // Check for password policy control

            PasswordPolicyControl ppolicy = PasswordPolicyControlExtractor.extractControl(ctx); 

            //bind was successful if we got here.

            result = new DirContextAdapter();

        } finally {
            LdapUtils.closeContext(ctx);
        }

        return result;              

    }
/*...*/

}

Then, I added my own LdapAuthoritiesPopulator (DatabaseAuthoritiesPopulator) , which queries the database to see if the user has any roles.

public class DatabaseAuthoritiesPopulator implements LdapAuthoritiesPopulator {

/*...*/

    public Collection<? extends GrantedAuthority> getGrantedAuthorities(DirContextOperations userData, String username) {

  
        logger.debug(String.format("Getting roles for %s", username));

        List<GrantedAuthority> result = new ArrayList<GrantedAuthority>();

        try {

            final String query = String.format("select roles from USERS where USERNAME = '%s'", username);     

            final String roles = this.jdbcTemplate.queryForObject(query, String.class);
    
            String splitRoles[] = null;
            if(roles != null && (splitRoles = roles.split(";")).length > 0){       
                for(String strRole : splitRoles){
                    result.add(new CsdbAuthority(strRole));
                }
            }

        }catch(Exception ex){
            logger.debug(ex);
        }

        logger.debug(String.format("%s roles found for %s", (result == null ? 0 : result.size()), username));
    
        return result;           

    }

/*...*/

}

The CsdbAuthority is nothing but an implementation of the GrantedAuthority interface.

So if the bind was successful, then I knew that the user was a legitimate AD user and then I looked in the application's database to see if the user was authorized to use the application.

At this point, you might be asking yourself, wouldn't it have been easier just to use the FilterBasedLdapUserSearch and DefaultLdapAuthoritiesPopulator classes from Spring?  First of all no because I didn't have any roles in LDAP to search for. As mentioned I wanted a quick solution and not one where I would need to do months of paper work to get the roles added to AD; this all took place within a very bureaucratic government agency. Secondly no, because I found the implementation of the FilterBasedLdapUserSearch  confusing, quirky and poorly documented and I figured that rather than messing around for hours trying to configure the class it would be easier just to write my own.