Matter faced during integration project

Wednesday, November 25, 2009

Using OWSM and BPEL PM behind a proxy

We use OWSM and BPEL PM to consumme services outside our network.
Nothing to fancy so far, we had to modify the JVM settings and add the following parameter:

Dhttp.proxySet=true
Dhttp.proxyHost = proxy_server
Dhttp.proxyPort = listen_port
Dhttp.nonproxyHost = *localhost|*domain.com

in our case this was:
Dhttp.proxySet=true
Dhttp.proxyHost = www-proxy.outsourcing.com
Dhttp.proxyPort = 80
Dhttp.nonproxyHost = *outsourcing.com

Do not forget to add your localhost as non proxy.
The JVM settings can be accessed through the OC4 console or by editing the file

ORACLE_HOME/opmn/conf/opmn.xml

Then we add to consumme an https service and the add the settings:
Dhttps.proxySet=true
Dhttps.proxyHost = www-proxy.outsourcing.com
Dhttps.proxyPort = 80
Dhttps.nonproxyHost = *outsourcing.com

The case has to be exactly that and only a restart of the OC4J instance is not enough- You have to restart the JVM.
Also note that this cannot be changed without a restart.

Tuesday, October 20, 2009

Consuming Web Services with Jakarta Commons HttpClient

Consuming web services with Jakarta Commons HttpClient can be useful to reuse all the API provided by HttpClient.
The example below query a web service that requires HTTP authentication. We could also use that mechanism for NTLM or proxy authentication.
More generally, for all authentication made in the HTTP layer rather than the SOAP layer, HttpClient is a good candidate.

To run this code you need the library of HttpClient and HttpCore. I used version 4.0 and 4.0.1

you also need commons codec: I used version 1.4

To consume Web Services protected with NTLM (sic). You can refer to this page: http://www.luigidragone.com/networking/ntlm.html

So here's the code:


package org.apache.http.examples.client;

import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.message.BasicHeader;
import org.apache.http.util.EntityUtils;

/**
* A simple example that uses HttpClient to execute an HTTP request against a
* target site that requires user authentication.
*/
public class ClientAuthentication {

private static final String XML_DATA =
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>" + "<soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"> "
+ "<soap:Body xmlns:ns1=\"http://xmlns.oracle.com/SensorBPEL\">" + "<ns1:SensorBPELProcessRequest><ns1:input>toto</ns1:input></ns1:SensorBPELProcessRequest>"
+ "</soap:Body></soap:Envelope>";

public static void main(String[] args) throws Exception {
DefaultHttpClient httpclient = new DefaultHttpClient();

httpclient.getCredentialsProvider().setCredentials(new AuthScope("ch0016188", 8888), new UsernamePasswordCredentials("frederic_agneray", "pass"));

HttpPost httpPost = new HttpPost("http://ch0016188:8888/gateway/services/SID0003006");
httpPost.setHeader(new BasicHeader("Content-Type", "text/xml;charset=UTF-8"));
httpPost.setHeader(new BasicHeader("SOAPAction", "process"));
StringEntity s = new StringEntity(XML_DATA, "UTF-8");
httpPost.setEntity(s);

System.out.println("executing request" + httpPost.getRequestLine());
HttpResponse response = httpclient.execute(httpPost);
HttpEntity entity = response.getEntity();

System.out.println("----------------------------------------");
System.out.println(response.getStatusLine());
if (entity != null) {
System.out.println("Response content length: " + entity.getContentLength());
System.out.println(EntityUtils.toString(response.getEntity()));
}
if (entity != null) {
entity.consumeContent();
}

// When HttpClient instance is no longer needed,
// shut down the connection manager to ensure
// immediate deallocation of all system resources
httpclient.getConnectionManager().shutdown();
}
}

Wednesday, September 2, 2009

Using OWSM on multi-domain Active Directory

Lately we had to use Oracle Web Service Manager to Authenticate and Authorize web service user against our Active Directory.
That is pretty simple if your users are in a single domain, however our directory is separated in 4 domains: EAME, APAC, AMR and B2B domains.

So we tried to figure out a way to authenticate against all those domains.
1. The AD authentication step of OWSM does not allow multiple domain search
2. The LDAP authentication can be used to authenticate against a AD. And the LDAP query can search against multiple directory.

3. In the same way, AD authorization cannot be used against multiple domain
4. Then LDAP authorization cannot be used because AD directory use the attribute "member" instead of "uniqueMember" for LDAP directory.

So to fix this issue we had to build a custome step within OWSM. We decompile LDAPAccessControlStep and built ILayerLdapAccessControlStep. It is the same piece fo code but "uniqueMember" is replaced by "member".

ETL Process Through Email Attachment

Last June, one of my post was "A BPEL process that saves e-mail attachment".
The aim was to try to insert data within the ERP via an e-mail attachment.
For example, to load the "daily mineral figures" of South-Africa, the South-Africa manager was sending a mail to etl.load@mycompany.com with an attached file (an xml file). This attachment would be processed and its containing data inserted into the ERP.
Security was insured by a x.509 certificate signature.

This process is very convenient as sending a mail is an easy process and can be done with a limited access to the web.

We tried to enable this process with "Oracle BPEL Process Manager", but the following issues arise:
1. Nearly impossible to authenticate the certificate signature
2. Difficult to retrieve the attachment with the correct encoding
3. Difficult to retrieve more than one file
4. Impossible to move mail in other folder after processing
5. To deal with all the operations, lots of Java embedded code had to be written.
6. Mail with attachment bigger than 1MB could not be processed
This is not exhaustive, but the feature for BPEL PM that enable to start a process by receiving a mail, is not meant to deal with complex mail content.

So we decided to change our option and use the Java Mail API.
We used Quartz to schedule our process and the process was written in Java.
The operations are:
1. Call the IMAP server every 5 minutes to check mail box for incoming mail
2. Retrieve the attachments and copy them to the correct location
3. Send e-mail if an error occurred or if the attachment was correctly copied
4. Move the mail from "Inbox" to "Processed", "Error" or "Discard" folder

The process is hosted within our J2EE server and is a reliable and robust solution.

Post a message if you want the source code of this process.

Tuesday, July 7, 2009

Web Services standard security implementation overview


Before Web Services can be used, security issues need to be addressed and security requirements such as authentication, access control, non-repudiation, data integrity, and privacy must be tackled by the underlying security technology.

Authentication: ensure the business partner is who he pretends to be

Access Control: check the partner is allowed to call this action on those data

Non-repudiation: the partner cannot withdraw his position after the order has been sent

Data Integrity: data received cannot have been modified by a third party

Privacy: no unauthorized person can see secure data

Auditing: any action on any data can be trace back

As Web Services are meant to be consumed by systems and not directly by users, authentication can be performed by either: providing the credentials of the user currently logged in the application, or to connect as a machine without providing user credential.

1. If providing the logged in user credentials: the system authenticates the user and provides the user credentials to the back-end Web Service. All calls to the Web Service are then made in the name of the user and access is allowed or denied depending on the logged in user rights.



Figure 1 - Delegating authentication to the Web Service provider

2. If connecting as a machine: the front-end system is responsible to authenticate the user. It calls the underlying Web Service without providing the logged-in user credential.

Thus security is insured by authenticating the calling machine. The machine can authenticate itself with a certificate or by providing a generic username and password. Although using a certificate is more secure, a username and password is commonly used, in the same way as an application accesses a database with encrypted security credentials.

Figure 2 - Authenticate to the Web Service provider as a machine

From the point of view of the Web Service, whether the authentication credentials come from an end-user or a machine does not make much difference. In the first case, the machine provides the user authentication credential provided to it, in the other case the machine provides its own authentication credential.

From there, authentication strategies are:

  • Provide the username and password of the user through HTTP header. This is the same process as when a user authenticate to a web site
  • Use WS-Security standard feature. WS-Security is part of the WS-* specifications proposed by OASIS which is the organization that defines Web Service standards. It describes a set of SOAP header extensions and allow authentication with:
    • Kerberos tickets: a standard and widely use authentication protocol published by the MIT
    • Username and/or password Token Profile: similar to sending username and password with http header, but this time using WS-Security header, thus allowing an end-point to end-point authentication as opposed to point to point authentication as for HTTP header.
    • SAML Token Profile: a specification that tries to solve Single Sign-On issues for Web Services
    • PKI certificate Token Profile: The most common authentication practice for Web Services. It is also the one use for credit card transaction over the internet. It consist of using a Public Key Infrastructure certificates which works as described in the diagram above:



Figure 3 - PKI certificate usage

User repository:

Then, the authentication itself must be performed against a user repository. The most standard user repositories are LDAP like directory tree such as Microsoft Active Directory. It allows administrators to simply manage a list of users and is a standard for any tool that needs to perform authentication.

Access control is to use to define the rights of the users concerning all actions and data of the application. Typically it defines:

  • Roles: like “Administrator”, “Clerk”, etc.
  • Actions: like “Create New Employee”, “Read Employee”, “Change Employee Salary”
  • Roles to Actions mapping: like
    • “Administrators” can “Create New Employee” and “Change Employee Salary”
    • “Clerk” can “Read Employee”

However, access control can be defined according to action or can be defined according to action and type of data. For example, the user “China Manager” can do the action “Modify Employee” only on the data “China Employee”. On the data “India Employee”, he can only do the action “Read Employee”.

The implementation of the first type of Access Control List, only according to action, is done by creating User Groups in the user repository: Users are positioned into User Group and all actions are associated to a User Group. So if the user does not belong to the associated User Group it cannot perform the action independently of the data manipulated.

The implementation of the second type of Access Control List can either be done by creating as many actions as there are types of data. It means that an action “Modify Geneva Employee” will be created and this action can only manipulate “Geneva employee”. This present the drawback of having to create a full set of action each time a new “region” is created.

Or the data can be flagged to belong to one “region” and all Users have a role according to a region. He could then only do a certain type of action on the data from a region.

This type of Access Control List implies that the data can be associated to a region, and that the user can also be positioned against this region. However Users and Access Control List usually resides in the user repository and data resides in the database, thus the concept of region need to exist and be consistent in both the database and the user repository.

Then the options are:

  • Regions need to be maintained between user repository and database. This present the risk of desynchronized data and maintenance needs to be setup either automatically or manually.
  • Database is used as User Repository. This implies that the user list is kept in a proprietary format, thus all tools that are used with LDAP protocol in mind cannot be used or need adapter.
  • A tool is used to synchronize or consolidate the data between the database and the repository. It can be done either through synchronization or through virtualization
    • Synchronization means that processes will regularly fetch the data from multiple user repositories (LDAP directory, Active Directory, Proprietary Database or any other format). Those processes will bring back the data to a single user repository, often an LDAP server. Thus only one repository contains all the data concerning the Users, their Access Control List and the “region” they belong to.
    • Virtualization means that a virtual directory is set in place and when called, it will actually launch processes to go and fetch the information from multiple repositories. It will then present the information to the requester as if it came from a single repository. With this solution, no information is duplicated unlike with solution above.

Access control standardization with XACML:

XACML stands for Extensible Access Control Markup Language, and its primary goal is to standardize access control language in XML syntax. A standard access control language results in lower costs because there is no need to develop an application-specific access control language or write the access control policy in multiple languages. Plus, system administrators need to understand only one language. With XACML, it is also possible to compose access control policies from the ones created by different parties.

Non-repudiation is the concept of ensuring that a party in a dispute cannot repudiate, or refute the validity of a statement or contract. So in case of dispute, a Trusted Third Party is necessary to guarantee that the refuting party did actually commit to the agreement.

On the digital side, the Trusted Third Party is often a Certificate Authority such as VeriSign or SwissSign. They provide Public Key Infrastructure Certificate to both party and, in case of dispute, can be requested to validate the signatures on the signed documents.

Certificate Authority also create, distribute and revoke certificate so that both party involve in a transaction are certain that the Public Key Infrastructure Certificates used for the transaction are valid before the transaction starts.

Figure 4 - Non-repudiation with PKI certificate

An alternative to the Public Key Infrastructure model is a Web of Trust which is a concept used by Open Source cryptography model like PGP, GnuPG and other OpenPGP compatible system.

In this model, digital signature and business agreement are exchanged in a face to face meeting before any transaction starts.

The Web of Trust presents the advantage to be unaffected by third party company failure

Data integrity, in terms of network security, is the insurance that no one can have modified the message between the sending and the reception. This is typically to prevent a “man in the middle” (MITM) attack where someone relays messages between communicators, making them believe that they are talking directly to each other over a private connection when in fact the entire conversation is controlled by the attacker.

Measures taken to ensure integrity and prevent MITM attack include, controlling the physical environment of networked terminals and servers, restricting access to data, and maintaining rigorous authentication and encryption practices.

Usage of Public Key Infrastructure certificates are commonly used to insure Data Integrity. It is quite all the time used in conjunction with encryption even though data integrity does not need Encryption as the aim is only to insure that the data has not been modified.

The typical usage to PKI certificate for Data Integrity and Data Privacy is explained on the diagram below:



Figure 5 - Data integrity and data privacy using PKI certificates

The key points from a Data Integrity point-of-view are step 1 - Bob sign the message, and step 8 - Alice verifies the hash of Bob’s signature.

A hash is like a digital "fingerprint" that uniquely identity a document. By recomposing the hash of the document, Alice cans checks:

  • It is Bob that has signed the message with his private key (or someone that has stolen Bob private key, which is normally impossible)
  • The hash reproduce by Alice is similar to the hash signed by Bob. If someone has changed the document, Alice could detect the modification by making a hash out of the document and check the difference.

In a way, Data integrity can also be insured through encryption. By making sure that nobody else could read the message, encryption insure that nobody can modify it.

Privacy is insured through a secure network (a VPN or an intranet) or through encryption. The 2 main encryption technologies for Web Services are:

  • SSL/TLS (Secure Socket Layer/Transport Level Security) for point-to-point security. Similar to HTTPS, SSL is commonly use for credit card transaction over the internet and present a level of security hat is sufficient for most transaction. However, Web service application topologies include all sorts of devices, PCs, proxies, demilitarized zones, gateways, etc. Consequently, many intermediaries come between two communicating parties. SSL/TLS may secure the path between any two, but not from one end to the other.

Furthermore, SSL/TLS encrypt the whole message and does not discern confidential to public data.

  • XML encryption allows encrypting only a part of the XML message, thus it allows end-point to end-point cryptography and allow better performance as it encrypts only part of the message.

Both SSL/TLS and XML encryption can use PKI certificate has explained on diagram Figure 5 - Data integrity and data privacy using PKI certificates. Then the public key of the certificate is used for encrypting the message and only the private key can decrypt it.

Auditing is to insure all actions that have been made were authorized and no abuse has been performed. Application log strategy needs to be defined to enabler log analyzer tool to search through the log file. This is usually automatically defined when choosing the tools.

Tuesday, June 23, 2009

Error while processing xpath expression *** the reason is FOTY0001: type error

Funny issue last time, might be useful to share it.
Worked with JDev on a BPEL process and opened a transform activity. Then before it loads the "Source Part" and the "Target Part", I clicked on the "ok" button to move out of there.

But JDev saved the transform activity with empty "Source Part" and "Target Part".
As the loading of source and target take quite some time because the .xsl is fairly big (on my PC anyway) it is easy to press the "ok" button before they are loaded. (as shown on the picture below).


Afterward, the process compiled fine, but a runtime exception appear at runtime complaining about the XSL transformation. The error message was not clear at all, and it took me some time to figure out what happened.
the complete error message was:

XPath expression failed to execute.
Error while processing xpath expression, the expression is "ora:processXSLT('OrderTransformation.xsl',bpws:getVariableData('inputVariable'))", the reason is FOTY0001: type error.
Please verify the xpath query.

Tuesday, June 16, 2009

Secure a web application in Oracle OC4J and reuse BPEL console credentials

This week, the team had to deploy a web application on OC4J. However we wanted to re-use the security settings of the BPEL console:


1. Created a web application with the following web.xml:


<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/j2ee">
   <display-name>webAccess</display-name>
   
   <security-constraint>
      <display-name>ILayer Scheduler</display-name>
      <web-resource-collection>
         <web-resource-name>web-resource-name</web-resource-name>
         <url-pattern>/*</url-pattern>
      </web-resource-collection>
      <auth-constraint>
         <role-name>toto-administrators</role-name>
      </auth-constraint>
   </security-constraint>

   <login-config>
      <auth-method>BASIC</auth-method>
      <realm-name>Ilayer FTP Application</realm-name>
   </login-config>

   <security-role>
      <description>Dummy Toto Administrators</description>
      <role-name>toto-administrators</role-name>
   </security-role>
</web-app>


2. Modified the file <$oracle_home>/j2ee/home/config/system-jazn-data.xml to add a role:


<role>
   <name>toto-administrators</name>
   <display-name>Scheduler Admin Role</display-name>
   <description>Admin role for scheduled application</description>
   <members>
      <member>
         <type>user</type>
         <name>oc4jadmin</name>
      </member>
   </members>
</role>


3. Then had to restart OC4J for the settings to be taken into account

Friday, June 12, 2009

Consuming a web service over https in JAVA

Had to consume a web service over https.
Here are the links:

how to build a web service over HTTPS (did not follow it, though)
http://www.pankaj-k.net/WSOverSSL/WSOverSSL-HOWTO.html

and how to consume it:
http://www.javaworld.com/javaworld/javatips/jw-javatip96.html

How to test a proxy with telnet

Lately I tried to test a proxy with telnet.
Here are the step I followed:

1. Open a telnet session
telnet www-proxy.oracleoutsourcing.com 80
2. Make the http request to the server you want to try
GET http://www.nfl.com HTTP/1.1
3. Give the host
host:www.nfl.com
(press enter twice)

This allow me to test our proxy from an environment we can only access through telnet.

Friday, May 29, 2009

A BPEL process that saves e-mail attachment

BPEL process that receive e-mails, get the attachment and save it somewhere in the disk.

The process looks like that:




Create a process that receives mail

follow the steps here:
http://blogs.bpel-people.com/2007/01/using-email-to-initiate-bpel-process.html

Save the attachment

There are 3 options:
1. Use a file adapter: you cannot easily define the file name, it always has to be the same. Furthermore there are some issues with the base64 encoding.

2. Use another web service to save the file (not a BPEL one): if you need to do some complex processing of the file it might be your only option.

3. Save the file with a "Java embedding" activities: this is the option I choose as I can easily set the name of the file + don't have to worry about the encoding.

The assign attachment is:



The java embedding is:



<bpelx:exec name="saveFile" language="java" version="1.5">
<![CDATA[java.io.Writer output = null;
try{
String name = (String)getVariableData("attachmentName");
int pos = name.lastIndexOf("name=");
String newName = name.substring(pos+6, name.length()-1);

java.io.File file = new java.io.File("c:\\" + newName);
output = new java.io.FileWriter(file);
output.write((String)getVariableData("attachment"));
} catch (Exception e) {
addAuditTrailEntry(e);
} finally {
try {
output.close();
} catch (java.io.IOException e) {
addAuditTrailEntry(e);
}
}]]>
</bpelx:exec>



http://forums.oracle.com/forums/thread.jspa?threadID=602700