This course is “Tiliting at Windmills 201.”

I’m your lecturer Don Quixote.

We will begin with a snap quiz.

You have two well-established computer systems A and B. In accordance with its original design system B makes use of a small but signifiant proportion of system A’s schema. It is proposed that system B should be enhanced, which will require accessing a few additional tables from system A’s schema.

Do you:

1. Go ahead and grab the data you need, along the lines of what is already being done?
2. First espouse the concept of a “common integration language” and then promote the idea of linking the two systems together via some to-be-announced-Real-Soon-Now “integration layer”?
3. Place a huge purchase order for the latest silver bullet product espoused in last month’s “What’s Next! A busy CIO’s guide to the latest trends in ICT” magazine, mandate that this product is now the “Enterprise Standard”, and require systems A & B should use this product for their new “Enterprise-level Interactions”?

I’ll give you a few seconds to think about it….

…time’s up.

Now: those of you who answered ‘A’…

Sigh. Did you learn nothing last year? Don’t you remember “Complification 101″? Don’t you remember the Three Principles: ‘Complicate’, ‘Obfuscate’, ‘Procrastinate’? I am disappointed in you…you’ll never get ahead in today’s enterprise with a can-do attitude like that.

OK. Who answered B?

That’s more like it! Glad to see you ‘B’s were paying attention in “Elementary Enterprise Operations 101″ last year. Remember that this alternative gives you the advantage of being able to wander around the whole organisation declaiming “Integration Layer” at every opportunity. This may not constitute a successful approach to building working software but it sure is a great way of building visibility with The Powers That Be…and what’s more important, after all, eh?

NO, Mr. “I answered ‘A’…I thought it was all about building software to do something useful”…it isn’t. Really! Maybe you should be rethinking your career path while the year is still young?

Did anybody answer C? Ah. you must be from the “Software and Sales” double degree. I see bright futures ahead for you guys. This is the way to go! Even you ‘B’s need to take notice here. Maybe even consider taking “Hype and Hysteria 204″? There’s always another silver bullet just around the corner and you don’t want to miss out on the opportunities.

Who’s left? You over there…you’re a “Software Law” student aren’t you? Kibitzing, I see…what are you looking so happy about? Wonderful, just wonderful: “Looking forward to all those failed implementations.” This, class, is what you should really be aspiring to if you really want to succeed! The software wrangling business is for mugs…why do you think that I am standing here in front of you and not sitting in front of a computer!

OK. With all this in mind, let’s begin, shall we?

Today’s lecture is entitled “A Picture Paints A Thousand Words. It Also Consumes Many Megabytes, Gives You RSI And Still Ends Up Less Useful Than A Single Line Of Code.” Please open your books to the chapter entitled “Oracle Service Bus”…

I’ve got to go now…some nice men in white coats have come to take me on holiday. How nice of them!

Seriously though…some 2004-vintage food for thought, entitled Jump off the bus, take a cab!:

…
A lot of companies have expressed the similarities between SOA and the concept of (hardware) Bus….I understand that a bus offers a “common interface infrastructure” and components can use the bus to communicate with each other, this is pretty much where the analogy stops for SOA. SOA requires very little in the middle (other than a standard communication infrastructure, that we already have for free) and is inherently point to point and asynchronous, unlike most bus architectures which require some centralized infrastructure and synchronous behavior. I would agree with the OASIS/BCM group that the only shared component in a SOA is a directory service.
…

Bear this quote in mind the next time some SOA-totin’ salesman attempts to bale up your checkbook with an order for their latest and greatest centralised server-based product.

Also worth an “honourable mention” is this oldie-but-a-goodie from InfoQ: ESB-Oriented Architectures considered harmful.

A final thought, as directed at the ‘real’ Don Quixote:

And you, Señor Don Quixote, your head is going to end up a stranger to your neck.

Concomitant with my attempts to be a good modern developer, one of the first entries on my Fings Wot I Need To Find Out More About list went something like:

Given that almost all of the actual ‘code’ in my system will be XQuery transformations, how do I unit-test those transformations with JUnit? It is obviously important that I use the weblogic API/Jars, not some external or third-party system.

Roughly six months later I am finally in a position to answer that question!

Took me a few goes (turns out that there are a number of different potential red-herring APIs to filter out and reject, including BEAs xdbc), a number of visits to other people‘s attempts and about 20 messages on the Oracle forum (gotta say I appreciated the comment “I just wanted to praise your discipline and devotion to research and investigation. Kudos!” that I picked up along the way…) but here I am.

This is the Groovy JUnit4 test class, just to prove that it can be done:

package transentia.test

import org.junit.*
import transentia.XQueryTransformer

class XQTest {
  def xqt

  @Before
   void setup() {
    xqt = new XQueryTransformer("xqueries/Messier.xq",
                                 src: new File("xml/Messier.xml"),
                                 str: 'hello, world',
                                 n: 42,
                                 b: true,
                                 dt: new Date(),
                                 f: Math.PI,
                                 nd: null as Date)
  }

  @Test
  void testTransform() {
    xqt.withString { str ->
      println str
    }
  }

  @Test
  void testSlurping() {
    xqt.withString { str ->
      def data = new XmlSlurper().parseText(str).declareNamespace(oth: "http://other")
      assert data.'oth:m'.size() == 5
      assert data.'oth:m'[3].text() == 'Scorpius'
    }
  }

  @Test
  void testParsing() {
    xqt.withString { str ->
      def data = new XmlParser().parseText(str)
      assert data.'oth:m'.size() == 5
      assert data.'oth:m'[4].text() == 'Serpens'
    }
  }
}

Pretty trivial.

This is the XQuery that is under test:

(Strangely, wordpress simply refuses to display this file without barfing. Can’t for the life of me work out why, hence the above is an image. Here’s the real file)

Notice the <destination:m?> bit? That trailing question mark is a BEA extension to the standard XQuery language: “Optional Indicator in Direct Element and Attribute Constructors.” Very useful but not widely advertised/supported (which basically accounts for why it took me 6 months to work out how to handle it).

Next is the XML file:

<?xml version="1.0" standalone="yes" ?>
<MESSIER xmlns="http://transentia">
  <M INDEX="1">
    <CONSTELLATION>Taurus</CONSTELLATION>
    <DESCRIPTION>Diffuse Nebula</DESCRIPTION>
  </M>
  <M INDEX="2">
    <CONSTELLATION>Aqarius</CONSTELLATION>
    <DESCRIPTION>Globular Cluster</DESCRIPTION>
  </M>
  <M INDEX="3">
    <CONSTELLATION>Canes Venatici</CONSTELLATION>
    <DESCRIPTION>Globular Cluster</DESCRIPTION>
  </M>
  <M INDEX="4">
    <CONSTELLATION>Scorpius</CONSTELLATION>
    <DESCRIPTION>Globular Cluster</DESCRIPTION>
  </M>
  <M INDEX="5">
    <CONSTELLATION>Serpens</CONSTELLATION>
    <DESCRIPTION>Globular Cluster</DESCRIPTION>
  </M>
  <M INDEX="6">
    <DESCRIPTION>ONLY FOR TESTING</DESCRIPTION>
  </M>
</MESSIER>

This file is just a tiny amount of fixture data for testing.

The missing <CONSTELLATION> element within the final <M> element should ‘trigger’ the XQuery engine to elide the empty <destination:m /> element that it would normally produce; thus the tests are all expecting 5 elements in the transformed XML, not 6 as in the source.

And now…without further ado…the pièce de résistance:

package transentia

import org.apache.xmlbeans.XmlObject
import org.apache.xmlbeans.XmlOptions
import org.apache.xmlbeans.XmlDateTime
import org.apache.xmlbeans.XmlBoolean
import org.apache.xmlbeans.XmlInteger
import org.apache.xmlbeans.XmlDouble
import org.apache.xmlbeans.XmlString

class XQueryTransformer {
  private final resultXML

  public XQueryTransformer(map, xQueryTransformSource) {

    XmlObject xmlObject = XmlObject.Factory.newInstance()
    XmlOptions options = new XmlOptions()

    options.setXqueryVariables(createOptions(map))

    XmlObject[] results = xmlObject.execQuery(new File(xQueryTransformSource).text, options)

    resultXML = results[0].xmlText(new XmlOptions().setSavePrettyPrint().setSavePrettyPrintIndent(2))
  }

  private createOptions(map) {
    def optionsMap = [:]
    map.each { k, v ->
      switch (v?.class) {
        case Date:
          XmlDateTime dt = XmlDateTime.Factory.newInstance();
          dt.setDateValue(v)
          optionsMap.put(k, dt)
          break
        case Boolean:
          XmlBoolean b = XmlBoolean.Factory.newInstance();
          b.set(v)
          optionsMap.put(k, b)
          break
        case [BigInteger, Integer, Long, Short, Byte, byte, short, int, long]:
          XmlInteger i = XmlInteger.Factory.newInstance();
          i.setBigIntegerValue(v as BigInteger)
          optionsMap.put(k, i)
          break
        case [BigDecimal, Float, Double, float, double]:
          XmlDouble xd = XmlDouble.Factory.newInstance();
          xd.setDoubleValue(v as double)
          optionsMap.put(k, xd)
          break
        case String:
          XmlString string = XmlString.Factory.newInstance();
          string.setStringValue(v);
          optionsMap.put(k, string)
          break
        case File:
          def txt = v.text

          // look for the prologue < ?xml ...?> processing instruction
          def m = txt =~ /(\< \?xml[^>]*\>)/
          if (m.count > 0) {
            def xmlPI = m[0][1]
            txt = """${xmlPI}
${txt[xmlPI.size()..-1]}
"""
          }
          else
            txt = "${txt}"

          // all the messing around with xml processing instruction/adding a new root above is because
          // selectChildren seems to be required. Don't know why...
          optionsMap.put(k, XmlObject.Factory.parse(txt).selectChildren("temp42", "root42")[0])
          break

        case null:
          XmlObject n = XmlObject.Factory.newInstance()
          n.setNil()
          optionsMap.put(k, n)
          break

        default:
          throw new Exception("Unhandled type: ${v.class}")
          break
      }
    }
    optionsMap
  }

  def withString(c) {
    c.call resultXML
  }
}

I’ve said it before and I’ll most likely say it again: love the Groovy switch statement!

A bit of clarification for the ‘File’ case is in order, I guess because it’s quite confusing! I found that I had to wrap a dummy root element around the original root element in order to immediately select only the single child element (the original )…this is why I need to isolate the processing instruction: the opening tag has to go between the PI and the tags. It seemed easiest to do this with simple string manipulation…

Here is the tricky bit…the various jars that need to be placed on the classpath:

oracle.nlsrtl_11.1.0\orai18n-collation.jar
oracle.xdk_11.1.0\xquery.jar
com.bea.core.xml.xmlbeans_2.0.0.0_2-5-1.jar
com.bea.core.xquery.xmlbeans-interop_1.3.0.0.jar
com.bea.core.binxml_1.3.0.0.jar
oracle.xdk_11.1.0\xml.jar
com.bea.core.xquery_1.3.0.0.jar
oracle.xdk_11.1.0\xmlparserv2.jar
com.bea.core.xquery.beaxmlbeans-interop_1.3.0.0.jar
weblogic.server.modules.xquery_10.3.3.0.jar

(Note the ordering: this is what IntelliJ X EAP worked out for me; who am I to argue…)

And to cut out any confusion, here’s what I have in my Intellij project:

C:\DEVELOPMENT\XQueryTester>tree /f /a
Folder PATH listing for volume Bugblatter
Volume serial number is 42E3-A91B
C:.
|   XQueryTester.iml
|
...
|
+---lib
|   |   com.bea.core.binxml_1.3.0.0.jar
|   |   com.bea.core.xml.xmlbeans_2.0.0.0_2-5-1.jar
|   |   com.bea.core.xquery.beaxmlbeans-interop_1.3.0.0.jar
|   |   com.bea.core.xquery.xmlbeans-interop_1.3.0.0.jar
|   |   com.bea.core.xquery_1.3.0.0.jar
|   |
|   +---modules
|   |   \---features
|   |           weblogic.server.modules.xquery_10.3.3.0.jar
|   |
|   +---oracle.nlsrtl_11.1.0
|   |       orai18n-collation.jar
|   |
|   \---oracle.xdk_11.1.0
|           xml.jar
|           xmlparserv2.jar
|           xquery.jar
|
...
|
+---src
|   \---transentia
|       |   XQueryTransformer.groovy
|       |
|       \---test
|               XQTest.groovy
|
+---xml
|       Messier.xml
|
\---xqueries
        Messier.xq

C:\DEVELOPMENT\XQueryTester>

All requisite jars can be found within the OSB 11g installation.

Update 2011 01 01; at least one person has read this and given me some feedback:

I needed to add com.bea.core.antlr_2.7.7.jar to the build path as
well. Maybe it’s because I use Eclipse and not IDEA.

Thanks, Morten!

The proof, as ‘they’ say is in the pudding…

Here’s the passing case:

Now for the failing case (with the ? taken out of the XQuery transformation):

Take a good look at the way the failed assertion is represented in the above screenshot. Something else to love about Groovy!

Hope this helps some other OSB user actually trying to embrace good software engineering practices.

The above isn’t perfect but it’s better than nothing. Use it; improve it; share it. Please let me know what you have done…

Restating something I said in the Oracle SOA Suite forums:

IMHO it is RIDICULOUS that what we are trying to achieve here is not clearly described and supported for OSB.

For something like Apache Camel or Spring Integration testing is a complete no-brainer. For OSB it seems almost impossible to do properly. For me this is a (/one of the) killer points AGAINST OSB.

Its a shame that this is not available from Oracle but then one can’t really expect them to go to all that trouble of carefully crafting a tool like OSB with maximum lock-in potential and then make the APIs openly accessible, can one?
(…and yes I do know the genesis of Apache XMLBeans…)

Ooohhhh…hope I’m not breaking any licensing rules here…don’t want to tangle with Oracle’s lawyers, after all!

I am currently working on a pretty simple systems integration task: read from a small bunch of database tables, munge the data and call a variety of webservices. Fundamentally a simpler integration problem than the exemplar for Spring Integration I worked up before. Pretty simple with Java and a framework like Spring Integration or Apache Camel; undoubtedly even simpler if I could add Groovy into the mix.

Organisation says: “Thou Shalt Use Oracle Service Bus.”

Ever heard the joke about regular expressions: Once upon a time a programmer had a problem. He decided to solve it using Regular Expressions. Then he had two problems.

Welcome to my world (with an OSB flavour) :-)

I’ll try and capture a few thoughts and experiences below.

It is very rare that I react so strongly against a technology….can’t really think of another example. I’m usually pretty gung-ho and even suffer from odd breakouts of evangelical fervour. I can even find a place for UML in the nerd-ish pantheon :-) For this product all bets are off. I’m struggling to find a silver lining in the cloud.

I hope that if I break things up into small doses, you won’t break down with sympathy tears…

Positioning

Why choose OSB vs a BPEL implementation? BPEL has an OASIS standard behind it, is the richer language, is more mature and more expressive, has better tooling, has multivendor support, can deal with state, timouts and long-running workflows…all the things that one needs in an integration setting. OSB does practically none of these things, so what’s it all about, Larry?

Ahhhh…perhaps that little ‘multivendor’ word gives us a clue? “So young to be so cynical”, I hear you say.

I have heard it said that OSB is “high level” and the alternatives like Apache Camel and Spring Integration are developers tools and as such are too “low level” to be true solutions. Sigh. I thought that old argument for CASE was put to bed long ago. It’s sad perhaps, but you still need developers to do complex development. Don’t believe me? Take a look at the average spreadsheet and then tell me how appropriate this “end-user engineering” idea is.

The task is not to find the most abstract or pretty tool but to put into the developers’ sweaty hands the toolset capable of getting into the gnarly nooks and crannies of a problem space and still come out on top. Next time you hire a carpenter or electrician or plumber, take a look into his/her toolbox. It won’t be pretty but it will be varied and capable. I’ll bet that it took the tradesperson quite a bit of training and experience to work effectively with everything in it too…there is no magic bullet.

Architectural

“Stored Procedures migrate to Session Beans” used to be the J2EE catch-call (for very good reasons, IMHO, YMMV). In my current OSB-oriented project, this seems to have become “Let’s do as much as possible in stored procedures because it’s too darned painful to even consider otherwise.”

This is really rubbing against the grain!

It’s interesting that these days, discussions regarding Systems Integration always bring up the “VETRO pattern“: Validate, Enhance, Transform, Route, Operate. This is great, as far as it goes. In my experience however VETRO is by far the easy side of SI. Prising and cajoling the requisite data out from the various niches, repositories, formats, access methods, security regimes, etc. tends to be much more tricky and is a necessary prerequisite. It’s sad, but I can’t think of a catchy management friendly phrase for all this nasty boring legwork. The term “Data Mining” is already taken, darn it! Regardless, I’m absolutely positive that no pretty picture is going to encapsulate the true reality of the SI task. I’m all for aspiring to do better but hey…even in Star Trek they sometimes need to reverse the polarity of their polaron fields in new and untested ways, and when they do they don’t draw pretty pictures but rip off the access panel to their nearest Jeffries tube, get down on their hands and knees and crawl.

In the integration community these days, it is common to tip the hat to Gregor and Hopke’s Enterprise Integration Patterns work. The book sits on every “Integration Architect”‘s shelf and provides a useful reference work. It’s very tempting to imagine architects sitting around designing systems using Star Trek-style dialog: “We’ll hook this Incoming Polling Consumer to this Aggregator, then to this Resequencer before Routing the incoming message to the outgoing Messaging Gateway.” Leaving aside the issue that this scenario and reality correspond to each other in roughly the same way that Star Trek corresponds to the NASA, let’s take a look at OSB’s support for patterns.

There’s a few: various species of polling consumer, a bit of routing, some basic message transformation and then…you’re on your own.

It is extremely cumbersome to do anything other than broad brush-stroke flows…I feel that I am being forced to spatter stored procedures, triggers, DBMS_ALERT calls and odd little support tables throughout my database, along with (as simple as possible) snippets of Java in the OSB application in a desperate attempt get OSB to do something useful.

I frequently repeat the following mantras: “OSB is just a dumb pipe” and “Let’s keep OSB as dumb as possible.”

I simply can’t bring myself contemplate the effort needed to deal with all the callouts to all the Java objects needed to deal with any sort of sophisticated API or complex processing.

I’m pretty sure that I could replace much of the mess that has grown up as a result of OSB’s pathetically limited capabilities with a couple of standard Java collection classes mixed in with a decent polling implementation such as in Apache Camel or Spring Integration.

Compare and contrast with Camel, which has direct support for about 50 of these patterns. That previous link also provides links to a nifty library of icons for them all, in various formats…very useful. Spring Integration also directly supports several of these patterns “out of the box.”

Seems to me (cynic that I am) that OSB successfully implements just two patterns: the “sell copies of Weblogic” pattern and the “extract lots of money from customers” pattern.

What goes into OSB, stays in OSB. For all but the simplest bit of work one typically needs to develop additional helper tools/applications/scripts, etc. Tools like JPA and Camel are by design flexible and make it possible to create these tools by reusing elements of the ‘main’ development path (a part of the object model, or a transformer, say). OSB does not…quite the reverse, in fact (it is very much [Oracle Weblogic] server-side-only, for example). Any ‘investment’/effort put into OSB is in reality not going towards infrastructure development but is instead just bringing into existence another closed, siloed system. An instant legacy problem. The only investment that one does with OSB is investing into Oracle’s share price. This is unacceptable. It need not be this way. We have tools available that can be used to enhance our infrastructure, not fragment it, that can work with our Objects and APIs, not make an all-out (but subtle) effort to supplant them.

Development/Engineering

OSB has an “everything is XML” approach (OK I acknowledge that it’s slowly divesting itself of this, but there’s a way to go yet before the alternatives are anything approaching useable). I like XML as much as the next guy, probably more; it’s a great interchange format, but requiring it for all internal channels as well is a bit much: one ends up touching the in-flight data (translating/extracting/recomposing) too many times. With JPA and JAX-RPC, one can in essence and after a little housekeeping do:

WebService.doSomething(DBEntity.findByPrimaryKey(pk));

The equivalent OSB (with Database JCA Adapters in place) is horrendous, simply horrendous!

Don’t get me started on the Database JCA Adapters! They are supported in OSB but cannot (yet) be created via OSB’s Eclipse-based tooling. This means that one is required to use JDeveloper 11g to create the adapter bindings appropriate for the underlying databases and then manually copy the resulting batch of files (a group of up to 5 files for each table being used) into the Eclipse project. Automate those build steps, I challenge you!

It gets worse: an OSB proxy created to service a JCA adaptor is extremely fragile: changes to a database table requires a regeneration of the adapter files (or more like a recreation from scratch…manual click, click, clickety, click). Any slight change to an adapter forces a regeneration of the associated OSB proxy…which blows away any business logic (== pretty drawing) associated with the message flow. One is then forced to recreate the flow from scratch by hand (more manual click, click, clickety, click). Ghastly. Tedious. Error prone. Nonsense. It’s in the nature of the systems integration task that changes happen often. One gets tired of (re)drawing OSB’s pretty pictures very quickly, let me testify.

One gets around this by splitting a flow into upper and lower levels: the upper level simply forwards to the lower level with as little effort as possible. The lower level flow does all the real work and is unaffected by changes to the adapter or upper level flow.

Some things might get cleaned up in the future (Oracle/BEA’s XQuery implementation is curently only partial [no module support], for instance) but I’m not holding my breath.

There’s even worse: OSB seems to confuse database polling frequency with database locking time: tell an adapter to poll every 10 secs and it seems to go away and lock all other processes out of the database for the next 10 secs, polling away in splendid isolation. This can lead to performance issues and effectively ties all adapters to a very short polling cycle. That’s a bit of a surprise and can lead to some subtle ‘issues’ arising, let me tell you…

I a great fan of these adapters…I simply love the way that they get ‘stuck’ every now and then (often when the underlying database is itself under some load). They often get ‘unstuck’ as well. Usually…sometimes…. I love going into production with systems that ‘usually’ work OK (that was sarcasm, in case you are sarcastically-challenged :-)).

OSB is completely stateless. To create something as simple as an “I have sent out a total of n messages” counter requires introducing a Java callout with an absurd amount of glue coding. Some things simply aren’t worth doing…

A good tool makes it easy to develop clear, relevant solutions close to the “problem space” without unnecessary obtuse ‘ceremony’ getting in the way. OSB is not such a tool, sadly.

You know the acronym DRY? Well, forget it!

One can try to apply a workaround and achieve a degree of DRY-ish-ness but the spaghetti flows that result aren’t worth the hassle. Here’s the issue: the process of munging a solution to fit in with OSB’s way of doing things results in a horrid unclear mess. Really. The cure ends up being worse than the disease: the resulting diagram becomes simply unintelligible…much less clear than the equivalent plain-old-text representation in Java.

Not convinced? OK, Mr./Mrs. Smarty-Pants: what values are being assigned and logged here?

To find out, you have to engage in a storm of clicking through the pretty icons and juggling Eclipe’s idiosyncratic idea of view focus control. After a while you will find that your wrists need to recover from their little workout, giving you time reflect on the fact that you’d be able to find out instantly when perusing a textual source, no clickety-click investigation needed. There’s a big difference between reading and scanning code; oftentimes the latter is what we need to do…and quickly.

On a related note, it’s hard to make the correspondence between trace logging output and the actual code. Consider, for illustration, that all service callouts have the same fixed label: “Service Callout.” It can be slightly awkward to pin down which “Service Callout” just logged something.

While it may be appealing to the naive or hopeful, Mouse-Hand Engineering…drawing pretty pictures…doesn’t really ease the development task. That’s just the nature of our complicated world…sad, but that’s life.

(You know, I was going to use the phrase “Right-Hand Engineering” but felt it was slightly too risque ;-))
(And don’t get me started [again] on UML, I’ve already thrown it a bone once in this posting :-))

Oh for the ability to “comment out” or bypass sections of a flow! It continually frustrates me that in most so-called modern languages, one cannot nest comments (something that Pascal had all worked out), but OSB drives me potty. Would the industry (yes, even this fad-ridden, silver-bullet-seeking, flavour-of-the-month driven IT industry) accept a textual language without a commenting convention, I wonder? This issue also relates to testing (see below).

Here’s a cute feature: every message that enters the system is given an unique message ID. Except that it isn’t actually unique, darn it! As a message is passed between services, each service allocates it a brand new ID. This is dumb, dumb, dumb! One is forced either to fiddle with and refer to header fields all the time, or do what I did and use a message’s database-allocated synthtic (surrogate) primary key as the message ID. Way to make it easy for us, OSB!

Error handling is broken. I know I’m being picky but something a little more than “something bad happened somewhere in some downstream flow or other” would be helpful. There are rumors of patches for this issue; good luck is all I can say.

One can’t get at system parameters/classpath resources, etc. without falling back to Java. To do anything slightly out of OSB’s (putative) “sweet spot”, one falls back to Java…so why not use a Java-based framework?

Testing. Ha ha haaaah!

How does one go about testing a drawing? This is probably the #1 question on the OSB forum.

Yes, the OSB console web application allows one to “test drive” the various componentry. Surely we can do better than hand-driven integration testing, however. Tools like citrus and SoapUI come close but don’t ring the bell for me…

Mark Smith gives an overview of the ridiculous process of creating testing stubs/proxies and the associated rewiring required when testing a proxy service’s flow. If a flow must be progressively rewired in order to be ‘tested’, is the flow actually being tested, I ask (rhetorically)?

Unit testing XQuery transformations should be simple, but isn’t…there’s no doco and a closed implementation. Writing testing flows in OSB seems ridiculous to me. Unit tests shouldn’t need a service bus and you shouldn’t be forced into integration-style testing for your atomic code units. Something smells quite wrong here.

I’ll probably pontificate on this some another time…for now, its in my “tried and failed to do something of value for the project” basket.

It may be a little churlish of me to point this out, but the XQuery editor always has some reason or other why it won’t let you do something…even dragging a dateTime onto a String:

The editor gets in the way many more times than it helps. As Roger van de Kimmenade also noted, it can get so completely messed up that it can’t do anything except show a blank screen. It can sometimes (usually when opening a file) even rewrite a transformation as it sees fit…blowing any hand-made changes away in the process. This is problematic ‘cos (as I have just pointed out) you have to do pretty much everything by hand. You Have Been Warned (hmmm…maybe Oracle should trademark this phrase. Hey: maybe I should…).

I particularly enjoy the fact that–although OSB is BEA Weblogic Server at its heart–OSB can’t readily create HTML. “Hang On a min., ” you say “isn’t HTML just a variant of XML at heart? Surely OSB should be happy with HTML!” Surely. After I asked roughly the same question on the OSB forums, Manoj Neelapu was kind enough to work up a way of doing it. The eventual solution requires a gnarly combination of two techniques, however.

Remember VETRO? In my current project, I have to process an incoming message which is defined by an XSD file included into an XSD file that is referenced by the service’s WSDL. If you followed that chain without getting confused you are doing better than OSB. Because of the include chain, OSB’s validate step won’t find the message type. The WSDL and XSDs are from a third-party vendor and so I’m loathe to touch/munge them.

This means that OSB is only capable of ‘…ETRO.’ Quality stuff, this!

Deployment is ‘strange’ as well. Took me a while to find the up-to-date doco telling you how to script the creation of an OSB domain from scratch. There are a few helpful posts out there telling you what to script after you get this beastie but so far this in in the same basket as XQuery unit testing. Seems to me that “Thou shalt not apply your existing deployment plan to a new domain.” Not if you want to keep your sanity, anyway.

Other

In no particular order:

• Eclipse IDE console’s “scroll lock” doesn’t work. You search through a long log trace looking for something and then–just as the line appears–the console refreshes, leaving you staring at the latest line to be added. Good grief but this is frustrating! I like Eclipse, have used it lots over the years (by choice, note!) but in this incarnation it is simply hateful, darn it!
• From the sublime to the ridiculous: sometimes the console doesn’t actually deign to show anything at all. Many times I hear: “Bob. A message has gone missing again.” I now know that 9 times out of 10 what has actually happened is that the Eclipse console is blocking the server as it is trying to send debugging to System.out (I’m not directly using System.out please note…I’m not that naive). A quick keypress into the window unfreezes everything and Lo! I have miraculously found the missing message. It’s strange but even miracles can get old, fast.
• JDeveloper periodically corrupts a project. One may work on a project for a week and then BAM! it fails to load up in the morning (this may be a local SOE issue but its still frustrating and I don’t get hassles from any other apps).
• There exists a fairly small user community. The Oracle forums are really not that active or useful. Apologies to those members out that that have tried their best to help me out…this comment is not intended to be an indictment of you guys (I AM grateful, really, I AM!). I am just trying to identify issues caused by community size & experience.
• OSB’s Eclipse tooling tries to deploy all the projects it can find…even if they are closed. This can lead to weird situations. You Have Been Warned…
• It’s not just the older versions, either. In the latest update, Eclipse seems to take ages to realise that a change has happened and that a deployment refresh is needed…and then sometimes fails to see that a redeploy has happened.
• …and what has happened to Weblogic? Even a simple change to a datasource’s configuration now requires a reboot. Shocking!
• The weblogic log browser is…’idiosyncratic.’ You can be eagerly following a chain of logging messages…clicking from one page to the next…homing in on the cause of all life’s woes…feeling the expectation of a debugging session drawing to a close welling up inside…and then bloody thing will decide to show page 0 again. Just for shits and giggles, it’ll be so confused that you will have to start a new web browser session. You quickly learn to munge the parameters displayed in the address bar, not that there’s anything wrong with that, of course…

My colleague David has introduced me to a term that sums all this up nicely: complification. To complify something is “To dramatically increase the difficulty of something for no good or apparent reason.”

Of course, I am an ingenious guy and I can pull some sort of solution to all of these things out of my expansive…err…mind. AFWSE is also my friend in many situations (but see my earlier comments regarding the size and vitality of the user community).

I am sure that you, dear browser, also pride yourself on your ingenuity and proven ability to wrestle a recalcitrant universe into shape.

This, Of course, is also the trouble! Ego comes riding into town…

My (rhetorical) question is: should we have to be so ingenious? Especially when many of the ‘issues’ are those that OSB in and of itself causes?

So here we are: back to the title…I can’t help but imagine that somewhere within Oracle there is a poor, downtrodden group afraid to tell King Larry that OSB just doesn’t make sense.

Guys, let me try. I’ll be the plucky little boy from the crowd (I hope that remember your Hans Christian Anderson fairy tale?) that doesn’t understand he should be following the party line…

Larry, listen up! In my (not-so-)humble-opinion there are much better ways to skin the integration cat that what I see here. OSB isn’t a worthy member of your product lineup. Don’t lead your customers down this particular garden path. The last decade has seen some good steady quiet progress in areas such as testing, interoperability and frameworks (to name but a few), please don’t set this industry back to the pre-2000s with this regressive, locked-down, locked-in, half-baked ‘thing.’ Please!

Phew! Said it. I Feel much better now!

Lest you think that I am losing it, I’m not the only person saying this sort of thing.

You know…the story doesn’t really say what happened to that plucky little boy from the crowd. Maybe he was showered in riches for saving the King’s honor. More likely, he was beheaded for embarrasing the high-and-mighty.

[edit]
Some almost immediate–and pretty much equally jaundiced and cynical (although slightly more generic)–reaction.

Thank you – a rather excellent way to start my day :-)

Now let me also share something. You may or may not agree. Oracle makes a lot (LOT) of money by sending in the consultants.

The consultants get called in when the local team comes up with zip because it’s so damned hard to get the Oracle tools to work.

Consultants are charged out at $2k / day and become an on-going income stream as the client needs lots of refresh (code change) activity.

It serves the likes of the big consulting houses to have internally trained specialists who know nothing else but these ghastly tools.

In fact some outside consultants cash in big time by spending all their time focusing on nothing but – discovering all the tips and tricks from the Oracle paid guys.

I think this has been going on since I first entered the industry and it’s a great model. Not for the industry nor the profession, but for money making monsters of which Oracle is one of the best.

There, my take…take it or leave it, but it’s ultimate raison d’etre of the Closed source movement and some clients just gravitate to it…

I’ll leave the sender anonymous…no sense in taking other people’s career down alongside my own ;-)

I’ve been known to say: “Don’t start a Java application without Spring. Ever.”

The primary reason for this is versatility. I can and have used Spring to:

• Develop various species of plain ‘ole web applications
• Develop webflow-based web applications
• Develop reporting applications incorporating eclipse BIRT
• Develop Swing-based desktop applications (using TeeChart)
• Develop batch processing systems
• Helped develop XML pipeline processing systems (alongside Stylus Studio‘s XML processing framework)
• Develop system integration processing flows
• …and probably a few more bits and pieces…

The Spring-oriented skills I have built up have let me work in a substantially standardized way regardless of the situation.

That’s pretty darned good, if you ask me!

Is this same story true of the obnoxious OSB? No.
Is it true for any implementation of the J(2)EE spec.? No. From any vendor? No.
Is it true for other frameworks like JBoss Seam. No.

Anyone know of any other framework out there offering similar flexibility, ‘cos I’m darned if I do.

The Spring framework isn’t standing still, either…it’s the foundation for both Grails and Griffon. As I have adopted these technologies, my productivity has increased greatly. I am “standing on the backs of Giants.” This is a Really Good Thing.

Let’s add a modicum of “Systems Thinking” here. Rather than wasting time training a multitude of staff in a multitude of (typically short-lived) siloed technologies…some of which are honestly horrid…it makes much more sense to build a solid core of experience with a solid core technology.

Whoops! Here I go advocating a standards-based ICT world…again.

‘Standards.’ “Systems Thinking.” How naive am I!

[...edit...]
Received an email from Manning the publishers a while back:

I know that this is not a scientific measure of ‘goodness’ (if one were even possible), but it speaks volumes to me…

Today was “first-look” day for the users of the system that I am currently working on.

After observing the users start in on the system and then watching them hack around the screens for a while, it occurred to me that I was seeing (at least, probably) three different types of behavior.

My background as a chemist came to the fore at this point and it occurred to me that I was actually seeing three states of matter at play.

The first state: solid. This type of user basically stayed where they were put. They logged in, typed what they were told to type and seemed happy to do nothing else.

Next were the liquids: these guys showed a bit more ability to ‘flow’ around the system than their colleagues. They found more ‘features’ and generally got more involved in things.

Then came the gases. These whizzed all over the place, escaping the confines of the system and generally finding leaks and holes all over the place. In the long run, these users were clearly going to be the most reactive and energetic. They were most likely to get the best return on their efforts.

Quite a nice analogy :-)

Now here’s the clincher.

As one develops and tests software one needs all three of these states at some point.

You can maximize your testing if you can arrange your user testing so that it…and your user communities…follows nature’s path: solid – liquid – gas.

You’ll find that the solid testers will pick up the easy, common, 80% of impactful bugs and –importantly–will be happy doing so. At the end of solid testing, you won’t have the perfect system, but you will have one that satisfies a good proportion of your user community’s needs.

Follow up with the liquids…they will help you trap some of those more grungy edge cases that always seems to be forgotten about. Again, the quality of the system will be increased and you will have another user community that is nicely ‘behind’ you.

Time to let the gasses roam free! They’ll get into the last remaining nooks and crannies and the result will be a good stable system. The gasses will be happy, knowing that they have contributed in a meaningful way without being waylaid by–to them–trivial and easy/’obvious’ issues.

Like all analogies, this one breaks down fairly quickly: in the chemical universe, there are so many more gasses than solids ‘out there.’ For testing, the reverse is true.

This reminds me of how systems like submarines are pressure tested with nitrogen, so there must be some truth in it, somewhere…

Sadly, today we didn’t follow nature’s “true path” to testing nirvana!

Still: epiphanies have to come some time…usually as a result of some quantity of pain.

For sure, I’ll be better able to express what I want from a testing process next time. We live and learn!

Testing philosophy, what next!

Eat your hearts out Kant, Heidegger, et. al…

[...edit...]
“Time to let the gasses roam free!” is not a reference to bodily functions! Shame on you, gentle reader (you know who you are)!

I came across my copy of the magazine when I was clearing one of my bedrooms out a few days ago (after heavy rain seeped in through the flashing that sits between slab and wall and started the carpet rotting. Sigh.).

The article was entitled “Under the J2EE Umbrella.” No prizes for guessing what the article was about :-)

Neither SEA nor ‘Software’ exist any longer (wound up in 2005, I believe) but sadly, I have no idea of the copyright/ownership of these golden words so I can’t actually publish the article.

Still, this should help me recall doing this, when I am old(er) and (more) forgetful.

I (currently) remember writing other articles for ‘Software’…don’t know if I still have copies of those lying around…

Transentia is Ten years old today.

Transentia is a baby of the dot-com era. We started out wrangling Java and are coming into the second decade having adopted Groovy as a technology platform.

This is a good time to reflect on the origin of the name…

It all started while I was a member of the “Business Development” team at DSTC.

After a session wrestling with a group of research staff (collective noun: a recalcitrant?) a colleague (who shall remain nameless) stormed into my office and yelled “I don’t know what we are doing, but it sure as hell isn’t technology transfer.”

He came up with the phrase “Technology Transferention”, explaining: “It sounds good and can impress people who don’t know any better, but it doesn’t actually mean a darned thing. Exactly right for what we do.” That phrase got a fair bit of mileage :-)

While casting around for a company name I recalled this event.

I decided that “Transferention Technologies” was a bit too much of a mouthful and munged it around a bit; thus the name Transentia was born.

(The alternative was a play on another phrase the team used when confronted with an absurd situation: “the sky is green, the trees are blue.” That got a lot of airplay too, but I couldn’t quite work it into a good company name: “Green Sky Technologies” sounds a bit too vomit-induc{ed/ing}.)

Nowadays, there’s a “Land of Transentia” in some online game, a (Japanese?) musician, and a cybersquatter is sitting on ‘transentia.com’ (and can continue to sit, as far as I am concerned. I deliberately chose an australian .com.au domain…we are a much more exclusive club :-))

Date: Friday 9 October 2009
Presenter: Tim Rubin
Venue: Augustine’s Mansions Private Dining Rooms

First (as always), I’d like to say “Thank You” to Oracle for inviting me along to this event. Even if they actually wanted “Dale Gately”, whoever that is (the email just happened to come to me via my domain’s default drop-box). I guess that I should say “Thank You, Dale” as well! Never look a gift-horse in the mouth :-)

I’m not sure that there was actually any ‘Strategy’, “Best Practices” OR “Lessons Learned” discussed in the presentation.

The guy sitting next to me summarized his response to Tim’s spiel as: “Please don’t buy into what our competitors are telling you…give us time to work something out…” I guess that’s about right.

I loved the opening Larry Ellison quote on Cloud Computing:

The interesting thing about cloud computing is that we’ve redefined
cloud computing to include everything that we already do. I can’t think of anything
that isn’t cloud computing with all of these announcements. The computer industry
is the only industry that is more fashion-driven than women’s fashion. Maybe I’m
an idiot, but I have no idea what anyone is talking about. What is it? It’s complete
gibberish. It’s insane. When is this idiocy going to stop?

We’ll make cloud computing announcements. I’m not going to fight this thing. But I
don’t understand what we would do differently in the light of cloud computing other
than change the wording of some of our ads. That’s my view.

The participants (all CIOs and CTOs…whoever Dale Gately is, he’s clearly more important than I am) were…a confused lot, it seems to me. This is what I picked up on:

• We are going to ‘do’ “Cloud Computing.”
• We don’t really know what it is.
• We don’t really know why we should be doing it.
• We are sceptical that it will buy us anything. At all. In fact, all we can see are a myriad of problems.
• Still, everyone tells us that we should ‘do’ “Cloud Computing”, so we will.

The event’s organiser/host made an interesting comment something along the lines of “In Brisbane, we always meet with a healthy ‘show-me-the-money’ attitude. Much more so than in the other states.” Decoded, this means: “Damn! You luddites are giving me a hard time again. I didn’t get this from Sydney or Melbourne.”

One question was repeated a couple of times with various phrasing whch I’ll summarize as: “If we do ‘do’ “Cloud Computing”, will we still have to do all that yukky Systems Integration?” I’d have hoped that a roomful of CTOs and CIOs would have a bit more of a grasp on things than they apparently have, but that’s life. To his credit, the presenter’s answer was quite correct: “Of course! You’ll still need to undertake a good round of Business Process Reengineering to get the benefits of this (or any new) technology.”

A few interesting tidbits/notes:

In a typical data center, the Uptime Institute found that up to 30% of servers are ‘dead’ (ie with <10% utilization).

According to Tim, the major benefit of Cloud Computing is reduced time/cost to provision a system.

According the to Oracle Cloud Computing survey:

• ~24% of respondents saw cloud computing as a way of saving (unspecified) costs
• ~20% saw (unspecified) ‘Architectural’ benefits
• ~18% considered Cloud Computing to be strategic, in some unspecified way
• ~15% considered that Cloud Computing offered a revenue-raising opportunity

Tim gently made fun of these statistics by pointing out how they seem to show a herd mentality following a fad (my interpretation of his comments).

Tim pointed out that Cloud Computing may actually cost more than self-hosting if you provision it properly and account for it fully (without all the funny intra-organizational ‘tweaks’ that are often so beloved).

The fact that Cloud Computing introduces a whole new swag of failure modes, critical infrastructure points and training issues was pretty well glossed over (and would probably have taken all the allotted time anyway!).

SLAs and legal issues were almost completely ignored.

IMHO the SLA is probably the most critical aspect of Cloud Computing adoption, and here’s why:

Let’s say I buy into a cloud on the basis that my cloud will be provisioned for a ‘normal’ load, but can be scaled on-demand to enable me to handle the bursts that come along every now and then. Everyone in an organization does their timesheet on a Friday morning, so that’s a good illustration. The timesheet system probably runs at 0.01% of maximum load during the rest of the week and so constitutes ‘waste’, according to the Cloud Computing credo. Consider that the cloud vendor is only going to make money by ensuring that his infrastructure is as maximally profitable as possible and so wants to handle the load with as little equipment/effort (= as highly shared between customers) as possible. How do you write an SLA that says: “I need my peak load traffic to be successfully dealt with, REGARDLESS of the traffic being offered at the same time by any of your other customers.” All the other customers have their timesheets to fill in, remember; their peak is Friday morning as well. The only way a cloud vendor could make an appropriate guarantee would be to provision a data center in exactly the same way as we do now: for maximal utilization and accept that the normal usage is going to be very wasteful. This will fly in the face of the vendor’s business case and make their life almost prohibitively expensive.

So: you want to be in the cloud? Expect to achieve worse service (in some unspecified/unspecifiable way). You may not get bitten, but you WILL need to decide how painful a bite you can potentially stand.

Tim makes the very good statement: “Any application that gives you a competitive advantage should NOT be in the cloud.” Good advice.

Overall, I came away feeling ‘unenlightened’ and a little miffed at the shallowness of the coverage.

I didn’t even get a desert at the end of the meal ‘cos the caterers had run out :-(

It has a tag on its ‘behind’ saying “Hi! I’m Tux The Linux Penguin” The badge has the heading “Penguin Power.”

The badge also uses the quaint spelling ‘LinuX’, which proves how old it is: who would spell it that way nowadays!

I was presenting a 2-day tutorial session on Linux (RedHat 5.2). I have mentioned another trophy before.

This T-shirt is a fairly recent acquisition, from the Energex/SPARQ TCE Project that I have written about before:

This one is a lot older:

It was given to me by one of the Enhydra guys in December 1999 at the Linux/Open Source Bazaar, Jacob K. Javits Centre, New York. I was presenting a 2-day tutorial session on Linux (RedHat 5.2).

My “body clock” was exactly 12 hours out of alignment but it was a lot of fun. For example: the organisers arranged for us to have free, unfettered access to the famous FAO Schwartz Toy Store for a night…the perfect entertainment for a bunch of geeks! Another choice moment: watching the audience practically boo Corel CEO Michael Cowpland (I think; might have been RedHat CEO Bob Young….my jet-lagged memory is a little hazy here but it keeps pushing a fedora at me…) offstage for daring to suggest that Linux would eventually have to be commercialised and become a money-making product before it could actually “rule the world.” Richard Stallman and Eric Raymond were practically having fits, as were many of the faithful in the audience. How times have changed…

This was pretty much the first ‘real’ Transentia engagement, so this T-shirt really does bring back a favourite memory.