Monday, 24 May 2010

ADF and WLS JNDI configuration errors

Another blog under the theme "let's document it so we don't get caught out again", which can also be tagged as "let's document it so others don't get caught out too".

In our Oracle JDeveloper 11g ADF applications at runtime, on occasion we encountered one of the following errors when running our applications under Oracle's WebLogic Server 10.3.2. I've included the beginning stack trace to assist Google searches into this page:

<Utils><buildFacesMessage> ADF: Adding the following JSF error message: JNDI failure. Unable to lookup Data Source at context jdbc/SomeJndi
oracle.jbo.DMLException: JBO-27200: JNDI failure. Unable to lookup Data Source at context jdbc/SomeJndi
at oracle.jbo.server.DBTransactionImpl.lookupDataSource(DBTransactionImpl.java:1414)
at oracle.jbo.server.DBTransactionImpl2.connectToDataSource(DBTransactionImpl2.java:309)
at oracle.jbo.common.ampool.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:203)
at oracle.jbo.server.ApplicationPoolMessageHandler.doPoolConnect(ApplicationPoolMessageHandler.java:553)
at oracle.jbo.server.ApplicationPoolMessageHandler.doPoolMessage(ApplicationPoolMessageHandler.java:409)
at oracle.jbo.server.ApplicationModuleImpl.doPoolMessage(ApplicationModuleImpl.java:8534)
at oracle.jbo.common.ampool.ApplicationPoolImpl.sendPoolMessage(ApplicationPoolImpl.java:4392)
at oracle.jbo.common.ampool.ApplicationPoolImpl.prepareApplicationModule(ApplicationPoolImpl.java:2388)
at oracle.jbo.common.ampool.ApplicationPoolImpl.doCheckout(ApplicationPoolImpl.java:2204)
at oracle.jbo.common.ampool.ApplicationPoolImpl.useApplicationModule(ApplicationPoolImpl.java:3088)
at oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:460)
at oracle.jbo.http.HttpSessionCookieImpl.useApplicationModule(HttpSessionCookieImpl.java:234)
at oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:431)
at oracle.jbo.common.ampool.SessionCookieImpl.useApplicationModule(SessionCookieImpl.java:426)
at oracle.adf.model.bc4j.DCJboDataControl.initializeApplicationModule(DCJboDataControl.java:513)
at oracle.adf.model.bc4j.DCJboDataControl.getApplicationModule(DCJboDataControl.java:856)
..or..
Caused by: javax.naming.NameNotFoundException: Unable to resolve 'jdbc.SomeJndi'. Resolved 'jdbc'; remaining name 'SomeJndi'
at weblogic.jndi.internal.BasicNamingNode.newNameNotFoundException(BasicNamingNode.java:1139)
at weblogic.jndi.internal.BasicNamingNode.lookupHere(BasicNamingNode.java:252)
at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:182)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:206)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:214)
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:393)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
..or..
<Utils><buildFacesMessage> ADF: Adding the following JSF error message: Unable to resolve 'jdbc.SomeJndi'. Resolved 'jdbc'
javax.naming.NameNotFoundException: Unable to resolve 'jdbc.SomeJndi'. Resolved 'jdbc'; remaining name 'SomeJndi'
at weblogic.jndi.internal.BasicNamingNode.newNameNotFoundException(BasicNamingNode.java:1139)
at weblogic.jndi.internal.BasicNamingNode.lookupHere(BasicNamingNode.java:252)
at weblogic.jndi.internal.ServerNamingNode.lookupHere(ServerNamingNode.java:182)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:206)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:214)
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:393)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
All of these errors occur because of a misconfiguration of a JNDI in one or more of the following locations:

1) the datasource within your WebLogic Server
2) an Application Module's JDBC Data Source within your main ADF application
3) an Application Module's JDBC Data Source within a Bounded Task Flow added to your application via an ADF Library
4) an Application Module's property jbo.server.internal_connection within your main application
5) an Application Module's property jbo.server.internal_connection within a Bounded Task Flow added to your application via an ADF Library

As readers know the JNDI typically takes the form "jdbc/SomeJndi".

Essentially the JNDI for the data source in #1 must be consistent with #2 and #3 otherwise the JNDI lookup fails. In addition if you make use of the jbo.server.internal_connection Application Module property, you'll require a second datasource for #1 that is consistent with that specified in #4 and #5.

This may seem simple, but during a recent development we suffered repeated mistakes where the JNDI was misconfigured causing us large amounts of grief when we thought we had it "sorted". Take the following error message, see if you can spot the issue:

Caused by: javax.naming.NameNotFoundException: Unable to resolve 'jdbc.SomeJndi '. Resolved 'jdbc'; remaining name 'SomeJndi '

Somewhat oddly this error message converts the JNDI such that the forward slashes are replaced with a fullstop. However that's a red herring. Can you see the real error?

If you've looked closely you'll note the JNDI name has an additional space character at the end, something very difficult to visually detect. This resulted in the error above.

For us this particular misconfiguration happened in the worst of places too. Rather than a JNDI used by our main ADF application which would have fallen over straight away, it was used by a secondary ADF Library Bounded Task Flow Application Module's jbo.server.internal_connection setting. As such the application when running under no stress would work fine, but only when the specific Bounded Task Flow was called, and only when the specific Bounded Task Flow was under stress and needed to passivate to the ps_txn table did we see ADF throw the above errors. Before we realized what was going on, a large flapping red herring was that the application would occasionally throw:
JBO-28030: Could not insert row into table PS_TXN, collection id 24,026, persistent id 1" followed by "JBO-27200: JNDI failure. Unable to lookup Data Source at context jdbc/SomeJndi "
...followed by one of the JNDI connections above. But we could see that records were inserted into the ps_txn table regardless of the previous error, and via our own testing, logging into the associated database account specified for the jbo.server.internal_connection, the db user had all the correct database privileges for working with the ps_txn table. Eventually we tracked it back to the misconfigured JNDI, such that the main application with the correct JNDI was happily inserting into PS_TXN, but the secondary Bounded Task Flow JNDI was intermittently causing this issue.

Lesson learned. Don't get your JNDIs wrong.

4 comments:

Simon Haslam said...

Thanks Chris - a very useful post.

It's interesting about your particular JNDI problem only being visible when passivation occurs including the BTF in an ADF Library. There's a common practice of unticking "Enable Application Module Pooling" during early testing (as recommended by Steve M & others) to test passivation on every page request, but that's not so obvious/easy when the AM is in a library (I don't know how you'd do that - you don't want two versions of the same library, though maybe it's part of the library testing?). Whilst that wouldn't have fixed the JNDI naming issue, it would have at least made it fail consistently (and intermittent bugs are usually the hardest to fix). Perhaps we need to come up with some sort of technique/best practice for this?

Chris Muir said...

Hi Simon

Agreed, the challenge under 11g is we're now much more dependent on our BTF programmers to get stuff right. In our case our JMeter stress tests brought the issue to the fore much earlier (thus all my recent JMeter stress tests posts), and was consistently reproducible, which helped resolve the issue much earlier (or at least it hadn't hit production yet!) One alternative key benefit with the stress tests is it brings all sorts of issues beyond this one to the fore, providing much better test coverage IMHO.

Cheers,

CM.

Juan Camilo Ruiz said...

Hi Chris,

I had a very similar error the other day but my application didn´t have any taskflows, the error was related to the definition of my DS on the AM and wls.

do you define the jdbc/someDS on both AM and WLS?

In my case I had to end up defining it as java:comp/env/jdbc/someDS on my App module

Then as jdbc.someDS on the WLS console.

Is it the same in your case?

Juan C.

Chris Muir said...

Hi Juan

With apologies on the slow reply, just returning from holidays.

Ya, certainly the error can occur for other reasons, I'm just attempting to document one issue that got us.

We do indeed define jdbc/someDS in both the AM and WLS. As you've also documented, JDev 11g has a short cut where you can use one of the predefined jdbc URLs such as java:comp/env/jdbc/someDS, that derives to jdbc/someDS once deployed to WLS. The rules on matching jdbc URLs still apply.

Cheers,

CM.