Friday, March 23, 2012

OBIEE 11g in-place upgrade issue with configuration lock

In-place upgrade within OBIEE 11 versions (from .3 to .5 o r .6 or from .5 to .6) can be a whole lot quicker than out-of-place ones but are also more interesting in terms of potential issues arising. After two uneventful in-place upgrades from 11.1.1.5 to 11.1.1.6 I had a more challenging third upgrade with some unexpected errors.

If you look at the official upgrade guide, all went fine up to item 6: "Upgrading system components". Executing upgradenonj2eeapp.sh, however, threw an error while re-deploying:
./upgradenonj2eeapp.sh -oracleInstance /bishiphome/Middleware/instances/myinstance/ -adminHost localhost -adminPort 7001 -adminUsername weblogic

Command failed: NonJ2EEManagement Applicationdeployment failed.

Looking into the provision.log (situated in MW_HOME/instances/myinstance/diagnostics/logs/OPMN/opmn) read as follows:
[Deployer:149163]The domain edit lock is owned by another session in non-exclusive mode - this deployment operation requires exclusive access to the edit lock and hence cannot proceed. If you are using "Automatically Aquire Lock and Activate Changes" in the console, then the lock will expire shortly so retry this operation.

I ran the upgradenonj2eeapp.sh a second time to see what it would do:
SEVERE: Command failed:
oracle.as.config.ProvisionException: Exception while unregistering the instance
    at oracle.as.config.impl.OracleASInstanceImpl.unregister(OracleASInstanceImpl.java:290)
    at oracle.as.config.impl.OracleASInstanceImpl.unregister(OracleASInstanceImpl.java:250)
    at oracle.as.config.provisioner.commands.UnregisterInstanceCommand.execute(UnregisterInstanceCommand.java:34)
    at oracle.as.config.provisioner.InstallerCmdLine.run(InstallerCmdLine.java:146)
    at oracle.as.config.provisioner.InstallerCmdLine.main(InstallerCmdLine.java:46)
Caused by: oracle.as.config.impl.SimpleProvisionException: Failed to unregister instance instance1. It does not exist on the adminserver.

So, a real no-go situation. Luckily, the OBIEE EMG already had a post which contained part of the solution. Root cause of the initial error was that there was a lock on the configuration I hadn't seen:


"Release Configuration" removed the immediate error source. Next step was re-running upgradenonj2eeapp.sh again to see what would happen and obviously the error "Failed to unregister instance instance1. It does not exist on the adminserver." still persisted since initially the first half of the script had run through and un-registered my instance. So off to some opmnctl commands to undo the damage.

1.) Deploy the instance again:
./opmnctl redeploy -oracleInstance /bishiphome/Middleware/instances/myinstance -adminHost localhost -adminPort 7001 -adminUsername weblogic

Command requires login to weblogic admin server (localhost):
  Username: weblogic
  Password:

Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl localhost:7001 -username weblogic -name NonJ2EEManagement -source /bishiphome/Middleware/Oracle_BI1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit

Task 1 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Task 1 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Target state: redeploy completed on Server AdminServer

Done
Command succeeded.

2.) Register the instance again:
./opmnctl registerinstance

Command requires login to weblogic admin server (localhost):
  Username: weblogic
  Password:

Registering instance
Command succeeded.

 After these steps, I was able to run the actual shell for the upgrade process again:

./upgradenonj2eeapp.sh -oracleInstance /bishiphome/Middleware/instances/instance1/ -adminHost localhost -adminPort 7001 -adminUsername weblogic
Unregister instance...

Command requires login to weblogic admin server (localhost):
Username: weblogic
Password:

Unregistering instance
Command succeeded.
redeploy NonJ2EEManagement.ear...

Command requires login to weblogic admin server (localhost):
Username: weblogic
Password:

Redeploying NonJ2EEManagement Application...weblogic.Deployer invoked with options:  -adminurl localhost:7001 -username weblogic -name NonJ2EEManagement -source /bishiphome/Middleware/Oracle_BI1/opmn/applications/NonJ2EEManagement.ear -redeploy -upload -noexit

Task 2 initiated: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Task 2 completed: [Deployer:149026]deploy application NonJ2EEManagement [Version=11.1.1] on AdminServer.
Target state: redeploy completed on Server AdminServer

Done
Command succeeded.
re-register instance...

Command requires login to weblogic admin server (localhost):
Username: weblogic
Password:

Registering instance
Command succeeded.
Successfully upgraded NonJ2EEManagement.ear and the registeration.
 One of those little things that can ruin your evening...

Cheers!

2 comments:

  1. Thanks Buddy ....You indeed saved a lot of time in debugging the issue.

    Thanks,
    Ram

    ReplyDelete
  2. you've spared me many hours of work, thank you very much!

    ReplyDelete