Directory Services

Using sed to clean up an LDIF file for import #Oracle #Identity #UNIX

I needed to import a group of users, into Oracle Internet Directory (OID) with attributes in a variety of backend data stores. I used Oracle Virtual Directory to virtualize the data stores into a single ldap view. I used the OVD adapter configuration to specify which attributes I wanted returned. I then exported using the export control from Apache Directory Studio. This resulted in an ldif file containing all of the records I needed with attributes. There were a few additional attributes as a result of using OVD that I now had to deal with.

I ended up with an ldif file that contained a lot of records like this:

dn: cn=Babs Jensen@ACME.GOV,ou=temp_user_load
objectclass: inetOrgPerson
objectclass: organizationalPerson
objectclass: person
objectclass: top
cn: 1234556677@ACME.GOV
cn: Babs.Jensen@ACME.gov
cn: Jensen, Babs
sn: Jensen
givenName: Babs
mail: BABS.JENSEN@ACME.GOV
vdejoindn: ou=acmeinfo_temp:cn=JENSEN\,BABS,ou=acmeinfo_temp
vdejoindn: AD_temp:CN=babs.jensen@ACME.GOV,OU=locations,OU=park,ou=ad_t
emp,dc=acme,dc=local
fascnDecoded: 1234567890987654321
guid: ABcdedghi1234567890
ssn: 12345678

…

Note: With the SED command you can make changes directly to the source file but I am creating a new target file with each change I can make so that I can always revert back if the command doesn’t work exactly the way I want it to.

I wanted to get rid of lines that don’t start with an attribute name (In my case I am free to get rid of lines that carry over into the second line … YMMV)

I also wanted to specifically wanted to get rid of all lines that start with “vdejoindn:” and there are also some vdejoindn lines that overrun onto a second line that won’t beremoved if I use sed to remove lines with the pattern matching vdejoindn:.

So, first I want to remove all lines that don’t contain a colon. This removes the overrun lines but also all blank lines.

$ sed ‘/:/!d’ input.ldif > tmp.ldif

this keeps the lines with a colon.

But now we don’t have breaks between the records

$ sed ‘s/^dn:/\n&/g’ tmp.ldif > tmp2.ldif

Ok, now I want to get rid of the lines that have “vdejoindn:”.

$ sed ‘/vdejoindn:/d’ tmp2.ldif > tmp3.ldif

Now at some point I ended up with “^M” at the end of each file … I don’t know if this is because I opened with VIM in Windows before moving to Linux … I am going to assume so but either way in this instance I want to remove these characters.

$ dos2unix tmp3.ldif > tmp4.ldif

Alright, Now, for me to import this into Oracle Internet Directory (OID) I’ll need to add the “changetype” directive. I am going to add the string “changetype: add” on a new line after each line with “ou=temp_user_load:” which is the temporary suffix I used in this export.

$ sed ‘/ou=temp_user_load/ a\changetype: add’ tmp4.ldif > tmp5.ldif

Now, should be the last step, prior to importing, is to correct the entries “DN” attribute. Essentially, we need to replace “ou=temp_user_load” with the correct suffix for where these users will be created.

$ sed ‘s/ou=temp_user_load/cn=Users,o=icam,dc=acme,dc=local/g’ tmp5.ldif > tmp6.ldif

At this point my ldif file (“tmp6.ldif”) is ready to import into my directory. You can use the ldapmodify command or since I am using OID you can use bulkload (which is recommended for large record sets).

Troubleshooting errors starting #OID #11g #Oracle #Identity #LDAP

I have an Oracle Identity 11g environment running on VirtualBox 4.0. This is a development environment that I use to test out various installations and configurations. I noticed the other day that I wasn’t able to start the Oracle Internet Directory (OID) instance.

When I checked the log file I can see that I am not able to connect to the Database. By the way, the log that is referenced doesn’t show anything of value. The log that actually contained the error is called: oidmon-0000.log

According to ora-code.com ora-28000 the error means that the user account that is connecting to the database ‘ODS’ is locked.

ORA-28000:	the account is locked
Cause:	The user has entered wrong password consequently for maximum number of times specified by the user’s profile parameter FAILED_LOGIN_ATTEMPTS, or the DBA has locked the account
Action:	Wait for PASSWORD_LOCK_TIME or contact DBA

It’s typically trivial to unlock an account from the sqlplus command line

So, we should be good now. I will try to start the process again.

But now my log shows

So, now I am getting an ORA-01017 error. Which means “Invalid username/password”. So, it seems that the Database doesn’t like the password that OID is supplying to connect to the ODS schema.

I’ll use SQL Developer to try and connect to the database with the ODS user

Interesting, SQL Developer is showing an ORA-28000 error.

Let’s try connecting using SQLPlus …

So, it seems we have a consensus (and yes, I did just include my password in the screenshot … it doesn’t matter)

Let’s see what the database has to say about this user. Make sure you reconnect to the DB as oracle.

Ok, didn’t we just unlock it? Let’s try again …

So, now what is the status?

Hey! This is good right? … the account seems to be open again.

So, let’s try to start OID again.

Ok, this is looking pretty ugly right about now…

… and the account is locked again. So, let’s see if we can figure out why this is happening.

Maybe the wallet that holds the ODS password for OID has become corrupt. We can recreate it using oidpasswd.

Note: Before you run oidpasswd it’s important to have your Oracle environment set up correctly. Here is what I am using (yours may vary):

ORACLE_SID=orcl

ORACLE_BASE=/opt/oracle

ORACLE_INSTANCE=/opt/oracle/Middleware/asisnt_1

ORACLE_HOME=/opt/oracle/Middleware/Oracle_IDM1

MW_HOME=/opt/oracle/Middleware

Now with this output … I have verified the location of the tnsnames.ora file and the information in it … so I am going to assume for the moment that the issue is with the password (at least until I prove otherwise).

Typically, changing the password will unlock the account

But here we are and the account is still locked.

… I am spending some time just fishing around on the Internet and looking around at my system

Wait a second … I wasn’t even thinking about ODSSM …

Change the ODSSM’s password and then unlock ODS.

So, both accounts should now be “OPEN”

Now restart the OIDMON process

What does the log say

Completely different error this time. At least I feel like we are making some progress …

hmmm … if the wallet can’t be read … maybe we can recreate the wallet. Let’s re-run the “create wallet” command that we tried earlier.

Hey! … it was successful this time. So, let’s try starting the OID processes

That was successful!

Now to check the status of the OPMN Processes

All of the OID related processes are now Alive. The ohs1 process is down because I turned it off earlier.

Directory Integration Platform (DIP w/Oracle DB #oracle #idm #dip #oid

DBImport Profile: (sync w/Oracle DB table)

Important Notes:

• Diptester utility has not been updated for 11g yet.
• Oracle recommends setting up sync profile to DB by using command line tools instead of EM UI.

Where is the profile stored:

Profile DN: orclodipagentname=”profile name”,cn=subscriber profile,cn=changelog subscriber,cn=oracle internet directory

Which files make up the profile:

Profile Name: “Profile Name”
Properties File: “profile name”.properties (Connection Info)
Configuration: “profile name”.cfg.master (SQL Query)
Map File: “profile name”.map.master (map columns to attributes)

DIP Log Location:
[/apps/oracle/]Middleware/Oracle_IDM1/ldap/odi/conf
Updating DBImport Profile: (After making changes to the config files you will need to update the profile)

$ manageSyncProfiles update -h hostname-p 7005 -D weblogic -pF “profile name”-f “profile properties file name”

Warning : Setting an incorrect value for the last change number could cause the profile to stop working or cause undesired sync operations
Do you want to continue [Y/N] Y
[Connected Directory Password] // Source password … DB user password in this case.
[Weblogic user password]
Connection parameters initialized.
Connecting at hostname:7005, with userid “weblogic”..
Connected successfully.

Profile “profile name” successfully updated.

Activate / Deactivate Profiles:

manageSyncProfiles activate -h dip_hostname -p 7005 -D weblogic -pf “profile name”
manageSyncProfiles deactivate -h dip_hostname-p 7005 -D weblogic -pf “profile name”
View the Subscriber Profile:
ldapsearch -h dip_hostname -p 3060 -D “cn=orcladmin” -w “orcladmin_password” -b “orclodipagentname=”profile name”,cn=subscriber profile,cn=changelog subscriber,cn=oracle internet directory” -s sub ‘objectclass=*’

Update the orclodipcondirlastappliedchgnum attribute in the sync profile:

// if there are any problems with the initial sync and you have to do it over … you will need to reset the last applied change number in the profile.

ldapmodify -h [diphostname] -p 3060 -D “cn=orcladmin” -w Passw0rd1 -f update_timestamp.ldif

update_timestamp.ldif:
dn: orclodipagentname=[diphostname],cn=subscriber profile,cn=changelog subscriber,cn=oracle internet directory
changetype: modify
replace: orclodipcondirlastappliedchgnum
orclodipcondirlastappliedchgnum: 20000101120000

Installing and Configuring Sun #DSEE 6.3 #Oracle #IDM #DS

To quickly install and configure Sun’s Oracle’s Directory Server Enterprise Edition (DSEE) 6.3:

1. Unpack the download (tar -xvzf)
2. CD into the DSEE_ZIP_DISTRIBUTION folder
3. Install the software:./dsee_deploy install -i /opt/ds6
4. Create a new instance:./dsadm create -p port -P SSL-port instance-path
5. Start the new instance:./dsadm start instance-path
6. Modify the example.ldif and then import (to create a new suffix)./dsconf create-suffix -h localhost -p 1389 dc=example,dc=com

   ./dsconf import -h localhost -p 1389 install-path/ds6/ldif/Example.ldif \ dc=example,dc=com

OVD 11g ForkJoin Plugin “FullOuterJoin” explanation #oracle #idm #ovd

I tried to implement the ForkJoin plugin today, for the first time.  The documentation is pretty good but not clear about one specific parameter (which happened to be the one that I needed).   When you add the plugin and then select to add a parameter, screenshot, one of the options is FullOuterJoin. According to the documentation is a setting under the JoinPolicy.  The implementation is a little different, as you can see if you click on the screenshot.  I wasn’t sure what to put here,  so I checked with Oracle and was told this:

“… that referenced Full Outer Join as being set to either true or false. ”  and “… they believe that setting Full Outer Join to true would mean full outer join is performed, set to false would mean left outer join, and to have standard join we would simply not install the plug-in.”

I was able to confirm that by setting FullOuterJoin to true does indeed allow entries from both (in my case) adapters to be returned.  The only caveat to this is that entries that should be joined … are no longer joining.  So, still trying to figure that part out.

Update (02/05/2010):

I heard back from OVD Dev as follows:

The documentation for plug-in configuration parameters has to be read as:
Names mentioned in bold are parameters that a plug-in supports.
Under each parameter name, description and semantics of all possible values are explained.

First, for Fork Join plug-in, SecondaryOnlyAttributes, PrimaryAndSecondaryAttributes & JoinPolicy are the only parameters supported. ‘FullOuterJoin’ is not a parameter, but one of the possible values for ‘JoinPolicy’ parameter. The other values are ‘StandardJoin’ & ‘LeftOuterJoin’. Please note that there are no spaces in parameter values.

Second, ODSM displays the list of parameters supported by a plug-in from the plugin manifest file. Since ForkJoin plugin manifest file incorrectly has “FullOuterJoin” as the parameter name instead of “JoinPolicy”, the incorrect parameter name is displayed in ODSM.

As ODSM would not allow specifying any other parameter to the plug-in configuration than what is listed in the plug-in manifest, the workaround, for now, is to please make a backup copy and then edit the <$ORACLE_INSTANCE>/config/OVD/<ComponentName>/adapters.os_xml file as follows, then re-start OVD server:

Change following line from:
<param name=”FullOuterJoin” value=”true”/>

To:
<param name=”JoinPolicy” value=”FullOuterJoin”/>

Third, since the value for “JoinPolicy” is not specified in the plug-in configuration, ForkJoin plug-in assumes “LeftOuterJoin” as the default and hence users that are only in secondary adapter (AD2) were not returned.”

Migrating Oracle 11g IdM from 11.1.1.1.0 (R0) to 11.1.1.2.0 (R1) #IdM #Oracle

Several months ago I installed Oracle IdM 11g (11.1.1.1.0) … I’ll call it Release 0 (zero).  I don’t know if it is officially called R0 but that will do for now.  This was a development environment.  We were holding off on installing R1 due to time constraints but have finally run into enough issues that we had to make the time.  Well, I am making the time (it’s 1AM right now).  I am planning on working through the migration from R0 to R1 and hopefully have it ready for use by tomorrow morning.   The two components that need to be upgraded our Oracle Internet Directory (OID) and Oracle Virtual Directory.  With 11g you have to upgrade Weblogic as well … so that is on my list for tonight also.

Our most compelling reason for upgrading is one particular plugin that has become available in OVD in R1.  That plugin is the ForkJoin Plugin as described from the Administration Guide:

“Supported only for Join View Adapters, the ForkJoin plug-in allows you to search against the primary adapter and/or secondary adapters in a Join View. During LDAP search, when a search filter contains one or more attributes that are available only in secondary adapter, without this plug-in, Oracle Virtual Directory cannot return Joined entries that satisfies the filter, as the entire search filter is sent only to the primary adapter. Using the ForkJoin Plug-in, Oracle Virtual Directory can search on attributes only in the primary adapter, only in the secondary adapter, and in both the primary and secondary adapters.”

According to the migration documentation from Oracle the upgrade process is actually a near complete reinstall and then a data migration. We have a lot of Oracle Access Manager Policies that are stored in OID and a few virtual dynamic groups stored in OVD. I am more concerned about the policies because of the time it took to implement.

Migration Steps

Step 1: Backups of course.

For backing up OID I just started at the root suffix and did an ldif export (from Apache Directory Studio).

Step 2: Install new Weblogic Home

Step 3: Install R1 components (OID, OVD)

When running the installer, make sure to select, “Install Software – Do Not Configure”. The Patch Assistant will migrate the middle-tier configuration. I noticed that when I selected, “Do not configure”, that the installer didn’t ask which components to install.

Step 4: Using the Patch Assistant to migrate schemas/data

Update the patchMaster.properties file with:

patchMaster.Domainhomenew

patchMaster.Domainapplicationshomenew

patchMaster.Domainhomeold

patchMaster.Domainusername

patchMaster.Oracleinstanceoldlist

patchMaster.Oraclehomeold

Interestingly enough the attribute Schemaprefix is marked as “optional” in the patchmaster file but the ant process complained that that attribute was null. I put “ODS” and it stopped complaining.

Create a WebLogic Domain for 11.1.1.2.0

Run the MW_HOME/oracle_common/common/bin/config.sh

Migrating the Domain

Type: ant master-patch-domain

Migrating an Instance

Type: ant master-patch-oinstance

Register the Instance

Type: ant master-register-oinstance

At this point I am getting an error because I missed the part that said, “Before registering the instance, ensure that your Administration Server is up and running”. I am searching for the new “StartWebLogic.sh” file but not able to find it. Strange! [time: 3:49 am]

At 4:15 AM I realized that I had skipped the part on creating the Weblogic Domain for 11.1.1.2.0 so, I went back and did that.

The migration completed successful but it looks like there is some cleanup that needs to be done. OID is not responding. OVD looks like it migrated successfully. I was able to log in and see the dynamic groups that I had created in the previous version.

[This is where I left off. I am planning on picking this up later today or tomorrow to complete the migration. I will update the post with the final details for migration.]

Step 5: Test new instances (Does OAM still work? Is all of the appropriate data available via OVD?)

Step 6: Decommission R0

UPDATE (02/05/2010)

I finally found that I had made a mistake with my patchMaster.properties file by leaving off “ODSM and OPMN” from the components list.  I ended up redoing the install and it worked perfectly.  Here is what the patchMaster.properties file should look like:

patchMaster.Componentlist=OID,OVD,ODSM,OPMN

patchMaster.Schemaurl=jdbc:oracle:thin:@idm.acme.com:1521:acme1 (note: acme1 is the service name)

patchMaster.Schemauser= System

patchMaster.Schemaprefix=ODS

patchMaster.Mwhomenew=/apps/Oracle/Middleware_R1

patchMaster.Mwhomeold=/apps/Oracle/Middleware

patchMaster.Domainhomenew=/apps/Oracle/Middleware_R1/user_projects/domains/IDMDomain

patchMaster.Domainhomeold=/apps/Oracle/Middleware/user_projects/domains/IDMDomain

patchMaster.Oracleinstancenewlist=/apps/Oracle/Middleware_R1/oid_1

patchMaster.Oracleinstanceoldlist=/apps/Oracle/Middleware/oid_1

patchMaster.Oraclehomenew=/apps/Oracle/Middleware_R1/Oracle_IDM1

patchMaster.Oraclehomeold=/apps/Oracle/Middleware/Oracle_IDM1

patchMaster.Domainadminhost=idmhost.acme.com

patchMaster.Domainadminport=7001

Starting the OID Replica Server

For two-way LDAP replication, you must start the Oracle Internet Directory replication servers at both the sponsor replica and the new replica, as follows:

Start or restart the replication server at the sponsor replica. Type:

1. Log into the sponsor server as oracle
2. oidctl server=oidrepld connect=oiddb instance=1 flags=”-h supplier_hostname -p 3060″ start

Start the replication server at the new replica. Type:

1. Log into the consumer replica server  as oracle
2. oidctl server=oidrepld connect=oiddb instance=1 flags=”-h consumer_hostname -p 3060″ start

To check the status of the services:

oidctl connect=oiddb name=oid_1 status

which results in this:

Exception java.lang.UnsatisfiedLinkError: libawt.so libXp.so.6: cannot open shared object file: No such file or directory

I encountered this error when trying to install Oracle Internet Directory on Redhat Enterprise Server.  The output looked like this:

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2009-06-15_09-53-57AM. Please wait ...[oracle@id-host2 Disk1]$ Oracle Universal Installer, Version 10.1.0.5.0 Production

Copyright (C) 1999, 2006, Oracle. All rights reserved.

Exception java.lang.UnsatisfiedLinkError: /tmp/OraInstall2009-06-15_09-53-57AM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory occurred..

java.lang.UnsatisfiedLinkError: /tmp/OraInstall2009-06-15_09-53-57AM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory

        at java.lang.ClassLoader$NativeLibrary.load(Native Method)

        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1560)

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1477)

        at java.lang.Runtime.loadLibrary0(Runtime.java:788)

        at java.lang.System.loadLibrary(System.java:834)

        at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.awt.NativeLibLoader.loadLibraries(NativeLibLoader.java:38)

        at sun.awt.DebugHelper.<clinit>(DebugHelper.java:29)

        at java.awt.Component.<clinit>(Component.java:506)

        at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.<init>(OiifmGraphicInterfaceManager.java:193)

        at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:174)

        at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:183)

        at oracle.sysman.oii.oiic.OiicInstaller.<init>(OiicInstaller.java:278)

        at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:714)

        at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:628)

Exception in thread "main" java.lang.NoClassDefFoundError

        at oracle.sysman.oii.oiif.oiifm.OiifmGraphicInterfaceManager.<init>(OiifmGraphicInterfaceManager.java:193)

        at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.createInterfaceManager(OiicSessionInterfaceManager.java:174)

        at oracle.sysman.oii.oiic.OiicSessionInterfaceManager.getInterfaceManager(OiicSessionInterfaceManager.java:183)

        at oracle.sysman.oii.oiif.oiifm.OiifmAlert.<clinit>(OiifmAlert.java:112)

        at oracle.sysman.oii.oiic.OiicInstaller.runInstaller(OiicInstaller.java:772)

        at oracle.sysman.oii.oiic.OiicInstaller.main(OiicInstaller.java:628)

This is typically indictive that I haven’t installed all of the required Redhat Packages.  I made a list of all of the RPMs that I usually have to install to then install Oracle Internet Directory.  This list is specific for Redhat (64 bit) … if you are on a 32 bit machine … look for the 32 bit alternative.

To install use the following command:  ”rpm -Uvh [package].rpm”  (insert the package name, from the list below, into the command where it says package).

xorg-x11-deprecated-libs-6.8.2-1.EL.52.x86_64.rpm
openmotif21-2.1.30-11.RHEL4.6.i386.rpm
openmotif-2.2.3-10.2.e14.x86_64.rpm
compat-glibc-headers-2.3.2-95.30.x86_64.rpm
compat-glibc-2.3.2-95.30.x86_64.rpm
compat-libstdc++-296-132.7.2.i386.rpm
compat-db-4.1.25-9.i386.rpm
compat-db-4.1.25-9.x86_64.rpm
libstdc++-devel-3.4.6-9.i386
libaio-0.3.105.2.x86_64.rpm
libaio-devel-0.3.105-2.x86_65.rpm
libobjc-3.4.6-10.x86_64.rpm
libieee1284-0.2.8-4.x86_64.rpm
sysstat-5.0.5-16.rhel4.x86_64.rpm
glibc-kernheaders-2.4-9.1.100.EL.x86_64.rpm
glibc-headers-2.3.4-2.39.x86_64.rpm
glibc-devel-2.3.4-2.39.i386.rpm
glibc-devel-2.3.4-2.39.x86_64.rpm
glibc-2.3.4-2.41.x86_64.rpm
gcc-3.4.6-9.x86_64.rpm
gcc-c++-3.4.6-9.x86_64.rpm
gcc-objc-3.4.6-10.x86_64.rpm
sane-backends-1.0.14-6.e14.1.x86_64.rpm

Oracle Internet Directory: Purge the Human Intervention Queue

Oracle recommends using the scripts, $ORACLE_HOME/ldap/bin/hiqpurge.sh (or hiqretry.sh) but I wasn’t able to get these to work even after modifying as indicated on metalink.  I was able to do the following and then confirmed with Oracle Support that this was correct.

Log in to the Database (Replica DB) as ODS user:

Determine what we are dealing with (I wanted to see what columns were in the table)

1. sqlplus ods/password List columns in asr_chg_log:

2. describe asr_chg_log List Records in asr_chg_log:

How many entries are in the HI queue: (there were zero rows in the master and 19 in the replica)

3. select chg_no, orclguid, chg_seq, retry_cnt from asr_chg_log; Delete entries whose RETRY_CNT is -1 (which means they are in the HIQ):

4. delete from asr_chg_log where RETRY_CNT = ‘-1′;

Restart OID (on both nodes: master and replica)

1. opmnctl stop

2. opmnctl start Use ODM to make change in master.

Confirm by watching for change in Replica.