This Article will guide you step-by-step how to set a Oracle 12 Database with ASM and iSCSI mounted disks in Oracle linux. This in quite straight forward and the only thing that I consider to be a hazel is mounting the iSCSI devices in a good manner. First we have to install the OS, but Im not going to cover that more than with a few words and mostly screenshots, since installing this should be easy for most. The thing to keep in mind is the host name, since the oracle installation needs to be able to lookup the local hostname (from DNS which is preferable in production or in the /etc/hosts file, my choice) . In my case my Linux is running as a virtual machine on VMWare
- Netmask: 255.255.255.0
First you have to choose language for the install Then: Just install the basic drivers, I’ll add the iSCSI support later on Also set IP by choosing Configure Network, and input your IP/Netmask and Gateway. Also choose AUTO CONNECT!! If you are using a virutal machine for this, then Use all space is your choice, if you are afraid of losing any data choose Replace Existing Linux System(s). In my case i used Create Custom Layout to set up my partiotioning as I please, because Oracle Linux uses LVM as default and I don’t feel comfortable working with that Installation done! If you are like me, and failed to set the default Gateway from the GUI you need to adding
(or what ever you have) in the /etc/sysconfig/network-scripts/ifcfg-eth0 to look something like this:
DEVICE=eth0 TYPE=Ethernet UUID=be4bfbb4-2b09-44f9-9f2c-a3e4ba674ec5 ONBOOT=yes NM_CONTROLLED=yes BOOTPROTO=none HWADDR=00:50:56:9E:8E:D8 IPADDR=192.168.110.25 GATEWAY=192.168.110.1 PREFIX=24 DNS1=192.168.100.14 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="System eth0"
and restart the network by /etc/init.d/network restart Once we’ve installed our Oracle Linux as a BASIC SERVER w/o KDE/GNOME etc and have the hostname/IP address configured. We can installing the Database related stuff: First turn off the firewall (or configure it for all the DB exceptions) since this is for testing only!.
[root@savdb01 ~]# service iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] [root@savdb01 ~]# chkconfig iptables off [root@savdb01 ~]#
Install Oracle 12c preinstall package with all packages and some settings and some users (oracle, oinstall).
yum install -y oracle-rdbms-server-12cR1-preinstall
Then we have to add the necessary groups for grid and add a few groups to the oracle user, and the the grid user is self.
Note that the oracle user will not be used in this article, since we only will install the grid component, but I know i will install oracle Database on here as well so.
groupadd asmadmin groupadd asmdba groupadd asmoper groupadd oper groupadd dba useradd -g oinstall -G asmoper,asmadmin,asmdba,dba -m grid usermod -G dba,oper,asmdba -m oracle passwd grid passwd oracle
Now we have all the prerequisite for ASM and the Database. No we need to configure the iSCSI.
yum install iscsi-initiator-utils
Configuring the iSCSI Initiator to Start on Reboot
Once the iSCSI Initiator package is installed, configure the package to start on reboot by completing the following steps: Check that the iSCSI initiator will start when the system is rebooted:
[root@savdb01 ~]# chkconfig --list iscsi iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off [root@savdb01 ~]# chkconfig --list iscsid iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
This example shows that both the iscsi and iscsid services are set to off at all run levels, so the iSCSI Initiator will not start at system reboot. If the settings at run level 3, 4, and 5 are not set to on for either the iscsi or iscsid service, enable them, as shown below:
chkconfig iscsi on chkconfig iscsid on [root@oel ~]# chkconfig --list iscsi iscsi 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@oel ~]# chkconfig --list iscsid iscsid 0:off 1:off 2:on 3:on 4:on 5:on 6:off service iscsi start service iscsid start
The iscsi and iscsid services are now set to start at system reboot. Next we have to identify the Host IQN. The Host IQN will be used by(most) the SAN and was auto-generated when we installed the initiator package.
cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1988-12.com.oracle:f5cf203ccf68
Then we need to the SAN to give us some big LUNs. This is done in many ways but if you like me are using an Oracle ZFS Storage Appliance you can follow this guide from Oracle Now we can try to identify the LUNs that we published before. First make sure the iscsi is started
service iscsi start iscsiadm -m discovery -t sendtargets -p 192.168.110.210 192.168.110.210:3260,2 iqn.1986-03.com.sun:02:c49183bc-8519-642a-c5ff-fa01e9b2bd79 iscsiadm -m node -p 192.168.110.210 -l tail /var/log/messages ... .. . Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:2: [sdd] Attached SCSI disk Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:5: [sdg] Attached SCSI disk Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:6: [sdh] Attached SCSI disk Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:4: [sdf] Attached SCSI disk Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:3: [sde] Attached SCSI disk Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:7: [sdi] Attached SCSI disk Apr 30 13:56:26 savdb01 kernel: sd 3:0:0:1: [sdc] Attached SCSI disk
Oracle has its own resolution for generating block device files especially for ASM, called ASMLib. This indeed works well, but how Oracle did to generate the devices, and how to troubleshoot the generation of the devices is for me utterly confusing, and i have not found any documentation on this. I’ve managed to mess my lab servers a few times, which is the reason I don’t like ASMLib, and do not feel comfortable with ASMLib. More on that can be read on ORAinternals/Riyaj Shamsudeen
When we’ve verified that the disks have attached we need to prep the disks using fdisk:
fdisk /dev/sdb fdisk /dev/sdc fdisk /dev/sdd fdisk /dev/sde fdisk /dev/sdf -->"n", "p", "1", "Return", "Return" and "w" You should now see the raw device and the partition we created: <strong>Next up we need to create the UDEV Rules</strong> To configure SCSI devices as trusted add 1options=-g
to the file /etc/scsi_id.config (note! that this file is probably missing so yu have to create it)
[root@savdb01 ~]# scsi_id -g -u -d /dev/sdb;scsi_id -g -u -d /dev/sdc;scsi_id -g -u -d /dev/sdd;scsi_id -g -u -d /dev/sde;scsi_id -g -u -d /dev/sdf; 3600144f0ebda77b600005268fa770003 3600144f0ebda77b600005268fa470002 3600144f0ebda77b600005268fa8c0004 3600144f0ebda77b600005268fa9f0005 3600144f0ebda77b600005280e58d0006 [root@savdb01 ~]#
This is the information needed to create your udev rules to map the SCSI devices. Create the file /etc/udev/rules.d/99-asmdevices.rules and add the YOUR SCSI_id by replacing the RESULT part.
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa770003", NAME="asm-disk1", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa470002", NAME="asm-disk2", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa8c0004", NAME="asm-disk3", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005268fa9f0005", NAME="asm-disk4", OWNER="grid", GROUP="oinstall", MODE="0660" KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="3600144f0ebda77b600005280e58d0006", NAME="asm-disk5", OWNER="grid", GROUP="oinstall", MODE="0660"
Then you can either reboot or do:
partprobe /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
Then we test, reload,start Udev and the disks should all pop up under /dev/
[root@savdb01 ~]# udevadm test /block/sdb/sdb1 [root@savdb01 ~]# udevadm control --reload-rule [root@savdb01 ~]# start_udev Startar udev: [ OK ] [root@savdb01 ~]# ls -al /dev/asm* brw-rw----. 1 grid oinstall 8, 17 30 apr 15.34 /dev/asm-disk1 brw-rw----. 1 grid oinstall 8, 33 30 apr 15.34 /dev/asm-disk2 brw-rw----. 1 grid oinstall 8, 49 30 apr 15.34 /dev/asm-disk3 brw-rw----. 1 grid oinstall 8, 65 30 apr 15.34 /dev/asm-disk4 brw-rw----. 1 grid oinstall 8, 81 30 apr 15.34 /dev/asm-disk5
Thats the OS part, all disks are now ready to be used by ASM Install Oracle Grid Infrastructure First we need to create a folder structure, i guess you all have your favorite structure, in this case I’ll go for /oracle as $ORACLE_BASE.
[root@savdb01 ~]# mkdir -p /oracle/app/grid/gi12101 [root@savdb01 ~]# chown -R grid:oinstall /oracle/app/grid [root@savdb01 ~]# mkdir -p /oracle/app/oracle/product/18.104.22.168/dbhome_1 [root@savdb01 ~]# chown -R oracle:oinstall /oracle/app/oracle/ [root@savdb01 ~]# chmod -R 770 /oracle/app/oracle/ [root@savdb01 ~]# chmod -R 770 /oracle/app/grid
We also have to create the grid user profile. To do that add the following to your profile (~/.bash_profile)
ORACLE_BASE=/oracle/app ORACLE_HOME=$ORACLE_BASE/grid/gi12101 ORACLE_SID=+ASM LD_LIBRARY_PATH=$ORACLE_HOME/lib PATH=$PATH:/usr/local/bin:/usr/sbin:/usr/bin:/usr/openwin/bin:/usr/ucb:$ORACLE_HOME/bin export ORACLE_BASE ORACLE_HOME ORACLE_SID LD_LIBRARY_PATH PATH
Now we download the grid binaries and unzip them in the grid user_home. Then we can run ./runInstaller. There is however some pre req. you need to pass first, memory part should be OK but you might run into trouble with the DISPLAY. You could write a book on this but, the easiest was is to connect from a Linux or Mac (with XQuartz) and use the ssh option -X(that is capital X) to enable x forwarding like:
[andre-mac:~ macora] ssh -X email@example.com
Now you probably forgot about the DNS or hosts entry. Add something like:
192.168.110.25 savdb01 savdb01.savecore.se savdb01.localdomain
to /etc/hosts or setup your DNS accordingly. Here we have change the discovery path to what se configured in the udev rules /dev/asm* And now w should see all (in this case) 5 disks as candidates. Select the disk and choose Normal (or extern for no ASM redundancy) For simplicity same password everywhere If you have create the groups OK they should be like this by default: Settings accoring to ORACLE_BASE and ORACLE_HOME for grid user profile oraInventory in ORACLE_BASE Skip this unless you have suders setup accordingly. You will probably have som issues here. Memory in this case is fine its a test system and just under the limit (4GB). The other can be fixed by “FIX and CHECK again” Run the script above and then press OK This can be ignored, by Clicking Ignore and then Next Yes (ffa) Review and install!
[root@savdb01 ~]# /oracle//oraInventory/orainstRoot.sh Changing permissions of /oracle//oraInventory. Adding read,write permissions for group. Removing read,write,execute permissions for world. Changing groupname of /oracle//oraInventory to oinstall. The execution of the script is complete. [root@savdb01 ~]# /oracle/app/grid/gi12101/root.sh Performing root user operation for Oracle 12c The following environment variables are set as: ORACLE_OWNER= grid ORACLE_HOME= /oracle/app/grid/gi12101 Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ... Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root script. Now product-specific root actions will be performed. Using configuration parameter file: /oracle/app/grid/gi12101/crs/install/crsconfig_params LOCAL ADD MODE Creating OCR keys for user 'grid', privgrp 'oinstall'.. Operation successful. LOCAL ONLY MODE Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. CRS-4664: Node savdb01 successfully pinned. 2014/05/02 01:18:38 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf' savdb01 2014/05/02 01:19:05 /oracle/app/grid/gi12101/cdata/savdb01/backup_20140502_011905.olr 2014/05/02 01:19:06 CLSRSC-327: Successfully configured Oracle Grid Infrastructure for a Standalone Server [root@savdb01 ~]#
Run the root script when prompted. Finally, this is what we want to see 🙂 #Done<