Oracle Grid Infrastructure v12.1 RAC installation + Oracle 11.2 Database RAC installation installed on two Linux RedHat servers. The datafiles were stored on the shared storage system via ASM. Both machines were wiped out with "rm -rf /". There was no file system backup (yes, i know what are you going to say about that - it's not my fault, i'm just trying to help ;). All Oracle data is still there because all storage LUNs were mounted as Linux block devices (raw devices), so nothing was removed from the storage system. But the OCR, Voting Disks and the whole clusterware is lost.
There were two independent ASM diskgroups for Grid Infrastructure and for data: +GRID and +DATA.
How can I restore the Clusterware (Grid Infrastructure) and get back an old DATA diskgroup with its contents?
Finally managed to restore all DBs! It was done in the following way:
. Overwrite GI ASM disk group (+GRID) devices with zeroes - in order to make them free (candidates) for the new GI installation. NOTE: you may want to make a backup of those devices/disks before overwriting them. It was enough to overwrite first 10MiB of each device with zeroes:
dd if=/dev/zero of=/dev/mapper/grid1 bs=1024K count=10 ... dd if=/dev/zero of=/dev/mapper/grid3 bs=1024K count=10
. Install 12.1 (latest version) of GI (Grid Infrastructure), using newly created +GRID ASM disk group, patch it with the latest GI PSU (Jan 2016)
. at this point you should be able to see your old DATA ASM disk group:
sqlplus / as sysasm alter diskgroup DATA mount;
. Install the same version of Database Home as before deleting/losing the OS - in my case 22.214.171.124.0
. Install the latest Database PSU for 126.96.36.199, which is 188.8.131.52.12
. Now I can start DBs manually, but when I was trying to register 11.2 DB against 12.1 CRS I was always hitting BUG: 13460353 (Registration of 11.2 DB fails against 12.1 CRS stack)
. download corresponding one-off patch, install it on DB home
. after that it was possible to register DBs and instances in CRS
. recreate old services