Wednesday 24 December 2014

How to know if a OPATCH requires downtime?

1. Download the particular opatch file
2. Check the file ..\\etc\config\inventory
3. If the following entry is present
(instance_shutdown) true (/instance_shutdown)
then it requires downtime.
Note: OPatch means rdbms patch.
For rdbms patch go to OPatch dorectory and select /etc/config/inventory

Post Clone Checks,Procedures for EBS 11i / R12

(Here DEV is the target and PROD is the source)
CHECKS
===========
THE CHECKS/STEPS LISTED HERE SHOULD BE PERFORMED AFTER DATABASE CLONE BEFORE APPLICATIONS CLONE
1 Check the Target – If its in ARCHIVELOG / NOARCHIVELOG MODE as desired.
2 Create the Temp datafiles , using the commands taken from the control file , if the temporary tablespace tempfiles is not existing.
ALTER TABLESPACE TEMP ADD TEMPFILE __________
Verify dba_temp_files, for all the temp file entries added to TEMP
SQL> select file_name from dba_temp_files;
Check for the Default Temporary file –
===========================
select * from database_properties where property_name = ‘DEFAULT_TEMP_TABLESPACE';
Confirm that this TEMP.
If the default is not TEMP,
SQL>alter database default temporary tablespace TEMP
3 Update global_name if it contains source
select * from global_name;
update global_name set global_name='<global db name>';
commit;
SQL> select * from global_name;
GLOBAL_NAME
——————————————————————————–
DEV
4 Restoration of files backed up on Target Instance prior to the clone.
- You may need to restore the tnsnames.ora , listener.ora (which was backed up for the target instance prior to clone)
– You may need to restore the spfile/init file , so that the target instance takes the original init parameters (memory/performance parameters) and restart the database.
5 Change the directories – if any need to be changed
set heading off
set pages 1000
set linesize 100
select ‘CREATE OR REPLACE DIRECTORY ‘||DIRECTORY_NAME||’ AS ‘||””||DIRECTORY_PATH||””||';’
from all_directories;
spool off
- Edit the commands by replacing PROD with DEV ie source with target and Execute.
- You would also be required to check and recreate the DB links as per tns entries and requirements in the Target Instance.
6 Update apps.wf_systems
(This should show the target Instance and not the source Instance)
SQL>select name,display_name from apps.wf_systems;
SQL>update apps.wf_systems set name=’<SID>’,display_name=’ <SID>.<hostname.domainname>’;
SQL>COMMIT;
Example after change
SQL> select name,display_name from apps.wf_systems;
NAME DISPLAY_NAME
—————————— ——————————————————————————–
DEV DEV.DBALOUNGE.COM
7 Update Notification status
(This needs to be bone before running adcfgclone on the apps tier , to avoid any Notifications to be sent from the Target Instance)
UPDATE wf_notifications SET status =’CLOSED’, mail_status =’SENT’, end_date =’01-JAN-01′ WHERE mail_status in (‘MAIL’,’INVALID’,’OPEN’) ;
Commit;
update wf_agents set address = replace ( address,’PROD’,’DEV’ ) ;
select name ,display_name from wf_systems;
update wf_systems set
DISPLAY_NAME = replace ( DISPLAY_NAME, ‘PROD.DBALOUNGE.COM’,’DEV.DBALOUNGE.COM’);
Commit;
8 Update the target node in CM fnd_concurrent_queues table :
update fnd_concurrent_queues set node_name='<target_conc_node>’ where node_name='<source_conc_node>';
commit;
9 Custom top file
R12 Error ‘Function not available to this responsibility’ When Trying to Access Custom Form. [ID 553014.1]
$ORA_CONFIG_HOME/10.1.2/forms/server/default.env ” and add custom in that as well.
THE CHECKS/STEPS LISTED HERE SHOULD BE PERFORMED AFTER APPLICATIONS CLONE BUT with APPS SERVICES DOWN
9 You may want to Restore the backed up target xml file (DEV xml) file to its proper location.
We may want to use the same xml file which was in place prior to the clone so that the same ports / configuration is in use.
10 remove the log files from APPLCSF/APPLLOG:
Make sure Dir are existing ; Use command to confirm
cd $APPLCSF/$APPLLOG
Delete all files under $APPLCSF/$APPLLOG
11 You can set the logfile to NULL
update fnd_concurrent_requests set logfile_name = null, logfile_node_name = null,
outfile_name = null, outfile_node_name = null;
commit;
12 You may need to run
exec FND_CONC_CLONE.SETUP_CLEAN;
commit;
to clear FND_NODES entries and rerun autoconfig (on all application tiers) as per step given below
13 Run Autoconfig
14. Change apps , system and sysadmin and a module specific passwords.
Change apps password in wdbsvr.app and CGIcmd.dat files
15 Log in to the target system application tier node as the applications User
Run the following tasks in adadmin for all products:
– generate JAR files
– generate message files
– relink executables
16 Run cmclean.sql
(This may be required to clean Concurrent Queue Status and Request Status)
THE CHECKS/STEPS LISTED HERE SHOULD BE PERFORMED AFTER APPLICATIONS CLONE with APPS SERVICES UP and RUNNING
17 Bring up all services
18 Set the Site Name Profile Option
20 Configure Workflow Mailer
Login to OAM. Click on
Notification Mailer => Edit Button => Advanced.
Make sure to type in the Target instance password.
PROCESS and DISCARD should exist on the Server for wf<sid> user.
The name of the Process and the Discard directory can be got from the workflow configuration
21 Check and Schedule the Concurrent Requests – these would include Gather Statistics , Purge old data and Workflow related Concurrent Requests.
You can submit these Conc. Requests using forms or use CONCSUB as described in Concsub-to-Quickly-submit-Concurrent-Requests
22 Do a comprehensive Status check
23 Release the Instance to Users

TEMP TABLESPACE Monitor Scripts

To check instance-wise total allocated, total used TEMP for both rac and non-rac
set lines 152
col FreeSpaceGB format 999.999
col UsedSpaceGB format 999.999
col TotalSpaceGB format 999.999
col host_name format a30
col tablespace_name format a30
select tablespace_name,
(free_blocks*8)/1024/1024 FreeSpaceGB,
(used_blocks*8)/1024/1024 UsedSpaceGB,
(total_blocks*8)/1024/1024 TotalSpaceGB,
i.instance_name,i.host_name
from gv$sort_segment ss,gv$instance i where ss.tablespace_name in (select tablespace_name from dba_tablespaces where contents='TEMPORARY') and
i.inst_id=ss.inst_id;
Total Used and Total Free Blocks
select inst_id, tablespace_name, total_blocks, used_blocks, free_blocks  from gv$sort_segment;
Another Query to check TEMP USAGE
col name for a20
SELECT d.status "Status", d.tablespace_name "Name", d.contents "Type", d.extent_management
"ExtManag",
TO_CHAR(NVL(a.bytes / 1024 / 1024, 0),'99,999,990.900') "Size (M)", TO_CHAR(NVL(t.bytes,
0)/1024/1024,'99999,999.999') ||'/'||TO_CHAR(NVL(a.bytes/1024/1024, 0),'99999,999.999') "Used (M)",
TO_CHAR(NVL(t.bytes / a.bytes * 100, 0), '990.00') "Used %"
FROM sys.dba_tablespaces d, (select tablespace_name, sum(bytes) bytes from dba_temp_files group by
tablespace_name) a,
(select tablespace_name, sum(bytes_cached) bytes from
v$temp_extent_pool group by tablespace_name) t
WHERE d.tablespace_name = a.tablespace_name(+) AND d.tablespace_name = t.tablespace_name(+)
AND d.extent_management like 'LOCAL' AND d.contents like 'TEMPORARY';
Temporary Tablespace groups
SELECT * FROM DATABASE_PROPERTIES where PROPERTY_NAME='DEFAULT_TEMP_TABLESPACE';

select tablespace_name,contents from dba_tablespaces where tablespace_name like '%TEMP%';

select * from dba_tablespace_groups;
To Check Percentage Usage of Temp Tablespace
select (s.tot_used_blocks/f.total_blocks)*100 as "percent used"
from (select sum(used_blocks) tot_used_blocks
from v$sort_segment where tablespace_name='TEMP') s,
(select sum(blocks) total_blocks
from dba_temp_files where tablespace_name='TEMP') f;
To check Used Extents ,Free Extents available in Temp Tablespace
SELECT tablespace_name, extent_size, total_extents, used_extents,free_extents, max_used_size FROM v$sort_segment;
To list all tempfiles of Temp Tablespace
col file_name for a45
select tablespace_name,file_name,bytes/1024/1024,maxbytes/1024/1024,autoextensible from dba_temp_files  order by file_name;

SELECT d.tablespace_name tablespace , d.file_name filename, d.file_id fl_id, d.bytes/1024/1024
size_m
, NVL(t.bytes_cached/1024/1024, 0) used_m, TRUNC((t.bytes_cached / d.bytes) * 100) pct_used
FROM
sys.dba_temp_files d, v$temp_extent_pool t, v$tempfile v
WHERE (t.file_id (+)= d.file_id)
AND (d.file_id = v.file#);
Additional checks
select distinct(temporary_tablespace) from dba_users;

select username,default_tablespace,temporary_tablespace from dba_users order by temporary_tablespace;

SELECT * FROM DATABASE_PROPERTIES where PROPERTY_NAME='DEFAULT_TEMP_TABLESPACE';
Changing the default temporary Tablespace
SQL> alter database default temporary tablespace TEMP;
Database altered.
To find Sort Segment Usage by Users
select username,sum(extents) "Extents",sum(blocks) "Block"
from v$sort_usage
group by username;
To find Sort Segment Usage by a particular User
SELECT s.username,s.sid,s.serial#,u.tablespace, u.contents, u.extents, u.blocks
FROM v$session s, v$sort_usage u
WHERE s.saddr=u.session_addr
order by u.blocks desc;
To find Total Free space in Temp Tablespace
select 'FreeSpace  ' || (free_blocks*8)/1024/1024 ||' GB'  from v$sort_segment where tablespace_name='TEMP';

select tablespace_name , (free_blocks*8)/1024/1024  FreeSpaceInGB,
(used_blocks*8)/1024/1024  UsedSpaceInGB,
(total_blocks*8)/1024/1024  TotalSpaceInGB
from v$sort_segment where tablespace_name like '%TEMP%'
To find  Total Space Allocated for Temp Tablespace
select 'TotalSpace ' || (sum(blocks)*8)/1024/1024 ||' GB'  from dba_temp_files where tablespace_name='TEMP';
Get 10 sessions with largest temp usage
cursor bigtemp_sids is
select * from (
select s.sid,
s.status,
s.sql_hash_value sesshash,
u.SQLHASH sorthash,
s.username,
u.tablespace,
sum(u.blocks*p.value/1024/1024) mbused ,
sum(u.extents) noexts,
nvl(s.module,s.program) proginfo,
floor(last_call_et/3600)||':'||
floor(mod(last_call_et,3600)/60)||':'||
mod(mod(last_call_et,3600),60) lastcallet
from v$sort_usage u,
v$session s,
v$parameter p
where u.session_addr = s.saddr
and p.name = 'db_block_size'
group by s.sid,s.status,s.sql_hash_value,u.sqlhash,s.username,u.tablespace,
nvl(s.module,s.program),
floor(last_call_et/3600)||':'||
floor(mod(last_call_et,3600)/60)||':'||
mod(mod(last_call_et,3600),60)
order by 7 desc,3)
where rownum < 11;
Displays the amount of IO for each tempfile
SELECT SUBSTR(t.name,1,50) AS file_name,
f.phyblkrd AS blocks_read,
f.phyblkwrt AS blocks_written,
f.phyblkrd + f.phyblkwrt AS total_io
FROM   v$tempstat f,v$tempfile t
WHERE  t.file# = f.file#
ORDER BY f.phyblkrd + f.phyblkwrt DESC;

select * from (SELECT u.tablespace, s.username, s.sid, s.serial#, s.logon_time, program, u.extents, ((u.blocks*8)/1024) as MB,
i.inst_id,i.host_name
FROM gv$session s, gv$sort_usage u ,gv$instance i
WHERE s.saddr=u.session_addr and u.inst_id=i.inst_id  order by MB DESC) a where rownum<10;
Identifying WHO is currently using TEMP Segments 10g onwards
SELECT sysdate,a.username, a.sid, a.serial#, a.osuser, (b.blocks*d.block_size)/1048576 MB_used, c.sql_text
FROM v$session a, v$tempseg_usage b, v$sqlarea c,
     (select block_size from dba_tablespaces where tablespace_name='TEMP') d
    WHERE b.tablespace = 'TEMP'
    and a.saddr = b.session_addr
    AND c.address= a.sql_address
    AND c.hash_value = a.sql_hash_value
    AND (b.blocks*d.block_size)/1048576 > 1024
    ORDER BY b.tablespace, 6 desc;

Basic Linux Commands

To check the Size of RAM in Unix

HP-UX

swapinfo –tm
Linux
free –m
free -g
AIX
lsdev -C|grep mem
Solaris
prtconf|grep -i mem

To Check number of CPU’s in the Server

Linux
cat /proc/cpuinfo | grep “processor”|wc –l
HP-UX
ioscan -C processor | grep processor | wc –l
SUN Solaris
psrinfo -v|grep “Status of processor”|wc –l
AIX
lsdev -C|grep Process|wc –l
To identify High Disk I/o using Unix command
/usr/bin/ps -eo pid,user,pcpu,args | grep -v %CPU |  sort -nbk 3 | tail -25
To report the current CPU Activity
sar -u 1 5
sar utility can be used to provide various information about the server resources.
In the above example the parameters used and the details are as mentioned below
-u option is for CPU related information
1 is the frequency of execution of the command in seconds
5 is the number of times the command should be executed.
So the command displays the CPU information every 1 second for 5 times.
It reports the CPU related usage by the user(%usr), system(%sys), the CPU Wait for I/O(%wio) and the Idle Percentage(%idle).
Sample Output
14:08:17    %usr    %sys    %wio   %idle   physc
14:08:18      25        0          0           75       4.00
14:08:19      25        0          0           74       4.00
14:08:20      25        0          0           75       4.00
14:08:21      25        0          0           75       4.00
14:08:22      25        0          0           75       4.03
Average       25        0          0           75       4.01

Finding files in Unix based on specific criteria

FIND FILES THAT WERE MODIFIED/ACCESSED WITHIN A SPECIFIED TIME
            $   find  /<your path>  -mtime   1  -type   f 
This command would return files modified  in the last 24 hours .
You can use -mtime option to return a list of files that were last modified N*24 hours ago. For example to find a file in last month (30 days) you would need to use -mtime +30 options.
  • -mtime +30   means you are looking for a file modified 30 days ago.
  • -mtime -30    means less than 30 days.
  • -mtime 30      If you skip + or – it means exactly 30 days
      -type f      searches only for files and not directories
To list the files in the directory tree that were modified within the past five minutes, type
$  find /<your path>  -mmin -5
To return a list of files that were accessed in the last 24 hours you would need to use the–atime option.
$  find  /<your path>   -atime   1  -type   f  ******************************************************************
FINDING FILES WITH A SPECIFIC EXTENTION WITHIN A DIRECTORY
         $  find   /<your path>   -name  “*.cfg”
The command –name matches the file names with the specified pattern
*****************************************************************
FINDING FILES BASED ON THE FILE-PERMISSIONS
Files with execute permission for group :
$ find /<your path>  -perm g=x   -type f 
Files with execute permission for others:
$ find /<your path>  -perm  o=x   -type f
Where ‘g’ denotes groups and ‘o’ denotes others. denotes execute permission.
*****************************************************************
KILLING ACTIVE UNIX PROCESSES FOR SPECIFIC COMPONENTS
ps -ef |grep applmgr |grep <Component Name> |grep -v grep |awk ‘{print $2}’ |xargs kill -9For example to kill all active forms processes, we can use the following command:
ps -ef |grep applmgr |grep frm | grep -v grep |awk '{print $2}' |xargs kill -9
ps -ef | grep FNDLIBR | grep -v grep | awk '{print $2}' | xargs kill -9        -- to kill all FNDLIBR


Linux commands:
to check the load average :
$ cat  /proc/loadavg
2.70 2.45 2.13 1/450 6959
Use the below command to check the number of CPU cores on the server.
$ grep 'process' /proc/cpuinfo | wc -l
32
[oracle@UATORADB2 ~]$ cat /proc/cpuinfo |grep core|wc -l
48
[oracle@UATORADB2 ~]$ cat /proc/cpuinfo |grep proc|wc -l
24
--To check process count by each user
ps hax -o user | sort | uniq -c
Finding the processes that are causing high I/O
iotop
to check the proccess reads and writes:
cat /proc/16528/io
Finding what files are being written too heavily:
lsof -p 16528



profile options recently changed in EBS R12

select p.profile_option_name SHORT_NAME, n.user_profile_option_name "PROFILE NAME",
decode(v.level_id, 10001, 'Site', 10002, 'Application',
10003, 'Responsibility', 10004, 'User', 10005, 'Server',
10007, 'SERVRESP', 'UnDef') LEVEL_SET,
decode(to_char(v.level_id), '10001', '',
'10002', app.application_short_name, '10003', rsp.responsibility_key,
'10005', svr.node_name, '10006', org.name, '10004', usr.user_name,
'10007', 'Serv/resp', 'UnDef') "CONTEXT", v.profile_option_value VALUE, v.LAST_UPDATE_DATE  
from fnd_profile_options p,
fnd_profile_option_values v,
fnd_profile_options_tl n,
fnd_user usr,
fnd_application app,
fnd_responsibility rsp,
fnd_nodes svr,
hr_operating_units org
where p.profile_option_id = v.profile_option_id (+)
and p.profile_option_name = n.profile_option_name
--and upper(n.user_profile_option_name) like upper('BNE%')
--and trunc(v.LAST_UPDATE_DATE) > trunc(sysdate-170)
and usr.user_id (+) = v.level_value
and rsp.application_id (+) = v.level_value_application_id
and rsp.responsibility_id (+) = v.level_value
and app.application_id (+) = v.level_value
and svr.node_id (+) = v.level_value
and org.organization_id (+) = v.level_value
and v.LAST_UPDATE_DATE is not null 
order by last_update_date desc, short_name, level_set;   

concurrent managers were not starting

Once after doing Rapid cloning some of concurrent managers were not starting.
The issue was that the node was wrongly defined as source node in Administer Manager screen.
Then i checked in the FND_NODES table which wrongly had the source nodes as well apart from the target nodes.
I performed the following action plan which resolved the issue->
* Shutdown the apps services

* EXEC FND_CONC_CLONE.SETUP_CLEAN;
   COMMIT;
   EXIT;

adrelink.sh force=y link_debug=y "fnd FNDLIBR"
adrelink.sh force=y link_debug=y "fnd FNDFS"
adrelink.sh force=y link_debug=y "fnd FNDCRM"
adrelink.sh force=y link_debug=y "fnd FNDSM"


* Ran AutoConfig on all tiers, firstly on the DB tier and then the APPS tiers.
* Start the apps services.
Actually FND_CONC_CLONE.SETUP_CLEAN clears up the tables with nodes information and when we run autoconfig, it repopulates these tables with correct node information.

Database monitor Scripts

DB UP TIME
select SYSDATE-logon_time "Days", (SYSDATE-logon_time)*24 "Hours"
 from sys.v_$session
 where sid=1 /* this is PMON */
 /
 DB SIZE (Database Size,Used space,Free space) 
col "Database Size" format a20
 col "Free space" format a20
 col "Used space" format a20
 select round(sum(used.bytes) / 1024 / 1024 / 1024 ) || ' GB' "Database Size"
 , round(sum(used.bytes) / 1024 / 1024 / 1024 ) -
 round(free.p / 1024 / 1024 / 1024) || ' GB' "Used space"
 , round(free.p / 1024 / 1024 / 1024) || ' GB' "Free space"
 from (select bytes
 from v$datafile
 union all
 select bytes
 from v$tempfile
 union all
 select bytes
 from v$log) used
 , (select sum(bytes) as p
 from dba_free_space) free
 group by free.p;

Database grouth year wise
select
 to_char(b.CREATION_TIME,'YYYY'),
 count(1) "NO_OF_FILES",
 sum( round(b.bytes/1024/1024,2)) "SIZE_IN_MB"
 from v$datafile b,dba_data_files a where a.file_id=b.file#
 group by to_char(b.CREATION_TIME,'YYYY')
 order by to_char(b.CREATION_TIME,'YYYY') desc;

Database grouth monthly wise
select to_char(creation_time, 'RRRR Month') "Month",
 sum(bytes)/1024/1024 "Growth in Meg"
 from sys.v_$datafile
 where creation_time > SYSDATE-365
 group by to_char(creation_time, 'RRRR Month');

Calculate total space for all datafiles within database.
select DF.TOTAL/1048576 "DataFile Size Mb",
 LOG.TOTAL/1048576 "Redo Log Size Mb",
 CONTROL.TOTAL/1048576 "Control File Size Mb",
 (DF.TOTAL + LOG.TOTAL + CONTROL.TOTAL)/1048576 "Total Size Mb"
 from dual,
 (select sum(a.bytes) TOTAL from dba_data_files a) DF,
 (select sum(b.bytes) TOTAL from v$log b) LOG,
 (select sum((cffsz+1)*cfbsz) TOTAL from x$kcccf c) CONTROL;

Calculate space for list of tablespaces within primary database.
select DF.TOTAL/1048576 "DataFile Size Mb",
 LOG.TOTAL/1048576 "Redo Log Size Mb",
 CONTROL.TOTAL/1048576 "Control File Size Mb",
 (DF.TOTAL + LOG.TOTAL + CONTROL.TOTAL)/1048576 "Total Size Mb" from dual,
 (select sum(a.bytes) TOTAL from dba_data_files a
 where tablespace_name in('SYSTEM','UNDOTBS1', 'SYSAUX', 'USERS')) DF,
 (select sum(b.bytes) TOTAL from v$log b) LOG,
 (select sum((cffsz+1)*cfbsz) TOTAL from x$kcccf c) CONTROL;

Schema SIZE
select sum(bytes)/1024/1024/1024 from dba_segments where owner=upper('&owner');