1.We upgraded the oracle database from 10g to 11g but at the post upgrade step
after 46% we found the ora 600 error
2.The alert.logh file and the trace.log file depicts the following ORA-600error message -:
Wed Apr 25 12:01:49 2012
Errors in file /oracle/JEP/saptrace/usertrace/jep_ora_7209412.trc:
ORA-00600: internal error code, arguments: [17069], [0x7000002BCF79448],Wed Apr 25 12:02:07 2012
Errors in file /oracle/JEP/saptrace/usertrace/jep_ora_6881478.trc:
ORA-00600: internal error code, arguments: [17069], [0x7000002BEA72D88],Shutting down instance: further logons disabled
Could you please provide the trace files as an attachment to the message.
Also, please provide the output of opatch lsinventory
3.Please find the attached trace file & lsinventory logs
4.Could You please let me know if all the pre upgrade tasks were performed. Also, did you encounter any INVALID objects prior to the upgrade.
Please perform the following tasks prior to the upgrade.
1) update the Dictionary Statistics as per SAP note
838725 - Oracle dictionary statistics and system statistics
2) run the following statements in SQLPLUS and provide the output:
select count(*) from v$rman_backup_job_details
where status = 'RUNNING';
check for invalid objects in the database:
SELECT substr(owner,1,12) owner,
SUBSTR(object_name,1,30) object,
SUBSTR(object_type,1,30) type,
status
FROM dba_objects
WHERE status <>'VALID' AND owner = 'SYS';
Please refer the Note 648203 when INVALID objects are encountered
5.I am providing you the output of your below commands.
Also we found around 3096 INVALID objects.
As per your note 648203 is for oracle upgrade 9i to 10G
We are doing upgrade of 10g to 11g so the note is compatible ?
6.Please execute the following command -:
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
Again check the output of the below command and update the message
with the output -:
SQL> select object_name,object_type,owner from dba_objects where
status='INVALID';
Also, the note 648203 can be referred to validate the INVALID segements
in this case also.
7.There are many objects with the INVALID status -:
orajep> sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Mon Apr 30 21:14:14 2012
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProducWith the Partitioning, Data Mining and Real Application Testing options
SQL> select count(*) from dba_objects where status='INVALID';
COUNT(*)
----------
5031
SQL>
8.Did you execute the scripts mentioned in my previous reply .
What was the operation perfromed recently as the number of INVALID
objects have increased from 3096 to 5031.
The creation date for many objects shows 25 April 2012.
Examples -:
---------- ------------------------------ --------------------
CREATED
--------------------
PUBLIC DBA_COMPARISON_ROW_DIF SYNONYM
25-APR-12
PUBLIC USER_COMPARISON_ROW_DIF SYNONYM
25-APR-12
PUBLIC _USER_COMPARISON_ROW_DIF SYNONYM
25-APR-12
SQL> select count(*) from dba_objects where status='INVALID' and createdbetween '25-APR-2012 00:00:00' and '26-APR-2012 00:00:00';
COUNT(*)
----------
1579
Could you please let me know the deatiled description of what
has happend till now -:
9.We are upgrading the oracle from 10g to 11g in that before down time we
have upgraded the software & applied the patch successful after that we
have started the down time & in that we are upgrading the database.
At that time first task database upgrade is completing successfully
after that it is going to the next step in assistant for post upgrade
and reaching to the 46% & giving all these errors. when I checked I am
getting ora-600 error every where.
The no of object is invalid increased after running the script
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
As per your previous mail.
Please check again & suggest
10. the script utlrp.sql would not
lead to the enhancing the numebr of invalid objects.
The recommended was provided on 27.04.2012 13:03:35.
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
The INVALID objects were created on 25-APR-2012.
SQL> select count(*) from dba_objects where status='INVALID' and createdbetween '25-APR-2012 00:00:00' and '26-APR-2012 00:00:00';
COUNT(*)
---------------------------------
1579
Since you have a vlaid backup of the system prior to the upgrade , I
would request you to please restore to the valid backup and again
and please do the preupgrade checks and update the message with the
preupgrade checks.
11. We have restored the backup before
downtime after completion of pre requisite steps.
SAP & DB is up & running.
Please check & suggest to upgrade the same.
12.I got the error at 7% and as I am aware of the issue as well.
We did not change the hostname in tnsnames.ora file at new
location /oracle/JEP/11203/network/admin
But as we discussed at any error I will send you the message so not re
correcting.
Change the ORACLE_HOME from 102_64 to 112_64 in listener.ora or tnsname.ora.
It is asking me to revert back the step. please suggest.
Please check the attached screen shot & suggest.
13.Could You please upload all the required files
-: ORACLE_Server.log
-: trace.log
-: Alert.log file
-: Output of the command uname -a
-: Files under the directory
$ORACLE_HOME/cfgtoollogs/dbua/logs
14.I got the error at 43% but i didn't press
ok button so in background process is running & it has completed to 50%
It is still running & completed 73% only in full night
15.This is not an
expected behaviour of the DBUA . It should have been completed
16. Solution 1st Step
This issue is related to the AIX operating system.
Please restore the valid backup , apply the fix mentioned below
and start the upgrade.
The following AIX fix needs to be applied in order to get rid of this
issue:
For AIX 5.3 apply IFIX IV10538:
https://www-304.ibm.com/support/docview.wss?uid=isg1IV10538
For AIX 6.1 apply IFIX IV09580:
https://www-304.ibm.com/support/docview.wss?uid=isg1IV09580
For AIX 7.1 apply IFIX IV09541:
https://www-304.ibm.com/support/docview.wss?uid=isg1IV09541
IBM also has pre-built the iFixes and uploaded to:
ftp://public.dhe.ibm.com/aix/efixes/
where the APAR numbers are:
5.3 TL11 - iv10538
5.3 TL12 - iv11158
6.1 TL4 - iv11167
6.1 TL5 - iv10576
6.1 TL6 - iv10539
6.1 TL7 - iv09580
7.1 TL0 - unaffected
7.1 TL1 - iv09541
Once the OS patch is applied, relink oracle binary so it will be
effective.
17.We have applied the below mentioned AIX fix & restored the backup again.
After that we made the Oracle & SAP up & now i have stop sap
apllication & listener & starting DB upgrade assistant but at the
starting point only I am getting ora-600 error.
Also checked the alertlog file & as per file we have set the parameter
_disable_image_check = true
So, please take the connection & suggest.Also attaching the screen shot
for your reference.
18.Also, please let me know if all the undescore parameters are commeneted
out for the upgrade
19.The issue depicted in the alert.log file is -:
Errors in file /oracle/JEP/saptrace/background/jep_m000_1179780.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() -
unknown KQ], [0], [], [], [], [], [], []
Thu May 10 16:30:33 2012
Errors in file /oracle/JEP/saptrace/background/jep_m000_1179780.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() -
unknown KQ], [0], [], [], [], [], [], []
This shows that the Oracle executables were not linked/compiled properly.The solution to the issue would be -:
Shutdown all instances and processes, relink the Oracle executable
and restart the database.( Note -: please ensure that the database,
listener etc. are brought down prior to relinking).
Please relink the oracle exevcutables for oracle 11g also.
Please refer the note -: 97953 and ensure that the oracle executables
are relinked properly.
20.At the time of relink all executables I am running the below command &
getting the error as per attachment.
make -f ins_rdbms.mk install
make: 1254-002 Cannot find a rule to create target hsodbc from
dependencies
21.please find out the log after executing command renink all.
ld: 0711-224 WARNING: Duplicate symbol: p_xrc
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more
information.
mv -
f /oracle/JEP/112_64/bin/nmosudo /oracle/JEP/112_64/bin/nmosudo0
mv /oracle/JEP/112_64/sysman/lib/nmosudo /oracle/JEP/112_64/bin/ /bin/make -f /oracle/JEP/112_64/sysman/lib/ins_emagent.mk
emtgtctl2
ld -b64 -o emtgtctl2 -L/oracle/JEP/112_64/lib/ -
L/oracle/JEP/112_64/sysman/lib/ -lld -lm
`cat /oracle/JEP/112_64/lib/sysliblist` -lm -brtl -bexpall -
bnoerrmsg -
blibpath:/oracle/JEP/112_64/lib/:/oracle/JEP/112_64/sysman/lib/:/oracle/JEP/11203/jdk/jre/bin/classic:/oracle/JEP/11203/jdk/jre/bin:/usr/lib:/lib
/oracle/JEP/112_64/sysman/lib//s0nmetgctl.o -lnmectl -lclntsh -brtl -L/oracle/JEP/112_64/lib -L/oracle/JEP/112_64/sysman/lib/ -lnmemso -
lcore11 -L/oracle/JEP/11203/jdk/jre/bin/classic -
L/oracle/JEP/11203/jdk/jre/bin -ljava -ljvm -lld -lm
`cat /oracle/JEP/112_64/lib/sysliblist` -lm
rm -f /oracle/JEP/112_64/bin/emtgtctl2
mv emtgtctl2 /oracle/JEP/112_64/bin/emtgtctl2
/bin/chmod 6751 /oracle/JEP/112_64/bin/emtgtctl2
Enterprise Manager native components relinked
IMPORTANT NOTE: To complete this Install/upgrade, please login as root
and
execute the root.sh script.
/bin/make -f /oracle/JEP/112_64/sysman/lib/ins_emagent.mk
relink_exe EXENAME=nmccollector
/bin/ld -b64 -
o /oracle/JEP/112_64/sysman/lib/nmccollector -L/oracle/JEP/112_64/lib/
`cat /oracle/JEP/112_64/lib/sysliblist` -
blibpath:/oracle/JEP/112_64/lib/:/usr/lib -berok -bbigtoc -
bnoerrmsg /oracle/JEP/112_64/sysman/lib/snmccolm.o /oracle/JEP/112_64/sysman/lib/libnmccol.a /oracle/JEP/112_64/sysman/lib/libnmcbuf.a
/oracle/JEP/112_64/lib/libclntsh.so /oracle/JEP/112_64/rdbms/lib/libdsga11.a -
bI:/lib/ksms.imp /oracle/JEP/112_64/lib/libserver11.a /oracle/JEP/112_64/lib/libgeneric11.a /oracle/JEP/112_64/lib/libttsh11.so -lld -lm
`cat /oracle/JEP/112_64/lib/sysliblist` -lm -
L/oracle/JEP/112_64/rdbms/lib
ld: 0706-003 Cannot find or read import file: /lib/ksms.imp
ld:accessx(): A file or directory in the path name does not
exist.
make: 1254-004 The error code from the last command is 255.
Stop.
make: 1254-004 The error code from the last command is 2.
Stop
22. Solution 2nd step
Please refer the note Note 97953 - UNIX: Relinking of the Oracle executables
I did set the ORACLE_HOME, PATH and LD_LIBRARY_PATH variables for
11g and the relink went fine.
I have executde the relink all.
orajep>cd $ORRACLE_HOME/install
orajep> relink all
writing relink log to: /oracle/JEP/112_64/install/relink.log
orajep>
Please check the logfiles and updat ethe message
23.I started the dbua.sap.sh again.
This time the database upgrade completed successfully upto 100%
after 46% we found the ora 600 error
2.The alert.logh file and the trace.log file depicts the following ORA-600error message -:
Wed Apr 25 12:01:49 2012
Errors in file /oracle/JEP/saptrace/usertrace/jep_ora_7209412.trc:
ORA-00600: internal error code, arguments: [17069], [0x7000002BCF79448],Wed Apr 25 12:02:07 2012
Errors in file /oracle/JEP/saptrace/usertrace/jep_ora_6881478.trc:
ORA-00600: internal error code, arguments: [17069], [0x7000002BEA72D88],Shutting down instance: further logons disabled
Could you please provide the trace files as an attachment to the message.
Also, please provide the output of opatch lsinventory
3.Please find the attached trace file & lsinventory logs
4.Could You please let me know if all the pre upgrade tasks were performed. Also, did you encounter any INVALID objects prior to the upgrade.
Please perform the following tasks prior to the upgrade.
1) update the Dictionary Statistics as per SAP note
838725 - Oracle dictionary statistics and system statistics
2) run the following statements in SQLPLUS and provide the output:
select count(*) from v$rman_backup_job_details
where status = 'RUNNING';
check for invalid objects in the database:
SELECT substr(owner,1,12) owner,
SUBSTR(object_name,1,30) object,
SUBSTR(object_type,1,30) type,
status
FROM dba_objects
WHERE status <>'VALID' AND owner = 'SYS';
Please refer the Note 648203 when INVALID objects are encountered
5.I am providing you the output of your below commands.
Also we found around 3096 INVALID objects.
As per your note 648203 is for oracle upgrade 9i to 10G
We are doing upgrade of 10g to 11g so the note is compatible ?
6.Please execute the following command -:
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
Again check the output of the below command and update the message
with the output -:
SQL> select object_name,object_type,owner from dba_objects where
status='INVALID';
Also, the note 648203 can be referred to validate the INVALID segements
in this case also.
7.There are many objects with the INVALID status -:
orajep> sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Mon Apr 30 21:14:14 2012
Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit ProducWith the Partitioning, Data Mining and Real Application Testing options
SQL> select count(*) from dba_objects where status='INVALID';
COUNT(*)
----------
5031
SQL>
8.Did you execute the scripts mentioned in my previous reply .
What was the operation perfromed recently as the number of INVALID
objects have increased from 3096 to 5031.
The creation date for many objects shows 25 April 2012.
Examples -:
---------- ------------------------------ --------------------
CREATED
--------------------
PUBLIC DBA_COMPARISON_ROW_DIF SYNONYM
25-APR-12
PUBLIC USER_COMPARISON_ROW_DIF SYNONYM
25-APR-12
PUBLIC _USER_COMPARISON_ROW_DIF SYNONYM
25-APR-12
SQL> select count(*) from dba_objects where status='INVALID' and createdbetween '25-APR-2012 00:00:00' and '26-APR-2012 00:00:00';
COUNT(*)
----------
1579
Could you please let me know the deatiled description of what
has happend till now -:
9.We are upgrading the oracle from 10g to 11g in that before down time we
have upgraded the software & applied the patch successful after that we
have started the down time & in that we are upgrading the database.
At that time first task database upgrade is completing successfully
after that it is going to the next step in assistant for post upgrade
and reaching to the 46% & giving all these errors. when I checked I am
getting ora-600 error every where.
The no of object is invalid increased after running the script
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
As per your previous mail.
Please check again & suggest
10. the script utlrp.sql would not
lead to the enhancing the numebr of invalid objects.
The recommended was provided on 27.04.2012 13:03:35.
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
The INVALID objects were created on 25-APR-2012.
SQL> select count(*) from dba_objects where status='INVALID' and createdbetween '25-APR-2012 00:00:00' and '26-APR-2012 00:00:00';
COUNT(*)
---------------------------------
1579
Since you have a vlaid backup of the system prior to the upgrade , I
would request you to please restore to the valid backup and again
and please do the preupgrade checks and update the message with the
preupgrade checks.
11. We have restored the backup before
downtime after completion of pre requisite steps.
SAP & DB is up & running.
Please check & suggest to upgrade the same.
12.I got the error at 7% and as I am aware of the issue as well.
We did not change the hostname in tnsnames.ora file at new
location /oracle/JEP/11203/network/admin
But as we discussed at any error I will send you the message so not re
correcting.
Change the ORACLE_HOME from 102_64 to 112_64 in listener.ora or tnsname.ora.
It is asking me to revert back the step. please suggest.
Please check the attached screen shot & suggest.
13.Could You please upload all the required files
-: ORACLE_Server.log
-: trace.log
-: Alert.log file
-: Output of the command uname -a
-: Files under the directory
$ORACLE_HOME/cfgtoollogs/dbua/logs
14.I got the error at 43% but i didn't press
ok button so in background process is running & it has completed to 50%
It is still running & completed 73% only in full night
15.This is not an
expected behaviour of the DBUA . It should have been completed
16. Solution 1st Step
This issue is related to the AIX operating system.
Please restore the valid backup , apply the fix mentioned below
and start the upgrade.
The following AIX fix needs to be applied in order to get rid of this
issue:
For AIX 5.3 apply IFIX IV10538:
https://www-304.ibm.com/support/docview.wss?uid=isg1IV10538
For AIX 6.1 apply IFIX IV09580:
https://www-304.ibm.com/support/docview.wss?uid=isg1IV09580
For AIX 7.1 apply IFIX IV09541:
https://www-304.ibm.com/support/docview.wss?uid=isg1IV09541
IBM also has pre-built the iFixes and uploaded to:
ftp://public.dhe.ibm.com/aix/efixes/
where the APAR numbers are:
5.3 TL11 - iv10538
5.3 TL12 - iv11158
6.1 TL4 - iv11167
6.1 TL5 - iv10576
6.1 TL6 - iv10539
6.1 TL7 - iv09580
7.1 TL0 - unaffected
7.1 TL1 - iv09541
Once the OS patch is applied, relink oracle binary so it will be
effective.
17.We have applied the below mentioned AIX fix & restored the backup again.
After that we made the Oracle & SAP up & now i have stop sap
apllication & listener & starting DB upgrade assistant but at the
starting point only I am getting ora-600 error.
Also checked the alertlog file & as per file we have set the parameter
_disable_image_check = true
So, please take the connection & suggest.Also attaching the screen shot
for your reference.
18.Also, please let me know if all the undescore parameters are commeneted
out for the upgrade
19.The issue depicted in the alert.log file is -:
Errors in file /oracle/JEP/saptrace/background/jep_m000_1179780.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() -
unknown KQ], [0], [], [], [], [], [], []
Thu May 10 16:30:33 2012
Errors in file /oracle/JEP/saptrace/background/jep_m000_1179780.trc:
ORA-00600: internal error code, arguments: [qksfroFXTStatsLoc() -
unknown KQ], [0], [], [], [], [], [], []
This shows that the Oracle executables were not linked/compiled properly.The solution to the issue would be -:
Shutdown all instances and processes, relink the Oracle executable
and restart the database.( Note -: please ensure that the database,
listener etc. are brought down prior to relinking).
Please relink the oracle exevcutables for oracle 11g also.
Please refer the note -: 97953 and ensure that the oracle executables
are relinked properly.
20.At the time of relink all executables I am running the below command &
getting the error as per attachment.
make -f ins_rdbms.mk install
make: 1254-002 Cannot find a rule to create target hsodbc from
dependencies
21.please find out the log after executing command renink all.
ld: 0711-224 WARNING: Duplicate symbol: p_xrc
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more
information.
mv -
f /oracle/JEP/112_64/bin/nmosudo /oracle/JEP/112_64/bin/nmosudo0
mv /oracle/JEP/112_64/sysman/lib/nmosudo /oracle/JEP/112_64/bin/ /bin/make -f /oracle/JEP/112_64/sysman/lib/ins_emagent.mk
emtgtctl2
ld -b64 -o emtgtctl2 -L/oracle/JEP/112_64/lib/ -
L/oracle/JEP/112_64/sysman/lib/ -lld -lm
`cat /oracle/JEP/112_64/lib/sysliblist` -lm -brtl -bexpall -
bnoerrmsg -
blibpath:/oracle/JEP/112_64/lib/:/oracle/JEP/112_64/sysman/lib/:/oracle/JEP/11203/jdk/jre/bin/classic:/oracle/JEP/11203/jdk/jre/bin:/usr/lib:/lib
/oracle/JEP/112_64/sysman/lib//s0nmetgctl.o -lnmectl -lclntsh -brtl -L/oracle/JEP/112_64/lib -L/oracle/JEP/112_64/sysman/lib/ -lnmemso -
lcore11 -L/oracle/JEP/11203/jdk/jre/bin/classic -
L/oracle/JEP/11203/jdk/jre/bin -ljava -ljvm -lld -lm
`cat /oracle/JEP/112_64/lib/sysliblist` -lm
rm -f /oracle/JEP/112_64/bin/emtgtctl2
mv emtgtctl2 /oracle/JEP/112_64/bin/emtgtctl2
/bin/chmod 6751 /oracle/JEP/112_64/bin/emtgtctl2
Enterprise Manager native components relinked
IMPORTANT NOTE: To complete this Install/upgrade, please login as root
and
execute the root.sh script.
/bin/make -f /oracle/JEP/112_64/sysman/lib/ins_emagent.mk
relink_exe EXENAME=nmccollector
/bin/ld -b64 -
o /oracle/JEP/112_64/sysman/lib/nmccollector -L/oracle/JEP/112_64/lib/
`cat /oracle/JEP/112_64/lib/sysliblist` -
blibpath:/oracle/JEP/112_64/lib/:/usr/lib -berok -bbigtoc -
bnoerrmsg /oracle/JEP/112_64/sysman/lib/snmccolm.o /oracle/JEP/112_64/sysman/lib/libnmccol.a /oracle/JEP/112_64/sysman/lib/libnmcbuf.a
/oracle/JEP/112_64/lib/libclntsh.so /oracle/JEP/112_64/rdbms/lib/libdsga11.a -
bI:/lib/ksms.imp /oracle/JEP/112_64/lib/libserver11.a /oracle/JEP/112_64/lib/libgeneric11.a /oracle/JEP/112_64/lib/libttsh11.so -lld -lm
`cat /oracle/JEP/112_64/lib/sysliblist` -lm -
L/oracle/JEP/112_64/rdbms/lib
ld: 0706-003 Cannot find or read import file: /lib/ksms.imp
ld:accessx(): A file or directory in the path name does not
exist.
make: 1254-004 The error code from the last command is 255.
Stop.
make: 1254-004 The error code from the last command is 2.
Stop
22. Solution 2nd step
Please refer the note Note 97953 - UNIX: Relinking of the Oracle executables
I did set the ORACLE_HOME, PATH and LD_LIBRARY_PATH variables for
11g and the relink went fine.
I have executde the relink all.
orajep>cd $ORRACLE_HOME/install
orajep> relink all
writing relink log to: /oracle/JEP/112_64/install/relink.log
orajep>
Please check the logfiles and updat ethe message
23.I started the dbua.sap.sh again.
This time the database upgrade completed successfully upto 100%
No comments:
Post a Comment