วันจันทร์ที่ 17 กันยายน พ.ศ. 2555

Warning "aiowait timed out x times"


When the above message is produced in alert.log, the database may also
appear to be hanging.

The message indicates that Oracle has encountered problems while trying
to perform asynchronous I/O.

When the Oracle instance is configured to run with asynchronous I/O
(disk_asynch_io = true), it will use the aiowait system call to obtain the
completion status of such I/Os.

When you using truss (On Solaris you will see)

28463/1:        open("/etc/passwd", O_RDONLY)                   = 3
28463/1:        kaio(5, 0xFFFFFFE8, 0xFFFFFFFF, 0x00000000, 0x00000000, 0x000000
00, 0xEF7F4298) = 0
28463/1:        kaio(AIOREAD, 3, 0x00020FF0, 1024, 0, 0xEFFFF9FC) Err#48
28463/1:        lwp_create(0xEFFFF628, 0, 0xEF781F44)           = 3
28463/3:        lwp_create(0x00000000, 0, 0x00000000)           = -277341360
28463/3:        lwp_self()                                      = 3
28463/1:        lwp_create(0xEFFFF628, 0, 0xEF778F44)           = 4
28463/4:        lwp_create(0x00000000, 0, 0x00000000)           = -277378224
28463/4:        lwp_self()                                      = 4
28463/1:        lwp_create(0xEFFFF628, 0, 0xEF6BAF44)           = 5
28463/5:        lwp_create(0x00000000, 0, 0x00000000)           = -278156464
28463/5:        lwp_self()                                      = 5
28463/1:        lwp_create(0xEFFFF628, 0, 0xEF6B1F44)           = 6
28463/6:        lwp_create(0x00000000, 0, 0x00000000)           = -278193328
28463/6:        lwp_self()                                      = 6          
28463/3:        pread(3, " r o o t : x : 0 : 1 : 0".., 1024, 0) = 1024
28463/4:        pread(3, " r y h a : x : 2 1 6 : 1".., 1024, 1024) = 1024
28463/5:        pread(3, " s 1 / j c a h i l l : /".., 1024, 2048) = 1024
28463/6:        pread(3, " s h\n s f a x : x : 5 0".., 1024, 3072) = 115

This is the clever bit. The KAIO kernel driver cannot operate on a filesystem file, so as a result of the first kaio(AIOREAD) failing, the aioread() function
spawns 4 lightweight processes (threads). Each of these threads is responsible for reading a different section of the file, but in parallel; thereby SIMULATING asynchronous I/O.

There are a number of actions that can be done to deal with this problem:

#1) Restart the database without asynchronous I/O

This is not an ideal solution and will only serve as a workaround until the underlying problem with asynchronous I/O is identified and fixed at the O/S level.

To operate without asynchronous I/O set the init.ora parameter 

disk_asynch_io = false

At the same time you should also set the following parameter, dbwr_io_slaves, to a value other than 0. This will allow Oracle to simulate asynchronous I/O through usage of multiple I/O slave processes where each such process performs I/O synchronously.

#2) Reduce I/O load on the system from this and other databases

Again this is only an attempt to avoid the underlying condition that causes asynchronous I/O to fail in cases where this happens under heavy load.

This is only a temporary measure until the problem is found and fixed e.g. by applying O/S patches.

Possible actions are:
- avoid running heavy batch jobs known to generate lots of I/O
- allocate more memory to the Buffer Cache and Sort Areas to reduce I/O from/to datafiles and temporary tablespaces
(relevant parameters: db_block_buffers, db_cache_size,
pga_aggregate_target, sort_area_size, hash_area_size etc.)
- tune checkpoints to reduce frequency of write I/O by DBWR

#3) Check all Oracle prerequisites for O/S and I/O subsystem configuration

You will need to refer to Oracle platform-specific Installation manuals and Administrator's references and make sure all required O/S patches and kernel parameters are in place and that Asynchronous I/O is correctly configured.

Note 169706.1 Oracle® Database on AIX®,HP-UX®,Linux®,Mac OS® X

#4) Check for known O/S issues with Asynchronous I/O operation

#5)Check filesystem's disk and try FULL export to /dev/null to identify possible corruption. 


Hope this help you to more understand :D
Chatchai Komrangded

ไม่มีความคิดเห็น:

แสดงความคิดเห็น