R12 Upgrade Performance Tuning: How to Dramatically Reduce Downtime
There are no magic touches or hacks or backdoors in this information – only a summary of the steps to reduce production downtime for major application upgrades (e.g., 11i to R12). Ready? Let’s go.
Infrastructure Tuning (or, “Trying to squeeze blood from the stone”)
- Get the maximum CPUs allowed by your license agreement with Oracle for your DB box. Use L-PAR, N-PAR hardware servers or pump up your Virtual Machine.
- Move your DB to the high-performance storage – for example, on Oracle ASM, you can simply attach volumes from new storage and do the sync on a background.
- If your DB is on NFS,, implement Oracle dNFS as soon as possible. Why? For one of our customers, dNFS increased the upgrade speed by 3.5x.
- Implement “huge pages” to offload your CPUs.
- Do not forget to fix Random Generator on your application servers – it will boost performance for java process.
Just a disclaimer – all parameter changes below need to be adjusted for the upgrade run only. Don’t forget to revert them back to your normal operational parameters after the upgrade is completed!
- Adjust the following parameters:
- Set job_queue_processes = # of CPUS
- Set parallel_max_servers = 2 X CPUs
- Double java_pool_size
- Make sure your streams_pool_size is not < 250M
- Set _pga_max_size
- Increase sga_target to 80% of your physical memory
- Turn off your RAC (if you have one). Keep only one DB node up for upgrade – multi-node RAC configuration will degrade performance of your upgrade process significantly. (By the way: I’ll try to describe HA/RAC configuration for EBS in my next post – stay tuned!)
- Re-create your redo-logs – no mirrors, 2GB+
- Make sure you’ve applied the latest PSU for your DB version (especially if you’re doing an 11i to R12 upgrade with DB upgrade as well).
- Turn off archive logging – use interim snapshot backups for rollback points.
- Gather statistics with higher estimate.
This process can be logically split into few parts:
- Skip what you can skip:
- SLA history – do only 3-6 months during cutover and do the rest of the history later on. (It can be done on background using SLA Hotpatch). Believe me – your Finance department will not use the whole history until the period closes.
- Application Accounting Definition (AAD) validation can be done after the upgrade.
- Discoverer EUL Upgrade, MFS and localization additions can be done in parallel with post-upgrade steps.
- Purge what you can purge.
- Communicate with your business SMEs to revise your purging policy for business transactions. Our team did a huge purging of business transactions for a customer and reduced their DB size by 30% of the original.
- Merge 12.1.1 with 12.1.1 pre-install RUP (or 12.2.0 with pre-install performance one-offs)
- Adjust the number of parallel workers (adworker) carefully. Maximum (of cause) is 2xDB CPUs cores, BUT in the case of 24 cores, DB box = 48 workers will slow down your upgrade.
- Distributed AD (use of multiple application nodes to run ADPATCH in parallel) is not really effective compared with staging (see below).
- Implement staging. We use a “three-stage schema.” It allows us to skip copy/generate portions and run only the DB portion of the upgrade. This gives a significant boost for multi-NLS environments. For example:
- Stage 1 = 12.1.1
- Stage 2 = Stage 1 + 12.1.3
- Stage 3 = Stage 2 + RPC + post-upgrade patches + NLS Sync patchsets + CEMLIs
- Order NLS Sync patchsets in Oracle instead of download and apply it for each patch.
- Push application development team to “package” remediated CEMLIs into the one pack.
- Use “more hands on keyboard” for final post-upgrade part.
« Back to blog