EWCP: CF Contention caused by XCF Signalling Flood Merged

Copied from old SHARE Requirements system SSEWCP03001: Voting Priority=2.8

+5=2 +4=4 +3=9 +2=2 +1=3 abstain=3 -1=1 Priority=2.8

Coupling Facility Contention caused by XCF Signalling Flooding (PDMD0502-452)
CF requests are generally non-prioritized. At times a CF may become flooded
with work in the form of CF requests. The requests may relate to both high and
low importance work. When CF CPU resources become constrained, CF response
times increase as a consequence. This is especially an issue for synchronous
CF requests such as IRLM lock requests, GRS ISGLOCK and ECS requests, but other
requests are affected as well. Under severe CF constraints the MVS sender cost
related to synchronous request may increase to an unacceptable level (most or
all of the remaining MVS cycles are lost as a consequence due to MVS
"dwelling").

Examples of CF "hogs" include but is not limited to VTAM High Performance
Routing (HPR). HPR uses XCF signalling. VTAM HPR traffic may for example be
due to DB2 DDF requests or file transfers (FTPs) of large amounts of data. We
generally have followed IBM recommendations for setup of a high availability
and performance XCF signalling environment. We only have limited use of VTAM
CTCs. In our sysplex systems we occasionally experience large bursts of
Communications Server (IP or VTAM) inter-MVS traffic. In addition to path
contention and subchannel contention, performance of the CF and thus
performance of the entire sysplex is affected negatively.

Further compounding these issues we have in our particular case large and
rather unpredictable exchanges of data between several sysplexes due to the way
we synchronize data across sysplexes. Also we experience occasionally heavy
queries from Business Intelligence (BI) type work. These data traffic elements
we expect to grow as times goes due to lower future network latency. The
business also requires the results from up-to-the minute BI type queries. Last
but not least, and in addition to the nagging performance problems described
above, we have experiences several system and/or sysplex outages due to the
severe CF constraint situations.

Workload manger (WLM) is generally unaware of CF contention. WLM generally
manages CPU, Storage and I/O resources. As a consequence WLM controls are not
able to distinguish between less important and critical work when it comes to
CF contention.

Although the emphasis in the description above has been on VTAM HPR XCF CF
messages it should be noted that any XCF message generator is potentially able
to affect the operation and availability of a sysplex since no guards or
governors are in place to limit their activity.

Problem Bypass:

Generally we have not found an effective bypass solution to our problem,
however several routes are being pursued:

We have considered to direct some or all of the traffic via other (non-XCF)
means for example by using Enterprise Extender via OSA channels. Intra-CEC
traffic may also in the future use Hipersockets to expedite matters.
We have considered to "throttle" the CS traffic, but this does not appear easy
and is generally not a pro-active mechanism but rather reacting to messages
etc.
Establish a separate CF configuration for the HPR traffic XCF messages, using
special and dedicated CF resources (CF LPAR with adequate CF CPU, storage and
link capacity). This solution makes systems programming and operation more
complex. Also we need dedicated links since they are not shareable at the
receiver end. This is a costly solution and may use CF resources in a
sub-optimal way.
Establishing full CTC connectivity between all our many MVS systems may be a
way to off-load the VTAM HPR traffic but performance and flexibility is
affected.
Change the network topology in such a way that inter MVS data traffic is
smaller. This is possible only if we can minimize extra hops when one MVS is
transmitting data to another.