TSO/E: Implement Pipes Facility

Can't you (IBM) decouple TSO Pipes from the Batchpipes product and release it as a way to improve customer productivity?

How much did IBM invest in adding pipes to Netview? It would have been a lot less if PIPES were part of the z/OS base which would then have only required the Netview team developing a few 'pipes' instead of the port. With z/OS 3.1 IBM has the perfect opportunity to improve the productivity of their customers who are looking at ways to improve productivity on their z systems.

Hi IBM,

The following comments are of my own and do not reflect that of my current or former empoyers in any way. I currently have (82817) open as well as a number of prior requests rejected over the course of several decades. I've tried formal as well as informal channels via our IBM SE over the years to no avail. Luckily this one is getting tracking (Thank you !!!). I'm not sure what it is that IBM would require from the end user-base to put this in as a no charge feature like is base code VM/CMS ? When it says "Future consideration" how far into the future would you say that is? This would help my day-to-day work and productivity greatly. I can't thank you (IBM) and Jon Hartmann enough. Please help us out! Thank you for or help and concideration with this matter.

Best regards,

Michael R.

This is what I find on my system: BatchPipes/MVS BatchPipeWorks, 5655-065/SCPSP 1.0109 (Version.Release/Mod) - Generated 12 Oct 1996 at 13:59:00

Yes it works, it is stable, but it is NOT current and the latest on z/VM has more capabilities.

Output of PIPEQ command:
FPLINX086I CMS/TSO Pipelines, 5654-030/5655-A17 1.0110 (Version.Release/Mod) - Generated 7 Apr 1998 at 14:00:22

Hi, I am surprised to learn that PIPE is not integrated within z/OS TSO Environment as it does affect Developers and z/OS Users across the Globe. We cannot develop Rexx coding to ship to our customers with PIPE commands as it does not provide a guarantee that they would have SmartBatch installed and is not even Free Product.

In times like now, the companies are not committing to Products which might increase the Budget and please integrate or make the product BatchPipes free of charge.

Lets do the RIGHT Thing by aligning with z/VM Platform.
This is what makes z/VM Folks laugh at z/OS development on the decisions used - Availability of PIPEs.

Regards
Jasi Grewal.

Oh, I spoke too soon. The same manual says: Level 1.1.9 of TSO Pipelines is incorporated into BatchPipes/MVS Release 2 (IBM Program Number 5655-065) under the name of BatchPipeWorks. This product is not being enhanced.

Reading the CMS Pipelines manual, it appears that TSO pipes is pretty much functionally the same; with obvious restrictions. So virtually all the data manipulation changes and/or filtering stages, that aren't operating system specific, are available in both environments. And the CMS manual documents TSO as well as CMS.

This should be embarrassing - the version of BatchPipes TSO Pipes is from the last millenium and is old enough to drink now.

BPW00086I BatchPipes/MVS BatchPipeWorks, 5655-065/SCPSP 1.0109 (Version.Release/Mod) - Generated 12 Oct 1996 at 13:59:00

I don't have native TSO Pipes so it would be interesting if someone with it could post the results of a PIPE Q command.

The requirement requests that TSO Pipelines be part of the base so that developers at IBM, customers, and ISVs can count on it and code for it. A separately licenced product does not address the requirement. Citing the licensed product is a smoke-screen. I encourage anyone who agrees to click "Report Abuse" on the comment by RalfMahlerREWEgroup.

This functionality (TSO PIPES) has been requested and is overdue for more than 25 years. It deserves higher priority than medium; I consider it just short of emergent. The fact that an out of date version is included in BatchPipes does not address the requirement. Charging z/OS customers for what z/VMers get for free and love makes no sense. Marketing the TSO PIPE command with BatchPipes is also irrelevant. The former is more flexible, easier to use and more general. The latter is limited, hard to use and useful only specific situations. The names, BatchPipes and BatchPipesWorks (TSO PIPE command) just create more confusion.

The functionality is there - you have only to license "5655-D45, IBM BatchPipes® for OS/390 V2.1, announced in Software Announcement 200-093, dated April 18, 2000"

There you can check which stages are supported:
http://publibfp.dhe.ibm.com/cgi-bin/bookmgr/BOOKS/asf1a411/CCONTENTS

There are slightly differncens between Netview Pipes and Batchpipeworks Pipes (i.e. SPECS instead of EDIT).

Please refer to Hobart Spitz's posting on IBM-MAIN dated Thu, 16Nov 2017, 08:36:09 titled "Pipelines in the z/OS base". Permission to include herein is granted by Mr. Spitz in the body of his post that follows below. Mr. Spitz is a long-standing proponent for fully implemented support of pipes throughout z/OS and TSO.

Date:    Thu, 16 Nov 2017 08:36:09 -0500
From:    Hobart Spitz
Subject: Pipelines in the z/OS base.

Someone on TSO-REXX wrote:
> Yes and if IBM would give us PIPES (BatchPipesWorks Product) then you can
use PIPE to read from a file into a STEM or STACK and vice-versa.

There have been other such posts.

For those who don't know TSO PIPEs is like an assembly line with reusable,
standardized stations.  (You can create your own, on the fly even.)

IMHO, making TSO PIPEs part of the z/OS base is long, long overdue.  Over
the last 20 years, I have been associated with two requirements for Pipes
in the OS base that have been submitted at SHARE, but which produced no
results.  I have it on good authority that that if you take all the
requests for TSO Pipelines, sort in REXX, and other requirements addressed
by TSO PIPEs, that there would be about 75% positive RFE votes.

Background on the term "piping":

The early versions of UNIX were written on small machines (32K?!), with
slow/small disk drives.  Writing temporary results out to disk was too slow
and inefficient, so they came up a pipe  operator ( | ), which connected
the input of one command to the output of the previous one, thus avoiding
physical I/O..  E.g. *ls | wc*.  The data passed thru an in-memory buffer
with both commands running, and UNIX dispatching each as needed.  It also
allowed intermediate results to be very much larger than the available disk
space, since those intermediate results didn't have to be written to disk.

CMS/TSO Pipelines does piping somewhat like this and much more.  Once
difference is that data doesn't have to move, record descriptors do that
job for unchanged records.

I was first exposed to this concept, when I was supporting a team of
previously UNIX-based developers working on TSO for the first time.  One of
the team came to me and asked how to connect the output of one program to
the input of the next in JCL.  On the whiteboard, I proceeded to write the
JCL for creating a temporary dataset in one job step (DD with DSN, UNIT,
DCB, DISP, etc.) and the JCL to read that dataset in the next job step.
When I turned around, the questioner was gone.  A short time later, his
officemate came in with the same question.  The original questioner thought
I was pulling his leg (since one could do that with a single character in
UNIX) and he had left before I was done.

For those that may not know:

  - TSO PIPEs implements dynamic, record-level, deterministic piping that
  is significantly more advanced than that on UNIX, Linux, and USS.  (More on
  that below.)
  - If installed, TSO PIPEs can be used by invoking the "pipe": command
  anywhere that a TSO command can be used, including:  READY Prompt, ISPF
  option 6, ISPF command line with the TSO prefix, REXX EXECs, CLISTs, TSO
  batch, and the ISPF service "select cmd(pipe ... )".
  - PIPEs is part of the z/VM base but is optional under z/OS.  Hence this
  post.
  - TSO PIPEs, TSO Pipelines, and BatchPipesWorks, are all names for
  essentially the same product, which is roughly a subset of CMS Pipes.  (My
  understanding is that TSO PIPEs is not the same as the USS command "pipe";
  you need to direct the command to TSO.)
  - Under z/VM, and under z/OS where available, it is well integrated into
  both environments.
  - On a very high-level conceptual level, PIPEs are somewhat similar to
  SQL and 4GLs in that many common, low level processing details (such as
  initialization, looping, and end of data) are taken care of internally.
  Thus, many types of selections, matchings, transformations, etc. can be
  done with compact commands and frequently without convention programming in
  languages like COBOL, PL/I, and C/C++.  (It is different from SQL and 4GLs
  in that Pipes can process many kinds of data, not just one type of data  or
  source.)
  - PIPEs can read both USS text and most binary files and can convert
  them to record orientation (address plus length descriptors).
  - PIPEs can also be invoked from JCL thusly:  //  EXEC
    PGM=PIPE,PARM='< IN.FILE|SORT|> OUT.FILE'

PIPEs in the z/OS base would greatly improve the competitiveness of
mainframe systems, while cutting development, hardware and support costs
for IBM, third-parties and customers.  By having Pipes available to all
internal and external developers, efficiency and productivity would
improve.  This would help IBM competitiveness and bottom line, as well as
that of mainframe sites, and software vendors.  If you've been following
IBM after quarterly reports are covered in the news, you'll know that IBM
needs a shot in the arm like this.  Results have been poor for many
quarters.  Over the long term it would mean improved profitability for the
entire mainframe division.  There might even be IBM customers who are saved
from moving off mainframes (because of costs) by such a move.  In short,
making PIPEs part of the z/OS base would more than pay for itself in
reduced costs, retained customers, and additional sales.  If I were an IBM
stockholder, a manager at a mainframe customer, or part of a third-party
development team, I would be asking for TSO PIPEs from anyone who would
listen, and maybe some who wouldn't.  :-)

Costs to IBM would not be an issue.  TSO PIPEs already exists as a subset
of CMS PIPEs.  Even in it's current subset state, it would satisfy the vast
majority of requirements (if not all) to deliver the product with the base,
thus minimizing the additional cost.

UNIX and variants cannot achieve what PIPEs can.  (I have some experience
with UNIX and Linux, but only a little exposure to USS, so feel free to
correct any misstatements or incorrect details, especially in this
section.)

  - The *NIX byte stream model requires that every character of a text
  file be inspected for end-of-record and forcing each and every character to
  the faster caches.  There is no way to skip an entire record in a text
  file.  The PIPEs model of record orientation means that records can be
  skipped, in their entirety, or by inspecting a part, without paging-in and
  cache-loading the whole record, should it span page or cache line
  boundary.  Certainly, the faster caches don't need to be flooded with
  unneeded, unused bytes.
  - In *NIX each and every byte must be copied from the sending process,
  to the buffer, to the receiving process.  In PIPEs, only the record
  descriptor (address and length) gets copied, while the data remains in
  place.  (Actually, it's probable that the descriptor is passed by address.)
  - Where binary (non-text) files are used (e.g. for performance or random
  access), this precludes the use of most *NIX/USS text-oriented built-in
  filters, without extra effort.  E.g., extra
encoding/translating/encapsulation
  to prevent binary data from being interpreted as end-of-record sequences,
  to ensure the logical records appear as such.
  - *NIX/USS piping byte stream can be split only with difficulty (think
  about one job step creating two temporary datasets), and, because the data
  flow is not deterministic, they can't be rejoined without special coding.
  With PIPEs, splitting and rejoining streams is routine and useful, because
  the record flow characteristics are known and documented for built-in
  stages.

PIPEs make REXX an even more compelling tool in batch.  I would nominate
PIPEs as a candidate for the JCL of the 21st century., but that's a long
story for another time.  In both conventional multi-step jobs and more
contemporary applications (e.g. ETL, data warehousing, etc.) multiple steps
can be achieved in a single command more efficiently and with less original
code than is generally done in z/OS today.

PIPEs has interfaces for DB2, PDS/Es, Byte Files, ISPF tables and
variables, as well as sequential files, to name a few.

If you have any doubt about the usefulness of Pipes, take a poll of your
z/VM friends.  They would tell you that they couldn't live without Pipes,
not because z/VM otherwise lacks anything, but because Pipes is so powerful
and efficient.

I urge anyone and everyone who uses REXX or JCL to do what they can to make
bring Pipes to the z/OS base.  Fee free to forward this post to anyone who
you think cares and/or who can help.

I hope that someone more vocal and persuasive than I can take up this issue
and get some needed action.

In the meantime, check to see if you have the TSO PIPE command already
installed.  Type "[tso] pipe q"; if you get a version message, you have
it.  Next step:  "[tso] pipe help menu" and/or "[tso] pipe ahelp menu".
Failing that, you can convince your management to order and install TSO
PIPEs.  Your site will benefit, and it might just be enough to get the
powers that be in IBM to wake up.  The TSO PIPE command was available
separately from BatchPipes.  I assume it still is.

IBM needs to get its act together.  The stock is right back to where it was
before the z encryption announcement, Buffet is selling his IBM holding
(possibly related), and the previous quarters have been a long string of
disappointments.  Putting PIPEs into the z/OS base may be just the step
needed.

My apologies for any errors, inaccuracies, or too harsh words.

Whew!  Kudos if you read the whole thing.

- Hobart Spitz

Having used z/OS NetView PIPELINE for years I really miss this in TSO. Productivity with NetView REXX/PIPE is much greater than in TSO REXX with no PIPE where I have to write and test PIPE-like REXX code for myself.

z/OS NetView has support for PIPEs that is extremely useful but they are sadly lacking from TSO/E.

Being a z/VM-er, forced to work on z/OS, I miss the functionality provided by CMS/TSO Pipes.
I have the told CMS/TSO Pipes code available, and that is a huge productivity booster, but the more recent (last couple of decades) enhancements are seriously lacking, requiring often to recreate "standard" functions in private REXX stages.
So getting a supported TSO/Pipes at the z/VM 6.4 level would be really appreciated...

This requirement has been outstanding for more than 25 years. Clearly, support for this requirement is forthcoming from the TSO user community.

One of the earliest SHARE presentations was SHARE 92 February 1990 "Plumbers Who Use TSO and the Users Who Love Them: An Introduction to TSO Pipelines", Session 2873, Hobart Spitz, Paine Webber.

"Plumbers Who Dump CVTs and the Control Blocks that Love Them: MVS/TSO Pipelines for Systems Programmers", SHARE February 1999, Session 2875, Hobart Spitz, Paine Webber

"Two Dimensional Pipes: How CMS Pipelines Differs from UNIX Pipes", SHARE 97, Summer 2001, Session 9109, Marty Zimelis, Computer Associates

This addition to the base z/OS environment would significantly improve the productivity of those developing and delivery extensions and enhancements to the platform. We know the code is available and that it isn't effectively marketed. Since it is apparently stable why not integrate it and ship if for free.

Creating a new RFE based on Community RFE #47699 in product z/OS.