Integrating z/OS applications with the rest of the world

Many mainframe applications were built in an era where little integration with other applications was needed. Where integrations were needed, this was mostly done through the exchange of files. For example, for the exchange of information between organizations.

In the 1990s the dominance of the mainframe applications ended and client-server applications emerged. These new applications required more extensive and real-time integrations with existing mainframe applications. In this period many special integration tools and facilities were built to make it possible to integrate z/OS applications and new client-server applications.

In this chapter I will highlight categories of these integration tools that are available on z/OS, from screen-scraping tools to modern integrations supporting the latest REST API interfaces.

File interfaces

The mainframe was designed for batch processing. Therefore integration via files is traditionally well catered for and straightforward.

You can use multiple options to exchange files between applications on z/OS and other platforms.

Network File System

Network File System (NFS) is a common protocol that you can use to create a shared space where you can share files between applications. Although it was originally mostly used with Unix operating systems, it is now built into most other operating systems, including z/OS. NFS solutions however are usually not a preferred option due to security and availability challenges.

FTP

The File Transfer Protocol (FTP) is a common protocol to send files over a TCP/IP network to a receiving party, and it is also supported on z/OS. With FTP a script or program can be written to automatically transfer a file as part of an automated process. FTP can be made very secure with cryptographic facilities.

FTP is built into most operating systems, including z/OS.

Managed File Transfer

Managed file transfer is also a facility to send files over a network, but the “Managed” in the category means a number of additional features are added.

Managed file transfer solutions make file transfers more reliable and manageable. A number of additional operational tasks and security functions related to file exchange are automated. Managed file transfer tools provide enhanced encryption facilities, some form of strong authentications, integration with existing security repositories, handling of failed transfers with resend functionality, reporting of file transfer operations, and more extensive API’s.

On z/OS a number of managed file transfer tools are available as separate products: IBM has Connect:Direct and MQ-FTE, CA/Broadcom has Netmaster file transfer and XCOM, BMC provides Control-M  and there are other less commonly known tools.

Message queueing

Message queuing is a generic manner for applications to communicate with each other in a point-to-point manner. With message queuing applications remain de-coupled, so they are less dependent on each other’s availability and response times. Applications can be running at different times and communicate over networks and systems that may be temporarily down. As we will see in the next section, when using alternative point-to-point protocols like web services, both applications and intermediate infrastructures must be available for successful application communications.

The basic notion of message queuing is that an application sends a message to a queue and another application asynchronously reads messages from that queue and (optionally) responds with another message over a queue. Besides the specific asynchronous nature of message queuing, a big advantage is that it can assure message delivery: messages will not get lost, and when the infrastructure is not available, messages remain stored until they can be delivered.

IBM’s MQSeries, or WebSphere MQ as it is called now, is a separate is one of the most well-known and robust solutions for message queuing available on z/OS.

The open API for messaging called Java Message Service (JMS) is implemented by WebSphere MQ and WebSphere Application Server on z/OS.

Applications using Message Queuing

Web services (SOAP, REST)

Web services is the modern technology that enables applications to communicate over the web protocol HTTP, the protocol we also use for browsing the web.

SOAP and REST are two different types of web services. SOAP is a bit older and exchanges XML messages. XML is more resource intensive because handling XML is a complex operation. REST is more modern and lightweight, and today’s API economy is mostly based on REST APIs.

The benefit of integration with web services is that no special infrastructure is needed for applications to integrate, apart from a capable web application server. Integrations are lightweight and can be very loosely coupled.

The downside of web service is that the HTTP protocol does not guarantee message delivery (as opposed to message queueing as we have seen above). Applications using web services have to implement their own recovery and retry mechanisms to cope with situations where connections are lost.

On z/OS today, most modern versions of application middleware on z/OS, like CICS, IMS, WebSphere Application Server, IDMS, and others support REST and SOAP interfaces.

Applications using Web Services

Enterprise Service Bus

Another form of integration can be achieved through Enterprise Service Bus tools. These tools probably give the widest variety of integration options. They can receive and send service requests over a number of different protocols. They can convert messages from and to many formats. And they can orchestrate complex message interactions between multiple applications. Enterprise Service Bus products in the market are Tibco Substation ES and IBM Integration Bus.

ESB solutions can be implemented on z/OS itself, which than often has the advantage of easier integration with the z/OS application side, but also a non-z/OS platform and integrate with z/OS agent software.

Enterprise Service Bus

Adapters

In many situations it may not be possible to refactor your old mainframe applications. The applications may not be designed properly in a layered manner, middleware technology may have limited options, skills may not be available, or the risk of a changing existing applications is too high. Or there may be other reason you do not want to touch the code.

For these situations, application adapters can help in opening up applications. In general, an adapter converts a proprietary middleware protocol like a CICS, IDMS or IMS API into a more common API or generic protocol, like a Java program, a web service or message queueing interface. Some adapters provide the option of converting a proprietary 3270 screen interface into a neat API through screen scraping.

I will highlight a number of the type of tools here.

Generic functioning of an adapter

CICS Transaction Gateway

CICS Transaction Gateway provides an API for Java and C programs to call a CICS transaction on z/OS.

CICS Transaction Gateway provides only a way to call functionality in CICS, but there is no possibility in this tool to reversely invoke an external program from CICS. CTG is only meant for external programs to call CICS.

CICS Transaction Gateway adapter

IMS Connect

IMS Connect provides a Java API through which you can invoke IMS functions form Java programs. Through IMS Connect you can access IMS transactions as well as data in IMS DB (see section Middleware for z/OS). As such it functions quite similar to CTG, although the native interfaces are of course different.

z/OS Connect

A recent product from IBM is z/OS Connect. This tool converts a REST API into one or more proprietary backend protocols, like a CICS or IMS transaction or call to Db2. Also, z/OS Connect makes it possible to call REST APIs from mainframe applications.

Thus, z/OS Connect provides a bi-directional adapter for REST API through which you expose and call RESTful APIs from existing z/OS programs in CICS, IMS, Db2, WebSphere Application Server and MQ.

z/OS Connect adapter

Screen scraping tools

You may have old legacy applications that are built as a silo, have only 3270 user interfaces and no decent program interfaces.

For this problem, screen scraping tools can be a last resort.

The integration problem of an application silo – refactoring is the ideal solution

A screen scraping tool provides the ability to simulate the interaction of a business user behind a screen, with the old application’s user interface. The screen scraper tool automates the workflow of the end-user by filling in the old application screen programmatically. With these automations such a tool can then aggregate and expose these interactions into higher level services. These higher level services can then be invoked through a modern API, such as a web service by other applications in your organization.

Integration with a screen scraping solution

The big problem with screen-scraping integrations is that you end up with more development artefacts that you need to maintain. Not only do you have the old application to maintain, but now also need to manage the screen scraping middleware and logic.

Screen-scraping should be considered a (very) temporary solution for a serious issue in your application landscape. Such a solution should be replaced by a strategic integration or new application as soon as possible.

Products like HostBridge, Rocket LegaSuite and IBM Host on Demand provide screen scraping facilities.

Legacy integration suites

There are many integration tools on the market that provide one or more of the forms of adapters that I have discussed in the above. For example, GT Software and Oracle Legacy Adapter provide functionality to bridge native z/OS interfaces including screen interfaces to and from modern applications.

Database access via JDBC, ODBC

So far, we have discussed application integration through application interactions – applications calling one another.

Applications on non-z/OS platforms alternatively can directly access data in databases on z/OS through the standard data access protocols ODBC and JDBC. All suppliers of database software for z/OS that I know provide drivers for ODBC and/or JDBC.

Integrating with JDBC and ODBC

From an architectural perspective it is not a preferred solution for integrating applications. Applications should manage their own data and access other applications’ data only through service interfaces, and follow the principle of loosely coupling for application architectures.

The Unix parts of z/OS

In the previous DBAOTM post I have introduced you to the z/OS operating system, the flagship operating system for the mainframe. In this post I will introduce you into the Unix side that z/OS has been equipped with over the past two decades.

Since the 1990s IBM has added Unix functionality to z/OS. The first extension was z/OS Unix System Services – z/OS Unix in short – and recently we IBM have added z/OS Container Extensions.

z/OS Unix

The Unix part of z/OS is a runtime environment that is an integral part of the z/OS operating system. z/OS Unix is fully Unix (Posix) compliant. z/OS Unix provides an interactive command interface that is called a shell in Unix terminology.

IBM has developed this part in the 1990s to make it easier to port applications from other platforms to z/OS. Many off-the-self and open source middleware and application packages that are available on z/OS make use of z/OS Unix. Examples are IBM’s own WebSphere Application Server and IBM Integration Bus, the banking product BASE24-eps from ACI, and open source tools like PHP and Git.

z/OS Unix has regular files the same as other Unix systems. In the z/OS Unix shell you can use normal Unix commands like ls, cd, more and many more of the standard Unix commands. You can set up a telnet or SSH session with z/OS Unix and do many more things you can also do with other Unix environments.

z/OS Container Extensions

A very recent development on z/OS (it came with z/OS 2.4, end 2019) is the possibility to run Linux Docker Containers on z/OS.

Docker containers are a hardware independent and lightweight manner to run many applications in a virtualized configuration. The Docker technology has been available on x86 platforms for a long time. With Docker containers you get a virtualization solution that does not need a complete virtual machine with an operating system running in it for every application. Instead your application is running in a small container that provides only a limited set of virtualization facilities. Just the things that the application needs. You can run many Docker containers – so applications – on a single real operating system image.

The interesting thing is that in a conceptual way Docker is a quite like z/OS as we have seen it in section Address Spaces are processes. On a z/OS operating system you can run many applications in “Address Spaces”. With Docker you run many container processes on a single real operating system image.

I will talk a bit more about Docker in section Linux in z/OS Container Extensions.

z/OS Address Spaces versus Docker containers

All Unix variants

A small elaboration, as you may already get confused with the Unix on the mainframe. I mentioned Linux for the mainframe, now I talk about z/OS Unix and Linux in Containers.

It is important to understand the difference between z/OS Unix, z/OS Container Extensions and Linux for Z.

z/OS Unix and z/OS Container Extensions are an integral part of z/OS. You get these options with z/OS.

z/OS Unix applications use the Unix flavour that z/OS provides, which is not Linux.

In z/OS Container Extensions you get the option to run applications with the Linux flavour of Unix, and run these in a containerized setup.

Linux for Z is an operating system that is in no way related to z/OS. Applications running on Linux for Z use the Linux flavour, and in an LPAR or Virtual Machine of its own.

I have tried to put all the Unix variants on z/OS in the picture below. You see z/OS Unix, as part of a z/OS operating systems, you see a z/OS container process running a Linux Docker container, and separate from z/OS there is an LPAR running Linux for Z.

The Unix flavours of z/OS and Linux for Z

What’s next

Now we have seen the basics of z/OS, we can turn to the more specialized and specific parts. In the next post I will discuss the unique clustering feature of z/OS, called parallel sysplex.