Posts Tagged ‘cobol’

NACA tools & Zaap Processors: transcode a Cobol application to Java and still run it (but much more cheaply…) on the IBM mainframe with zOS!

Monday, February 1st, 2010

In the NACA project (offload from a mainframe to Linux on Intel servers through 100% automatic transcoding of the 4+ millions lines of Cobol of the in-house application), in order to reach the maximum level of savings, we took a radical approach (downsize to Linux and stop the mainframe) to free up as much financial resources as possible. We wanted to maximize the investments for upcoming projects in our IT budgets:

  • replace IBM operating system zOS by Linux and Cobol by Java (through our automatic transcoder now open-.sourced)
  • replace the zSeries architecture of the mainframe by very simple Intel servers as soon as our benchmarks demonstrated that it was feasible at no risk for our level of activity (750′000 CICS transactions per day). Thanks to Moore’s Law, the Intel Pentium proved itself to be highly powerful to handle high transactional workloads.

This very radical approach made us eventually very satisfied: immense cost reductions, availability and performances of new system, rollout of an internal disaster recovery mirror system, etc.

But, since the open sourcing of our NACA transcoding tools, we had the opportunity to discuss with many companies evaluating our technologies or already running transcoding projects with them on 4 continents (only Africa missing as of now!…).

Those discussions led us to develop a smoother and more “fluid” approach for corporations wanting simultaneously to leverage their mainframe assets and their strengths and to rejuvenate their legacy application:

  • Rejuvenate the application functionally: via the NACA tools, the Cobol application is automatically transcoded to Java source code and enjoys then following benefits: (a) as new “official” source code is Java, ability to leverage the very sophisticated environment of Java Virtual Machine still evolving very quickly (b) availability of many (open source) Java packages to integrate with the newly transcoded application to augment its capabilities (c) use of a state-of-the art user interface fully web-based (HTML-CSS-AJAX) as transcoded from the original 3270 BMS maps with no human intervention.(d) use of very modern development IDE (Eclipse through a specific plug-in) that strongly increases their productivity of legacy programmers as they continue seeing their initial and familiar Cobol code to maintain it.
  • Rejuvenate the application technically: the Java world is still highly dynamic and improving on a recurring basis. When on JRE, an application can progress just by “being here” and not doing much: we saw a 30% improvement in performances just by upgrading from Java 5 JRE to Java 6 JRE. This is in line with official Sun benchmarks. The Java community improves the JVM day after day to make Java now a clear rival of very efficient languages like C/C++.
  • Leverage mainframe assets and strengths: IBM wants its current clients to avoid such radical decisions as ours to keep them in the mainframe community. Consequently, IBM has launched some specialized processors dedicated to handling Java workload on the mainframe. They are called Zaap pour “zSystem Applicat ion Assist Processor”. By using them and without any major evolution of the mainframe configuration in place, the naca-transcoded application now runs on those dedicated processors and consequently preserves a very stable and deeply known zOS environment.



Those Zaap processors have a double financial advantage as created by the IBM marketing:

  • they are themselves much cheaper than a regular mainframe processor (GCP)
  • their mips don’t add themselves on top of the mips of the regular processors: they don’t as such have a negative influence on the monthly software bill of the mainframe for zOS essentially based on mainframe performance. As a matter of fact, they even tend to reduce this bill as the workload run on Zaaps means less mips needed in the regular processors. See details of Zaaps below

Consequently, it is possible to use our NACA transcoding tools to transcode 100% automatically to Java and to simultaneously stay in the “comfort” of the established mainframe environment after transcoding. The application will then run under Websphere (or even in the transactional monitor CICS or IMS but in Java) in a JVM powered by Zaap under direct control of zOS. For batch programs, same architecture but simpler: only the JVM is needed.

Such a mainframe-hosted Java architecture is needed by corporations who want to leverage all the benefits of a transcoded application to Java and web-based UI (see above) but who want to keep it on the mainframe. The company can this way maintain all the integration of the transactional activity with other mainframe subsystems (batch, archiving, printing, etc) sometimes developed over many decades.

Everything stays “in the same box” but the end users can benefit right away from additional functionnalities of their application brought to state-of-the-art.

As additional benefit, this architecture can reduce the mainframe total cost of ownership at double level: (1) the Mips of the transactional workload have now migrated from the costly regular processors to the cheap Zaaps (2) the “regular” (Cobol/CICS) mainframe workload has become smaller, replaced by functionally equivalent Java workload, and the monthly software bill can be reduced as less mips are consumed (see IBM below)

The resulting effects are triple:

  • The legacy application has become state-of-the art, from both a user and a developer standpoint: 100% Java and Web
  • A smooth and fluid evolution: a potential mainframe-less future is prepared at no risk as current machine, very well mastered by teams in place, remains the unique vector of activities.
  • Savings generated by optimized configuration: all possible workload is transferred to the cheapest possible Java environment and the requirements of expensive GCP mips is highly reduced through maximum use of Zaaps wherever possible through the CICS workload migrated to Websphere or Tomcat.

This is the clear proof that gradual paths exist to massively leverage the financial and functional advantages of new technologies (Java) for legacy applications. The existing runtime system is not jeopardized: the operational competencies accumulated on the mainframe for decades can further be leveraged and at the same time the business competencies accumulated by in-house are also preserved as they continue maintaining familiar structures of source code. No leap jump is required when such is a the wish!

============== IBM Zaap processors

Source for schemes of this article: IBM presentations of Kathy Walsh.

Zaap processors are a significant step in the continuous investment by IBM to maintain the competitiveness of its mainframe zOS platform in the market, when facing fierce rivalry from Open Source (Linux) highly due to the intrinsic expensiveness of zSeries.

To achieve this goal, IBM consistently introduces new processors dedicated to some specific workloads. Those workloads transferred on cheaperinternal processors reduce the TCO of the global machine.

By Publicitas, we did start the NACA project on this path of dedicated processors: we initially started on an IFL processor (IFL = Integrated Facility for Linux) integrated in our ultimate mainframe in order to run another operating system than zOS: zLinux (linux distribution for the mainframe). We decided to move to linux on Intel servers only when internal benchmarks demonstrated that there was almost no risk and significant additional financial advantage to do so.

Since then, IBM has strengthened the concept of dedicated processors beyond the IFL original concept: the Zaap doesn’t require another operating (zLinux) to offload activities but runs directly under the control of zOS. The slide below demonstrates the technical and marketing goals of IBM in this technology ( the yellow rectangle at bottom is to be read in details):



Through a marketing “secret sauce”, it is all about migrating “central” mips to Java mips on a dedicated processor. The emergence of this new technology brings the opportunity of selling it on a different price scale and also the opportunity of eliminating it of the mips total used to bill the zOS software. So, the current software renting by the mips total remains unquestioned by the community of corporation running their core activities on mainframe. At the same time, the Zaap can be presented as the “white knight” reducing the total of mips hence the price.

As a result, the competitiveness of zOS is clearly improve din its current fight against the linux adoption for critical applications.

[EN] Project NACA: open sourcing GPL/LGPL of the tools for transcoding Cobol programs to Java

Thursday, July 17th, 2008

Update of March, 4th 2009: version 1.1 is now available: NACA-Package 1.1 (Runtime, transcoder, Tests + docs) (1134)].   Details and change log in this post.

Publicitas publishes under Open Source license (GPL/LGPL) the source code of the tools developped to execute the project called NACA now successfully completed. Its aim was to migrate a homegrown Cobol application (named PUB 2000 - 4 millions lines of Cobol) running on an IBM/zOS mainframe toward its 100% iso-functional Java equivalent running on Intel-propelled Linux-based servers.

For those in hurry to get the package, the complete zip file is here:NACA-Package 1.0 (Runtime, transcoder, Tests + docs) (4200)

This version is clearly v1.0, i.e a first delivery: it will get improved over time through the feedback and contributions returned by external testers running the tools on their own infrastructure. New versions of runtime libraries (JLib and NacaRT) can also be expected because they are still heavily used in our own environment on a daily basis for the transcoded version of Pub2000.

Why this choice of licenses? Because we want to avoid that professional service consultancies takes this code back to the proprietary zone for their own single benefit. For those players, GPL brings a set of constraints that maximizes the return of their improvements to the community by preventing them from locking the source code of their changes. For end-user companies, the freedom brought by GPL is totally appropriate.

From another perspective, the LGPL + GPL combination allows application software editors to migrate their applications through our tools (NacaTrans), link them to our runtime libraries (NacaRT and Jlib), distribute the source code of those libraries but not the source code of their own application so that they can remain competitive in the industrial sector of their application.

The tools that we deliver today (v1.0) in the zip package:

  • Doc: a set of documents explaining in details the tools and libraries. Your feedback around this documentation, its missing points, etc. is essential in order to improve it.
  • NacaTrans (license GPL - approx. 83′00 lines of code code & 690 Java classes): our transcoder that allowed us to convert 100% automatically the 4 millions lines of our PUB 2000 application in COBOL to Java. It is very much based on compiler technologies. It analyzes the structure of the initial COBOL programs (supposed 100% valid) to bring them in an intermediate xml structure before generating the final Java code that extensively calls functions and uses classes of the runtime library NacaRT, itself depending on JLib. This new Java source code was very carefully designed: each line of Cobol generates very intentionally a single corresponding line of Java. The aim is to look like as much as possible like the original COBOL code in order to ease the maintenance by the original developers / maintainers who master the structure of their original Java programs. The completeness of the accepted syntax for all variants of Cobol is of course not guaranteed. But our own 4 millions of lines of code as well as additional tests on other external application tend to prove that the current coverage of Cobol by NacaTrans is already very high. We want to improve this coverage through your feedback for valid constructs that we don’t support yet.
  • · NacaRT & Jlib (license LGPL - approx 153′000 lines of code & 890 Java classes): those are the 2 runtime librairie who provide all the runtime transactional services for the application. They emulate all teh functions of a classical transactional monitor like CICS from IBM. At the same time, they also support all the COBOL constructs (for example, COMMARÈA structure with multiple data representation masks, management of specific data format like COMP-X, etc.)
  • NacaRTTest (license GPL): this is a test suite allowing us to test the correct output of the transcoder on a set of reference COBOL constructs. It’s the way to validate part of our transcoding algorithms. For a new user of NACA, this is definitely the place to start: when this runs on oyur infrastructure, you can feel pretty confident about your installation of the package.

All those components are also delivered with Eclipse project description files in order to facilitate their setup in a standard Java environment.

Now, the answer to your last question already on your lips: why does Publicitas undertake this delivery?

  • Because Publicitas has received quite a lot from the OSS community for our NACA project. We had to give back as much as we can: with this contribution, we want to return our highest possible contribution to the dynamics of Open Source.
  • Because we believe that this set of tools heavily tested in our environment can be an excellent starting point for other similar projects by other companies also wishing to leverage Open Source software as we did (see our various articles on this topic)
  • Because we still want to improve our tools and libraries as they remain used on a productive basis to handle our 750′000 transactions / day . We are interested in improving both the runtime libraries since Pub2000 runs on them but also the transcoder NacaRT itself . We still use NacaRT on a very regular basis for the transcoding of some programs that our developers decided (for valid reasons) to maintain from their COBOL source code. We even developed an Eclipse plugin to make this task simpler and more efficient.
  • Finally, because Consultas, our internal IT entity, mostly aimed at serving other group entities, developed through this project a set of very specific competences that they would love to reuse in other similar contexts. Consultas’ engineers love this kind of challenges! If you plan your own NACA, they may want to be part of it.

Now, it’s you turn: take the package NACA-Package 1.0 (Runtime, transcoder, Tests + docs) (4200), install it, run it and let us know: we’d love your feedback in the comments of this post.

PS: for more specific contacts around your particular case, you can email us at naca-contact@publicitas.com

[FR] NACA: Présentation du projet aux Rencontres Mondiales du Logiciel Libre (Mont-De-Marsan)

Monday, July 14th, 2008

Comme promis aux participants à la conférence et peut-être d’intérêt pour les lecteurs de ce blog, voici donc ma présentation faite hier aux RMLL 2008 à Mont-De-Marsan.

fichier à télécharger = NACA - Presentation RMLL 2008 (489)

Vu le contexte de cette conférence (orientation entreprises), j’ai ajouté des considérations supplémentaires aux autres billets de la saga rappelés en fin de cet article.

Application critique et Open Source:

  • Risque: vous n’êtes pas seul - Google, NASA, Boeing, Airbus, Areva, Wall-Street etc.. l’utilisent aussi et à des échelles énormes
  • Qualité: relire “la Cathédrale et le Bazar” de E. Raymond
  • Performances: le RoadRunner (#1 au monde - 1.026 pétaflop) fonctionne sous Linux !
  • Sécurité: la “sécurité par l’ignorance” (logiciels propriétaires) n’est pas bonne - La dissection par la communauté Open Source est bien plus solide
  • Support: seulement les forums ? ou plus….. ? notre choix pour NACA de la distribution RedHat avec son support professionnel

Bénéfices intangibles:

  • Open Source = standards.En migrant vers le libre, on respecte naturellement les standards (de facto ou de jure) de l’informatique: XML, J2EE, SOA, HTML, etc..
  • Migration -> modernisation: interface graphique pour tous les services systèmes (Webmin, etc.) outils modernes et très “pointus”
  • Nouvelles fonctions “bonus”: qq. exemples
    • arrivée dans le monde Java / Linux ouvert: intégration globale beaucoup plus simple (communication RPC synchrone et temps réel)
    • PDF très cher / compliqué sur le mainframe ? naturel sur Linux (permet une digitalisation complète
    • Système d’archivage (Knowledge Tree) permet une recherche “full-text.
    • Architecture à croissance horizontale plutôt que verticale ? plus de décisions d’upgrade complexes
    • remplacement de fonctions maisons en Cobol par des packages Open Source (tris, etc….)

Points critiques:

  • Le business case initial doit être clair: les avantages financiers énormes de l’Open Source sont la motivation la plus évidente directe du projet
  • La progression à multiples petits pas et la réversibilité de chaque mutation sont essentielles
  • Les inventaires des fonctions (applicatives et système) en production doivent être 100% à jour -> nous aurions pu faire mieux….
  • L’iso-fonctionnalité est un must: permet le transcodage automatisé et le fabrication quasi-gratuite de jeux de tests critiques au succès du projet

Bénéfices:

  • Economies (vrais “cash-outs”) de plusieurs millions d’euros / an
  • Grande stabilité et excellentes performances
  • Un système technologiquement à l’état de l’art ? la base technologique pour bâtir le successeur de l’application PUB 2000 (chantier démarré)
  • Construction d’un propre centre de backup (qq. serveurs…)
  • Architecture à croissance horizontale avec très faibles incréments
  • des équipes mutées à ces nouvelles technologies (pari humain réussi !)


Pour ceux qui auront au bout de la présentation, ils verront que nous avons annoncé durant cette conférence la mise en Open Source prochaine de nos outils de transcodage Cobol vers Java: NacaTrans, NacaRT et NacaTools. Les détails seront annoncés prochainement sur ce blog.

[Rappel: cet article fait partie d'une série qui décrit le projet NACA ayant conduit au remplacement d'un mainframe IBM sous MVS/OS390 par des serveurs Intel sous Linux. Le projet a été lancé en Janvier 2003 et s'est terminé avec succès au 30 Juin 2007.

Il a été réalisé volontairement de manière 100% iso-fonctionnelle (i.e. sans aucune modification pendant et par le transcodage) pour l'application et a engendré la conversion automatisée de 4 millions de lignes de Cobol vers leur équivalent Java. L'économie en cash-outs - paiements externes - est de plus de 85% de leur montant annuel = initial d'environ 3 millions d'euros annuels

Articles déjà parus:

]