Posts Tagged ‘transcoding’

[EN] Project NACA: open sourcing GPL/LGPL of the tools for transcoding Cobol programs to Java

Thursday, July 17th, 2008

Update of March, 4th 2009: version 1.1 is now available: NACA-Package 1.1 (Runtime, transcoder, Tests + docs) (1134)].   Details and change log in this post.

Publicitas publishes under Open Source license (GPL/LGPL) the source code of the tools developped to execute the project called NACA now successfully completed. Its aim was to migrate a homegrown Cobol application (named PUB 2000 - 4 millions lines of Cobol) running on an IBM/zOS mainframe toward its 100% iso-functional Java equivalent running on Intel-propelled Linux-based servers.

For those in hurry to get the package, the complete zip file is here:NACA-Package 1.0 (Runtime, transcoder, Tests + docs) (4200)

This version is clearly v1.0, i.e a first delivery: it will get improved over time through the feedback and contributions returned by external testers running the tools on their own infrastructure. New versions of runtime libraries (JLib and NacaRT) can also be expected because they are still heavily used in our own environment on a daily basis for the transcoded version of Pub2000.

Why this choice of licenses? Because we want to avoid that professional service consultancies takes this code back to the proprietary zone for their own single benefit. For those players, GPL brings a set of constraints that maximizes the return of their improvements to the community by preventing them from locking the source code of their changes. For end-user companies, the freedom brought by GPL is totally appropriate.

From another perspective, the LGPL + GPL combination allows application software editors to migrate their applications through our tools (NacaTrans), link them to our runtime libraries (NacaRT and Jlib), distribute the source code of those libraries but not the source code of their own application so that they can remain competitive in the industrial sector of their application.

The tools that we deliver today (v1.0) in the zip package:

  • Doc: a set of documents explaining in details the tools and libraries. Your feedback around this documentation, its missing points, etc. is essential in order to improve it.
  • NacaTrans (license GPL - approx. 83′00 lines of code code & 690 Java classes): our transcoder that allowed us to convert 100% automatically the 4 millions lines of our PUB 2000 application in COBOL to Java. It is very much based on compiler technologies. It analyzes the structure of the initial COBOL programs (supposed 100% valid) to bring them in an intermediate xml structure before generating the final Java code that extensively calls functions and uses classes of the runtime library NacaRT, itself depending on JLib. This new Java source code was very carefully designed: each line of Cobol generates very intentionally a single corresponding line of Java. The aim is to look like as much as possible like the original COBOL code in order to ease the maintenance by the original developers / maintainers who master the structure of their original Java programs. The completeness of the accepted syntax for all variants of Cobol is of course not guaranteed. But our own 4 millions of lines of code as well as additional tests on other external application tend to prove that the current coverage of Cobol by NacaTrans is already very high. We want to improve this coverage through your feedback for valid constructs that we don’t support yet.
  • · NacaRT & Jlib (license LGPL - approx 153′000 lines of code & 890 Java classes): those are the 2 runtime librairie who provide all the runtime transactional services for the application. They emulate all teh functions of a classical transactional monitor like CICS from IBM. At the same time, they also support all the COBOL constructs (for example, COMMARÈA structure with multiple data representation masks, management of specific data format like COMP-X, etc.)
  • NacaRTTest (license GPL): this is a test suite allowing us to test the correct output of the transcoder on a set of reference COBOL constructs. It’s the way to validate part of our transcoding algorithms. For a new user of NACA, this is definitely the place to start: when this runs on oyur infrastructure, you can feel pretty confident about your installation of the package.

All those components are also delivered with Eclipse project description files in order to facilitate their setup in a standard Java environment.

Now, the answer to your last question already on your lips: why does Publicitas undertake this delivery?

  • Because Publicitas has received quite a lot from the OSS community for our NACA project. We had to give back as much as we can: with this contribution, we want to return our highest possible contribution to the dynamics of Open Source.
  • Because we believe that this set of tools heavily tested in our environment can be an excellent starting point for other similar projects by other companies also wishing to leverage Open Source software as we did (see our various articles on this topic)
  • Because we still want to improve our tools and libraries as they remain used on a productive basis to handle our 750′000 transactions / day . We are interested in improving both the runtime libraries since Pub2000 runs on them but also the transcoder NacaRT itself . We still use NacaRT on a very regular basis for the transcoding of some programs that our developers decided (for valid reasons) to maintain from their COBOL source code. We even developed an Eclipse plugin to make this task simpler and more efficient.
  • Finally, because Consultas, our internal IT entity, mostly aimed at serving other group entities, developed through this project a set of very specific competences that they would love to reuse in other similar contexts. Consultas’ engineers love this kind of challenges! If you plan your own NACA, they may want to be part of it.

Now, it’s you turn: take the package NACA-Package 1.0 (Runtime, transcoder, Tests + docs) (4200), install it, run it and let us know: we’d love your feedback in the comments of this post.

PS: for more specific contacts around your particular case, you can email us at naca-contact@publicitas.com

[FR] NACA: Présentation du projet aux Rencontres Mondiales du Logiciel Libre (Mont-De-Marsan)

Monday, July 14th, 2008

Comme promis aux participants à la conférence et peut-être d’intérêt pour les lecteurs de ce blog, voici donc ma présentation faite hier aux RMLL 2008 à Mont-De-Marsan.

fichier à télécharger = NACA - Presentation RMLL 2008 (489)

Vu le contexte de cette conférence (orientation entreprises), j’ai ajouté des considérations supplémentaires aux autres billets de la saga rappelés en fin de cet article.

Application critique et Open Source:

  • Risque: vous n’êtes pas seul - Google, NASA, Boeing, Airbus, Areva, Wall-Street etc.. l’utilisent aussi et à des échelles énormes
  • Qualité: relire “la Cathédrale et le Bazar” de E. Raymond
  • Performances: le RoadRunner (#1 au monde - 1.026 pétaflop) fonctionne sous Linux !
  • Sécurité: la “sécurité par l’ignorance” (logiciels propriétaires) n’est pas bonne - La dissection par la communauté Open Source est bien plus solide
  • Support: seulement les forums ? ou plus….. ? notre choix pour NACA de la distribution RedHat avec son support professionnel

Bénéfices intangibles:

  • Open Source = standards.En migrant vers le libre, on respecte naturellement les standards (de facto ou de jure) de l’informatique: XML, J2EE, SOA, HTML, etc..
  • Migration -> modernisation: interface graphique pour tous les services systèmes (Webmin, etc.) outils modernes et très “pointus”
  • Nouvelles fonctions “bonus”: qq. exemples
    • arrivée dans le monde Java / Linux ouvert: intégration globale beaucoup plus simple (communication RPC synchrone et temps réel)
    • PDF très cher / compliqué sur le mainframe ? naturel sur Linux (permet une digitalisation complète
    • Système d’archivage (Knowledge Tree) permet une recherche “full-text.
    • Architecture à croissance horizontale plutôt que verticale ? plus de décisions d’upgrade complexes
    • remplacement de fonctions maisons en Cobol par des packages Open Source (tris, etc….)

Points critiques:

  • Le business case initial doit être clair: les avantages financiers énormes de l’Open Source sont la motivation la plus évidente directe du projet
  • La progression à multiples petits pas et la réversibilité de chaque mutation sont essentielles
  • Les inventaires des fonctions (applicatives et système) en production doivent être 100% à jour -> nous aurions pu faire mieux….
  • L’iso-fonctionnalité est un must: permet le transcodage automatisé et le fabrication quasi-gratuite de jeux de tests critiques au succès du projet

Bénéfices:

  • Economies (vrais “cash-outs”) de plusieurs millions d’euros / an
  • Grande stabilité et excellentes performances
  • Un système technologiquement à l’état de l’art ? la base technologique pour bâtir le successeur de l’application PUB 2000 (chantier démarré)
  • Construction d’un propre centre de backup (qq. serveurs…)
  • Architecture à croissance horizontale avec très faibles incréments
  • des équipes mutées à ces nouvelles technologies (pari humain réussi !)


Pour ceux qui auront au bout de la présentation, ils verront que nous avons annoncé durant cette conférence la mise en Open Source prochaine de nos outils de transcodage Cobol vers Java: NacaTrans, NacaRT et NacaTools. Les détails seront annoncés prochainement sur ce blog.

[Rappel: cet article fait partie d'une série qui décrit le projet NACA ayant conduit au remplacement d'un mainframe IBM sous MVS/OS390 par des serveurs Intel sous Linux. Le projet a été lancé en Janvier 2003 et s'est terminé avec succès au 30 Juin 2007.

Il a été réalisé volontairement de manière 100% iso-fonctionnelle (i.e. sans aucune modification pendant et par le transcodage) pour l'application et a engendré la conversion automatisée de 4 millions de lignes de Cobol vers leur équivalent Java. L'économie en cash-outs - paiements externes - est de plus de 85% de leur montant annuel = initial d'environ 3 millions d'euros annuels

Articles déjà parus:

]