2020-05-03

DevOps with Informatica PowerCenter



Before to begin.

What is DevOps?

Was a role invented by Google closing the gap between business requirements, development cycle and operations.

Pretty bad developments are complex to maintain out of standards or unstable applications. Imagine going to google and search something and get an error? Unacceptable right?

There’re so many definitions, for me it's closing the gap between development requirements and operations. In a traditional development cycle takes a while to start creating deliveries and the cycle is slow and require so many people.

DevOps use/is Agile, that means that in a short time you can have several iterations increasing the quality at the code, process and functionality focused on business requirements. DevOps encourage Continuous Integration and Delivery using tools, techniques reducing development time and outages.

It's possible to implement DevOps with Informatica on premise?

Partially.

Yes. this is a tricky title. Let’s match and review why is partially:

DevOps tools and components:

  • Requirement and incidents:
    • You can use a tool like Jira and sync it with Jenkins (lets talk later about that later)
  • Version Control
    • First missing component, Informatica versions stays inside the repository (metadata) service, for now there’s no way to have a globally VMS like git.
    • You can use git for data sources definitions like text files, SQL server scripts (DDL and DML XREFS), etc.
  • Code Quality and build:
    • Second missing component, there’s no way to integrate Maven or IntelliJ
    • Informatica includes code quality and build embedded at Informatica Client Tools. This means when you save your work a component is marked as valid or invalid, based on the result of your development.
    • You can build and/or use an external app to review if naming conventions and standards are correct but does not means that the jobs will work as expected.
  • Testing
    • Third partial component missed. You can use Selenium for some test cases based on web services calls (if you have web service hub) or can call a test job or DVO using Control-M, Jenkins or any other scheduler, this means is a manual and really innovative work from the development side.
  • Continuous integration:
    • Informatica versioning deployment is internal (missed too), let’s say that you have 3 repositories (let say DEV, UAT and PRD). It’s easy to move from one env to another using tags and deployment groups (this is functionality part of Informatica platform).
    • By instance you can integrate Jira with Jenkins and when you mark that the ticket is resolved at DEV and target is UAT, Jenkins or any other scheduler can execute a script that deploys the changes. For Informatica can use client tools utilities (command utilities) API part of informatica to move the deployment. Also, you can move and execute any SQL script need it for DB changes (DML mainly). You need to develop all this, there’s no many products that do what you need a 100% or for reference some developments requires cloud databases or uses Oracle and SQL and the code can differ.
    • Many developers uses as well Control-M or external schedulers orchestrations, this are out of the scope for now (but if there’s an API can create one more step in a config file and use it).
  • Configuration Management
    • Partial missed as well. Informatica OS level configurations are not common, so we can discuss that the configurations are at jobs level, commonly can be handled using parameters files, job scripts (bat or bash files) and SQL Server scripts.
    • You can handle that with Jenkins as well, using a “implementation script”, honestly using Puppet, Chef, Ansible or Docker does not apply (because PowerCenter is on premise) – we don’t create any image and change libraries on the fly.
    • Some configurations like folders, privs and connections - including ODBC/DSN entries needs to be manually updated.
  • Performance and monitoring:
    • There are two approaches for monitors, the first one is based at the services and resources used at Informatica platform servers; these ones are created and always used (there’s no dynamic or continues creation of these). You can use SiteScope, Nagios, AppDynamics or any other desired monitoring tools.
    • For business process/jobs; developers are responsible of configure emails on success or failure sessions/tasks. As sysadmins we use a metadata queries to report power users any critic failed jobs every week. This runs internally at Informatica (like any other Workflow).
  • Incidents and on call:
    • Business jobs and SiteScope monitors can create incidents to ServiceNow and if is critic can create a SMS/mail to PagerDuty.
  • Log Analysis:
    • Informatica servers can uses ELK Stack and Splunk forwarder for server level logs and applications (service logs). Jobs logs are included on failure sessions mails.
  • Deployment cloud:
    • Does not apply as well, we use our own servers and for now is not planned to use AWS or Azure clouds.
  • Deployment Container:
    • For now, I heard some rumors that Informatica on premise can use Docker Image, for now we don’t use it (also includes Kubernetes), containers are weird things in heavy applications like Informatica because the jobs needs to use the more CPU, RAM and IO possible for the jobs. This is one of the main reasons why Informatica is still on premise.

It’s possible to have DevOps at Informatica?

Partially, if you implement best practices and development standards you can automate deployments (Informatica Objects and Configuration files/scripts).

Also some teams needs to change they dev standards or have more privileges to execute the deployment groups with other tools.

I will say for now we can have like the most critical processes with DevOps (like 60%) and like around 50% of the teams are using it.

Have a nice night.

2019-04-19

Sync a molle

I've been doing a little bit of hacking...

OneDrive TB (I hope still secure) as been integrated with my Linux server and Mac OS.

Part of the game integrates my backups using rsync and symlinks (that way I don't have to move the files from any of my local partitions and locations in Ubuntu).

By last I used a little bit of cronjobs at free Unix and payed Unix... I'm on fire!

That way I can have what I want at the server or Mac and avoid some risks like loosing pics or critical files...

Boring!!!

2019-02-15

4 days with a Macbook Air / MacOS

So far so good, I'm still understanding a little bit the MacOS experience, it's a really good OS but (I don't want to make you wait to the critics). Since always Apple wants to be unique but I hate the keyboard and miss so many keys from any Windows (standard Keyboard, like begin, end, sup, etc). And MacOS can do it with 2-3 and even 4 keys combos (really?).

On the other side reminds me a lot to Ubuntu before Unity. That GNome Skin was almost the same as Mojave.

But by the way is really easy to hack and recover admin rights and user (for a reason the office give me the laptop with a user and password without the second one)...

The best: Thanks to Chrome I don't miss Windows at all, on the other side Office + OneDrive works awesome, sometimes OneDrive is pretty slow to sync small changes (files)... and installing apps is almost the same as Windows (Wizard with next, next)...

The second best is the performance and battery time... I will keep posting the small things I learn like the apps:
- Spectacle, to use fast key combos to arrange and move the windows in 1 or 2 screens...
- CheatSheet, when you hold pressed the Command key shows the keyboard shortcuts for that app. Awesome...

The last comments; several years a go supposed to be more apps and integrations with Social networks and now facebook and linkedin are removed.

2018-08-27

BOM! Encontre una manera de como sacar provecho a 1TB OneDrive

El titulo alternativo sería, como administrar OneDrive entre distintas nubes como Dropbox y Google Drive... Incluyendo un equipo con Linux.

Bueno, la cosa esta así. Tengo años siendo cliente de Google y estuve pagando por un tiempo el espacio que usaba... pero un día Google me dice ya te acabaste el espacio y entre conversaciones anonimas un amigo me regaló una cuenta de Office 365... Que incluye un terabyte de OneDrive.

No es comercial pero este modelo/método por subscripción de Microsoft esta muy bien... Es una cuenta familiar con 5 miembros que sale lo que cuesta casí una membresia de Dropbox anual (para una persona)... e incluye office para "N" equipos por miembro.

En resumen, era evidente que tenia que matar mi Google-dependencia con Google Drive. Además desde años atras no he entendido como es posible que si exista Dropbox para Linux pero no OneDrive y Google Drive (cuando es notorio que Google Drive servers usan Linux).

Despues de mi drama viene lo siguiente...

Tengo cosas en Google Drive que no van a cambiar, ahí se quedaran para siempre, ademas por contingencia debo poder manejar ambas clouds por algunos documetnos y necesidades...

Tengo cosas en un server de Linux que deben de estar en OneDrive y que a la vez quiero acceder y/o compartir en un clic.

BOM!

Hice esto...

  • Configure en mi Windows un symlink de Google Drive a OneDrive
  • Configure en Linux OneDrive usando onedrive-d
Tengo dos syncs en Windows, uno de Google y otro de OneDrive, cuando sincronizo algo en Google Drive (ahora sync), en automatico se sincroniza a OneDrive y al sincronizarse a OneDrive en automatico se sincroniza a Linux.

En Linux tengo unos crontabs que necesitaba tener una copia alterna a la local (aunque tenga varios discos); que resolví con symlinks y scripts bash.

Años años con Linux y no había hecho esto.

Kudos for me!

2018-07-31

Starting with Talend ETL

So. I've been working with Informatica PowerCenter (ETL) for almost 10 years. Believe me is the best product if the companies can pay it.

Well... time to time I like to look around and learn new tools, my new goal was Pentaho and ODI (Oracle Data Integrator), but a really good friend told me about a project with Talend that needs maintenance and told me the reason "why he choosed Talend > Pentaho"... The main reason; Licensing and the second he need it an Open Source solution for 6 months integrations.

Today I received a great introduction to Talend and I started comparing the "elements" and components used to create jobs and how they work.

As everyone I started from the beginning at talend.com and now I'm checking this great Blog "Talend by Example".

I will keep you posted!
------------

Many position changes... Talend is now on hold but I'm sure would be a must in a long term...

------------

- Juan


2018-07-27