2020-05-03

DevOps with Informatica PowerCenter



Before to begin.

What is DevOps?

Was a role invented by Google closing the gap between business requirements, development cycle and operations.

Pretty bad developments are complex to maintain out of standards or unstable applications. Imagine going to google and search something and get an error? Unacceptable right?

There’re so many definitions, for me it's closing the gap between development requirements and operations. In a traditional development cycle takes a while to start creating deliveries and the cycle is slow and require so many people.

DevOps use/is Agile, that means that in a short time you can have several iterations increasing the quality at the code, process and functionality focused on business requirements. DevOps encourage Continuous Integration and Delivery using tools, techniques reducing development time and outages.

It's possible to implement DevOps with Informatica on premise?

Partially.

Yes. this is a tricky title. Let’s match and review why is partially:

DevOps tools and components:

  • Requirement and incidents:
    • You can use a tool like Jira and sync it with Jenkins (lets talk later about that later)
  • Version Control
    • First missing component, Informatica versions stays inside the repository (metadata) service, for now there’s no way to have a globally VMS like git.
    • You can use git for data sources definitions like text files, SQL server scripts (DDL and DML XREFS), etc.
  • Code Quality and build:
    • Second missing component, there’s no way to integrate Maven or IntelliJ
    • Informatica includes code quality and build embedded at Informatica Client Tools. This means when you save your work a component is marked as valid or invalid, based on the result of your development.
    • You can build and/or use an external app to review if naming conventions and standards are correct but does not means that the jobs will work as expected.
  • Testing
    • Third partial component missed. You can use Selenium for some test cases based on web services calls (if you have web service hub) or can call a test job or DVO using Control-M, Jenkins or any other scheduler, this means is a manual and really innovative work from the development side.
  • Continuous integration:
    • Informatica versioning deployment is internal (missed too), let’s say that you have 3 repositories (let say DEV, UAT and PRD). It’s easy to move from one env to another using tags and deployment groups (this is functionality part of Informatica platform).
    • By instance you can integrate Jira with Jenkins and when you mark that the ticket is resolved at DEV and target is UAT, Jenkins or any other scheduler can execute a script that deploys the changes. For Informatica can use client tools utilities (command utilities) API part of informatica to move the deployment. Also, you can move and execute any SQL script need it for DB changes (DML mainly). You need to develop all this, there’s no many products that do what you need a 100% or for reference some developments requires cloud databases or uses Oracle and SQL and the code can differ.
    • Many developers uses as well Control-M or external schedulers orchestrations, this are out of the scope for now (but if there’s an API can create one more step in a config file and use it).
  • Configuration Management
    • Partial missed as well. Informatica OS level configurations are not common, so we can discuss that the configurations are at jobs level, commonly can be handled using parameters files, job scripts (bat or bash files) and SQL Server scripts.
    • You can handle that with Jenkins as well, using a “implementation script”, honestly using Puppet, Chef, Ansible or Docker does not apply (because PowerCenter is on premise) – we don’t create any image and change libraries on the fly.
    • Some configurations like folders, privs and connections - including ODBC/DSN entries needs to be manually updated.
  • Performance and monitoring:
    • There are two approaches for monitors, the first one is based at the services and resources used at Informatica platform servers; these ones are created and always used (there’s no dynamic or continues creation of these). You can use SiteScope, Nagios, AppDynamics or any other desired monitoring tools.
    • For business process/jobs; developers are responsible of configure emails on success or failure sessions/tasks. As sysadmins we use a metadata queries to report power users any critic failed jobs every week. This runs internally at Informatica (like any other Workflow).
  • Incidents and on call:
    • Business jobs and SiteScope monitors can create incidents to ServiceNow and if is critic can create a SMS/mail to PagerDuty.
  • Log Analysis:
    • Informatica servers can uses ELK Stack and Splunk forwarder for server level logs and applications (service logs). Jobs logs are included on failure sessions mails.
  • Deployment cloud:
    • Does not apply as well, we use our own servers and for now is not planned to use AWS or Azure clouds.
  • Deployment Container:
    • For now, I heard some rumors that Informatica on premise can use Docker Image, for now we don’t use it (also includes Kubernetes), containers are weird things in heavy applications like Informatica because the jobs needs to use the more CPU, RAM and IO possible for the jobs. This is one of the main reasons why Informatica is still on premise.

It’s possible to have DevOps at Informatica?

Partially, if you implement best practices and development standards you can automate deployments (Informatica Objects and Configuration files/scripts).

Also some teams needs to change they dev standards or have more privileges to execute the deployment groups with other tools.

I will say for now we can have like the most critical processes with DevOps (like 60%) and like around 50% of the teams are using it.

Have a nice night.