Handling the flow of growing amounts of messy data from multiple sources throughout the enterprise is a complex process. Whether moving transactional data so that it can be reported upon, migrating application data from old systems to new ones, or integrating data from external suppliers or partners, the reliable management of data streams can present a challenge to IT organizations. Because these movements of data can be scheduled on a regular basis or executed based on specific triggers, ETL and data warehousing processes inherently lend themselves to automation. IT automation brings a host of benefits to these processes including faster delivery times, improved productivity, reduced cost, decreased risk of error, and higher levels of data quality, among others. But the automation tools commonly available often don’t provide the flexibility that is needed for granular and end-to-end automation. For most organizations a different approach, a unified workload automation solution, is needed to achieve all the benefits that automation can provide.
Analyzing Three Traditional Approaches to ETL Automation
In order to facilitate the extraction, transformation, and loading (ETL) of data used in building enterprise data warehouses, many organizations have turned to Informatica’s industry-leading data management solutions, Informatica PowerCenter and Informatica Cloud. When it comes to automating and scheduling Informatica PowerCenter and Cloud processes, there are a few different approaches that are traditionally taken. Typically, we see organizations automate their Informatica data warehousing/ETL processes using one of three methods outlined below:
As you can see, the challenge with these approaches is that they limit users in realizing the full benefits that automation can provide, both from the perspective of automating powerful Informatica PowerCenter and Cloud ETL processes, as well as incorporating Informatica processes with non-Informatica processes, such IT operations, business processes, file operations, BI/reporting, and more, to achieve true end-to-end automation.
Informatica & Beyond: The Case for a Unified Workload Automation Solution
Enterprise workload automation and job scheduling solutions provide the most comprehensive, advanced way to automate both Informatica workflows and tasks as well as incorporate those workflows and tasks with non-Informatica applications.
Think about why you implemented an ETL tool in the first place. Your organization, with its ever-increasing amount of data, needed to evolve from manual management of this data to sophisticated and complex systems, including CRM, ERP, and others, each with its own data store. Different parts of the organization, maybe in different physical locations or perhaps not, were running different applications on different architectures. So what happened when you needed to compile a report for an audit, for example? You pulled the required data from each disparate system and manually consolidated it into a report, but that was neither practical nor efficient and almost certainly resulted in errors. An ETL tool, like Informatica PowerCenter, solved those problems by pulling data from multiple sources and transforming it in a way the business could use and easily and reliably report on.
The case for workload automation isn’t much different.
According to research by Gartner, most large organizations have between three and eight different automation tools in their environments, and managing these disparate automation tools brings additional layers of complexity. Instead of moving forward faster, IT departments are held back. Compounding the problem, IT budgets aren’t increasing much; according to John-David Lovelock, Research Vice President at Gartner, global IT spending will only rise 1.6% this year and will only grow 2-3% from 2017 to 2020.
The only way to combat these issues is a modern, architectural workload automation strategy that consolidates silos of automation within a single framework, giving organizations the IT agility they need while reducing the cost of change.
Picking the Right Workload Automation Solution
As an Informatica PowerCenter or Cloud user, there are special considerations, requirements, and features you should look for when selecting a workload automation solution. Here is a list of just some of those features, many of which specifically apply to the unique challenges of automating PowerCenter and Cloud workflows.
- Flow control and conditional logic capabilities that do not require custom scripting (if/then/else statements)
- Ability to ensure dependencies have been met, such as ensuring that a particular file is closed, before starting other dependent workflows
- Ability to trigger workflows sourced in and across multiple folders
- Ability to dynamically set runtime parameters to reduce the number of workflows you need to manage and maintain
- Complex event-based scheduling capabilities to more easily manage particular automation requirements, including synchronization and priorities
- Advanced date/time scheduling functionality to handle even the most complex custom schedules (like incorporating holidays, weekends, Fiscal and Gregorian calendars, and more) without needing to write special code
- A library of pre-built, pre-tested, templated integrations that allow you to easily, yet powerfully drag-and-drop job steps for Informatica and non-Informatica processes in order to build end-to-end workflows
- Ability to build and automate end-to-end workflows across Informatica and non-Informatica applications without custom scripting
- Built-in runtime monitoring to proactively monitor a job’s progress and send an alert if an ETL job or other job is running longer than expected; monitoring capabilities for jobs with SLAs to ensure meeting of expected completion requirements
- Advanced runbook, auditing, and logging capabilities
- Workflow optimizing tools to reduce the time to completion and insight
Get a Full List of Key Requirements for Workload Automation & Enterprise Job Scheduling Software
ActiveBatch is Transforming IT Automation
ActiveBatch® Workload Automation and Enterprise Job Scheduling provides Informatica PowerCenter and Cloud users a unique and powerful approach with capabilities beyond the built-in PowerCenter and Cloud schedulers and PMCMD. The ActiveBatch Integrated Jobs Library provides pre-built, pre-tested, templated integrations that can be dragged and dropped to easily and reliably automate PowerCenter and Cloud workflows. Users can also build workflows into end-to-end processes that pass data and manage dependencies without the need for custom scripting.
Advanced Systems Concepts, Inc., maker of ActiveBatch, is a recognized leader in the IT automation industry by leading analyst firms such as Gartner, Forrester, and EMA. The ActiveBatch Extensions for Informatica PowerCenter and Informatica Cloud are part of the Informatica Marketplace and have each earned the Informatica Seal of Approval.
Learn more about how you can transform your Informatica and data warehousing processes by reading how ActiveBatch customer Xcel Energy uses IT automation to streamline its Informatica PowerCenter processes.
Learn How You Can Transform Your Informatica PowerCenter and Cloud Automation:
