Friday 26 August 2016

Which ETL tool?

You are wandering what is the best tool to gather, meaning connect and extract from your data sources, and nanage data, that is, updating and backing up in the database. Well you have a few choices like:

  1. Oracle Warehouse Builder (OWB)
  2. Oracle Data Integrator (ODI)
  3. SAP Data Services
  4. IBM Infosphere Information Server
  5. SAS Data Management. 
  6. PowerCenter Informatica. 
  7. Elixir Repertoire for Data ETL 
  8. Data Migrator (IBI) 
  9. SQL Server Integration Services (SSIS)
  10. Talend
  11. Pentaho
  12. many others...
But, why be pushed to buy a tool when perhaps the best method is basic SQL Scripting?

SQL Scripting will probably be cheapest choice and the easiest way to implement and maintain your own data warehouse. You can build your own queries and run different extract and processing tasks using SQL scripts and scheduling using SQL Server Jobs or the easy option that is Windows Scheduler. Don't reinvent the wheel and if you want raw data in your data warehouse you need only to identify and connect to the source and write some basic select insert into queries, you have your base data. Once you begin using different tools you will start loosing focus on the core task at hand, finding and understanding the data, systems and processes.

Many times companies build complex ETL processes to collect data with complex data flows and auditing procedures that usually are of no use or value for the stakeholder and end-user. The usual arguments are compliance, control and efficiency, but the business case for complicated methods is flaud. It becomes so difficult to maintain and change that it does not answer what it was build for. The costs increase, with high maintenance like licensing and payroll and usually because it does not fulfill the requirements of users you are bringing to life shadow IT and end-user computing with users finding their way to the information they want and building the databases they need, also breaking the compliance requirement.

Keep it simples, flexible and use easy to use tools that can also be made available to end-users so it becomes an collaborative and value-adding endeavour.

No comments:

Post a Comment