If you need data prepared in advance, like reports and aggregations, without any time delays, opt for ETL. However, if your focus is on data analysis and running ad-hoc queries, ELT is the better choice.

To prioritize ETL processes, I first identify which data sources offer the most valuable information for business goals. This ensures alignment with strategic objectives and maximizes impact. Next, I prioritize based on business rule complexity and data volume. Even small data sources with intricate rules can significantly affect ETL performance. I've found success using ELT (Extract, Load, Transform) instead of ETL. This approach enables building multiple pipelines and intermediate datasets, allowing parallel and interconnected execution for data transformations. It also enhances maintenance, encapsulates business rules effectively, and improves data quality monitoring and governance.

Switch to ELT process instead of ETL due to the cheaper storage cost vs spending more time on cleaning it before load. Once the data is stored safely, prioritize the sources that are business critical applications first to transform with clean data.

In general, below data set are faster for ETL: 1. 1-1 mapping data set (without transformation) are faster. 2. Smaller data set are faster. 3. Data set that supports delta capability are faster due to less volume. 4. Good quality data are faster (since corrections are not required during ETL) 5. In many cases, native connector are faster compared to generic connector ETL priority among data sources depends on: 1. Business critical and then Non-critical 2. Operational (Daily) and then strategic (Daily, Weekly, Monthly) 3. Single source data set and then multi-source data set (federated data) Priority within a source system: a. Master data b. Transaction data (with no dependency) c. Transaction data (with dependency on a and/or b)

In my experience with ETL tools, I have extensively worked with Informatica, which I regard as one of the best solutions in the field. When dealing with an overwhelming number of data sources, the priority should be on those that are critical for business insights, have the highest frequency of updates, and require minimal transformation to maintain speed. Informatica's robust features, such as efficient data mapping and seamless integration capabilities, significantly contribute to optimizing these processes, ensuring both speed and reliability in data handling.

Overwhelmed with ETL data sources? Prioritize for speed by focusing on key factors. Analyze data: prioritize high volume, velocity, and structured formats. Consider source system performance and data quality. From a business perspective, focus on valuable, time-sensitive, or regulated data. Technically, optimize extraction methods, ensure strong connectivity, and consider data virtualization or Change Data Capture (CDC). Employ the 80/20 rule and data profiling to pinpoint bottlenecks. This combined approach of data, business, and technical considerations will accelerate your ETL processes.

Decide most critical datasources you need , define which datasources could be used with lower frequency (slowly changing dimensions for example), which data you need on daily, weekly / monthlt basis. Stop integrating data "for future usage" and focus on really necessary data. Between those identified prioritize those which are well structured already and needs low investment into complex transformation.

Prioritizing ETL data sources can be overwhelming, especially if there are more than a 100. There are several ways to optimize, and they are more or less relevant by context. 1. I would not touch any source before understanding my business data flows. Relevance of specific data elements is key. Any data elements that are not used often can typically wait 2. Split the source files to Key or critical, Useful, maybe useful and partition vertically. You may load into a landing zone to do this. 3. Evaluate for sparsity. Any field that is very sparse may have limited general value. Evaluate for ref data mapping. If the hierarchies are not leveling with your target, that is important before you load 4. Size, frequency, conformance all matter

Do ETL processes feel like juggling bowling balls? Here's how to take control of the chaos: Imagine you're a retailer drowning in data sources - point-of-sale systems, online transactions, social media analytics. Instead of tackling everything at once, you focus on the biggest player: your online data, which makes up 70% of your total volume. A few smart optimizations later, you've halved the processing time. Next, you tackle the messy customer feedback data, streamlining the cleaning process so it’s as easy as pitting cherries. The result? Faster insights, less stress, and finally time for your second coffee. So, start with the heavy hitters - clean, important, and easy to integrate. What’s your go-to ETL strategy for cutting the chaos?