Como você pode experimentar a estratégia de dados mantendo a qualidade e a governança?

Forge a data strategy symphony: Define objectives and metrics, crafting a melody of experimentation. In this orchestrated dance, innovation harmonizes with quality and governance, ensuring your data strategy evolves dynamically without missing a beat.

Like a recipe guides cooking - first, decide the dish you want to make, then list key ingredients and steps needed. This focus helps efficiently shop and cook. Likewise with data strategy - first define business objectives, then identify metrics to track. This alignment provides a clear roadmap to demonstrate value, not just chopping and mixing data aimlessly.

It is imperative to also get support from various business areas on how can they support this Everything else kept apart, data initiatives will need personas like data stewards, data custodians to use the capabilities that are built as part of data architecture

In my workplace experience, this is almost always a story/ticket that your team builds out with well defined acceptance criteria. The key things to consider here are deadline, performance impact, and data quality impact. As an example, my company recently had my team compare a data processing change to our current state in a replica test environment. At the outset, we needed to know how many fields of the data elements in question changed, if the change aligned with a 3rd party source, and if the overall data processing pipeline suffered reduced performance.

Based on evolving business strategy , data strategy will be also evolving and dynamically changing to suit business models. But data quality and governance will be underlying fundamental fabric for now and future, should run through the evolving data strategy at any point in time.

The sandbox is crucial, providing a secure space to test data innovations without affecting production. It ensures adherence to quality and governance standards. For example, it's valuable for trying new integration techniques or validating machine learning algorithms before deploying to production.

I agree with this. A sandbox that can help in quickly turn ideas to value should be provisioned but should also have inbuilt capabilities safeguard data and handle concerns like privacy, secret data

In some ways, this step is self explanatory, right? If you want to test a change, don't do it live in production. However, in my experience theres a critical detail in how you create your sandbox. Many times I have been burned or saw others get burned by assuming randomized fake data will suffice for testing. Sure, the easiest way to test is on that random mock data, but the best way is to test on a model as close as possible to the data you are actually working with in production. If at all possible avoid taking the easy way out, test on a sanitized copy of real data unless you're certain its not relevant.

Remember have independent testing and development environment with replicated prod mode data to work: the more realistic to the production environment is better. Your CI/CD pipeline can help to build those environments.

In a hypothetical scenario, a company could pilot a new market analysis technique, and data quality checks would ensure the resulting insights are valid and actionable. These checks keep the company's data credible and trustworthy, which is vital when the business outcome relies on launching innovative products based on data-driven insights. Such strategic experimentation, along with solid data quality practices, allows for confident decision-making and maintains governance standards.

Data quality should be implemented at source and this DQ rules should be democratisesd for people to use and add rules to DQ profiling should be inbuilt as part of ETL pipelines

In a scenario where a business is running predictive analytics to enhance customer experience; data governance principles guide the collection and use of customer data in a way that is ethical and compliant with privacy regulations. This responsible approach safeguards the company against risks and solidifies trust with customers. Maintaining rigorous governance during strategic experimentation is instrumental in achieving advancements in data utilization without compromising on the core values of the organization.

Few practical examples of data governance capabilities are 1) Data lineage system to track how data moved within org and how physical data like table and column are related to high level data model and terminologies. 2) Data catalogue to make data searchable 3) defined Golden sources for data. 4) Data stewards to own the data and keep everyone aware about data 5) DQ rules.

When you experiment with production ready options for specified defined components in data strategy; you need a contingency plan to revert this change to keep system available and expected results. This plan need to define how change components, how keep sincronize in data as possible, integration between components and so on. Part of responsabilit in data governance is manage these cases.

When considering data governance principles, it's important to anonymize PII to ensure data privacy. This must be implemented throughout the data engineering and architecture process.

Your data strategy should be solid. It should support your business strategy. It really isn't something you experiment with. There are different ways you can achieve your data strategy. There can be different techniques and tools you can use. You can try an approach, realize it isn't working, and switch to another approach. But you're still striving to achieve the exact same data strategy. You might have to change your data strategy if the business strategy changes, but that's a natural part of business.

In my experience, the true meaning of experimenting with data strategy lies in learning and adaptation. Documenting and communicating experiment details, along with objectives and results, is key to building institutional knowledge. Leveraging feedback and metrics provides valuable insights into the effectiveness of your data solutions

Working in data strategy bitacora to document your experiments and how it's related with data strategy. For each experiment you need to say: - Goal of experiment. - Problem want to solve or improve; based both business view and technical view. - Define options to validate and why choose this options. - Definition of metrics, parameters, acceptance thresholds and validations criterios to measure each options. - Show result of options by friendly way. - Staff work in experiments: both on the business and technical side with detailed about them. - Used resources: platform, paterns, time, cost and others related things. - Taken decision of experiment from both on the business and technical side.

In data strategy, balancing innovation through experimentation with strict governance and quality is crucial. The 'fail fast, learn fast' approach is key, urging data teams to quickly prototype and learn from failures, a method especially useful in dynamic sectors like telecom and retail. Adopting agile methodologies helps in shortening project timelines, and keeping pace with changing business landscapes. Maintaining data quality is vital, necessitating regular quality checks to ensure reliable, actionable insights. Collaborative experimentation involving various stakeholders ensures experiments are not only technically sound but also business-aligned. This balanced approach fosters innovation while upholding data integrity and utility.

Remember the experiment with data strategy is part of evolution and adaptation of data system to support business strategy: part of cycle in data architecture.

For ex, A retail company wants to improve its customer segmentation using ML algorithms. To experiment with different segmentation approaches without affecting production data, a data sandbox is created. We apply data governance principles to ensure data privacy, security during experimentation process. We implement data quality checks to verify the accuracy, completeness of the data used in the experiments. Throughout the process, we monitor and track the experiments to assess impact on data quality and governance. We gather feedback from stakeholders and incorporate it into experimentation process. By following these strategies, the retail company can effectively experiment with data strategy while maintaining data quality and governance.

Synthetic data and self-service infrastructure are essential in modern data strategies for rapid experimentation and innovation. Synthetic data, artificially generated to mimic real datasets, enables safe and scalable testing without compromising privacy - a critical factor in fields like AI, where data is the lifeblood but often sensitive. Self-service infrastructure empowers teams to access and manipulate data resources independently, speeding up processes and reducing bottlenecks. By integrating these elements, organisations can significantly enhance their agility and capacity for innovation, enabling them to adapt quickly and effectively to new challenges and opportunities.

There are so many tools and tech available in data space , you should focus on how do you consolidate various things and build a reusable framework rather than vendor lock in solutions