Vytas Jelinskas ☁’s Post

Are you looking to enhance the resilience of your application's user authentication system? We've developed a robust solution to replicate #AWS Cognito user data across multiple regions, perfect for maintaining seamless operations during unexpected outages. This blog is by Akshay Mewara walks you through a serverless architecture using #AWSLambda and #DynamoDB global tables. Read the full blog now to get started with cross-region user data replication - https://lnkd.in/grMZGX2M #AWSCognito #CloudReplication #ServerlessArchitecture #DisasterRecovery #CloudSecurity #UserAuthentication #DevOps #CloudComputing

Here is a post by a Databricks Resident Solution Architect Arun Wagle on "The Keys to Production Data and AI Applications with Azure Databricks." Check it out! I really like the section on Databricks Asset Bundles (DABs), and the relevent examples for how to integrate DABs using Azure Devops Pipelines and CI/CD. #dataengineering #datascience #machinelearning #mle #llms #dataanalyst #de #largelanguagemodels #dataanalyst

We’re excited to announce the deployment of #Trino—a powerful distributed SQL query engine—using a highly available Amazon ECS setup on EC2, fully automated with Terraform. #Trino’s ability to query massive datasets across multiple sources with speed and efficiency makes it a perfect solution for handling big data analytics. This setup offers high availability, resilience, and optimal scaling, ensuring uninterrupted performance for our data-intensive operations. Our team’s hard work paid off in configuring this robust infrastructure to handle data challenges seamlessly. This setup is ideal for real-time analytics, multi-source querying, and large-scale data lake management. #Trino and #ECS are a match made in heaven for scalable, reliable data querying. We encourage others to give it a try for their big data needs—it’s a game-changer! Kudos to Raj Singh for his hard work in making this implementation a success! #TechDeployment #CloudSetup #InfrastructureAsCode #DevOpsSuccess #TrinoQueryEngine #ECSDeployment #AWSonECS #QueryEngine #TerraformMagic #IaC #AutomateEverything #BigDataSolutions #DataAnalytics #DataQuerying #DataDriven #TeamworkMakesTheDreamWork #TechMilestone #ProjectSuccess #EngineeringExcellence

𝗦𝘂𝗰𝗰𝗲𝘀𝘀𝗳𝘂𝗹𝗹𝘆 𝗗𝗲𝗽𝗹𝗼𝘆𝗲𝗱 𝗮 𝗛𝗶𝗴𝗵𝗹𝘆 𝗔𝘃𝗮𝗶𝗹𝗮𝗯𝗹𝗲 𝗘𝗖𝗦 𝗦𝗲𝘁𝘂𝗽 𝗳𝗼𝗿 𝗧𝗿𝗶𝗻𝗼 𝗤𝘂𝗲𝗿𝘆 𝗘𝗻𝗴𝗶𝗻𝗲 𝗼𝗻 𝗘𝗖𝟮 𝘄𝗶𝘁𝗵 𝗧𝗲𝗿𝗿𝗮𝗳𝗼𝗿𝗺! 🚀 For those unfamiliar, 𝗧𝗿𝗶𝗻𝗼 is a powerful distributed SQL query engine for high-performance analytics across data lakes, cloud storage, and databases, all within a single query interface. Its scalability and SQL compliance make it essential for big data analytics. Here's a breakdown of my Terraform-based setup: 🔹 𝗠𝗮𝘀𝘁𝗲𝗿 𝗦𝗲𝗿𝘃𝗶𝗰𝗲: • 𝗛𝗶𝗴𝗵 𝗔𝘃𝗮𝗶𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆: Configured with a load balancer for traffic distribution. • 𝗗𝗲𝗱𝗶𝗰𝗮𝘁𝗲𝗱 𝗖𝗮𝗽𝗮𝗰𝗶𝘁𝘆: Runs in an auto-scaling group (ASG) with only on-demand instances to ensure reliability. • 𝗦𝗶𝗻𝗴𝗹𝗲 𝗧𝗮𝘀𝗸 𝗗𝗲𝘀𝗶𝗴𝗻: One active master task maintains centralized control and efficiency. 🔹 𝗪𝗼𝗿𝗸𝗲𝗿 𝗦𝗲𝗿𝘃𝗶𝗰𝗲: • 𝗖𝗼𝘀𝘁-𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗦𝗰𝗮𝗹𝗶𝗻𝗴: Supports up to 10 workers with a 30-70 on-demand to spot instance ratio. • 𝗙𝗹𝗲𝘅𝗶𝗯𝗹𝗲 𝗦𝗽𝗼𝘁 𝗔𝗹𝗹𝗼𝗰𝗮𝘁𝗶𝗼𝗻: Uses multiple instance types to enhance spot instance allocation. • 𝗗𝗼𝘄𝗻𝘁𝗶𝗺𝗲 𝗔𝘃𝗼𝗶𝗱𝗮𝗻𝗰𝗲: Ensures the first instance is always on-demand, with workers connecting directly to the master, eliminating the need for a load balancer. 💡 𝗞𝗲𝘆 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀: • High Availability and Cost Optimization to support demanding Trino queries. • Scalable, Resilient Architecture for reliable analytics across large data sets. #ECS #AWS #Terraform #CloudComputing #DevOps #Trino #BigData #CostOptimization #devops #devopsengineer #devopsengineering #devsecops #sre #ai #terraform

🚀 What Are Consistent Reads from Cache in Kubernetes? 🚀 In Kubernetes, the control plane manages cluster state through the etcd datastore. Traditionally, to ensure strong consistency, reads require contacting multiple etcd nodes (quorum reads), which can be resource-intensive and create bottlenecks. Consistent reads from cache allow Kubernetes to serve read requests using cached data instead of querying etcd directly every time. This approach maintains data accuracy while reducing the load on the datastore. How It Works: 1)Caching Mechanism: Kubernetes maintains a cache of recent data, which is refreshed periodically or when changes occur. 2)Fallback to Quorum Reads: If the cache cannot provide the needed data, Kubernetes will perform a quorum read to ensure consistency. Key Benefits: 1)Performance Improvement: Reduced Latency: By using cached data, read operations are faster, minimizing delays in data access. Lower Resource Usage: Decreases the demand on etcd, freeing up resources for other processes. 2)Scalability: Efficient Scaling: As clusters grow, the ability to handle more read requests without overwhelming etcd is crucial for maintaining performance. Use Cases: 1)Large Clusters: Particularly beneficial for large-scale environments where reducing etcd load can significantly improve overall performance. 2)Frequent Reads: Workloads that require frequent state checks benefit from faster access to data #Kubernetes #CloudNative #DevSecOps #PerformanceBoost #OpenSource

DevOps Day 12 Before: I was deploying the Azure Data Factory using ARM templates generated by clicking the "Validate" and "Publish" button. Every time there was a change, I had to manually click both the buttons which used to update the changes in Azure Repo. Now: An automation for this also has been created. Now we can create a build pipeline which will generate the ARM templates for us every time a change has been made and hence trigger the release. Limitation of this method: There is a Node Script to export the ARM template. If we are going to utilize the same template without much change then we can go ahead. But if we have any modification in ARM template then we won't be able to use this method. For example: if lots of parameterization is required then it won't be suitable to use this method. Thanks to Yash Pratap Singh for sharing the below doc https://lnkd.in/gQc5NAEQ #devops

As an Azure Data Engineer, I’ve seen how adopting CI/CD pipelines can transform data projects. Here’s why CI/CD is a game-changer in the data space: ⚡ Automation: No more manual deployments—CI/CD ensures consistent, reliable data pipeline updates. 📈 Faster Releases: Deploy changes to Azure Data Factory, Synapse, or Databricks seamlessly. 🔍 Quality Assurance: Automated testing catches issues early, maintaining data accuracy and integrity. 🔄 Version Control: Track changes and roll back effortlessly with tools like Git and Azure Repos. 📊 Continuous Feedback: Real-time monitoring and feedback improve efficiency and collaboration. From optimizing ETL workflows to enabling real-time analytics, CI/CD helps us build scalable, high-performance solutions. 💡 How has CI/CD transformed your data engineering workflows? Let’s discuss best practices! 👇 #DataEngineering #Azure #CICD #DevOps #CloudComputing

Is Kubernetes ready for stateful workloads? This quarter, we're planning to migrate #Clickhouse, which powers the "fast query" layer in our analytics stack, to Kubernetes. The motivation is the same as I described in my "Kubernetes for Data Engineering" post 3 years ago - Kubernetes is easier for us data engineers than a mix of EC2, custom AMIs, Ansible, and black magic. We also had moderate success using the #Altinity operator to run Clickhouse for experimental projects. That said, all my backend colleagues are giving me strange looks. Can anybody share any advice - what are the primary pitfalls I should expect?

Day 19 of #90daysofdevops 🔹 Docker volumes are essential for managing data persistence and sharing between containers in multi-container setups. 🔹 They decouple data storage from container lifecycles, ensuring data integrity, scalability, and easy management. 🔹 With Docker Compose, volumes enable seamless data sharing between services, promoting modularization and enhancing security. Read the complete post: https://lnkd.in/g5VCb3wM 🔹 Docker volumes facilitate collaboration among containers without the constraints of container boundaries, simplifying data management in multi-stage environments. 🔹 Mastering Docker volumes is crucial for efficient container orchestration and robust application deployment. #Docker #Containerization #DevOps

Deploying, scaling, and managing #storage for #Kubernetes applications involve more than just ensuring #data persistence outside of the #container lifecycle. As Kubernetes #applications scale quickly and dynamically, they introduce additional challenges. Read through on how #Portworx can help you overcome these challenges. #PureStorage #Ease of #Use.