Datavolo’s Post

Last week we launched a technical capability that’s been in my head for years, and today I want to tell you about how we built it. When we started work on Oso, we knew that one of the hardest problems to solve is how to draw a clean boundary around authorization-relevant data. With last week’s launch of Distributed Authorization, we let you store common authorization data in Oso Cloud and leave the rest of your application data (e.g., resource relationships) in your database. We achieved this by building Polar, a logic language for declaratively configuring authorization, compiling Polar down to SQL, and making it possible to run + evaluate Polar across multiple databases. I wrote a detailed post on this, A logic language for distributed SQL queries.

📣🎉We are excited to announce that the Postgres CDC connector for ClickPipes is now in Private Preview! https://lnkd.in/d8H3i-zK With this connector, you can natively replicate your Postgres data to ClickHouse Cloud in just a few clicks for blazing-fast analytics, eliminating the need for external tools that are often expensive and slow. Key benefits include. 🚀 Blazing Fast Performance: Achieve 10x faster initial loads with replication latency as low as a few seconds. 🎯 Super Simple: Start replicating in just a few clicks and a few minutes. 🛠️ Native Features: Supports Postgres and ClickHouse native features such as schema changes, custom order keys, and more. 🔒 Enterprise-grade Security: All data is encrypted in transit, with support for features such as SSH tunneling and Private Link. 🌐 No Vendor Lock-in: Powered by open-source PeerDB https://lnkd.in/dA864RUs This launch marks a significant milestone following the PeerDB (YC S23) acquisition. Many customers, including SpotOn , Vueling Airlines , Daisychain and others, are already leveraging this technology to continuously replicate their Postgres databases to ClickHouse for analytics. You can sign up for the private using this link - https://lnkd.in/dzzst5fz Our team will reach out within a day to provide you with access.

We got a lot of questions about this. After yesterday's post on Apache Iceberg, we realised that there are a lot of people that are comparing Iceberg and Delta Lake Comparing Apache Iceberg and Delta Lake: When choosing between Apache Iceberg and Delta Lake, it's important to look at both business and technical points. Business Questions Data Rules and Compliance: How do Apache Iceberg and Delta Lake follow rules like GDPR or CCPA? Which one is better at tracking changes and keeping records to follow these rules?  Both Apache Iceberg and Delta Lake can track changes and keep records well. Delta Lake has a slight advantage because it has built-in tools for auditing and compliance, making it a bit easier to follow strict rules. Cost and Resource Management: Question: How much does it cost to use Apache Iceberg vs. Delta Lake? Think about licensing, storage costs, and infrastructure. Which one saves more money and resources? Apache Iceberg is free to use, which can save money at the start. Delta Lake is also free, but it often works with Databricks, which might cost extra. Both save money by managing data efficiently. The best choice depends on what your company already uses and its budget. Vendor Lock-In and Compatibility:  How do these technologies fit with our current systems and cloud providers? Which one offers more flexibility and works better with other tools we use?  Apache Iceberg is very flexible and works with many cloud providers and big data tools, reducing the risk of being locked into one vendor. Delta Lake works great with Databricks but may create some dependency if you use Databricks a lot. Both work well with popular tools like Spark and Hive. Technical Questions Performance and Scalability: How do Apache Iceberg and Delta Lake compare in speed and ability to grow? Look at things like query speed, data processing, and handling large amounts of data. How do they perform with many users and big datasets? Delta Lake is very fast, especially with Databricks, offering quick queries and high data processing speeds. Apache Iceberg is also fast and can handle large data well. Both are designed to support many users and big datasets effectively. Data Consistency and Transactions:  How do Apache Iceberg and Delta Lake ensure data consistency and handle transactions? How do they manage complex transactions and conflicts to keep data reliable? Both Apache Iceberg and Delta Lake support ACID transactions, which keep data consistent and reliable. Delta Lake uses a transaction log to manage conflicts, while Iceberg uses metadata management. Both are good for complex data tasks and ensure reliable operations with multiple users. Apache Iceberg and Delta Lake are strong choices for managing data. The right one depends on your company's needs, current systems, and goals.

I've talked before about how creative and thoughtful the team at Oso is. This article is a great example. There's a lot of fascinating stuff here - logic programming, SQLite in production, declarative vs imperative languages. Check it out if that's your kind of thing.

Deeply technical post on the language we've built and how we compile it down to SQL to run SQL queries across multiple databases

This is the first result of the PeerDB (YC S23) acquisition by ClickHouse that happened a few months ago. I am super proud and humbled of what the team was able to achieve in just a few months. Now there is a native way to integrate your Postgres databases to ClickHouse in just a few clicks! I want to thank all our customers, including SpotOn, Daisychain, Vueling Airlines, Adora, and others, who have been running production workloads for several months. Without you, this integration wouldn’t have been possible! I am a strong believer that customers will find the maximum value using purpose-built databases, as they are designed and with full flexibility, rather than relying on alternatives that retrofit one database engine into another, compromising the full feature set of each. This has been our approach with Postgres and ClickHouse—Postgres for transactional (OLTP) workloads, ClickHouse for analytical (OLAP) workloads, and a seamless Postgres CDC to bring them closer, forming a powerful data stack. We are seeing this trend across various customers who are using this data stack to solve most of their data challenges efficiently. I love this testimonial from one of our customers, Nathan Woodhull from Daisychain, which concisely summarizes our approach and is a testament to what we are building. “Clickpipes helps us reliably keep data from Postgres in sync with Clickhouse, while we rapidly improve our product. Clickhouse and Postgres are the peanut butter and chocolate of databases (they go well together!) and Clickpipes makes it easy to build that architecture at a fast-moving startup” ClickHouse and Postgres are the peanut butter and chocolate of databases. We are very excited for what is in store for the future. The vision is very clear - making it magical for customers to use Postgres and ClickHouse together. We will continue driving towards that effort. More to the future! 🚀 cc Kaushik Iska, Ryadh Dahimene Ph.D, Pete Hampton, Amogh Bharadwaj, Kunal Gupta, Kevin Biju, Philip Dubé, Cristina Albu, Mikhail Shustov, Kelsey Schlarman, Tanya Bragin

Here is our next blog with #tidb where in we delve in to the journey of writes #tidb #distributedsql #datamanagment #databases #mydbops #remotesupport https://lnkd.in/gEGUYnJY

👋 Some interesting news in the world of #distributedsql, #postgresql, #cassandra, and #yugabytedb #database Combine Transactional Integrity and Data Lake Operations with YugabyteDB and Apache Hudi Discover how integrating YugabyteDB with Apache Hudi transforms data lakehouse capabilities, offering unparalleled real-time processing and scalability. This blend of technologies not only manages vast data volumes efficiently but also ensures transactional consistency. The post Combine Transactional Integrity and Data Lake Operations with YugabyteDB and Apache Hudi appeared first on Yugabyte. 👀 Read more at https://lnkd.in/grEDKfSF 🎓 For courses, training, workshops, and certifications, enroll for FREE at https://lnkd.in/dubG63pp🏆

Apache Kafka has gained significant popularity in recent years, with companies like Netflix, LinkedIn, and Uber leveraging its capabilities to handle high-volume data streams efficiently. To help you grasp the key concepts of Kafka, I've created a handy diagram that breaks them down in a simple and easy-to-understand way. 𝗣𝗿𝗼𝗱𝘂𝗰𝗲𝗿: A Kafka producer is an entity responsible for publishing data to specific topics within the Kafka cluster. These producers act as the sources of data streams, which can originate from various applications, systems, or sensors. They push records into Kafka topics, with each record consisting of a key, a value, and a timestamp. 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿: A Kafka consumer, on the other hand, pulls data from the Kafka topics to which it subscribes. Consumers process the data and often belong to a consumer group. Within a group, multiple consumers can read from a topic in parallel, with each consumer responsible for reading from specific partitions, ensuring efficient data processing. 𝗧𝗼𝗽𝗶𝗰: A topic in Kafka represents a category or feed name to which records are published. Topics are multi-subscriber, meaning they can be consumed by multiple consumers and consumer groups simultaneously. To enable data scalability and parallel processing, topics are divided into partitions. 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻: A topic can be split into partitions, which are essentially subsets of the topic's data. Each partition is an ordered, immutable sequence of records that is continually appended to. By splitting data across multiple brokers, partitions allow topics to be parallelized for enhanced performance. 𝗕𝗿𝗼𝗸𝗲𝗿: A broker refers to a single Kafka server that forms part of the Kafka cluster. Brokers are responsible for maintaining the published data, and each broker may have zero or more partitions per topic. 𝗖𝗹𝘂𝘀𝘁𝗲𝗿: A Kafka cluster is composed of one or more brokers working together to provide scalability, fault tolerance, and load balancing. The cluster manages the persistence and replication of message data across the brokers. 𝗥𝗲𝗽𝗹𝗶𝗰𝗮: A replica is a copy of a partition that Kafka creates to ensure data is not lost if a broker fails. Replicas are classified as either leader replicas or follower replicas, each serving a specific purpose. 𝗟𝗲𝗮𝗱𝗲𝗿 𝗥𝗲𝗽𝗹𝗶𝗰𝗮: For each partition, one broker is designated as the leader. The leader replica handles all read and write requests for that partition, while other replicas simply copy the data from the leader. 𝗙𝗼𝗹𝗹𝗼𝘄𝗲𝗿 𝗥𝗲𝗽𝗹𝗶𝗰𝗮: Follower replicas are copies of the leader replica for a partition. Their purpose is to provide redundancy and take over as the leader if the current leader fails. They replicate the leader's log but do not serve client requests directly. #python #data #datascience #programming #dataanalysis #analytics #productivity #innovation #coding #tech #softwareenginee #developer #technology #techcommunity #aicommunity

Apache Kafka has gained significant popularity in recent years, with companies like Netflix, LinkedIn, and Uber leveraging its capabilities to handle high-volume data streams efficiently. To help you grasp the key concepts of Kafka, I've created a handy diagram that breaks them down in a simple and easy-to-understand way. 𝗣𝗿𝗼𝗱𝘂𝗰𝗲𝗿: A Kafka producer is an entity responsible for publishing data to specific topics within the Kafka cluster. These producers act as the sources of data streams, which can originate from various applications, systems, or sensors. They push records into Kafka topics, with each record consisting of a key, a value, and a timestamp. 𝗖𝗼𝗻𝘀𝘂𝗺𝗲𝗿: A Kafka consumer, on the other hand, pulls data from the Kafka topics to which it subscribes. Consumers process the data and often belong to a consumer group. Within a group, multiple consumers can read from a topic in parallel, with each consumer responsible for reading from specific partitions, ensuring efficient data processing. 𝗧𝗼𝗽𝗶𝗰: A topic in Kafka represents a category or feed name to which records are published. Topics are multi-subscriber, meaning they can be consumed by multiple consumers and consumer groups simultaneously. To enable data scalability and parallel processing, topics are divided into partitions. 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻: A topic can be split into partitions, which are essentially subsets of the topic's data. Each partition is an ordered, immutable sequence of records that is continually appended to. By splitting data across multiple brokers, partitions allow topics to be parallelized for enhanced performance. 𝗕𝗿𝗼𝗸𝗲𝗿: A broker refers to a single Kafka server that forms part of the Kafka cluster. Brokers are responsible for maintaining the published data, and each broker may have zero or more partitions per topic. They can handle data for multiple topics simultaneously. 𝗖𝗹𝘂𝘀𝘁𝗲𝗿: A Kafka cluster is composed of one or more brokers working together to provide scalability, fault tolerance, and load balancing. The cluster manages the persistence and replication of message data across the brokers. 𝗥𝗲𝗽𝗹𝗶𝗰𝗮: A replica is a copy of a partition that Kafka creates to ensure data is not lost if a broker fails. Replicas are classified as either leader replicas or follower replicas, each serving a specific purpose. 𝗟𝗲𝗮𝗱𝗲𝗿 𝗥𝗲𝗽𝗹𝗶𝗰𝗮: For each partition, one broker is designated as the leader. The leader replica handles all read and write requests for that partition, while other replicas simply copy the data from the leader. 𝗙𝗼𝗹𝗹𝗼𝘄𝗲𝗿 𝗥𝗲𝗽𝗹𝗶𝗰𝗮: Follower replicas are copies of the leader replica for a partition. Their purpose is to provide redundancy and take over as the leader if the current leader fails. They replicate the leader's log but do not serve client requests directly. Have I overlooked anything? Please share your thoughts—your insights are priceless to me.