From the course: AWS Certified Machine Learning - Specialty (MLS-C01) Cert Prep: 1 Data Engineering

Unlock this course with a free trial

Join today to access over 24,100 courses taught by industry experts.

Job styles batch vs. streaming

Job styles batch vs. streaming - Amazon Web Services (AWS) Tutorial

From the course: AWS Certified Machine Learning - Specialty (MLS-C01) Cert Prep: 1 Data Engineering

Start my 1-month free trial Buy for my team

Job styles batch vs. streaming

“

- [Instructor] AWS has a couple different ways to think about jobs. One is Batch and the other is streaming. If we take a look at Batch, three services that really take a lot of the Batch terminology and put it together are Glue, AWS Batch, and AWS Step Function. In the case of Glue, what it does is it creates metadata that allows you to perform operations on something potentially that's in S3 or a database, and also do transformations. And this is a good example of how you could batch up something that's a really large amount of data. AWS Batch is really a general purpose batch-based framework where you can process anything at scale, and it's really good at doing many things in containers, as well as training models with GPUs. AWS Step functions allow you to really parametrize and put together different steps, and a good example would be orchestrating lambda functions with inputs that go from one to the other. In terms…

Contents