Using Azure Synapse Serverless Pools for Data Engineering | Data Exposed
Data engineering is one of the most involved, dull, and repetitive tasks in the data industry today. Bringing up the subject is a sure way to instantly clear the room at a party or make your friends groan in despair. The work includes a lot of complicated tools, many, many lines of code, and a not-insignificant amount of gnashing of teeth. But what if I told you there is a way to quickly, easily, and cheaply do both data engineering and data exploration on most data without even having to import it into a database? What if I told you that Azure Synapse has a just about magical tool that can instantly connect to flat files and make them available as nice, well-behaved tables? I'm talking about Azure Synapse serverless pools, a great tool for both initial exploration and surprisingly complex data engineering. I'll walk you through what it is, how to set it up, and give you a few examples of basic data exploration, as well as a tad more complex data engineering. Oh, and did I mention you can actually store the end result in several different formats as well...?
[00:56 ] Azure Synapse Serverless Pools Overview
[01:54 ] Demo
[11:05 ] Getting started
Andy Cutler's blog on serverless pools
Andy Mallon's blog on bucketing with T-SQL
About Alexander Arvidsson:
Alexander is a principal solutions architect at Attollo in Sweden. Nobody really knows what it means to be a principal solutions architect, but he spends his days speaking, teaching courses or helping clients of all shapes and sizes to take better care of their data.
He has spent the last 25 years toying with data, databases and related infrastructure services such as storage, networking and virtualization, occasionally emerging from the technical darkness to attend a Star Wars convention somewhere in the world.
He is a frequent international speaker, Data Platform MVP, podcaster, Pluralsight author, blogger and a Microsoft Certified Trainer, focusing on the Microsoft data platform stack.
Microsoft Most Valuable Professionals, or MVPs, are technology experts who passionately share their knowledge with the community. They are always on the "bleeding edge" and have an unstoppable urge to get their hands on new, exciting technologies. They have a very deep knowledge of Microsoft products and services, while also being able to bring together diverse platforms, products, and solutions, to solve real-world problems. MVPs make up a global community of over 4,000 technical experts and community leaders across 90 countries/regions and are driven by their passion, community spirit, and the quest for knowledge. Above all and in addition to their amazing technical abilities, MVPs are always willing to help others - that's what sets them apart. Learn more: https://aka.ms/mvpprogram