Video Transcript:
- What if I told you you can quickly spot real-time indicators of issues as they unfold without the need to poll or manually monitor changes in your data and without writing a single line of code. That’s the goal of the new Real-Time Intelligence service, part of Microsoft Fabric’s platform. It extends Fabric to the world of streaming data across your IoT and operational systems. As a data analyst or a business user, you can easily explore high-granularity, high-volume data, and spot issues before they impact the business, and as a data engineer, you can more easily track system-level changes across your data estate to manage and improve your pipelines, and today I’m joined by Courtney Berg, who also helped build Microsoft Fabric. Welcome.
- Hey, Jeremy. Thanks for having me on the show.
- Thank you, and congrats on the availability of Real-Time Intelligence in Microsoft Fabric today. Now, most of us are familiar with event orchestration systems, but this is a lot different than that, so what makes this different?
- Well, this will give you an intelligent, unified, no-code way to listen and analyze real-time changes in your data wherever it lives. So, for example, you might have telemetry data collected from your business systems sitting in other clouds, along with streaming data of your IoT devices on the edge. You can pull from the sources you want and transform and combine the data in real time. And then with interactive real-time analysis, you can explore the data, spot emerging patterns, and isolate that single data point that could be the first indicator of an issue. We make query authoring easier with generative AI to help you quickly discover insights, and you can act faster by establishing rich conditions for active monitoring and defining what should happen next, so whether it’s notifying the right team or triggering automated workflows for system-level remediations. So, you can build your own custom integrated and automated systems to detect changes in data across your estate, analyze these events in context, and trigger early actions, all without writing a single line of code.
- Right, and if we compare that to most event-based systems out there today that are tied to specific data sources and those aren’t listening to changes in data at an aggregate level, this is changing things significantly, so what’s behind that?
- Well, as you mentioned, Real-Time Intelligence is part of Microsoft Fabric, and what it does is it orchestrates the process of being able to ingest streaming and event data in real time, analyze and transform it, and then act on it. The way that it does this is through a number of capabilities. So, first is Eventstream, which lets you bring in data, whether it’s from Microsoft Services, external sources, or even change data feeds from operational databases using available connectors, and a major thing we’re solving for here is how to capture real-time data in motion so that you could directly act on insights while they’re fresh. So here, underlying everything is the new Real-Time hub. This provides a single location for streaming data, as well as discrete event data across your organization happening at a system level. Now importantly, as a central location, it’s also cataloging the data, and it makes it easy to search for and discover real-time data: something that’s been historically difficult. And from here, you can take two paths to make a decision for your available data. So first, as data comes in, you can take immediate action using our Reflex capability, which is part of Data Activator, to look for rich conditions to trigger notifications or specific processes. Secondly, your data can go directly into our Eventhouse that provides a unified workspace to work with all of your data and is optimized for time series data. From there, you can easily query it with KQL and visualize it on our Real-Time Dashboard before acting on it with Reflex.
- So, just to pause you there for a second, so where do existing components for Fabric like Synapse Real-Time Analytics then fit into the picture?
- Yeah, so what we’re doing is we’re removing those tech silos so that we could better orchestrate the entire lifecycle, from capturing, analyzing, and acting on Real-Time Intelligence. We’ve built Real-Time Intelligence on top of proven technologies while adding new functionality. So, capabilities from Synapse Real-Time Analytics and Data Activator in Fabric have been unified, and there’s also a number of real-time streaming analytics and data exploration features from the Azure platform, along with Fabric’s strength in data visualization from Power BI that we bring in under the covers.
- Right, this is really going to provide a familiar experience, while expanding the approach to addressing the specific challenges of working with real-time data.
- That’s right, and all your Entra ID, information protection and governance policies, they all apply here.
- So, can you walk us through an example with all this running?
- Sure, I’m going to show you a scenario focused on business events where we take change data recorded from one system to another that causes a butterfly effect. In this case, we’re a direct-to-consumer food retailer. So, imagine it’s a hot spell and our sales and marketing team want to improve customer loyalty and satisfaction. They’ve come up with this idea of an aggressive discount on ice cream, which sounds like a great idea, but there’s a chain of dependencies with different teams and systems on point, and it’s not until you integrate these systems together that you can catch and react to what’s unfolding in real time. Plus, we want to catch that sweet spot of early indicators, and Real-Time Intelligence will do that for you with zero code. Here in our Real-Time Dashboard, you can see we’ve brought in all the relevant information across multiple systems into one view. Data is coming in from our sales and stock system, which gets updated hourly, I have real-time information from our IoT sensors with the refrigeration temperatures in our stores, and on the right here, I see data from Postgres backend of our mobile delivery app, showing orders ready to be picked up and available drivers, and in fact, we see average temperatures across our freezers are increasing over the last few hours.
- So, just because we have perishable goods here, and like you said, it’s a hot day, we’ve also got logistical dependencies, including different teams that are on point, there is a lot of things that could potentially go wrong here, so how do we get ahead of something like this?
- Yeah, listen closely, because the devil’s in the details. You’ll notice that our dashboard currently shows aggregate numbers across multiple freezers across different departments. So, the first gotcha is I don’t have a view of the individual freezer levels to be able to spot hidden issues with the freezers containing the ice cream, so let’s dig in a little bit more. Now, I can manually query to see what’s going on, but to save time, I’ll ask Copilot to do this for me. I’ll paste in my prompt, “Show the average temperature by department as column chart,” and it generates a KQL query that I can insert to get my chart. This looks pretty useful, so I’ll pin it to a dashboard to keep me updated in real time. Now, I’ll select the existing dashboard, give the tile a name, and add it. I’ll move it where I want it, resize it, and now I have visibility over freezers by each department.
- So, I can tell there’s a lot of different things to look into here, so from the report directly, can we slice and dice that data?
- Yes, everything is filterable live and can be queried on each tile. Let’s explore the freezer data some more. I’m going to drill into the aggregate freezer temperature. This should normally be pretty flat over time. I’ll start by removing the summarization to look at the data across all of the stores. There’s way too much here, so I can aggregate it in different ways. I’ll start with the average temperature, group it by timestamp, and also by department. Now, I can see the frozen dessert department is trending up over time. Now, we found something in the information that’s interesting. The temperatures in the frozen dessert freezers are moving up, which is not a good thing when you’re dealing with ice cream, so of course, I get notified of the issue from any of these tiles. I could set up alert if a threshold is exceeded, but that wouldn’t be super useful in our case, because we’ve alerted it to change the aggregate temperature across multiple freezers.
- That makes sense. So, could we zero it in then on the freezers that we care about, maybe the frozen dessert ones that have our ice cream?
- Yeah, absolutely. You can get pretty granular here. To go into particular data streams, the Real-Time hub is the best option. Here, I can find and use all the streaming data in Fabric. I can filter with these options on top and I can search through these streams. I’ll start typing “IoT” to pull up all of my IoT sensors. This top row represents our freezer sensors. In the details, I can see here what other items are using the stream, and then over here on the right, I get to preview the actual events coming in with the details about each event, and from here I can also set an alert. I’ll do that, and at this time, I’ll set my condition to be on event grouped by. In the grouping field, I’ll select FreezerID, in when, I’ll choose temperature, for the condition, I’ll select it becomes greater than, and for the value, I’ll set it to 29. So now that if any freezer goes above that threshold, I can get alerted in Teams, and I’ll use the same workspace as before and the same item name, TemperatureAlerts.
- So now, you can see all these alerts per freezer as they happen.
- Yeah, and the logic can get more sophisticated to look at business conditions and how they’re changing when the alert condition happens over a period of time, not just event by event. So, let’s look here. In fact, if I go into the reflex that was created, I can see the trigger and the individual streams that it’s monitoring. So, it looks like freezers D1 and D2 have gone over our threshold, so we’re seeing a few indicators of issues.
- So, that’s one sign, but you might also recall that we saw some orders and wait times that were trending up in the dashboard, so what can we tell from that?
- Yeah, that’s valuable data. That might give us an early warning from our mobile customer order app. So, if I go back into Real-Time hub and search for your orders data, I can’t find that information yet, so let’s bring that data in from our app’s database. I can add more data to integrate a complete view from Real-Time hub. I click Get Events. You can see all the data sources that you connect to, like Confluent, a few Azure services, Google Pub Sub, and Amazon Kinesis. In this list, there are a few marked CDC, which use the open-source Debezium framework that we host for you. Here, I’ll connect a Postgres database and listen for the changes. From there, you’ll add the connection details, the server, and then the database instance. And I need to specify other Eventstreams that are going to manage the connection stream, And finally, I’ll put in the name of the table that I want to monitor for changes. In this case, it’s delivery orders. Now, I just need to confirm it and that’s it.
- So, is that then going to streamify our CDC data feed in this case so that we can get real-time updates from our Postgres database?
- Exactly, in just a few steps. In the Eventstream, I can see and transform the events. This is a preview of the list of all the changes to the orders as they come in. If I go into Edit, I can do all sorts of transformations, like aggregate, expand, filter, group, or join the data to integrate multiple events to a cleaned up feed before I publish it back to Real-Time hub. I’ll choose Manage Fields, and now I can select the fields by reaching into that schema under the payload, and then order_id, customer_id, order_type, delivery_type, and waitTime. I’ll refresh, and now preview shows me just those fields. This looks pretty good. It’s the sort of output I need, so I can choose from the stream output to publish it back to the Real-Time hub for my orderWaitTime feed, and now it’s configured.
- Okay, so now the data from the mobile delivery app is flowing into the Real-Time hub.
- Yeah, it just takes a moment for those events to come in. Now, I can try the same search as before in Real-Time hub. I’ll search again for those orders, and then there’s the orderWaitTime feed I just created. When I open it, I get a preview of the number of events that have been generated and flowing through the stream. I can see connected items, the event stream, and things that are subscribing to the events, and I can create alerts directly from here as well. To save time, since I showed you this before, I’ll fast forward and head right over to Data Activator, because I already have my trigger ready for when the wait time crosses nine minutes, and it will send out an email.
- And there’s nothing better than getting an email or a Teams notification, except for maybe a mobile app notification, right?
- It’s funny that you mention that. This is right up your alley: workflow automation. So, in this case, I might change the promotion to something that reduces the number of orders but balances it out increasing the average amount of each order. You’ve probably seen examples, where instead of getting 25% off a single item, the promotion might be something like when you spend $100, you save $25. So, instead of an email, I can actually have it start a Power Automate workflow. In fact, in this tab, I’ve created one here. It listens to when the reflex trigger fires, then it creates an approval, and an Adaptive Card is posted in the Teams channel to start a new campaign. This should reduce the number of smaller customer orders and drive higher value orders so that we don’t tie up our drivers. Back in my reflex item, I just need to change the action that I want to have take over to this custom action for the campaign approval, and that’s all I need to do.
- And that’s going to then kick off a Power Automate flow effectively every time that trigger fires, so does this only work then with business-related events?
- It also works with system events too. Every Fabric item generates a system event, and activity in Azure storage does too. We can use these events just like business events. You’ll remember that our inventory system only updates hourly, and we can make that closer to real time. Starting from the pipeline, I’m going to create a new trigger, and then ask it to listen for a particular event in Real-Time hub. Our inventory stock system writes all the recent transactions into Azure storage account. I’ll connect to existing account and choose the correct subscription. Now I’ll choose my storage account, ContosoStockOutput. In Eventstream name, I’ll paste in ContosoStorageEvents. Then I can choose the event types. In my case, I only want the create events, so I’ll deselect everything else. Then I just need to create, and after that, hit Save, and like before, I need to fill in details for the workspace and the new item. I’ll name it EventBasedDataLoad this time, and that’s it. If I head back over to the monitor, you’ll see that the batch sales load succeeded, so now Real-Time Intelligence is listening for events where files are added to my Azure storage account, and will kick off the pipeline automatically. The analytics over the stock system are event driven, so you’ll see updates faster than the hourly poll we had previously.
- And I can see a lot of cases where this would be really useful to Real-Time Intelligence, whether that’s for recommendation engines or for things like generative AI. It could also be used to ground large language models with up-to-date information for a lot more accurate responses.
- Yeah, and of course, you can use system events to start other Fabric jobs. For example, you could run a notebook to train an LLM using real-time streaming data routed into OneLake via Real-Time Intelligence. So, basically any data activity in Fabric can now be event driven rather than scheduled.
- So, it’s really great to see all the updates for Microsoft Fabric, so where can all the folks watching right now go to learn more?
- Yeah, Real-Time Intelligence is in public preview today, and you can learn more at aka.ms/RealTimeIntelligence, and for all things Microsoft Fabric, check out microsoft.com/fabric.
- Thanks so much for joining us today, Courtney, and of course, keep watching Microsoft Mechanics for all the latest tech updates. Be sure to subscribe if you haven’t already, and as always, thank you for watching.
