Momentum One Academy · article
Real-Time Intelligence and KQL in Microsoft Fabric
A production system does not become real time simply because a dashboard refreshes often. The useful question is: how quickly can the team detect a meaningful change, query the evidence, and take a safe action? For Momentum One Academy, that is the practical purpose of Module 9. We use Microsoft Fabric Real-Time Intelligence to move from “the batch report will tell us tomorrow” to a controlled operating loop for data in motion.
The scenario can be a production platform receiving application telemetry, device readings, or operational events. Some questions need an answer within seconds or minutes: Is a service error rate rising? Has a device stopped reporting? Did a pipeline fail before its downstream report was refreshed? Other work is not truly real time: a nightly backfill, a dimensional model rebuild, or a large historical correction. The design challenge is to put each workload on the right path instead of forcing every problem into streaming.
Start with the problem, not the product list
Before creating an Eventhouse, write down the service objective. Define the event timestamp, the time the platform receives the event, the acceptable freshness, the action that follows a breach, and the period for which the team needs to investigate. “Near real time” is not a universal number. It is a contract between producers, the data platform, and the people responsible for the outcome.
This distinction also prevents a common production mistake: treating a fast query as proof that the pipeline is healthy. A KQL query can return quickly over an empty or stale table. A useful design measures both data freshness and data completeness. Keep an explicit event identifier where possible, retain source and ingestion timestamps, and decide how late or repeated events will be handled before the first incident.
Fabric’s Real-Time Intelligence experience is broader than a query editor. Microsoft describes it as an end-to-end set of capabilities for ingesting, transforming, storing, modeling, analyzing, visualizing, tracking, and acting on data in motion. Real-Time hub is the discovery and catalog surface; Eventstreams connect and process flows; Eventhouse and KQL databases store and analyze events; dashboards and alerts make results usable. 1
The Fabric mental model
Think of the architecture as a set of responsibilities rather than as a single “real-time database.”
Real-Time hub is where teams discover and manage streams and event-driven sources. It can expose active Fabric streams and connectors for sources such as Event Hubs, IoT Hub, database change feeds, and workspace events. That makes it a useful starting point for a production inventory, but it is not a substitute for an agreed data contract. 1
Eventstream is the low-code movement and processing layer. It brings in events, applies transformations such as filtering, field management, aggregation, grouping, unions, expansion, or joins, and routes the result to destinations. The official overview distinguishes the existing Eventstream experience from a schema-aware preview experience, so a production workshop should record which experience is being used rather than assuming every tenant has the same behavior. 3
Eventhouse is the real-time analytics environment. An Eventhouse can contain multiple KQL databases, and a KQL database contains tables. Eventhouse is intended for time-based streaming events and structured, semistructured, or unstructured data. Fabric describes the stored data as organized for fast searching by arrival time, which suits telemetry, logs, time series, and other event-shaped workloads. 2
KQL is the read-only query language used to explore and analyze the tables. A query flows through operators separated by pipes. That makes the sequence visible: restrict the time range, select useful columns, parse or enrich, then aggregate. KQL is particularly suitable for telemetry, metrics, logs, time-series operations, text parsing, and aggregation. A management command is a different category from a query, so do not treat a read-only investigation as a schema change. 5
Data Factory pipelines handle orchestration. A pipeline is a logical group of activities that can move, transform, and control work. Fabric supports on-demand, scheduled, and event-based pipeline runs. In the Momentum One operating model, a pipeline is usually the right tool for repeatable batch work, backfills, dependencies, notifications, and controlled recovery—not for pretending that a scheduled loop is an event-processing engine. 7
Monitor hub is an operational surface for job health across Fabric items. Its Job runs page centralizes run status, duration, history, and details, including pipeline runs. As of this review, the new Monitor hub and Job runs page are documented as preview experiences, and the page does not update continuously; refresh it when checking an active or recent run. 8
Architecture choices for a Momentum One production build
Pattern 1: Eventstream to Eventhouse for live operations
Use this as the default workshop pattern when the source is a continuous event flow and the first consumers need fast investigation. The path is:
source -> Eventstream -> Eventhouse/KQL database -> KQL queryset or dashboard
Choose direct ingestion when the Eventhouse should receive the stream with no Eventstream processing. Choose event processing before ingestion when filtering, aggregation, or other operators should run before data lands in the Eventhouse. Fabric documents both ingestion modes for an Eventhouse destination. 4
For production, keep the raw or minimally transformed path available when replay and forensic analysis matter. Put business-specific rollups in a separate table or derived stream when that makes ownership clearer. The trade-off is straightforward: preprocessing can reduce noise and query work, but it can also remove evidence that an incident responder later needs. Do not discard fields merely because the first dashboard does not use them.
Pattern 2: Eventstream plus Lakehouse for dual consumption
Route a stream to an Eventhouse for operational queries and to a Lakehouse when the organization also needs Delta tables, batch analytics, or warehouse-oriented processing. Eventstream documentation describes the Lakehouse destination as converting real-time events to Delta format before storing them in Lakehouse tables. 3
This pattern adds a second destination to govern, test, and monitor. It is worthwhile when the same events serve two different latency contracts. It is wasteful when the second copy exists only because the team has not decided which system owns the data. Establish ownership, retention, schema, and replay rules for each destination.
Pattern 3: Pipeline-led batch, backfill, or recovery
Use a Data Factory pipeline when the work has explicit activities and dependencies: load a historical partition, validate a source extract, run a notebook, refresh a downstream item, or notify an owner. A pipeline can also be triggered by an event, but the activity graph is still orchestration. It should not be used as a substitute for the continuous ingestion path when the business needs an answer as events arrive.
A practical hybrid is often best: Eventstream and Eventhouse support the live view, while a pipeline performs hourly or daily reconciliation and backfill. The reconciliation should compare counts, time windows, and known event identifiers rather than silently overwriting the live table.
Implementation guidance: build the smallest trustworthy path
Create the contract. Name the event time and ingestion time columns, their data types, the producer, the expected grain, and the retention need. Make timestamps UTC-compatible and preserve the source identifier. If the payload is JSON or another semistructured format, identify the fields that must be queryable as typed columns.
Provision the container and database. In an enabled Fabric workspace, create an Eventhouse and then a KQL database in its context. Microsoft’s creation guidance states that one workspace can hold multiple Eventhouses, an Eventhouse can hold multiple KQL databases, and each database can hold multiple tables. Creating a KQL database also creates an attached KQL queryset environment for exploration and management. 2 3
Connect the source deliberately. In the Eventstream canvas, configure the source, transformations, and destination. Preview a representative window before publishing. Check that field names, types, null behavior, and time semantics are what the destination expects. Do not assume that a successful connection means the data is useful.
Choose the ingestion mode and table shape. Use direct ingestion for an as-is stream. Use processing before ingestion for an intentional filter, windowed aggregation, or normalization. For schema-enabled sources, the Eventhouse destination supports choosing a combined table or separate tables for schemas and selecting payload-only or metadata-plus-payload output. 4
Publish in an observable order. Eventstream documentation warns that ingestion and routing can start at different times after publishing. In a CDC scenario, initial data may be ingested into the Eventstream before routing is ready, so some data may not reach the destination. For a controlled cutover, configure the destination without activating ingestion, publish, then activate ingestion from an appropriate earlier time when the documented options support that recovery. 4
Prove the query. KQL queries should be bounded by time and should return only the columns and rows needed. For a telemetry table with a ReceivedAt column typed as datetime, DeviceId, and numeric TemperatureC, this is a small, production-friendly starting point:
Telemetry
| where ReceivedAt > ago(15m)
| summarize
Events = count(),
Devices = dcount(DeviceId),
AvgTemperatureC = avg(TemperatureC)
by bin(ReceivedAt, 1m)
| sort by ReceivedAt asc
The explicit bin() creates one-minute groups. The early time filter limits the scan. Microsoft’s KQL guidance recommends reducing the amount of data processed, applying where close to the table reference, using datetime for time columns, and using explicit binning for time ranges. 6 Treat the table and column names in this example as a contract you must adapt, not as built-in Fabric names.
- Expose an operational view. A dashboard is useful only when it shows freshness, volume, errors, and the business signal together. Include a “last event received” value and a count over the expected window. If a threshold should trigger an action, define the owner and the safe response before enabling automation.
Trade-offs and failure modes
The first trade-off is latency versus evidence. Filtering and aggregating in Eventstream can make a dashboard cheaper and simpler to query, while raw ingestion preserves more options for later analysis. The second is simplicity versus reuse. One broad Eventhouse may reduce duplicated ingestion, but separate KQL databases can give teams clearer ownership and permissions. The third is elasticity versus readiness. Eventhouse is designed to suspend when not in use; reactivation can add a few seconds, so a workload with strict startup sensitivity should assess the documented capacity-planning options rather than assume a permanently warm service. 2
When data is missing, trace the path in order: source connector, Eventstream incoming events, transformations, destination output, and Eventhouse ingestion. Eventstream monitoring exposes incoming and outgoing events, bytes, backlog, runtime errors, data-conversion errors, deserialization errors, and watermark delay for supported nodes. Its Runtime logs view provides warning, error, and information entries. 10
When data is present but the result is wrong, inspect schema mapping and time columns before rewriting the KQL. A string timestamp, local-time assumption, unexpected null, or schema drift can make a correct query appear broken. Then check for late, repeated, or out-of-order events. If the business requires exactly-once outcomes, that requirement must be designed and tested across the source and destination; a fast query alone does not provide it.
When a query becomes slow, first reduce the time range and projected columns. Avoid wildcard table scans, use selective predicates early, and consider materialized views for repeated aggregations where the workload justifies them. 6 Monitor Eventhouse system overview for ingestion rate, activity, queried databases, query errors, and cache misses. 9
When a service call or ingestion request fails intermittently, implement retry behavior in the calling application or integration. Microsoft’s reliability guidance calls out transient failures, connectivity interruptions, and throttling as conditions applications should be prepared to retry. It also states that Eventhouse does not provide native backup and restore or point-in-time restore for KQL databases. For recovery planning, use supported export or upstream rehydration patterns and test the replay runbook. 11
Production checklist
- Is the freshness target stated in minutes or seconds, with an owner and an escalation path?
- Are event time, ingestion time, source identifier, and schema documented?
- Is the Eventhouse/KQL database boundary aligned to ownership and access needs?
- Is Eventstream used for continuous movement and shaping, while pipelines handle orchestration and backfill?
- Has the team chosen direct ingestion or processing before ingestion for a stated reason?
- Has a representative event been traced from source to destination and queried in KQL?
- Does the live view show both business metrics and data-health metrics?
- Are backlog, watermark delay, conversion errors, deserialization errors, and runtime logs reviewed during a test run?
- Are pipeline runs monitored in the Monitor hub and, where deeper analysis is needed, in workspace-monitoring KQL logs?
- Are retry, replay, late-event, duplicate-event, capacity, and regional-recovery procedures documented and exercised?
- Has every preview feature been labeled as preview in the solution record and accepted by the service owner?
Workspace monitoring is useful for a deeper operational slice: when enabled, Fabric creates a monitoring Eventhouse and read-only KQL database. Pipeline-level and activity-level logs can then be queried for status, timestamps, duration, and failure type. Microsoft recommends combining this detailed analysis with the Monitor hub’s quick operational checks; the workspace-monitoring page also documents limitations, including unsupported error details and diagnostics. 12
The Module 9 workshop connection
In the Momentum One production workshop, teams should leave with more than a working chart. Start with a small event contract and a measurable freshness target. Build one Eventstream-to-Eventhouse path, inspect its data insights and runtime logs, and write the bounded KQL query that proves the data is arriving. Then add one pipeline for reconciliation or backfill and use the Monitor hub to compare live data health with job health.
The final exercise is an incident rehearsal: stop or delay the source, observe the backlog or stale-data symptom, identify the failing layer, and execute the agreed recovery step. That practice turns Real-Time Intelligence from a feature tour into an operating discipline. It also makes the boundary clear: KQL helps the team understand what is happening, Eventhouse makes event-shaped data queryable, Eventstreams move and shape the flow, and pipelines make repeatable operational work explicit.
Ready to turn the design into a tested Fabric operating pattern? Continue with the Momentum One Academy Fabric Production Workshop.
References
Plan your workshop