I spent close to two decades building distributed systems and streaming platforms before co-founding Factor House in 2019, including several years advising banks on Kafka and Cassandra deployments, so I keep reading how other engineering teams run Apache Flink once it is carrying real production traffic rather than sitting on a slide. After Netflix’s Flink fleet, Airbus is a useful next stop precisely because it isn’t a consumer internet company: it’s an aerospace manufacturer whose AirSense unit built a real-time global aircraft-tracking service on top of Flink. I read seven sources for this one - Airbus’s own newsroom announcement, a Flink Forward Berlin 2019 conference talk by a named Airbus engineer, a joint success story published by the implementation partner that quotes Airbus’s tech lead directly, two independent pieces of German tech journalism reporting on the same talk, and Ververica’s own case study - all read directly rather than taken from a secondhand summary.
What makes AirSense worth writing up is the specific engineering problem underneath it: reconciling several independent, imperfect data feeds about the same physical aircraft into one trustworthy real-time picture, at a volume of more than two billion events a day. That’s a different shape of problem to a company deciding how to route microservice events internally. It’s closer to the kind of multi-source reconciliation work I’ve seen come up in financial-services data platforms, just applied to airspace instead of markets, and it’s a good illustration of why “just pick the biggest vendor” rarely survives contact with a real latency and correctness requirement.
Company overview
Airbus Defence and Space built AirSense as a real-time, global aircraft-tracking and analytics service for the wider aviation industry: air navigation service providers, airports, airlines, and government aviation agencies. Rather than relying on any single radar or ADS-B (Automatic Dependent Surveillance-Broadcast) data provider, AirSense ingests position reports from multiple independent sources and fuses them into a single, continuously updated view of where every tracked aircraft is, tuned to the roughly 10,000 to 15,000 flights in the air at any given moment and around 69 million flight hours a year across the market it monitors.
| Date | Milestone |
|---|---|
| 2018-07 | Airbus launches AirSense and signs a strategic ADS-B data partnership with Aireon |
| 2018 (mid) | AirSense’s real-time flight-data analysis pipeline goes into production |
| 2019-08-23 | Jesse Anderson (Big Data Institute) and Hassene Ben Salem (Airbus AirSense) present “Airbus makes more of the sky with Flink” at Flink Forward Berlin |
| 2019-08-23 | German tech press reports on the talk, quoting Heiko Udluft on Flink’s stateful-processing edge over Spark Streaming |
| Post-2019 | Airbus replaces its custom Java Flink job-management framework with Ververica Platform |
Airbus’s Apache Flink use cases
AirSense - real-time global aircraft tracking and event detection
AirSense’s core job is to answer, continuously and for any tracked aircraft anywhere in the world, where it is and what it’s doing right now. As Jesse Anderson and Hassene Ben Salem described at Flink Forward Berlin 2019, the service ingests ADS-B position reports - transmitted by aircraft at a minimum of twice a second - from multiple independent data providers, fuses them, and infers higher-level flight events such as takeoff and landing from the resulting stream, rather than depending on any provider to supply that context directly.
Fuel-burn analytics and downstream data quality
Beyond raw tracking, AirSense’s Flink pipeline feeds fuel-consumption analytics and validates data quality before it reaches external consumers. Per Big Data Institute’s success story, quoting Heiko Udluft (Tech Team Lead, Airbus AirSense), the pipeline’s cleaning stage filters and validates the fused position stream so that more than ten downstream consumers - airlines, airports, and other aviation stakeholders - receive a consistent, trustworthy data product rather than each having to reconcile provider noise themselves.
Scale and throughput
- 2+ billion events processed per day, independently corroborated across the Big Data Institute success story and informatik-aktuell.de’s report on the same Flink Forward talk.
- 2,000+ messages per second from primary data providers, rising to roughly 24,000 messages per second in aggregate once all providers and downstream fan-out are counted.
- 10,000 to 15,000 concurrent flights tracked at any given time, drawn from a market of around 69 million flight hours a year.
- ~100 GB of data volume processed per day.
- Avro binary serialization cuts payload size by 90-95% compared with the original per-provider JSON feeds, a compression step that matters directly for keeping that message rate affordable to move through the pipeline.
Airbus’s Apache Flink architecture
An Azure-native pipeline, end to end
AirSense runs on Microsoft Azure. Incoming ADS-B data lands in Azure Event Hubs, whose Kafka-compatible API lets the ingestion layer speak the same protocol Flink connectors already understand. Flink jobs process that stream on Azure Kubernetes Service, and the results are served from Azure Managed PostgreSQL with geospatial extensions, with raw and historical data archived to Azure Blob Storage. Azure Cosmos DB was evaluated as a proof of concept for part of the serving layer, per Big Data Institute.
A seven-stage pipeline from raw feed to served analytics
The Flink pipeline itself runs as a sequence of distinct stages: ingest the raw, inconsistent JSON coming from each data provider; transform it into a unified Avro binary schema and discard any record missing an ICAO aircraft identifier; normalise across the different schemas each provider uses; run a fusion algorithm that reconciles overlapping position reports and compensates for each provider’s own delay and positional uncertainty; clean and validate the fused result; tag inferred flight events such as takeoff and landing; and finally serve the resulting analytics to more than ten downstream consumers. Source: Big Data Institute.
A separate batch path for historical analysis
Not everything runs through Flink. Historical data captured in Blob Storage is analysed separately through managed Databricks, giving Airbus’s data scientists a Python-based, Spark-backed environment for retrospective analysis that doesn’t have to share infrastructure - or a real-time latency budget - with the live tracking pipeline.
Special techniques and engineering innovations
A fusion algorithm that trusts no single provider
Rather than treating any one ADS-B provider’s feed as authoritative, AirSense’s fusion logic reconciles overlapping and sometimes conflicting position reports for the same aircraft across providers, explicitly compensating for each provider’s own latency and positional uncertainty. That’s a materially harder problem than deduplicating identical events from a single source, and it’s the piece of this pipeline I’d flag first to any team dealing with more than one upstream feed for the same physical entity: build the reconciliation logic to distrust every source a little, not just the ones you already suspect.
Avro normalisation at the edge of the pipeline
Every provider’s raw JSON is converted into a single Avro binary schema early in the pipeline, before fusion or event tagging happens. Per Big Data Institute’s account, this cuts payload size by 90-95% relative to the original JSON, which simplifies every stage downstream and materially reduces what the pipeline has to move to sustain tens of thousands of messages per second in aggregate.
Inferring flight events statefully, rather than sourcing them
Takeoff, landing, and other flight events aren’t supplied directly by any ADS-B provider - they’re inferred from the pattern of position reports over time, which requires the pipeline to hold state per aircraft rather than treating each incoming report as independent.
Operating Apache Flink at scale
A 24/7 uptime bar, by aviation-industry expectation
AirSense runs continuously, with no tolerated downtime window, consistent with the service-quality expectations of the air navigation providers, airports, and airlines it serves. Source: Ververica’s case study.
Managed job operations instead of in-house tooling
After moving off its original custom-built job-management framework, Airbus operates its Flink jobs through Ververica Platform, using its service desk for day-to-day query resolution and its dashboard for visibility into every job’s current state, rather than maintaining that operational tooling itself.
Challenges and how they solved them
A custom Java framework couldn’t survive interruptions
Airbus’s original approach to running Flink in production was a custom-built Java solution for deploying and managing jobs.
Problem: the custom framework required manual handling of jobs and struggled to recover cleanly from unexpected interruptions.
Root cause: it had no automated recovery or job-state management layer suited to operating many concurrent, long-lived streaming jobs continuously.
Solution: Airbus adopted Ververica Platform to handle Flink job deployment, monitoring, and recovery, rather than continuing to extend the in-house framework.
Outcome: streamlined operations without an ongoing in-house maintenance burden, clearer visibility into every job’s state, and improved stability of the real-time streaming pipelines, per Ververica’s case study.
Choosing a processor that could actually hold state at low latency
Processing roughly two billion events a day for real-time tracking meant Airbus needed both low latency and the ability to hold per-aircraft state for fusion and event tagging - a combination that ruled out its first instinct toward a batch-oriented tool.
Problem: AirSense needed real-time latency and per-aircraft state, for a use case where micro-batch delay was not acceptable.
Root cause: Spark Streaming’s micro-batch model added latency the tracking use case couldn’t tolerate, and lacked the stateful processing primitives the fusion and event-tagging logic depended on.
Solution: the team evaluated Apache Flink, Spark Streaming (via Databricks), and Azure Durable Functions, and selected Flink for its stateful processing model and materially lower latency.
Outcome: a production pipeline sustaining 2,000+ messages per second from primary providers (24,000+ per second in aggregate) at real-time latency, presented by Jesse Anderson and Hassene Ben Salem at Flink Forward Berlin 2019 and independently reported by informatik-aktuell.de and infopoint-security.de.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Stream processing | Apache Flink | Core engine for multi-source fusion, event tagging, and low-latency stateful analytics on the aircraft-position stream |
| Messaging | Apache Kafka (via Azure Event Hubs’ Kafka-compatible API) | Ingestion backbone for incoming ADS-B position data from multiple providers |
| Serialization | Apache Avro | Binary schema normalising heterogeneous provider JSON feeds; cuts payload size 90-95% |
| Batch analytics | Apache Spark (via managed Databricks) | Historical, Python-based analysis for data scientists, kept separate from the real-time Flink path |
| Job management | Ververica Platform | Managed deployment, monitoring, and recovery for Flink jobs, replacing Airbus’s original custom Java framework |
| Container orchestration | Azure Kubernetes Service | Runs the Flink processing layer |
| Serving store | Azure Managed PostgreSQL (geospatial extensions) | Queryable store for processed aircraft position and analytics data |
| Archival storage | Azure Blob Storage | Raw and historical data store feeding the Databricks analytics path |
| Serving layer (proof of concept) | Azure Cosmos DB | Evaluated for part of the serving layer; not confirmed as production |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Hassene Ben Salem | Chief Engineer, Airbus AirSense (Airbus Defence and Space) | Co-presented “Airbus makes more of the sky with Flink” at Flink Forward Berlin 2019; one of the original members of Airbus’s advanced analytics team, later Product Owner and Lead Architect for AirSense |
| Heiko Udluft | Tech Team Lead, Airbus AirSense | Named, quoted source in Big Data Institute’s success story and in German tech press describing AirSense’s architecture and its Flink-over-Spark rationale |
| Jesse Anderson | Managing Director, Big Data Institute (implementation partner) | Co-presented the Flink Forward Berlin 2019 talk with Ben Salem; led the technology evaluation across Flink, Spark Streaming, and Azure Durable Functions |
Key takeaways for your own Apache Flink implementation
These are the patterns from Airbus’s own account that I’d point any team toward, particularly one dealing with more than one upstream source of truth for the same real-world entity:
- Evaluate stream processors against your actual latency and state requirements, not reputation. Airbus ran a direct comparison between Flink, Spark Streaming, and Azure Durable Functions before picking Flink, specifically because its stateful processing model and lower latency matched what real-time tracking needed - a decision made on measured behaviour rather than defaulting to whichever tool the team already knew.
- Build explicit fusion logic when you have more than one feed for the same entity. AirSense’s fusion algorithm treats every ADS-B provider as imperfect, compensating for each one’s own delay and positional uncertainty rather than picking a “primary” source and patching around its gaps.
- Normalise and compress early, not late. Converting every provider’s JSON into a single Avro schema right at ingestion cut payload size by 90-95% and simplified every stage that came after it - a cheap change to make once, and an expensive one to retrofit after a pipeline is already fanning out to a dozen consumers.
- Don’t let custom job-management tooling become your single point of operational risk. Airbus’s original custom Java framework for deploying Flink jobs couldn’t recover cleanly from interruptions; replacing it with a managed platform improved both stability and the team’s own visibility into job state, without the ongoing cost of maintaining that tooling in-house.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Airbus, “Airbus launches AirSense and signs strategic partnership with Aireon” - https://www.airbus.com/en/newsroom/press-releases/2018-07-airbus-launches-airsense-and-signs-strategic-partnership-with (2018)
- Jesse Anderson (Big Data Institute), Hassene Ben Salem (Airbus AirSense), “Airbus makes more of the sky with Flink,” Flink Forward Berlin - https://www.youtube.com/watch?v=sYlbD_OoHhs (2019)
- Flink Forward Berlin 2019 speaker listing (Hassene Ben Salem, Chief Engineer, Airbus AirSense) - https://www.flink-forward.org/berlin-2019/speakers (2019)
- Big Data Institute, “Airbus Success Story,” quoting Heiko Udluft (Airbus AirSense) - https://www.bigdatainstitute.io/success-stories/airbus-success-story/ (undated)
- Herbert Wieler, “Airbus setzt auf Flink zur Bewältigung der Datenflut in Echtzeit,” infopoint-security.de - https://www.infopoint-security.de/airbus-setzt-auf-flink-zur-bewaeltigung-der-datenflut-in-echtzeit/a20995/ (2019)
- “Anwendungen von Apache Flink und Ausblick in die Zukunft,” informatik-aktuell.de - https://www.informatik-aktuell.de/entwicklung/methoden/anwendungen-von-apache-flink-und-ausblick-in-die-zukunft.html (2019)
- Ververica, “Case study - Airbus” - https://www.ververica.com/case-study/airbus (undated)
I read every one of these directly rather than working from a summary, and I’ve linked the specific claim to its source throughout the piece above, not just in this list.
Enterprise tooling for Apache Flink
Airbus’s own story here is a job-management story as much as a stream-processing one: a custom-built framework that worked until it had to survive real interruptions at scale, replaced by a managed platform instead of another round of in-house patching. That’s exactly the operational gap my team and I built Flex to close - an enterprise UI and API for Apache Flink covering job visibility, deployment, and recovery, so you’re not the one maintaining that tooling yourself when it matters most.