Securing streaming data access in all market conditions

As data volumes proliferate, companies seek new ways to consume information reliably

What is conflation?

With a backdrop of sustained volatility, an explosion of streaming market data message rates means many firms are challenged to find new ways to efficiently process information. Here, conflation can play a critical role.

Conflation is generally defined by the combining of data in multiple messages to form one single message. The intention is to control and throttle the distribution of data. The parameters for delivery can be controlled by vendors or by subscribers. These parameters range from intervals of time to different types/styles of conflation.

The benefits

Conflation offers several benefits. The sheer volume of streaming data available to many firms is so large that it’s not necessarily physically possible - or desirable - to consume it all. Consider the typical desktop use case: the desktop user can only possibly observe data flashing by at 100-150 milliseconds per interval. Anything faster than this would be invisible to the human eye and would suggest firms are using unnecessary processing power and paying for unused data with no benefit at all to the user.

Because conflation works to reduce the volume of data consumed, it can help cut bandwidth requirements too and as a result, companies can optimize their infrastructure and related costs. For illustrative purposes, below is a generalized approximation of the data consumed based on the conflation type in a random, 30 minute window, on June 9th 2021, sometime between 10:30am-11:30am est.

Conflation isn’t a binary concept though; and it’s important to recognize the need for customization. How to customize will depend on the real requirements of the consuming user or application, in combination with considering physical limitations the user or application is constrained by, both in terms of bandwidth availability (static or variable) across the network stack, CPU power of the receiving application, and other technical factors. The objective is to tailor and optimize the user or application experience by carefully considering what data points to receive, and at what intervals.

A tailored experience

Recognizing the increasing customer reliance on sophisticated conflation techniques, any ICE customer subscribing to the Consolidated Feed - which offers low-latency, real-time or delayed streaming data across 600+ markets - has been given access to a range of conflation options as part of their subscription. ICE has opted to offer a fully customizable conflation experience, at the client application level, by conflation type (trade-safe or intervalized), by rate (time interval for conflation to occur, between a range of 1ms to 1 hour) and instrument type (such as a single stock). For example, a client might choose to conflate all updates on a specific big-cap stock listed on the NYSE but not to conflate updates for a given small-cap name. Alternately, excessive data rates from a particular source may mean a client may choose to conflate a few instruments to manage data volumes by decreasing data volumes at market open in North America, while opting for the full volume of data at market open in Europe. With the ICE Consolidated Feed, clients have the ability to increase or decrease the level of customization of their conflation preferences at any time.

The Consolidated Feed offers two standard conflation types: Intervalized and Trade-Safe. Intervalized conflation means a conflation interval is pre-determined (such as one second) with quote and trade information to be updated with the latest changes at that interval. Trade-Safe conflation means that only quote related information is conflated for a pre-determined interval, to ensure clients receive conflated quote data but are not likely to miss a trade.

Streaming data access in times of stress

For those customers who don’t want “regular” conflation options (e.g. receive intervalized or trade-safe updates only at a predetermined time intervals, configured by the user), the ICE Consolidated Feed also offers a more advanced, dynamic, conflation capability, which, once enabled, only applies Trade Safe Conflation during market dislocation or when data spikes mean the receiving application can’t keep up with the increasing message rates (also referred to as a slow consumer). This mode of conflation, “just-in-time (JIT)” conflation, can protect a client’s connection to the market when their bandwidth or application CPU would otherwise be unable to keep up. JIT conflation is applied when backpressure is detected, and a client is then notified that conflation is in effect for all instruments. The JIT conflation applies a trade-safe conflation mode where the rate is continuously and dynamically adjusted to manage back pressure. As soon as the edge device determines that the dislocation event has passed, the application is notified that conflation has been removed and reverts to receiving tick-by-tick data. In this way, in times of market upheaval or unexpected stress on a client’s network infrastructure or application JIT conflation can help support continuity of information flow.

For major banks, asset managers, hedge funds and redistributors, secure access to a data feed through all market conditions is often critical to their operations. Random, unpredictable data loss can be disruptive - and often costly - and applying conflation techniques depending on the firm’s specific circumstances and needs can help mitigate these risks. Applying the right conflation mode may also help to better optimize the required network infrastructure and bandwidth to receive critical information as it allows the customer to cut out unwanted information or avoid having to customize for extreme market conditions. The tailored conflation approach made available in connection with the ICE Consolidated Feed can help support these needs.