Dear Traders,
At Dhan, we love building technology & infrastructure from the grounds up - unlike most of the broking platforms and apps in the market, including the much larger ones who use off the shelf third-party systems and products.
We have discussed at length few of these capabilities on this forum, like our proprietary trading engine - DEXT (Dhan Exchange Trading System), and our APIs that ride on top of DEXT, news services like ScanX, and many more that we will speak of. Today we talk about one more such capability that Dhan has developed in-house - our ability to read and decipher exchange data feeds much faster than the industry benchmarks.
For a Trader, every tick is important! And this post will help you understand how Dhan delivers the same to you.
Every time you open Dhan, thousands of market updates are already moving through our underlying technology infrastructure to give you an experience that you see on your mobile app, web platforms, charts or consume via our APIs.
A stock price changes. A new bid enters the order book. An existing order is modified or cancelled. A futures contract trades. Liquidity shifts across multiple price levels.Within a very short time, these changes appear on your screen. For a trader or investor, this experience should feel effortless.
But behind every price movement or even behind every tick is a continuously running market data platform designed to receive exchange feeds, process every incoming packet, reconstruct market depth and distribute updates to users across mobile, web, desktop and APIs.
This is how Dhan processes exchange market data and delivers it to retail investors - faster and seamlessly, keeping its accuracy intact.
Market Data Is a Continuous Stream.
Most software applications follow a request and response model. A user sends a request. The application performs some processing, queries a database and sends a response.
Market data works differently!
Once trading begins, exchanges continuously broadcast updates across equities, derivatives, commodities and indices. There is no pause between two updates. The stream continues whether anyone is watching it or not.
Every trade generates new information. Every new order can change available liquidity. Every modification or cancellation can alter the order book. The engineering challenge is therefore not simply to store market data. It is to process and distribute a continuous stream of updates without becoming a bottleneck.
At Dhan, this led us to build a streaming-first platform around one principle:
Market data should keep moving.
Receiving market data through multiple paths.
Dhan receives exchange market data through multiple independent connectivity paths.
The first path consists of exchange leased lines terminating at multiple partner data centres where Dhan infrastructure is hosted. These leased lines provide reliable exchange connectivity across geographically and operationally separate facilities.
The second path comes from infrastructure deployed inside exchange co-location facilities.
By processing the exchange feed directly from co-location, we can begin working on incoming market data close to where it originates. This reduces the number of network hops before processing begins. Both the partner data-centre feeds and the co-location feed are then securely transported into Dhan’s infrastructure on AWS, where the primary processing and distribution pipeline operates.
During peak market activity, our platform ingests and processes nearly 35+ million market-data events per minute from the exchanges - over 550,000 events per second (approximately 550 events every millisecond).
This multi-path design provides two important advantages:
- Lower latency through direct co-location processing
- Greater resilience through multiple independent feed and connectivity paths
Speed is valuable, but speed without redundancy is not sufficient for a production-grade market data platform.
Dhan’s Market Data Architecture
The following simplified architecture illustrates how market data travels from the exchanges, through co-location and partner data centres, into Dhan’s core infrastructure and finally to clients.
This is a simplified view of Dhan’s real-time market data architecture. Feed ingestion, processing, streaming, distribution and backpressure-aware queues operate inside Dhan’s infrastructure. Client applications consume the resulting market stream outside the core infrastructure boundary.
The design separates the system into clearly defined responsibilities:
- Exchange connectivity and feed acquisition
- Feed decoding and order-book reconstruction
- Symbol partitioning and parallel processing
- Data-quality and sequence checks
- Canonical event-stream generation
- Live snapshot management
- Stateless real-time distribution
- Backpressure-aware client queues
- Asynchronous storage and analytics
The live path is optimized for continuous movement, while historical and analytical consumers operate asynchronously without delaying real-time distribution.
A Streaming Platform, Not a Traditional Backend
Market data is not processed like a conventional business transaction.
The exchange does not wait for us to ask whether a price has changed. It sends the next packet as soon as an event occurs. Our systems therefore operate around events rather than polling.
Instead of repeatedly asking: “Is there a new price?”
our platform continuously receives: “Here is the next exchange update.”
This distinction influences almost every architectural decision — from how packets are processed to how services scale and how slow client connections are handled.
Zero Database in the Hot Path
One of the most important decisions we made was keeping databases outside the critical live-distribution path.
Databases are essential for many purposes:
- Historical charts
- Research and analytics
- Compliance records
- Market replay
- Reporting
- Debugging and reconciliation
But a live market-data packet should not wait for a database write before it can reach the user.
The critical path is therefore intentionally short:
Exchange → Feed processing → Event stream → Distribution → Client
The live pipeline does not synchronously wait for:
- Database inserts
- Transaction commits
- Storage acknowledgements
- Replication
- Disk flushes
- Analytical workloads
Historical storage, replay and reporting systems consume the market stream asynchronously. This separation ensures that a slower storage or analytics workload cannot delay the live market feed.
The fastest database operation in a latency-sensitive hot path is the database operation that does not need to happen there.
Packet-by-Packet Processing
Markets do not move in batches, so our live feed is not designed to wait for batches either. Every packet received from the exchange is processed as it arrives.
A simplified packet journey looks like this:
Receive → Decode → Validate → Reconstruct → Normalize → Publish → Distribute
The packet is decoded according to the exchange protocol. Sequence information is checked. The relevant order book or market state is updated. The result is normalized into Dhan’s internal event format and immediately published to the downstream distribution pipeline.
We avoid holding packets merely to create a larger batch for convenience. Batching can improve throughput in some types of systems, but it can also introduce waiting time. For live market data, our priority is to keep each update moving.
Feed Decoding and Order-Book Reconstruction
Every exchange publishes data using its own defined protocols and message structures.
The feed-decoding layer converts these exchange-specific packets into structures that Dhan’s systems can process consistently.
This stage performs tasks such as:
- Decoding exchange messages
- Validating packet and message sequences
- Identifying instruments
- Processing trades, additions, modifications and cancellations
- Reconstructing market depth
- Detecting missing or inconsistent sequences
Full-depth market data is not simply a list of prices received periodically.
It is a continuously changing state reconstructed from a stream of order-book events. Each new event may add liquidity, remove liquidity, alter quantities or change the ordering of available prices.
Maintaining this state correctly at high packet rates is one of the most performance-sensitive parts of the system.
Creating a Canonical Event Stream
NSE, BSE and MCX do not necessarily represent every market event in exactly the same way. Allowing every downstream application to understand every exchange-specific format would introduce unnecessary duplication and complexity.
Instead, exchange-specific messages are converted into a canonical internal event format. This creates a common market-data language for the rest of Dhan’s platform. Once normalized, downstream services can consume events without needing to understand where each event originated or how the original exchange packet was encoded.
The canonical stream also preserves ordering where ordering matters, particularly at the instrument or symbol level.
This gives us a clean internal contract:
Exchange protocols may differ, but downstream systems consume one consistent Dhan market-data stream.
Symbol-Based Parallelism
A single processing loop should not become responsible for the entire market. Dhan distributes market-data processing across multiple processors operating in parallel.
The stream is partitioned into symbol or instrument groups. Each processing worker handles a portion of the overall market while preserving the required ordering within its assigned partition.
This allows many different instruments to be processed simultaneously.
Horizontal processing provides several advantages:
- Additional capacity can be introduced by adding processing nodes
- Workloads can be distributed across independent partitions
- A high-activity group does not need to block unrelated symbols
- Capacity can be planned around peak market conditions
- Individual components remain smaller and easier to operate
Instead of continuously building a larger single machine, the platform can scale by distributing work across more processors.
Risk, Integrity and Quality Checks
Fast data must also be correct data.
Before an update enters the common distribution stream, the processing layer performs integrity and quality checks.
These checks can include:
- Packet-sequence validation
- Gap detection
- Instrument mapping validation
- Invalid or malformed message detection
- Timestamp consistency
- Order-book integrity checks
- Data normalization checks
When a sequence gap is detected, the system must identify that the local state may no longer be complete and initiate the appropriate recovery or reconciliation process.
The objective is not merely to forward packets quickly. It is to deliver a reliable representation of the market.
Live Snapshot Cache
A user may connect to Dhan at any point during the trading session.
The user should not have to wait for the next trade or replay the entire day’s event stream before seeing the current state of an instrument. For this reason, the architecture maintains a lightweight live snapshot of the latest market state.
When a new client subscribes, the platform can first provide the latest available snapshot and then transition the client to the continuous live stream.
Conceptually, the flow is:
- Client subscribes to an instrument.
- The latest available state is sent.
- Live incremental updates continue from the event stream.
This provides a fast initial experience while avoiding the need to query a traditional database in the live subscription path.
The snapshot cache supports the stream; it does not replace it.
Stateless Real-Time Distribution
Once events have been decoded, reconstructed and normalized, they enter Dhan’s real-time distribution fabric. This layer is designed to remain stateless wherever practical.
No single distribution server should become the permanent owner of the market stream or a critical point of dependency.
A stateless distribution design gives us operational advantages such as:
- Horizontal scaling
- Faster service replacement
- Rolling deployments
- Rapid failover
- Better infrastructure utilization
- Easier recovery during failures
If additional distribution capacity is required, more nodes can be introduced without redesigning the processing pipeline.
Event-Driven Delivery
The distribution platform is event-driven. There is no need for clients or internal services to repeatedly poll for price changes. When the market changes, an event is published.
When an event is published, subscribed consumers receive it. This model reduces unnecessary requests and helps updates move through the platform quickly.
The event itself triggers delivery. That is fundamentally different from systems that refresh data every few seconds or repeatedly check whether something has changed.
Backpressure-Aware Client Queues
Not every client consumes data at the same speed.
One user may be connected through high-speed broadband. Another may be using an unstable mobile network. An API client may process messages slower than the rate at which the exchange produces them.
If slow consumers are not isolated, they can create head-of-line blocking and affect faster consumers.
Dhan’s distribution layer uses backpressure-aware client queues inside Dhan’s infrastructure to manage these differences.
The important principle is:
One slow consumer should not slow the entire market-data pipeline.
Client delivery paths are isolated through bounded queues. When a consumer cannot keep up, its condition is managed independently instead of allowing an unbounded backlog to affect shared resources.
Depending on the channel and type of update, the system can prioritize the latest relevant state rather than forcing a slow consumer to process a large backlog of outdated intermediate states.
This is especially important for market data. A user generally needs the most current order-book or price state—not thousands of stale updates delivered after the market has already moved.
Full Market Depth for Retail Investors
Historically, detailed market-depth infrastructure was associated primarily with institutions and professional trading firms.
The reason was not only access. Full-depth data is significantly more demanding to process and distribute than basic price snapshots.
It requires:
- High-throughput feed processing
- Accurate order-book reconstruction
- Large-scale state management
- Efficient symbol partitioning
- Low-latency fan-out
- Resilience during burst traffic
- Client-side rendering capable of handling frequent updates
By processing exchange feeds through co-location and building a streaming pipeline designed for scale, Dhan has been able to bring richer market-depth information to retail users.
We believe advanced market information should not remain limited to institutions simply because the underlying infrastructure is difficult to build.
The Processing and Distribution Backbone
After exchange data reaches Dhan through partner data centres and co-location connectivity, the core processing and client-distribution systems operate within Dhan’s infrastructure.
The core infrastructure includes:
- Feed ingestion
- Exchange protocol decoding
- Order-book reconstruction
- Symbol partitioning
- Quality and integrity checks
- Canonical event-stream processing
- Live snapshot management
- Stateless WebSocket distribution
- Backpressure-aware client queues
Only after the client queues does data leave the AWS-hosted infrastructure and travel over the internet to user devices and external API consumers.
This model allows Dhan to combine proximity to exchanges with the elasticity and operational capabilities of cloud infrastructure.
Exchange-side infrastructure helps us acquire data quickly. AWS helps us process and distribute it at retail scale.
Horizontal Scaling by Design
Market activity is highly variable.
A normal afternoon is very different from:
- Market opening
- Weekly or monthly expiry
- Budget announcements
- RBI policy decisions
- Election results
- Major IPO listings
- Unexpected global events
During such periods, packet rates and user activity can increase sharply within seconds. Our systems are designed to add processing and distribution capacity horizontally.
That means scaling is not limited to making one machine larger. Additional workers and distribution nodes can be introduced to share the load.
This improves both scalability and fault isolation.
Built for Burst Traffic
Average traffic is not the most meaningful test for a financial-market platform. A system may perform well for most of the day and still fail during the few minutes when users need it most.
We therefore design around bursts, not only averages.
Capacity planning and performance testing must consider sudden increases in:
- Exchange packet rates
- Order-book changes
- User subscriptions
- WebSocket messages
- API consumption
- Simultaneous client connections
The objective is for infrastructure to remain invisible even when the market becomes unusually active.
Built-In Redundancy
Market data is a critical service. No single connectivity path, server or processing node should become an avoidable single point of failure.
Redundancy exists across multiple parts of the architecture:
- Multiple partner data centres
- Exchange leased-line connectivity
- Exchange co-location processing
- Multiple processing workers
- Horizontally scalable distribution nodes
- Independent asynchronous consumers
- Recoverable snapshots and replay systems
The purpose of redundancy is not only to recover after a failure.
The stronger objective is to continue serving market data while the affected component is being recovered.
Storage and Analytics Outside the Hot Path
Live distribution and historical analysis have different requirements. Live distribution prioritizes low latency, predictable processing and continuous movement.
Historical systems prioritize durability, query flexibility and long-term retention. Trying to solve both problems within the same synchronous path creates unnecessary coupling.
Dhan therefore keeps historical storage, reporting, analytics and replay consumers asynchronous.
They receive the market stream, but they do not block it.
This allows the live platform to remain independent of:
- Reporting workloads
- Long-running analytical queries
- Historical database maintenance
- Data exports
- Research jobs
- Replay and debugging tasks
Different systems perform different jobs, while consuming the same underlying stream.
Designed for Failure
Distributed systems will eventually encounter failures. Networks can degrade. Machines can restart. Processes can crash. Client connections can become slow. Downstream consumers may temporarily become unavailable.
The architecture must assume these events will happen.
This is why services are designed to be replaceable, processing is horizontally distributed, client queues are bounded and critical paths have multiple sources and recovery mechanisms.
A reliable system is not one that assumes nothing will fail. It is one that continues operating when individual components do.
The Engineering Principles Behind the Platform
A few principles guide the way we build Dhan’s market-data infrastructure:
- Keep the hot path short.
- Process packets as they arrive.
- Do not wait for databases during live distribution.
- Normalize exchange-specific events once.
- Preserve ordering where it matters.
- Scale horizontally before creating oversized components.
- Keep the distribution layer stateless.
- Prevent slow consumers from affecting fast consumers.
- Keep storage and analytics asynchronous.
- Build for burst traffic, not average traffic.
- Introduce redundancy across every critical layer.
- Start processing as close to the exchange as practical.
These principles are simple to state, but applying them consistently across a large real-time platform requires significant engineering discipline.
Making Infrastructure Invisible
The best market-data infrastructure is infrastructure the user never has to think about.
Users should not need to understand packet sequences, symbol partitions, streaming pipelines or backpressure queues.
Traders should simply open Dhan and see the market.
That experience is powered by a platform combining exchange leased lines, partner data centres, co-location processing, real-time feed processing, zero-database hot paths, packet-by-packet streaming, full-depth reconstruction, horizontal scalability and backpressure-aware distribution.
While we do not control how the market moves, we can control how efficiently market information moves through our systems.
And that is what we continue to engineer for every day.
Kind Regards,
Alok