How To Scale To “Trillion Events Per Minute” - With Ease And Without Downtime

Project Parsec is one of the ways we’re offering unprecedented speed and performance to applications built on the Macrometa GDN. It is an ongoing project still in development and available only in Macrometa.

What’s better than a map? A map that offers turn by turn navigation, and uses real-time traffic information to chart the best route to your destination. A map that’s aware that your favorite coffee shop is closed and routes you to the next nearest outlet. In short, a map that takes the guesswork out of your commute and routes you to whatever you need - quickly and simply.

What’s better than a router? A router that dispatches you to the nearest datacenter with the lowest latency- after confirming that it's active. If it’s not, it dispatches you onto the next nearest datacenter, so that - no matter what - you always get to your data, quickly and simply.

Meet our internal project, Parsec, the geo-aware and latency-aware router that helps you do more with data. Whether you want to super-size to a trillion events per minute scale, run applications at "blink of an eye" speed, or build new "never possible before" features, Parsec helps us help you get there.

What is Project Parsec?

The Macrometa Global Data Network is organized as a distributed mesh that lets you bind data to specific regions and use any combination of countries and clouds to store, process, and query data. Parsec functions as an intelligence layer that sits atop the Global Data Network spanning across the globe and helps your teams reach your data quickly with ultra-low latency. 

In today’s cloud model, the practice has been to move the data from the source to where the services are running i.e., in a hyperscale cloud data center that is thousands of miles away. Macrometa flips this model and moves the computing closer to the data source.

Modern applications need predictable low latency. So forcing requests to make a trip across the globe to reach data is a bad user experience: it will introduce huge latencies, low reliability, and higher computing costs. Not to mention how applications today have to deal with regulations on data like GDPR etc. It’s critical that appropriate PII data never leaves the applicable regions to avoid violation of regulations. Macrometa’s Project Parsec layer helps you do just that i.e., by automatically routing requests to the nearest location with the right availability and lowest latency.

How does Project Parsec work?

Parsec is a geo-distributed coordination-free routing & processing engine connected by Anycast to serve the requests originitation from anywhere in the world. When an App or API makes a request to the GDN, the request is routed, processed, and served by our Parsec engine that is closest to request source.

The Parsec engine applies pre-defined policies to quickly determine the right Macrometa GDN Pool to route the requests to. A pool (PoP) is a group of nodes within a data center that holds data in a region. By keeping copies of the data in different regions, we are able to reduce the access times since the data is closer to the people requesting it.

Parsec is used to find the optimal pool (based on the configured latency and geo policies) where a user's data is stored. Optimal - in the Parsec world - is defined as first, the user always has access to their data no matter what and second, they are able to access the data in the fastest possible way.

Parsec also provides intelligent request routing and tenant placement among geo-pools. It automatically manages data navigation, request routing, scaling, and load balancing - with the goal of making data always accessible at the lowest possible latency.

What’s the big deal with Parsec?

There’s quite a few things, actually.

Limitless scale

Need to handle a trillion events per minute? No problem, Parsec helps your data teams scale easily. Macrometa’s decentralized/distributed data, compute, and governance architecture makes it incredibly easy - also very quick and highly cost efficient.

Traffic isn’t directed to one central location, but geo-distributed. This means, each city’s traffic goes to that closest Parsec engine instance. So, people In India are not contending with traffic from the US.

As regional traffic grows, more pools are added or size of a pool is extended to handle this growth. If a region becomes over provisioned, Parsec can redirect traffic to other regions to help lighten the load. This response to changing demands is managed by Parsec with an eye to reduce latency while maintaining reasonable costs. As demand subsides, Parsec dismantles any additional resources it brought up to handle the increased load. In this way, Parsec helps to maintain the right balance between availability, performance, and cost.

Zero downtime

Parsec's number one priority is to always make your data available to you. While it strives to give the fastest possible access to data, it does not sacrifice access. If a pool (data center) isn’t available to deliver data, it redirects the request to other data centers. As resources are added (in step with increasing demand), no downtime is experienced. In other words, when pools or regions are added and removed, users' access to their data is never disrupted.

Effortless load balancing

Parsec offers a smart way to automatically balance workloads without any effort from your teams.

Parsec continuously measures the load and latency of all pools. These measurements are used to determine the most effective pool to service the data request. When it detects increases in either, it schedules more resources to be brought up to that region. This effectively means as the number of events increases we can direct more resources to that region with the intent of handling trillion of events per second.

Strong tenant isolation

All data is accessed via unique URLs. Parsec maps the unique URL to the correct pool in the correct region for each customer. If a customer grows and is moved to a new pool with more capacity, Parsec automatically knows how to route the traffic to the correct pool. And it does this transparently for customers. So no interruptions to customer. Internally this also allows us to isolate customers to their own pools thereby providing strong isolation between customers.

Parsec Key Benefits

• Makes data accessible faster
• Drives superior performance
• Assures strong tenant isolation
• Improves productivity
• Transparent to the user

Photo by Vincentiu Solomon on Unsplash