Explore insights on real-time edge computing, AI, and cloud solutions. Learn how our technology optimizes performance across industries.

Blog

, a writer and artist from Cleveland wrote something that I've pondered often in the last few months. She asks, "Do you think the universe fights for souls to be together? Some things are too strange and strong to be coincidences."

In the world of business, discussing souls, togetherness, and coincidences is taboo. Yet as I sit here writing these words - I'm struck by the profound serendipity of just having raised a 

 and all the wonderful lessons, people, and experiences that have happened in the short time since closing the round.

 by our first investor, board member, my extraordinary and dear friend, 

. Yash led our pre-seed financing in 2017 and took a risk on a crazy napkin sketch of a database that would run in 100s of locations worldwide. I've been pitching products, companies, and ideas for almost two decades now and know the fundraising game at some depth. I've met hundreds of funds, principals, and partners over the last 15 years. Yet here I was, meeting Q over zoom and unable to put my finger on the incredibly rich presence, thoughtfulness, and authenticity he possessed. A few days later, Q invited me to join him on a hike (socially distanced) to talk more. What unfolded on that hot June day by the San Francisco bay's foaming sands in Foster city was incredibly special. As Q and I spoke - it became very evident to me that he and DNX are special. The valley and Sand Hill Road are full of brilliant, accomplished, and ambitious people. But Q was more than brilliant, accomplished, and ambitious - there was a "Je ne sais quoi" about him - I still don't know what it is, but I've put it down to a powerful combination of authenticity, a genuine curiosity in people and a very refreshing self-effacing honesty and humility that frankly far too few people on Sandhill Road possess.

Today, I'm incredibly pleased and excited to announce our oversubscribed $7M seed financing lead by DNX. All our incredible investors from the last pre-seed round participated - my heartfelt gratitude to 

 (Nitin) for underwriting the crazy napkin sketch in 2017.

 are joining our board of directors as independents. Rob was one of the architects of Netflix as it transformed from a DVD by mail company to the global streaming giant and viscerally faced all the problems we strive to solve at Macrometa while building Netflix. Jon was the CEO of Auth0 and Xnor.ai (acquired by Apple recently), and he invented and wrote the book on building products and companies that primarily sell to developers and architects. At Macrometa, we do just two things: (1) We build great products, and (2) We sell great products. We are incredibly fortunate to get Rob and Jon's help to build and sell products we hope will enable developers to do things locked in their imaginations and held back by the complexities and arcane architectures of the cloud.

Finally, I'm super excited to say Welcome to 

, who joins us to lead our marketing, developer experience, and relations. Justin is an inspiration and a hero to me - I've followed his work at Keen.io and then at StackPath, where he helped build and market exceptional products and then formed vibrant developer communities around them.

Durga and I would be remiss if we didn't thank our team members for their hard work from the first day we started in 2017. To the Macrometa family spread across the US, Europe, and India - congrats to all of us - the edge is our oyster.

 in engineering, marketing, sales, and customer success - we don't care where you live or work from - we have always been, and will always be about enabling you to do your best work from where ever you are happiest doing it).

Announcing our $7M Seed raise lead by DNX

", Josh Breilinger, a VC at Jackson Square ventures, makes a powerful argument that one of the most effective forms of competition comes in the form of newcos who aspire to take chunks out of an existing platform by better addressing the needs of a specific vertical within that platform

In the past, I've viewed edge computing as a new kind of tier to cloud computing and one that potentially eclipses the centralized clouds - because it's a natural place to run anything that is interactive, needs to analyze large volumes of shared state data and quickly compute against short time value data. Seen from Josh's perspective it almost seems inevitable to me that edge computing will eclipse cloud computing and that new companies will not just challenge but dethrone the big cloud incumbents.

The incumbent cloud vendors will always see edge computing as a specialized sub market to cater to - with extensions to their current technology stack. And there lies the fallacy, that the current centralized stack can be extended to work with edge computing. Edge computing demands its own architecture - one that eschews centralization for distribution, with the goal of creating revenue instead of saving money.

Event driven programming is wasted on centralized clouds. It is at the edge that event sourcing and reactive technologies really open up trillions of dollars of possibilities in real time, low latency edge computing - in robotics, data analytics, gaming, smart things and machine to machine interactions. When events can be detected, processed and reacted to in single digit milliseconds, the cloud of today will seem quaint and fill us with nostalgia for a simpler time much as a Casio LCD calculator does when compared to an iPhone. 

Just as AirBnB took a submarket of Craigslist and turned it into a standalone multi-billion dollar a year business, Macrometa and other edge computing startups will take today's submarket of edge and turn it into its own hundreds of billions of dollars a year of business in the future.

The "Edge" Is A New Kind Of Vertical Cloud 

Ages ago I had the privilege of building out the first field engineering team for VMware’s Cloud Service Providers. Back then it was an easy message for users to understand – we could take your existing workloads and give you improved resilience and operational costs, while providing the flexibility to deliver enterprise applications in a familiar way.

Since then, a few things have changed in our industry.

User communities have grown, mobile and distributed applications have become a real value to Enterprise. The current iteration of that transition is edge computing, which not only distributes the app closer to user populations enabling things like data privacy, but also drastically improves performance. The days of centralization for resilience are over.

What about operational costs? When the options were managing it yourself, sourcing talent, and buying infrastructure, 

 made a ton of sense. We’re past that now. Distributing applications to the network edge doesn’t change that dynamic, but it does create operational efficiencies for your users which ultimately increases productivity and either revenue or the marketability of the application.

This is the basic thesis for Content Delivery Networks (CDNs). You take the data that can be placed closer to the user and just do it. No more long transit times and transit costs to fetch an image for a web page or a key from a Key/Value store. Put it closer to the user and enjoy the benefits of increased performance and efficiency. We needed a similar paradigm for dynamic data, and that is what Macrometa is all about.

The cloud model holds true for Macrometa. We can now provide even better resilience, reduced operational costs, and still deliver applications in a familiar way – while allowing developers to build apps in a familiar way. From day one Macrometa has been committed to supporting the stacks and frameworks that developers are comfortable with. To put it simply, build your app in the same way but deliver it faster and with more end user performance by leveraging a ready-made backend that is edge-native.

The cloud will continue to be relevant for a long time due to its ability to quickly stand up infrastructure and scale it. But the new reality is that the way we access data is becoming just as important as the way we host it, and for some applications like CRM and mobile apps it is even more important to reduce the costs and delays that backhauling to a cloud datacenter creates. Basically, we're reclaiming the value of resilience and operational costs that cloud gave us 10 years ago and applying it to the brave new world of edge applications. 

Are you ready to take the next step and join our community of hundreds of developers building the next generation of apps? Click on the link below to create a free developer account on Macrometa’s Global Data Network.

The Cloud Is Now The Limitation

What if every food option you ever wanted was just a few miles away?

Spending my career in Dallas, TX while working for companies in California has been fun and offered me the experience of traveling all over the world. Business travel is anything but glamorous – if you’ve done much traveling you’ve probably shared my experience of watching the last flight out of a random city be cancelled due to weather somewhere else on the continent while you’re desperately trying to get home before a holiday weekend.

Having said that, there are some perks. One of these was In-N-Out Burger. For decades this franchise was a California phenomenon. It was another place like Shake Shack, NY Pizza and White Castle. Delicious and totally unavailable in Dallas. I enjoyed the opportunity occasionally to grab a great In-N-Out Burger in California, but that highly centralized model meant if I was sitting at my home in Texas and wanted one I was completely out of luck. It was the opposite of fast food for me, and required a lot of planning and waiting to make it happen.

On a brighter note, over the last few years they’ve been popping up in Texas. Now there are 3 convenient locations near me, so I can grab one any time I feel like it (or more appropriately, my kids can have one now instead of listening to me tell them it is really good… if they only had the time and opportunity to make the trip out West.)

Hyper-locality of resources is a huge consumer benefit, and a way to service and acquire customers in a much shorter amount of time. When it comes to applications and backend services, there really isn’t much difference between my experience with In-N-Out Burger and my experience with companies like AWS and Google. It’s a great service, but requires me to cross huge distances to access what I need. To put it another way, when I’m hungry in Dallas the In-N-Out in Santa Clara doesn’t do me any good, and even if I decided to go there I’m going to be starving by the time my flight lands. If I need data in 10-25 ms, then traversing a thousand miles also doesn’t make sense.

When my apps are hungry for data and performance, a datacenter a thousand miles away isn’t satisfying. What if instead of using 5 or 6 fixed datacenters in distant locations (that I didn’t even get to choose) I could use 5 or 6 datacenters right here in Texas? Now my app is blazingly fast.

That’s the value of Macrometa. Our globally distributed edge network provides 5 or 6 datacenters 

 instead of forcing me to perform a costly data backhaul and wait for functions to cold start somewhere in a faraway cloud. The end result is a massive improvement in application latency and user experience.

Macrometa has been proven to eliminate over 240-425ms of backend round trip latency for apps that need dynamic content at the edge. For developers who know where their user populations live across the globe, they also have the freedom of selecting 

 datacenters serve their app. Why should I shoehorn my app into a geographically limited design, when I can run it 

? This is hyper-locality, and it is the foundation of a concept Macrometa has pioneered called the Global Data Network (GDN). Like a CDN for static data and global footprint, the GDN is purpose built for dynamic data 

 global distribution - unlike existing cloud datacenters which are built for local scale and centralization.

The GDN provides multi-model database capabilities including Doc, Graph and Time Series, along with global pub/sub, Streams, and rich APIs. End users never have to send data requests a thousand miles away and wait, they can make local requests and get local performance from the GDN.

Want to see what Macrometa’s GDN can do for your application? Sign up for a free developer account by clicking the button below.

The Merits Of Application Hyper-Locality

A friend of mine was a construction contractor and his customer mantra was simple:

“I can build it good, fast, and cheap. Pick 2.”

You’ve probably heard that before, and of course it applies whether you’re building a house or a cloud native application. The hard part is picking which two criteria are most important for you, and knowing which criteria is expendable. For my contractor friend, if the customer picked fast and cheap his mantra to the construction crew became “it doesn’t look perfect, but it will look fine from the next job site.”

For developers, there is no shortage of tools and SaaS/PaaS products to help them build. I started thinking about how difficult it must be to sort through the available options and select a product that you will essentially be married to for months or years, and the difficulty of balancing quality, performance, and cost.

To get some perspective, I sat down with our Chief Architect to pick his brain about the vendor selection process. In the enterprise space, many product decisions regarding commodity technology are about “good and cheap” – or at least good enough. To put it another way it is about comparing value to price and finding an acceptable compromise. In the developer space, there are other nuances.

One of the big takeaways from the conversation is that cost is measured in different terms by developer organizations. We typically think of cost in terms of licensing, supporting infrastructure, or monthly SaaS charges. As Durga pointed out, it isn’t quite so straightforward for a dev shop.

“Obviously the first selection criteria is ‘can it handle my workload.’ Sometimes the workload itself narrows the field down to just one or two products that can accomplish the task. An example requirement we’re familiar with would be a database that is consistent across a dozen global regions. A hosted NoSQL or cloud platform that can only replicate across 6 locations may be more common, but if I truly needed more locations or a global footprint, then Macrometa would be the right platform.”

This is where the conversation became even more interesting. I asked if there were multiple products that fit the workload needs, how would he narrow down the options and make a final selection?

“The most important thing is the cost – but not in terms of the actual product pricing. Unless the cost is vastly different, the monthly or one-time expense isn’t enough to influence a decision. I evaluate products based upon my most important costs – Time.”

This made a lot of sense to me. If Product A is $500 and Product B is $700, but Product B will let me produce my product (an app) 50% faster, then the additional time to market savings greatly offset the additional expense. Tying up 10 developers for 2 months is better than committing 10 developers for 4 months – not to mention the loss of use or revenue impact of unnecessarily delaying an application release.

“The second most important element is simplicity and ease of use. This is also a major contributor to the true cost in time, since it is faster to leverage existing expertise on the team without having to learn new skills. The same is true for customer support. If we run into issues, it is imperative that the solution be well-documented and that help is readily available.”

Macrometa shines in the time to value category. Providing a ready made, geo-distributed, global data network (GDN) that can serve as a cloud backend across 100s of locations means that a developer's time can now be 100% focused on getting code completed and products shipped. If you’re interested in seeing just how easy it is to accelerate your development cycle with Macrometa and your team’s existing coding skills, just click the button below and create a free developer account.

Good, Fast Or Cheap - Pick Two

Note - there is an interactive code tutorial at the bottom of the post that shows you how a CDN model can be applied to databases for serving data globally like CDNs serve static content.

 have received a lot of attention in the last few years, with the emergence of newer and richer feature capabilities and features. Companies like Fastly and CloudFlare have entered the market with a focus on helping developers leverage the power of edge computing and hyper-locality to users by caching static content close to where it is being used.

That said, CDNs cannot do much for app and API developers, because while today’s apps are bootstrapped with static content from a CDN, everything they do once loaded involves backhauling dynamic data requests to and from a cloud database hundreds of miles and thousands of milliseconds away. Today’s web services, APIs and applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in data centers that are close to their users. Applications need to elastically respond in real time to large changes in usage at peak hours, store ever increasing volumes of data, and make this data available to users in milliseconds. We need data systems that store data at the edge, at distances that are milliseconds away from users and devices.

We should be able to think of data — in the cloud or at the edge — as part of a single global system. In short, our data infrastructure should give us data that can be accessed where, when, and by whom we want with minimal latency and effort. This is a natural CDN model, but current CDN providers have not delivered the ability to exploit dynamic edge interactions.

Static content distribution is simple, with non mutable content assets replicating to CDN PoPs. Dynamic content, such as data stored in databases requires different replication and consistency guarantees. In a nutshell, this is the edge distributed data problem - and the answer to this problem is a Global Data Network or GDN.

Performance and Availability are Critical

For computers & networks, distance is a performance killer. If the images of a website are stored locally, but the infrastructure serving dynamic portions of a user experience are extremely remote, the entire user experience suffers - regardless of how quickly a CDN can serve the static portions of content. This is the fundamental reason we use CDNs and stateless function services at the edge wherever we can.

For a database, availability is also a crucial factor. Traditional cloud approaches using a centralized database create single points of failure and significant risk for the business. A failure means lost revenue, damaged reputation, and of course remediation. Depending on the business, every hour of outage can cost millions.

One of the biggest issues in using dynamic application data at the edge is the footprint required to maintain hyper-locality to users. Most modern databases and dynamic data backends work well in a single datacenter, but placing the data closer to users requires massive, global distribution. Maintaining consistency across these regions is one of the most challenging aspects of designing distributed applications. Eventual consistency can be attained, but in limited regions with data taking as long as 10 seconds to reach consistency. This creates an enormous exposure to conflicts and severely limits performance. This is why there is so much interest in Conflict Free Replicated Data Types (CRDTs).

A CDN for Stateful Application Data – Macrometa’s GDN and Fast Data Platform

Most research on CRDTs currently is focused on data types. What a developer needs is not individual data types but regular database and query languages with CRDT technology transparently managed underneath.

Macrometa’s GDN and Fast Data platform pushes the boundaries of what databases are using cutting edge technologies like 

 and provides a higher level abstraction to developers i.e., a multi-model (key-value, doc, graph, streams) database along with a query layer (C8QL) to handle both data-at-rest and data-in-motion use cases. For those interested in learning further the technology parts, you can read more about it here - 

Best way to understand a technology is to just try it. At least that is how Dad did it, America does it and how we do it. So for folks who are interested, We strongly recommend that you get a cup of strong coffee and get a free developer account on Macrometa GDN and Fast Data platform 

Can your CDN do this? A quick interactive demo of Macrometa's GDN

Macrometa provides a geo distributed platform that delivers a real-time noSQL streaming database in 25 global PoPs. The code sample below shows how data can be stored globally across these regions with strong consistency and high performance. Macrometa provides a free developer account with access to 4 regions but that should be sufficient as the platform behavior is same whether it is on 4 regions or on 25 regions. Following is a small interactive program that you can use to play around with it as a starting point.

The CDN Is Dead, Long Live The GDN*

(Note - there is an interactive code tutorial at the bottom of the post that shows you how a CDN model can be applied to dynamic application data for instant performance.)

CDNs have received much attention in the last few years, with the emergence of newer and richer feature sets. Companies like Fastly and CloudFlare have entered the market with a focus on helping developers leverage the power of edge computing and hyper-locality to users by caching static content close to where it is being used.

That said, much of the promise of CDN has not been realized by developers, as static content can only do so much to accelerate an application that is consistently backhauling dynamic data requests to a cloud datacenter. Some CDNs have responded with Key/Value stores at the edge, but generally a rich user experience requires more than a K/V store. Why?

Today’s web services, APIs and applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in data centers that are close to their users. Applications need to elastically respond in real time to large changes in usage at peak hours, store ever increasing volumes of data, and make this data available to users in milliseconds. We need data systems that store data at the edge, at distances that are milliseconds away from users and devices. We should be able to think of data — in the cloud or at the edge — as part of a single global system. In short, our data infrastructure should give us data that can be accessed where, when, and by whom we want with minimal latency and effort. This is a natural CDN model, but current CDN providers have not delivered the ability to exploit dynamic edge interactions.

Static content distribution is simple, with content assets replicating to CDN PoPs. Dynamic content, such as is stored in databases requires different replication. In a nutshell, this is the edge distributed data problem - and prevents CDN providers from servicing the most complex applications at the edge.

A CDN for Stateful Data – Macrometa’s Geo Distributed Fast Data Platform

Macrometa’s Fast Data platform pushes the boundaries on CRDTs and provides a higher level abstraction to developers i.e., a multi-model (key-value, doc, graph, streams) database along with a query layer (C8QL) to handle both data-at-rest and data-in-motion use cases.

Following is a high-level architecture view of Macrometa’s Fast Data platform. For those interested in learning further the technology parts, you can read more about it here - 

Best way to understand a technology is to just try it. At least that is how I do it. So for folks who are interested further, I recommend to go ahead and get a free developer account on Macrometa Fast Data platform. You can do that by signing up 

Here's an Interactive Program to show how a CDN can work like DB

Replit

Stateful Data At The Edge Needs A New Kind Of CDN

Over the last few years the focus of enterprises has been on cloud computing, but increasingly the focus is shifting towards edge computing which provides benefit of low latency and real-time processing. But many real world edge applications require state to provide meaningful results whether that edge application is for:

Industrial automation for monitoring, predictive maintenance, etc.

Advanced persistent threat detection against botnets & DDoS attacks

Augmented Reality (AR) & Virtual Reality (VR) apps

The state in an application can be broadly divided into following three categories.

This state is about the environment in which the subjects of the edge application are operating under. The context also could be related to the subject behavior and management in the application like configuration, roles, rules and policies specified by the administrator. Finally context is not a fixed entity i.e., the context changes with time.

For example in a fleet tracking app, the subjects could be the vehicles, fleet, depots, etc. The context of the vehicle could be the data that describes the vehicle, vehicle specification, location of the vehicle etc. Similarly the context of the fleet could be the metadata of the fleet, number of vehicles in the fleet and other details.

Some types of context like vehicle specification may not change often, whereas other types of context like geo-location of a vehicle could change quite frequently. In between these extremes we have other types of context that may change but more gradually like policies, roles, etc.

This state is about the relationships of the subjects in the edge application. The relationships could be

The relations provide deeper insights into the system that are not easily observable from the context alone. Using fleet tracking as an example, the relationships could provide insight into how delays with one subject (vehicle) can have a cascading impact on the other vehicles and route plans in the fleet. Or the relationship of a vehicle's current speed to its past speed could provide insights into when and why traffic violations happened. Recommender & ML engines are other areas that rely heavily on relationships.

Just like context, relations also do not necessarily need to be static and can change with time, e.g. the merchants in a credit card fraud detection app or route of a vehicle in fleet tracking app.

This state is about the current activity of the edge application. The activity could be many things like real-time sensor data, logs, commands & responses by the subject, etc. Also, it could be derived data from the current activity.

Taking fleet tracking as an example, the activity could be a vehicle’s speed, fuel level, geo location, or status. In a JAMStack app, it could be user clickstreams. The derived data could be the generated aggregations, transformations, filters and projections on the activity data.

, most real-work applications involve state. What I found often useful is to look at the state requirements of a system or application via these questions to build a good mental model i.e.,

What constitutes context for the application? How often does the context change?

What are the various types of relationships within the system? How often does these relationships change?

What constitutes activity and derived activity for the application?

The next important question is then how do we store, manage and process this state?

Databases have been around for a long time, and each of us have some ideas about what they are - and what they can be. Basically most data stores are organized around a single data model that determines how data can be organized, stored, and manipulated.

For example, the relational data models became popular after its publication by 

 in 1970. This was the de-facto standard until NoSQL databases became common around 2009.

NoSQL databases use a variety of data models, with document, graph and key-values models being popular. Similarly, in domains that deal with data-in-motion event driven processing models like Complex Event Processing (CEP), Messaging (pub-sub, queues), and Stream processing became very popular. Each of these data store types have their own, unique advantages.

Let’s consider the state we talked in previous section. To briefly summarize, the state of an application can be broadly divided into 3 categories -

, probably the most appropriate data store is either a relational, document or key-value database. If the application requirements involve querying data, filtering, or joins then probably document or relational database is a better bet. In addition if the edge application state spans multiple regions then a document database is likely a better bet. Many document databases now also provide ACID semantics within a region.

, the most appropriate database is a graph database. When you want a cohesive picture of your data, including the connections between subjects, graph database is the right model for it. In contrast to relational & document data stores, graph databases store data relationships as first class objects. This means fewer disconnects between your evolving schema and your actual database.

Also, the flexibility of a graph data model allows to add new nodes and relationships without expensive data migration issues. With data relationships at their center, graph databases are incredibly efficient when it comes to query speeds, even for deep and complex queries. Relationship traversals and other graph operations is a nightmare to do in relational and document databases with joins.

 probably the most appropriate data store is either a time series database (if activity is time series data) or a stream store (if activity is log data). Further depending on the processing requirements of the activity state, one may as well need a stateful/stateless stream processing solution.

While each of these data stores can handle well specific types of state created by an application they also create a very big problem. Each of these data stores create a technical silo i.e., a data store with boundaries between them that are both physical (because the data is stored in different places) and conceptual (because the data is stored in fundamentally different forms).

Until recently, this forced a difficult choice:

Choose one datastore model (i.e., relational / key-value/ document / graph / stream model) and work within the constraints of the datastore by moving the complexity to the developer.

Alternatively, choose a few data store models and cobble them together with ETL jobs and integration code - yet again moving the complexity to the developer.

Edge complicates this further. Imagine having these cobbled systems globally across dozens of locations to process data at the edge. There should be a better way!

) the health data collected from thousands of critical care patients in an Intensive Care Unit (ICU). The anonymized health data is publicly available at 

. It contains context state i.e., structured data such as demographics and medications; unstructured text such as doctor and nurse reports; relations state i.e., caregivers to patient relations, patient to lab relations, etc. and activity state i.e, time–series data of physiological signals such as vital signs and electrocardiogram (ECG).

Further some insights are more valuable shortly after events have happened, with the value diminishing quickly over time. So you also need a streaming platform (for receiving events) and stream processing platform (for processing events) to provide insights faster, often within milliseconds to seconds from the trigger.

Also the analytics often will cross the boundaries of a single data modality, such as correlating information from a doctor’s note against the physiological measurements collected from a particular sensor in realtime. In that case, you need another platform to run the ETLs and other data pipelines. Now imagine running and maintaining this business application across 10s and 100s of locations at the edge.

What developers and organizations really want is a way to use all their data in an integrated way,

 so why shouldn’t a data platform support this out of the box by natively integrating data storage, access and processing across different data models and paradigms (data-at-rest, data-in-motion). This is what I mean by freeing database from its cages.

A database that is free of cages supports multiple data models and paradigms (data-at-rest, data-in-motion) within a single, integrated backend, and provides uniform api, query and indexing capabilities. Queries are extended or combined to provide seamless query across all the supported data models and paradigms. Indexing, parsing, and processing appropriate to the data model and paradigm are included in the core database product.

 I suppose one reason could be that most databases are tightly coupled monolithic systems collapsing the query layer , indexing layer, processing layer and storage layer into one. Same with streaming platforms which also have these same layers. Another reason could be the herd mindset (i.e., the comfort & security in staying with current status quo) which silently constrains our ability to see and question. Or maybe it is some other reason. IMO why databases have these cages does not matter much. What matters is can we free the database from these cages?

So can we free the database from these cages? 

I certainly think so. For example, Macrometa Fast Data platform currently supports the data models and paradigms mentioned above with more features like search to come in near future. This is certainly not the only way to free the database from cages. I am sure in next few years we will see more around this idea of freeing databases from cages as edge becomes the center.

Finally, a sales pitch :-) Macrometa provides a managed service, enabling you and your teams to easily build, deploy and run cross-region or global, data driven, multi-modal & real time apps and APIs. Give it a try with a 

Freeing the Database from Cages for Edge Data

By Christopher Meiklejohn
Consulting Scientist, Macrometa

When deploying web and mobile applications today, developers typically start with an initial deployment of their application in a single data center. Here, the primary state for the application is kept in a highly-available database, and clients located around the world issue read and write requests against this database through an application server also located in the same data center. In this model, not all clients get the same quality of service from the database – some clients, depending on how geographically distant they are from the data center, see much higher costs in terms of latency, leading to slower application performance and a degraded user experience. Therefore, in order to provide users located all over the world with a rich, native application experiences, developers typically have to resort to geo-replication.

Geo-replication serves two purposes: high-availability and fault-tolerance. First, geo-replication allows applications to survive network and data center outages by routing users to other data centers in the event of failure. Second, geo-replication places replicas all over the globe, thereby allowing users to communicate with the data center that is geographically closest, resulting in lower latency interactions with the application. While these two properties together help to provide users with a fluid, always-on experience with the application, they raise a number of concerns around how to remain consistent between replicas when things inevitably go wrong and failures occur.

When a failure occurs that prohibits one or more of the data centers from communicating with the others, the system is forced to make a choice when presented with a request from the application. Either the system can refuse the request (or block it arbitrarily), or the system can allow the request to proceed, resulting in a potential inconsistency between the data centers in the system and the possibility of data loss under subsequent failures; systems making the former design choice are typically referred to as Consistent (CP) systems, whereas systems making the latter design choice are referred to as Available (AP).

Consistent systems aim to provide users with a “single system image” experience - where the distributed system is given the appearance of a single machine and the failures of individual components are invisible to the end user. These systems have a gold standard for correctness known as linearizability (or in transactional systems, strict serializability) which roughly states that read operations for an object will reflect the result of the most recent write to that object in a manner that is compatible with the real time order of events. Even under a certain number of faults these guarantees must not be sacrificed. To achieve these guarantees, the system typically must ensure that all (or a majority) of the data centers acknowledge writes from an elected coordinator of the group who is responsible for ordering the operations, before responding to the application.

Available systems sit on the opposite side of the spectrum. Acknowledging that failures are unavoidable, available systems typically allow the application to proceed as soon as the operation is acknowledged at one (or more) of the data centers. However, these systems typically place a burden on the application developer for a number of reasons. First, if an application has to fail over to another data center, an object that was just written may no longer be readable; in fact, without a guarantee of ordering any number of the previous writes may no longer be visible during failover. Second, without involving a majority of the data centers and using a single data center for ordering the reads and writes that occur in the system, conflicting writes to the same object can occur at different data centers at the same time: these writes do not even need to occur at precisely the same second, but only in a window that occurs between when the two data centers last communicated, to induce a conflict that must be manually resolved. Third, if a write is only acknowledged at a single data center, failure of that data center before the value can be transmitted to the other data centers might result in data loss.

In edge systems, these issues are unavoidable. Edge systems are designed specifically for reducing the user perceived end-to-end latency for applications. This requires that applications and the data needed to run these applications is replicated as close to the user as possible. Given the goals of achieving low latency for all users of the applications, these replicas must be numerous – an order of magnitude more than the numbers of traditional data center deployments. These additional networks, additional data centers, and additional replicas only increase the probability of failure, thereby exacerbating the aforementioned issues in modern stateful edge deployments.

Since edge deployments are designed to be close to the user in order to reduce user-perceived latency for read and write operations, they must be able to accept these writes locally without blocking. If these nodes wait to coordinate with a primary datacenter, either by using consensus or another mechanism, users will observe the cost of a coordinated write: the very round-trip operation that they were trying to avoid by placing a replica close to the user. Therefore, conflict resolution is unavoidable.

One mechanism for automatic conflict resolution is Conflict-Free Replicated Data Types (CRDTs.) Conflict-Free Replicated Data Types are abstract data types that codify deterministic merge behavior into the data structure itself. There are two main flavors of CRDT, each designed for use in different environments. For environments where causal ordering can be provided, there are operation-based CRDTs. For environments where no ordering can be guaranteed at all, there are state-based CRDTs.

When no ordering can be guaranteed, state-based CRDTs are ideal. State-based CRDTs are constructed using a mathematical structure called a semilattice as the base for building a data type. Semilattices, in particular bounded join-semilattices, have a unique property where for any divergent state (or, concurrent) there is always a well-defined join point. This maps nicely into the domain of distributed systems where for any two divergent replicas, they always can be merged without coordination to the same result. In order to ensure this merge behavior under any ordering, these data structures typically need to store metadata internally that models, for each change, information summarizing the changes already observed so that when an update is potentially duplicated or delivered out of order on the network, the data structure can identify this. Similarly, this data structure must also encode enough information that it can detect divergent state. There’s been significant work in the area of CRDT on reducing the amount of metadata required for state-based CRDTs, with some work reducing the metadata as far as a constant overhead and linear with the number of elements for collection types.

When causal ordering can be provided, we can do better. Causal ordering ensures that updates are delivered with respect to the order in which updates were performed – with no real-time bound on visibility. Think of it this way: each node’s updates sent to its peers are delivered in FIFO (first-in-first-out) order, and the transitive closure of those updates is also delivered in order – if node A generated update 2 after observing update 1 from node B – node C will observe the updates in any order that is compatible. Therefore, some of the information typically stored in the state-based CRDT around message ordering is now stored somewhere in the network. It’s important to realize that the system still need this information somewhere, it’s about where that information is stored and enforced. Operation-based CRDTs exploit this to reduce the overhead required for object storage. Instead of modeling abstract data types as a lattice, we model changes to the objects state as operations. These operations only need to guarantee one property, commutativity, which ensures that two updates that occurred at the same logical time, when permuted in anyway, arrive at the same result.

However, designing data types remains a challenge. If we consider a set that allows concurrent additions and removals of items in the set, it’s rather straightforward to map many of the invariants of the sequential specification to the concurrent and distributed setting: concurrent adds of the same element commute, concurrent removals of the same element commute. But, not all of these invariants can be easily mapped into the distributed setting. For example, what happens when an element is concurrently added and removed at the same time at different replicas? In this case, in order for the system to remain deterministically convergent without coordination, the designers of these data types need to decide how these operations will be arbitrated up-front. This has led to many variations of these data types: one data type variant for each pair of concurrent operations that doesn’t commute. In the case of the concurrent addition and removal on a set, there’s two variants: one that biases towards keeping the addition and ignoring the removal, and one that biases towards removals and ignores the additions.

The challenges do not stop there. One of the most primitive and valuable data types, the register, poses similar challenges. Registers do not have the benefit of different types of operations by which we can arbitrate (as in the previous example we had both add and remove operations on a set). Registers only have a single operation: set, to modify the value in place. Therefore, every concurrent operation in the system will conflict and we must make a decision ahead of time how we will resolve these conflicting updates. There are several approaches taken by many of the different systems that implement distributed registers. First, we could keep both values - however, this is problematic if an application is expecting to read a single value from the database and instead is provided with a set when reading from the database. Second, we could associate a timestamp with each write and have the system pick the most recent update - however, clocks are known to drift and the system may pick the wrong value (furthermore, a clock drifting into the future can prevent any writes at the current time, a phenomenon in Apache Cassandra called “doomstones.”) Finally, another possible option (and, not the only remaining option) arbitrarily picks one of the updates – however, to remain correct this must be done in a manner that all nodes pick the exact same update, without requiring coordination in the critical path.

An open question remains: is there a way to make practical choices on how to resolve these conflicts in a manner that is easy to implement for developers, while admitting the difficulty of solving these in a mechanism that suits every possible case and every possible conflict resolution strategy? That will be the subject of our next post: how the Macrometa convergence engine works.

Motivating CRDTs

Macrometa is the hyper distributed cloud platform for enterprises building global applications at the Edge. 

PhotonIQ

Global Data Network

Learning Hub

Case Studies

Developer Hub

Demos

Support

About

Partners

Press

Careers

Contact Us

Terms of Service

Privacy Policy

Trust Center

Macrometa vs. Edgio

Are you an Edgio customer facing uncertainty? Discover how Macrometa ensures continuity and delivers high-performance solutions for your business.

Archive