Explore insights on real-time edge computing, AI, and cloud solutions. Learn how our technology optimizes performance across industries.

Blog

A friend of mine was a construction contractor and his customer mantra was simple:

“I can build it good, fast, and cheap. Pick 2.”

You’ve probably heard that before, and of course it applies whether you’re building a house or a cloud native application. The hard part is picking which two criteria are most important for you, and knowing which criteria is expendable. For my contractor friend, if the customer picked fast and cheap his mantra to the construction crew became “it doesn’t look perfect, but it will look fine from the next job site.”

For developers, there is no shortage of tools and SaaS/PaaS products to help them build. I started thinking about how difficult it must be to sort through the available options and select a product that you will essentially be married to for months or years, and the difficulty of balancing quality, performance, and cost.

To get some perspective, I sat down with our Chief Architect to pick his brain about the vendor selection process. In the enterprise space, many product decisions regarding commodity technology are about “good and cheap” – or at least good enough. To put it another way it is about comparing value to price and finding an acceptable compromise. In the developer space, there are other nuances.

One of the big takeaways from the conversation is that cost is measured in different terms by developer organizations. We typically think of cost in terms of licensing, supporting infrastructure, or monthly SaaS charges. As Durga pointed out, it isn’t quite so straightforward for a dev shop.

“Obviously the first selection criteria is ‘can it handle my workload.’ Sometimes the workload itself narrows the field down to just one or two products that can accomplish the task. An example requirement we’re familiar with would be a database that is consistent across a dozen global regions. A hosted NoSQL or cloud platform that can only replicate across 6 locations may be more common, but if I truly needed more locations or a global footprint, then Macrometa would be the right platform.”

This is where the conversation became even more interesting. I asked if there were multiple products that fit the workload needs, how would he narrow down the options and make a final selection?

“The most important thing is the cost – but not in terms of the actual product pricing. Unless the cost is vastly different, the monthly or one-time expense isn’t enough to influence a decision. I evaluate products based upon my most important costs – Time.”

This made a lot of sense to me. If Product A is $500 and Product B is $700, but Product B will let me produce my product (an app) 50% faster, then the additional time to market savings greatly offset the additional expense. Tying up 10 developers for 2 months is better than committing 10 developers for 4 months – not to mention the loss of use or revenue impact of unnecessarily delaying an application release.

“The second most important element is simplicity and ease of use. This is also a major contributor to the true cost in time, since it is faster to leverage existing expertise on the team without having to learn new skills. The same is true for customer support. If we run into issues, it is imperative that the solution be well-documented and that help is readily available.”

Macrometa shines in the time to value category. Providing a ready made, geo-distributed, global data network (GDN) that can serve as a cloud backend across 100s of locations means that a developer's time can now be 100% focused on getting code completed and products shipped. If you’re interested in seeing just how easy it is to accelerate your development cycle with Macrometa and your team’s existing coding skills, just click the button below and create a free developer account.

Good, Fast Or Cheap - Pick Two

Note - there is an interactive code tutorial at the bottom of the post that shows you how a CDN model can be applied to databases for serving data globally like CDNs serve static content.

 have received a lot of attention in the last few years, with the emergence of newer and richer feature capabilities and features. Companies like Fastly and CloudFlare have entered the market with a focus on helping developers leverage the power of edge computing and hyper-locality to users by caching static content close to where it is being used.

That said, CDNs cannot do much for app and API developers, because while today’s apps are bootstrapped with static content from a CDN, everything they do once loaded involves backhauling dynamic data requests to and from a cloud database hundreds of miles and thousands of milliseconds away. Today’s web services, APIs and applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in data centers that are close to their users. Applications need to elastically respond in real time to large changes in usage at peak hours, store ever increasing volumes of data, and make this data available to users in milliseconds. We need data systems that store data at the edge, at distances that are milliseconds away from users and devices.

We should be able to think of data — in the cloud or at the edge — as part of a single global system. In short, our data infrastructure should give us data that can be accessed where, when, and by whom we want with minimal latency and effort. This is a natural CDN model, but current CDN providers have not delivered the ability to exploit dynamic edge interactions.

Static content distribution is simple, with non mutable content assets replicating to CDN PoPs. Dynamic content, such as data stored in databases requires different replication and consistency guarantees. In a nutshell, this is the edge distributed data problem - and the answer to this problem is a Global Data Network or GDN.

Performance and Availability are Critical

For computers & networks, distance is a performance killer. If the images of a website are stored locally, but the infrastructure serving dynamic portions of a user experience are extremely remote, the entire user experience suffers - regardless of how quickly a CDN can serve the static portions of content. This is the fundamental reason we use CDNs and stateless function services at the edge wherever we can.

For a database, availability is also a crucial factor. Traditional cloud approaches using a centralized database create single points of failure and significant risk for the business. A failure means lost revenue, damaged reputation, and of course remediation. Depending on the business, every hour of outage can cost millions.

One of the biggest issues in using dynamic application data at the edge is the footprint required to maintain hyper-locality to users. Most modern databases and dynamic data backends work well in a single datacenter, but placing the data closer to users requires massive, global distribution. Maintaining consistency across these regions is one of the most challenging aspects of designing distributed applications. Eventual consistency can be attained, but in limited regions with data taking as long as 10 seconds to reach consistency. This creates an enormous exposure to conflicts and severely limits performance. This is why there is so much interest in Conflict Free Replicated Data Types (CRDTs).

A CDN for Stateful Application Data – Macrometa’s GDN and Fast Data Platform

Most research on CRDTs currently is focused on data types. What a developer needs is not individual data types but regular database and query languages with CRDT technology transparently managed underneath.

Macrometa’s GDN and Fast Data platform pushes the boundaries of what databases are using cutting edge technologies like 

 and provides a higher level abstraction to developers i.e., a multi-model (key-value, doc, graph, streams) database along with a query layer (C8QL) to handle both data-at-rest and data-in-motion use cases. For those interested in learning further the technology parts, you can read more about it here - 

Best way to understand a technology is to just try it. At least that is how Dad did it, America does it and how we do it. So for folks who are interested, We strongly recommend that you get a cup of strong coffee and get a free developer account on Macrometa GDN and Fast Data platform 

Can your CDN do this? A quick interactive demo of Macrometa's GDN

Macrometa provides a geo distributed platform that delivers a real-time noSQL streaming database in 25 global PoPs. The code sample below shows how data can be stored globally across these regions with strong consistency and high performance. Macrometa provides a free developer account with access to 4 regions but that should be sufficient as the platform behavior is same whether it is on 4 regions or on 25 regions. Following is a small interactive program that you can use to play around with it as a starting point.

The CDN Is Dead, Long Live The GDN*

(Note - there is an interactive code tutorial at the bottom of the post that shows you how a CDN model can be applied to dynamic application data for instant performance.)

CDNs have received much attention in the last few years, with the emergence of newer and richer feature sets. Companies like Fastly and CloudFlare have entered the market with a focus on helping developers leverage the power of edge computing and hyper-locality to users by caching static content close to where it is being used.

That said, much of the promise of CDN has not been realized by developers, as static content can only do so much to accelerate an application that is consistently backhauling dynamic data requests to a cloud datacenter. Some CDNs have responded with Key/Value stores at the edge, but generally a rich user experience requires more than a K/V store. Why?

Today’s web services, APIs and applications are required to be highly responsive and always online. To achieve low latency and high availability, instances of these applications need to be deployed in data centers that are close to their users. Applications need to elastically respond in real time to large changes in usage at peak hours, store ever increasing volumes of data, and make this data available to users in milliseconds. We need data systems that store data at the edge, at distances that are milliseconds away from users and devices. We should be able to think of data — in the cloud or at the edge — as part of a single global system. In short, our data infrastructure should give us data that can be accessed where, when, and by whom we want with minimal latency and effort. This is a natural CDN model, but current CDN providers have not delivered the ability to exploit dynamic edge interactions.

Static content distribution is simple, with content assets replicating to CDN PoPs. Dynamic content, such as is stored in databases requires different replication. In a nutshell, this is the edge distributed data problem - and prevents CDN providers from servicing the most complex applications at the edge.

A CDN for Stateful Data – Macrometa’s Geo Distributed Fast Data Platform

Macrometa’s Fast Data platform pushes the boundaries on CRDTs and provides a higher level abstraction to developers i.e., a multi-model (key-value, doc, graph, streams) database along with a query layer (C8QL) to handle both data-at-rest and data-in-motion use cases.

Following is a high-level architecture view of Macrometa’s Fast Data platform. For those interested in learning further the technology parts, you can read more about it here - 

Best way to understand a technology is to just try it. At least that is how I do it. So for folks who are interested further, I recommend to go ahead and get a free developer account on Macrometa Fast Data platform. You can do that by signing up 

Here's an Interactive Program to show how a CDN can work like DB

Replit

Stateful Data At The Edge Needs A New Kind Of CDN

Over the last few years the focus of enterprises has been on cloud computing, but increasingly the focus is shifting towards edge computing which provides benefit of low latency and real-time processing. But many real world edge applications require state to provide meaningful results whether that edge application is for:

Industrial automation for monitoring, predictive maintenance, etc.

Advanced persistent threat detection against botnets & DDoS attacks

Augmented Reality (AR) & Virtual Reality (VR) apps

The state in an application can be broadly divided into following three categories.

This state is about the environment in which the subjects of the edge application are operating under. The context also could be related to the subject behavior and management in the application like configuration, roles, rules and policies specified by the administrator. Finally context is not a fixed entity i.e., the context changes with time.

For example in a fleet tracking app, the subjects could be the vehicles, fleet, depots, etc. The context of the vehicle could be the data that describes the vehicle, vehicle specification, location of the vehicle etc. Similarly the context of the fleet could be the metadata of the fleet, number of vehicles in the fleet and other details.

Some types of context like vehicle specification may not change often, whereas other types of context like geo-location of a vehicle could change quite frequently. In between these extremes we have other types of context that may change but more gradually like policies, roles, etc.

This state is about the relationships of the subjects in the edge application. The relationships could be

The relations provide deeper insights into the system that are not easily observable from the context alone. Using fleet tracking as an example, the relationships could provide insight into how delays with one subject (vehicle) can have a cascading impact on the other vehicles and route plans in the fleet. Or the relationship of a vehicle's current speed to its past speed could provide insights into when and why traffic violations happened. Recommender & ML engines are other areas that rely heavily on relationships.

Just like context, relations also do not necessarily need to be static and can change with time, e.g. the merchants in a credit card fraud detection app or route of a vehicle in fleet tracking app.

This state is about the current activity of the edge application. The activity could be many things like real-time sensor data, logs, commands & responses by the subject, etc. Also, it could be derived data from the current activity.

Taking fleet tracking as an example, the activity could be a vehicle’s speed, fuel level, geo location, or status. In a JAMStack app, it could be user clickstreams. The derived data could be the generated aggregations, transformations, filters and projections on the activity data.

, most real-work applications involve state. What I found often useful is to look at the state requirements of a system or application via these questions to build a good mental model i.e.,

What constitutes context for the application? How often does the context change?

What are the various types of relationships within the system? How often does these relationships change?

What constitutes activity and derived activity for the application?

The next important question is then how do we store, manage and process this state?

Databases have been around for a long time, and each of us have some ideas about what they are - and what they can be. Basically most data stores are organized around a single data model that determines how data can be organized, stored, and manipulated.

For example, the relational data models became popular after its publication by 

 in 1970. This was the de-facto standard until NoSQL databases became common around 2009.

NoSQL databases use a variety of data models, with document, graph and key-values models being popular. Similarly, in domains that deal with data-in-motion event driven processing models like Complex Event Processing (CEP), Messaging (pub-sub, queues), and Stream processing became very popular. Each of these data store types have their own, unique advantages.

Let’s consider the state we talked in previous section. To briefly summarize, the state of an application can be broadly divided into 3 categories -

, probably the most appropriate data store is either a relational, document or key-value database. If the application requirements involve querying data, filtering, or joins then probably document or relational database is a better bet. In addition if the edge application state spans multiple regions then a document database is likely a better bet. Many document databases now also provide ACID semantics within a region.

, the most appropriate database is a graph database. When you want a cohesive picture of your data, including the connections between subjects, graph database is the right model for it. In contrast to relational & document data stores, graph databases store data relationships as first class objects. This means fewer disconnects between your evolving schema and your actual database.

Also, the flexibility of a graph data model allows to add new nodes and relationships without expensive data migration issues. With data relationships at their center, graph databases are incredibly efficient when it comes to query speeds, even for deep and complex queries. Relationship traversals and other graph operations is a nightmare to do in relational and document databases with joins.

 probably the most appropriate data store is either a time series database (if activity is time series data) or a stream store (if activity is log data). Further depending on the processing requirements of the activity state, one may as well need a stateful/stateless stream processing solution.

While each of these data stores can handle well specific types of state created by an application they also create a very big problem. Each of these data stores create a technical silo i.e., a data store with boundaries between them that are both physical (because the data is stored in different places) and conceptual (because the data is stored in fundamentally different forms).

Until recently, this forced a difficult choice:

Choose one datastore model (i.e., relational / key-value/ document / graph / stream model) and work within the constraints of the datastore by moving the complexity to the developer.

Alternatively, choose a few data store models and cobble them together with ETL jobs and integration code - yet again moving the complexity to the developer.

Edge complicates this further. Imagine having these cobbled systems globally across dozens of locations to process data at the edge. There should be a better way!

) the health data collected from thousands of critical care patients in an Intensive Care Unit (ICU). The anonymized health data is publicly available at 

. It contains context state i.e., structured data such as demographics and medications; unstructured text such as doctor and nurse reports; relations state i.e., caregivers to patient relations, patient to lab relations, etc. and activity state i.e, time–series data of physiological signals such as vital signs and electrocardiogram (ECG).

Further some insights are more valuable shortly after events have happened, with the value diminishing quickly over time. So you also need a streaming platform (for receiving events) and stream processing platform (for processing events) to provide insights faster, often within milliseconds to seconds from the trigger.

Also the analytics often will cross the boundaries of a single data modality, such as correlating information from a doctor’s note against the physiological measurements collected from a particular sensor in realtime. In that case, you need another platform to run the ETLs and other data pipelines. Now imagine running and maintaining this business application across 10s and 100s of locations at the edge.

What developers and organizations really want is a way to use all their data in an integrated way,

 so why shouldn’t a data platform support this out of the box by natively integrating data storage, access and processing across different data models and paradigms (data-at-rest, data-in-motion). This is what I mean by freeing database from its cages.

A database that is free of cages supports multiple data models and paradigms (data-at-rest, data-in-motion) within a single, integrated backend, and provides uniform api, query and indexing capabilities. Queries are extended or combined to provide seamless query across all the supported data models and paradigms. Indexing, parsing, and processing appropriate to the data model and paradigm are included in the core database product.

 I suppose one reason could be that most databases are tightly coupled monolithic systems collapsing the query layer , indexing layer, processing layer and storage layer into one. Same with streaming platforms which also have these same layers. Another reason could be the herd mindset (i.e., the comfort & security in staying with current status quo) which silently constrains our ability to see and question. Or maybe it is some other reason. IMO why databases have these cages does not matter much. What matters is can we free the database from these cages?

So can we free the database from these cages? 

I certainly think so. For example, Macrometa Fast Data platform currently supports the data models and paradigms mentioned above with more features like search to come in near future. This is certainly not the only way to free the database from cages. I am sure in next few years we will see more around this idea of freeing databases from cages as edge becomes the center.

Finally, a sales pitch :-) Macrometa provides a managed service, enabling you and your teams to easily build, deploy and run cross-region or global, data driven, multi-modal & real time apps and APIs. Give it a try with a 

Freeing the Database from Cages for Edge Data

By Christopher Meiklejohn
Consulting Scientist, Macrometa

When deploying web and mobile applications today, developers typically start with an initial deployment of their application in a single data center. Here, the primary state for the application is kept in a highly-available database, and clients located around the world issue read and write requests against this database through an application server also located in the same data center. In this model, not all clients get the same quality of service from the database – some clients, depending on how geographically distant they are from the data center, see much higher costs in terms of latency, leading to slower application performance and a degraded user experience. Therefore, in order to provide users located all over the world with a rich, native application experiences, developers typically have to resort to geo-replication.

Geo-replication serves two purposes: high-availability and fault-tolerance. First, geo-replication allows applications to survive network and data center outages by routing users to other data centers in the event of failure. Second, geo-replication places replicas all over the globe, thereby allowing users to communicate with the data center that is geographically closest, resulting in lower latency interactions with the application. While these two properties together help to provide users with a fluid, always-on experience with the application, they raise a number of concerns around how to remain consistent between replicas when things inevitably go wrong and failures occur.

When a failure occurs that prohibits one or more of the data centers from communicating with the others, the system is forced to make a choice when presented with a request from the application. Either the system can refuse the request (or block it arbitrarily), or the system can allow the request to proceed, resulting in a potential inconsistency between the data centers in the system and the possibility of data loss under subsequent failures; systems making the former design choice are typically referred to as Consistent (CP) systems, whereas systems making the latter design choice are referred to as Available (AP).

Consistent systems aim to provide users with a “single system image” experience - where the distributed system is given the appearance of a single machine and the failures of individual components are invisible to the end user. These systems have a gold standard for correctness known as linearizability (or in transactional systems, strict serializability) which roughly states that read operations for an object will reflect the result of the most recent write to that object in a manner that is compatible with the real time order of events. Even under a certain number of faults these guarantees must not be sacrificed. To achieve these guarantees, the system typically must ensure that all (or a majority) of the data centers acknowledge writes from an elected coordinator of the group who is responsible for ordering the operations, before responding to the application.

Available systems sit on the opposite side of the spectrum. Acknowledging that failures are unavoidable, available systems typically allow the application to proceed as soon as the operation is acknowledged at one (or more) of the data centers. However, these systems typically place a burden on the application developer for a number of reasons. First, if an application has to fail over to another data center, an object that was just written may no longer be readable; in fact, without a guarantee of ordering any number of the previous writes may no longer be visible during failover. Second, without involving a majority of the data centers and using a single data center for ordering the reads and writes that occur in the system, conflicting writes to the same object can occur at different data centers at the same time: these writes do not even need to occur at precisely the same second, but only in a window that occurs between when the two data centers last communicated, to induce a conflict that must be manually resolved. Third, if a write is only acknowledged at a single data center, failure of that data center before the value can be transmitted to the other data centers might result in data loss.

In edge systems, these issues are unavoidable. Edge systems are designed specifically for reducing the user perceived end-to-end latency for applications. This requires that applications and the data needed to run these applications is replicated as close to the user as possible. Given the goals of achieving low latency for all users of the applications, these replicas must be numerous – an order of magnitude more than the numbers of traditional data center deployments. These additional networks, additional data centers, and additional replicas only increase the probability of failure, thereby exacerbating the aforementioned issues in modern stateful edge deployments.

Since edge deployments are designed to be close to the user in order to reduce user-perceived latency for read and write operations, they must be able to accept these writes locally without blocking. If these nodes wait to coordinate with a primary datacenter, either by using consensus or another mechanism, users will observe the cost of a coordinated write: the very round-trip operation that they were trying to avoid by placing a replica close to the user. Therefore, conflict resolution is unavoidable.

One mechanism for automatic conflict resolution is Conflict-Free Replicated Data Types (CRDTs.) Conflict-Free Replicated Data Types are abstract data types that codify deterministic merge behavior into the data structure itself. There are two main flavors of CRDT, each designed for use in different environments. For environments where causal ordering can be provided, there are operation-based CRDTs. For environments where no ordering can be guaranteed at all, there are state-based CRDTs.

When no ordering can be guaranteed, state-based CRDTs are ideal. State-based CRDTs are constructed using a mathematical structure called a semilattice as the base for building a data type. Semilattices, in particular bounded join-semilattices, have a unique property where for any divergent state (or, concurrent) there is always a well-defined join point. This maps nicely into the domain of distributed systems where for any two divergent replicas, they always can be merged without coordination to the same result. In order to ensure this merge behavior under any ordering, these data structures typically need to store metadata internally that models, for each change, information summarizing the changes already observed so that when an update is potentially duplicated or delivered out of order on the network, the data structure can identify this. Similarly, this data structure must also encode enough information that it can detect divergent state. There’s been significant work in the area of CRDT on reducing the amount of metadata required for state-based CRDTs, with some work reducing the metadata as far as a constant overhead and linear with the number of elements for collection types.

When causal ordering can be provided, we can do better. Causal ordering ensures that updates are delivered with respect to the order in which updates were performed – with no real-time bound on visibility. Think of it this way: each node’s updates sent to its peers are delivered in FIFO (first-in-first-out) order, and the transitive closure of those updates is also delivered in order – if node A generated update 2 after observing update 1 from node B – node C will observe the updates in any order that is compatible. Therefore, some of the information typically stored in the state-based CRDT around message ordering is now stored somewhere in the network. It’s important to realize that the system still need this information somewhere, it’s about where that information is stored and enforced. Operation-based CRDTs exploit this to reduce the overhead required for object storage. Instead of modeling abstract data types as a lattice, we model changes to the objects state as operations. These operations only need to guarantee one property, commutativity, which ensures that two updates that occurred at the same logical time, when permuted in anyway, arrive at the same result.

However, designing data types remains a challenge. If we consider a set that allows concurrent additions and removals of items in the set, it’s rather straightforward to map many of the invariants of the sequential specification to the concurrent and distributed setting: concurrent adds of the same element commute, concurrent removals of the same element commute. But, not all of these invariants can be easily mapped into the distributed setting. For example, what happens when an element is concurrently added and removed at the same time at different replicas? In this case, in order for the system to remain deterministically convergent without coordination, the designers of these data types need to decide how these operations will be arbitrated up-front. This has led to many variations of these data types: one data type variant for each pair of concurrent operations that doesn’t commute. In the case of the concurrent addition and removal on a set, there’s two variants: one that biases towards keeping the addition and ignoring the removal, and one that biases towards removals and ignores the additions.

The challenges do not stop there. One of the most primitive and valuable data types, the register, poses similar challenges. Registers do not have the benefit of different types of operations by which we can arbitrate (as in the previous example we had both add and remove operations on a set). Registers only have a single operation: set, to modify the value in place. Therefore, every concurrent operation in the system will conflict and we must make a decision ahead of time how we will resolve these conflicting updates. There are several approaches taken by many of the different systems that implement distributed registers. First, we could keep both values - however, this is problematic if an application is expecting to read a single value from the database and instead is provided with a set when reading from the database. Second, we could associate a timestamp with each write and have the system pick the most recent update - however, clocks are known to drift and the system may pick the wrong value (furthermore, a clock drifting into the future can prevent any writes at the current time, a phenomenon in Apache Cassandra called “doomstones.”) Finally, another possible option (and, not the only remaining option) arbitrarily picks one of the updates – however, to remain correct this must be done in a manner that all nodes pick the exact same update, without requiring coordination in the critical path.

An open question remains: is there a way to make practical choices on how to resolve these conflicts in a manner that is easy to implement for developers, while admitting the difficulty of solving these in a mechanism that suits every possible case and every possible conflict resolution strategy? That will be the subject of our next post: how the Macrometa convergence engine works.

Motivating CRDTs

At some point in the last 12 months, something historic happened quietly and right under everyones' noses. Without much fanfare, the center of gravity for cloud computing moved from the centralized databases and compute in giant hyper-scale data-centers to the edge of the network where CDNs live. This tectonic shift changes the way you (dear developer) will build applications, APIs and all things programmable in the months, years and decades to come.

The shift started with AWS introducing a radical new way to build applications with functions as a service 2 years ago. Dubbed "lambdas", AWS' serverless functions let you create functions of code that are spun up on demand in response to events in the real world - or in other adjacent systems. Without the burden of any infrastructure to operate or manage, Lambdas let anyone create web scale backends that scale elastically to meet demand with a powerful cost model - you only pay for the resources your functions consume while they run. As great as lambdas are they still sat in a single AWS cloud region and that meant any calls to your lambdas had to travel across the internet, over hundreds of miles, thousands of milliseconds of time to get to the runtime to execute the function. That coupled with the fact that your lambdas might be cold (no running instances to route the request to) meant that they were slow for many real time use cases. AWS solved the first problem by introducing Lambda @ edge - a node.js lambda engine that serves JavaScript functions at 100s of globally distributed locations around the world. With Lambda @ edge your requests could now go to the closest AWS edge and be served by a JavaScript function in milliseconds. Problem solved? Almost. The cold start problems remained.

Serverless functions and workers at the edge - exciting possibilities

There's still one giant problem waiting to be solved for the next era of edge cloud computing to really take off. While we've moved compute to the edge, the data is still sitting hundreds of miles and thousands of milliseconds away in lonely, centralized databases in a single cloud region and not at the edge.

Now imagine if like Lambda @edge, CloudFlare Workers and Fastly Lucet/Terranium, you had a database that was globally geo-distributed and ran in 100s of locations worldwide and delivered accurate, highly consistent, mutable data to your functions in real-time?

CloudFlare workers are similar to Lambda @ edge in philosophy but entirely different in design and implementation. Eschewing node.js, CloudFlare Workers instead use the V8 Javascript engine (the same V8 that runs in your Chrome browser but extensively reengineered to run at the CDN edge) to execute JavaScript functions inside their lightning fast edge locations spread around the world. Using their GUI Worker developer tools or the command line based wrangler tool, you can write, package and deploy a worker to all their 200+ locations in seconds and have your own global, geo-distributed workers responding to requests at insanely low latencies for fractions of a penny per invocation.

Also, the awesome folks at Fastly have their own vision and implementation of serverless workers with Lucet and Terrarium. Lucet is incredibly exciting - it can fire up a worker from (or isolate from) the cold in under 15 microseconds (that's 

). I've chosen not to write about WASM in this post - but will share thoughts on WASM in a separate post.

Macrometa and AWS Lambda @ edge - a global stateful data layer for serverless functions in the cloud and at the edge

Macrometa is an ultra scale, geo-distributed serverless database platform. Put simply, its a new kind of database that doesn't run in just one cloud region or data center - it runs across hundreds just like the serverless runtimes provided by Lambda @ edge, CloudFlare Workers and Fastly Lucet/Terrarium. But like conventional databases it provides strong consistency guarantees, a query language and a modern interface (multi model - K/V, JSON DocDB, GraphDB, Stream, pub/sub and more in one interface) and serverless API for working with it. Unlike previous attempts to build these databases that tradeoff consistency for speed, Macrometa provides a rich set of consistency primitives to ensure that data is not just mutable at the edge but is also globally and shared with accuracy and low latencies.

By coupling the Macrometa serverless database with your functions on any of these edge computing runtimes, developers can now write apps, APIs and all kinds of incredible things that automatically run globally in seconds across 100s of locations serving data and compute instantly - something that previously only the giant web scale tech companies like Google, Facebook could do.

To give you a taste of the awesome power that the combination of these technologies provide - let's switch gears and build a stateful geo distributed AWS Lambda @ edge function that runs globally and queries data from the Macrometa database edge location closest to where the function is running.

Writing a global stateful Lambda function in 5 easy steps

In this example we will serve a globally geo-distributed address book that has a bunch (about 500) names, email addresses and zip codes that are stored in the Macrometa global database across 25 global locations worldwide.

https://4uiu9z43qj.execute-api.us-west-1.amazonaws.com/default/addressbookFunc

When your client (mobile app or web browser) calls the URL of the Lambda @ edge function, DNS will resolve the address using global latency based routing to send the request to the closest Lambda @ edge PoP (a regional/local datacenter where the function is stored and runs). For context there are about 180 PoPs that AWS provides which means that users are never more than 25ms away from the vast majority of them. When the https request is received, the Lambda @ edge function is invoked and it makes a call to the Macrometa database using a light weight graphQL end point (we call it RESTQL - you can learn more about it by reading about our query language 

 - Get a free Macrometa developer account at 

 - Create a global fabric and data collection in Macrometa

Once on the main console page, click on collections and add collections

Click on create collection and use the dialog to provide details

Skip the key - we will use an autogenerated one

Give the collection a name - lets call it addresses and click save

 - Put some data in your global collection


In the query editor copy paste the address book data:

Copy the query and paste in the Macrometa query window

Click on Execute to run the query and insert the data

You can click on the collection / Addresses and see that the data has been inserted

Cool feature - on the main console click the top link global fabric and click on one of the map pins - by clicking on the >> you can open a direct console connection to that data center and use the console to browse data and perform operations on that location. You can click collections on a console in a different datacenter and see that the data has replicated.

 - Write a query to enumerate it and expose it as a geo-distributed RESTful end point


execute it and it will enumerate all the docs in the addresses collection

Click "Save as" and give it a name in the dialog - call it showAddresses

Click on queries and you should see it at the top of the list - click the Play button to run/test it

Now comes the magical part. When you saved the query, Macrometa automatically turned the function into a globally distributed RESTful end point

Click on Support in the main console, in the API viewer scroll down to the RESTQL section

Find the API end point for "Execute restql by name" (the second one under RESTQL) and click on it and then click "Try it out"

Provide your tenant id (you got this when you signed up)

Provide your fabric name (_system if you didn't create a new one - the default is _system)

Now click execute and you should see the API execute and its response displayed below

There's a section which shows you the curl command - we're going to need this in the next section

You can copy past the curl command on your command line and run it if you have curl installed.

IMPORTANT - copy the url after POST between the quotes

it looks like this (copy the highlighted part from the API explorer) - "https://<host>macrometa.io/_tenant/<tenantname>/_fabric/fabricname/restql/execute/showAddresses"</tenantname></host>

Paste in your favorite text editor and change the tenant name and fabricname to your tenant name and the fabric you created before

Now copy the part after the bearer (this is your authentication token)

It looks like this (copy the part highlighted in yellow from the API explorer in the UI console under support)-
"Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NjY2OTEyNTYsImlhdCI6MS41NjQwOTkyNTYwMzQyMTIyZSs2LCJpc3MiNOTAREALTKEN3VzZXJuYW1lIjoicm9vdCIsInRlbmFudCI6Il9tbSJ9.GC5wZG0Ng6AM0wEna1un7-8G0PY6Q7ROnzCnxxrtxMA="

Copy the part AFTER the bearer till the end of the string and paste in your favorite editor

 - Integrating the Database RESTful API with the Lambda functions -

Assuming you already have a AWS account, create a new Lambda function and make sure it is placed to run at US-EAST-1

Once you get to the code editor, copy and paste the following code from pastebin


Now carefully copy the url from your text editor and paste the URL you copied from the Macrometa Console where it says host and path (make sure you don't inadvertently add or remove anything while copy pasting).

Change the tenant name to your tenant name

Next copy the authentication token and paste it inside the quotes starting from bearer in the headers section where it says 'Authorization' :

Click save and test and you should see it run inside the console and retrieve data from the nearest Macrometa databased edge location

Assuming you already have a AWS account, create a new Lambda function and make sure its placed to run at US-EAST- virginia

 - if you've not missed a step or made any typing mistakes so far, your Lambda will now deploy and automatically pull data from the closest Macrometa Fast Data PoP.

Use the API end point provided by the AWS gateway and test to see if your Lambda fetches data from on the 25 global data Pops closest to it that Macrometa provides.

Note that this lambda is not geo distributed yet

 - that will need you to setup a cloudfront distribution which I will cover in a separate post. But the steps to do that are easy and you can find plenty of examples on google for that if you want to try it out now.

So there you have it - a completely serverless, stateful, global, geo-distributed API end point that will serve dynamically queried database responses in milliseconds all around the world

Macrometa & AWS Lambda - Geo Distributed Stateful Data For Serverless Functions - A Step By Step Guide

, we discussed the challenges of conflict resolution when supporting a highly-available distributed database that operates at the edge. Let’s recap the challenges.

Moving applications to the edge is primarily done for two reasons: latency and availability. First, by moving application code and associated application data to the edge, user requests can be processed entirely at the edge and responses returned to the user quicker; this serves to reduce the user-perceived latency. Second, in the event of a data center or connectivity failure, user requests can be rerouted to another geographically close Point-of-Presence preventing periods of application inaccessibility, allowing the user to continue using the application without an interruption of service. Even further, if the user’s geographically closest Point-of-Presence is unable to contact other data centers because of connection failure, users can operate on data local to that location without having to experience an interruption of service.

Enabling this type of user experience ultimately means that conflicts at the data store are inevitable. Without waiting for agreement from all of the data centers in the system before modifying an object in the system – the key to remaining available under network partitions that are a result of data center failures and connectivity loss – concurrent modifications to the same object in the data store may conflict. These conflicts need not occur simultaneously – but only within a time window where a relationship between the two modifications cannot be determined.

To provide a practical example, if Alice and Bob both modify a document in their address book containing the current address of a relative – even if the values written are the same or minor variations using abbreviations – the system will detect a conflict and will be unable to proceed. Generally speaking, in this case the system has to make one of two choices: either keep both values, and upon the next read performed by the user return the user multiple values, when they may have been expecting a single value; or, pick one of the updates as the “winning” updates, by using some sort of arbitration procedure to figure out which of the modifications “overwrites” the other. Some systems (e.g., Riak, Bayou, Dynamo) return both values to the user and allow the user to replace the multiple values with a single value – a form of application-specific conflict resolution; whereas other systems (e.g., Cassandra) choose a single value based on a user provided time stamp. In some of the formal literature on the topic the former is known as a Multi-Value Register, the latter a Last-Writer-Wins Register. However, since applications work with richer data types than just registers – they use dictionaries, sets, and counters – keeping both values or arbitrarily choosing one might not provide the correct application behavior for the user. To provide a pathological example, if a counter is modeled as a register, for every conflicting set of writes a single increment or decrement of the counter would be lost – this may be very bad for the application!

Coming up with interesting ways to converge conflicting operations across different data types has a whole field of study and body of work behind it called Conflict-Free Replicated Data Types (CRDTs). We touched on the challenges of designing CRDTs in our previous post. In this post, we discuss how the Real-Time Intelligent Convergence Engine – at the heart of Macrometa’s C8DB offering – handles conflict resolution.

Real-Time Intelligent Convergence Engine (RICE)

Given that Macrometa’s platform must face the challenges of convergence and conflict resolution head-on to remain available at the edge when network partitions inevitably occur, it was paramount to establish a data model that was both usable from the application point-of-view and contained conflict resolution logic that was transparent to the user.

In C8DB, application users store unstructured documents – similar to JSON objects, or documents in a document-oriented database such as CouchDB. Documents are values that are associated with keys, and applications can either access a document by key – using a point query – or use a SQL-inspired query language that allows for aggregations and selections across documents located in the system. Documents can be arbitrarily nested and therefore are recursive data structures containing keys that map to values (which, themselves can be dictionaries mapping further keys to values.) To perform modifications to a document, users use the query language to perform INSERT or UPDATE operations against the data store – these operations can modify either a single document or a collection of documents.

Internally, these operations are totally ordered at each data center. Therefore, the types of conflicts the application developer needs to reason about are conflicting operations that happen concurrently at different data centers. These conflicting operations occur when the system cannot reason about which update preceded the other, nor when the system can determine some precedence between the different types of operations. In our example before, if Alice and Bob were to update the street address for the same record concurrently at different data centers, we have to make a choice. In lieu of using different timestamps provided by the user to arbitrarily pick a write – which, has been known to result in serious consistency problems in Cassandra when clock skew occurs – RICE instead chooses one of the updates arbitrarily from the conflicting writes based on the data center’s unique identifier. While this may not capture user intent, it ensures convergence and prevents clock skew from rendering the system unwritable if a clock happens to skew forward in time. These updates are performed per field in the document and when update-update conflicts occur where the system cannot determine an answer that reflects the true user intent.

When more information is available, the system can do better. If we consider the case where we have a document containing multiple keys and concurrently Alice removes a field while Bob is updating that same field to a new value, we can order Bob’s update after Alice’s, under the assumption that Alice’s removal is performed given the current value of the field – therefore, Bob’s update should survive and not be removed based on Alice’s modification. Again, we can perform this arbitration on a per field basis, and try to preserve user intent. While this might seem reasonable, this also can lead to unintended consequences.

Consider the case of an address book entry as a document containing street address, state, and a boolean flag to identify whether or not the user lives in Washington state. Alice updates the document to modify the address from Washington to California and untoggles the boolean; simultaneously, Bob updates the address to a different address in Washington state. Alice is located in DC2, Bob in DC1. In this case, we run into trouble using any methods when we try to converge.

Using data center identifiers and per field convergence, if we send only the modified fields to the system for convergence, we result in a checked boolean and an address in California. This is clearly wrong.

If we model our boolean as a register, we will not converge correctly, unless Bob sends an update for the boolean as well.

If we model our boolean using semantics where booleans should always merge from a false value to a true value, and model Bob’s change as an update of both the state (to its current value) and new address, this merges correctly, but only because Alice’s updates originated in DC2. If Alice was located in DC1, it would not have.

If Bob is located in DC1, we merge correctly.

The difficulty here is that the value of the boolean depends on the value of the address: this is an invariant that exists within the document itself, and converging documents based only on the fields in the document is not sufficient to perform convergence while preserving the invariant. Therefore, we need another mechanism to ensure that we can preserve these invariants under conflict resolution.

In RICE, these groups of updates to a document can be performed atomically. Atomic groups of updates ensure that convergence is performed per document where all of the updates for a given document are ordered together when performing conflict resolution – thereby helping the developer to ensure invariants. However, these atomic updates are only good for ensuring one-way invariants: where a given value depends on another value in the document; they are insufficient for preserving two-way invariants where the values of two fields of a document are interdependent and conditional on one another. For that, we refer you to the next post in our series on transactions.

Macrometa's CRDT Edge Database - Implementing Coordination Free Convergence of Data for Edge Computing

If you think about making your own twitter-ish app, you will soon stumble upon a few super interesting challenges.

A twitter style app will allow anyone to sign up, tweet, and then to follow people and see their tweets.

The feature Signing Up requires some kind of database to store the users. If this is just for your closest friends or a few thousand tweeters, SQLite will be fine.

Of course, you also need some back-end to handle the sign-ups and talk with the database. You could use a LAMP stack, Ruby on Rails or roller-skates, or Node.js or whatever. It doesn't really matter. Unless you want your app to be scalable, then you may want to make the back-end a stateless micro-service that talks with a stateful database like MariaDB, Postgres or MongoDB - each of which can be scaled up if you put some effort into it.

Once signed up, your users must have a way to become followers. In other words; they need a way to discover other users. That means that our app must have some search functionality. So - when someone signs up they could search for their friends, or people with similar interest (or opposite interests for the evangelists). What about other local users? How cool would it be to be able to find and follow people in your own village or neighborhood? Or in the same coffee shop or nightclub?

Since all internet connected devices have some kind of location awareness (so that advertisers can feed us the most relevant ads), how about exploiting that to locate nearby people? Getting your location from your device is trivial. But a simple database like SQLite has no geo-location awareness. So we need a database that knows about geo-locations to implement that feature. A proper geo-location aware database should allow us to directly query for the n closest entities of any geo-location (latitude, longitude) in the world.

Another way to find followers is to see who else is following the people we follow. That can be implemented using a traditional SQL database - to a certain extent. If we want to walk several levels into the follower network we are really in the realm of a graph database. A graph database tracks relations between objects, and allow us to make simple and fast queries to traverse the graph to fetch whatever objects we desire.

But the most important feature of them all is to make sure all of our followers get our tweets immediately. Not like Google Chat or Skype, that may take days to push a message notification. When we have something on our tongue, we want people to get it immediately so that they can react to it (read: Like it!).

The traditional way to implement a thing like this, using web technology, is polling. The client polls every second to see if any of the people it follows has sent any new messages. That's fine on a residential Internet connection with a PC, but not so much on a phone, where you pay for the traffic and where traffic drains the battery - not to mention the back-end resources. If you make your twitter for your friends, you just need one micro-service instance with Node.js, 5 with Ruby or 42 with Java, to serve 20 friends. On the other hand, if your app goes viral you have a real challenge. One million users, each polling each second, will be rather expensive to handle as you will need thousands of micro-services. A better approach is to push the messages to the device. That can be accomplished with grpc, push notifications, or just plain old web-sockets.

In a micro-service architecture, one way to handle pushing messages in real time to the clients is a pub-sub service. Think of it as queues or persistent streams, where each user has a personal queue or a stream she subscribes to. When someone publishes a message, the back-end copies the message to the queue or stream for each of her subscribers. As soon as the message is pushed to a queue or stream, the pub-sub service checks if the subscriber is connected, and immediately sends the message to those who are. Those who are off-line will get the messages as soon as they connect to their queue or stream.

So for our twitterish app, we need a trivial database for the users, a geo-location database, a graph database, and some kind of pub-sub service for the notifications. That's 4 stateful technologies that our stateless micro-service has to deal with (and we have to read up on, comprehend, deploy and interact with from the micro-service).

If the app goes viral, we also have to scale it from tens to thousands to millions of users. That's hard, especially if our users are spread all around the world. If we deploy our gang of 4 in Central Europe, users from USA, India, China, Australia will get latency and connectivity issues. (Even Netflix and HBO are unable to serve a global user base in a timely manner. I often switch off my TV before either of them starts to play the movie I want to watch). If your app is slow, you lose users. It's that simple.

So what if we could just make up some imaginary fairy technology that our stateless micro-service could connect to that would have low latency, scale to any size, and deal with the gang of 4 stateful technologies we require - from anywhere in the world? Then we could deploy our micro-services in Australia, in Asia, in Europe in USA, in UK - anywhere large clusters of users subscribed - and provide all of them with wonderful, reliable, low latency access to the latest fake news and trolling on the net? (Let's be realistic - if your app goes viral, all of the bad people will be there, but so will all of the good people, and we are the majority!)

Well - good news. The "fairy" technology is here!

I have spent the last few months working on it, after joining a small team of dedicated, talented people in India, Europe and USA. When it went online with an early free preview a few weeks ago, I wanted to spend a weekend or two making an app that consumed all of our main features - the gang of 4 - to get an idea about how easy or difficult it would be to actually use our service as a software architect and developer. So I made a simple app in Qt/QML that I called 

 (I'm more a dog person then a bird person) using our document database (a basic key/value store where the values are Json documents) for users and messages (barks), the graph database (edges attached to Json documents) for relations (followers), the Geo-location index type (for the Json documents) for Geo-location, and streams over websockets to receive new messages in real time.

It took me 46 hours and 39 minutes to do this. About 1/3 of the time was spent on figuring out how to implement all the client functionality in QML (usually I use C++ for all the logic - but I wanted to learn something new - and if someone wants to use my code as an example, JavaScript is much simpler to understand for most people than C++), 1/3 was spent working around bugs, figuring out undocumented features and tuning queries (don't worry - our documentation team and developers got a steady stream of feedback - so all of that is/will be fixed), and 1/3 was spent on implementing the thing.

I ended up with a very simple stateless micro-service back-end in Node.js dealing with subscriptions to the service, login, distributing barks (what Twitter refers to as tweets) and a few other things.

I could have distributed the barks in the client, but if the client is shut off or loses connectivity in the process, some subscribers may not see the barks - which will be devastating for the barker, so it was better architecture to put it in the back-end .

For me this was an expedition into unknown technological territory. I have not used geo-locations or graph databases in the past, and I have never designed a stateful, distributed application that's supposed to scale to any size just by deploying more back-end resources.

Archive