Has Macrometa Cracked The Code For Delivering Data Value?
Nov 12, 2022
Excerpts from this article were originally published in the Datanami article “Has Macrometa Cracked the Code for Global, Real-Time Data?” by Alex Woodie.
There is an old saying that time is money. Data has a time, location, concurrency, and actuation value. However, trying to combine real-time and historical data often ends in complex or expensive solutions that may not be able to keep up with the required low latency demands. In this blog, we will explore the recent Datanami article about how Macrometa’s architecture and features deliver real-time data value. The article highlights how Macrometa offers a Global Data Mesh, provides a unique use of conflict-free replicated data types (CRDT)s, and integrates the Global Data Mesh with Edge Compute and Data Protection in a “three-layer cake” within the Global Data Network (GDN).
Macrometa CEO Chetan Venkatesh kicks off talking to Datanami about a typical customer example, Cox Communications, and “how the amount of data they were sending in, the scale at which they had to build and integrate these systems made it too complex and slow” with a cloud provider. Once Cox Communications engaged Macrometa, they were able to “reduce costs by 90% and have a picture of reality within the hundreds of milliseconds of latency” vs minutes to hours delay. Let’s run through the Macrometa components that provide these real-time benefits.
A solution for big and fast data - Global Data Mesh
The issue today is that data has to make the roundtrip to a centralized data center - whether on premises or in the cloud - and this is far away from customers, systems, staff, partners, and suppliers. This delay can lower the value of data, if a customer goes to another website with a better response time, or if a logistics company doesn’t see a traffic accident notification. Chetan explains to Datanami how Macrometa addresses this problem with a Global Data Mesh, that is a distributed resource, not a centralized resource like typical cloud providers.
In Macrometa's data mesh model, data can be stored, accessed, and processed where it originates and/or where it is needed. The result is a flexible, ultra-low-latency data layer purpose-built for global, real-time, and event-driven use cases. Data is automatically replicated across the Macrometa GDN, and API requests are routed to the nearest available location. There are over 175 points of presence around the world. With Macrometa, developers have the tools to build exciting new applications and they can use the data model that makes the most sense for their application.
“The storage engine is a very small but integral part of our stack,” Chetan said to Datanami. “The real value is in the approach we have for the way we ingest data into the log, the way we collapse that log into objects in real time, and most importantly, the way we can replicate that data across hundreds of locations with transactional and consistency guarantees. That’s always been the missing piece.”
The secret sauce: CRDTs
In developing Macrometa future data volumes were a consideration - which could easily be ingesting trillions of events per second per the article. At its heart, Macrometa's architecture is a geo distributed event source with a materialized view engine and CRDT based replication. Macrometa manages changes at the atomic field level and merges changes made across the network to maintain a single version of the truth via CRDTs, showcasing the power and versatility of CRDTs as a regular JSON database and API.
The article depicts how Macrometa transforms all data changes into a CRDT operation, which is then replicated to all the Macrometa locations using a vector clock. “With a vector clock and a causal tree of changes, Macrometa ‘s system essentially serializes and provides serialization ACID-like guarantees,” as Chetan explains in the Datanami article. “This ensures consistency without having to exchange a lot of messages with different apps.” Macrometa can store, process, and serve data within milliseconds to multiple locations around the world. In contrast, the internal messaging required within distributed systems can significantly lower performance if they expand past four or five locations around the world.
Three-layered cake approach
The article describes the Macrometa GDN ”three-layered cake”: a real-time, geo-distributed storage layer, a compute layer to process data to run real-time apps close to your customers, and a data governance layer to help organizations comply with regulatory and legal standards.
Base layer to store and ingest data - Global Data Mesh
As big and fast data ebbs and flows in all directions, developers and enterprises and everyone that relies on data for decisions need a simplified way to quickly address different types of data. The Global Data Mesh has all the benefits of a NoSQL database with KV, doc, and graph stores - plus Macrometa Streams with pub/sub and messaging queues.
Middle layer bringing it together - Edge Compute
“In our platform, streams and tables are the same things,” Chetan explained to Datanami. “It’s just that streams are real time user tables. And so you can interact with your data in both real time, in-motion fashion with streams, via pub/sub, or you can query it using request-response with SQL.”
What separates Macrometa from other data storage solutions is that it brings compute closer to people and where data is generated, regardless of where that is. Query Workers lets developers create simple REST APIs on top of the data mesh.
Stream Workers enables Complex Event Processing (CEP) functions and Stream Processing workloads - in minutes instead of days or months. With Stream Workers, data could come from different sources (GDN streams, GDN Database, HTTP endpoints, Kafka, and MQTT etc.) and can be published to different sinks or stores. Stream Workers also allows you to transform data from one data type to another, aggregate streams, and clean and filter data. This CEP engine automatically creates objects like streams, collections, and even tables and remote databases.
Top layer keeping data separate and safe - Data Protection
Macrometa’s Data Protection supports geo-pinning or geo-fencing to address data sovereignty use cases. Developers and operations can easily set up localized data fabrics for regional and global data, and adjust locations with just a toggle. Macrometa is SOC 2 certified and offers user, token-based, and API keys authentication.
The icing on the cake - a complete Edge-as-a-Service platform
Chetan summarized for Datanami, “At the end of the day, the Macrometa platform essentially deliver a full database, a pub/sub system like Kafka, and a complex event processing system Flink along with compute engine so you can build a real-time data application, completely distributed using us” via the 175-site GDN.
Try it out today
With many tutorials, QuickStarts, and easy-to-follow example documentation, you can get started with Macrometa in minutes. I mean you can actually build a streaming app in 10 mins or less! Start by requesting a 30 day trial, or schedule a demo with one of our experts.
