Out-Thinking Attackers With Graphs: Why AppSecEngineer Chose Macrometa Over Cloudwatch

This is a guest post from Abhay Bhargav, the Founder and Chief Research Officer of AppSecEngineer.

“Attackers think in Graphs, but Security thinks in Lists” is a famous saying amongst security practitioners like myself. And this is very true.

John Lambert - who popularized this saying - pointed out that defenders or security teams think in terms of asset lists while attackers think of the relationships between different assets. They breach a system using one (or more) of the assets and find multiple paths to the highest value asset by navigating the “graph” of interlinked vulnerable assets. 

On the modern web, attackers look to identify and exploit vulnerabilities. Once they compromise an application(s), they pivot into internal applications and connected cloud services through chained vulnerabilities and exploits. These attack sequences lead to the eventual outcome that the attackers desire-, which could be anything from stealing sensitive information to deleting mission-critical datasets and disrupting operations.

Attackers honestly think in graphs. Their ability to form relationships between connected assets gets them the ultimate prize.

Security teams need to pivot from list-thinking to graph-thinking to defend their systems successfully. This means they must visualize the entire network as a series of security dependencies and implement controls to reduce the attack surface.

That’s precisely what we set out to do at AppSecEngineer.

About AppSecEngineer

We’re a leading EdTech Platform for Application Security. We provide labs, cyber ranges, and lessons on security-focused areas like Application Security, Cloud Security, Kubernetes Security, and more. As a result, thousands of students, from job seekers to experienced professionals, use our platform daily to upskill themselves and improve their job prospects.

Being a security training platform, we can’t help but take security seriously. One significant aspect of “taking security seriously” is to capture and enrich logs and possible security events. The idea is to capture logs and events we ingest and further transform them into data we can query and investigate as the need arises.

Our platform is serverless from the get-go as we were “born in the cloud”. We use AWS Lambda and API Gateway. We use Serverless Databases like DynamoDB and Amazon S3 for object storage. Our application and user base have grown exponentially over the past year that we’ve been in business. 

When we initially launched, we used Cloudwatch for log streaming, alerts, etc. However, we quickly realized this could have been more user-friendly and scalable. Cloudwatch is an excellent logging service, but it also has some significant downsides for our use case: 

• Its native query interface is challenging to work with, especially for several thousand functions with multiple API Gateway instances you need to query. We found this frustrating, to say the least.
• Cloudwatch comes with Log Limits. And that is fine. But, we wanted to retain logs for an indefinite period. This proved to be expensive with Cloudwatch.
• Dashboarding and Alerts are not the easiest with Cloudwatch. It becomes a significant bottleneck and management overhead to manage all of this.

The Onset of Macrometa + AppSecEngineer

Looking at the downsides of Cloudwatch, we decided to move to Macrometa. As the Chief Technologist, I was already a fan of Macrometa because of its ability to ingest multiple data types, from document to key: value to graph, etc.

Why Macrometa?

In addition, since Macrometa is serverless, it makes management overhead “not our problem,” - which we use as a basis to evaluate every solution we use. We have a small team, so “managing” a lot of resources is a NO-GO. In addition, I was interested in leveraging Macrometa’s graph capabilities to query events based on relationships. 

For example - If I want to query all the related security logs on API Gateway, Lambda Function, and User Security Activity for a given service at a given time, creating a graph makes the most sense. With this, I can get a complete set of data points for all these complex relationships without spending much time writing and optimizing SQL queries.

So we set to implementing this functionality with Macrometa.

We started to stream logs from Cloudwatch to Macrometa, thereby ensuring that we didn't have to rely on Cloudwatch at all.

We stored all the logs in Macrometa as documents (structured logs). Then, using Lambdas, we added valuable transformations in the form of additional context to the structured logs with service information and graph relationships for the dataset.

This is super-easy with Macrometa because all we have to do is just create a query worker with specific parameters that you pass. This is an API that we invoke with the suitable parameters, and the data is ingested into Macrometa. 

And tada! We were able to get up and operate very quickly. We were able to capture events quickly with zero human intervention and most importantly, with the graphing capability, we are now able to model complex relationships and identify security anomalies and events very soon on our stack.

The result? The much-needed observability into our stack allows us to out-think attackers without ever having to “manage” things.

Try out Macrometa

Like how AppSecEngineer implemented Macrometa to overcome its logging challenges with CloudWatch, you can try out Macrometa to expand its query workers and graph capabilities. Get started by requesting a 30 day trial.