Skip to main content

Quickstart

Overview

Today’s applications are required to be highly responsive and always online. They need to be deployed in datacenters closer to their users and can access data instantly across the globe.

Macrometa global data network (GDN) is a fully managed realtime materialzed view engine that provides access to data instantly to Apps & APIs in a simple & single interface.

This article is an introduction to working with documents in GDN with pyC8 and jsC8 drivers.

In the drivers, a document is a dictionary/object that is JSON serializable with the following properties:

  • Contains the _key field, which identifies the document uniquely within a specific collection.
  • Contains the _id field (also called the handle), which identifies the document uniquely across all collections within a fabric. This ID is a combination of the collection name and the document key using the format {collection}/{key} (see example below).
  • Contains the _rev field. GDN supports MVCC (Multiple Version Concurrency Control) and is capable of storing each document in multiple revisions. Latest revision of a document is indicated by this field. The field is populated by GDN and is not required as input unless you want to validate a document against its current revision.

Here is an example of a valid document:

    {
'_id': 'students/bruce',
'_key': 'bruce',
'_rev': '_Wm3dzEi--_',
'first_name': 'Bruce',
'last_name': 'Wayne',
'address': {
'street' : '1007 Mountain Dr.',
'city': 'Gotham',
'state': 'NJ'
},
'is_rich': True,
'friends': ['robin', 'gordon']
}

Edge documents (edges) are similar to standard documents but with two additional required fields _from and _to. Values of these fields must be the handles of "from" and "to" vertex documents linked by the edge document in question. Here is an example of a valid edge document:

    {
'_id': 'friends/001',
'_key': '001',
'_rev': '_Wm3dyle--_',
'_from': 'students/john',
'_to': 'students/jane',
'closeness': 9.5
}

A Graph consists of vertices and edges. Edges are stored as documents in edge collections. A vertex can be a document of a document collection or of an edge collection (so edges can be used as vertices). Which collections are used within a named graph is defined via edge definitions. A named graph can contain more than one edge definition, at least one is needed. Graphs allow you to structure your models in line with your domain and group them logically in collections and giving you the power to query them in the same graph queries.

In SQL you commonly have the construct of a relation table to store n:m relations between two data tables. An edge collection is somewhat similar to these relation tables. Vertex collections resemble the data tables with the objects to connect.

While simple graph queries with fixed number of hops via the relation table may be doable in SQL with several nested joins, graph databases can handle an arbitrary number of these hops over edge collections - this is called traversal. Also edges in one edge collection may point to several vertex collections. Its common to have attributes attached to edges, i.e. a label naming this interconnection.

Edges have a direction, with their relations _from and _to pointing from one document to another document stored in vertex collections. In queries you can define in which directions the edge relations may be followed i.e.,

  • OUTBOUND: _from_to
  • INBOUND: _from_to
  • ANY: _from_to.
note

If you are new to Macrometa GDN, we strongly recommend reading Essentials of Macrometa GDN.

Pre-requisite

Let's assume your

Driver download

    pyC8 requires Python 3.5+. Python 3.6 or higher is recommended

To install pyC8, simply run

$ pip3 install pyC8

or, if you prefer to use conda:

conda install -c conda-forge pyC8

or pipenv:

pipenv install --pre pyC8

Once the installation process is finished, you can begin developing applications in Python.

Connect to GDN

The first step in using GDN is to establish a connection to a local region. When this code executes, it initializes the server connection to the region URL you sepcified.

    print("--- Connecting to C8")
# Simple Way
client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')

# To use advanced options
client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443)

Get GeoFabric Details

To get details of fabric,

    from c8 import C8Client
client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')
print("Get geo fabric details...")
print(client.get_fabric_details())

Create Collection

We can now create collection in the fabric. To do this, first you connect to fabric and then create a collection called employees.

The below example shows the steps.

    client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')
client.create_collection(name='employees')

Create Edge Collection

An edge collection contains edge documents and shares its namespace with all other types of collections. You can manage edge documents via standard collection API wrappers, but using edge collection API wrappers provides additional safeguards:

  • All modifications are executed in transactions.
  • Edge documents are checked against the edge definitions on insert.
    # Simple Approach
client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')
client.create_collection(name='employees')
if client.has_graph('school'):
print("Graph exists")
else:
print("Create: ", client.create_graph(graph_name='school'))

You can manage edges via graph API wrappers also, but you must use document IDs instead of keys where applicable.

Insert Documents

Let's insert documents to the employees collection as shown below.

    client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')
client.insert_document(collection_name='employees', document={'_key':'Jean', 'firstname': 'Jean', 'lastname':'Picard', 'email':'[email protected]'})

docs = [
{'_kefabricy':'James', 'firstname': 'James', 'lastname':'Kirk', 'email':'[email protected]'},
{'_kefabricy': 'Han', 'firstname': 'Han', 'lastname':'Solo', 'email':'[email protected]'},
{'_kefabricy': 'Bruce', 'firstname': 'Bruce', 'lastname':'Wayne', 'email':'[email protected]'}
]

client.insert_document(collection_name='employees', document=docs)

Create Graph

A graph consists of vertices and edges. Vertices are stored as documents in vertex collections and edges stored as documents in edge collections. The collections used in a graph and their relations are specified with edge definitions.

    from c8 import C8Client

client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')


# List existing graphs in the fabric.
client.get_graphs()

# Create a new graph named "school" if it does not already exist.
# This returns an API wrapper for "school" graph.
iif client.has_graph('school'):
school = client.graph('school')
else:
school = client.create_graph('school')

Graph Traversals

A graph consists of vertices and edges. Vertices are stored as documents in vertex collections and edges stored as documents in edge collections. The collections used in a graph and their relations are specified with edge definitions.

    from c8 import C8Client
import pprint

# Variables - Queries
global_url = "gdn.paas.macrometa.io"
email = "[email protected]"
password = "xxxxxx"
geo_fabric = "_system"
collection_people = "CDRpeople"
collection_calls = "CDRcalls"
collection_cellsites = "CDRcellsites"
collection_graph = "CDRgraphdocs"
read_people = "FOR person IN CDRpeople RETURN person"
read_calls = "FOR call IN CDRcalls RETURN call"
person = "Lou Feaveer"
graph_traversal1 = "FOR c IN CDRpeople FILTER c.full_name == \"{}\" FOR v IN 1..1 INBOUND c CDRcalls RETURN v".format(person)
graph_traversal2 = "FOR c IN CDRpeople FILTER c.full_name == \"{}\" FOR v IN 1..1 OUTBOUND c CDRcalls RETURN v".format(person)

pp = pprint.PrettyPrinter(indent=4)

# Initialize the C8 Data Fabric client.
# Step1: Open connection to GDN. You will be routed to closest region.
print("1. CONNECT: federation: {}, user: {}".format(global_url, email))
client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email=email, password=password,
geofabric=geo_fabric)


# Step2: Create collections if not exists
print("2a. CREATE_PEOPLE_VERTEX_COLLECTION: region: {}, collection: {}".format(global_url, collection_people))
if client.has_collection(collection_people):
peopleCol = client.collection(collection_people)
else:
peopleCol = client.create_collection(collection_people)

print("2b. CREATE_CALLS_EDGE_COLLECTION: region: {}, collection: {}".format(global_url, collection_calls))
if client.has_collection(collection_calls):
callsCol = client.collection(collection_calls)
else:
callsCol = client.create_collection(collection_calls, edge=True)

# Step3: Insert data into collections.
print("3a. INSERT_PEOPLE_DATA: region: {}, collection: {}".format(global_url, collection_people))

# insert documents into the collection
docs = [
{
"full_name": "Kiel Dummer",
"first_name": "Kiel",
"last_name": "Dummer",
"city": "Burbank",
"state": "CA",
"address": "40317 5th Crossing",
"calling_nbr": "757-697-9065",
"_key": "757-697-9065"
},
{
"full_name": "Pernell Winspare",
"first_name": "Pernell",
"last_name": "Winspare",
"city": "San Diego",
"state": "CA",
"address": "596 Packers Pass",
"calling_nbr": "718-208-8096",
"_key": "718-208-8096"
},
{
"full_name": "Ava Kermath",
"first_name": "Ava",
"last_name": "Kermath",
"city": "Berkeley",
"state": "CA",
"address": "2 Doe Crossing Junction",
"calling_nbr": "765-623-5328",
"_key": "765-623-5328"
},
{
"full_name": "Tremain McGrah",
"first_name": "Tremain",
"last_name": "McGrah",
"city": "Torrance",
"state": "CA",
"address": "079 Russell Street",
"calling_nbr": "859-783-3227",
"_key": "859-783-3227"
},
{
"full_name": "Vidovik Boddam",
"first_name": "Vidovik",
"last_name": "Boddam",
"city": "Los Angeles",
"state": "CA",
"address": "3 Brentwood Crossing",
"calling_nbr": "703-265-1313",
"_key": "703-265-1313"
},
{
"full_name": "Oralie Goward",
"first_name": "Oralie",
"last_name": "Goward",
"city": "Los Angeles",
"state": "CA",
"address": "922 Columbus Park",
"calling_nbr": "617-815-8610",
"_key": "617-815-8610"
},
{
"full_name": "Lou Feaveer",
"first_name": "Lou",
"last_name": "Feaveer",
"city": "San Jose",
"state": "CA",
"address": "55223 Hooker Crossing",
"calling_nbr": "716-463-8993",
"_key": "716-463-8993"
},
{
"full_name": "Peria King",
"first_name": "Peria",
"last_name": "King",
"city": "Stockton",
"state": "CA",
"address": "8 Troy Plaza",
"calling_nbr": "713-707-8699",
"_key": "713-707-8699"
}
]
peopleCol.insert_many(docs)

print("3a. INSERT_CALL_RECORDS_DATA: region: {}, collection: {}".format(global_url, collection_calls))
docs = [
{
"calling_nbr": "757-697-9065",
"called_nbr": "716-463-8993",
"_from": "CDRpeople/757-697-9065",
"_to": "CDRpeople/716-463-8993",
"call_date": "1/4/2020",
"call_time": "13:33",
"call_duration": 30,
"cell_site": 4044703906
},
{
"calling_nbr": "716-463-8993",
"called_nbr": "713-707-8699",
"_from": "CDRpeople/716-463-8993",
"_to": "CDRpeople/713-707-8699",
"call_date": "1/28/2020",
"call_time": "3:02",
"call_duration": 18,
"cell_site": 2289973823
},
{
"calling_nbr": "765-623-5328",
"called_nbr": "713-707-8699",
"_from": "CDRpeople/765-623-5328",
"_to": "CDRpeople/713-707-8699",
"call_date": "1/28/2020",
"call_time": "3:02",
"call_duration": 18,
"cell_site": 2289973823
}
]
callsCol.insert_many(docs)

#Step4: Create a graph
print("4. CREATE_GRAPH...CDRgraph")
graph = client.create_graph(collection_graph)
register = graph.create_edge_definition(
edge_collection=collection_calls,
from_vertex_collections=[collection_people],
to_vertex_collections=[collection_people]
)

# Step5: Read Data
print("5a. GRAPH_TRAVERSAL: Find outbound calls TO: {}".format(person))
cursor = client.execute_query(graph_traversal1)
docs = [document for document in cursor]
pp.pprint(docs)
print("5b. GRAPH_TRAVERSAL: Find inbound calls FROM: {}".format(person))
cursor = client.execute_query(graph_traversal2)
docs = [document for document in cursor]
pp.pprint(docs)

# Step6: Delete Data
print("6. DELETE_DATA...")
#callsCol.truncate()
#peopleCol.truncate()
client.delete_graph(collection_graph, drop_collections=False)

Outbound Traversal

    # Step4: Read Data
print("4a. GRAPH_TRAVERSAL: Find outbound calls TO: {}".format(person))
cursor = client.execute_query(graph_traversal1)
docs = [document for document in cursor]
pp.pprint(docs)

Inbound Traversal

    print("4b. GRAPH_TRAVERSAL: Find inbound calls FROM: {}".format(person))
cursor = client.execute_query(graph_traversal2)
docs = [document for document in cursor]
pp.pprint(docs)

Delete Graph

    from c8 import C8Client

# Initialize the C8 Data Fabric client.
client = C8Client(protocol='https', host='gdn.paas.macrometa.io', port=443,
email='[email protected]', password="xxxxxx",
geofabric='_system')

# This returns an API wrapper for "school" graph and deletes the graph
client.delete_graph('school')