What’s in your Data Store?

Brass KeyAs of 7.2.0, HPCC Systems platform users have access to a new Web service called WsStore. The service is a general purpose key-value (KV) store. A key-value store is a type of non-relational database that uses a simple key-value pair mechanism to store data in which the key serves as a unique identifier to index associated value entries. Keys and values can be anything–ranging from simple literals to complex compound objects.

The obvious question is: 

What can WsStore do for me and my project?

In general, KV data stores are used to store intermediary information, stateful data, user preferences, lookup tables, or other non-transactional information your process needs fast access to.

For example, if your ECL query requires a URL as the target of a SOAPCALL method, you might currently keep that information in a static configuration or other rigid means. This information could be stored in a WsStore key-value pair and updated in real-time, as needed. The value is then retrieved at runtime to resolve the URL to use in the SOAPCALL.

Interaction with WsStore is simple and straightforward via SOAP or REST, but we also provide native ECL, Java, and JavaScript libraries, making access possible from ECL queries, Java applications, and Web UIs via JavaScript.

How is the data stored?

The KV data is based on top level stores which must be created by administrators, below the stores we have namespaces to segregate related data, and within the namespaces there are global and user-specific buckets which house the KV pairs. In this hierarchy, multiple applications within the same business organization could target a single store MyStore and distinguish the application data based on namespaces MyApp and OtherApp.

Individual applications can then read/write KV pairs into the global bucket or user-specific buckets, depending on business needs.

WsStore Image

WsStore can contain multiple stores (this example has a single store called MyStore), within each store, related KV pair sets can be isolated by namespaces (this example has 2 namespaces — MyApp and OtherApp. Each of these namespaces has Global and user-specific buckets which contains KV pairs.

In version 7.2.0 and later, an instance of the WsStore service is deployed by default alongside the ECLWatch ESP. This default instance is pre-configured with a default store called “HPCCApps”. As mentioned before, new stores can be created by administrators of the target HPCC Systems platform instance. This version of WsStore uses a Dali back-end and is not intended to be used for high load read/write scenarios. WsStore is extendable, and can plugins can be written to target other no-SQL backend data stores.

Sample usage using Java constructs:

String connString = "http://myhpcc:8010";
String hpccUser = "myname";
String hpccPass = "mypass";

Connection connection = new Connection(connString);
connection.setCredentials(hpccUser, hpccPass);

HPCCWsStoreClient client = HPCCWsStoreClient.get(connection);
	
String storename = "MyStore";
String appname = "MyApp";

//Only create once
boolean success = client.createStore(storename, "KV Store for my app", "");

System.out.println("Setting " + storename + "." + appname + "." + "mykeyname=somevalue");
client.setValue(storename, appname, "mykeyname", "somevalue", true)); 
                                                         //Set in global bucket
String fetchedVal = client.FetchValue(storename, appname, "mykeyname", true); 
                                                         //fetch from global bucket
client.setValue(storename, appname, "isActive", "true", false));
                                                         //Set in myname's bucket
fetchedVal = client.FetchValue(storename, appname, "isActive", false); 
                                                        //fetch from myname's bucket

Useful information and related resources can be found in these locations:

ECL Module (Dan Camper) 

JavaScript interface (Gordon Smith)

Java Client (Rodrigo Pastrana)