Tue Aug 21, 2018 2:50 am
Login Register Lost Password? Contact Us


examples of Roxie based apps?

Topics specific to using ECL from a Data Analyst standpoint

Thu Jul 28, 2011 2:14 pm Change Time Zone

Not sure where to put this post...

We have legacy applications that build and execute dynamic SQL against a reporting database. I think Roxie could replace the reporting database but the implementation seems a little foreign to me, coming from the relational world. Can someone list a few Roxie implementation scenarios/examples to illustrate how it typically meshes with reporting apps?

Could Roxie handle open ended decision support apps that traditionally use a data warehouse or OLAP? The data questions are unknown at design-time which seems to conflict with Roxie - this could be a misunderstanding on my part though.

Could it be used as a dynamic computation engine for tiny datasets where all of the necessary data (and maybe even ECL) are input arguments?

Thanks in advance!
aintnomyth
 
Posts: 86
Joined: Wed Jul 13, 2011 7:40 pm

Thu Jul 28, 2011 2:52 pm Change Time Zone

Gurgle - that is one of those open-ended questions I always answered badly at college! Let me try to narrow things down a little and I may be able to help:

1) Roxie (at least used properly) expects you data -model- to be fairly fixed. Obviously you can change it day-to-day but not minute-to-minute (unless you are in the experimental stage)

2) One use for which Roxie is extremely well suited is running a few (hundreds) pre-compiled queries billions of times a day on different smallish (megabytes) datasets. Think: use an index or two, retrieve a handful of data, smack it around, maybe do a couple more index fetches, smack it around some more and hand it back.

3) Another use which -appears- the same in when it is possible to pre-build some aggregated statistics on large sets of data (gigabytes/terabytes) and then using those aggregated or partially aggregated results it is possible to -appear- to be running global reports billions of times a day.

4) 'Roxie on demand' - The ECL system has the ability to 'compile and execute' a roxie query as a one-shot, one time deal via the ECL server. You can think of this as very similar to SQL <semantics, not syntax>. The ECL server will take your ECL, optimize it, compile it, deploy it and get back the results. The roxie itself executes at full speed; but there is overhead for the compile/optimize etc on the ECL server. This allows you to query ANYTHING you wish that the indexes you build support. Speed depends upon the number of ECL servers you have - but you now need to be thinking of low-seconds latency rather than tens of milliseconds.

5)In memory roxie with an 'interpreter' front end: this is the way we did our Pentaho demo (an OLAP tool). We use PRELOAD to get the data (compressed) into memory in roxie. We then wrote ECL that 'interpreted' a range of queries that were passed to it - this range supported all of the query types required to support an OLAP front end.

6) In my 'used properly' quote earlier you might have noticed a slice edge - there is another way to use Roxie and Thor that is extremely flexible IF you have ample resources for the volume of data you are processing. Your 'tiny' dataset would be an example of this. Essentially you can use the ECL Parse statement to process a 'language' of your design. You would typically represent the data using some completely generic data model (such as file/property/value triples) and you could process it according to the 'commands' in the language typed in. (The GRAPH statement is excellent for this - I have a simple example of using it here:http://www.dabhand.org/ECL/construct_a_simple_bible_searchII.htm)

As I said, I hope that helps - but I can probably give a more specific answer with more information to go on ....

David
dabayliss
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 109
Joined: Fri Apr 29, 2011 1:35 pm

Fri Jul 29, 2011 8:46 pm Change Time Zone

David, thanks for the information. That gives me a lot to work with.

You mentioned the Pentaho demo, is Roxie actually feeding their BI demo at demo.pentaho.com? Their demo looks impressive, any idea how much data is behind that?
aintnomyth
 
Posts: 86
Joined: Wed Jul 13, 2011 7:40 pm

Fri Jul 29, 2011 10:00 pm Change Time Zone

I don't believe so - their front end can be ran off of a number of different SQL back ends (it operates through JDBC). We have done the Pentaho demo at a number of shows but to the best of my knowledge there is nothing up on the web yet for people to play with ...

David
aintnomyth wrote:David, thanks for the information. That gives me a lot to work with.

You mentioned the Pentaho demo, is Roxie actually feeding their BI demo at demo.pentaho.com? Their demo looks impressive, any idea how much data is behind that?
dabayliss
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 109
Joined: Fri Apr 29, 2011 1:35 pm


Return to ECL for Analysts

Who is online

Users browsing this forum: No registered users and 1 guest

cron