Sat Aug 18, 2018 2:34 pm
Login Register Lost Password? Contact Us


Roxie needs deployed queries to remain on Thor

Post questions or comments on how best to manage your big data problem

Sat Sep 24, 2011 1:49 pm Change Time Zone

built indexes in Thor, deployed the query to Roxie, then deleted the index from Thor. Unfortunately, this caused the Roxie query to stop working even though when
I deployed the query to Roxie the data got copied into the Roxie data directory and was still there after I had deleted it from Thor.

To re-create using _Certification:
1.
Run on Thor: _Certification.BuidDataFiles
2.
Run on Thor: _Certification.build_index
3.
Compile on Roxie, then Publish: _Certification.ReadIndexService
4.
Verify that the Roxie service works
5.
Log into the VM and verify that the index (full_test_distributed_index) exists in both:
a.
/var/lib/HPCCSystems/hpcc-data/thor/certification
b.
/var/lib/HPCCSystems/hpcc-data/roxie/certification
6.
Using ESP, browse to the files in certification and delete them.
a.
This step won’t actually complete while Roxie is up – it’s holding a lock on the files.
b.
Rebooting the VM releases the lock, and the index gets deleted.
7.
Reboot the VM.
8.
Log into the VM and verify that the full_test_distributed_index files now exist only in: /var/lib/HPCCSystems/hpcc-data/roxie/certification
9.
The Roxie service now fails with error:
a.
Query readindexservice.1 is suspended because Could not resolve filename certification::full_test_distributed_index

Since the index was copied to Roxie, and still exists in roxie’s data directory, the query should still work. (Right now, it appears data has to be double-stored in order for Roxie to work properly in the OSS VM.)
thildebrant
 
Posts: 18
Joined: Mon Apr 11, 2011 4:39 pm

Wed Sep 28, 2011 12:07 pm Change Time Zone

Todd,

Deleting a logical file deletes all copies of the file, including Roxie's and Thor's. Please keep in mind that Dali is aware of both, Thor and Roxie.

It probably wasn't obvious, but the platform was preventing the removal of the files in Roxie by keeping a lock on them. By rebooting the VM, you effectively circumvented that protection and allowed for the deletion process to complete.

In any case, there are two factors that may make this slightly confusing:

    1. If you were used to the old behavior (Roxie creating an independent copy of the data), the current platform can do this but this is not the default.
    2. The delete process should have probably failed right away rather than just block on a lock until the lock was released, and then completing the removal of the file. It might be a good idea to change the current behavior to fail immediately if the user tries to delete a file for which there is a query deployed to Roxie.

Flavio
flavio
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 73
Joined: Wed Apr 27, 2011 8:59 pm

Wed Sep 28, 2011 12:19 pm Change Time Zone

Flavio,
Thank you for your reply.
What is the method to enable Roxie to maintain independent copies of the data?
And can you elaborate on the decision to change to only one logical file? What are the benefits?

Thank you,
Todd
thildebrant
 
Posts: 18
Joined: Mon Apr 11, 2011 4:39 pm

Wed Sep 28, 2011 3:06 pm Change Time Zone

Todd,

To maintain independent copies of the data, you can put Roxie on a separate Dali. This may be an EE only feature, though.

Having said this, you should be able to delete the redundant copy of the data from Thor without affecting the copy of the data in Roxie, but that's not the same as deleting the logical file. I need to check to see if the former is currently implemented and how.

The reason behind the change in Dali's behavior with respect to Roxie was to avoid having these "orphan" files, and also to provide consistency across the environment. This should ensure a more seamless user experience (tool-wise, for example).

Flavio
flavio
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 73
Joined: Wed Apr 27, 2011 8:59 pm


Return to Managing Big Data

Who is online

Users browsing this forum: No registered users and 1 guest

cron