Sun Jul 21, 2019 8:54 am
Login Register Lost Password? Contact Us


Removing Unused Roxie Indexes Issue

Questions related to node architecture, redundancy and system monitoring

Fri Jun 14, 2019 10:12 am Change Time Zone

Hi All,

I am hoping someone can help with the below.

When I check my roxie for unused indexes using :

ecl roxie unused-files myroxie

and then I take that list and attempt to use

dfuplus action=remove

to attempt to remove the indexes from the list it returned, it just sits and both won't remove the index AND the nowait appears to be ignored.

I think Roxie is locking those indexes even though the package has updated and doesn't reference them any more.

1. Is there a way to get Roxie to release them so that we can clean them out properly?

2. What does ecl roxie detach and attach do?

3. If we detach does it mean all the queries stop working? (can we detach from dali and the queries still know where all the indexes are?)

Also, there are cases where the roxie runs out of space and can't copy index parts to local. Once we clean things up we want to poke the roxie to force it to try and copy the indexes the package references. The only successful way we have managed it is to stop and start the roxie, which takes it out of commission for a while and we would like to avoid this.

1. Is there a way to trigger the roxie to try to refresh all the local indexes without deploying a new package or stop and starting roxie?

2. What does reload do? (We have tried using that to force roxie to continue to copy index parts from the thor but it doesn't seem to do anything).

Thanks in advance
amillar
 
Posts: 18
Joined: Fri Oct 16, 2015 7:32 am

Fri Jun 14, 2019 3:15 pm Change Time Zone

First, just as a sanity check, to make sure that the files are not actually in use, can you verify that the files listed by unused-files can be deleted if you do restart roxie? That would tell us that the locks really aren't needed, and help weed out the possibility that we are dealing with an issue determining which files are actually in use.

There can be a delay before roxie releases the locks are you trying to run the cleanup immediately after updating the packagemap?

Detaching roxie should release the locks, but make sure there isn't an issue with which files are being listed before using detach as a way of forcing the ability to delete them. You don't want to reattach roxie just to find out the wrong files were deleted.

When roxie is detached it can still use files it had already copied and files it's opened remotely, but it won't be able to be updated with new queries or packagemaps until it is reattached. It's basically in a frozen state until it's re-attached to dali.

Reload should try to re-resolve files and start copying files that it can. If you can verify that after making space reload doesn't seem to cause files to be copied as you would expect please open a JIRA.
anthony.fishbeck
 
Posts: 51
Joined: Wed Jan 30, 2013 10:18 pm

Mon Jun 17, 2019 10:36 am Change Time Zone

Hi Anthony,

Thanks for the quick reply its very much appreciated.

When we pull the unused file list it also won’t let us remove those files even though they are listed as unused.
amillar
 
Posts: 18
Joined: Fri Oct 16, 2015 7:32 am

Tue Jun 18, 2019 2:35 pm Change Time Zone

Hi, yes, understood. I was just trying to help determine if:

1. This is a timing issue where roxie just hasn't released the locks yet but soon will.
--This is just a question about how soon you are calling the delete after changing the packagemap or removing queries.

2. If there may be a bug in what we are reporting as unused files.
-- If after restarting roxie you still can't delete the files this may be the case.

3. If there may be an issue with roxie holding locks.
-- If you've waited a long time, but still can't delete, but restarting roxie allows you to delete the files.

In any case detaching roxie should allow you to delete. But if the problem is something like #2 then it could be dangerous. That's the reason I asked the questions.

Another possibility would be that you have other clusters (roxie or otherwise) using the same DALI, accessing the same files.

To help determine that, as another sanity check (if there are multiple clusters on the same DALI), and if you have access to the DALI box you could use the daliadmin command to show the locks on the files.

"/opt/HPCCSystems/bin/daliadmin . dalilocks"

Then look for one of the locked file paths and check the IP of the server(s) locking the file.
anthony.fishbeck
 
Posts: 51
Joined: Wed Jan 30, 2013 10:18 pm


Return to System Health

Who is online

Users browsing this forum: No registered users and 1 guest

cron