Fri Nov 16, 2018 8:21 pm
Login Register Lost Password? Contact Us


Expanding or moving dropzone location volume

Questions or comments related to Cloud Computing and the HPCC Systems Instant Cloud for AWS

Thu Apr 30, 2015 3:45 pm Change Time Zone

I'm have a difficult time figuring out how to increase the dropzone folder capacity of the instance which is setup to be the drop zone location.

I'm able to attach a new volume to the mnt partition, but not the device where where the dropzone is currently pointing to: xvda1

So if I could just point the dropzone location to /mnt or attach a larger volume to the xvda1 device, I think I'd be good to go.

Though when I spray the dropped file and when I run the job and it creates temp files, I'm probably going to need to increase disk space available for that as well. Is there an easy way to bump up the storage space where it needs to be increased? I'm trying to process an input file that is 45 GB XML. It would be handy if the cluster create command asked for the disk space needed up front in addition to the number of nodes.
dnordahl
 
Posts: 14
Joined: Fri Apr 24, 2015 6:11 pm

Thu Apr 30, 2015 5:53 pm Change Time Zone

Another aspect that is adding to the difficulty of figuring out how to do this is that all the info I find online about re-sizing a partition involves shutting down a node first. But that option is not available for my individual instances and the only option on the cluster status screen is terminate.
dnordahl
 
Posts: 14
Joined: Fri Apr 24, 2015 6:11 pm

Thu Apr 30, 2015 6:23 pm Change Time Zone

It sounds like you have the 45GB file in the directory, /mnt. So, you should be able to mount that directory onto the dropzone.

Is the file in an S3 bucket?
tlhumphrey2
 
Posts: 256
Joined: Mon May 07, 2012 6:23 pm

Thu Apr 30, 2015 6:24 pm Change Time Zone

Yes, its on S3. I've figured out how to use the s3cmd tool to pull it over, but I'm running out of disk space. How to I mount the /mnt directory to the dropzone?
dnordahl
 
Posts: 14
Joined: Fri Apr 24, 2015 6:11 pm

Fri May 01, 2015 8:51 pm Change Time Zone

Code: Select all
# Put the directory s3cache someplace where there is a lot of disk space. Here
# I put it under /var/lib/HPCCSystems because there is a lot of space there.
sudo mkdir /var/lib/HPCCSystems/s3cache
sudo chmod 777 /var/lib/HPCCSystems/s3cache
sudo chmod 777 /var/lib/HPCCSystems/mydropzone

# Uncommenting 'user_allow_other' in /etc/fuse.conf so 'allow_other' works in f3fs
sudo sed "s/# *user_allow_other/user_allow_other/" /etc/fuse.conf > t
sudo mv t /etc/fuse.conf

# Do mount onto /var/lib/HPCCSystems/mydropzone. Be sure to put your bucket name
#  in bucket_name. Plus, make sure you have the environment variables: S3_ACCESS_KEY
# and S3_SECRET_KEY with yours.
bucket_name=<your s3 bucket name>
s3fs -o rw,allow_other,use_cache=/var/lib/HPCCSystems/s3cache,uid=33,gid=33 \
$bucket_name /var/lib/HPCCSystems/mydropzone
tlhumphrey2
 
Posts: 256
Joined: Mon May 07, 2012 6:23 pm

Mon May 04, 2015 1:06 pm Change Time Zone

I ended up transferring my large file to a new folder I created in mnt where there was about 300GB allocated by the HPCC cluster creation. Then I placed a symbolic link in the mydropzone folder to the file I uploaded in the subfolder of mnt.

Luckily the output files I needed to pull back into the drop zone after processing did not exceed the small size allocated to the partition where the mydropzone folder is located.
dnordahl
 
Posts: 14
Joined: Fri Apr 24, 2015 6:11 pm


Return to Cloud

Who is online

Users browsing this forum: No registered users and 1 guest

cron