Is there any way to increase disk space in the VM Image?

[dsun]

Hi,

Since it will produce much more data(output and temp) after running the ecl code, there is no enough space in the disk, is it possible that we can increase the disk space?

Thanks,
Dongliang

[bforeman]

Hi,

The HPCC VM Edition runs on a single node, has a limit of 20 GB in it workspace, and doesn't support custom configurations.

That said, you can still use the HPCC VM Image to deeply evaluate ECL and ETL on a single node. My advice is that if you are running low on disk space, try filtering the output to a smaller recordset result. Anything larger, you should probably build a test cluster of 3 or 4 nodes. If you have an Amazon Web Services account, you can test larger datasets using the one-click THOR cluster for a nominal fee.

Regards,

Bob

[BrianB644]

If the HPCC Systems VM isn't configured appropriately for your needs, I think the first thing to consider would be to rebuild a VM or system that was better suited to your needs. It seems to be fairly straight-forward to do so, and the resulting system will probably save time, effort, and frustration in the long run.

That being said, I recently found myself in a situation where I had a short-term need and circumstances conspired so that the desktops we needed to run on were isolated and couldn't get access to the resources required to create a reconfigured VM.

In the end what was easy and safe to configure was the following. I loaded the VM-ware tools in the VM, and defined a shared folder on the VM Host System with sub-directories for my dropzone, sprayed files, and result files. Within the VM Guest ... I verified the shared folder was working correctly, and I used symbolic links (man 'ln -s') to link the directories into the HPCC file system. Using this configuration, I was able to: 1) copy files into the shared folder on the host and have them "appear" on the landing zone (no VM disk space is used), 2) spray the files normally and have the results written to the shared folder on the host (no VM disk space used), and 3) have the result files written to the shared folder on the host (no VM disk space used). Spill and temporary files still used the VM disk ... which I imagined to be faster than writing the files externally.

Using this configuration we were able to write and test quite a bit of code before our full system was available. We also did some full file data analysis, and join trials. When our full hardware was configured, we were able to hit the ground running.

FYI - in our case, we also updated the VM to add a CPU and bumped memory a bit to better leverage the resources we had available.

Brian