Sun Nov 29, 2020 4:46 pm
Login Register Lost Password? Contact Us


Listing Files Details from Data Store

Post questions specific to installation or configuration for the HPCC Systems platform

Tue May 19, 2020 9:41 pm Change Time Zone

Hi guys,

I need to develop a tool to keep tracking the existence of files in OS level and also in DFU. In order to list files from data store Im using the dfuplus command as following:

Code: Select all
dfuplus action=list name=* server=http://<ecl_watch:8010> username=<myuser> password=<mypass>


The command executes fine, however, in my configuration I have four cluster being managed by the same ECL Watch and I would like to gather more details of these files, at least, in which cluster they are stored. Does somebody know a command or another parameter to get this kind of information?

Thanks in advance.

Att.
Artur Baruchi
abaruchi
 
Posts: 17
Joined: Thu Apr 18, 2019 4:50 pm

Wed May 20, 2020 12:44 pm Change Time Zone

Artur,

You can use ECL code to do this, too. The Standard Library has a GetLogicalFileAttribute() function that gets all the attributes of any file. The one you're interested in is this:
Code: Select all
STD.File.GetLogicalFileAttribute(filename,'clusterName')

Here's a function I wrote using that function to return all the attributes of any set of files:
Code: Select all
IMPORT STD;
GetLogicalFileAllAttributes(SET OF STRING files) := FUNCTION
  rec := RECORD
      STRING filename;
      STRING recordSize;
      STRING recordCount;
      STRING size;
      STRING clusterName;
      STRING directory;
      STRING numparts;
      STRING owner;
      STRING description;
      STRING partmask;
      STRING name;
      STRING modified;
      STRING protected;
      STRING format;
      STRING job;
      STRING checkSum;
      STRING kind;
      STRING csvSeparate;
      STRING csvTerminate;
      STRING csvEscape;
      STRING headerLength;
      STRING footerLength;
      STRING rowtag;
      STRING workunit;
      STRING accessed;
      STRING expireDays;
      STRING maxRecordSize;
      STRING csvQuote;
      STRING blockCompressed;
      STRING compressedSize;
      STRING fileCrc;
      STRING formatCrc;
      STRING ECL;
   END;

  ds := DATASET(files,{STRING file});

  RetDS := NOTHOR(
    PROJECT(ds,TRANSFORM(rec,
      ThisFile := '~' + LEFT.file;
      GetAttr(STRING attr) :=
        STD.File.GetLogicalFileAttribute(ThisFile,
                                         attr);
      SELF.filename := LEFT.file;
      SELF.recordSize := GetAttr('recordSize');
      SELF.recordCount := GetAttr('recordCount');
      SELF.size := GetAttr('size');
      SELF.clusterName := GetAttr('clusterName');
      SELF.directory := GetAttr('directory');
      SELF.numparts := GetAttr('numparts');
      SELF.owner := GetAttr('owner');
      SELF.description := GetAttr('description');
      SELF.partmask := GetAttr('partmask');
      SELF.name := GetAttr('name');
      SELF.modified := GetAttr('modified');
      SELF.protected := GetAttr('protected');
      SELF.format := GetAttr('format');
      SELF.job := GetAttr('job');
      SELF.checkSum := GetAttr('checkSum');
      SELF.kind := GetAttr('kind');
      SELF.csvSeparate := GetAttr('csvSeparate');
      SELF.csvTerminate := GetAttr('csvTerminate');
      SELF.csvEscape := GetAttr('csvEscape');
      SELF.headerLength := GetAttr('headerLength');
      SELF.footerLength := GetAttr('footerLength');
      SELF.rowtag := GetAttr('rowtag');
      SELF.workunit := GetAttr('workunit');
      SELF.accessed := GetAttr('accessed');
      SELF.expireDays := GetAttr('expireDays');
      SELF.maxRecordSize := GetAttr('maxRecordSize');
      SELF.csvQuote := GetAttr('csvQuote');
      SELF.blockCompressed := GetAttr('blockCompressed');
      SELF.compressedSize := GetAttr('compressedSize');
      SELF.fileCrc := GetAttr('fileCrc');
      SELF.formatCrc := GetAttr('formatCrc');
      SELF.ECL := GetAttr('ECL'))));
  RETURN RetDS;
END;   

filelist := NOTHOR(STD.File.LogicalFileList());
// MyFiles := filelist(owner='rtaylor' and name[1..5]='class');
MyFiles := filelist(Std.Str.find(name,'lookupcsz',1)<>0);

GetLogicalFileAllAttributes(SET(MyFiles,name));

I'm using NOTHOR because this is only working with the DFU's metadata so there's no need to run on Thor. I'm also using the STD.File.LogicalFileList() function to get all the filenames, then filtering that result to get just the files I'm interested in.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1568
Joined: Wed Oct 26, 2011 7:40 pm


Return to Installation

Who is online

Users browsing this forum: No registered users and 2 guests

cron