Thu Sep 20, 2018 8:31 pm
Login Register Lost Password? Contact Us


Working with a flat text file

Questions around writing code and queries

Wed May 02, 2018 3:12 pm Change Time Zone

I sprayed a file as a csv with no delimiters and no quotes. The first 2 lines would be posted in the 1st reply.

I also passed the maxRecordSize as 2147483646 (1 less than INT.MAX) because of the large record size / line.

The ECL Record structure is :
RECORD
STRING field1;
END;

I had the following questions :
1. Def : UNKNOWN char field1
Why UNKNOWN?
2.XML :
<?xml version="1.0"?>
<Table>
<Field ecltype="string" label="field1" name="field1" position="0" rawtype="-983036" size="-15" type="string"></Field>
<filename>file1</filename>
</Table>

Why is the rawtype negative?

My corresponding ECL code is as follows:

Code: Select all
dataset_name := '~benchmark::file1';
rs := RECORD
    STRING field1;
END;

histogram_dataset2 := DATASET(dataset_name, rs, THOR);

PATTERN histogramRule1 := PATTERN('(?<=_)[1]');
RULE userRatingRule1 := histogramRule1;
matchText1 := {STRING100 out3 := MATCHTEXT(userRatingRule1)};         
userRatingAnswerOne := PARSE(histogram_dataset2, field1, userRatingRule1, matchText1, BEST, MANY, NOCASE);


ecl run HistogramRating.ecl --target=thor --server=master:8010

W20180502-145535 failed
<Result>
<Exception><Code>4294967295</Code><Source>eclagent</Source><Message>System error: -1: Graph graph1[1], diskread[2]: SLAVE #1 [172.31.15.1:20100]: CFileSerialStream::get read past end of stream, CFileSerialStream::get read past end of stream - handling file: /var/lib/HPCCSystems/hpcc-data/thor/benchmark/file1._1_of_2 - caused by (-1, CFileSerialStream::get read past end of stream)</Message></Exception>
</Result>

How to resolve this error?
rsghatpa
 
Posts: 13
Joined: Wed Mar 28, 2018 5:13 pm

Wed May 02, 2018 3:12 pm Change Time Zone

Here are 2 lines of the same :

1:1488844_3,822109_5,885013_4,30878_4,823519_3,893988_3,124105_4,1248029_3,1842128_4,2238063_3,1503895_4,2207774_5,2590061_3,2442_3,543865_4,1209119_4,804919_4,1086807_3,1711859_4,372233_5,1080361_3,1245640_3,558634_4,2165002_4,1181550_3,1227322_4,427928_4,814701_5,808731_4,662870_5,337541_5,786312_3,1133214_4,1537427_4,1209954_5,2381599_3,525356_2,1910569_4,2263586_4,2421815_2,1009622_1,1481961_2,401047_4,2179073_3,1434636_3,93986_5,1308744_5,2647871_4,1905581_5,2508819_3,1578279_1,1159695_4,2588432_3,2423091_3,470232_4,2148699_2,1342007_3,466135_4,2472440_3,1283744_3,1927580_4,716874_5,4326_4,1546549_5,1493697_1,880166_5,535396_2,494609_4,1961619_5,883478_4,793564_4,1567202_2,573537_4,1972040_4,1838912_3,411705_4,2244518_5,584542_5,667730_5,2488120_5,1926776_1,38052_3,1196100_4,314933_3,1792741_2,769643_1,2477242_5,1421006_3,729846_4,1719610_2,1696031_4,1817215_4,406057_4,636262_1,1245406_4,1834590_3,593225_3,1011918_4,1665054_4,2630337_5,1155747_3,2439493_1,479924_5,530789_5,765860_4,231001_3,1493615_5,1850615_2,68959_3,147386_5,624035_5,782308_5,1116080_5,421374_5,1158759_4,1025798_3,1215397_5,2475251_4,321111_2,2162676_3,2635437_4,2389367_1,485622_5,235553_4,831869_4,99400_5,684876_4,1871179_3,1107678_5,642036_3,700890_5,2289956_5,2040859_1,1524964_3,121318_4,317050_5,2287003_5,59052_2,893742_4,1346257_3,55016_3,30245_5,743633_4,1596531_5,1125499_5,706832_4,2465337_3,2291422_1,1777406_3,1904905_4,2450433_3,1348967_2,638020_3,2217779_4,194280_1,493009_4,1567167_4,850327_5,520386_3,320540_2,1188228_2,57961_4,1113230_3,1374216_2,595778_3,209573_4,2354601_5,2563596_4,835265_4,1819474_3,1447104_3,1100940_1,143274_3,2329565_4,181592_4,936396_2,1125797_3,2283366_3,514495_4,1772176_3,1877347_4,1287892_4,255443_2,890669_4,1989766_4,2315073_4,14756_4,907623_3,991423_4,1604238_4,1027056_3,2025883_5,732936_5,563962_5,799442_4,352635_5,2537543_5,1564395_4,1655178_4,573434_4,1141189_4,383247_5,1763921_5,1943970_5,322009_3,2333817_3,2095681_2,1149588_4,2354740_5,2421360_5,496087_2,2191781_1,1694083_4,818416_3,701960_5,2090477_4,1664010_5,2583822_5,369646_5,2234063_4,259799_4,1077982_4,2631796_4,1122383_3,1508526_3,1600207_5,1283117_5,1727869_5,1522799_4,1394012_5,1558286_3,1155602_3,361066_3,1743210_5,1148389_4,2268101_4,519684_5,767518_5,122197_1,2112162_4,1073367_3,400162_5,1524343_5,741245_4,2563768_3,1406595_4,1137010_4,60343_5,225765_4,2530404_3,437881_3,1935793_1,134001_4,2607300_3,1008986_4,94565_4,828410_4,1805202_4,1922925_4,1435717_5,2277395_4,2305014_5,166041_4,2413320_4,87113_2,722591_5,2291306_1,2010770_4,255383_5,1873429_4,1647618_4,608234_5,42930_3,1462072_5,685565_5,3321_3,2554942_4,1874547_4,2269844_5,34907_3,1779903_4,2576424_4,230112_3,508727_3,1603525_3,172264_4,1182185_4,2275470_2,491531_5,1346432_4,1554712_5,1450941_5,1714116_3,2016488_4,1782762_4,1343170_5,2565752_4,435841_3,2242821_5,638824_5,2256485_1,101597_5,623036_5,1559445_5,1723381_5,1824586_4,2233105_4,682963_3,2529547_5,504620_2,1682104_4,16272_4,2491785_5,978412_5,2054145_3,2444240_3,547732_3,811790_5,31913_4,437111_4,640588_4,2625019_3,2605190_5,915_5,1430587_4,2544219_5,2603381_5,305344_1,2569099_1,2430356_4,885165_4,2380806_5,1512406_1,1774623_4,2226525_4,2537076_4,2060858_4,498469_5,68033_4,1819146_5,2088415_4,473070_5,1823641_5,1839976_2,14924_5,1852606_4,453694_5,921487_2,1022254_5,2464081_4,1228324_4,1563530_4,1181170_3,1357013_3,21722_4,288420_5,1739170_5,2584676_3,2013504_4,1245176_4,269524_3,661344_3,652324_3,2239213_3,863302_4,758850_4,1884755_2,544833_3,1562707_1,810700_5,837756_5,155164_4,493945_5,1565175_5,2005193_4,1605780_4,1294335_2,608576_4,659505_4,1604707_4,2630797_5,402266_5,752642_3,1906145_4,389872_2,1462866_2,1952116_4,54774_4,1776980_5,1494196_5,253794_5,1569513_3,596728_2,1107588_1,1133763_3,1398076_4,1178171_4,984369_3,2618594_4,1653834_4,2322840_3,2207647_4,1994111_4,1824044_4,2255037_3,2056022_3,1458179_4,1508350_4,1168571_5,766489_3,1424199_5,2054180_3,448902_5,1547173_3,1751103_4,121073_5,2609436_4,1398626_2,1311231_3,2279000_3,236921_5,2566259_5,758937_4,2260684_4,1190829_4,136106_3,344753_3,568930_5,206115_4,2390644_3,2078679_5,1682651_4,386915_4,972136_3,1806515_3,11589_3,2118461_5,444411_3,691108_4,332401_3,1278488_4,358776_4,387418_1,872408_4,646098_4,396595_5,1366860_4,1046882_3,470861_5,1455257_4,1274780_3,379184_4,1273630_4,492291_3,145873_3,1388284_5,712610_4,1116065_1,660499_1,1918987_4,1357894_3,190418_3,1060658_3,1443203_4,1772839_5,2385774_3,1059319_3,831775_4,881346_5,1066317_4,13651_3,208920_4,308753_5,2564257_3,565041_4,1602153_4,173930_4,202811_3,353369_3,1201176_4,2047577_3,685113_4,1686060_5,151004_5,2126192_3,1981464_4,1862581_4,1255780_5,1962300_3,1515355_3,1001779_4,2093105_3,1123959_3,1876297_5,1364481_4,998236_5,328415_3,1347129_4,1117062_4,1033930_3,45117_5,1005769_5,712609_4,740495_4,2497991_4,1017324_4,120491_5,1645794_4,1658790_3,2451020_4,1878798_4,1790903_4,1254683_1,874943_5,121456_4,1140108_4,515436_1,272689_5,1247177_3,263240_3,2539549_3,2565654_5,334701_3,42921_3,2011399_5,433945_5,2151149_4,1415954_2,1086360_3,2419258_4,2380848_5,1550216_1,596533_5,287901_5,188613_4,1654508_3,1313126_5,51334_4,2374451_4,2031093_4,548064_5,946102_5,1790158_4,1403184_3,1535440_4,1426604_4,1815755_5
2:2059652_4,1666394_3,1759415_4,1959936_5,998862_4,2625420_2,573975_3,392722_4,1401650_4,988104_3,977632_4,2557870_4,1793899_5,1340535_5,1888322_5,1283598_3,1784150_4,2271251_5,65932_3,1828884_5,1878728_4,1922778_3,1176404_4,2265116_3,1078701_4,1832577_4,748922_5,1013802_1,1131325_2,2244378_4,494639_2,636262_1,1903158_4,220427_4,2439493_1,2225116_4,1445632_5,2592823_4,1288603_5,2556926_3,1190070_4,1312846_3,2226229_3,1563935_1,69809_5,1349753_3,785768_3,426476_5,810636_4,468713_5,222290_4,349407_5,311232_2,2596999_4,1025601_5,1743759_4,2385553_5,1374216_1,526466_4,2648861_3,1210631_3,2314531_4,618272_1,2532807_3,412535_4,1315005_4,1358911_5,507603_1,1507649_5,845529_5,1479907_5,236271_2,2422676_3,1636093_5,995594_5,1664010_4,2431481_3,1980668_5,402321_4,1344564_3,1632603_3,2567280_3,1623166_3,521932_4,105086_5,2072554_5,2231529_3,2103439_2,261764_1,193476_5,1576540_4,1783594_5,503334_4,183903_5,2606799_1,1236127_4,2375962_3,2212071_3,1252841_3,247898_5,970975_3,305344_1,1581186_4,1129620_3,584750_3,11409_5,1875495_2,1403217_2,2147527_1,2418486_4,1476323_5,2345723_4,2640085_5,1803154_2,1251170_2,527491_4,391517_4,1398626_3,828919_5,196494_5,715897_5,268917_2,41422_4,1806515_3,2118461_4,387418_1,2019055_5,348960_1,1167731_4,2468831_5,219925_4,1025193_5,630887_5,1461435_1,1838586_1,1515430_3,1807053_5,1172326_5,1785842_3,803752_3,1581265_3,515436_1,1824543_4,1283204_3,1272122_5
rsghatpa
 
Posts: 13
Joined: Wed Mar 28, 2018 5:13 pm

Fri May 11, 2018 2:52 pm Change Time Zone

rsghatpa,

I tried your code like this:
Code: Select all
ds := DATASET([{'1:1488844_3,822109_5,885013_4,30878_4,823519_3,893988_3,124105_4,1248029_3,1842128_4,2238063_3,1503895_4,2207774_5,2590061_3,2442_3,543865_4,1209119_4,804919_4,1086807_3,1711859_4,372233_5,1080361_3,1245640_3,558634_4,2165002_4,1181550_3,1227322_4,427928_4,814701_5,808731_4,662870_5,337541_5,786312_3,1133214_4,1537427_4,1209954_5,2381599_3,525356_2,1910569_4,2263586_4,2421815_2,1009622_1,1481961_2,401047_4,2179073_3,1434636_3,93986_5,1308744_5,2647871_4,1905581_5,2508819_3,1578279_1,1159695_4,2588432_3,2423091_3,470232_4,2148699_2,1342007_3,466135_4,2472440_3,1283744_3,1927580_4,716874_5,4326_4,1546549_5,1493697_1,880166_5,535396_2,494609_4,1961619_5,883478_4,793564_4,1567202_2,573537_4,1972040_4,1838912_3,411705_4,2244518_5,584542_5,667730_5,2488120_5,1926776_1,38052_3,1196100_4,314933_3,1792741_2,769643_1,2477242_5,1421006_3,729846_4,1719610_2,1696031_4,1817215_4,406057_4,636262_1,1245406_4,1834590_3,593225_3,1011918_4,1665054_4,2630337_5,1155747_3,2439493_1,479924_5,530789_5,765860_4,231001_3,1493615_5,1850615_2,68959_3,147386_5,624035_5,782308_5,1116080_5,421374_5,1158759_4,1025798_3,1215397_5,2475251_4,321111_2,2162676_3,2635437_4,2389367_1,485622_5,235553_4,831869_4,99400_5,684876_4,1871179_3,1107678_5,642036_3,700890_5,2289956_5,2040859_1,1524964_3,121318_4,317050_5,2287003_5,59052_2,893742_4,1346257_3,55016_3,30245_5,743633_4,1596531_5,1125499_5,706832_4,2465337_3,2291422_1,1777406_3,1904905_4,2450433_3,1348967_2,638020_3,2217779_4,194280_1,493009_4,1567167_4,850327_5,520386_3,320540_2,1188228_2,57961_4,1113230_3,1374216_2,595778_3,209573_4,2354601_5,2563596_4,835265_4,1819474_3,1447104_3,1100940_1,143274_3,2329565_4,181592_4,936396_2,1125797_3,2283366_3,514495_4,1772176_3,1877347_4,1287892_4,255443_2,890669_4,1989766_4,2315073_4,14756_4,907623_3,991423_4,1604238_4,1027056_3,2025883_5,732936_5,563962_5,799442_4,352635_5,2537543_5,1564395_4,1655178_4,573434_4,1141189_4,383247_5,1763921_5,1943970_5,322009_3,2333817_3,2095681_2,1149588_4,2354740_5,2421360_5,496087_2,2191781_1,1694083_4,818416_3,701960_5,2090477_4,1664010_5,2583822_5,369646_5,2234063_4,259799_4,1077982_4,2631796_4,1122383_3,1508526_3,1600207_5,1283117_5,1727869_5,1522799_4,1394012_5,1558286_3,1155602_3,361066_3,1743210_5,1148389_4,2268101_4,519684_5,767518_5,122197_1,2112162_4,1073367_3,400162_5,1524343_5,741245_4,2563768_3,1406595_4,1137010_4,60343_5,225765_4,2530404_3,437881_3,1935793_1,134001_4,2607300_3,1008986_4,94565_4,828410_4,1805202_4,1922925_4,1435717_5,2277395_4,2305014_5,166041_4,2413320_4,87113_2,722591_5,2291306_1,2010770_4,255383_5,1873429_4,1647618_4,608234_5,42930_3,1462072_5,685565_5,3321_3,2554942_4,1874547_4,2269844_5,34907_3,1779903_4,2576424_4,230112_3,508727_3,1603525_3,172264_4,1182185_4,2275470_2,491531_5,1346432_4,1554712_5,1450941_5,1714116_3,2016488_4,1782762_4,1343170_5,2565752_4,435841_3,2242821_5,638824_5,2256485_1,101597_5,623036_5,1559445_5,1723381_5,1824586_4,2233105_4,682963_3,2529547_5,504620_2,1682104_4,16272_4,2491785_5,978412_5,2054145_3,2444240_3,547732_3,811790_5,31913_4,437111_4,640588_4,2625019_3,2605190_5,915_5,1430587_4,2544219_5,2603381_5,305344_1,2569099_1,2430356_4,885165_4,2380806_5,1512406_1,1774623_4,2226525_4,2537076_4,2060858_4,498469_5,68033_4,1819146_5,2088415_4,473070_5,1823641_5,1839976_2,14924_5,1852606_4,453694_5,921487_2,1022254_5,2464081_4,1228324_4,1563530_4,1181170_3,1357013_3,21722_4,288420_5,1739170_5,2584676_3,2013504_4,1245176_4,269524_3,661344_3,652324_3,2239213_3,863302_4,758850_4,1884755_2,544833_3,1562707_1,810700_5,837756_5,155164_4,493945_5,1565175_5,2005193_4,1605780_4,1294335_2,608576_4,659505_4,1604707_4,2630797_5,402266_5,752642_3,1906145_4,389872_2,1462866_2,1952116_4,54774_4,1776980_5,1494196_5,253794_5,1569513_3,596728_2,1107588_1,1133763_3,1398076_4,1178171_4,984369_3,2618594_4,1653834_4,2322840_3,2207647_4,1994111_4,1824044_4,2255037_3,2056022_3,1458179_4,1508350_4,1168571_5,766489_3,1424199_5,2054180_3,448902_5,1547173_3,1751103_4,121073_5,2609436_4,1398626_2,1311231_3,2279000_3,236921_5,2566259_5,758937_4,2260684_4,1190829_4,136106_3,344753_3,568930_5,206115_4,2390644_3,2078679_5,1682651_4,386915_4,972136_3,1806515_3,11589_3,2118461_5,444411_3,691108_4,332401_3,1278488_4,358776_4,387418_1,872408_4,646098_4,396595_5,1366860_4,1046882_3,470861_5,1455257_4,1274780_3,379184_4,1273630_4,492291_3,145873_3,1388284_5,712610_4,1116065_1,660499_1,1918987_4,1357894_3,190418_3,1060658_3,1443203_4,1772839_5,2385774_3,1059319_3,831775_4,881346_5,1066317_4,13651_3,208920_4,308753_5,2564257_3,565041_4,1602153_4,173930_4,202811_3,353369_3,1201176_4,2047577_3,685113_4,1686060_5,151004_5,2126192_3,1981464_4,1862581_4,1255780_5,1962300_3,1515355_3,1001779_4,2093105_3,1123959_3,1876297_5,1364481_4,998236_5,328415_3,1347129_4,1117062_4,1033930_3,45117_5,1005769_5,712609_4,740495_4,2497991_4,1017324_4,120491_5,1645794_4,1658790_3,2451020_4,1878798_4,1790903_4,1254683_1,874943_5,121456_4,1140108_4,515436_1,272689_5,1247177_3,263240_3,2539549_3,2565654_5,334701_3,42921_3,2011399_5,433945_5,2151149_4,1415954_2,1086360_3,2419258_4,2380848_5,1550216_1,596533_5,287901_5,188613_4,1654508_3,1313126_5,51334_4,2374451_4,2031093_4,548064_5,946102_5,1790158_4,1403184_3,1535440_4,1426604_4,1815755_5'},
{'2:2059652_4,1666394_3,1759415_4,1959936_5,998862_4,2625420_2,573975_3,392722_4,1401650_4,988104_3,977632_4,2557870_4,1793899_5,1340535_5,1888322_5,1283598_3,1784150_4,2271251_5,65932_3,1828884_5,1878728_4,1922778_3,1176404_4,2265116_3,1078701_4,1832577_4,748922_5,1013802_1,1131325_2,2244378_4,494639_2,636262_1,1903158_4,220427_4,2439493_1,2225116_4,1445632_5,2592823_4,1288603_5,2556926_3,1190070_4,1312846_3,2226229_3,1563935_1,69809_5,1349753_3,785768_3,426476_5,810636_4,468713_5,222290_4,349407_5,311232_2,2596999_4,1025601_5,1743759_4,2385553_5,1374216_1,526466_4,2648861_3,1210631_3,2314531_4,618272_1,2532807_3,412535_4,1315005_4,1358911_5,507603_1,1507649_5,845529_5,1479907_5,236271_2,2422676_3,1636093_5,995594_5,1664010_4,2431481_3,1980668_5,402321_4,1344564_3,1632603_3,2567280_3,1623166_3,521932_4,105086_5,2072554_5,2231529_3,2103439_2,261764_1,193476_5,1576540_4,1783594_5,503334_4,183903_5,2606799_1,1236127_4,2375962_3,2212071_3,1252841_3,247898_5,970975_3,305344_1,1581186_4,1129620_3,584750_3,11409_5,1875495_2,1403217_2,2147527_1,2418486_4,1476323_5,2345723_4,2640085_5,1803154_2,1251170_2,527491_4,391517_4,1398626_3,828919_5,196494_5,715897_5,268917_2,41422_4,1806515_3,2118461_4,387418_1,2019055_5,348960_1,1167731_4,2468831_5,219925_4,1025193_5,630887_5,1461435_1,1838586_1,1515430_3,1807053_5,1172326_5,1785842_3,803752_3,1581265_3,515436_1,1824543_4,1283204_3,1272122_5'}],
{STRING field1});

histogram_dataset2 := ds;

PATTERN histogramRule1 := PATTERN('(?<=_)[1]');
RULE userRatingRule1 := histogramRule1;
matchText1 := {STRING100 out3 := MATCHTEXT(userRatingRule1)};         
userRatingAnswerOne := PARSE(histogram_dataset2, field1, userRatingRule1, matchText1, BEST, MANY, NOCASE);
userRatingAnswerOne;
and it worked -- no errors.

Therefore, I suggest that the problem may be that you sprayed the file as a CSV but defined it as a THOR file (flat file). Try changing it to CSV and you may get past that error.

HTH,

Richard
rtaylor
Community Advisory Board Member
Community Advisory Board Member
 
Posts: 1376
Joined: Wed Oct 26, 2011 7:40 pm


Return to Programming

Who is online

Users browsing this forum: No registered users and 1 guest