Loading large CSV using Generic indexer

[DDP021]

We loaded a large test CSV file to CMOD.  File loaded successfully although the 87 message contained  this information:

Loading started, 570696197 bytes to process
Warning:  A document of size >570696197< bytes was processed.  It is not recommended to store documents in OnDemand greater than 50MB in size.  Although data may
OnDemand Load Id = >90353-10-0-1FAA-0-0-90354<
Loaded 1 rows into the database
Document compression type used - OD77.  Bytes Stored = >44760080< Rows = >1<
Loading completed

When the users attempted to view in Content Navigator they received this, FILE NOT LOADED COMPLETELY

Spreadsheet contained 1196248 rows but they were only able to see 1048576.

Does anyone know if CMOD has a limit to size it can ingest?

[jsquizz]

What version of CMOD?

[DDP021]

10.1.0.5

[Justin Derrick]

What is your Application Configuration in CMOD?  Are you treating this document as line data?  You may have exceeded a 'number of lines per page' limit, since it appears to cut off at 2¹⁰.

It's probably best to treat this as a user defined data type, so that OnDemand treats it as a binary file.

-JD.
   

[DDP021]

Justin,

We do have it defined as USER DEFINED (CSV).  We did increase the Max Rows on the Application Group to 20,000,000 and having them retry sending file again.

[Justin Derrick]

Hi.  Max rows is for the database table, not documents. 

I'd open a ticket with IBM to see which limitation you're running into -- I haven't seen this specific issue previously.

Is there any chance you could break up this CSV file somehow?  If it's truly archival (i.e., end users won't access it), you could pre-compress it with .zip or .gz, and store it as a binary.

-JD.

[DDP021]

Agree..changing the max rows made no difference.  After loading file was not able to view all rows in CSV.  We are going back to application to see if they are able to split the file.  If they are resistant, we will have to go to IBM.  As always, appreciate the quick response!!   Take care

Dave

[jsquizz]

Agree..changing the max rows made no difference.  After loading file was not able to view all rows in CSV.  We are going back to application to see if they are able to split the file.  If they are resistant, we will have to go to IBM.  As always, appreciate the quick response!!   Take care

Dave

Since you mention that..

I've seen this before, Granted- IT was a very VERY old CMOD system.. Maybe 8.5?

Our business partners were sending CSV files with like, 200-300k rows. They wanted to pickup something on each row of the file. CMOD would get to like the ..10th to last row and just quit. PMR didn't resolvge anything. We just told them to send it as smaller files.

[DDP021]

Thanks for the reply!...Not going to bother opening a PMR for this.   Application end is looking to splitting this up in 2 pieces. 

[jsquizz]

Thanks for the reply!...Not going to bother opening a PMR for this.   Application end is looking to splitting this up in 2 pieces.

Ironic. I'm trying to open up a very large data dump from oracle with excel, it's a .del file with maybe 500k rows? Excel isnt liking it. I got the same error message you got.

Sounds to me like it's probably hitting some kind of resource limit on local PC, memory or cpu..or something

[Justin Derrick]

Yup, there's a limit in Excel:  2²⁰ rows:  https://support.microsoft.com/en-us/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3#ID0EBABAAA=Newer_versions

-JD.

[DDP021]

Good information to know!!!!�.Especially if the application comes back to us blaming CMOD!!!  I appreciate it!!

Take care