File System Question for Using Generic Indexer

[JeanineJ]

I currently have my arsload libraries defined for the CMOD arsload daemon as /app/arsacif/acif1 /app/arsacif/acif2 and /app/arsacif/acif3.
I want to define another file system /app/arsacif/acif4 that will be strictly for input coming to be indexed with the Generic Indexer with the load triggered by a scheduled script. The expected payload could be (more or less) 20k of PDF files (the source doesn't want to put them in one large file anymore because it becomes too large for them manipulate).
Is that a good or bad idea?
I'm trying to brainstorm ideas to keep this payload in CMOD. I've already told the client that we won't accept individual pdf files to load one at a time. Been there, done that and loading took too long.

[Mehmet S Yersel]

You already plan to load the PDFs from the new file system using a special script triggered on schedule.

What if that script has a step where 100 of ".IND" files are merged into one large ".IND" file?

This way, the sender would send the 20K single PDFs into the landing zone, just as they prefer, and you will have a script to group them into larger .IND files for efficient CMOD storage and move the merged .IND files into a directory CMOD monitors. Your PDF landing zone will be used by your script to prepare the merge.IND files for the monitored directory.   

[Justin Derrick]

If you can convince them to send the index files *after* all the PDFs have been uploaded, that would be best, and you can leave an arsload daemon running that will ingest the files when that index file arrives.

Otherwise, having an entire filesystem set aside is a good idea -- having more than 10,000 files in a single directory usually causes headaches with performance when you are using the 'ls' command or trying to use wildcards.

Also, I'll provide the standard advice -- invest some time in building a utility that will read the generic index file, and ensure that each of the files it describes actually exist in the directory before starting the load. 

-JD.

[JeanineJ]

I'm putting my functional requirements document together and step one will be send the pdf's first with the caveat that if they don't we'll have more administrative overhead. There's a custom process that uses the CMOD user exit for additional processing.
The vendor is going to be responsible for quality control.
As far as the daemon starting the load, it won't unless you know something other than what is stated in the IBM documentation. The files the daemon will automatically load are .ARD and .PDF. From the manual (online) "The ARSLOAD program will process any input file that has a file type of ARD or PDF."
That's why the new location will be out of the ARSLOAD daemon "view" and under the control of a script.