RESTYPE=NONE means you aren't getting any of the benefits of CMOD's 'deduplication' of PDF resources.  What sort of documents are you storing?  Are these PDF statements, or do they include something like cheque images, where each PDF file is large because it contains mostly-unique images?  If that isn't your use case, then it may still make sense to store SOME resources, since you *are* using the PDF indexer, and could benefit from storing fonts as resources.

-JD.

If you can convince them to send the index files *after* all the PDFs have been uploaded, that would be best, and you can leave an arsload daemon running that will ingest the files when that index file arrives.

Otherwise, having an entire filesystem set aside is a good idea -- having more than 10,000 files in a single directory usually causes headaches with performance when you are using the 'ls' command or trying to use wildcards.

Also, I'll provide the standard advice -- invest some time in building a utility that will read the generic index file, and ensure that each of the files it describes actually exist in the directory before starting the load. 

-JD.

I did get a few "Hey, I can't reach ODUG" eMails and texts over the past two weeks...  :D

-JD.

Broadly speaking, I advise *against* using AI/LLM output for anything important.  I've had a few occasions where people have written to me, letting me know that machine generated summaries (of information clearly found on the Forums or Wiki) were not accurate.  I informed them to use the source itself and not summaries, because I can't stop LLMs from generating garbage output.

I will *also* say that many, many folks have moved on -- I haven't counted the full total yet, but I'm going to estimate I have nearly 100 eMail bounces waiting for me to deal with.

EDIT:  About 10% of eMails were bounces, or recipients asked to be removed from the Forums, we've got a little over 800 users, down from a high of around 1100.

-JD.

So, it's time to have the discussion...  It's approaching 5 years since our last technical conference...  Five since the last Board of Directors meeting, Three since ODUG ceased to exist according to its Bylaws...  Two years since the old forums went offline suddenly and I managed to recover them from a routine database backup...

ODUG has had a rich history spanning 30+ years... but it's future is currently a blank canvas.  So, with the new year, I ask the question to the community...

What do the Users want to happen to their User's Group?  I'm open to any and all input and feedback, good, bad, and/or ugly.

-JD.

Hey everyone...

While trying to move the Forums to a new cloud provider over the holidays (with a better reputation for mail delivery) and doing some upgrades at the same time, I made a tremendous mistake, and deleted the *real* ODUG Forums, instead of the test machine I *thought* I was working on.  The real catastrophe came when I realized that the backups I was using to do the migration only existed on the real server, and the only offsite copies were several months old.

Unfortunately, we've lost about 5 months of data from the Forums, including posts, replies, and new user memberships...  but also small fixes and improvements I'd made to the server -- tightening firewall rules, database performance tweaks, among others.

To ensure this never happens again, I took a few parts from my old office, and built a 30TB backup server with 3x redundant storage -- 5 disks of ZFS RAIDz3.  Local database backups happen nightly.  Whole-server file-level backups are initiated from the new backup server, which is currently living on my desk.  Offsite copies to a 'tough' USB stick will be rotated out at least monthly.

I've always taken the reliability of ODUG very seriously, and I'm sorry for this lapse that's resulted in data loss.

Having said all that...  ODUG has been extraordinarily quiet lately, and the actual volume of new users / posts / replies has been very low.  That's something I've been meaning to work on as well.  I'll start another thread about that shortly.  :)

-JD.

 

Yes.  The snag is that encryption happens at load time -- all new documents will be encrypted, but old documents will remain un-encrypted.  You can reload the documents to force the data to be encrypted at rest.  This also remains true for Globally Unique IDs and Hash values -- only new data will receive the features.

-JD.

There's a few reasons not to do this...

- The file it generates may be *enormous*, especially if the loaded files were compressed.
- Older versions of CMOD had issues generating generic index files over 2GB in size, rolling over the byte offset back to 0.
- Loading a single large file with data that spans years may break expiration processing, especially if the expiration type is 'Load'.
- Any error retrieving data from CMOD will result in the entire extract failing.
- If there's more than a couple thousand documents, you'll quickly reach a point where your filesystem will be overwhelmed, and simple commands (ls / cp / mv / rm) will take several minutes.

Most folks extract by Load ID or date or by a mostly-unique ID like customer number, etc.

If this is a particularly large job, reach out to me via eMail -- I have a fast & automated extraction utility I use for migrations.

-JD.

Sounds like it would be a good enhancement request.  :)

-JD.

Just a quick note to let you know we were down for a little over 24 hours due to a blackout in the area -- the VM server came back up after power was restored, but the individual VM didn't restart automatically!

We'll be fixing that issue by moving to a small cloud server in the next week or so.

-JD.

Excellent.  I'm glad to see you got the problem resolved.  Take care!

-JD.

That adds some clarity, but unfortunately we still can't answer your question, because we don't know what your site's backup strategy is, and if it relies on being able to access these files to complete a database restore to a particular point in time.  You'll have to determine this with the help of your in-house database administrators and enterprise backup teams.

-JD.

We don't have enough information on your system design to answer the question.  That's not a standard folder name (as per the Installation Guide) so we don't know what's being stored in there.

-JD.

Just guessing, but there's probably more than one record pointing to the same document.  I've seen this at banks where more than one customer is listed on a mortgage -- two records in the database table will point to the same document for a mortgage statement.  You should try using the options -agcNv to export them with the Generic Index File, then look for two records with the same offset & length.

Hrm.  I was hoping the indexing parameters would be helpful, but I don't see anything there. 

It sounds like these are temporary files - is your ARS_TMP configuration parameter set to the same directory that you're loading from?  If so, changing it to another directory might move these out of the way, although it doesn't answer the question about what's in them.

You can use the UNIX/Linux 'file' command to see if the OS recognizes the file format:
  file Load.File.PDF.out?

This should give you the best guess for each data type.  If it can't determine what it is, the response will be 'data'.

-JD.