Best way to preserve deep digital archives for future generations

[janice6]

In preserving archives, there are many variables depending on what your goal is.  If you want to keep a record for the visual concept, degradation in terms of resolution, depends on the specific user. 

Some time just the major data contributions are good enough and great deterioration can be acceptable bercause you are only triggering memories.  Family records are only demanding in the sense that the reader/viewer has prior knowledge.  Without prior knowledge the context is lost and within a few generations no one knows who is in the record, and worse, many won't care.

If you want detail for later decision making. high resolution with no degradation is a very difficult goal to achieve.  In the terms of the government requirements, retaining ALL the resolution is paramount, and that means that indefinite storage in terms of hundreds of years is almost unachievable, since you not only have to preserve the resolution in its entirety, but you have to be able to display it on a mechanism that is at least as good as the one you made the record with. 

Many forms of re-recording/duplicating of the data introduce noise.  At first it's not observable, but with continued copying, the noise starts to add up and the uncertainty of the bit begins to play a significant part in even in the digital the record.   For minimum resolution in digital data, you require a minimum of 6 db signal to noise ratio to discern a "one" from a "zero", any less and uncertainty renders your data unusable.

The discussions here have involved family records, which have the least requirement for preservation of detail. 

The preservation of written word is the least demanding since it only depends on the reader being able to decipher the record.  Detail is a variable here since in terms of resolution, the written word is so very limited in its resolution, it's not even worthy of concern.  As in the Dead Sea Scrolls, you begin to lose part of the records since the medium is spread out over many volumes and they begin to get dispersed over time, thus losing context and for this discussion, data.

In terms of The longest time of preservation, you not only have the problem of the storage media, but you add into the problem of displaying the data on rapidly obsolete mechanisms.  You also have the problem of people.  You can make the data as comprehensive as possible initially, but in just a little time the desire for additional information not part of the original data set, begins to complicate even the interpretation of those records.

My example in my original post had requirements on the recorded data due to government regulation.  The problem was compounded by the fact that each year the government changed the regulations on what was acceptable in the video recorded.  This demanded that the video record was studied to determine if there were components in the video record that were now not allowed under this year's regulations.  However, when you recorded the data you had no idea of what detail might be demanded in the future.

So, either the video record was of extremely high resolution to detect the presence of components that may be now affected by the new regulation, or the sealed 55 gallon custom made Stainless Steel containers were cut open to examine each and every substance within.  Since when I last looked at this problem there were 55,000, 55 gallon drums of transuranic waste accumulated to date, and this number was being added to every year, it turned into a massive project all by itself of examining each and every drum, and removing and repacking what was allowed and providing for new containers for what was removed and consolidated anew. 

We are talking hundreds of millions of dollars on a continuing yearly basis, simply because the detail in the recorded data was degrading or inadequate to the new evolving congressional demands in the first place.

Because of the intelligence and foresight of our Congress, every year the rules would change and Congress demanded that the new rules be adhered to.  Now you begin to see just how important the finely detailed recording needed to be.  You had to be able to know the most minute piece of information contained within each barrel,  or cut them open the up to see what really existed.

Compounding the information problem, was the fact that some of the radioactive components were dangerous to the observer, but since it was in a Stainless Steel barrel, you couldn't measure the intensity or type of radioactivity in the barrel from the outside.  Think of high level Alpha radiation for example.  So now you added in the ability to sift through each barrel by automated means, so the observer wouldn't be exposed to deadly radiation.
 

This is why the emphasis was on recording extremely high resolution, and being able to keep that recording and it's detail ad infinitum.  Simply because each meeting of congressional politicians wanted to be unique in their control and regulation of something they knew nothing about. 

[Scotsman]

https://en.wikipedia.org/wiki/5D_optical_data_storage

Give it a few years and you'll be set.

[janice6]

9 minutes ago, Scotsman said:

https://en.wikipedia.org/wiki/5D_optical_data_storage

Give it a few years and you'll be set.

Many demonstrations of advanced technology have been offered over the years, but most are still in the theoretical stage.  Because you can store data using research level laboratory instrumentation, is not a guarantee that it will ever come to fruition.  Advanced concepts are just that.  In many instances, the laboratory level announcements are superseded by other technical and cost breakthroughs making the first technology obsolete before it's on the market.

It will be a very long time before Femtosecond pulsed lasers are available to the public for use in a product.  I have worked with them and various research level lasers for many years and they yet are to be available to the public.  Because you can do something in the laboratory doesn't mean it lends itself to general use.

My cohorts were making true laser holograms for years before the public had the means, then all of a sudden, visible light holograms flooded the market and the laser display technology wasn't needed.

Remember all the promises of "Things to Come" that still haven't arrived.  Every couple of years or so we are told to expect flying cars and I know some people that spent their lives "waiting".

Eventually a means of lossless recording will come, but given the complexity and technology, it will take many years to accomplish.

[RenoF250]

On 8/18/2019 at 10:51 AM, JimBianchi said:

As a retired commercial photographer who also shot a few hundred gigs of 1080p at the end, I had this issue when I converted to digital in 2000 and again when I retired in 2012.

For my clients, I gave them the hard drives, cables and power supplies when I closed up shop. (those I could find)

Smaller clients (weddings and family's) were given the CD-R and DVD's, I kept a backup on (3x) hard drives.  Could not find more than half the clients, I will hang on to them indefinitely.  (I had a bride contact me 10 years after the fact for pics of her mom at the wedding, she had died and they wanted all I had.)

My personal stuff is on hard drives and SSD's.  SSD's are shelf stable for a 50 years at least.

Yes, My filing/labeling system is pretty good.  Not perfect, but I can find anything in a few minutes to a an hour.

I still have a decade's worth of 35mm and 120mm film, unscanned.  I looked at some of it last year after 20+ years in storage (kept in an Air Con'ed room and air tight containers) and they held up very well.

 

Most of that stuff is not worth a tinker's dam to anyone, but I keep it anyways.

 

As technology progress I will address it in the future.  I probably have a few hundred terabytes already and 2x that in unscanned film, so the cloud is not a serious option for long term storage.  I do use private setting on facebook photo storage for any client who needs it right now.  Seems to work.  But no guarantee of privacy.

 

All this is contingent on my memory holding out.  I recently got diagnosed with TBI.  (traumatic brain injury)  So if I remember, I will stay abreast of new technologies and such.

You are putting way too much faith in SSDs.  High end SLC NAND SSDs may last 50 years if stored properly but most of the stuff made now is 3D MLC made for price and speed.  I would not rely on them for long life.  Regardless, store them cold.

[PATCHMAN]

In addition to how/where to store digital images, you will also need a way to easily find them.  If it's only family photos, usually not a problem, especially if get re-naming software such as Name Mangler to give unique names (Dad's 90th B-day or Jenny's Christening 1998) to each folder and each photo within the folder.     

Edited August 20, 2019 by PATCHMAN

[kerbie18]

The photos I have from 150 plus years ago are hard copies. Would digital photos last longer for future generations if I make hard copies to pass down? How do I do that?

[Huaco Kid]

1 hour ago, kerbie18 said:

How do I do that?

[PATCHMAN]

6 hours ago, kerbie18 said:

The photos I have from 150 plus years ago are hard copies. Would digital photos last longer for future generations if I make hard copies to pass down? How do I do that?

150 years ago?  Do you just have a print, or do you also have the glass negative(s)?

If you only have the print, scan the print(s) you have.  Save them digitally as jpeg (for your purposes jpeg will be fine).   Then print a new photo from the jpeg.  You can pay extra to print up the photo(s) in archival quality.  Or you can print up the photo(s) in a photo inkjet and reprint as necessary whenever you think the photo(s) have faded.     

The digital image(s) you scanned and saved will not fade or "go bad."  Just be sure you save the images in a few locations in case the computer or the external hard drive(s) it's saved in goes bad, is damaged, lost, etc...