Sometimes you get lucky

In line with this year’s 2023 World Digital Preservation Day theme, “Digital Preservation: A Concerted Effort” hosted by the Digital Preservation Coalition, we thought it an opportune moment to share a digital preservation experience that may inspire other organisations to dive back into those old systems. Systems that have been gathering virtual dust since migration or decommission waiting for someone with more time, technical expertise, and resources. 

Around 15 years ago we took possession of a server running an instance of a common early 21st century EDRMS, containing the records of a short-lived but significant government agency. We were under pressure to act fast, the agency was winding up, its subject matter experts were leaving, and ongoing technical support for the system was uncertain. So, we decided to take a copy of the system, together with administrator access, and extract and prepare the records for transfer to our digital archive ourselves. We did this by using agency designed queries to apply an approved disposal authority that identified records required as archives. We then exported the metadata and content for conversion to our long-term preservation format (the VERS Encapsulated Object, or VEO), for subsequent ingest to our digital archive.

Given the speed at which we had to work, the loss of subject matter expertise, and the sheer number of records we were dealing with (close to one million), we knew there was a risk some records could be overlooked. This was borne out by the mysterious absence of content for some of the folders exported from the system. We had archived the paper equivalent of cardboard file covers without any content. Believing the number of missing records to be relatively small, and with pressure to move on to other transfers, the issue was parked until we had the time and resources to investigate further. That time finally arrived, 15 years later, in September 2023.

Checklist of things that could go wrong in 2023: Did we know where the server was stored? Would the server still fire up after 15 years? Did we still have the user id and password?

Tick yes to all the above, so what could go wrong? Turns out the account had expired, and the fix required an administrator to log-in and update it. Not much use given we were already trying to log-in using administrator credentials.

That appeared to be the end of the matter. But given we had full access to the server we went exploring and to our good fortune we discovered a master database file that contained all the EDRMS tables and metadata, including links between the metadata in the tables to the content store that held the records. We could then verify that the database contained records of interest by reconciling it with records captured from the EDRMS 15 years ago. 

The challenge now was to understand the workings of an EDRMS system, at the table level. Which tables contained the key descriptive metadata of interest? How were relationships between records stored? How were records linked to their content files in the content store? How were the original queries used to identify permanent records stored? Not so easy given this was a fully functioning, sophisticated system, containing some 127 tables.

Again, we were lucky to have both the technical skills at hand, and staff who had been involved in the original transfer who were familiar with the data. We also have a relatively new archival management system, which allowed us to run more complex queries over the metadata of records captured in the original transfer, enabling comparisons to be made between the records in the EDRMS system, and those in the PROV digital archive.

At the end of the day, we were able to identify the missing records – some 19,000 in total, far more than we had anticipated - and export the metadata and content for ingest to our digital archive. Given what we now know about the system, we also think it would be possible to archive the database using the Swiss Federal Archives SIARD tool (Software Independent Archival of Relational Databases), something to consider in future. 

A good news digital preservation story but not one we are keen to repeat.

#WDPD2023