Tag: archiving

While on winter break I decided to tackle some winter cleaning that I’ve put off for years. I decided to clean up my files and data and organize them into something more useful. This was part of the personal automation effort that I mentioned in my goals for 2023.

I have files that go back to my college days. Looking at my data, I see that the oldest document file I have in my archives goes back to March 10, 1993, when I was junior in college. The data that I have spans just about 30 years. And it was something of a mess.

In order to avoid organizing things, I created “temp” folders to store old stuff. Within those temp folders, I also had “bak” folder with still older stuff. Sometimes, I had multiple copies of these things spread across various parts of my filesystem. Over the break, I decided it was finally time to clean this all up.

A profusion of storage options

To clean things up, I first needed to figure out where to put things. Over the years, I’ve built up a profusion of storage options. Locally, I’ve got a laptop, a Mac Mini and my iPhone. The Mac Mini hosts 2 external disks with 6 TB of storage capacity.

I have an iCloud account with 2 TB of storage. I have Dropbox subscription giving me an additional 2 TB of storage. I have a Office 365 account for the family which gives me somewhere around 5 TB of storage. And I have a Google account with Google Drive and some amount of storage there. And I had files scattered about all of these data sources.

Simplifying my file system

I decided to start by simplifying my file system. I’ve had interesting arc over the years. For a long time, was a strong proponent of cloud storage for the obvious benefits of accessibility. But time and experience has taught me that local access with sufficient backup is most desirable for me, as you never know when a service might go away or be priced out of reach. Also, it’s nice to have everything centralized in one place so that I don’t have to remember which data I store where.

I decided, therefore, that the primary source for my data would be local and that I’d use cloud services as a mechanism for making the data accessible between systems, but not as primary storage.

I decided that there are really three types of data that I work with on a regular basis:

1. Working documents (source code, spreadsheets, etc.)
2. Notes (Obsidian, all of my writing, daily notes, diaries, etc.)
3. Archive (all of my data that in not “active” in the sense that I don’t work with it regularly, but is of great historical value to me)

Working documents

I decided that my working documents would be stored on my local machine, in my Documents folder, and that folder is part of my iCloud Drive, so that whatever is stored there is synchronized with other devices that I use.

Given that one of my goals is to see if I can pare down the tools I use to the smallest possible set¹, I found that I was able to consolidate my Documents folder down to just three top-level folder:

The Repos folder contains code repositories for projects I am actively working on–the key word being “actively.”

The Settings folder is where I store various configuration settings, templates, custom fonts I use, and various branding artifacts like profile photos, etc.

Finally, the Wolfram Mathematica folder contains Wolfram Language notebooks I am actively working on. This folder may go away, however. I found that if I enable the “Detect All File Extension” option in Obsidian, I can store my notebooks there, link to the notebooks from other notes, and open the notebooks directly from the links, which is far more useful to me.

For working documents, this is pretty much all I have.

Notes and writing

All of my notes and writing are stored in Obsidian. These days, this is where the vast majority of my daily output goes. I use the Obsidian Sync service to sync my notes between devices. I’ve found the services to be virtually flawless, fast, and extremely reliable. I’ve currently got about 4,000 notes in my Obsidian vault and Obsidian Sync has been perfect in keeping my devices up-to-date. It works so well, that I basically forget it is even running.

I don’t keep my Obsidian vaults on iCloud. There have been issues with vault synchronization when vaults are stored in cloud services like iCloud or Dropbox, and it can occasionally lead to some odd behavior when those services sync files. Instead, I have a separate Vaults folder on my local machine that is not part of a cloud sync service–except for Obsidian Sync–and all of my vaults are stored within that folder.

Archive

As I mentioned, I have files going back 30 years. The most time-consuming part of my winter cleaning was cleaning and organizing that archive, eliminating duplicates, moving things into the archive that belong there and getting rid of things that don’t.

There are really two parts to my archive: Data and Installers. The data portion of my archive is about 15 GB and contains all of the files I’ve worked on over the last 30 years or so. The Installers is much newer than that. Generally, I try to keep copies of older version of software installers whenever I get a new version, so that if I ever run into compatibility issues, I can go back to an older version of a piece of software. This makes up about 8 GB of my archive, so that the total archive currently stands at about 24 GB.

It is important to mention what is not in the archive. Photos and videos I store in Apple Photo which is part of iCloud. I don’t have the time or patience to go through the 30,000 photos and videos I’ve accumulated over the years and pare them down, so I just leave them alone.

I debated where to store my archive. Should I include it iCloud so that it is accessible from all my devices? Or should I keep in one location? After considerable thought, I decided on the latter. My archive is stored on external 3 TB drive connected to my Mac Mini. This drive, along with my Mac Mini, and indeed, all of our household computers, is backed up using CrashPlan, so that if something where to happen to the drive, I wouldn’t lose any data. (There are additional redundancies in place for the archive, but that is a topic for a separate post.) Given that I don’t access the archive regularly, it didn’t seem necessary to keep it in an active cloud service.

Next, I had to figure out how to organize the archive. Over the years, I’ve played around with all kinds of organizational structures, including, most recently, PARA, or even no organization and relying on search functionality to find what I am looking for. But I decided to go old-school and use a more traditional, hierarchical structure to organize the archive instead. The reason is that more and more, I think about how my family would access this data if I wasn’t around. Structures like PARA or arbitrary searches might not work for them. A more obvious hierarchy of topics would be more useful.

Ultimately, I ended up with the following structure:

I tend to sort things from most recently to least recently modified, which explains the order in which the folders appear. The “Photos” folder is not my Apple photos, but rather curated photos that I’ve specifically moved into the archive. Many of these folders contains sub-folders. Here, for instance, is what it looks like inside my Writing folder:

The archive contains all of my personal and work email. I tend to perform these archival functions annually and then zip up the resulting email archives for storage. The earliest email message I have in my archive dates to October 17, 1994.

The archive also contains big social media archives. For instance, when I stopped using Evernote in favor of Obsidian, I did an expert of all of my Evernote data to an archive. Similarly, when I stopped using Facebook, I archived all of my Facebook data. And when things were looking iffy with Twitter, I grabbed an archive of my Twitter data as well. These are all within the Cloud Services section of my archive.

I rely heavily on the file meta-data for finding what I am looking for, particularly filename, modification and creation dates. It is helpful that I’ve kept the original files in many cases because it makes it easier to search the archive in a given timeframe. As I mentioned, some of my files go back as far as 30 years. Here is an example of a few of my files from 30 years ago:

Cleaning up cloud services

In centralizing my files locally, I also took the opportunity to clean up what I had on the various cloud services I use. I had a lot of random stuff on Dropbox that I moved to my archive because I rarely access it these days. Instead, I now use Dropbox as a convenient way to share files, and for some application settings where the application prefers Dropbox over other services for its settings. Also, Dropbox is where my writer’s group stores its stuff, so it is convenient to keep it around. But what I have there is mostly ephemeral now.

We have a family OneDrive from Office 365 and there are a few files I’ve stored there for convenience, but I rarely use Office tools these days outside of work. I moved much of the writing-related documents I had in OneDrive to my archive. What is left there is a few things that are shared between family members.

Still to-do

For several years, between 2013-2016, I used Google Drive almost exclusively for my writing. This is one place that I have yet to tackle cleanup. It is a mess and I imagine it will be a challenge to get it all cleaned up. It should be made easier by the fact that much of what I wrote there should already be in my archive. But it will take time, and I may end up putting off this task until next winter. One indicator of whether I need something is how frequently I access it, and I haven’t needed to access my files in Google Drive in a long time.

Backups

I’ve had a robust strategy for backing up my data, and indeed, all of my family’s data for a long time now. But as it is somewhat off-topic for this post, I’ll save the details of the backup strategy for later.

A feeling of relief

It is amazing what a winter cleaning like this does for the soul. When i completed it, when I had everything setup the way I wanted it, a feeling of relief washed over me. It was a similar feeling to looking over a freshly mown lawn, or a recently cleaned desk surface. Everything was in its place, and everything had a place to go. It’s nice to know that when I create something, there is a clear and obvious place to put it.

It was also a relief to know that I’d finally organized my archive and eliminated all of the duplicate files there.

I am now working on a README file with a target audience of my family that should make it quick and easy for them to find something in the archive in my absence.

Written on 15 January 2023.