Tag: obsidian

Winter Cleaning

black classic car inside the garage
Photo by Mike B on Pexels.com

While on winter break I decided to tackle some winter cleaning that I’ve put off for years. I decided to clean up my files and data and organize them into something more useful. This was part of the personal automation effort that I mentioned in my goals for 2023.

I have files that go back to my college days. Looking at my data, I see that the oldest document file I have in my archives goes back to March 10, 1993, when I was junior in college. The data that I have spans just about 30 years. And it was something of a mess.

In order to avoid organizing things, I created “temp” folders to store old stuff. Within those temp folders, I also had “bak” folder with still older stuff. Sometimes, I had multiple copies of these things spread across various parts of my filesystem. Over the break, I decided it was finally time to clean this all up.

A profusion of storage options

To clean things up, I first needed to figure out where to put things. Over the years, I’ve built up a profusion of storage options. Locally, I’ve got a laptop, a Mac Mini and my iPhone. The Mac Mini hosts 2 external disks with 6 TB of storage capacity.

I have an iCloud account with 2 TB of storage. I have Dropbox subscription giving me an additional 2 TB of storage. I have a Office 365 account for the family which gives me somewhere around 5 TB of storage. And I have a Google account with Google Drive and some amount of storage there. And I had files scattered about all of these data sources.

Simplifying my file system

I decided to start by simplifying my file system. I’ve had interesting arc over the years. For a long time, was a strong proponent of cloud storage for the obvious benefits of accessibility. But time and experience has taught me that local access with sufficient backup is most desirable for me, as you never know when a service might go away or be priced out of reach. Also, it’s nice to have everything centralized in one place so that I don’t have to remember which data I store where.

I decided, therefore, that the primary source for my data would be local and that I’d use cloud services as a mechanism for making the data accessible between systems, but not as primary storage.

I decided that there are really three types of data that I work with on a regular basis:

  1. Working documents (source code, spreadsheets, etc.)
  2. Notes (Obsidian, all of my writing, daily notes, diaries, etc.)
  3. Archive (all of my data that in not “active” in the sense that I don’t work with it regularly, but is of great historical value to me)

Working documents

I decided that my working documents would be stored on my local machine, in my Documents folder, and that folder is part of my iCloud Drive, so that whatever is stored there is synchronized with other devices that I use.

Given that one of my goals is to see if I can pare down the tools I use to the smallest possible set1, I found that I was able to consolidate my Documents folder down to just three top-level folder:

Folder in my working documents

The Repos folder contains code repositories for projects I am actively working on–the key word being “actively.”

The Settings folder is where I store various configuration settings, templates, custom fonts I use, and various branding artifacts like profile photos, etc.

Inside my Settings folder

Finally, the Wolfram Mathematica folder contains Wolfram Language notebooks I am actively working on. This folder may go away, however. I found that if I enable the “Detect All File Extension” option in Obsidian, I can store my notebooks there, link to the notebooks from other notes, and open the notebooks directly from the links, which is far more useful to me.

For working documents, this is pretty much all I have.

Notes and writing

All of my notes and writing are stored in Obsidian. These days, this is where the vast majority of my daily output goes. I use the Obsidian Sync service to sync my notes between devices. I’ve found the services to be virtually flawless, fast, and extremely reliable. I’ve currently got about 4,000 notes in my Obsidian vault and Obsidian Sync has been perfect in keeping my devices up-to-date. It works so well, that I basically forget it is even running.

I don’t keep my Obsidian vaults on iCloud. There have been issues with vault synchronization when vaults are stored in cloud services like iCloud or Dropbox, and it can occasionally lead to some odd behavior when those services sync files. Instead, I have a separate Vaults folder on my local machine that is not part of a cloud sync service–except for Obsidian Sync–and all of my vaults are stored within that folder.

Archive

As I mentioned, I have files going back 30 years. The most time-consuming part of my winter cleaning was cleaning and organizing that archive, eliminating duplicates, moving things into the archive that belong there and getting rid of things that don’t.

There are really two parts to my archive: Data and Installers. The data portion of my archive is about 15 GB and contains all of the files I’ve worked on over the last 30 years or so. The Installers is much newer than that. Generally, I try to keep copies of older version of software installers whenever I get a new version, so that if I ever run into compatibility issues, I can go back to an older version of a piece of software. This makes up about 8 GB of my archive, so that the total archive currently stands at about 24 GB.

It is important to mention what is not in the archive. Photos and videos I store in Apple Photo which is part of iCloud. I don’t have the time or patience to go through the 30,000 photos and videos I’ve accumulated over the years and pare them down, so I just leave them alone.

I debated where to store my archive. Should I include it iCloud so that it is accessible from all my devices? Or should I keep in one location? After considerable thought, I decided on the latter. My archive is stored on external 3 TB drive connected to my Mac Mini. This drive, along with my Mac Mini, and indeed, all of our household computers, is backed up using CrashPlan, so that if something where to happen to the drive, I wouldn’t lose any data. (There are additional redundancies in place for the archive, but that is a topic for a separate post.) Given that I don’t access the archive regularly, it didn’t seem necessary to keep it in an active cloud service.

Next, I had to figure out how to organize the archive. Over the years, I’ve played around with all kinds of organizational structures, including, most recently, PARA, or even no organization and relying on search functionality to find what I am looking for. But I decided to go old-school and use a more traditional, hierarchical structure to organize the archive instead. The reason is that more and more, I think about how my family would access this data if I wasn’t around. Structures like PARA or arbitrary searches might not work for them. A more obvious hierarchy of topics would be more useful.

Ultimately, I ended up with the following structure:

Structure of my archive

I tend to sort things from most recently to least recently modified, which explains the order in which the folders appear. The “Photos” folder is not my Apple photos, but rather curated photos that I’ve specifically moved into the archive. Many of these folders contains sub-folders. Here, for instance, is what it looks like inside my Writing folder:

Inside my Writing folder in my archive

The archive contains all of my personal and work email. I tend to perform these archival functions annually and then zip up the resulting email archives for storage. The earliest email message I have in my archive dates to October 17, 1994.

The archive also contains big social media archives. For instance, when I stopped using Evernote in favor of Obsidian, I did an expert of all of my Evernote data to an archive. Similarly, when I stopped using Facebook, I archived all of my Facebook data. And when things were looking iffy with Twitter, I grabbed an archive of my Twitter data as well. These are all within the Cloud Services section of my archive.

I rely heavily on the file meta-data for finding what I am looking for, particularly filename, modification and creation dates. It is helpful that I’ve kept the original files in many cases because it makes it easier to search the archive in a given timeframe. As I mentioned, some of my files go back as far as 30 years. Here is an example of a few of my files from 30 years ago:

Some 30-year old files in my archive

Cleaning up cloud services

In centralizing my files locally, I also took the opportunity to clean up what I had on the various cloud services I use. I had a lot of random stuff on Dropbox that I moved to my archive because I rarely access it these days. Instead, I now use Dropbox as a convenient way to share files, and for some application settings where the application prefers Dropbox over other services for its settings. Also, Dropbox is where my writer’s group stores its stuff, so it is convenient to keep it around. But what I have there is mostly ephemeral now.

We have a family OneDrive from Office 365 and there are a few files I’ve stored there for convenience, but I rarely use Office tools these days outside of work. I moved much of the writing-related documents I had in OneDrive to my archive. What is left there is a few things that are shared between family members.

Still to-do

For several years, between 2013-2016, I used Google Drive almost exclusively for my writing. This is one place that I have yet to tackle cleanup. It is a mess and I imagine it will be a challenge to get it all cleaned up. It should be made easier by the fact that much of what I wrote there should already be in my archive. But it will take time, and I may end up putting off this task until next winter. One indicator of whether I need something is how frequently I access it, and I haven’t needed to access my files in Google Drive in a long time.

Backups

I’ve had a robust strategy for backing up my data, and indeed, all of my family’s data for a long time now. But as it is somewhat off-topic for this post, I’ll save the details of the backup strategy for later.

A feeling of relief

It is amazing what a winter cleaning like this does for the soul. When i completed it, when I had everything setup the way I wanted it, a feeling of relief washed over me. It was a similar feeling to looking over a freshly mown lawn, or a recently cleaned desk surface. Everything was in its place, and everything had a place to go. It’s nice to know that when I create something, there is a clear and obvious place to put it.

It was also a relief to know that I’d finally organized my archive and eliminated all of the duplicate files there.

I am now working on a README file with a target audience of my family that should make it quick and easy for them to find something in the archive in my absence.

Written on 15 January 2023.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. I mentioned two tools in my Goals post, Obsidian and Mathematica/Wolfram Language

Goals for 2023

pexels-photo-13088176.jpeg
Photo by Engin Akyurt on Pexels.com

Over the years I have spent quite a bit of times on various experiments. In the 2010s, I considered the idea of the paperless office, embraced Evernote as a tool for paperless productivity, and wrote a popular series of posts on the subject. So far, in the early 2020s, my experimenting has shifted somewhat. I’ve re-embraced the idea of plain text files as my fundamental unit of work. A plain text file is both versatile and long-lasting. Text files created in the 1970s can still be read today. I found Obsidian, and took lessons from my Going Paperless experiment, to see what I “practically” paperless[1. As in “practical” uses, as opposed to absolute migration to a paper-free lifestyle.] lifestyle looked like using primarily plain text files.

In 2022, I’ve been pleased with how things are working out with Obsidian and plain text files, but there is room for improvement. I’ve identify three areas to experiment with throughout 2023. My hypothesis for this experiment is that improvements in these three areas will lead to more creative time in the years beyond. The three areas are: (1) Consolidation; (2) Simplification; (3) Automation.

Consolidation

When I look at my computer, I am often overwhelmed by all of the apps and tools I find on it, the vast majority of which I rarely use. It would be nice, I tell myself now and then, to get rid of those tools that I never use, but I am sometimes reluctant to do so, thinking that I might need the tool some day. To some extent, this is no different than trying to declutter at home.

While thinking about this recently, I realized that I made huge strides in consolidation over the last two years in terms of my work products, the vast majority of which are now plain text files. With a work product as simple and versatile as a plain text file, it should be much easier to give up apps I no longer need to manipulate those files.

The more I think about this, the more I believe I can get away with two primary work product formats: plain text files used in conjunction with Obsidian; and Wolfram Language notebooks (.nb files) used in conjunction with Wolfram Language and Mathematica.

Using text files as work products of Obsidian is pretty obvious and I’ve written extensively about my use of Obsidian. But Mathematica1?

I’ve been a very casual, hobbyist user of Mathematica for years now. But the more I play with it, and the more I learn about its symbolic and functional structure, the more impressed I am with the language. Moreover, the language now reaches into all aspects of computing so that it seems useful as a general purpose language. Finally, as a developer, I’ve already mastered more common languages like Python, JavaScript, C#, Perl, PHP, Ruby, etc., to my personal satisfaction. Becoming equally proficient in Wolfram Language would something new.

I am mentally dividing this consolidation into two parts:

  1. Plain text files as my primary work product: in the form of notes and writing.
  2. Wolfram Language scripts and notebooks as my primary tool for development and automation.

As I write this, there are about 100 applications in my Application folder on my laptop. That, of course, does not include the command line tools inherent in any Unix-based computer system. I’m using this number of 100 applications as a baseline to see how much I can consolidate over the course of 2023. Obviously, I need more than just Obsidian and Mathematica. For instance, I use Photoshop for photo editing. However, Mathematica has powerful photo- (and video- and sound-) editing tools that may be able to automate and supplant most of what I do in Photoshop. This is an example of the consolidation I am looking to do.

It will be interesting to see how well this consolidation works over the course of 2023.

Simplification

As I worked through my Practically Paperless series, I noted that I was building up more and more complex structures for my notes, as well as relying increasingly on plug-ins, tags, etc.

In the months since finishing that series, I’ve been attempting to simply how I use Obsidian with plain text files. In the process of simplifying, I have the following things in mind to guide me:

  1. My primary work product is a text file. Whether a note, a blog post, a list, a trip summary, a reading note, the primary form in which the content appears is a plain text file.
  2. Using tools like Obsidian is a good for me, but I also keep in mind that these text files need to be useful in the absence of a tool like Obsidian. Keeping the work products themselves simple help in this regard. For example, the structure of markdown makes for simple way of parsing and interpreting these files outside of Obsidian.
  3. I’m trying to simplify my file structure, both within Obsidian and as a whole, including cloud platforms.

Over 2023, I’m looking to continue this simplification of my work products. With the following ideas in mind:

  1. My notes need to be usable outside Obsidian.
  2. My notes and files need to be easy to understand, not just for myself, but for others, especially Kelly and our kids.
  3. My notes and files need to be easy to find for me and for the family.

Automation

I spend a lot of time trying to automate processes. The idea here is to automate the stuff that is repeatable, so that I can spend more time on creative stuff. But often, the time it takes to implement such automation offsets the time it saves. Put another way, if it takes me 2 hours to write a script to automate a task that currently takes me 5 minutes per day, it will take 24 days once the automation is in place to pay back the time it it took to build the automation before it starts paying off.

In the past I’ve used all kinds of tools to implement automations: Python scripts, tools like Alfred or Keyboard Maestro, Apple Shortcuts, etc. In 2023, I am looking to use Wolfram Language as much as possible to automate things, in part because I’m fascinated to see the possibilities in the language, and in part as a way of becoming proficient in the language.

One simple example of this, which I will talk about in more detail in a subsequent post, is adding a list of files I created or updated on a given day to my Daily Notes. it is convenient for me to be able to see at a glance all of the files I worked on in a given day. The way I structure my daily notes makes the daily note the perfect place to store this information. But I don’t want to spend time searching my file system each day for all of the files I created or modified. So I’ve created a Wolfram Language script to do this automatically.

An example of my current Daily Note format, using today's note.
An example of my current Daily Note format, using today’s note.

Throughout 2023, I’ll be looking at how I work2 to identify inefficiencies, things that can be eliminated, and tasks and processes that can be automated, particular:

  1. Things I do frequently and are repeatable. In addition to the above, examples, might include: taking a blog post in note form and automatically publishing it to WordPress; or cropping and editing a photo in a repeatable way
  2. More complex tasks that I perform less frequently but at regular intervals, like archiving information.

Follow along on the journey

Because I know that other people are interesting in this sort of thing (a quick search of “productivity” gives an idea of just how much), I plan to write about this journey of mine throughout the year. As always, I am experimenting to see what works best for me. I know, for instance, that using Wolfram Language for a primary automation and scripting tool is probably outside the norm for many people. But I’m fascinated by the idea and that makes it fun for me. These will tend to be more technical posts, but they might be of interest to others, so, feel free to follow along, ask questions, and offer suggestions.

And Happy New Year, everyone.

Written on 29 December 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. There are other reasons for me to use this, but they are too technical to get into here.
  2. To be honest, I’m doing this constantly, but this year, I’ll be putting a special emphasis on it.

A Simple, Unified Reading List in Obsidian Publish

assorted books on book shelves
Photo by Element5 Digital on Pexels.com

Recently, I have been working on simplifying my notes in Obsidian1. One of the things I have wanted to do for a while is make the list of books I’ve read since 1996 available in simple way that is easy to maintain, but extensible, so that I can eventually include notes about books I’ve read. So I subscribed to the Obsidian Publish service, and after experimenting with various formats, decided to keep things very simple and created a page that lists what I have read since 19962.

The result looks something like this (which may evolve over time, especially if you are reading this post sometime after 15 November 2022):

a screenshot of my new reading list site

Some elements of the new reading list:

  1. Clicking on the “Guide to the reading list” will provide some additional information about what appears on the list.
  2. The entire list of everything I’ve read since 1 January 1996 is in a single page. However, you can jump to a given year using the index at the top of the page.
  3. At the beginning of each year is a “Year in review” section in which I list notable reading-related things for that year. As of this writing, I have not yet completed these for every year on the list.
  4. Some book titles contain links to notes about the book in question. These links are mostly just stubs at this point, as I play around with how I want to manage this.

Reading stats

I’ve also added a reading stats page, which has some basic year-by-year stats for the reading that I do. Over time, I see the information on this page growing, almanac-like to cover various aspects of what I’ve read.

Now

Finally, I’ve added what I call a “now” page that lists things I’m working on now, and things I’m currently reading. This is a conventient place to direct people from various online profiles. (You can see it, for instance, in my Twitter profile, for as long as that remains actively useful.)

a screen capture of my Twitter profile

The reading list site is in its most basic form right now and I see if growing and evolving over time. I’ll post significant updates here on the blog. But you can see more detailed information on the Reading List change log.

Written on 15 November 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. Details in a future post.
  2. Technical detail for those interested in such things: I’ve hosted my reading list in various places online since the late 1990s. The URL has therefore changed over the years. One of the things I wanted to do with this effort was have a single URL on my site that I could in the future no matter where my list was actually hosted. So I created a stub page in WordPress at the following URL: https://jamierubin.net/reading-list/. On this URL, I setup a redirect that takes you to where my list is hosted on my Obsidian Publish site: https://notes.jtrwriter.com/reading/lists/reading-list. I did this so that what I link to my list in the future, I can link the the short WordPress link, and it will always point to the current location of my reading list, no matter where it is hosted. End technical note.

Practically Paperless with Obsidian, Episode 26: Use Case: Managing My Blog Writing in Obsidian

a vintage typewriter
Photo by Suzy Hazelwood on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

In Episode 25, I described how I managed my “professional” writing in Obsidian. I also mentioned that I looked to Obsidian as the one place to do all of my writing. That includes the writing I do here on the blog, so I thought I’d use this episode to describe how I use Obsidian to write for my blog.

WordPress and the Block Editor

I use WordPress for my blog services, and I have been incredibly happy with the service. I like the block editor for writing and editing posts, too. However, when I finally decided to do all of my writing in Obsidian, I intended to do my blog writing there, too. There are a number of advantages to this, but the main one is a single interface and set of commands for all of my writing. Also, all of my writing is now stored in plain text files, using markdown formatting, and readily accessible locally on my computer within my Obsidian vault.

Writing for the Blog

The bulk of the writing I do each week is for the blog. Readers who come for the Practically Paperless posts see just one of eight to ten posts I publish each week. I’ve been writing here on the blog since late 2005, about 17 years, and in that time, I’ve published more than 7,000 posts. Since January 1, 2021, I’ve made it a goal to publish at least one post everyday. As of this writing, I have published at least one post every day for 467 consecutive days.

I generally try to write 2 posts per day, scheduling them out so that I build up a backlog. I do this for two reasons:

  1. It keeps me writing, and keeps me thinking, both of which I enjoy doing.
  2. It acknowledges the truth of writing for me, which is that there are some days where I just can’t bring it. I’m either too busy, too tired, or I write something that I just don’t like. Having a backlog takes the pressure off publishing a post every day.

For instance, as of this writing (I am writing this on April 3, 2022), I have posts scheduled out through April 23. I sometimes leave gaps in the schedule, like I did for this post, since these Practically Paperless posts go out on Tuesdays.

There are two ways that Obsidian helps me with the blog writing: (1) Collecting ideas, and (2) writing posts.

Collecting ideas for the blog

Over the years, I’ve realized how important having a list of ideas is to writing posts whenever I have time. I’ve gotten into the habit of jotting down every idea I get. I don’t always use the ideas, but I jot them down regardless. There have been too many times when I told myself I would remember an idea, only to forget it.

If I am away from the computer, I’ll jot an idea in my Fields Notes notebook. That idea will get transferred to the current day’s daily note at the end of the date. I detailed some of this back in Episode 24. If I am sitting by the computer, the idea goes directly into the daily note as a task. The task gets tagged with “#post-idea”. These tasks, uncompleted and completed are collected using the Dataview plug-in a note called “Post Ideas.” When I am ready to write each day, I’ll pull up this note and skim through the ideas to see if there is anything in particular I want to write about. This note also shows the list of ideas that I have either completed writing or discarded.

Example of my Post Ideas file -- showing the section on ideas I've either written about or discarded.
Example of my Post Ideas file — showing the section on ideas I’ve either written about or discarded.

Writing posts for the blog

When I am ready to start writing, I make use of a template and the QuickAdd plug-in to generate the note in which I compose my post. The template and plug-in prompt me for information about the kind of writing I am doing, generate the note, and automatically file the note in my Writing/Blog/Posts folder in my Obsidian vault. At this point, I start writing. Below you can see the process for creating a new post note:

Animated gif showing how I create a new blog entry in Obsidian using Templater and the QuickAdd plug-ins.

Here is an example of what a post note looks like after I’ve started to write. I’ve used this post for my example:

draft of the current post in Obsidian

I try hard to keep most posts between 500-600 words. That makes writing 2 posts per day much more managable, given my time constraints. It also helps me practice writing to a target length, which is useful when doing professional writing and an outlet requests a piece of, say, 500 or 800 words. Some posts (like many of the posts in this series, are significantly longer). WordPress tells me that for the 114 posts I have published so far as of today, the average length is 784 words.

Publishing to WordPress

Once I finish writing my post, immediately schedule it in WordPress. Usually, I schedule it for the next open date on my calendar. As of today, the next open date is Sunday, April 24, but since I left a gap in my calendar for this post, I would schedule this one on Tuesday, April 12.

This is a manual process for me, and it goes as follows:

  1. Copy the text of the post out of Obsidian.
  2. Create a new post in WordPress and paste the copied text into the body.

The combination of Obsidian and WordPress make this a very simple process and it usually takes just a few seconds. The reason it is so simple is that my posts are written in Obsidian using Markdown formatting and WordPress knows how to interpret Markdown formatting when it is pasted into a post. All my formatting comes through cleanly, which saves a lot of time.

Once I have the post in WordPress, I schedule it for its future date. I change the status on my Obsidian note to “scheduled” and add the date that it was scheduled for.

Managing My Posts

I have “Blog Post MOC” note that i use to manage my posts. There are three sections to this post, each using a different dataview query to display a list of posts:

  1. Posts scheduled tomorrow. This lists any posts that are schedule for the next day. I use this to proofread the post the night before and try to intercept any obvious typos I happen to notice.
  2. Posts scheduled today. This lists any posts scheduled to be published on the current day. This reminds me what is being published. I also use this to update the meta-data in the note to reflect the status (published) and the link to the published post.
  3. Published posts. This is a list of all the published posts, with a link to the published URL for the post in question.
my blog MOC showing the tomorrow, today, and published posts sections
A look at my blog MOC.

Comments on the Blog

As I said, I try to capture all of my writing in Obsidian. That includes significant comments I make on my blog (or on others, for that matter). I have template for blog comments and I use it to write out my comments before posting them to the blog. This has a few advantages for me:

  1. It keeps all of my writing in Obsdian. I can use the Vim keyboard mappings I am used to and store my comments locally as part of all of my writing captured in my vault.
  2. It allows me to think through my comments and write them with the same care I’d use for any other writing. When I wrote comments on the fly, in the spur of the moment, I tend to (a) make mistakes, and (b) miss some important points I want to make. Writing them out in Obsidian ahead of time let’s me think through what I want to say.

The process for creating a new comment note in Obsidian is similar to the process for my other writing. It makes use of a template and the QuickAdd plug in. After I select the destination as “Blog” the template gives me the following options:

Selecting "comment" in my template.
Selecting “comment” in my template.

This provides a quick way for categorizing the note as a comment to a blog post. I also use this for other significant social media posts: posts and comments to Reddit, to various forums, and to other blogs, for instance. I find three advantages to this:

  1. It allows me to do all of my writing in Obsidian.
  2. Writing out a comment or reply in Obsidian allows me to to think about what I am writing and edit it much easier than if I did it in a text box of a blog or a social media site like Reddit. I don’t feel rushed. I can draft a comment, then come back to it later and edit it before posting.
  3. It allows me to collect all of my writing in one place, whether that is my “professional” writing, blog writing, or social media posts and comments.

Final thoughts

In my attempt to collect all of my writing in text files in Obsidian, I’ve shown how I manage my professional writing, and my blog writing. There is one final bit of writing that I now do and capture in Obsidian. In next week’s episode, I’ll go through my process for writing my journal entries in Obsidian.

See you back here next week.

Prev: Episode 25: Five Use Cases for Managing My Writing in Obsidian
Next: Episode 27: Use Case: Writing Journal Entries in Obsidian (coming April 19, 2022)

Written on April 3 and April 11, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Practically Paperless with Obsidian, Episode 25: Five Use Cases for Managing My Writing in Obsidian

shallow photoghrapy of black and gray type writer keys
Photo by mali maeder on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

Years before I began using Obsidian, I’d wanted to consolidate all of my writing into a single format–preferably text files. I wanted all of my old writing files accessible in plain text, and wanted to do all future writing in the same format. Ideally, I would use a single editor for all of this and would only have to know a single set of keyboard mappings. Once I started using Obsidian, and learned more and more of its features, I began to do what I’d I’d always wanted: centralize all of my writing–all of it–in a single place.

This episode, and the several that follow will describe how I’ve centralized all of my writing in Obsidian. I’ll begin with how I use Obsidian for my “professional” writing, illustrated through 5 use cases.

Professional writing?

In addition to this blog and my day job as an application developer and project manager, I also write stories and articles for publication. I began to write with the idea to sell stories almost 30 years ago, while still in college. It took 14 years of writing, submitting, and collecting rejections before I made my first professional story sale. Sales came quicker, and I branched out from writing stories to writing articles as well. Writing short fiction and nonfiction pieces was never going to take over from my day job as my primary source of income, but it was my avocation: something I’d always wanted to do and something that I really enjoyed doing.

Over the years, I’ve used all kinds of tools for my writing. When I began to write in college I was composing my stories in Microsoft Word 5.5 for DOS–still my favorite word processor to this day. I moved onto other tools. I was a long-time user of Scrivener, and later, Google Docs. When I began using Obsidian more than 15 months ago, I decided that I wanted to do all my writing there. Moreover my vault could serve as the repository for all of my writing.

Writing as a profession–even one as humble as mine–involves more than just typing words onto a page. I have identified 5 tasks I perform as part of the overall process of managing my writing. They are:

  1. Writing drafts
  2. Managing writing projects
  3. Tracking submissions
  4. Tracking sales and contracts
  5. Seeing the big picture

Prerequisites

There are a couple of tools I use in conjunction with Obsidian to make the overall process smooth and seamless.

  • Templater plug-in. I use this to build a set of writing-related templates so that I am not re-inventing the wheel every time I create a new draft manuscript note, or a new submissions note.
  • Quick Add plug-in. I use this to speed up the process of creation and to populate some of the meta-data in my templates.
  • Dataview plug-in. I use this in my “writing project” notes for collecting together related information about the project in one place. Because the dataview plug-in does not actually link notes together, I have, as you will see, taken additional steps to ensure that all of my writing notes have links within them to keep them related to their projects. This serves as a kind of backup to the dataview plug-in.
  • Pandoc. I use this to automate compiling a draft note into standard manuscript format. Standard manuscript format is used by most of the professional publications that I have worked with. It is a simple set of guidelines for formatting a manuscript that takes the job of figuring out how to format a document out of the writer’s hand. This is a good thing since getting bogged down in formatting is a good way to avoid writing.

Use Case 1: Writing Drafts

My process for actually sitting down to write in Obsidian is straight-forward:

  1. Create a new note using a New Manuscript Draft template via the Quick Add plug-in. This creates a note with the appropriate meta-data at the top (the YAML frontmatter) and a callout footer with a link to the project. It also automatically files the note into my “Working Drafts” folder.
  2. Write. This is the activity where I try to spend the bulk of my time.

There isn’t much formatting involved in the types of manuscripts I produce. I use simple markdown for things like italicized text. I use markdown headings for sections or parts of a story or article. Otherwise, I just write. I do, however, have a process for working through my drafts that has evolved over the years and I have built that process into my workflow in Obsidian.

1st draft and 2 second drafts

When I create a new note using my New Manscript Draft template via the Quick Add plug-in, I am prompted for several pieces of information:

  • Draft title. This is often a working title. My first published story was called “When I Kissed the Learned Astronomer,” for instance.
  • Project title. A name of the project that this piece is associated with. This is often an abbreviated version of the title. In the case of my first published story, the project name was “Learned Astronomer”
  • Draft: The draft version, 1st draft, 2nd draft, etc.

After providing this information, a new note is available to me with the meta-data in the YAML frontmatter of the note file, and a footer containing a hard Obsidian link to the project note. Now I can begin writing. Here is what it looks like in action:

My New Manuscript Draft template in action
My New Manuscript Draft template in action

At the bottom of my template, you’ll notice a callout section called “Note” which contains a link to the project name. Many of these notes are surfaced through the dataview plug-in, as you will see. However, the dataview plug-in does not create actual links between notes. The “footer” in my templates create links between the note and the project: in this case, between the draft and the project. This serves as a way to see the relationships in backlinks, or the graph view. It also serves as a kind of backup to the dataview itself.

The footer of my template, which creates a relationship between the draft and the project note.

For me, a typical story or article goes through two drafts. On rare occasions, I’ll have a third draft. Each draft gets its own note so I can see the evolution of the piece from one draft to another. These drafts are accessible from the project note, as I will demonstrate in Use Case 2.

Submitted and publication drafts

In addition to “1st”, “2nd”, and “3rd” draft, my template also provides me with two additional options: “Submitted draft” and “Publication draft.”

  • Submitted draft. This is a version of a piece that is formally submitted to a specific market. There can sometimes be several submitted drafts, each one tied to a specific market, slightly different based on feedback I have received, or changes I have made between submissions.
  • Publication draft. This is a version a piece that is ultimately published. This may differ slightly from the submitted draft. Some magazines and publishers provide authors with “galleys” of their piece set in type from which minor corrections can be introduced prior to publication. Changes I make on a galley get reflected in the publication draft.

Writing

Ultimately the creation of the new manuscript draft note takes just a few seconds. The templates and plug-ins help speed that process along and keep everything standardized so that I can get to the writing itself, which is what matters.

These templates and plug-ins eliminate a lot of overhead and allow me to focus on writing.

Compiling a manuscript with Pandoc

When I have a submission draft locked down, I will compile a manscript using Pandoc. Pandoc, for those who aren’t familiar with it, is a tool that takes one format of text and converts it to another format. For my purposes, it takes a plain text markdown file and generates a Word document (or sometimes, a PDF) in standard manuscript format.

I have a simple command I run at the command line to do this. Because I’ve already set up a “standard manuscript” template in Pandoc, I run my command against a given note in Obsidian, and out comes a properly formatted Word document ready for submission. After running my command, Pandoc generates the manuscript. Here is an example of the first page of a manuscript that is produced (header information is made up for this purpose):

sample first page of "compiled" manuscript draft via Pandoc
Sample first page of a manuscript compiled from a text file using Pandoc.

Once again, this saves me time. I want to spend as much of my available “writing” time actually writing.

Use Case 2: Managing writing projects

If I manage to complete a draft, I create a “Writing Project” note. (If I give up, I trunk the draft.) I have a “New Writing Project” template that I use to create the project. Like the “New Manuscript Draft” template, it prompts me for a bunch of information, and the resulting project file gets places in a project folder.

animated gif showing how I create a new writing project
Creating a new writing project.

There are five sections to my New Writing Project template:

  1. Drafts: a dataview table listing all of the drafts associated with the project.
  2. Submissions: a dataview table listing all of the submissions associated with the project.
  3. Contracts: a dataview table listing all of the contracts associated with the project.
  4. Appearances: currently, a manually maintained list of places the piece has appeared, with links if available.
  5. Notes: notes and other items related to the project.

Here is an example of a Writing Project note from my first published story from back in 2007:

Writing Project note for my first published story, showing four of the five sections
Writing project note for my first published story.

Everything on the template, with the exception of the “Appearances” is automated so long as I use the same project name throughout the process. Below is what my actual Writing Project template looks like in source mode so that you can see the dataview queries:

a sample of my new writing project template
Template for my New Writing Project.


Once again, because the “footer” in my Manuscript Draft, Submission, and Contract templates contains a link to the project note, there are hard links between the notes, in addition to surfacing the related notes through the dataview plug-in. For instance, from the project note, I can see the backlinks to all of the related notes (drafts, submissions, etc.):

A view of a writing project with showing the backlinks to related notes
A view of a writing project with showing the backlinks to related notes

As indicated, the {{VALUE:Project Name}} comes from a Template configuration I have in the QuickAdd plug-in. That configuration looks as follows:

Quick Add plug-in configuration for my New Writing Project template
Quick Add plug-in configuration for my New Writing Project template

The result is that when I create a new draft, submission, or contract that uses the same project name in the meta-data, it will appear on the MOC for the project in question, surfaced in the dataview tables as well as the backlinks to the note.

I have a writing project MOC for every writing project for which I completed a first draft, even if the story or article was ultimately trunked. More on this in Use Case 5 below.

Use Case 3: Tracking submissions

When a story or article is ready for submission, I use my “New Submission” template. This prompts me for two pieces of information:

  • Project name
  • Market name

The market name is typically an abbreviated version of the market to which the piece is submitted (e.g., IGMS for InterGalactic Medicine Show, Daily Beast for The Daily Beast, etc.)

When the note is created, it is automatically filed in a Submissions folder. I then manually add a couple of additional pieces of information:

  • Submission date
  • A note link to the the submitted manuscript draft.

Recall that some projects may have multiple submission drafts. This link is what ties a specific draft to a specific market. Here is what a submission looks like:

animated gif showing how my New Submission template works
Creating a new submission.

When I hear back from a market, I will update the submission note. I’ll update the status, as necessary, and the status date. I will also make running notes in the note itself. For instance, if an editor requests changes, I’ll note the changes in the submission note that need to be incorporated into the published draft.

As I make these changes, or add new submissions, they are automatically captured on the Writing Project MOC for the project in question.

Here is a look at my New Submission template:

My New Submission template
My New Submission template

Use Case 4: Tracking sales and contracts

When a piece is sold, a contract usually follows that contains the terms of the sale, how much I am to be paid, and what rights I am selling. When this happens, I create Contract note using my New Contract template via the QuickAdd plug-in. The contract template collects information like:

  • project name
  • market
  • contract date
  • payment
  • contract terms/rights

This information goes into the meta-data of the note. I then manually add an embedded link to the PDF version of the contract. This note gets autmatically filed into my _documents folder. The contract is automatically listed in the “Contracts” section of the project MOC for the piece. For some projects, there may be more than one contract. This will happen for reprints or foreign sales, for instance.

Here is an example of what a contract note looks like. This one is for my story, “Take One For the Road” which appeared in the June 2010 issue of Analog Science Fiction:

Contract note for my story "Take One for the Road" (Analog SF, June 2011)
Contract note for my story “Take One for the Road” (Analog SF, June 2011)

The contract is automatically listed in the “Contract” section of the corresponding project note. Like my other templates, the contract template also has a “footer” with links to the project note and the corresponding market note, thus creating hard links in Obsidian between those notes.

Use Case 5: Seeing the big picture

Finally, I have a “Writing Projects MOC” note that lists all of my writing projects using the data view. It has three parts:

  • projects in progress
  • projects that have been complete/published
  • projects that have been trunked

This is the 50,000 foot view of my writing. It lets me see everything and then drill down into those project MOCs that I am interested in seeing in greater detail. Here is what my Writing Project MOC looks like today:

my master writing project note, listing all of my writing projects, active, completed, and trunked
Master “Writing Projects” note listing all of my writing projects, active, completed, and trunked.

The screen capture above cuts off after my first four “trunked” stories, but there are at least a hundred of them in that list, going back, as you can see to as early as January 1993, when I first began submitting.

There are other views I have as well. I can see tables showing me how much I was paid in a given year–useful for tax season. Or I can see a listing of publications by market. One thing I am working on is creating a template for appearances; this will essentially automate my bibliography.

Final thoughts

No solution is perfect for everyone. This one works well for me because it allows not only to do all of my writing in Obsidian (and in plain text files), but it allows me to manage my writing in plain text files as well. The dataviews are convenient for this, but not required. As I have shown, my templates also create hard links between projects and related notes so that I can see the relationships in backlinks, and graph view, in addition to the dataview tables.

There is certainly room for some improvement in my process. But that comes with time. I should also point out that I use this process for my paid writing. I have a similar, process for how I managed my writing here on the blog — but that will be the subject of next week’s episode.

I know there is a lot in this post. I am happy to try to answer any questions, technical or otherwise about managing my writing in Obsidian in the comment thread.

Prev: Episode 24: Use Case: How I Capture Field Notes in Obsidian
Next: Episode 26: Use Case: How I Manage My Blog Writing in Obsidian (coming April 12, 2022)

Written on March 24-26 and April 1, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Practically Paperless with Obsidian, Episode 24: Use Case: How I Capture Field Notes in Obsidian

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

I have been a dedicated user of Field Notes brand notebooks since 2015. Since then, I have rarely been without a Field Notes notebook in my back pocket. These Field Notes notebooks represent my short-term memory. They contain fleeting notes, lists, ideas, names of people (if I don’t write them down I am liable to forget them), and just about anything else I need to remember. Here are 2 annotated pages from a notebook from 2016.

Capturing my Field Notes in Obsidian

My process for capturing my notes in Obsidian is straight-forward. At the end of each day, I open my Daily Notes file for the day and do the following:

  1. Flip through notes in my Field Notes notebook looking for anything worth saving.
  2. Tranfer those items worth saving into my daily notes.
  3. Elaborate on these as necessary

Most of what I jot in these notebooks stays in the notebooks. The most common things to go into Obsidian are:

  • blog post ideas
  • notes jotted about things I read or listed to
  • notes from experiences, like tours, museum visits, etc1.

I will frequently elaborate on notes as I enter them. For instance, if I entered a post idea for “capturing field notes in obsidian” in my notebook, when I add that note to my daily notes, I might expand it, add some sub-bullets, flesh it out a bit, or clarify it so that it is more useful than what I scribbled in the notebook.

For a while, I prefaced these items in my daily notes with an “FN” to indicate that they came from a Field Notes notebook, but I gave that up as completely unnecessary.

I do try to fit the notes into the rhythm of the day in my daily notes. If I jot down a blog post idea on my morning walk, that will go into the earlier part of my daily notes for that day. If I wanted to note a particularly good restaurant where we ate dinner, that will go in the latter part of the day

Usually, I don’t add a whole lot and probably spent less than 5 minutes each day transferring notes from my notebook into Obsidian.

Below is a page from my current Field Notes notebook from March 6, 2022, followed by my daily notes for the same day. I’ve highlighted the notes in the notebook page that I moved into Obsidian, and highlighted them in Obsidian so you can see the end result.

Page from my Field Notes notebook. Items highlighted in the red boxes were  transferred to my Obsidian daily notes
Page from my Field Notes notebook. Items highlighted in the red boxes were transferred to my Obsidian daily notes

And below, here are the my Obsidian daily notes for the same day:

My daily notes from March 6 -- items in the red boxes came from my Field Notes notebook.
My daily notes from March 6 — items in the red boxes came from my Field Notes notebook.

Why not just capture these notes directly in Obsidian?

People who see me with my Field Notes notebook frequently ask why I don’t use a note-taking app for these notes. “Aren’t you the paperless guy?” they’d ask back when I was Evernote’s paperless ambassador. Plenty of people do capture their notes directly in Obsidian and it works perfectly fine for them, there are 5 reasons why I use a notebook for these fleeting notes instead of an app.

1. A notebook is faster for me

In my experience, nothing is faster or more convenient than a pen and a notebook. Believe me, I have tried. I’ve measured the time it takes me on paper and in a dozen or more note-taking apps over the years. A notebook is always faster. I think there are few reasons for this:

  • In the time it takes to pull out my phone, unlock it, open the app I want, and create a new note, I’ve already jotted the note in my Field Notes notebook and moved on to other things.
  • Over the years I’ve developed a kind of shorthand that makes jotting notes even faster.

2. I enjoy using a notebook

I like using a notebook. There is a tactile difference to jotting notes with pen and paper that I enjoy and that I probably wouldn’t give up even if an app was developed that was more convenient than paper.

3. A notebook doesn’t run out of battery life

I don’t have to worry about a dead or dying battery with a notebook. I may run out of pages, but when I am down to the last few blank pages in a notebook, I always have a second with me. I may run out of ink, but I always carry two pens.

4. A notebook gets me off screens, for at least some of the day

I try to avoid screens for everything. When I walk in the morning and have an idea for a post, or want to jot a note on the book I am listening to, I don’t want to look at a screen. My notebook provides a convenient way to capture fleeting thoughts without depending on my phone.

5. A notebook acts as a good filter for fleeting information

As I said earlier, not everything that goes into my notebook needs to be kept. I don’t need to put shopping lists into Obsidian. I don’t need to record the name of our server in the restaurant we’re eating at in Obsidian. For those things that are worth keeping, the shorthand in my notebook reminds me of them and I when they do go into Obsidian, I can elaborate on them as needed.

What about the notes I don’t capture in Obsidian?

For a while, I considered scanning the pages of my notebooks and storing them as PDFs in Obsidian, but that seemed like too much work for too little gain. Put another way, it seemed impractical to do that. Instead, I’ve found that capturing just those things that I find useful in the future is enough.

As for all of the other notes: when I fill up a notebook, it goes into a box with all of the other Field Notes notebooks I’ve filled up over the years. Any time I want, I can flip through them and see the stuff that I needed to remember on a given day. It is difficult to search the notebooks this way. I once spent quite a bit of time searching for a beer brand in my notebooks. But that’s why I lean toward keeping notes in Obsidian that I think will be useful in the future. In Obsidian, I could easily locate what I am looking for.

A box of my Field Notes notebooks
A box of my Field Notes notebooks

In next week’s episode, I’ll talk about how I use Obsidian to manage my writing, illustrated through 5 use cases

Prev: Episode 23: Protecting My Data in Obsidian
Next: Episode 25: 5 Use Cases for Managing My Writing in Obsidian

Written on March 17, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. Yup, I’m that guy with a notebook out jotting furiously as a tour guide leads us through Monticello, Mount Vernon, or some other place.

Practically Paperless with Obsidian, Episode 23: Protecting My Data in Obsidian

black android smartphone on top of white book
Photo by Pixabay on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

As I go through the process of migrating from Evernote to Obsidian, I have moved all kinds of notes and documents into Obsidian. Some of these, like tax documents, for instance, raise the question: Is it safe to put sensitive data into Obsidian?

This is nothing new. When I was writing my Going Paperless posts using Evernote ten years ago, one of the most common questions I got was, “But don’t you worry about exposing your data?” The question came in different forms–what if Evernote is hacked? Don’t you worry that Evernote staff can read your notes and documents?–but the general idea was always the same. People were worried about others gaining access personal information and want to know if a tool is safe to use in this regard. I thought I would spend this Episode talking about how I think about data protection and describing some of the tools I use to protect my in the context of going practically paperless with Obsidian.

Layers of a security onion

When I think of protecting the notes I keep in Obsidian, I think in terms of layers of an onion, with each layer representing one method of protection that I am comfortable with. Each method alone is not perfect. Combined, however, they provide me with enough sense of security that I don’t lose any sleep over the matter. These layers remind me of those doors that Maxwell Smart passed through to get to and from his office each day.

In the case of the data I keep in Obsidian there are three layers of security working together to protect my data: physical access, digital access, and data encryption. I illustrate this as follows:

diagram showing the security layers as if layers of an onion, with my obsidian vault at the center.

1. Physical access to Obsidian data

One of the big selling points of Obsidian is that data resides locally by default. My Obsidian notes are stored on the hard disk of my desktop or laptop computer. Let’s set aside cloud-syncing services and mobile devices for the moment–I’ll come back to it later, I promise. For now, my Obsidian vault sits in a folder on a hard disk on my Mac Mini, and that Mac Mini sits on shelf in my office.

If someone wants to access my Obsidian notes, they need access to my Mac Mini and that means they need access to my office. Access to my office requires access to my house. This is illustrated in the “physical access” layer in the image above. On the right side, the house represent physical access to my Obsidian data.

Now let’s pretend that instead of storing my notes in Obsidian, I followed Niklas Luhmann’s Zettelksaten model and all of my notes were on slips of paper stored in a slipbox that resided in my office where my Mac Mini currently sits. This is illustrated in the “physical access” layer in the image above. On the left side, the house represents physical access to my slip box.

So far physical access to my notes is identical in either case. A person needs to gain access to my office to get to either the Mac Mini or the slipbox. Physical security of my notes is the same in both cases. I make this point because I sometime hear arguments that people don’t want to store certain types of data in digital form. This is perfectly reasonable. Everyone needs to find their own comfort level. But from my viewpoint, physical security for locally stored data in Obsidian and the same data stored on paper in a slipbox is identical.

Physical security itself can include a number of layers. (1) We lock the door to our house. (2) We may lock the door to our office within our house1. (3) We may have an alarm system installed to discourage an unauthorized breach.

I never lost sleep about data security when most of my files were on paper and stored in a filing cabinet in my office, long before Evernote or Obsidian. Given that the physical security is the same when storing my data locally in Obsidian, I don’t lose sleep over the physical security of my data their either.

2. Digital access to Obsidian data

Suppose, however, that someone managed to breach the physical security layer and was able to sit in front of my Mac Mini2. The second layer of the onion–protecting digital access to my Obsidian data–kicks in.

To gain access to my notes in Obsidian, a person would first need to be able to sign into my Mac Mini using an account that has permissions to the vault. At its most basic level, that means a person needs a password to sign into the computer. Good password practices help here. For instance, I use a different password for everything. Gaining access to one password only gets a person access to one specific device or service. A password manager helps with this. I’ve also made it a practice to use long passwords as they are much harder to crack.

Suppose, however, for the sake of our exercise, that the person who managed to break into my house has figured out my unique and hard-to-guess password for my computer. What then? Just as physical access can have many elements to it (keys, alarms, etc.) so does digital access. In addition to a strong, unique password, I also make use of multi-factor authentication. A password alone isn’t enough to get into the computer. A second form of verification is required. There all kinds of multi-factor authentication: biometric, an authenticator tool, a text message to another device, or a separate device entirely like a YubiKey.

The likelihood of someone breaching my physical security, figuring out my password, and getting past the multi-factor authentication is so small that it really isn’t worth worrying about, at least in my view. Suppose, for instance, that my notes were stored on paper in a slipbox, as my illustration shows on the left side. In this case, the digital access layer of the security onion doesn’t exist. Here, then, is a side-by-side comparison of what someone has to do to gain access to my notes in Obsidian in case, and on paper (slipbox) in another:

LayerSlipbox (Paper)Obsidian (Digital)
Physical access1. Access to house (key, alarm code, etc.)
2. Access to office
2. Access to slipbox (key?)
1. Access to house (key, alarm code, etc.)
2. Access to office
Digital accessNone1. Know my strong, unique password.
2. Have access to my second form of authentication

It would actually be easier to gain access to my data if it were stored on paper in my office than in Obsidian with the vault stored locally on my computer.

3. Data encryption

Suppose our frustrated villain, able to get into my office, but unable to breach the digital security of my computer, decides to just take the computer with them, with the idea off gaining direct access to the data on the hard disk within, perhaps by mounting it on another device. At this point, the scenario is getting a bit preposterous, but let’s just go with it. This is where the data encryption layer comes into play.

The hard disks on my computers are all encrypted using FileVault, MacOSs native disk encryption software. FileVault uses 128-bit AES encryption with a 256-bit key. The data on the disk is encrypted at rest until proper authorization has been granted at which point the data is decrypted. “Proper authorization” in this case means the access required in part 2 above. Without a password and valid authentication from a separate authenticator, there is no practical way to decrypt the data. Our villain would have obtained a computer, but it would be useless to them unless they wiped the drive, in which case, they lose access to the data.

With these three layers of the onion working in conjunction. I feel perfectly safe storing my notes, even sensitive ones, in my local Obsidian vault. The risk of a breach is not impossible, but it is so unlikely that I don’t worry about it.

Syncing notes and cloud security

Ah, but what if I want to access my notes from multiple devices? Don’t I worry about someone gaining access to my data in the cloud? I don’t.

For Obsidian data to sync across multiple devices, some kind of syncing service must be used. I have experience with two of these services: iCloud and Obsidian Sync. When I began using Obsidian in January 2021, I used iCloud to sync data between my computer and my iPhone. In that scenario, the data lives on my hard disk (which is still encrypted) is synchronized through the iCloud service to iCloud servers, which in turn synced the data to my iPhone (where the data is also encrypted). Indeed the entire synchronization process is encrypted.

Eventually, I ran into sync issues that seemed to be entirely related to iCloud and I switched to Obsidian’s Sync service. This is a fast, reliable sync service that I setup once and then never had to mess with again. It just works.

There are two models to Obsidian Sync: end-to-end encryption and managed encryption. To ensure complete privacy, I use the end-to-end encryption model. In this model, the data (already encrypted on my hard disk thanks to FileVault) is encrypted when being sent to and from Obsidian’s sync service. It is encrypted at rest on Obsidian’s servers. Best of all, I am the only one who can access the data. The encryption uses a password and Obsidian developers do not have access to that password. The downside of this is that if I ever forget my password, I would not be able to access the vault in the Obsidian Sync service3.

The 3 layers of my security onion still apply even with data stored in Obsidian Sync. Someone would need physical access to the server where the data was stored. They would need digitial access to the data on the server, and they would need a password for decrypting that data. Those three layers of security are good enough for me.

But what about a mobile device?

Possibly the weakest link in this chain is my iPhone, and I say it is the weakest link only because it is the most likely place where someone can gain physical access. This doesn’t even have to be malicious. My phone could slip out of my pocket and I might not notice it is gone right away.

Even here, however, the other layers of the security onion kick in. If someone finds my phone, they need to gain digital access to my phone. And data on iPhones is encrypted so direct access to the solid state storage does no good. Moreover, security is set up on my phone so that after a certain number of failed attempts to sign into the phone, the phone will wipe its data, making it useless to anyone who was after the data. Also, I can wipe the data remotely, if I realize the phone is missing and the phone is turned on and able to access a cellular network.

Some additional protections

Protecting data is more than just preventing unauthorized access to the data. It is also making sure that I have access when I need it. Here are some additional ways that I protect my data (including my Obsidian vault).

1. VPN service

I use a VPN service to make sure my data is encrypted end-to-end, even when I am on 5G, cellular, or unfamiliar WiFi networks. This way, I don’t have to worry about someone snooping on networks that I have no control over. The VPN service ensures my data is encrypted end-to-end from the moment it leaves my device.

2. Data backups

My data is regularly backed up. There are local and cloud backups:

  • I use TimeMachine to continually backup my data to an external disk on my home machines. This allows for quick local restores for times that I make a dumb mistake.
  • All of our computers are also backed up to the cloud using CrashPlan’s Small Business backup plan. This service costs some money, but I look at it the same way I look at paying for insurance. The few times I’ve needed it, it has been there and restored all of my data quickly.

In each case, the backed-up data is encrypted and only I can decrypt it.

Finding a security sweet spot

It is one thing for me to list out all of the reasons why I don’t worry about unauthorized access to my Obsidian data. It is another for others to feel comfortable on their own setup. Some people will insist that the three layers I find reassuring are just not reassuring enough. Others will think it is overkill. That’s fine. Everyone needs to figure out their own security sweet spot. Below is a diagram I sketched out for a post on this subject I wrote just about 10 years ago. I’m reusing it here because I think it is as true now as it was then. Given the practical theme of these posts, I might change the “low reward” and “low risk” to “low practicality” and “highly practical”, but otherwise, it stands.

Find a sweet spot you are comfortable with. And remember, you always have the option of not storing something in Obsidian, or in digital form period.

Summary of Tools and Services

The table below summarizes some of the tools and service I make use of in each of the layers discussed about. I should point out that I’ve had these practices in place for a long time. They do more than just protect my Obsidian data. They protect all my data.

LayerTools/Services
1. Physical accessLock and key
Security service
2. Digital accessStrong, unique password (LastPass for password manager)
Multi-factor authentication (biometric / YubiKey)
3. Data encryptionFileVault (128-bit AES encryption with 256-bit key)
Express VPN to ensure end-to-end encryption on “untrusted” networks
Obsidian Sync for end-to-end encryption for syncing Obsidian data between devices
Additional ToolsTime Machine (MacOS) for local continuous backups
CrashPlan for Small Business for cloud backups

Coming attractions

I have covered a lot of ground in the first 23 episodes of this series. A lot of this ground was me exploring the basic principles and functions of Obsidian, and figuring out how to make practical use of them. Over the next several episodes, I’ll be exploring specific use cases — practical examples of how I use Obsidian on a day-to-day basis.

I also wanted to put out a call for suggestions on topics you’d like to see covered here. I can’t say I’ll be able to meet every demand: I tend to avoid writing about situations for which I have no need or for which I have no experience, but I’ll do my best.

Thanks again for reading, and I’ll see you back here next weel.

Prev: Episode 22: Daily Notes Revisited: The Best of Both Worlds?
Next: Episode 24: Use Case: How I Capture Field Notes In Obsidian

Written on March 12 and 20, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. For those home offices with doors. Mine home office has a doorway, but no door.
  2. Given that my Obsidian data would be completely uninteresting to anyone but myself, I have a hard time conceiving of someone going through the exercise of breaking into the house to gain access, but l’ll go with it for the purposes of this illustration.
  3. Of course, I’d still have my local vault, and any backups I have made

Practically Paperless with Obsidian, Episode 22: Daily Notes Revisited: The Best of Both Worlds?

brown framed eyeglasses on a calendar
Photo by Olya Kobruseva on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

A couple of months ago, in Episode 15, I wrote about how I use daily notes as an index to my life. I was impressed by a post I’d read on someone who kept all of his daily notes in a single file, and I aimed to try to reproduce that in Obsidian. Since writing that post, I’ve tweaked things somewhat–enough that I felt it was worth reporting back on. Since this series is an ongoing experiment in going practically paperless, it isn’t unusual for me to change the way I do things when I find ways and methods that work better.

Two competing requirements

In the case of daily notes, I found myself pinned in by two competing requirements:

  1. I want the ability to see more than just “today’s” note. I like to be able to easily scroll through past daily notes. This is a quick and easy way to review things without having to search through a bunch of files, or open a bushel of notes windows to reference something I am looking for. This is what attracted me to the single daily notes file in the first place.
  2. I want to use the native daily notes functionality that comes with Obsidian. This functionality, however, is based on a single-note-per-day, and I’ve found it just doesn’t work nearly as well when all daily notes are in a single file. The main problem was that if I linked to another note from my daily note file, the backlink referred to the daily note file, but not the date in the file under which the link was listed.

So: I want to use a single file for all of my daily notes, but I want the daily notes functionality that is based on one note per day. What to do?

First solution

In order to address this conundrum, I decided to go back to the single-note-per-day. That immediately gained me the native daily note functionality in Obsidian. In a single-note-per-day model, if I link to another note, the backlink shows the date of because it is the name of the daily note file itself. For instance, I list out the blog posts I write on a given day within my daily notes file. In the case of this post, the backlink shows what day I wrote the post:

screenshot showing date reference (in title) to a daily note in a backlink

This is much more useful than when I used a single note for all my daily notes. In that situation, the backlink would show up as follows:

screenshot showing reference to the "daily" file (no date) to a daily note in a backlink

and “Daily” doesn’t tell me the date. That is a big drawback of the single-file model. As I pointed out in Episode 15, one thing that would help would be if the section path that the link was under was displayed as part of the backlink — that at least would pick up the date. I even submitted a feature request for this. But it doesn’t exist now and I have to work with what I’ve got.

One thing I did to attempt to get the best of both worlds was to setup my workspace so that the previous day’s note always appeared in a window above the current day’s note. That way I could easily scroll through the previous day’s note if I needed to reference something there. I pin the current day’s note, but leave the previous day’s note unpinned so I can quickly switch to another day using the calendar.

my obsidian workspace with a 3 windows, 2 on the left split vertically with yesterday's daily note on top and today's pinned below. window the right is whatever I am working on

I played around with this model and it worked okay, but it just wasn’t as useful as being able to scroll through all of my daily notes in a single file. It did, however, get me the benefit of the native daily note functionality in Obsidian, so that was a partial win.

Second solution

After giving it some thought, I came up with another solution that works even better: a single daily notes file made up of transcluded links (embeds) to individual daily notes. This is achieved as follows:

  1. Daily notes are created normally, one-file-per-day, using native daily note functionality in Obsidian.
  2. At the end of each day, I add a transcluded link to the current day’s daily note to the top of a note file called “Daily”

The “Daily” note file looks like this in source mode:

screenshot of Daily file in source mode showing just a list of embedded links to daily note files

When viewed in Live Preview mode, however (which is my default), this same file looks as follows:

screenshot showing the same Daily file in Live Preview mode which renders the embedded daily notes so that they all appear to be in a single file

This allows me to scroll through all of my individual daily notes as if they are in a single file, even though the daily notes themselves are each in a separate file. It allows me to work in a workspace where I keep the Daily file pinned at all times, even while having today’s daily note open in a separate window, like this:

screenshot showing my new workspace with the Daily file pinned on the left and today's daily note pinned on the lower right, with my working file above

In this setup, I can easily scroll through all of my daily notes going back to when I started intending to keep them in a single file (December 28, 2021). What’s more, as I add new items to my current daily note (the window on the lower-right), they appear instantly in the file on the left.

One change I made in the “Daily” file from my previous attempt at a single file is that I reverse sort the entries so that the most recent day appears at the top of the file. I found that I was constantly scrolling to the bottom of the single file to add new notes, and while I like the strict chronology, it was more practical to reverse sort it. Several readers pointed out to me that this is what they did, and it made a lot of sense to me.

While the embedded text within the file is not searchable in source mode or Live Preview mode, it is searchable in Read mode. And I can leave the file in read mode because I only have to made an update to it once a day, to add the current embedded daily note file. That means, if I hit Cmd-F to search in the file, results show for all of the embedded files. For instance, if I search for the term “- read” here is what the results look like:

screenshot of unified daily notes search results spanning across multiple days

Note that the search results span multiple days in this view, meaning they span multiple embedded files. This is what I was hoping for when I began looking into this solution.

The best of both worlds?

This solution — using individual daily note files and a separate Daily file with embedded notes via transcluded links — is better and gets me closer to the two competing requirements I’ve been aiming for. I can use native daily note functionality; I can see references to daily notes in backlinks, and I can scroll through the daily notes as if they are all in a single file. But it is not perfect. There is still at least one challenge to overcome, and a relatively easy one at that.

Challenge: Automating the Daily file

As I said, currently, at the end of each day, I add a new transcluded link to the current daily note at the top of my Daily note file. If I forget to do this, I won’t see the previous day’s note. This isn’t all that cumbersome, but it is something that I want to automate.

A simple shell or Python script can take care of that. The script will add the trancluded link to the new daily note at the beginning of each day so that I can see it there at the start of the day, instead of just at the end. This really is a simple script and when I get it written, I’ll post a link to it on GitHub1.

Daily summary and Dataview

There is one other adjustment I’ve been playing around with in my daily notes. Now that each day is in a separate file, my daily notes template includes YAML frontmatter for capturing some information that I care about. Currently, that template looks as follows:

---
locations: ["Arlington, VA"]
sleep: 1
summary: Good writing day, busy work day. Upgraded Obsidian. Good response to latest PP post
---

Here is what this data is for:

  • Locations: a list of places that I was at on a given day. Not every place I go, but rather where I was based on that day. On days I travel, this list contains multiple entries, like [“Arlington, VA”, “Wisp Resort, McHenry, MD”]
  • Sleep: a rating of the quality of my previous nights sleep on a scale of -2 to 2, where 0 is an average night, -2 is a absolutely terrible sleep, and 2 is a dreamless, perfect sleep. For those curious, I learned of this method listening to Jim Collins describe this method on the Tim Ferris Show podcast.
  • Summary: A short, one line summary of my day, written at the end of the day.

I use this in a Dataview table that provides a kind of index to my daily notes. At a glance I can see where I was on a given day. I can also use the summary to help remember what happened on that day. I can then drill into the specific note for more details. I have a note called “Daily Notes Index” and within that note, use the following dataview query:

```dataview
TABLE summary, locations FROM "Daily Notes"
SORT file.name DESC```

Here is what this Daily Notes Index looks like for March 2020 so far:

screenshot of the dataview table rendering a summary of my daily notes with 3 columns: file, summary, and locations.

This hasn’t replaced my Daily file containing embedded daily notes. Instead, I find this file more useful as a high-level review of my days, something I can scan through at the end of the month or end of the year, or a place I can go to quickly see where I was or what I did on a certain day without flipping through the details of the individual daily notes file.

Putting it all together

To sum up the current state of my daily notes:

  1. I have gone back to using a single-note-per-day to get the most benefit from Obsidian’s native daily note functionality. I always have “today’s” note open in a pinned window.
  2. I maintain a “Daily” note that emulates the all-daily-notes-in-one-file that I started with back in Episode 15. This allows me to easily scroll through all of my daily notes and search them without having to open multiple files. Given the way that I work, this is more efficient for me.
  3. I use the YAML frontmatter in my daily notes to populate a dataview table in a Daily Notes Index file that I use periodically for skimming my days

This is as close as I’ve been able to get to the best of both worlds: each day in its own note, and all notes in a single file. So far, it seems to be working for me, but as always, I am open to ideas an suggestions for improvment.

In next week’s post, I’m going to change pace a bit and talk about my personal views on information security with respect to my notes in Obsidian. I’ve been asked questions about security quite a bit. This was also try when I was writing about using Evernote a decade ago. The post will summarize how I think about security, some tools I use to protect my data, and what I do and won’t worry about. See you back here next week!

Prev: Episode 21: Tags in Theory and Tags in Practice (And Never the Twain Shall Meet?)
Next: Episode 23: Protecting My Data in Obsidian

Written on March 9 and 11, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. I suppose I could try to make this a plug-in but (a) this is too niche a use case, and (b) I’d have to deal with all of the overhead involved in a plug-in. A Python script on a scheduled job is much simpler for this.

Practically Paperless with Obsidian, Episode 21: Tags in Theory and Tags in Practice (And Never the Twain Shall Meet?)

blank tags in close up photography
Photo by Eva Elijas on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

This episode is less of a “how-do-I” and more of a “how-I-struggle with…” post. I have been working with tags for more than a decade and still struggle with them. I wanted to give fair warning at the top in case anyone is expecting brilliant revelations about tagging notes in Obsidian here.

I can’t remember when I first learned of the concept of tags. As someone who grew up during the personal computer revolution, my approach to organizing information was naturally hierarchical. From my earliest days using computers, operating systems organized files in a hierarchical structure, and it seemed natural to follow that model.

Moreover, my schooling, which from 5th grade on paralleled my experience with personal computers–seemed to encourage a hierarchical form of information organization. Textbooks had tables of contents that were organized by Part / Chapter / Section. We used outlines to plan arguments. School notebooks were organized by tabs (remember Trapper-Keepers?). Even something as unbiquious as TV Guide was organized hierarchcally: by Date / Time / Station. Indeed. the closest thing to “tags” that I can think of were the indexes that accompanied many nonfiction books. Entries in these indexes (which themselves were sometimes hierarchical) spanned the hierarchy of the book itself. That is, an entry for “atmosphere, of Earth” might have references in multiple parts, sections, and chapters of the book in question.

First page of the index of the book The Nine Planets by Franklyn M. Branley, the first book I ever checked out of a library.
The first page of the index of The Nine Planets by Franklyn M. Branley, the first book I ever checked out of the library, and the book that introduced me to science and astronomy,

The best I can say is that my awareness of tags probably came out of the various photo-storage apps and services that began to appear around 2004 or so, a decade after the Internet as we know it began to take shape.

Tags: In theory

These photo apps (Flickr was an early one I recall using) allowed users to “tag” photos with arbitrary keywords. This was a revelation to me. After living in a hierachical box for so long, tags introduced two ways that freed me from that box.

  1. The keywords were completely arbitrary. There was no set list. I could make them up, or choose from words that had already been used.
  2. One was not limited to a single keyword. I could tag an image with multiple keywords, placing it in multiple categories at once.

The latter point especially was important. Hierarchies, by their nature, limit where you can place something. A photo, for instance, can be placed in a folder path called /Family/Trips/Italy, or in /Buildings/Cathedrals. To put the photo in both places requires duplicating the photo file and then maintaining it in both places. Tags compliment the hierarchy, allowing one to tag a photo as “Italy” and “Cathedral” simulatenously, without having to duplicate the file.

There was another dimension to tags that I found interesting: crowd-sourcing. Because I first discovered through Internet services like Flickr, tags were often crowd-sourced, meaning anyone could apply tags to photos, thereby creating a kind of public melange of the categorization of images.

At the time, I remember thinking how useful it would be to have the ability to artitrarily tag files in a file system. MacOS introduced such a feature in OS X 10.9 in 2013. When that happened, however, I didn’t jump on it and start tagging my files.

The problem for me was that tags seem like a great idea in theory, but are much more difficult to implement in practice. I discovered this in my first concerted attempt to use tags — back when I was writing my Going Paperless series for Evernote between 2012-2016. Evernote allows you to tag notes with an arbitrary number of user-generated tags. When I began using Evernote (around 2010) I did what I suspect many users did: I started tagging everything without a whole lot of thought about it.

Tags: In practice, a.k.a., by intuition

In practice, tagging has been tricky for me: a classic case of having enough rope to hang myself. It turns out that, for me, at least, tagging comes with the same problems as hierarchical organization: if you just start arbitrarily creating tags, you create a mess of confusion that it is difficult to escape from. Escape is difficult because the process is self-perpetuating, and once started, I find I’d rather go with the flow than to start over from scratch.

What is needed for tags is some kind of taxonomy: a set of rules to follow for when to apply tags, when to create a new tag, and how tags relate to the information they are organizing. Here, however, I come up against a limitation of personal knowledge and experience: I am not a researcher, nor do I have expertise in subjects like library sciences that might provide some kind of guidance in the development of a useful taxonomy for tagging. Ultimately, I am tagging by intuition, which is probably not the best approach.

So back in my Evernote days, I began to think about practical ways in which tagging fits into my note organization, and from that thinking.

1. Tags can span hierarchies

Hiearchical organization can be useful because its very structure provides a map for locating something. I discussed the practical utility (for me) back in Episode 18, when I talked about how I use folders to quickly locate Maps of Content notes. The main limitation to hierarchies from my vantage point is that a note falls into only one path.

Tags can span hierarchies. If you imagine folders and tags as two dimensions of a grid, then the organizational model of notes would look something like this, where folder hierarchies cross horizontally, and tags (in this case, notes tagged “#jamie”) slice vertically across all folders.

an illustration of hierarchical folders show on a horizontal axis and tags shows on a vertical axis

Tags, therefore, compliment hierarchies by allowing you to create categories of notes that span multiple hierachies. In a practical sense, this allows two quick ways of finding a note or collection of notes: by either following a path in a single hierarchy, or by tag for specific collection of notes that span multiple paths in a hierarchy.

2. Tags can be used to quickly identify an arbitrary collection of notes that have a practical everyday utility

I frequently have to find a note related to a specific person. Kelly might ask, “Do you have Grace’s 4th grade report card from the spring?” Or maybe I need my son’s “School Health Entrance Form” for a camp application. One practical arbritrary collection of notes, therefore, is to tag notes by person. I do this way many notes, and these notes often span multiple paths in my folder structure.

This kind of tagging allows me to quickly locate all notes tagged “jamie”, for instance no matter where they are located, and then quickly whittle those notes down to the specific note I am looking for by adding additional search criteria. In the above illustration, the tag #jamie serves as that arbitrary collection of notes.

Similarly, I use tags to identify document types: forms, statements, correspondence, receipts, confirmations, etc. Again, the documents themselves may be spread throughout a file structure, but if I am looking for all school-related forms for my daughter, Grace, then my search would begin with notes tagged “grace” and also tagged “form” — from that result set, I can quickly locate the specific forms in question, regardless of where they are in the file system.

3. Tags can be placeholders for future ideas and concepts

I’ve also found tags useful as placeholders for future ideas and concepts. I described one of these uses in Episode 20, where I illustrated how I tag task lines in my daily notes files with the “post-idea” tag for ideas that I want to write about here on the blog. I then use the dataview plug-in to collect all of the “incomplete” tasks with the tag “post-ideas” in a query that lists the open ideas in a single place.

image showing a checkbox list of blog post ideas, rendered using the dataview plugin and filtered by the tag "post-idea"

There are also subjects that I have a broad interest in, and I’ve taken to using a tag/sub-tag model for captuing notes related to these interests. A few examples include:

  • theme/theory-of-notes
  • theme/theory-of-work
  • theme/value-of-reading
  • sports/baseball

This provides a quick way for me to collect notes together around a theme or concept that I am interested in.

Tagging problems I still struggle with

Despite these practical uses, I still feel like an amateur when it comes to tagging, and no doubt this is reflected in my tag structure in Obsidian today. Indeed, just looking at my list of tags in Obsidian makes me cringe a little. It still feels too arbitrary.

There is a time investment required to tag a note, but at the time I tag the note, it is not clear whether or not there is a clear return on that investment of time. Adding a bad tag is worse than adding no tag at all because time invested to add the tag is either wasted or, worse, does not help in locating the tagged document later on.

Another problem is that, intuitively, I feel that I should be using the fewest tags required to find what I am looking for. And yet tags, like rabbits, seem to proliferate faster than I can wrangle them in.

All of this points to a lack of taxonomy. A clearly defined taxonomy provides a scope for tagging, and more importantly, removes any abiguity from the tagging process. In other words, there is no confusion between two tags, no flitting about wondering, should I tag this note using tag A or tag B. It is always clear from the taxonomy how best to tag a note. But a taxonomy requires knowledge and experience that I don’t have, and this comes to the crux of the tagging problem that I struggle with today:

I don’t know enough about what I need to know to define a useful tagging taxonomy.

I have started to tackle this problem using sub-tags to try to identify those areas of interest I want to collect, but even this seems tenuous at best. It is an ongoing process, and the better handle I have on what goes into Obsidian, the better chance I have of coming up with a reasonably useful taxonomy in the long-run.

For now, my biggest take-aways when it comes to tagging are to ask myself the following questions when tagging notes:

  1. Does this tag help to locate the note or collection of notes that it takes?
  2. Is this tag clear enough that I will remember it in the future without much thought?
  3. Is tagging this note absolutely necessary? Can I find this note easily without a tag?

Outside of some simple, practical use cases that I’ve outlined above, I’m still wary of tags, and yet, ironically, I use them more than I should. I’ve been doing this for more than a decade with only small hints toward a useful taxonomy. Finding the taxonomy that works for me will be the information-theory equivalent of the pot of gold at the end of the rainbow.

Prev: Episode 20: Experimenting with the Dataview Plug-In.
Next: Episode 22: Daily Notes Revisited: The Best of Both Worlds?

Written on March 7, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Practically Paperless with Obsidian, Episode 18: How I Organize My Notes

depth of field photography of file arrangement
Photo by Mike on Pexels.com

During the years that I was Evernote’s paperless ambassador, one thing I learned was that organization was a very personal thing. This came as no surprise. Each person is unique, has unique requirements, and the result is a unique was of organizing their notes. What was a bit surprising at the time was how vocal people were about what method of organization was best. I’m not doing that here.

Recently I wrote about what I think of as “how-to versus how-I.” My migration from Evernote to Obsidian is an experiment. My use of plain text files for the bulk of my output is an experiment. Some things work, and I keep those things around. Other things don’t work and I toss them or try to find ways to improve them. So when I talk about how I organize my notes in Obsidian, I am talking about an experiment that I am working on that meets my own unique requirements. What works works for me, and may not work for anyone else. But I think it is good to have examples of what people are doing because there is always the possibility to learn something.

Lessons-Learned

Lessons from Evernote

Consider some lessons I took from how I organized my notes in Evernote:

  1. The organization grew organically without much initial thought.
  2. I considered notebooks topical and that all notes related to the topic should go into the notebook.
  3. I thought of tags as themes: a mechanism that allowed me to span or collect topics together.

This resulted in an overly complicated set of notebooks, where it was sometimes difficult to find something because topically, I put it into one notebook when, thematically, I looked for it elsewhere.

Lessons from Obsidian

I have been using Obsidian for over a year now, and in that time, I’ve experimented quiet a bit. One thing that I have found, is that my notes fall into several general categories:

  1. Attachments: PDFs, images files and other files stored in my vault that are generally not plain text. Example: a PDF of a W-2
  2. Documents: Documents are attachments with context. They are wrappers for the attchments. That is, a note in which I transclude an attachment file, in order to provide additional meta-data and context to that file either through YAML frontmatter, note text, or links. Example: a note that transcludes a PDF of W-2 and also contains meta-data (tags, etc.) about the document, as well as any other remarks or links that I add. For more on how I think about document notes, see Episode 1.
  3. Permanent notes: I am borrowing this term from Luhan’s Zettelkasten methodology. Permanent notes are notes about one thought that can stand on its own, written in my own words, with context added in the form of links and tags. Example: a note about how power differs in the Senate and executive branch, based on reading I have done.
  4. Maps of Content: Notes that pull together links to other notes in a unified context that allows for easy access to relavant information. Example: a note about a specific vehicle, that links to other notes and documents related to the vehicle, and provides additional context in the form of a timeline or backlinks.
  5. Writing: Notes that contain writing: stories, letters, blog posts drafts, essays, etc. Example: The note containing the draft of this blog post.

Looking through the notes in my vault, I can generally put each one into one of these buckets. There are some exceptions, of course, like templates or my daily notes file. But the vast majority fit into this framework.

As I was thinking about my requirements for organizaing my notes, I tried to keep all of these lessons in mind.

My Requirements

Since the basis of organization stems from one’s requirements, let me talk about the requirements I had in mind when I was organizing my notes in Obsidian. As I thought about how I might better organize my notes in Obsidian, two requirements began to emerge:

  1. Optimize for finding things later as opposed to organizing them now. This is something I have emphasized in previous episodes. It takes time to maintain a taxonomy for notes. The important thing for me is not so much how the notes are organized, but how quickly I can find them when I need them.
  2. Remember that I may not be the only one that needs to access these notes and plan accordingly. Kelly may need access to the notes and she needs to be able to find what she is looking for. We jokingly call this “succession planning” but in reality, if something happened to me, access to my notes, which contain all kinds of information that she would need, would be vital. Finding that information quickly would take some burden off her mind.

(Re)organizing My Vault

These two requirements led to some decisions that I made about how I’d structure my vault.

  1. Attachements, documents and permanents notes can have a completely flat structure. I typically access these note in one of two ways: via an MOC note, or via a link on another note. It is rare for me to search for these notes directly. The allows for a flat structure.
  2. Maps of content (MOCs) need to be very easy to find. Because they are topical and provide sturcture and context, they lend themselves to a more hierarchical structure. MOCs are also the most useful mechanism for Kelly to find what she is looking for.
  3. Writing notes could go into their own folder and be organized in a relatively simple hierarchy, the way I might organize them on a file system.

I went about making these changes. It was not hard to do because Obsidian handles all of the linking and indexing behind the scenes as notes move around. I was tempted to have just four or five top level folders, so that my vault would look something like:

  • attachments
  • documents
  • slipbox
  • writing
  • MOCs

But I decided for ease-of-use to keep to a flatter hierarchy and implement the taxonomy through a different mechanism. At the top level, therefore, my vault looks like this:

The vast majority of my note can be found in the attachments, document, and slipbox folders. I’d guess they contain 90%+ of all of my notes. The structure for MOCs is flat, too, but contains some structure to make it easy to find the MOC I’m looking for. Still, the MOCs make up a small fraction of all the notes, even though they link to many of them.

Some examples I used to test this structure

Example 1: Tax season – MOCs to the rescue

As I write this, I am preparing to send all of our tax information to our accountant for preparation and filing. An MOC for all tax-related documents makes this a simple exercise. As tax documents come in, I add the PDFs to my vault as attachments, create document notes for them to capture context and meta-data, and then add a reference to the document on the MOC note for taxes.

Here is what the “Tax Documents” MOC looks like (with some annotations in red):

In the Tax Documents section, there is a corresponding year for each year in the Tax Returns section and under each section is a list of all of the tax documents for that year, each of which in turn transcludes the PDF attachment as well as meta-data for the yearly summary table.

I’ve tried to make this a one-stop-shopping document for anything tax related. It is useful for more than just tax season. We recently refinanced our house (squeezing it in before the rates started to go up again) and needed to provide some tax documents as part of that process. This made it easy. If Kelly needs some tax information, she know to go to this note, and it will point her to everywhere else she needs to go.

When I go to sent the documents to our accountant, I’ve already got everything in one place. It’s just a matter of uploading the documents to their secure portal.

This MOC meets both by requirements: (1) optimizing for finding things later, and (2) making sure Kelly can find them too.

Example 2: Managing services – herding kittens

I miss the old days when you had your phone bill, you gas bill, and electric bill, and maybe a newspaper subscription to keep track of. Today, everything is a service: from streaming media to software to digital subscriptions, there are tons of services that need to be managed and coralled.

Within my MOC structure, I have folder called “Services” and within that folder is an MOC note for each and every service we subscribe to. Combining the usual utilities with subscriptions and software services, it is a long list. Each service is billed in its own way, on its own cycle. Each costs something different. It can be hard to stay on top of. I do it through MOCs using a simple set of guidelines:

  1. Each service gets its own note.
  2. Each note has a specific set of YAML frontmatter at the top to keep track of the basics. I’ll come back to this in a moment. Currently, it looks like this:
  1. The note itself has two main sections: (1) Service info, which includes things like where the service is billed to, links to managing the account, how to cancel the service, etc. (2) Service notes: if I have to contact support, the notes related to the contact go in this section. The dates are links back to the daily note on the date of the contact. Here’s an example:
  1. With a note for each service populated with the meta-data, I have a Service Summary MOC note. This note uses the Dataview to summarize all of our “active” services with links to the service note, and listing the monthly and annual costs, and the due date of each. This allows us to view, at-a-glance, all of the services we have, how much they are costing us, and when they are do. If we cancel a service, I mark it’s meta-data status as “canceled” and it drops off our active list. Here is a snippet from that summary note:

This model has worked very well for keeping up with all of our services. The summary note makes it easy to find anything we are looking for. The MOC note for each service contains everything we need to know about that service and its history. There only one place we need to go to look for the information so it is easy to find.

Example 3: Continuity planning – a step-by-step guide

Back when I was Evernote’s paperless ambassador, one of the more popular pieces I wrote was one I called 6 Steps for Life Continuity Planning in Evernote. In our house, we jokingly call this “succession planning” to take away some of the discomfort of thinking about this kind of thing. As part of my migration from Evernote, I’ve been working on using the features of Obsidian to improve upon how we document this continuity planning. An MOC I call “Succession Plan” plays a central role in this.

The Succession Plan MOC centralizes everything that Kelly or I would need if something happened to one of us. It is also written so that if something happened to both of us, the folks who would be responsible for our affairs would have a detailed roadmap.

The document is broken into two parts:

  1. A step-by-step guide for everything that needs to happen, in a logical order, with links to all of the relevant documents, contacts, phone numbers, email addresses, etc. It is all right there in one place. Go to that note and you have everything you need at your fingertips.
  2. A kind of annotated index to my Obsidian vault, with links, so that if there is something that is needed that is not in the step-by-step section, there is a framework for locating it in the index section.

I try to keep this up-to-date, and it is still a work-in-progress, but there is enough there that it would be useful in the unlikely event of a water landing. Here is a high-level glance at the structure:

Once again, I’m focusing on finding things later when they are needed as opposed to a deep hierarchical structure to how the notes are organized.

Summary

Like any organizational structure, this is a work-in-progress. It evolves as I test things out, figure out what works and what doesn’t, and make adjustments. But this is the closest I’ve come to accomplishing an organization that doesn’t require me to constantly think about where I need to put something, and instead focus on linking to it in ways that will be easy to locate later on. It is an easy system to work with and so far, it is working well for me.

In next week’s post, I’ll talk about archiving notes.

Prev: Episode 17: Six Ways I Use Note Links
Next: Episode 19: Archiving Notes

Written on February 2-13 , 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Automating My Journal Entry Numbers in Obsidian with Keyboard Maestro

colorful toothed wheels
Photo by Digital Buggu on Pexels.com

Not long ago I wrote about how I went back to keeping my journal in Obsidian. I was going to try this experiment for all of 2022 to see how it worked out. I’ve flip-flopped on this over the last year or so, but you can check out the recent post as to why I made this decision.

Since I began this particular incarnation of my journal, back in 2017, I began giving each entry a unique entry number. I took this idea from Isaac Asimov, who used a similar method to simplify the indexing his of book Asimov’s Biographical Encyclopedia of Science and Technology. The journal spans multiple volumes (9 large Moleskine books and now, one file) and the entry numbers allow me to index the journal without worrying about which volume or page number the entry is on.

I am always looking for small efficiencies in workflow, and as I have been keeping my journal in Obsidian, I wondered if there was an easy way to automate entering the next entry number heading. It’s a small thing, but given that the journal is a text file, and each entry number a level 2 heading, it should be do-able. Here is how I managed to automate this:

Obtaining the next entry number

First, I wrote a command to search my journal file for all level 2 headings (that is, heading that begin with ## in Obsidian) and are followed by a numeric sequence. A typical heading entry looks like this:

## 2345

It took about five minutes of messing around with some Unix commands to do the trick. Here is what I came up with:

egrep "^##\s(\d+)" ~/Documents/DFC/Writing/Journal/2022\ Journal.md | tail -1 | sed 's/## //'

Running this command returns the number part of the last level 2 header in the journal file. For instance, when I run it right now, it returns: 2155

Here is how it works:

  • egrep "^##\s(\d+)": searches for any lines in the file that begin with ## followed by a space and then a sequence of one or more digits. Since ## 2155 matches this pattern, any lines in this format will be returned.
  • ~/Documents/DFC/Writing/Journal/2022\ Journal.md: the name of the file to search. That is, my current journal file.
  • | tail -1: take the output of the egrep command about, which will be a list of all the heading level 2 entry numbers in the file (like ## 2155) and filter it through the tail -1 command, which returns the last item in the list.
  • | sed 's/## //': take the last heading that comes from the previous command and filter it through the sed command to strip out everything but the number itself. What I am left with after this is just the entry number.

Getting the entry number into Obsidian

I use Keyboard Maestro for a lot of text expansion and miscellaneous automation. I decided I could use it here to get the entry number into Obsidian. I created a Keyboard Maestro macro called “New Journal Entry” that is triggered whenever I type ;;dd. (Note, this doesn’t apply to just Obsidian, it will do it when I type that sequence of keys anywhere.) The following macro is run when I type that key combination:

Here is how it works:

  • first, it executes the Unix command discussed above to obtain the next entry number from the file, and stores the result in a variable called CurEntry. If I ran this right now, the value of CurEntry would be “2155”.
  • next, it increments the value of the CurEntry by 1, making it 2156.
  • finally, it prepends the number with ## and inserts the value at the position of the cursor in the document. Keyboard Maestro automatically handles replacing the triggering text (;;dd) with the inserted value.

Here is what it looks like in action:

This might seem like a lot of effort to type out a number, but keep in mind, it took less time for me to create the automation than it did to write this post. Also, I’ve already got more than 2,100 entries in my journal and each time, I find myself having to check the previous entry number before entering the new one, and occassionally, I make a mistake, which is a nuisance to correct. This little macro eliminates all of that.

Perhaps even more important, it is one less thing I have to think about. Instead of sitting down to journal and first having to figure out the entry number, now I can just start writing.

There are probably other tools that could be used to achieve the same results. I’ve just happened to be a Keyboard Mastro user for a long time and have a cache of automations that I’ve created over the years stored there, so it seemed the logical place for this one.

Written on February 13, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Practically Paperless with Obsidian, Episode 17: Six Ways I Use Note Links

selective focus photoraphy of chains during golden hour
Photo by Joey Kyber on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

At a high level, I had three main requirements when I was looking for a note-taking app. I wanted an app that:

  1. Uses plain text files: first because they are accessible anywhere and require no special or proprietary software to access; and second because they are essentially future-proof.
  2. Does not require a cloud service or SaaS to use. Such a service could be optional, for syncing notes between devices, for instance, but the core functionality of the app does not require this.
  3. Has the ability to link between text files.

There are plenty of text editors in the world. Notepad on Windows and Vi/Vim in Unix environments, which have been around for decades, can read and edit text files. There are plenty of apps that are based on cloud services: Evernote is one example. Obsidian met all three of these requirements, but the one that really impressed me was its implementation of note-linking. So today, I’m going to focus on that and discuss six aspects of note linking that I find incredibly useful in Obsidian.

Anyone who has used Obsidian knows how easy it is to create a link to another note. You just tap the link key twice and begin typing the name of the note you want to link to. Once you have it, you hit enter, and your note is now linked. The ability to be able to link between notes is something I’d wanted in a text editor for a long time. Of course, it is possible to do this in a rudimentary way in other tools, like Evernote and OneNote, but neither tool uses plain text files locally on your computer, and neither links notes with the power that Obidian does. Two examples of that power that I find incredibly useful:

I can link and not worry about losing the connection. Obsidian handles any updates to links between notes behind the scenes. That is, if I have file A linked to file B and I move file B, the link between the notes remains in tact because Obsidian is keeping it up-to-date with any changes. The same is true if I rename the note.

I can link to any section within a note. In addition to being able to link to a note, I can link to any section within a note. I use this frequently in my daily notes file. Each top-level heading represents a single day, and if I want to link to a day in my daily notes file, I can do it using a link like this [[Daily#2022-01-28 Fri]]

2. Using note aliases

Another powerful feature of linking that I use is note aliases. A note alias is piece of YAML frontmatter that allows you to refer to a note by a different name. I use this for notes about people. I have, in my vault, notes related to family members which act as a place to see where I have mentioned them. For instance, suppose I have a note for James Bond. Anytime I refer to Bond in a note, I do so with a link to the note.

Now, because I want the list of people to appear alphabetically by last name, I name the people notes using the format “Last Name, First Name.” Bond’s note, therefore, is titled “Bond, James.”

Within the YAML frontmatter of the note, however, I have an alias setup that allows me to refer to the note as “James Bond” which looks better inside the text of a note.

The alias allows me to refer to a note either way: “Bond, James” or “James Bond” by automatically adding an alias to the note link when I insert it. Either alias points to the same note file underneath.

When I use the alias as a link, Obsidian automatically inserts the note alias into my note link so that it appears as “James Bond” in my note, even though it links to “Bond, James.”

I can think of all kinds of uses for note aliases, but this is a very practical one that I’ve been using quite a bit.

3. Linking to dates in daily notes

Last week I wrote about how I use my daily notes as an index to my life. In that post, I described how I use a single file for my daily notes rather than one-note-per-day the way the out-of-box behavior for daily notes is setup in Obsidian. One of the advantages of the one-note-per-day form of daily notes is that the title of the note is a date and the natural language date plug-in can create links based on a date, which is an easy way to link back to a daily noet for a given day.

It turns out, however, that this is easy to do using a single file for daily notes, and I’ve been doing this more and more. If I have a note that refers to a date, I’ll create the date in a link and the link will link back to the section of my daily notes file for the day in question. For example:

I’m writing this on January 28, 2022. I draft all of these posts in Obsidian so by using a note link for the date in this link, I am connecting back to my daily notes file as a backlink to that file. The date link itself looks like this: [[Daily#2022-01-28 Fri]]. If I now go look at my daily notes, I’ll see the backlink to this draft because I’ve linked to the section of my daily notes file containing the date in question:

I can do this for dates (sections) that already exist in my daily notes file, or I can do it for a future date. Once that future date appears in my daily notes file, the backlink to the referenced note will appear as well.

One feature I’d like to see added to Obsidian related to backlinks is the ability to indicate which section of the note file the link appears in. In addition to showing the note from which the backlink comes, it would show the path to the section in the note. So if the reference took place in a section of the document under an H1 head called “Posts” and an H2 heading called “Series”, the backlink would indicate the document name followed by something like “Posts > Series.” That would be a useful improvement.

Backlinks provide additional context for a note I am looking at. I’ve come to rely on backlinks quite a bit. As I demonstrated in the previous section, a backlink is a link to the current note from another note. By referencing other notes using links, I can easily get a summary of activity, for instance on a particular note, or see how that notes relates to other notes, ideas, or thoughts.

For instance, I recently had to change the battery in the car fob for our car. I’ve done this before and had instructions written for it. Not long after that, I had to replace the battery in Kelly’s fob. Looking at the instruction note with the backlinks, I can see references to changing the battery in my daily notes file:

If I click on the backlinks, I can see the exact date on which I replaced these batteries. When I replace them again, I’ll know how long they last. This is also an example of where I think it would be useful for a backlink should also indicate the section of the note the link appears in. If this feature were available, I could see the dates directly in the backlink because I use dates as a section header in my daily notes file.

Sometimes, when writing notes, I find it convenient to add placeholder links while I am typing, and come back to them later to fill them in. Obsidian makes this easy. I can just create a note link as I would for any other note, and move on. Clicking on the link will create the note, but I don’t have to create it first and then link to it later. This has advantages because I may not want to lose a train-of-thought. It helps me to organize thoughts at a high-level , creating a pseudo-structure. I tend to use placeholder links in two scenarios:

  1. When writing thoughts on something I’ve read, and trying to see how that fits in with other things I’ve read or thought about.
  2. When referring to future dates. Linking to a future date means the link will work once that date section appears in my daily notes file.

I can see these placeholder links the graph view. They are the dimmer dots. This provides an easy way to see all these “to-be-created” notes at a high level:

![[Pasted image 20220201174346.png]]

My notes frequently link outside of my vault–that is, to web links to other sites. One plug-in that has saved me a ton of time for these external sites is the “Auto-link title” plug-in. I can grab a link from my browser and when I paste the link into Obsidian, it automatically goes out to the site, extracts the title of the page, and creates an Obsidian link that includes the title so that it renders cleaner in my notes

For instance, if I paste in a link to the list of books I’ve read since 1996, the plug-in will grab the title of that page and convert it to a nicely formatted link in my note:


In next week’s post, I’ll talk about how I organize my notes in Obsidian (so far). See you back here then!

Prev: Episode 16, Finding Notes Quickly
Next: Episode 18: How I Organize My Notes in Obsidian

Written on January 28-February 1, 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com