Tag: obsidian

Practically Paperless with Obsidian, Episode 2: The Basics: My Obsidian Configuration

settings android tab
Photo by Pixabay on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.

In the first episode in this series, I discussed the basic objects in Obsidian: notes and documents. In this episode, I want to step back and talk more broadly about Obsidian itself. In order to best understand I how I use Obsidian as an Evernote substitute, I want to highlight some features of Obsidian that I use in order to get the most out of it for my paperless notes. There are two reasons I am doing this:

  1. There are some features that are not enabled by default but that I have found particularly useful.
  2. I know from experience that people will ask about what themes and plug-ins I am using in Obsidian and this makes a nice post to which I can point people for an answer.

Also, I am trying something new beginning with this episode:

  • I am creating animated GIFs to help illustrate some of the things I discuss1, something I never tried with my Going Paperless posts, so please bear with me as I figure this out. It’a a learning experience for me.

My preferred Obsidian theme: Yin and Yang

One limitation I found in Evernote is that I never had much control over the look and feel of the tool. The current version2 as of this writing allows you to configure Light or Dark mode, but that’s about it. This may not seem that important. After all, Evernote and Obsidian specialize in storing information (the former in the cloud, the latter locally in your filesystem). More and more, however, modern text editors and IDEs are being designed with a great deal of flexibility in how they look and feel. Editors like Atom, Sublime, and IDEs like Visual Studio Code all allow customization of the user interface through the creation of themes that manipulate the styles of objects that appear on the screen. Obsidian is among these tools. In Obsidian, themes are nothing more than CSS files that you can download from a community, or even create on your own.

My preferred Obsidian theme is called Yin and Yang and can be found in the Obsidian by going to Settings > Appearance > Themes, and clicking the Manage button to view a list of community themes. Obsidian themes can be used dark and light mode. Given how much time I spend on screens, I prefer dark mode.

So how do themes alter the look and feel of Obsidian? Let me illustrate. Below is what Obsidian looks like out-of-the-box in light mode (and notice that the Obsidian Help is just another Obsidian vault):

And here is what Obsidian looks like when I change it from light to dark mode:

Switching to Dark mode in Obsidian

With those images in mind, here is how Obsidian looks with the Yin and Yang theme. The image on the left is a note in edit mode; on the right is the same note in preview mode.

This is my own preference. You can keep the default theme, use one of the 70+ community themes, or create a new theme entirely on your own3

A few additional useful UI tweaks

Folding headings and lists

As I mentioned in Episode 1, notes are nothing more than plain text files that use Markdown to format the content. One nice feature that Obsidian comes with out of the box is the ability to fold your headings and lists. These features have to be enabled in the Editor settings as shown below.

Once you’ve enabled these features, you can open and close headings to show and hide the text within the heading. Using my note on my Retro Posts above as an example, here is how folding headings work:

The same works for lists. If you have a list, like an outline, each level of the list is foldable. I find both of these features very useful for focus. If I want to concentrate on just one part of a note, I can easily fold other parts so that they don’t distract me.

Useful “core” plug-ins

Obsidian comes with “core” plug-ins that are packaged with the application. There are also community-based plug-ins. In this section, I’ll talk about the “core” plug-ins that I find most useful in my pursuit of going practically paperless.

Daily notes

Daily notes are a fundamental part of Obsidian, and they are also a fundamental part of my efforts to go practically paperless with Obsidian. They are so important, that I’ll have an entire episode dedicated to them in January4. Daily notes are simply notes that you can associate with a given day. These notes have a special naming convention that uses the date to form the name of the note. This makes it useful when linking other notes to this date. Obsidian knows about all of the “backlinks” to a note–that is, all of the other notes in your vault that are linked to that note. I have actually automated my daily notes so that they are generated automatically each night, and pull in information from my calendars, making them even more useful.

Daily notes act as a kind of index to many of my other notes in Obsidian. They take the place of my timeline concept in Evernote.

An example of a daily note in Obsidian
An example of a daily note in Obsidian

Daily notes can be enabled in Settings > Core Plugins.

Starred notes

At any given moment, I have one or more notes that I use frequently, and want to be able to access quickly. This is what “starred” notes are for. After enabling this core plug-in from the Settings > Core Plugins menu, you can “star” a note. Once starred, that note will be available for quick access on the Star panel on the lefthand side of the screen. Starring a note is like creating a note shortcut in Evernote.

The starred note panel displays a list of notes that have been "starred" for quick access.
The starred note panel displays a list of notes that have been “starred” for quick access.

Zettelkasten prefixes

This one is a mouthful, but it refers to a fascinating method for organizing notes. I don’t use this method in its strict interpretation, but I have borrowed liberally from it for my own notes–especially my reading notes. Because of Obsidian’s ability to link notes and illustrate the relationships between notes, it has become a particularly useful tool for those who wish to have a digital Zettelkasten.

But back to the plug-in. This plug-in does 2 things:

  1. It allows you to define a “prefix” for your note titles based on a date format. If you’ve been paying attention, you may have noticed that many of my note titles begin with a long number combination like 202110111506. This number is just the current date and time in yyyymmddhhmm format. When I create a note using a prefix, Obsidian automatically generates the prefix and I can add more to the title if I want. Having the date in this format is more powerful than it may seem. When searching for notes, for instance, I have a convenient way of searching by date, something I will discuss in much more detail in Episode 11.
  2. You can also, optionally, define a template for your note so that when your note is created, not only is there a title prefix, but there can be other information preloaded in the note. This can save a lot of time, and help with standardization. When I create a new note, here is what my basic template looks like:

I’ll have more to say about Zettelkasten and note titles in Episode 6.

Useful community plug-ins

In addition to having community-developed themes, there are also some amazing community plug-ins that have been developed for Obsidian and the list keeps growing. Here are the community plug-ins that I find most useful:

Calendar

The calendar plug-in provides a quick way to get to your daily notes (if you are using them). Since I use daily notes constantly, the calendar provides an easy way to navigate quickly to the note I am looking for. In this example, I am using the calendar to navigate to my daily note from October 3:

The dots in the calendar are days in which I have daily notes. The more dots on a given day, the longer the note. It is a very cool plug-in and the first community plug-in I installed when I began using Obsidian.

Natural language dates

Dates are important in my paperless taxonomy because they tell when things occurred. Obsidian allows you to format dates in any desired format. My daily notes use the format yyyy.mm.dd.DDD, or 2021.10.11.Mon. If I create a note link in Obsidian to “2021.10.11.Mon” it will link to my daily note from that date.

The Natural Language Dates allows you to quickly create these note links using natural language, like “yesterday”, “today”, or “tomorrow”. I’ll use this frequently on in timeline sections of notes that refer to dates.

Using the plug-ins

These plug-ins make it much easier for me to zip through creating notes, linking them together the way want to, and have the notes appear the way I want on the screen. As we progress through future episodes, you’ll see me using these plug-ins frequently. I wanted to have a post to which I could refer people to list those plug-ins and setting that I find most useful. There are other plug-ins that I use, but not as frequently, and I’ll discuss them in the context of how I use them to perform certain activities in future episodes.

In episode 1 I showed the basics of notes and documents in Obsidian. In this episode, I discussed the settings, themes, and plug-ins most useful to my note-taking in Obsidian. Both episodes set the stage for next week when I illustrate how I emulate basic Evernote features in Obsidian. Hopefully this will provide a like-for-like comparison of how I did basic things in Evernote and how I can do those some basic things (with improvements) in Obsidian. See you back here next week!

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. For those wondering, I am using GIPHY Capture for the Mac to create these GIFs.
  2. v10.22.3
  3. At one point, as an experiment, I created a theme to make Obsidian look like Word for DOS 5.5, my all-time favorite word processor.
  4. Episode 16, if you are curious.

Practically Paperless with Obsidian, Episode 1: The Basics: Notes and Documents

white stacked worksheets on table
Photo by Pixabay on Pexels.com

Welcome to my blog series, “Practically Paperless with Obsidian.” Foran overview of this series, please see Episode 0: Series Overview.

Nearly 10 years ago, I began an experiment to see if the elusive paperless office was actually possible. That series, Going Paperless, was my attempt to use Evernote and other tools to go completely paperless. After several years, my conclusion was that it was not really possible for me to go completely paperless. In the years since, I’ve returned to paper for several things, but still enjoy the efficiencies of having information I need at my fingertips. In examining the lessons I learned from my paperless experiment, I realized that, as with most things, moderation is key. There is a difference between going completely paperless, and looking to be paperless in the practical sense. That is what this series is all about.

Instead of Evernote, I’ve decided to use Obsidian instead. I’ve written about why I want to use Obsidian elsewhere, but the gist of it is:

  • files are plain text, which makes them essentially future-proof;
  • files are stored locally instead of on someone else’s server (unless you want to store your files in a cloud system like iCloud, Dropbox, OneDrive, etc.);
  • it has a great note-linking function that I will make heavy use of as we progress through this series.

Of course, if you are following along, you don’t have to use Obsidian. Evernote still works for much of what I’ll be discussing. If you are not going to use Obsidian, you can safely skip the first three episodes of this series, as they focus on setting a kind of baseline with the tool for moving forward through subsequent episodes.

The first 20 episodes in this series build upon one another. I am using them as a guidepost for getting me to where I want to be. The first 3 episodes establish some basics, beginning here with how I plan on storing my notes. A note can be anything, text, a document, and image. When I think about what I want to be able to capture in digital form, I think of notes in two categories: notes and documents.

Notes in Obsidian

A note is just a markdown file (.md) file in Obsidian. Markdown, for those not familiar, is a plain text file in which special markup can be used to format the note. This is light markup, not as elaborate as, say HTML. In a plain-text markdown file, for instance, if I want to bold some text, I surround it with a double asterisk **like this**.

For me, notes are distinguished from documents in that a note is a markdown file. A document is something else, like a PDF or an image file. I’ll discuss those files in a moment. Notes can be viewed in two ways within Obsidian. They can be viewed in edit mode, where you can see the markup’s that you add to the note; and they can be viewed in Preview mode, which renders the notes fully formatted. Here is an example the same note rendered in edit mode and preview mode in Obsidian. You can use the slider bar to see how they look different.

Figure 1: Comparing a note in edit and preview mode.

Notes are where the majority of my paperless stuff goes. The note in the example above is from my “commonplace” notebook, a collection of notes and highlights from my reading. At its most basic, a note in Obsidian is a file on your file system. The note has title1, and it has the same file attributes as any file on your file system: create date, modified date, permissions, etc. I’ll have more to say about note titles in Episode 6.

Obsidian uses the concept of a “vault” to store notes. A vault is nothing more than a folder on your computer. Obsidian controls and monitors the files and folders within that vault folder. This is incredibly useful. It means that you can move notes around within your vault and Obsidian will take care of maintaining the links that notes have to other notes automatically.

Note-linking is a key reason why I love Obsidian and I’ll have a lot more to say on it in Episodes 17 and 19. For now, note links are simply links to other notes in your vault. In the example above, the “[[202109220957 Natural Questions]] on the “source” line is an example of a note link. Clicking on the link takes you to that note.

Just like in Evernote, notes can have tags. Obsidian uses the hashtag format for tags. In the example above, you can see two tags: #discovery and #favorite. One difference between tags in Obsidian and Evernote is you can refer to tags anywhere in a note in Obsidian. And because Obsidian’s search capabilities are very granular, that means that sections or even lines of a note can appear in search results, making tags quite powerful. I’ll have more to say about tagging in Episodes 7-10.

Notes, then, are containers for information you want to capture. They are the basic unit of storage in Obsidian. You can create notes quickly with a hot key and start typing. Obsidian saves as you type so you don’t have to worry about remembering to click a Save button.

Notes have one other very powerful feature in Obsidian that they lack in Evernote: transclusion. Transclusion allows you to include a note within another note. When I read, I highlight passages, and those passages, after I review them, each get their own note in Obsidian. Here is an example from when I was reading Franklin D. Roosevelt: A Political Life by Robert Dallek:

Figure 2: An example note in Obsidian

For books with notes, I keep a “source” note to collect all of the notes related to that book together, as well as any other thoughts I might have about the book not captured in a specific note. Rather than have to copy and paste the notes into the source note, I can “transclude” the note into the source instead, thus reusing the existing note, almost as if it were a subroutine. Here is my source note for the book in Edit mode:

Figure 3: A note with transcluded note links in edit mode

In the “Notes” section, note that all I did was include links to my notes on the books. Note the ! that precedes each note link? That is what tells Obsidian to transclude the note in Preview mode. So when I look at this note in Preview mode, what I see is:

Figure 4: A note with transcluded note links in Preview mode.

Each of the two note links are transcluded–they include the entire note–within the note in which they are references. This turns out to be incredibly useful with documents.

Documents in Obsidian

I think of a document as something other than a note. Much of what I collected in Evernote over the years were scanned PDFs, or PDFs automatically sent to Evernote through a service like FileThis. Think: bank statements, tax forms, official documents, instructions for household appliances, etc.

Obsidian has the ability to keep track of and render certain types of documents files, among them PDFs and image files. You have the ability to store these files right along with your notes, or you can separate them out into their own folders in a number of different ways. I’ll have more to say on this in Episode 2.

In order to keep things simple, I created a folder called “_attachments” in which any and all document files go. This includes PDFs, image files, and any other files that fit this category. The reason for this is that I don’t just use the bare attachment file, but I couple it with a note in order to gain the benefit of all of the Obsidian functionality that comes with notes. Let me give an example.

We recently got a new microwave oven, which I ended up installing myself. Information about electronics and appliances is something that I actually use from time-to-time, and just as I did in Evernote, I created a note for the new microwave in Obsidian.

Figure 5: My new microwave oven note

I use a similar format for all electronics and furniture because it means I have one centralized place to go for anything related to that thing. Obsidian has the ability to create templates for notes, similar to Evernote. I’ll discuss this in more detail in Episode 8. The “Timeline” section is a running timeline of events related to the microwave. If, for instance, I had to call for support, I’d add an entry to the timeline to record information about that call to support. All of the information I need is right there and easy to locate.

Note that there is a translcuded note link for the Owner’s Manual PDF file. I put the PDF file in my “_attachments” folder, but I don’t have to worry about where it is. When I go to add the link, I just start typing and Obsidian presents me with a list of matches anywhere in my vault. Because it is a transcluded note link, when I look at this note in Preview mode, what I see is:

Figure 6. The microwave note in Preview mode with the PDF included

I have the ability to scroll through the pages of the PDF, or print the PDF if I want to. It’s all right there, included as part of the note on the new microwave oven. Many notes are just the document itself, so the note will contain nothing but a title, some tags, and then a transcluded link to the actual note, say a bank statement or tax form. This gives the added benefit of searching the meta-data in the note to find what I am looking for. I’ll illustrate more examples of this when I talk about finding note in Episodes 15-17.

Notes are just files in the filesystem

I want to stress the point that these note are just files in my file system. This is a big difference from Evernote, which stored notes as objects on their server, which could be downloaded to your client. Here is a look at what my vault in Obsidian looks like (on the left) and a similar look at what this looks like on my filesystem (on the right):

The “DFC” is the name I gave my vault in Obsidian. It stand for “Digital Filing Cabinet.” I can open any of these notes outside of Obsidian in a text editor and still be able to read and use (and even update) the note. For instance, opening the microwave note looks as follows on my Mac’s TextEdit app:

Figure 8. An Obsidian note opened in a text editor app.

Establishing a baseline

I wanted to begin this series with something simple, illustrating how notes are captured in Obsidian, because I wanted to establish a baseline. I wanted to give people who are used to using a tool like Evernote or OneNote an idea of how their notes might be stored in a tool that is essentially a fancy text editor. To that end, I identified two types of notes that I capture–notes and documents–and showed how I am using these separately and in combination to capture notes in Obsidian similar to how I captured them in Evernote. I was trying to answer the question: what would my notes look like in Obsidian?

Again, as I attempt to go practically paperless, I’m using Obsidian because it is simple, future-proof, and doesn’t require paying for cloud service if you don’t want one. Documents are stored locally and because they are plain text files with some PDFs and images in the mix, you can use your OS to manage the files, and even search the files. Running a Spotlight search for “LG Microwave” on my Mac instantly returns the following:

Figure 9. A spotlight search turns up files in my Obsidian vault.

But I like Obsidian because of its note-linking capability, as well as its ability to manage the vault, keeping links updated even as I move notes around in the vault. It also has some powerful search capabilities that even Evernote lacks (like regular expression searches). And it provides an interface that separates my notes from other things that I do. This series will focus on using Obsidian as I attempt to go practically paperless, but I hope it is clear that other tools can work as well. This just happens to be the one that I think is best suited for this task.

Next week: Continuing down the path of establishing the basics for this experiment, next week will focus on Obsidian and how I have configured it to take advantage of features and plug-ins that I think are most useful for managing my notes. A week later, in Episode 3, I’ll show how I emulate some of Evernote’s useful features in Obsidian.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. You’ll notice that my note titles begin with a long number. That is by choice, I will explain why I do this in more detail in Episode 6. For now, don’t worry about it.

Practically Paperless with Obsidian

I. Birth of an idea

For a while now, I have wanted to clean up my Evernote instance. I’ve got over 12,000 notes there. The problem is:

  • I’ve never looked at 80% of the notes. They get in automatically through services like FileThis and I’ve never had need to access those notes.
  • I’ve looked perhaps once at another 10% of the notes. For some of those notes that is once in over 10 yerrs.

It is the remaining 10% (or less) that I interact with on an ongoing basis. Out of 12,000 notes that’s less than 1,200. With this in mind, I’ve been considering starting from scratch with Evernote, taking all of the lessons I’ve learned and applying them anew. It seemed to me there were two ways to do this:

  1. Create a new Evernote account and start from scratch there, unfettered by the clutter in my existing account. I could then plan my attack and over time begin migrating those notes that I wanted to keep to the new account. When I’d migrated everything I wanted, I could export the remaining notes to an export file and archive it so that I could access it if I ever needed it. After that, I could delete all of the unused notes from Evernote and proceed with the new account. The downside would be maintaining two Evernote accounts for some unknown period of time.
  2. Create scaffolding within my existing Evernote account, partitioning with notebook stacks, or tags, or some other mechanism, and then moving things around as necessary to create the structure I am looking for and eliminate all of the notes that I don’t need.

While thinking about this, a third option began to form in my mind, aided by an email exchange with reader, and frequent commenter Jaap van Dodeweerd: if I am going to start fresh, why not start completely fresh with a different tool? Why not try going paperless with Obsidian? I have been using Obsidian since January, and have written frequently about it over the past several months.

Obsidian

For those who are not familiar with Obsidian, it is a knowledge base tool that combines a plain text markdown editor with tools that allow you to link notes together and visualize the links between them. Evernote’s slogan was “Remember everything.” Obsidian’s is similar: “A second brain for you, forever.” Obsidian is “future-proof” in two regards:

  1. The notes are composed text files that use markdown to style them. That eliminates the need for future compatibility. Text files are the most basic form of human-readable digital files. They have been around for half a century and can be read, updated and viewed with tools on any computer.
  2. The text files live in a folder structure on your local computer. That eliminates the dependency on cloud-based services for those who don’t want them.

Of course, Obsidian renders those plain text files nicely. And, of course, if you local folder structure happened to be in iCloud, or Dropbox, or OneDrive, well, then you’d have those notes available anywhere you needed them. But it isn’t required. Here is an example of what a plain text markdown file looks like when rendered in Obsidian.

Example of a plain text file rendered in Obsidian
Example of a plain text file rendered in Obsidian

Obsidian also offers several features I’ve wanted in Evernote for a long time now:

Linking notes

An easy way to link note, and different ways of seeing those note links. You can link notes in Evernote, but I have always found it cumbersome. And there is no way to see backlinks–what notes are linked to a specific note. In Obsidian, it is incredibly easy to link to a note just by typing. A sidebar will display backlinks, for instance:

An example of note links in Obsidian

Transclusion

Transclusion is an incredibly powerful feature that allows you to make a note file act as a subroutine. You can create a note, and then “transclude” (that is, include a link to the first note in such a way as that it is rendered in the second note) the note within other notes. The first note will appear as though it is part of the other notes. Changing or updating the first note will update it in the other notes as well. This makes note reusable without duplicating them everywhere.

Last Viewed Date

One thing that makes it difficult to purge Evernote of unneeded notes is that there is no way of searching for notes by the last time they were opened or viewed. Evernote allows you to search note by create date and modified date, which can be useful. But if I wanted to see how many notes hadn’t been viewed in the last 10 years, I have no way of doing that with create and modified dates.

Because Obsidian uses your operating system and plain text files, you can use your operating system functionality to search notes by “last opened date” (on MacOS) or last accessed time in Linux and Windows. With that, there is an easy way of seeing notes that haven’t been opened (accessed) in more than 10 years (or whatever your preference is) and this is an incredibly useful tool for getting rid of noise.

Obsidian versus Evernote

In thinking this through, I tried to consider the features I use in Evernote and if there was a similar feature in Obsidian. It wouldn’t make sense to move from one to the other and lose functionality. The table below summarizes my thoughts on this.

FeatureEvernoteObsidian
Formatting notesEvernote has been improving its ability to format notes and the most common formatting features (including, most recently, code blocks) are available.Obsidian uses Markdown to format notes. Anything you can do in Markdown, you can do in Obsidian. This is fairly robust formatting, and probably more than I’d ever need.
NotebooksEvernote allows you to organize notes in notebooks. Notebooks cannot be nested, but you can collected notebooks into a notebook stack.In Obsidian file system folders are the equivalent of notebooks. They can be nested as deeply as you like.
TasksEvernote recently introduced the concept of tasks into notes. Tasks can have due dates, reminders, and be assigned to people.Obsidian has the concept of tasks in markdown. A task can be completed or incomplete.
TagsEvernote notes can be tagged and searches can use tags to narrow the field.Obsidian has the concept of tags and some themes even render them nicely in the UI. See the note link image above for an example.
ShortcutsEvernote provides a quick way of getting to notes, notebook, saved searches, tags, etc. by having a shortcut to the object in question.In Obsidian, a note can serve as a shortcut to another note (via links or transclusion). Searches can be embedded within notes as well, which replicates the “filtered notes” feature of Evernote.
Mobile appEvernote has a robust mobile app that syncs with your notes anywhere.Obsidian has a mobile app that also allows you to sync with your notes anywhere, if you choose to keep your notes in some cloud-syncing service like iCloud or Dropbox. Obsidian also provides its own syncing service.
Note sharingEvernote allows you to share notes with others.In Obsidian, since notes are just files on the OS, you can share them with others the same way you’d share any file, either through a cloud service, or simply by copying the file into an email or other kind of instant message.
SearchingEvernote has robust searching capabilities that include searching within the text of PDFs and images. Its advanced search grammar can be tricky to learn, but it is powerful.Obsidian comes with powerful native search capabilities. PDFs and image files can be stored in Obsidian vaults and if the PDFs are scanned with full text, then they can be searched by the OS. Also, Obsidian notes can be searched by regular expressions, something I’ve wanted in Evernote for a long time.
A comparison of features between Evernote and Obsidian

Going down the list, I found that there was nothing that I regularly did in Evernote that I couldn’t readily do in Obsidian. Feature limits would not stand in my way. I could begin this new adventure.

II. Learning from my mistakes and improving upon the past

Well, I wasn’t quite ready to begin. I decided that if I was going to go through the effort of going (mostly) paperless in Obsidian, I should look to my past mistakes with going paperless and also see how I can improve upon what I’d done before. For the former, I have a number of ideas in mind. For the latter, I have 136 posts I’ve written on going paperless which serve as a wide-ranging buffet ripe for improvement.

The lesson for me: Going paperless is not a binary state. It means using paper where it is useful and practical. It means capturing notes and digital documents that are useful. It means setting a clear bar for what goes in Obsidian, what goes on paper, and what I don’t need to worry about at all.

Learning from my mistakes

Paperless is a not a binary state

In my original going paperless experiment, my goal was to see if it was possible to go completely paperless. I was chasing the elusive paperless office to see if such a thing was possible. Two important lessons came out of my 4-year experiment between 2012-2016:

  1. I could go completely paperless but the world was not going paperless anytime soon. I still routinely dealt with other people’s papers and tried to develop a simple process for managing that paper in 10 minutes a day.
  2. I found that I didn’t want to go completely paperless. For some things, paper is more useful; for other things, paper is a more enjoyable experience. In the former case, I have never found a notes app (Obsidian included) that allows me to jot down notes as quickly and easily as I can with a pen and my Field Notes notebooks. In the latter case, I simply prefer keeping my journals on paper like John Adams, John Quincy Adams, Henry David Thoreau, Samuel Pepys, and countless others.

Then, too, I kept everything I got in that four year stretch, much of which (80%) I’ve never even looked at.

The lesson for me: Going paperless is not a binary state. It means using paper where it is useful and practical. It means capturing notes and digital documents that are useful. It means setting a clear bar for what goes in Obsidian, what goes on paper, and what I don’t need to worry about at all. It was these last two categories that I never really considered in my first experiment. This time around I’m looking toward Practical Paperless: the idea that its important to keep some things well organized in a digital archive, and other things on paper.

Improving on what’s come before

By now it should be clear that I’m not just planning on copying notes out of Evernote and into Obsidian. I am looking for ways to improve upon everything I’d when using Evernote to go paperless. It means a full review of all the posts I wrote to see where improvements can be made. It means eliminating things that didn’t work for me, or weren’t that useful. It means beginning with a plan.

III. A new “practically paperless” series

Of course, in taking off in this new direction, I plan on writing about it here on the blog. I’m calling the new series “Practically Paperless” to distinguish it from my original series. There is equal emphasis on both words with “practically” used an adjective for practical: that is, I plan on writing a series about going paperless with Obsidian, where it is practical to do so.

The series will begin at the beginning, with a plan for moving forward, something I lacked with my original series. That plan will include the requirements that are important to me in this effort: what stays on paper? What gets captured in Obsidian? From there, I’ll move into posts on the basics: setting up a framework that allows me to meet the requirements. And from there? I suspect we’ll all be discovering new things along the way.

My Going Paperless series started as weekly thing, and eventually moved biweekly. Based on that experience, I think that the new Practically Paperless series will start on a biweekly schedule and I’ll adjust things from there. More than likely, each post will appear on Tuesday mornings, like this one. That means you can expect Episode 1 to appear two weeks from today, on Tuesday, October 5.

IV. What about Evernote?

I am not moving away from Evernote because of any problems I’ve had with the service, or any problems I foresee in their future. I tend to be a creature of habit in most areas of my life, but when it comes to software, I a kind of wanderlust. I’m always looking to try new things that I think can help me improve whatever it is I am trying to do. In this case, I’ve been using Obsidian long enough to see that it works really, really well for me. Not only do I think it can replace Evernote, I think it can improve upon what I was doing with Evernote. That is what this new series is all about.

That said, not everyone will like Obsidian. Not everyone works the same way that I work. Many, many people are happy with Evernote and if you are happy with something, you should stick with. This is an experiment my part, one that I hope will prove successful, but you never know. My experiment with Evernote worked out well, and I was happy user for more than ten years. I’m still a happy user, but that wanderlust of software is calling me again.

So I’m going practically paperless with Obsidian. As Al Bean said to Pete Conrad in H.B.O.’s From the Earth to the Moon, “Y’all can come along with me if you like.”

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

The Weekly Playbook #4: Finishing a Notebook: Transcribe or Scan?

For an overview of this series, see the debut post on my morning routine.

Background

I am rarely without a Field Notes notebook in my back pocket. Several times a day, I pull out my notebook to jot something down: an idea for a post; notes from a podcast; the names of people I meet; items to pick up at the grocery store; the name of the server in the restaurant we ate at; funny lines I hear at a gathering. I do this so that I can remember these things later. Some of them find their way into posts I write, some into stories. Other things are more ephemeral, but even a server’s name in a restaurant can be useful if I am searching for a character name in a story.

I’ve filled more than 30 of these notebooks since 2015. They sit in a nice row on a shelf in my office. Occasionally I go back to them, to look for something, like when I was searching for a particular brand of beer recently. The problem is, I only have access to them when I am sitting here in my office. It would be nice to have access to them no matter where I was.

my 30 completed field notes notebooks with an index notebook on top
My 30 completed Field Notes notebooks, with an index notebook on top.

This weekly playbook is a kind of experiment. I began with the idea that I wanted to be able to access these notes anywhere. I had two ideas:

  1. Transcribe the notebooks into Obsidian, where my other notes live, or
  2. Scan them into Evernote

I decide to try both in order to see what worked better for me. The playbook section below has the procedures I followed for each. In each case, I used my most recently completed notebook, book #30. I’ll describe my findings in the commentary.

Playbook

Transcribing notebook into Obsidian

  1. Create a Field Notes folder in Obsidian
  2. Create a new note called “Book 30 – March to June 2021.
  3. Begin typing in the notes using the following guidelines:
    • Make each “day” a header in the notes
    • If my handwriting is unintelligible, put question marks and move on.
    • Wherever I have a dividing line in my notebook, include a divider in the notes file
    • Use only one file per notebook

Scanning notebook into Evernote

  1. Create a Field Notes notebook in Evernote.
  2. Using the Scannable app by Evernote, scan in all 48 pages of my notebook #30, including the cover and inside cover.
  3. Once scanned, put the note in the Field Notes notebook
  4. Title the note “Book 30 – March to June 2021”
  5. Set the create date of the note to March 1, 2021
transcribed notebook page in obsidian
A transcribed notebook page in Obsidian
scanned notebook page in evernote
A scanned notebook page in Evernote

Commentary

It probably took me an hour to transcribe the first 15 pages of the Field Notes notebook into Obsidian. After an hour I stopped. It is easy enough to estimate that a full notebook would take me a little over 3 hours to transcribe.

On the other hand, it took about 15 minutes to scan the entire notebook into Evernote using the Scannable app. (I think Evernote’s Scannable app does a slightly better job at scanning than the regular iOS app does.)

For me, the Evernote scan is the better over all option. There are several reasons for this:

  1. It is quick enough to make it worthwhile. Investing 15 minutes to have the contents of the notebook available to me anywhere is a worthwhile investment of time. 3 hours is a little much. I am not likely to invest 3 hours, but 15 minutes is no big deal.
  2. The notebook really is available anywhere. The screenshot above is from my phone. I can flip through the pages just as I can with any PDF.
  3. Scanning preserves everything in my notes, include occasional sketches and diagrams that I make.
  4. Evernote uses its AI to attempt to make the PDF searchable. It is supposed to be able to recognize handwriting. I made several attempt, but I think my handwriting is too messy. Still, for people with very neat writing, the notebook is searchable. I keep the notes in their own notebook in Evernote for this reason: when I want to search for something in a Field Notes notebook, I can limit the search to notes in the Field Notes notebook so that I don’t get results from other sources.

There are a few cons to using Evernote over Obsidian:

  1. The notebook is not as searchable as it would be if I transcribed it into Obsidian. I could probably find things faster in Obsidian.
  2. My notes would be in plain text format and could be manipulated like any plain text.
  3. I could do more dynamic linking of my notes to other notes using Obsidian. (You can link to other notes in Evernote, but there is no practical way to do this in scanned documents.)

Another consideration is that I want to get my entire backlog of notebooks in a format that I can access anywhere. Transcribing 30 notebooks into Obsidian would be an investment of nearly 100 hours of my time. Scanning 30 notebooks into Evernote is an investment of 7-1/2 hours. From a practical standpoint, this is a no-brainer.

Then, too, since the notes already exist, they fit into the model of using Evernote for curation and collection, and using Obsidian for creation.

Remember, my goal at the outset was to be able to access the notebooks from anywhere. My goal wasn’t to make them as searchable as they could be. I’m fine flipping through a PDF to find what I am looking for. It usually doesn’t take very long, so it seems like the investment in time to manually transcribe all of my notes would be overkill.

Going forward, when I finish a notebook, I’ll follow the procedures for scanning that notebook into Evernote.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Small Efficiencies in Workflow

With my recent plan to focus on my writing and improve my overall well-being (a.k.a. Project Sunrise), I have been hunting for small efficiencies in workflow that can have an outsized impact on my day. My morning routine takes about two and a half hours to complete. While developing the routine, I teased out actions or tasks that I could eliminate or improve upon to maximize the use of my time. Two examples come to mind.

Writing in my journal: content versus medium

Since 2017, I have been writing my journal longhand in large Moleskine notebooks. I’ve written about the advantages and disadvantages of having a paper journal versus a digital one in a piece called The Paradox of Journaling. I like the feeling of writing longhand, and I understand and believe in the durability of paper. But there are two tradeoffs to consider when time is limited and my goals depend on data:

  1. The speed and clarity with which I can write.
  2. The speed an accuracy with which I can find what I wrote about.

With limited time, I had to consider what is more valuable to me now, the content of my journaling or the medium in which it is stored. Today it is the content. Since I can type much faster than I can write longhand, since my typing is more clear than my handwriting, and since I express thoughts more clearly through a keyboard than a pen, it seemed prudent to switch my journaling to a digital form instead of a paper one. This is why for the last week, I have been composing it as a text file using Obsidian, despite what I wrote in February when I initially rejected the idea. The reasons I rejected it were valid then, but circumstances have changed, and I think this little efficiency will have long-term benefits.

One of those benefits is the speed with which I can find what I wrote about. It is much easier to search a text file than volumes of journals, even when they are roughly indexed. And time is the key. I want to spend as much of my time as possible on creative tasks. That said, to improve, I need to look back at the data I’ve collected so that I can apply it going forward. I can do this much more quickly searching a text file than books. Practical considerations–speed of input, clarity, and speed of retrieval–have overridden my desire to continue writing my journal longhand, at least for the duration.

Composing in WordPress

For a long time, I composed my blog posts in an external editor. That editor has changed over the years. I’ve written drafts in Scrivener, in Word, and most recently, in Obsidian, my current editor of choice. With my recent migration to WordPress hosting, and conversion to a modern WordPress theme, I have found WordPress’s native Gutenberg editor to be comfortable and easy to compose in directly. This saves a good deal of time. Prior to composing directly in WordPress my process looked like this:

  1. Write the post in Obsidian (or other editor)
  2. Copy the text out of Obsidian
  3. Paste it into a blank WordPress post
  4. Fix any formatting issues
  5. Publish.

For the last week I have been composing directly in WordPress which allows me to eliminate the administrative steps I was doing before. This shaves a little time spent on each post, which I get back for creative work, like writing the posts themselves.

These are small efficiencies. They don’t save huge chunks of time each, but the affect is cumulative. I journal in the morning and evening, so I am saving a little time each journaling session. I tend to write in the mornings, sometimes one post, sometimes more than one, and I save a little time with each draft. In a cumulative sense, over the long haul, I think small efficiencies like these have outsized results.

I am always looking for small efficiencies like these because of their magnified results over time. Do you have small efficiencies that you have discovered? If you feel like it, share them in the comments.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Evernote and Obsidian: Collecting vs. Creating

As I have continued to increase my use of Obsidian over the last few months, I have occasionally felt a little guilty at my neglect for Evernote. I keep coming back to the question: why not use Evernote for the stuff that I am putting into Obsidian? Yesterday, while thinking about this question, I came up with the answer. It is all about collecting versus creating.

Collecting

If you look back at my Going Paperless posts, they are all about collecting and categorizing information in some form or another. From the early days when I described my process for going paperless and tips on how I use Evernote to remember everything to the way I used Evernote as a mobile paperless office or capturing technology setup instructions, everything I wrote about, everything I experimented with was about how to collect, categorize, and find the information I needed at moment’s notice. What I didn’t write about much, because I simply didn’t do it, was create new stuff in Evernote.

Why not create in Evernote? Two reasons come to mind.

  1. It is still faster to jot something (an idea, a list) down in a pocket notebook than it is to pull out my phone, unlock it, fire up the Evernote app, and tap out the note on the touchscreen. I wrote about this as far back as 2013 when I discussed how I performed time trials of using a Field Notes notebook instead of Evernote.
  2. I’ve never been convinced that Evernote’s note-taking interface was an improvement over similar WYSIWYG interfaces. I find I spend too much time doing things like trying to get the formatting correct, and not enough time in the process of actual creation. Evernote is not unique in this regard. This is a problem I have with most WYSIWYG interfaces.

I still collect things in Evernote just as I always have. I still categorize them, refining my taxonomy, finding ways to make it easier to search for what I am looking for. But I still don’t create things in Evernote.

Creating

When it comes to writing things down, Obsidian has become the only tool I use for creation. Indeed, while I still carry around a Field Notes notebook wherever I go, and while I still fill them up, I have started to transcribe those notes into Obsidian to have the content of the 25 notebooks I have filled thus far accessible when I need it. But I transcribe them into Obsidian because I see those notes as acts of creation on my part, not me collecting things for which I had no part in the creation.

Obsidian is an entirely text-based editor that uses markdown to allow the separation of the content versus the presentation layer, something I find to be of utmost importance in a writing tool. (Scrivener does this as well.) That said, it is no faster at jotting down content than my Field Notes notebook because I still have to go through the same steps I’d go through with Evernote: pulling out the phone, unlocking it, etc.

But I do think it has important improvements over WYSIWYG interfaces that make it much for useful for creation:

  1. I can use markdown to format how I like things without spending much time worry about the formatting. I can use a third party tool like Pandoc to export the notes into whatever format I want.
  2. I can create links between my notes with ease and visualize how the notes are related to one another. The former can be done in Evernote, but not the latter.
  3. There is little to distract during the creation process so I find it easier to focus on what I am creating.
  4. The things I create in Obsidian, because they are plain text, lend themselves more readily to automation. Take my daily notes, for example.
Current map of my note links in Obsidian
Current map of my note links in Obsidian

Balance

This notion of collecting versus creating has helped provide some balance to how I think of these tools. Evernote is my digital filing cabinet–it has been since I first began using it nearly 11 years ago.

But for my own creations: notes, stories, blog posts, essays, lists, anything that is the product of my brain, Obsidian is a living archive, one that makes it easy to create content and provide context to the creation by through its innovating note linking. I now have a much better sense of what goes where, and why.

A Developer’s Logbook

Working scientists use logbooks to record their work so that they can (a) reproduce results, and (b) establish priority in discoveries. As a working developer, I use a logbook, too, which also serves two primary purposes: (a) capture what I did during the day (sometimes in order to reproduce things), and (b) as an index to more detailed notes for specific things.

My logbook had evolved over the years. It’s present incarnation is a text (markdown) file in Obsidian. For work, I use Obsidian’s Daily Note feature for my logbook. I have one logbook “page” for each day. I have this logbook page open on one screen at all times. Usually my Obsidian window is split with my logbook on the left and other notes files on the right. Here is what a typical screen looks like:

My screen with my logbook and other notes open in Obsidian

Yesterday I mentioned how I was in crunch time for a software system my team has been building for the last 13 months. We go-live on Monday and we are preparing for roll out. We are in what I call the “junk drawer” phase of the project. I think of it like moving houses. All of the big furniture had been moved and all that is left is the stuff in the closets and junk drawers. That’s where we are. Yesterday, my workday started at 7am and ended at 12:30 am (this morning!). Here is my logbook entry for yesterday:

Yesterday's page from my logbook

Those of you who spend your days making software are probably familiar with the pattern of these last few days before roll out. Those who aren’t can at least get a little glimpse of what these days are like for me.

As you can see from the logbook, I finally got to bed around 1 am (about 4 hours later than normal) and I was up before six this morning so that I could get in my morning walk before taking my girls to school and writing this post.

Now it’s back to where I left off yesterday. I have a new blank “page” open in my logbook and I’m ready to fill it.

My Obsidian Daily Notes Automation Script is Now Available on GitHub

Since I am on vacation and happened to find myself with an empty hour this afternoon, I managed to clean up my code enough to where I was willing to put my Obsidian daily notes automation script on GitHub. This is the script that I use to automate the creation of my daily notes in Obsidian.

You can find the repo here.

As I say in the README:

I’m posting this software as-is. It works for me, and a number of folks have requested it and I’m happy to put it here to share it. But I have no time to support it. If I make improvements, I’ll try to post them, but there’s no guarantee there either. I realize that this may not work perfectly on non-Mac systems, but the whole point of posting the code is to let folks see it, fork it, and roll your own from it. Hopefully it works for you the first time. If not, the code’s there for you to mess with.

For those who choose to use it, keep in mind that I run my script on a Mac, and icalBuddy, which I use for pulling in my agenda, is designed for Apple’s calendar app. You may need to look for alternatives for other platforms.

I’m always eager for feedback and suggestions, but as I said, my schedule is such that I don’t have time to provide any kind of support for getting the script working for you.

Obsidian and Vim Mode

For the last several days, I have been playing around with Obsidian in Vim mode. Vim, for those who don’t know, is a powerful text editor that can take some getting used to. It uses different “modes”: for editing, for navigating and issuing commands. It’s keyboard commands are designed for touch typists so that you can do anything you need to do without your fingers ever leaving the keyboard.

Obsidian offers a “Vim mode” which gives some of Vim’s capabilities. I like the idea of Vim but I’m not completely sold on its implementation in Obsidian yet.

Because of how navigation work (basic cursor movements use the h, j, k, and l keys) a fixed-width font is better for Vim. The theme that I use, Pisum, doesn’t make use of a fixed-width font. That meant I needed to edit the styles in the theme to get what I wanted.

This was a useful side-effect of experimenting because I found that it was pretty easy to edit the styles. I copied the styles for the Pisum theme into another .css file and edited there so I didn’t mess up the original theme. I made two basic changes:

  1. I switched to a fixed-width font in the editor (but not in preview mode).
  2. I modified the emphasis style to show an underline. I like seeing the underlying for emphasis because this is what I am used to from decades of writing manuscripts in standard format–in which italicized text is represented with underlines.

Here is an example of what this looks like:

An example of my Obsidian theme changes
An example of my Obsidian theme changes

But there are some serious limitations. For one thing, Obsidian is not Vim, and that is a good thing, since Obsidian is really focused on something different from what Vim attempts to do. It means, however, there is limited support for .vimrc files in which various setting, keyboard mappings, and other configurations that customize the editor are stored. There is a plug-in that provides some limited support for .vimrc files, but it is limited

It is nice that Obsidian includes the Vim mode option because it makes the transition to Obsidian easier for people used to Vim’s keyboard mappings. But after playing around with Vim mode for several days, I think I am going to turn it off. I asked myself how often, while writing do I need to do some of the fancy things that Vim’s commands let me do? The answer is rarely.

All is not lost, however. I learned that it is easy to edit themes in Obsidian–by far easier than any other editor I’ve played with. And some of my edits I’m keeping. I’m turning off the fixed-width font. I prefer the default theme font for notes. But I’m leaving my underlined emphasis in place. And there are probably other tweaks I’ll make. I have it in my mind to produce a theme that looks like Word for DOS 5.5 (which was my all-time favorite word processor). I’d do this more for learning than actual use. After all, these days, everyone want to use a dark theme because it’s easier on the eyes, and that bright blue background in Word for DOS is the antithesis of a dark theme.

Once I have my personal theme stable, I’ll write a separate post about it.

A Quick Update on My Obsidian Automation Scripts

I’ve had a number of people reach out recently to inquire about my Obsidian automation scripts. I will get them posted to Github in the relatively nearly future. In their current state, they are not ready for public consumption. There are lots of hard-coded paths, to say nothing of the code is fairly ugly. I’ve cleaned up about half the code and moved it into a Python package. I still need to work on the other half.

The problem, as always, is time. I am working on a day-job crunch as I prepare to roll out some software my team has been developing for the past year. That really limits my time in March and April. So at this point, May is looking like when I will most likely have time to work finish my cleanup of the Python code.

I promise to post an update once I’ve put the code on GitHub. Until then, thanks for your patience.

Journal in Obsidian Notes?

Once I got the hang of how Obsidian worked for me, once I realized the power of its linking capabilities, and that it really did everything I wanted a note-taking app to do, it was natural to consider what could go into my vault. Daily notes were a given, of course. All of my reading notes, and even a version of my reading list could go in there. Borrowing some concepts from Zettelkasten, it could become a kind of digital commonplace book, something I’ve always wanted. What about my journal? With all of the other information in one place, linkable and searchable, it seemed to make sense that my journal should go there as well.

The thing is, my journal has always been handwritten, going back to 1996. There were times when I experimented with it in a digital form, but I always came back to the handwritten form. In the current incarnation (since late 2017), they fill eight Moleskine Art Collection sketchbooks.

My collection of journals.
My collection of Moleskine journals

As it turns out, how I keep my journal lends itself Obsidian linking. Rather than an entry-per-day, I number entries, beginning at 1. Each discrete entry gets its own number. I date the first one of each day, but there may be two or three entries in a day, each of which will have its own number. I did this thinking ahead: if I ever wanted to index the thing, I wouldn’t have to worry about what volume or page and entry was on. All I’d need was its entry number. (I took this lesson from Isaac Asimov’s description of how he numbered entries in his Biographical Encyclopedia of Science and Technology in order to avoid the tedious work of tying index entries to page numbers.) Thus, I have 1,782 unique “entries” each with its own number. This makes it ideal for linking in Obsidian.

Several weeks ago, I decided to give it a try, and I began writing my journal entries in Obsidian, giving each entry a unique number, continuing from where I left off. I liked being able to link these entries to other notes.

Journal entries listed in Obsidian
Journal entries in Obsidian

Something nagged at me, however. I missed writing in my journal. I missed how the pages contain more than just writing. I paste pictures and clippings in the pages. Sometimes I sketch things. It just didn’t feel the same typing the entries rather than writing them out in my journal.

A typical multimedia page from my journal.
A typical “multimedia” journal entry

It occurred to me that I might have the best of both worlds with a little effort. At the end of each week, for instance, I could type up the entries I’d written in the Moleskine notebook, copying the entries into Obsidian. Then they’d be there for searching and linking. After a little thought, that felt like a monumental waste of time.

Last night, I decided not to keep my journal in Obsidian and to continue with the notebooks. I did this for several reasons:

  1. I still think there is a compelling argument for how long paper lasts. Digital media has been around half a century or so. Paper has been around centuries. Witness John Adams’s diaries or Leonardo da Vinci’s notebooks.
  2. I don’t actually search through my journals that much to make it worthwhile to put every entry into digital form.
  3. When I do search them, I enjoy the feel of flipping through them, seeking out what I am looking for.

But there was one other thing that occurred to me that sealed the deal for me. My Daily Notes in Obsidian serve as an index to my life. If I needed to know when I wrote about something in my journal, I need go no further than my daily notes. I can search them for the appropriate reference and then use the date of those notes to look up any entries in my journal. Moreover, if I write something in the journal and want to make sure I can find it easily, I can just add a reference to the entry number in my daily notes.

That seemed to satisfy me, and with that, I began this morning, transcribing those entries I made in Obsidian back to my Moleskine notebook. Going forward, the journal will stay in a notebook, but I’ll rely more and more on the daily notes as a kind of compass for finding what I need.

How I Capture Reading Notes in Obsidian

In addition to automating my daily notes with Obsidian, it quickly became clear to me that Obsidian‘s note-linking capabilities would allow me to capture my reading notes in Obsidian in a really useful way. Moreover, because of Obsidian’s powerful linking capability, it occurred to me that my Obsidian vault could serve as a database for my reading. To describe how I managed to do this (so far) in a step-by-step manner will required a little history first.

A Brief History of My Reading List

I began keeping a list of every book I finished reading back on January 1, 1996. Although I am no longer certain of why I started keeping the list (was it part of a New Year’s resolution?) I am fairly certain that I was influenced by an early reading list I found on the Internet, Eric W. Leuliette’s “What I Have Read Since 1974“.

As a developer (even back then), I decided I would build an elaborate relational database to store my reading list. Over the years, it went through many iterations, and forms. When time became short, I moved the list out of the database and into Excel, or Google Sheets. Finally, several years ago, I settled on a plain text file using Markdown format, and that is how I’ve kept my list ever since.

But I’ve been bothered by shortcomings on this list. There are redundancies I don’t like about it. I have no easy way of referring to books or authors separate from the list. There are things I’d like to automate about it but that the format makes tricky.

A Brief History of My Reading Notes

With all of the reading I do, I have trouble remembering important details of what I read about. So I started keeping notes on my reading. This evolved out of how I kept notes on my reading back in college, and has continued to evolve over the decades since. It was in college that I first decided it was okay for me to write in my books. After all, if I was spending so much money on them, I might as well make them my own, right?

These days, I highlight books, writing margins, and with e-books, I highlight and make short notes on my Kindle devices and apps. But I still have no good way of aggregating these notes into useful groups, categories, and certainly no way of readily searching them.

As I started using Obsidian, and began to see how I could better organize my books and reading lists in its vault structure, I began to get a hint of ways that I might start to link my reading notes back to the books they are associated with, my reading, and other notes.

Enter Zettelkasten

I’d never heard of Zettelkasten before I started using Obsidian. Zettelkasten was originally invented as a way to link paper notes together to be able to easily create connections (links) between then. While it was workable on paper, such a process could be greatly improved with hypertext tools, and it so happens that Obsidian’s note-linking capability is idea for this.

One important idea from Zettelkasten is that a note should contain a single thought or piece of information (say, a passage highlighted in a book). That note is given a unique identifier. In addition to the passage, one would add their own thoughts to the note, and perhaps further link that note to other notes and ideas that are related to it. Zettelkasten has its own unique numbering system for “naming” the notes. Obsidian has a plug-in for creating a “Zettelkasten number” for this purpose that is based on the date/time the note is created. I wasn’t particularly fond of that identifier because it duplicates information already contained in the note itself. After all, the note is just a file in the file system, and has its own create and modified date/times as part of the file. A good identifier does’t embed real data. It’s just an identifier.

I also struggled a bit to figure out how this would work for my reading notes. I originally imagined that if I had a note for each book I read, I could simply add my highlights and annotations to that note. Zettelkasten, however, suggested that rather than adding that highlight to the book note, I’d create a separate note for just the highlight or annotation, and then link it to the book note–as well any other notes it might make sense to link it to. This took a while for me to process, and I thought about it a lot as I built out my reading library in Obsidian.

My Obsidian Library

So how did I decided to structure my reading notes in Obsidian? I’ll try to go through the step-by-step process I have for putting this all together, in case someone is interested in reproducing this.

Step 1: Establishing the structure

I decided that because of Obsidian’s great linking capability, I could use the file system itself as a relational database. In deciding this, I further decided that there were 3 main “objects” I wanted to be able to capture at a kind of atomic level. That is, three things that make up the structure of my reading library:

  1. Things I read, e.g., books, articles, stories, etc.
  2. Authors: the people who write the things in #1.
  3. My notes as they relate to #1 and #2.

From this, I established the following structure of folders in within my Obsidian vault:

My folder structure for Reading notes in Obsidian.
  • Commonplace Book contains all of my reading notes.
  • Library contains all of the “atomic” notes that make up my reading library:
    • Authors: a single note for each unique author in my library
    • Articles: a single note for each unique article (often not tied to a book) in my library.
    • Book: a single note for each unique book in my library
    • Essays: a single note for each unique essay in my library; these are often related to books.
    • Stories: a single note for each unique story in my library.

Step 2: Deciding what goes into a note

Once I had my structure, I had to decide what goes into a note of each type. What is it I want to know about authors, books, stories, etc.? This was fairly easy for me as I’ve been thinking about it for a long time (years, actually). I had in mind an idea that I could write an API that uses these files as a database to query them and produce results. With that in mind, I decided to start by keeping things simple, knowing that I could add detail as needed going forward.

For authors, I wanted just some basic information. Here is a typical author note, in this case, for Alan Lightman, whose new book I read earlier this week:

A sample author note for Alan Lightman.

The backlinks section is generated automatically by a script that I have that runs nightly. I know that I could just click on the “Linked mentions” in Obsidian to see all of the backlinks, but I wanted the related books on the note as a reference in case I access the file outside of Obsidian.

For books (or essays, stories, articles), I also kept things simple. A typical book (or essay, or article, or story) looks like this:

A sample title note for In Praise of Wasting Time

Note that in both authors and books there are links back and forth between the files. The book file refers to the author. The author file has link references back to the books. Moreover, you’ll note that in the book, there is an “Annotations” section with a list of links. These are auto-generated links to my notes and highlights for the book. I’ll have more to say on these shortly, but the important thing is that each note and highlight is a separate file (in the Zettelkasten vain) and is included with the book as a “transclusion” link, meaning that when I view the note in preview mode, it “includes” the links files as part of the note, like this:

Title note in preview mode with transcluded annotations visible.

Step 3. Populating the database

Once I had the structure I wanted, I needed to populate my database. I was fortunate in this regard on 2 counts: (1) I happened to recently create a SQLite database of my books, and (2) I can write code relatively easily. I wrote a script that crawled my book database, and from it, creating the notes for books and authors in Obsidian. This turned out to be a surprisingly simple exercise. (The Python script was 130 lines.)

My digital commonplace book

I first learned of commonplace books reading a biography of Thomas Jefferson (in this case, it was Williard Sterne Randall’s Thomas Jefferson: A Life.) Jefferson (and others in his time) would copy passages from their reading into a book. This helped with memorization, but it also provided a resource where they could add notes and observations. I’ve always liked this concept, and I decided that Obsidian would finally allow me to put it into action in a way I’d envisioned.

It is trivial to create a note and add it to the note containing the book to which it is related. But what if the note ultimately relates to more than one thing? Reading about Zettelkaten provided me with insights into how I might handle this. The naming convention in Zettelkasten (and the way it is implemented in Obsidian) bothered me. Neither made much sense. How do you search for things with essentially coded filenames?

I was in the shower when I finally had a breakthrough insight on this. I’m not searching for a filename, I’m searching for file content. If each annotation and highlight I can link it to as many notes as makes sense. Furthermore, I can add tags to each note. The name of the file doesn’t matter. What matter is how it links to other notes, and that all files are searchable.

I still didn’t like the file-naming scheme for Zettelkasten in Obsidian, which essentially uses a datetime stamp down to the current second. So a file might be named: 20210215084456. Given that one is not likely to create two of these notes within the same second, it guarantees uniqueness. But from a database perspective, identifiers like these are not supposed to embed any information. They should be strictly identifiers. Moreover, the with the date embedded in the note title, I would be duplicating information that already exists in the file properties.

I decided instead to use a Guid, or what is sometimes called a UUID. This is another form of a unique identifier that doesn’t embed information, just produces a unique code. (For those tech-savvy folks reading this, I used Python’s UUID4 which doesn’t use the MAC address as part of the identifier.)

When I have a new note or highlight for a book, it goes into my Commonplace Book folder in Obsidian. These notes also have a specific structure. A typical one looks like this:

A typical note, Zettelkasten style.

Each annotation begins with a Source that links back to the source for that annotation. It may or may not have tags associated with it. That is followed by the body of the annotation, which may be a highlighted passage. Finally, there are my own notes related to the specific passage. In the above example, my notes also link to another book, making this particular annotation related to more than one note. That is, a link has been created between Creativity, Inc by Ed Catmull and Amy Wallace, and On Writing by Stephen King.

Automating my annotations

Over the weekend, I got a start on automating these annotations. I wrote a Python script that reads a CSV files exported from Kindle, and creates a unique note for each annotation in the file, relating it back to the source book in my Library. My process is roughly this (I say roughly because this is still new):

  1. When I finish reading a book, I export the annotations from my Kindle, which sends me an email. That email has a CSV attachment which I save in a folder.
  2. A script runs, and processes and CSV files I have in the folder, creating the notes and links.
  3. The script, outputs a list resulting annotations for each file. I copy this and paste it into the “Annotations” section of the source book or article. That makes it easy to view the annotations inline when previewing the note. An example of the output from the script looks like this:
Output from my annotation import script.

Toward an API for my books and annotations

I am able to do the above automation because I have a standardized structure to my books and author notes. That standardization allowed me to write an API for my book library. From this API I can, for instance, check to see if a title exists in my library already. I can grab information about a book or author and then use it in some way. The API typically returns data in JSON format. For instance, if I call the function biblio.search_by_title("Beyond"), I get a JSON formatted return containing the following:

[
   {
      "title":"_Beyond Band of Brothers: The War Memoirs of Major Dick Winters_",
      "link":"[[Beyond Band of Brothers (334)]]",
      "type":"book",
      "authors":[
         {
            "author":"Winters, Richard",
            "authorFirstLast":"Richard Winters",
            "authorLink":"[[Winters, Richard]]",
            "gender":"None"
         }
      ],
      "source":"",
      "date":""
   },
   {
      "title":"_Beyond Apollo_",
      "link":"[[Beyond Apollo (58)]]",
      "type":"book",
      "authors":[
         {
            "author":"Malzberg, Barry N",
            "authorFirstLast":"Barry N Malzberg",
            "authorLink":"[[Malzberg, Barry N]]",
            "gender":"m",
            "alternateNames":[
               {
                  "name":"Barry, Mike",
                  "nameLink":"[[Barry, Mike]]"
               }
            ]
         }
      ],
      "source":"",
      "date":""
   },
   {
      "title":"_Beyond the Blue Event Horizon_",
      "link":"[[Beyond the Blue Event Horizon (259)]]",
      "type":"book",
      "authors":[
         {
            "author":"Pohl, Frederik",
            "authorFirstLast":"Frederik Pohl",
            "authorLink":"[[Pohl, Frederik]]",
            "gender":"None"
         }
      ],
      "source":"",
      "date":""
   }
]

The results so far

I’ve linked all of this together using my master reading list note. This note contains a list of everything I have read since 1996 and serves as a kind of index to my reading:

A sample from my reading list index note.

A big part of the way Obsidian works is that it can show you the relationships between your notes. While I am still working on importing all of the notes I have in my Kindle, I can already see a a network of relationships when I view the graph of my Obsidian vault:

A graph of the relationships between all of my notes.

Most of my notes are book and reading-related at this point. That big dot in the center is the master reading list illustrated above. If I highlight it, this is what I see:

Sample of a highlighted node on the graph.

From there, you can see other nodes and relationships that have started to form. For instance, if I hover over one of the Alan Lightman books I finished yesterday, In Praise of Wasting Time, you can see a little network of links coming off that book:

Some of those links point to annotation files. Another points back to the note for Alan Lightman. And a few of the annotation links point to seemingly unrelated notes.

Here is another example. One of the big nodes is for John W. Campbell, editor of Astounding Science Fiction in the late 1930s through his head in the early 1970s. I read many of those old issue when I was taking my Vacation in the Golden Age of Science Fiction. So Campbell shows up a lot on my master reading list:

Highlighting an author node on the graph.

You can see that Campbell is linked to all of the issues of Astounding that I have read. I have started to bring my notes in for those issues. If we look at the July 1939 issue, for instance, you can see this is related to all of the stories and articles and authors in that issue:

Currently, the notes for each story are part of the story note, but I plan on breaking those out into their own Zettelkasten-style notes as I’ve done for my other notes and annotations.

Conclusions

Keep in mind, that this is all being done with plain text files, something that I like because the format is compatible virtually anywhere. This could be done as easily on a Windows machine as a Mac. It could be done easily on a Linux machine. The openness and longevity of plain text (which has been around for fifty years now) is a big part of what I like about this system.

The linking that Obsidian provides from within its application makes all of this useful. But once established, those links are just as useful outside Obsidian with a little coding–as I’ve done with my API for books and authors. And this API is extensible. This week, I plan to add capability for the API to return any annotations when returning a “book” object. So in addition to what is returned by the JSON format illustrated above, that will soon contain a node for annotations related to that book.

Mostly, I am satisfied that I now have a simple way of keeping my reading notes in a useful form. These are easily searchable, they are easily linked. I can continue to capture highlights and brief notes as a I read. The import function allows a nice step to expand on my annotations as I review them after they’ve been pulled into Obsidian.

It did take me some time to get the infrastructure in place, but now that it is there, I am able to focus on reading, notes, and let the system organize them for me.