Tag: obsidian

The Weekly Playbook #4: Finishing a Notebook: Transcribe or Scan?

For an overview of this series, see the debut post on my morning routine.

Background

I am rarely without a Field Notes notebook in my back pocket. Several times a day, I pull out my notebook to jot something down: an idea for a post; notes from a podcast; the names of people I meet; items to pick up at the grocery store; the name of the server in the restaurant we ate at; funny lines I hear at a gathering. I do this so that I can remember these things later. Some of them find their way into posts I write, some into stories. Other things are more ephemeral, but even a server’s name in a restaurant can be useful if I am searching for a character name in a story.

I’ve filled more than 30 of these notebooks since 2015. They sit in a nice row on a shelf in my office. Occasionally I go back to them, to look for something, like when I was searching for a particular brand of beer recently. The problem is, I only have access to them when I am sitting here in my office. It would be nice to have access to them no matter where I was.

my 30 completed field notes notebooks with an index notebook on top
My 30 completed Field Notes notebooks, with an index notebook on top.

This weekly playbook is a kind of experiment. I began with the idea that I wanted to be able to access these notes anywhere. I had two ideas:

  1. Transcribe the notebooks into Obsidian, where my other notes live, or
  2. Scan them into Evernote

I decide to try both in order to see what worked better for me. The playbook section below has the procedures I followed for each. In each case, I used my most recently completed notebook, book #30. I’ll describe my findings in the commentary.

Playbook

Transcribing notebook into Obsidian

  1. Create a Field Notes folder in Obsidian
  2. Create a new note called “Book 30 – March to June 2021.
  3. Begin typing in the notes using the following guidelines:
    • Make each “day” a header in the notes
    • If my handwriting is unintelligible, put question marks and move on.
    • Wherever I have a dividing line in my notebook, include a divider in the notes file
    • Use only one file per notebook

Scanning notebook into Evernote

  1. Create a Field Notes notebook in Evernote.
  2. Using the Scannable app by Evernote, scan in all 48 pages of my notebook #30, including the cover and inside cover.
  3. Once scanned, put the note in the Field Notes notebook
  4. Title the note “Book 30 – March to June 2021”
  5. Set the create date of the note to March 1, 2021
transcribed notebook page in obsidian
A transcribed notebook page in Obsidian
scanned notebook page in evernote
A scanned notebook page in Evernote

Commentary

It probably took me an hour to transcribe the first 15 pages of the Field Notes notebook into Obsidian. After an hour I stopped. It is easy enough to estimate that a full notebook would take me a little over 3 hours to transcribe.

On the other hand, it took about 15 minutes to scan the entire notebook into Evernote using the Scannable app. (I think Evernote’s Scannable app does a slightly better job at scanning than the regular iOS app does.)

For me, the Evernote scan is the better over all option. There are several reasons for this:

  1. It is quick enough to make it worthwhile. Investing 15 minutes to have the contents of the notebook available to me anywhere is a worthwhile investment of time. 3 hours is a little much. I am not likely to invest 3 hours, but 15 minutes is no big deal.
  2. The notebook really is available anywhere. The screenshot above is from my phone. I can flip through the pages just as I can with any PDF.
  3. Scanning preserves everything in my notes, include occasional sketches and diagrams that I make.
  4. Evernote uses its AI to attempt to make the PDF searchable. It is supposed to be able to recognize handwriting. I made several attempt, but I think my handwriting is too messy. Still, for people with very neat writing, the notebook is searchable. I keep the notes in their own notebook in Evernote for this reason: when I want to search for something in a Field Notes notebook, I can limit the search to notes in the Field Notes notebook so that I don’t get results from other sources.

There are a few cons to using Evernote over Obsidian:

  1. The notebook is not as searchable as it would be if I transcribed it into Obsidian. I could probably find things faster in Obsidian.
  2. My notes would be in plain text format and could be manipulated like any plain text.
  3. I could do more dynamic linking of my notes to other notes using Obsidian. (You can link to other notes in Evernote, but there is no practical way to do this in scanned documents.)

Another consideration is that I want to get my entire backlog of notebooks in a format that I can access anywhere. Transcribing 30 notebooks into Obsidian would be an investment of nearly 100 hours of my time. Scanning 30 notebooks into Evernote is an investment of 7-1/2 hours. From a practical standpoint, this is a no-brainer.

Then, too, since the notes already exist, they fit into the model of using Evernote for curation and collection, and using Obsidian for creation.

Remember, my goal at the outset was to be able to access the notebooks from anywhere. My goal wasn’t to make them as searchable as they could be. I’m fine flipping through a PDF to find what I am looking for. It usually doesn’t take very long, so it seems like the investment in time to manually transcribe all of my notes would be overkill.

Going forward, when I finish a notebook, I’ll follow the procedures for scanning that notebook into Evernote.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Didn’t like it so much? Let me know why, either in the comments, or by reaching out to me directly. I’m always looking for ways to improve. Already a reader or subscriber to the blog? Thanks for being a part of this community!

Follow Jamie Todd Rubin on WordPress.com

Small Efficiencies in Workflow

With my recent plan to focus on my writing and improve my overall well-being (a.k.a. Project Sunrise), I have been hunting for small efficiencies in workflow that can have an outsized impact on my day. My morning routine takes about two and a half hours to complete. While developing the routine, I teased out actions or tasks that I could eliminate or improve upon to maximize the use of my time. Two examples come to mind.

Writing in my journal: content versus medium

Since 2017, I have been writing my journal longhand in large Moleskine notebooks. I’ve written about the advantages and disadvantages of having a paper journal versus a digital one in a piece called The Paradox of Journaling. I like the feeling of writing longhand, and I understand and believe in the durability of paper. But there are two tradeoffs to consider when time is limited and my goals depend on data:

  1. The speed and clarity with which I can write.
  2. The speed an accuracy with which I can find what I wrote about.

With limited time, I had to consider what is more valuable to me now, the content of my journaling or the medium in which it is stored. Today it is the content. Since I can type much faster than I can write longhand, since my typing is more clear than my handwriting, and since I express thoughts more clearly through a keyboard than a pen, it seemed prudent to switch my journaling to a digital form instead of a paper one. This is why for the last week, I have been composing it as a text file using Obsidian, despite what I wrote in February when I initially rejected the idea. The reasons I rejected it were valid then, but circumstances have changed, and I think this little efficiency will have long-term benefits.

One of those benefits is the speed with which I can find what I wrote about. It is much easier to search a text file than volumes of journals, even when they are roughly indexed. And time is the key. I want to spend as much of my time as possible on creative tasks. That said, to improve, I need to look back at the data I’ve collected so that I can apply it going forward. I can do this much more quickly searching a text file than books. Practical considerations–speed of input, clarity, and speed of retrieval–have overridden my desire to continue writing my journal longhand, at least for the duration.

Composing in WordPress

For a long time, I composed my blog posts in an external editor. That editor has changed over the years. I’ve written drafts in Scrivener, in Word, and most recently, in Obsidian, my current editor of choice. With my recent migration to WordPress hosting, and conversion to a modern WordPress theme, I have found WordPress’s native Gutenberg editor to be comfortable and easy to compose in directly. This saves a good deal of time. Prior to composing directly in WordPress my process looked like this:

  1. Write the post in Obsidian (or other editor)
  2. Copy the text out of Obsidian
  3. Paste it into a blank WordPress post
  4. Fix any formatting issues
  5. Publish.

For the last week I have been composing directly in WordPress which allows me to eliminate the administrative steps I was doing before. This shaves a little time spent on each post, which I get back for creative work, like writing the posts themselves.

These are small efficiencies. They don’t save huge chunks of time each, but the affect is cumulative. I journal in the morning and evening, so I am saving a little time each journaling session. I tend to write in the mornings, sometimes one post, sometimes more than one, and I save a little time with each draft. In a cumulative sense, over the long haul, I think small efficiencies like these have outsized results.

I am always looking for small efficiencies like these because of their magnified results over time. Do you have small efficiencies that you have discovered? If you feel like it, share them in the comments.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Didn’t like it so much? Let me know why, either in the comments, or by reaching out to me directly. I’m always looking for ways to improve. Already a reader or subscriber to the blog? Thanks for being a part of this community!

Follow Jamie Todd Rubin on WordPress.com

Evernote and Obsidian: Collecting vs. Creating

As I have continued to increase my use of Obsidian over the last few months, I have occasionally felt a little guilty at my neglect for Evernote. I keep coming back to the question: why not use Evernote for the stuff that I am putting into Obsidian? Yesterday, while thinking about this question, I came up with the answer. It is all about collecting versus creating.

Collecting

If you look back at my Going Paperless posts, they are all about collecting and categorizing information in some form or another. From the early days when I described my process for going paperless and tips on how I use Evernote to remember everything to the way I used Evernote as a mobile paperless office or capturing technology setup instructions, everything I wrote about, everything I experimented with was about how to collect, categorize, and find the information I needed at moment’s notice. What I didn’t write about much, because I simply didn’t do it, was create new stuff in Evernote.

Why not create in Evernote? Two reasons come to mind.

  1. It is still faster to jot something (an idea, a list) down in a pocket notebook than it is to pull out my phone, unlock it, fire up the Evernote app, and tap out the note on the touchscreen. I wrote about this as far back as 2013 when I discussed how I performed time trials of using a Field Notes notebook instead of Evernote.
  2. I’ve never been convinced that Evernote’s note-taking interface was an improvement over similar WYSIWYG interfaces. I find I spend too much time doing things like trying to get the formatting correct, and not enough time in the process of actual creation. Evernote is not unique in this regard. This is a problem I have with most WYSIWYG interfaces.

I still collect things in Evernote just as I always have. I still categorize them, refining my taxonomy, finding ways to make it easier to search for what I am looking for. But I still don’t create things in Evernote.

Creating

When it comes to writing things down, Obsidian has become the only tool I use for creation. Indeed, while I still carry around a Field Notes notebook wherever I go, and while I still fill them up, I have started to transcribe those notes into Obsidian to have the content of the 25 notebooks I have filled thus far accessible when I need it. But I transcribe them into Obsidian because I see those notes as acts of creation on my part, not me collecting things for which I had no part in the creation.

Obsidian is an entirely text-based editor that uses markdown to allow the separation of the content versus the presentation layer, something I find to be of utmost importance in a writing tool. (Scrivener does this as well.) That said, it is no faster at jotting down content than my Field Notes notebook because I still have to go through the same steps I’d go through with Evernote: pulling out the phone, unlocking it, etc.

But I do think it has important improvements over WYSIWYG interfaces that make it much for useful for creation:

  1. I can use markdown to format how I like things without spending much time worry about the formatting. I can use a third party tool like Pandoc to export the notes into whatever format I want.
  2. I can create links between my notes with ease and visualize how the notes are related to one another. The former can be done in Evernote, but not the latter.
  3. There is little to distract during the creation process so I find it easier to focus on what I am creating.
  4. The things I create in Obsidian, because they are plain text, lend themselves more readily to automation. Take my daily notes, for example.
Current map of my note links in Obsidian
Current map of my note links in Obsidian

Balance

This notion of collecting versus creating has helped provide some balance to how I think of these tools. Evernote is my digital filing cabinet–it has been since I first began using it nearly 11 years ago.

But for my own creations: notes, stories, blog posts, essays, lists, anything that is the product of my brain, Obsidian is a living archive, one that makes it easy to create content and provide context to the creation by through its innovating note linking. I now have a much better sense of what goes where, and why.

A Developer’s Logbook

Working scientists use logbooks to record their work so that they can (a) reproduce results, and (b) establish priority in discoveries. As a working developer, I use a logbook, too, which also serves two primary purposes: (a) capture what I did during the day (sometimes in order to reproduce things), and (b) as an index to more detailed notes for specific things.

My logbook had evolved over the years. It’s present incarnation is a text (markdown) file in Obsidian. For work, I use Obsidian’s Daily Note feature for my logbook. I have one logbook “page” for each day. I have this logbook page open on one screen at all times. Usually my Obsidian window is split with my logbook on the left and other notes files on the right. Here is what a typical screen looks like:

My screen with my logbook and other notes open in Obsidian

Yesterday I mentioned how I was in crunch time for a software system my team has been building for the last 13 months. We go-live on Monday and we are preparing for roll out. We are in what I call the “junk drawer” phase of the project. I think of it like moving houses. All of the big furniture had been moved and all that is left is the stuff in the closets and junk drawers. That’s where we are. Yesterday, my workday started at 7am and ended at 12:30 am (this morning!). Here is my logbook entry for yesterday:

Yesterday's page from my logbook

Those of you who spend your days making software are probably familiar with the pattern of these last few days before roll out. Those who aren’t can at least get a little glimpse of what these days are like for me.

As you can see from the logbook, I finally got to bed around 1 am (about 4 hours later than normal) and I was up before six this morning so that I could get in my morning walk before taking my girls to school and writing this post.

Now it’s back to where I left off yesterday. I have a new blank “page” open in my logbook and I’m ready to fill it.

My Obsidian Daily Notes Automation Script is Now Available on GitHub

Since I am on vacation and happened to find myself with an empty hour this afternoon, I managed to clean up my code enough to where I was willing to put my Obsidian daily notes automation script on GitHub. This is the script that I use to automate the creation of my daily notes in Obsidian.

You can find the repo here.

As I say in the README:

I’m posting this software as-is. It works for me, and a number of folks have requested it and I’m happy to put it here to share it. But I have no time to support it. If I make improvements, I’ll try to post them, but there’s no guarantee there either. I realize that this may not work perfectly on non-Mac systems, but the whole point of posting the code is to let folks see it, fork it, and roll your own from it. Hopefully it works for you the first time. If not, the code’s there for you to mess with.

For those who choose to use it, keep in mind that I run my script on a Mac, and icalBuddy, which I use for pulling in my agenda, is designed for Apple’s calendar app. You may need to look for alternatives for other platforms.

I’m always eager for feedback and suggestions, but as I said, my schedule is such that I don’t have time to provide any kind of support for getting the script working for you.

Obsidian and Vim Mode

For the last several days, I have been playing around with Obsidian in Vim mode. Vim, for those who don’t know, is a powerful text editor that can take some getting used to. It uses different “modes”: for editing, for navigating and issuing commands. It’s keyboard commands are designed for touch typists so that you can do anything you need to do without your fingers ever leaving the keyboard.

Obsidian offers a “Vim mode” which gives some of Vim’s capabilities. I like the idea of Vim but I’m not completely sold on its implementation in Obsidian yet.

Because of how navigation work (basic cursor movements use the h, j, k, and l keys) a fixed-width font is better for Vim. The theme that I use, Pisum, doesn’t make use of a fixed-width font. That meant I needed to edit the styles in the theme to get what I wanted.

This was a useful side-effect of experimenting because I found that it was pretty easy to edit the styles. I copied the styles for the Pisum theme into another .css file and edited there so I didn’t mess up the original theme. I made two basic changes:

  1. I switched to a fixed-width font in the editor (but not in preview mode).
  2. I modified the emphasis style to show an underline. I like seeing the underlying for emphasis because this is what I am used to from decades of writing manuscripts in standard format–in which italicized text is represented with underlines.

Here is an example of what this looks like:

An example of my Obsidian theme changes
An example of my Obsidian theme changes

But there are some serious limitations. For one thing, Obsidian is not Vim, and that is a good thing, since Obsidian is really focused on something different from what Vim attempts to do. It means, however, there is limited support for .vimrc files in which various setting, keyboard mappings, and other configurations that customize the editor are stored. There is a plug-in that provides some limited support for .vimrc files, but it is limited

It is nice that Obsidian includes the Vim mode option because it makes the transition to Obsidian easier for people used to Vim’s keyboard mappings. But after playing around with Vim mode for several days, I think I am going to turn it off. I asked myself how often, while writing do I need to do some of the fancy things that Vim’s commands let me do? The answer is rarely.

All is not lost, however. I learned that it is easy to edit themes in Obsidian–by far easier than any other editor I’ve played with. And some of my edits I’m keeping. I’m turning off the fixed-width font. I prefer the default theme font for notes. But I’m leaving my underlined emphasis in place. And there are probably other tweaks I’ll make. I have it in my mind to produce a theme that looks like Word for DOS 5.5 (which was my all-time favorite word processor). I’d do this more for learning than actual use. After all, these days, everyone want to use a dark theme because it’s easier on the eyes, and that bright blue background in Word for DOS is the antithesis of a dark theme.

Once I have my personal theme stable, I’ll write a separate post about it.

A Quick Update on My Obsidian Automation Scripts

I’ve had a number of people reach out recently to inquire about my Obsidian automation scripts. I will get them posted to Github in the relatively nearly future. In their current state, they are not ready for public consumption. There are lots of hard-coded paths, to say nothing of the code is fairly ugly. I’ve cleaned up about half the code and moved it into a Python package. I still need to work on the other half.

The problem, as always, is time. I am working on a day-job crunch as I prepare to roll out some software my team has been developing for the past year. That really limits my time in March and April. So at this point, May is looking like when I will most likely have time to work finish my cleanup of the Python code.

I promise to post an update once I’ve put the code on GitHub. Until then, thanks for your patience.

Journal in Obsidian Notes?

Once I got the hang of how Obsidian worked for me, once I realized the power of its linking capabilities, and that it really did everything I wanted a note-taking app to do, it was natural to consider what could go into my vault. Daily notes were a given, of course. All of my reading notes, and even a version of my reading list could go in there. Borrowing some concepts from Zettelkasten, it could become a kind of digital commonplace book, something I’ve always wanted. What about my journal? With all of the other information in one place, linkable and searchable, it seemed to make sense that my journal should go there as well.

The thing is, my journal has always been handwritten, going back to 1996. There were times when I experimented with it in a digital form, but I always came back to the handwritten form. In the current incarnation (since late 2017), they fill eight Moleskine Art Collection sketchbooks.

My collection of journals.
My collection of Moleskine journals

As it turns out, how I keep my journal lends itself Obsidian linking. Rather than an entry-per-day, I number entries, beginning at 1. Each discrete entry gets its own number. I date the first one of each day, but there may be two or three entries in a day, each of which will have its own number. I did this thinking ahead: if I ever wanted to index the thing, I wouldn’t have to worry about what volume or page and entry was on. All I’d need was its entry number. (I took this lesson from Isaac Asimov’s description of how he numbered entries in his Biographical Encyclopedia of Science and Technology in order to avoid the tedious work of tying index entries to page numbers.) Thus, I have 1,782 unique “entries” each with its own number. This makes it ideal for linking in Obsidian.

Several weeks ago, I decided to give it a try, and I began writing my journal entries in Obsidian, giving each entry a unique number, continuing from where I left off. I liked being able to link these entries to other notes.

Journal entries listed in Obsidian
Journal entries in Obsidian

Something nagged at me, however. I missed writing in my journal. I missed how the pages contain more than just writing. I paste pictures and clippings in the pages. Sometimes I sketch things. It just didn’t feel the same typing the entries rather than writing them out in my journal.

A typical multimedia page from my journal.
A typical “multimedia” journal entry

It occurred to me that I might have the best of both worlds with a little effort. At the end of each week, for instance, I could type up the entries I’d written in the Moleskine notebook, copying the entries into Obsidian. Then they’d be there for searching and linking. After a little thought, that felt like a monumental waste of time.

Last night, I decided not to keep my journal in Obsidian and to continue with the notebooks. I did this for several reasons:

  1. I still think there is a compelling argument for how long paper lasts. Digital media has been around half a century or so. Paper has been around centuries. Witness John Adams’s diaries or Leonardo da Vinci’s notebooks.
  2. I don’t actually search through my journals that much to make it worthwhile to put every entry into digital form.
  3. When I do search them, I enjoy the feel of flipping through them, seeking out what I am looking for.

But there was one other thing that occurred to me that sealed the deal for me. My Daily Notes in Obsidian serve as an index to my life. If I needed to know when I wrote about something in my journal, I need go no further than my daily notes. I can search them for the appropriate reference and then use the date of those notes to look up any entries in my journal. Moreover, if I write something in the journal and want to make sure I can find it easily, I can just add a reference to the entry number in my daily notes.

That seemed to satisfy me, and with that, I began this morning, transcribing those entries I made in Obsidian back to my Moleskine notebook. Going forward, the journal will stay in a notebook, but I’ll rely more and more on the daily notes as a kind of compass for finding what I need.

How I Capture Reading Notes in Obsidian

In addition to automating my daily notes with Obsidian, it quickly became clear to me that Obsidian‘s note-linking capabilities would allow me to capture my reading notes in Obsidian in a really useful way. Moreover, because of Obsidian’s powerful linking capability, it occurred to me that my Obsidian vault could serve as a database for my reading. To describe how I managed to do this (so far) in a step-by-step manner will required a little history first.

A Brief History of My Reading List

I began keeping a list of every book I finished reading back on January 1, 1996. Although I am no longer certain of why I started keeping the list (was it part of a New Year’s resolution?) I am fairly certain that I was influenced by an early reading list I found on the Internet, Eric W. Leuliette’s “What I Have Read Since 1974“.

As a developer (even back then), I decided I would build an elaborate relational database to store my reading list. Over the years, it went through many iterations, and forms. When time became short, I moved the list out of the database and into Excel, or Google Sheets. Finally, several years ago, I settled on a plain text file using Markdown format, and that is how I’ve kept my list ever since.

But I’ve been bothered by shortcomings on this list. There are redundancies I don’t like about it. I have no easy way of referring to books or authors separate from the list. There are things I’d like to automate about it but that the format makes tricky.

A Brief History of My Reading Notes

With all of the reading I do, I have trouble remembering important details of what I read about. So I started keeping notes on my reading. This evolved out of how I kept notes on my reading back in college, and has continued to evolve over the decades since. It was in college that I first decided it was okay for me to write in my books. After all, if I was spending so much money on them, I might as well make them my own, right?

These days, I highlight books, writing margins, and with e-books, I highlight and make short notes on my Kindle devices and apps. But I still have no good way of aggregating these notes into useful groups, categories, and certainly no way of readily searching them.

As I started using Obsidian, and began to see how I could better organize my books and reading lists in its vault structure, I began to get a hint of ways that I might start to link my reading notes back to the books they are associated with, my reading, and other notes.

Enter Zettelkasten

I’d never heard of Zettelkasten before I started using Obsidian. Zettelkasten was originally invented as a way to link paper notes together to be able to easily create connections (links) between then. While it was workable on paper, such a process could be greatly improved with hypertext tools, and it so happens that Obsidian’s note-linking capability is idea for this.

One important idea from Zettelkasten is that a note should contain a single thought or piece of information (say, a passage highlighted in a book). That note is given a unique identifier. In addition to the passage, one would add their own thoughts to the note, and perhaps further link that note to other notes and ideas that are related to it. Zettelkasten has its own unique numbering system for “naming” the notes. Obsidian has a plug-in for creating a “Zettelkasten number” for this purpose that is based on the date/time the note is created. I wasn’t particularly fond of that identifier because it duplicates information already contained in the note itself. After all, the note is just a file in the file system, and has its own create and modified date/times as part of the file. A good identifier does’t embed real data. It’s just an identifier.

I also struggled a bit to figure out how this would work for my reading notes. I originally imagined that if I had a note for each book I read, I could simply add my highlights and annotations to that note. Zettelkasten, however, suggested that rather than adding that highlight to the book note, I’d create a separate note for just the highlight or annotation, and then link it to the book note–as well any other notes it might make sense to link it to. This took a while for me to process, and I thought about it a lot as I built out my reading library in Obsidian.

My Obsidian Library

So how did I decided to structure my reading notes in Obsidian? I’ll try to go through the step-by-step process I have for putting this all together, in case someone is interested in reproducing this.

Step 1: Establishing the structure

I decided that because of Obsidian’s great linking capability, I could use the file system itself as a relational database. In deciding this, I further decided that there were 3 main “objects” I wanted to be able to capture at a kind of atomic level. That is, three things that make up the structure of my reading library:

  1. Things I read, e.g., books, articles, stories, etc.
  2. Authors: the people who write the things in #1.
  3. My notes as they relate to #1 and #2.

From this, I established the following structure of folders in within my Obsidian vault:

My folder structure for Reading notes in Obsidian.
  • Commonplace Book contains all of my reading notes.
  • Library contains all of the “atomic” notes that make up my reading library:
    • Authors: a single note for each unique author in my library
    • Articles: a single note for each unique article (often not tied to a book) in my library.
    • Book: a single note for each unique book in my library
    • Essays: a single note for each unique essay in my library; these are often related to books.
    • Stories: a single note for each unique story in my library.

Step 2: Deciding what goes into a note

Once I had my structure, I had to decide what goes into a note of each type. What is it I want to know about authors, books, stories, etc.? This was fairly easy for me as I’ve been thinking about it for a long time (years, actually). I had in mind an idea that I could write an API that uses these files as a database to query them and produce results. With that in mind, I decided to start by keeping things simple, knowing that I could add detail as needed going forward.

For authors, I wanted just some basic information. Here is a typical author note, in this case, for Alan Lightman, whose new book I read earlier this week:

A sample author note for Alan Lightman.

The backlinks section is generated automatically by a script that I have that runs nightly. I know that I could just click on the “Linked mentions” in Obsidian to see all of the backlinks, but I wanted the related books on the note as a reference in case I access the file outside of Obsidian.

For books (or essays, stories, articles), I also kept things simple. A typical book (or essay, or article, or story) looks like this:

A sample title note for In Praise of Wasting Time

Note that in both authors and books there are links back and forth between the files. The book file refers to the author. The author file has link references back to the books. Moreover, you’ll note that in the book, there is an “Annotations” section with a list of links. These are auto-generated links to my notes and highlights for the book. I’ll have more to say on these shortly, but the important thing is that each note and highlight is a separate file (in the Zettelkasten vain) and is included with the book as a “transclusion” link, meaning that when I view the note in preview mode, it “includes” the links files as part of the note, like this:

Title note in preview mode with transcluded annotations visible.

Step 3. Populating the database

Once I had the structure I wanted, I needed to populate my database. I was fortunate in this regard on 2 counts: (1) I happened to recently create a SQLite database of my books, and (2) I can write code relatively easily. I wrote a script that crawled my book database, and from it, creating the notes for books and authors in Obsidian. This turned out to be a surprisingly simple exercise. (The Python script was 130 lines.)

My digital commonplace book

I first learned of commonplace books reading a biography of Thomas Jefferson (in this case, it was Williard Sterne Randall’s Thomas Jefferson: A Life.) Jefferson (and others in his time) would copy passages from their reading into a book. This helped with memorization, but it also provided a resource where they could add notes and observations. I’ve always liked this concept, and I decided that Obsidian would finally allow me to put it into action in a way I’d envisioned.

It is trivial to create a note and add it to the note containing the book to which it is related. But what if the note ultimately relates to more than one thing? Reading about Zettelkaten provided me with insights into how I might handle this. The naming convention in Zettelkasten (and the way it is implemented in Obsidian) bothered me. Neither made much sense. How do you search for things with essentially coded filenames?

I was in the shower when I finally had a breakthrough insight on this. I’m not searching for a filename, I’m searching for file content. If each annotation and highlight I can link it to as many notes as makes sense. Furthermore, I can add tags to each note. The name of the file doesn’t matter. What matter is how it links to other notes, and that all files are searchable.

I still didn’t like the file-naming scheme for Zettelkasten in Obsidian, which essentially uses a datetime stamp down to the current second. So a file might be named: 20210215084456. Given that one is not likely to create two of these notes within the same second, it guarantees uniqueness. But from a database perspective, identifiers like these are not supposed to embed any information. They should be strictly identifiers. Moreover, the with the date embedded in the note title, I would be duplicating information that already exists in the file properties.

I decided instead to use a Guid, or what is sometimes called a UUID. This is another form of a unique identifier that doesn’t embed information, just produces a unique code. (For those tech-savvy folks reading this, I used Python’s UUID4 which doesn’t use the MAC address as part of the identifier.)

When I have a new note or highlight for a book, it goes into my Commonplace Book folder in Obsidian. These notes also have a specific structure. A typical one looks like this:

A typical note, Zettelkasten style.

Each annotation begins with a Source that links back to the source for that annotation. It may or may not have tags associated with it. That is followed by the body of the annotation, which may be a highlighted passage. Finally, there are my own notes related to the specific passage. In the above example, my notes also link to another book, making this particular annotation related to more than one note. That is, a link has been created between Creativity, Inc by Ed Catmull and Amy Wallace, and On Writing by Stephen King.

Automating my annotations

Over the weekend, I got a start on automating these annotations. I wrote a Python script that reads a CSV files exported from Kindle, and creates a unique note for each annotation in the file, relating it back to the source book in my Library. My process is roughly this (I say roughly because this is still new):

  1. When I finish reading a book, I export the annotations from my Kindle, which sends me an email. That email has a CSV attachment which I save in a folder.
  2. A script runs, and processes and CSV files I have in the folder, creating the notes and links.
  3. The script, outputs a list resulting annotations for each file. I copy this and paste it into the “Annotations” section of the source book or article. That makes it easy to view the annotations inline when previewing the note. An example of the output from the script looks like this:
Output from my annotation import script.

Toward an API for my books and annotations

I am able to do the above automation because I have a standardized structure to my books and author notes. That standardization allowed me to write an API for my book library. From this API I can, for instance, check to see if a title exists in my library already. I can grab information about a book or author and then use it in some way. The API typically returns data in JSON format. For instance, if I call the function biblio.search_by_title("Beyond"), I get a JSON formatted return containing the following:

[
   {
      "title":"_Beyond Band of Brothers: The War Memoirs of Major Dick Winters_",
      "link":"[[Beyond Band of Brothers (334)]]",
      "type":"book",
      "authors":[
         {
            "author":"Winters, Richard",
            "authorFirstLast":"Richard Winters",
            "authorLink":"[[Winters, Richard]]",
            "gender":"None"
         }
      ],
      "source":"",
      "date":""
   },
   {
      "title":"_Beyond Apollo_",
      "link":"[[Beyond Apollo (58)]]",
      "type":"book",
      "authors":[
         {
            "author":"Malzberg, Barry N",
            "authorFirstLast":"Barry N Malzberg",
            "authorLink":"[[Malzberg, Barry N]]",
            "gender":"m",
            "alternateNames":[
               {
                  "name":"Barry, Mike",
                  "nameLink":"[[Barry, Mike]]"
               }
            ]
         }
      ],
      "source":"",
      "date":""
   },
   {
      "title":"_Beyond the Blue Event Horizon_",
      "link":"[[Beyond the Blue Event Horizon (259)]]",
      "type":"book",
      "authors":[
         {
            "author":"Pohl, Frederik",
            "authorFirstLast":"Frederik Pohl",
            "authorLink":"[[Pohl, Frederik]]",
            "gender":"None"
         }
      ],
      "source":"",
      "date":""
   }
]

The results so far

I’ve linked all of this together using my master reading list note. This note contains a list of everything I have read since 1996 and serves as a kind of index to my reading:

A sample from my reading list index note.

A big part of the way Obsidian works is that it can show you the relationships between your notes. While I am still working on importing all of the notes I have in my Kindle, I can already see a a network of relationships when I view the graph of my Obsidian vault:

A graph of the relationships between all of my notes.

Most of my notes are book and reading-related at this point. That big dot in the center is the master reading list illustrated above. If I highlight it, this is what I see:

Sample of a highlighted node on the graph.

From there, you can see other nodes and relationships that have started to form. For instance, if I hover over one of the Alan Lightman books I finished yesterday, In Praise of Wasting Time, you can see a little network of links coming off that book:

Some of those links point to annotation files. Another points back to the note for Alan Lightman. And a few of the annotation links point to seemingly unrelated notes.

Here is another example. One of the big nodes is for John W. Campbell, editor of Astounding Science Fiction in the late 1930s through his head in the early 1970s. I read many of those old issue when I was taking my Vacation in the Golden Age of Science Fiction. So Campbell shows up a lot on my master reading list:

Highlighting an author node on the graph.

You can see that Campbell is linked to all of the issues of Astounding that I have read. I have started to bring my notes in for those issues. If we look at the July 1939 issue, for instance, you can see this is related to all of the stories and articles and authors in that issue:

Currently, the notes for each story are part of the story note, but I plan on breaking those out into their own Zettelkasten-style notes as I’ve done for my other notes and annotations.

Conclusions

Keep in mind, that this is all being done with plain text files, something that I like because the format is compatible virtually anywhere. This could be done as easily on a Windows machine as a Mac. It could be done easily on a Linux machine. The openness and longevity of plain text (which has been around for fifty years now) is a big part of what I like about this system.

The linking that Obsidian provides from within its application makes all of this useful. But once established, those links are just as useful outside Obsidian with a little coding–as I’ve done with my API for books and authors. And this API is extensible. This week, I plan to add capability for the API to return any annotations when returning a “book” object. So in addition to what is returned by the JSON format illustrated above, that will soon contain a node for annotations related to that book.

Mostly, I am satisfied that I now have a simple way of keeping my reading notes in a useful form. These are easily searchable, they are easily linked. I can continue to capture highlights and brief notes as a I read. The import function allows a nice step to expand on my annotations as I review them after they’ve been pulled into Obsidian.

It did take me some time to get the infrastructure in place, but now that it is there, I am able to focus on reading, notes, and let the system organize them for me.

Automating My Daily Notes with Obsidian

ETA: 4/1/2021: I have now posted my code for my daily notes automation script on GitHub. See this post for more details.

For nearly a week now my Daily Notes file in Obsidian is generated automatically each night after midnight. I wrote a Python script that creates the note, and having the script running as a Launchd job–which these days is recommended on Mac OS over cron jobs. So far, this has been working out beautifully for me. Of course, this is in large part because Obsidian itself works so well for me.

I use my daily notes in much the same way as I used daily notes in my Bullet Journal days. Indeed, I store my daily notes in my Obsidian vault in a folder called “Bullet Journal”. In addition to having reference information right up front, I use the Today’s Notes section of my notes for jotting various notes down throughout the day, that serve as a kind of log of activity, ideas, and things to do.

The basis of the design of my Daily Note drew inspiration from this post on the Obsidian forums.

Here is a look at my annotated Daily Notes file for today, Monday, February 8.

An annotated version of a daily notes files automatically generated by my scripts.

1. Note title

My Python script generates a note title for the note in yyyy.mm.dd.www format. I used this format back when my Bullet Journal was on paper and so I’ve kept it going here.

2. Navigation/weather bar

My script generates a navigation bar to the previous day’s notes, and to tomorrow’s note (which doesn’t exist yet, of course). It also make a call to a weather app to pull in a very short version of the weather for the day for my location.

3. Agenda

Next, the script generates the “Agenda” section of my note. It does this by calling a command-line program called “icalBuddy” (for Mac OS) which allows you to grab information from your calendar. The published version of icalBuddy had an issue with accessing calendars from a cron job or launchd, but I found post where a person was provided with an updated version of the binary that fixes the problem. The only place I’ve been able to find that binary is in that post, but it worked perfectly once I replaced the original binary.

Each night, a call to icalBuddy is made and the resulting events on that day’s calendar are returned. I then render those events in the Agenda section of my Daily Note.

4. To-Do

This section lists all of the tasks I have that have not yet been checked off. This was a little tricky. The script searches for all tasks in my notes and splits them into 2 groups, completed and incomplete. It then hashes the tasks and removes any “incomplete” tasks that appear in the “completed” task list. This prevents seeing uncompleted tasks that were completed on a subsequent day. The result is that I only see tasks that I haven’t completed.

4a. Illustrates what happens when tasks are carried over day-to-day without being completed. My script appends a note indicating how long the task has been open. For instance, that “Add functionality for notes timeline” task has been open for 2 days (since Saturday).

Throughout the day, I’ll check off tasks here and add new ones and those get incorporated into the next day’s runs.

5. Reading

I try to read at least one feature article from the various magazines I subscribe to every day. This is where I record what I read. I’ve actually developed a fairly elaborate reading and book notes system in Obsidian that I will write about in a future post. What I do here is jot down the article title, author, and source in a specific format (e.g., “Article Title” by Some Authors | Source: Magazine (Month Year)) and a separate script adds this information to my Library notes.

6. Today’s notes

This is where my bullet journal-style daily notes go. These can be virtually anything that I think is noteworthy. As you can see, when I opened the paper this morning, I saw that George Shultz died, and noted that here.

I’ll put to-do items here. If I am working on a something, I’ll make notes about what i am working on. Ideas get logged here. Funny remarks I hear throughout the day might find their way here. It can be quite a mish-mash of stuff, but it ends up being a useful reference.


I suspect that some people will ask if I can share my scripts. I am happy to do that, but not quiet yet. I tend to write scripts the way I write stories, the first draft is quick and dirty and functional. The second draft is cleaned up and organized. Once I get the “second draft” written, I’ll post what I’ve written on GitHub and anyone who wants can make use of them.

Some of this functionality is built into existing Obsidian plug-ins, but it is either not automated, or not quite the way I wanted it, so the scripts that I have written are tailored to my specific needs and behaviors. It’s nice to wake up in the morning, open Obsidian, and see my daily notes file waiting for me.

Notes with Obsidian: My Initial Impressions

Now that I have been using Obsidian fairly exhaustively for the last 5 days, I think I’ve got enough experience with it to share my initial impressions.

I mentioned earlier that Obsidian was the notetaking tool I’ve been looking for all my life. That begs the question: what is it that I’ve been looking for in notetaking software to begin with? Here are a few of my requirements. (Your mileage may vary.)

My requirements

Notes should be plain text

There are two kinds of notes I deal with:

  1. Ephermeral notes, or those notes that I am jotting something quickly for later use, but that can ultimately get thrown away because they will be transformed into something else. Examples of this might include a person’s name, or phone number, or a rough set of notes from a phone conversation.
  2. Permanent notes: the kind that I’ll continue to use, update, and search into the forseeable future. Examples of these include how-to notes, daily notes, notes and clippings from reading. Drafts of blog posts, lists I maintain.

When I think of notes, I think of the need for versatility. I keep a Field Notes notebook in my pocket at all times to capture many of the type 1 notes above. I’ve found over the years that is much quicker than pulling out my phone, unlocking it, finding a notes app, and tapping out what I want to capture. To that end, the most versatile file format for digital notes, for me, anyway, is plain text. Plain text has many advantages:

  • I don’t have to worry about the file format. Plain text files created in the 1980s can still be read today with any old text editor.
  • Plain text files are easy vehicles for automation. Automation is a big reason why I’d want to keep notes in digital form in the first place.
  • Plain text files are cross-platform compatible.
  • Plain text files are easy to search and are the perfect targets for advances searches with regular expressions.
  • With Markdown, even plain text files can be rendered with formatting.

Notes should be rendered as Markdown

When I have looked to convert all my notes (and even my writing) to plain text in the past, one thing that has help me back is the lack of a good tool for rendering the Markdown in the files. It never looked like how I wanted it. Most text editors are really for writing code, and they bias their features in that direction.

This is one place where Obsidian excels. It has a great UI, one that is designed for notetaking (instead of code) and one that renders Markdown well. It is also flexible enough to allow custom themes so that really, you can make your plain text notes look pretty much however you want.

Notes should be easily linkable

A few years back, I tried to put together a system that allowed me to create links between my notes. I understood intuitively how useful this would be. But it seemed like a difficult problem, and really, I didn’t want to spend my time writing code, I wanted to work with my notes.

Obsidian’s note-linking capability–one of its central functions–eliminates this problem, and removes all the roadblocks to linking notes. Not only have I jumped on this feature, I have even started down the Zettelkasten rabbit hole (something I will discuss in a future post). Being able to easily link notes, follow those links, and see the relationships they form is a huge plus in favor of Obsidian.

Notes should be kept locally

I like that underneath it all, these are just plain text files sitting on my local file system. Of course, I can store them in iCloud or Dropbox or some other cloud service if I want to, but the actual files are local. This just helps with automation.

My initial impressions of Obsidian

Where to begin?

  • I just love the UI. It is clean, and easy to use. It makes me want to live in the app when I am working on the computer.
  • The linking is fantastic.
  • I also love that is has a folding feature built in which allows you to easily show and hide sections of a document.
  • I love that moving notes around (which I do often for organizational purposes) keeps the links up-to-date. I don’t have to think about it, it just work.
  • I love not having to save the notes. They save as I type. Important since more and more tools work this way and I’ve gotten use to not having to manually save documents.
  • Great searching capabilities! I can even search regular expressions, which comes in hand.
  • The Daily Notes feature is awesome and I’ve started using it to create a kind of digital bullet journal.

I’ve liked what I’ve seen enough that I became an Obsidian Supporter. I find the forums and documentation both useful.

Some use cases to illustrate and illuminate

Let me illustrate some of what I like about Obsidian with some real use cases that I’ve got just in my first five days using the tool.

Reading notes

I take notes when I read. I mark up books (real and digital). Ultimately, I like to get those notes together in some useful fashion. (This is the one area in which I have really started to explore Zettelkasten, and I’ll have more to say about it in the future.) Here is one simple example of what I have started to do in Obsidian.

I try to read one feature article from the various magazines I get every day. When I finish an article, I create a note for the article with information about the author(s), source, date, a link to the online version, if available. I also tag the note.

In order to build a kind of searchable index, I have started to create a note for each author. On the article note, I’ll link to the author note, like this, for example:

An Obsidian note for an article in Smithsonian by Glenn Adamson

Note that I have linked to the the author on this note. So when I go and look at the note I have for that author, I see this:

Obsidian note for author with backlinks section listing all of the backlinks

The “Backlinks”section on the note is created automatically through some automation I’ve written. Again, I’ll talk about how I automated this in a future post. I know that I can just go to the backlinks section on the sidebar to see backlinks, but I like seeing the actual references in the note itself.

Presenting with notes

Block folding is incredibly useful. That is, being able to close some blocks and keep others open while looking at a note. Here is an example of this post (yes, written in Obsidian) so far with most of the blocks closed.

Obsidian note with folded headers

I find this allows me to use my notes as presentations in meeting. I’ll jot down the topics I want to discuss in the meeting. List out the points under those topics, and then, before the meeting starts, close all of the headings, so that I can walk through them one by one, sharing my notes on the screen. It allows us to focus only on the open block at hand, while still having a context for the rest of the discussion. No need for slides!

Searching for daily notes

One test I have for any good editor is how well it can search. In addition to the great capabilities that Obsidian has for searching already, it also passes my test for being able to do regular expression searches. Here is a regular express search I have ato surface all of my Daily Notes:

A regular expression search for my daily notes: /^\d{4}\.\d{2}/

I have actually gotten quite a bit of mileage already out of the daily notes features, especially after I added some automation to make it even better.

How I’ve got my Obsidian configured

I’ve already had people asking me thing like what themes I am using, so here is a summary of how I have Obsidian configured for my Macs:

![[Pasted image 20210131113120.png]]

Obsidian configured with my preferences.
My Obsidian configuration

Theme

I’m using Pisum (from the community themes). Incidentally, I did spend time loading every community theme to find one that fit well. I kept a note open and moved the ones I liked to the top of the list as I discovered ones I liked better.

Plug-ins

I’m currently using the following plug-ins:

  • Calendar
  • Daily Notes
  • Zettelkasten prefixer

Other settings I’m using

  • Fold Heading
  • Tab size: 8

Questions I’m still pondering

There are some things I am still trying to figure out, and I’ve been searching the forums to see how others approach these issues:

  1. What should be in my vault. When I started, I created my vault as a Notes folder within my Documents folder. From there, a sub-folder structure began to evolve. But I’ve found that I want to link to documents that are not in the vault, so I am beginning to wonder if really, the vault should be my entire documents folder. I’m looking for advice on the scope of a vault in Obsidian.
  2. Zettelkasten. This seems like overkill for a lot of the notes I have. Put it seems like the perfect solution for how I have envisioned organizing notes from my reading. I used to have a note for a book. All my comments and highlights would be in that note. Now, I have a note for a book, but highlights each get their own note with (a) a link back to the book in question, and (b) a reason for the link. I’ve even started linking some of these extracts to other notes, formiing the web of relationships that Zettelkasten is all about.
  3. Best format for my Daily Notes. I’ve actually done some automation here and getting happier with what I’ve got. But I’m still figuring it out. Here is an example of my Daily Note for today:
An example of a daily note
An example of a Daily Note

Automation preview

So far, I have automated 3 things using Obsidian notes and the plain-text framework:

  1. Collect “to-read” references into a single note. This collects tags marked “to-read” throughout my notes into a single “To Read” note so that I have a single list to look at. My script will ignore items tagged to read if they have already been checked off.
  2. Update People index. This is the script that scrapes notes for backlinks to specific people notes and adds the “Backlinks” section to that person’s notes.
  3. Daily Note generation. This genarates the daily note for the day (just after midnight). I pulls in the agenda from my iCloud calendars. It pulls the to-dos from incomplete to-dos found in other notes, and from any incomplete items in my Apple Reminders. It then formats the other sections and creates the note.

I’ll write about these automations in more detail in a future post.


So there you have it, my initial impressions of Obisian. I’m looking forward to exploring it more. And I’m eager for that mobile app that’s “coming soon.”

One More Note on (Obsidian) Notes

Okay, okay, you’ve convinced me. I’ve been playing around with Obsidian (h/t to Matt Banks for initially bringing it to my attention) for a couple of days now. I think it is the notes system that I have been looking for all my life!

I’m still getting used to some of the ins-and-outs, but I really like what I see so far, and I’ve started converting my Apple Notes (the ones that are not throw-aways) to Obsidian. I am also experimenting with using it as a digital bullet journal, similar to how I use OneNote in my day job for the same purpose. Indeed, I’m using my work to test out the bullet journal aspects of using Obsidian.

It is going to take me a few days to get my thoughts together, as I need time to play with it more. But I was convinced enough not only to try it, but to become a supporter as well.

Thanks again to all who mentioned it. I’ll have more to say about it soon. In the meantime, I’ll leave you with this preview of an Obsidian note for reading notes from a book I recently finished.

A sample Obsidian note