Tag: automation

My New and Improved (and Automated!) Reading List

A stacked date list plot of the number of books I've read by year since 1996

Late last year, I set 3 goals for myself for 2023: (1) consolidate the apps that I use; (2) simplify; and (3) automate repetitive tasks.

One repetitive task I’ve been dealing with for decades is maintaining and publishing my reading list. I started keeping my reading list back in 1996 in order to track a goal that I’d set for myself: to read one book per week. Initially, the reading list was in a text file. That was followed by Excel, HTML on an early website, a SQL database, more HTML, a plain text file on GitHub, and most recently, a markdown file that I host through Obsidian Publish.

None of my “routine” tasks have proved more time-consuming than maintaining this list. Frequently, I update in one place, and then have to make similar updates in other places to keep everything in sync. If I could automate this, it would be a huge win. Recently, this is just what I have done.

I created an Excel spreadsheet which is now my canonical master reading list. This is the only place I make updates. I wrote a Wolfram Language script that generates my reading list markdown file as well as a new reading stats page from the spreadsheet. I then simply publish the changes in Obsidian Publish, and presto! A single process for doing it all!

You can see the results on my reading list site. In addition to automatically generating the list, my script also produces the new Reading Stats page, which has all kinds of charts and visualizes of my reading over the decades, with more in the works. I no longer have to do this manually myself, and it has proven a big timesaver already.

To cap things off, I wrote a new About the List page, which gives the history of my reading list and provides an FAQ. Eventually, I’m looking to automate this further by having Wolfram Language detect when my reading list file is updated and automatically produce the updated files, and publish them.

I’m pleased with this new process since it saves measurable amounts of time (considering how much I read and, therefore, how frequently I’m updating my list), which means more time for other things. Like writing more here on the blog.

Written on 6 May 2023.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

Goals for 2023

pexels-photo-13088176.jpeg
Photo by Engin Akyurt on Pexels.com

Over the years I have spent quite a bit of times on various experiments. In the 2010s, I considered the idea of the paperless office, embraced Evernote as a tool for paperless productivity, and wrote a popular series of posts on the subject. So far, in the early 2020s, my experimenting has shifted somewhat. I’ve re-embraced the idea of plain text files as my fundamental unit of work. A plain text file is both versatile and long-lasting. Text files created in the 1970s can still be read today. I found Obsidian, and took lessons from my Going Paperless experiment, to see what I “practically” paperless[1. As in “practical” uses, as opposed to absolute migration to a paper-free lifestyle.] lifestyle looked like using primarily plain text files.

In 2022, I’ve been pleased with how things are working out with Obsidian and plain text files, but there is room for improvement. I’ve identify three areas to experiment with throughout 2023. My hypothesis for this experiment is that improvements in these three areas will lead to more creative time in the years beyond. The three areas are: (1) Consolidation; (2) Simplification; (3) Automation.

Consolidation

When I look at my computer, I am often overwhelmed by all of the apps and tools I find on it, the vast majority of which I rarely use. It would be nice, I tell myself now and then, to get rid of those tools that I never use, but I am sometimes reluctant to do so, thinking that I might need the tool some day. To some extent, this is no different than trying to declutter at home.

While thinking about this recently, I realized that I made huge strides in consolidation over the last two years in terms of my work products, the vast majority of which are now plain text files. With a work product as simple and versatile as a plain text file, it should be much easier to give up apps I no longer need to manipulate those files.

The more I think about this, the more I believe I can get away with two primary work product formats: plain text files used in conjunction with Obsidian; and Wolfram Language notebooks (.nb files) used in conjunction with Wolfram Language and Mathematica.

Using text files as work products of Obsidian is pretty obvious and I’ve written extensively about my use of Obsidian. But Mathematica1?

I’ve been a very casual, hobbyist user of Mathematica for years now. But the more I play with it, and the more I learn about its symbolic and functional structure, the more impressed I am with the language. Moreover, the language now reaches into all aspects of computing so that it seems useful as a general purpose language. Finally, as a developer, I’ve already mastered more common languages like Python, JavaScript, C#, Perl, PHP, Ruby, etc., to my personal satisfaction. Becoming equally proficient in Wolfram Language would something new.

I am mentally dividing this consolidation into two parts:

  1. Plain text files as my primary work product: in the form of notes and writing.
  2. Wolfram Language scripts and notebooks as my primary tool for development and automation.

As I write this, there are about 100 applications in my Application folder on my laptop. That, of course, does not include the command line tools inherent in any Unix-based computer system. I’m using this number of 100 applications as a baseline to see how much I can consolidate over the course of 2023. Obviously, I need more than just Obsidian and Mathematica. For instance, I use Photoshop for photo editing. However, Mathematica has powerful photo- (and video- and sound-) editing tools that may be able to automate and supplant most of what I do in Photoshop. This is an example of the consolidation I am looking to do.

It will be interesting to see how well this consolidation works over the course of 2023.

Simplification

As I worked through my Practically Paperless series, I noted that I was building up more and more complex structures for my notes, as well as relying increasingly on plug-ins, tags, etc.

In the months since finishing that series, I’ve been attempting to simply how I use Obsidian with plain text files. In the process of simplifying, I have the following things in mind to guide me:

  1. My primary work product is a text file. Whether a note, a blog post, a list, a trip summary, a reading note, the primary form in which the content appears is a plain text file.
  2. Using tools like Obsidian is a good for me, but I also keep in mind that these text files need to be useful in the absence of a tool like Obsidian. Keeping the work products themselves simple help in this regard. For example, the structure of markdown makes for simple way of parsing and interpreting these files outside of Obsidian.
  3. I’m trying to simplify my file structure, both within Obsidian and as a whole, including cloud platforms.

Over 2023, I’m looking to continue this simplification of my work products. With the following ideas in mind:

  1. My notes need to be usable outside Obsidian.
  2. My notes and files need to be easy to understand, not just for myself, but for others, especially Kelly and our kids.
  3. My notes and files need to be easy to find for me and for the family.

Automation

I spend a lot of time trying to automate processes. The idea here is to automate the stuff that is repeatable, so that I can spend more time on creative stuff. But often, the time it takes to implement such automation offsets the time it saves. Put another way, if it takes me 2 hours to write a script to automate a task that currently takes me 5 minutes per day, it will take 24 days once the automation is in place to pay back the time it it took to build the automation before it starts paying off.

In the past I’ve used all kinds of tools to implement automations: Python scripts, tools like Alfred or Keyboard Maestro, Apple Shortcuts, etc. In 2023, I am looking to use Wolfram Language as much as possible to automate things, in part because I’m fascinated to see the possibilities in the language, and in part as a way of becoming proficient in the language.

One simple example of this, which I will talk about in more detail in a subsequent post, is adding a list of files I created or updated on a given day to my Daily Notes. it is convenient for me to be able to see at a glance all of the files I worked on in a given day. The way I structure my daily notes makes the daily note the perfect place to store this information. But I don’t want to spend time searching my file system each day for all of the files I created or modified. So I’ve created a Wolfram Language script to do this automatically.

An example of my current Daily Note format, using today's note.
An example of my current Daily Note format, using today’s note.

Throughout 2023, I’ll be looking at how I work2 to identify inefficiencies, things that can be eliminated, and tasks and processes that can be automated, particular:

  1. Things I do frequently and are repeatable. In addition to the above, examples, might include: taking a blog post in note form and automatically publishing it to WordPress; or cropping and editing a photo in a repeatable way
  2. More complex tasks that I perform less frequently but at regular intervals, like archiving information.

Follow along on the journey

Because I know that other people are interesting in this sort of thing (a quick search of “productivity” gives an idea of just how much), I plan to write about this journey of mine throughout the year. As always, I am experimenting to see what works best for me. I know, for instance, that using Wolfram Language for a primary automation and scripting tool is probably outside the norm for many people. But I’m fascinated by the idea and that makes it fun for me. These will tend to be more technical posts, but they might be of interest to others, so, feel free to follow along, ask questions, and offer suggestions.

And Happy New Year, everyone.

Written on 29 December 2022.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. There are other reasons for me to use this, but they are too technical to get into here.
  2. To be honest, I’m doing this constantly, but this year, I’ll be putting a special emphasis on it.

Keyboard Maestro: An Almost Invisible Superhero of Productivity

a person typing on a keyboard
Photo by Florenz Mendoza on Pexels.com

There are some tools that I use that are so seamlessly integrated with my workflow that they are almost invisible. They work so well that I simply don’t think about them. That really isn’t fair since tools like these do far more than they get credit for. And since I had a particular tool in mind today, for reasons I will explain, I thought I’d write about one such invisible superhero of productivity: Keyboard Maestro.

Keyboard Maestro is made by Stairways Software and it is Mac-only. If you are a Windows users, you can read with envy. If you are a Linux users, you probably don’t need a tool like Keyboard Maestro in the first place.

I have been using Keyboard Maestro for just over 8 years1 and I’m not sure I can quantify how much time it has saved me. Keyboard Maestro is an automation tool that combines three useful functions in one:

  1. Text expander
  2. Clipboard manager
  3. Automation engine

I think I first began using Keyboard Mastro as a substitute for TextExpander in order to get the expansions that I wanted. Keyboard Maestro can do text expanding, but even there it is powerful. It can take a simple expansion like ;;email and expand it to my email address. Honestly, I can’t remember the last time I had to type out my email address. Quite possibly it has been years. Like many text expanders you can set one phrase and have it expand to another.

But it can do more. It can perform transformations on the data. I wrote about a fairly simple version of this when I described how I used Keyboard Mastro to automatically add the next entry number to my journal in Obsidian. I have all kinds of useful text expansions and they have become so innate that I don’t even notice them. It would be fascinating to know just how much typing Keyboard Maestro has saved me over the years (and how many typos it has prevented).

Keyboard Maestro also serves as my clipboard manager. I am constantly copying things to the clipboard and pasting them other places. Keyboard Maestro keeps a history of my copies and allows me to access that history when I paste. Here is a glimpse of what that clipboard history looks like at the moment:

an image of my clipboard history in keyboard maestro, showing the last 5 copies I made, all ready for pasting


In addition to containing text, the clipboard can contain images. And when I paste something from the clipboard, I have the option of transforming it as I paste. Examples of this transformations include:

  • changing the capitalization of the text (all caps, all lowercase, title case, etc.)
  • manipulating an image by flipping it, rotating it, changing its size

This takes a lot of manual effort out of routine and reptitive tasks.

It was this clipboard history functionality that got me thinking about writing this post. I occasionally made use of PasteBot for this, but recently decided to consolidate this functionality into one app–much as I did with TextExpander. And since Keyboard Maestro does this so well, that was what I went with.

Keyboard Maestro can also do all kinds of automation, something I have only experimented with lightly over the years2, but something I am interested in making more use of.

As I said, Keyboard Maestro works so well, it is virtually invisible. I often forget I am even using it when I am typing. So I wanted to call attention to it, because it really is a powerful, useful, time-saving tool.

Written on March 11, 20223.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

  1. I just verified this, going back through my email to find the receipt from my first order, dated March 3, 2014.
  2. For an example, see how I used Keyboard Maestro, years ago, to automate pulling notes and highlights of my Kindle. I don’t use this automation anymore, but it is a good example of what is possible.
  3. Yes, even this dateline that you see at the end of each post is generated by a Keyboard Maestro macro that I created

“Hey Siri, Let’s Nap”

pexels-photo-699122.jpeg
Photo by Tyler Lastovich on Pexels.com

More than a year ago I began exploring the Shortcuts feature that Apple introduced to iOS (and later, into MacOS). It took me a while to figure out what the shortcuts were all about and longer to figure out a practical use for shortcuts. There were plenty of examples of things that people were using shortcuts for, but I was looking for something practical for me. Eventually, I found it.

At the time, it was my job to put our youngest daughter down for her nap, and I usually napped with her. We napped at lunch and our routine was always the same. I’d look for when my next meeting was–if I had one within the next 2 hours–and set an alarm so that I’d wake up before my meeting. Then I’d adjust the volume of my phone and put on a playlist that we’d listen to while we napped. Inevitably, my phone would buzz with some alert or other so I’d then remember to put my phone into Do Not Disturb mode. Of course, I’d forgot to take it out of DnD and wonder why I saw no notifications for the rest of the afternoon.

Could this all be handled by a Shortcut?

It took me an hour of two of playing around with shortcuts, but I finally created one that did just what I wanted it to do. Because Siri can activate a shortcut by voice command, I named my shortcut “Let’s nap.” When we were ready to take our nap, I’d set my phone down and say, “Hey Siri, Let’s Nap.” And Siri would take care of everything else.

Here is what my shortcut does:

First, it sets the volume on my phone to 22%, which experiment has shown is the perfect volume for napping. Not too loud, and not too soft:

Next, the shortcut plays a playlist that we listen to while we nap. The playlist is called “Grace Music” because it is a playlist that I used to play for my older daughter when she would nap. The shortcut sets the playlist to repeat indefinitely.

This is where things get interesting. All I wanted to do when it was time for us to nap was say “Let’s nap” and have the Shortcut take care of everything else. So, what it does next is check my calendar for the next meeting that shows up on the day the shortcut runs:

Having gotten the next event, I filter the list of upcoming events for things on my calendar. It seems like I am filtering twice here, but for some reason that I can’t recall at the moment, this was required to find the next event.

If the event is within the next 2 hours, then I grab the start date of the calendar event and subtract five minutes from it. I use that time to create an alarm so that I get woken up before the next meeting (assuming I am not already awake, which I usually am.)

I also turn on Do Not Disturb until the same time the alarm goes off:

Now, if there is no event on my calendar, or there is an event more than 2 hours in the future, then there is no need to set an alarm. I just set Do Not Disturb until 2 hours from now and that’s it.

My youngest daughter stopped napping for good at the end of the summer when school started and she began kindergarten. But I enjoy my lunchtime naps and I still use this shortcut nearly every day. It works like a charm. I eat a quick lunch, lay down, say, “Hey Siri, let’s nap” and Siri and my shortcut take care of everything else for me.

Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!

Follow Jamie Todd Rubin on WordPress.com

A Quick Update on My Obsidian Automation Scripts

I’ve had a number of people reach out recently to inquire about my Obsidian automation scripts. I will get them posted to Github in the relatively nearly future. In their current state, they are not ready for public consumption. There are lots of hard-coded paths, to say nothing of the code is fairly ugly. I’ve cleaned up about half the code and moved it into a Python package. I still need to work on the other half.

The problem, as always, is time. I am working on a day-job crunch as I prepare to roll out some software my team has been developing for the past year. That really limits my time in March and April. So at this point, May is looking like when I will most likely have time to work finish my cleanup of the Python code.

I promise to post an update once I’ve put the code on GitHub. Until then, thanks for your patience.

How I Capture Reading Notes in Obsidian

Note: December 9, 2021: I wrote this post in February 2021 after I’d been using Obsidian for a little over a month. In the 10+ months since, I’ve learned a lot more and modified my reading notes process based on lessons I’ve learned in what you see below. If you are interested, you can check out this update on how I manage my reading notes in Obsidian today. The stuff below is still useful stuff. But my process has changed some from what you see below.

In addition to automating my daily notes with Obsidian, it quickly became clear to me that Obsidian‘s note-linking capabilities would allow me to capture my reading notes in Obsidian in a really useful way. Moreover, because of Obsidian’s powerful linking capability, it occurred to me that my Obsidian vault could serve as a database for my reading. To describe how I managed to do this (so far) in a step-by-step manner will required a little history first.

A Brief History of My Reading List

I began keeping a list of every book I finished reading back on January 1, 1996. Although I am no longer certain of why I started keeping the list (was it part of a New Year’s resolution?) I am fairly certain that I was influenced by an early reading list I found on the Internet, Eric W. Leuliette’s “What I Have Read Since 1974“.

As a developer (even back then), I decided I would build an elaborate relational database to store my reading list. Over the years, it went through many iterations, and forms. When time became short, I moved the list out of the database and into Excel, or Google Sheets. Finally, several years ago, I settled on a plain text file using Markdown format, and that is how I’ve kept my list ever since.

But I’ve been bothered by shortcomings on this list. There are redundancies I don’t like about it. I have no easy way of referring to books or authors separate from the list. There are things I’d like to automate about it but that the format makes tricky.

A Brief History of My Reading Notes

With all of the reading I do, I have trouble remembering important details of what I read about. So I started keeping notes on my reading. This evolved out of how I kept notes on my reading back in college, and has continued to evolve over the decades since. It was in college that I first decided it was okay for me to write in my books. After all, if I was spending so much money on them, I might as well make them my own, right?

These days, I highlight books, writing margins, and with e-books, I highlight and make short notes on my Kindle devices and apps. But I still have no good way of aggregating these notes into useful groups, categories, and certainly no way of readily searching them.

As I started using Obsidian, and began to see how I could better organize my books and reading lists in its vault structure, I began to get a hint of ways that I might start to link my reading notes back to the books they are associated with, my reading, and other notes.

Enter Zettelkasten

I’d never heard of Zettelkasten before I started using Obsidian. Zettelkasten was originally invented as a way to link paper notes together to be able to easily create connections (links) between then. While it was workable on paper, such a process could be greatly improved with hypertext tools, and it so happens that Obsidian’s note-linking capability is idea for this.

One important idea from Zettelkasten is that a note should contain a single thought or piece of information (say, a passage highlighted in a book). That note is given a unique identifier. In addition to the passage, one would add their own thoughts to the note, and perhaps further link that note to other notes and ideas that are related to it. Zettelkasten has its own unique numbering system for “naming” the notes. Obsidian has a plug-in for creating a “Zettelkasten number” for this purpose that is based on the date/time the note is created. I wasn’t particularly fond of that identifier because it duplicates information already contained in the note itself. After all, the note is just a file in the file system, and has its own create and modified date/times as part of the file. A good identifier does’t embed real data. It’s just an identifier.

I also struggled a bit to figure out how this would work for my reading notes. I originally imagined that if I had a note for each book I read, I could simply add my highlights and annotations to that note. Zettelkasten, however, suggested that rather than adding that highlight to the book note, I’d create a separate note for just the highlight or annotation, and then link it to the book note–as well any other notes it might make sense to link it to. This took a while for me to process, and I thought about it a lot as I built out my reading library in Obsidian.

My Obsidian Library

So how did I decided to structure my reading notes in Obsidian? I’ll try to go through the step-by-step process I have for putting this all together, in case someone is interested in reproducing this.

Step 1: Establishing the structure

I decided that because of Obsidian’s great linking capability, I could use the file system itself as a relational database. In deciding this, I further decided that there were 3 main “objects” I wanted to be able to capture at a kind of atomic level. That is, three things that make up the structure of my reading library:

  1. Things I read, e.g., books, articles, stories, etc.
  2. Authors: the people who write the things in #1.
  3. My notes as they relate to #1 and #2.

From this, I established the following structure of folders in within my Obsidian vault:

My folder structure for Reading notes in Obsidian.
  • Commonplace Book contains all of my reading notes.
  • Library contains all of the “atomic” notes that make up my reading library:
    • Authors: a single note for each unique author in my library
    • Articles: a single note for each unique article (often not tied to a book) in my library.
    • Book: a single note for each unique book in my library
    • Essays: a single note for each unique essay in my library; these are often related to books.
    • Stories: a single note for each unique story in my library.

Step 2: Deciding what goes into a note

Once I had my structure, I had to decide what goes into a note of each type. What is it I want to know about authors, books, stories, etc.? This was fairly easy for me as I’ve been thinking about it for a long time (years, actually). I had in mind an idea that I could write an API that uses these files as a database to query them and produce results. With that in mind, I decided to start by keeping things simple, knowing that I could add detail as needed going forward.

For authors, I wanted just some basic information. Here is a typical author note, in this case, for Alan Lightman, whose new book I read earlier this week:

A sample author note for Alan Lightman.

The backlinks section is generated automatically by a script that I have that runs nightly. I know that I could just click on the “Linked mentions” in Obsidian to see all of the backlinks, but I wanted the related books on the note as a reference in case I access the file outside of Obsidian.

For books (or essays, stories, articles), I also kept things simple. A typical book (or essay, or article, or story) looks like this:

A sample title note for In Praise of Wasting Time

Note that in both authors and books there are links back and forth between the files. The book file refers to the author. The author file has link references back to the books. Moreover, you’ll note that in the book, there is an “Annotations” section with a list of links. These are auto-generated links to my notes and highlights for the book. I’ll have more to say on these shortly, but the important thing is that each note and highlight is a separate file (in the Zettelkasten vain) and is included with the book as a “transclusion” link, meaning that when I view the note in preview mode, it “includes” the links files as part of the note, like this:

Title note in preview mode with transcluded annotations visible.

Step 3. Populating the database

Once I had the structure I wanted, I needed to populate my database. I was fortunate in this regard on 2 counts: (1) I happened to recently create a SQLite database of my books, and (2) I can write code relatively easily. I wrote a script that crawled my book database, and from it, creating the notes for books and authors in Obsidian. This turned out to be a surprisingly simple exercise. (The Python script was 130 lines.)

My digital commonplace book

I first learned of commonplace books reading a biography of Thomas Jefferson (in this case, it was Williard Sterne Randall’s Thomas Jefferson: A Life.) Jefferson (and others in his time) would copy passages from their reading into a book. This helped with memorization, but it also provided a resource where they could add notes and observations. I’ve always liked this concept, and I decided that Obsidian would finally allow me to put it into action in a way I’d envisioned.

It is trivial to create a note and add it to the note containing the book to which it is related. But what if the note ultimately relates to more than one thing? Reading about Zettelkaten provided me with insights into how I might handle this. The naming convention in Zettelkasten (and the way it is implemented in Obsidian) bothered me. Neither made much sense. How do you search for things with essentially coded filenames?

I was in the shower when I finally had a breakthrough insight on this. I’m not searching for a filename, I’m searching for file content. If each annotation and highlight I can link it to as many notes as makes sense. Furthermore, I can add tags to each note. The name of the file doesn’t matter. What matter is how it links to other notes, and that all files are searchable.

I still didn’t like the file-naming scheme for Zettelkasten in Obsidian, which essentially uses a datetime stamp down to the current second. So a file might be named: 20210215084456. Given that one is not likely to create two of these notes within the same second, it guarantees uniqueness. But from a database perspective, identifiers like these are not supposed to embed any information. They should be strictly identifiers. Moreover, the with the date embedded in the note title, I would be duplicating information that already exists in the file properties.

I decided instead to use a Guid, or what is sometimes called a UUID. This is another form of a unique identifier that doesn’t embed information, just produces a unique code. (For those tech-savvy folks reading this, I used Python’s UUID4 which doesn’t use the MAC address as part of the identifier.)

When I have a new note or highlight for a book, it goes into my Commonplace Book folder in Obsidian. These notes also have a specific structure. A typical one looks like this:

A typical note, Zettelkasten style.

Each annotation begins with a Source that links back to the source for that annotation. It may or may not have tags associated with it. That is followed by the body of the annotation, which may be a highlighted passage. Finally, there are my own notes related to the specific passage. In the above example, my notes also link to another book, making this particular annotation related to more than one note. That is, a link has been created between Creativity, Inc by Ed Catmull and Amy Wallace, and On Writing by Stephen King.

Automating my annotations

Over the weekend, I got a start on automating these annotations. I wrote a Python script that reads a CSV files exported from Kindle, and creates a unique note for each annotation in the file, relating it back to the source book in my Library. My process is roughly this (I say roughly because this is still new):

  1. When I finish reading a book, I export the annotations from my Kindle, which sends me an email. That email has a CSV attachment which I save in a folder.
  2. A script runs, and processes and CSV files I have in the folder, creating the notes and links.
  3. The script, outputs a list resulting annotations for each file. I copy this and paste it into the “Annotations” section of the source book or article. That makes it easy to view the annotations inline when previewing the note. An example of the output from the script looks like this:
Output from my annotation import script.

Toward an API for my books and annotations

I am able to do the above automation because I have a standardized structure to my books and author notes. That standardization allowed me to write an API for my book library. From this API I can, for instance, check to see if a title exists in my library already. I can grab information about a book or author and then use it in some way. The API typically returns data in JSON format. For instance, if I call the function biblio.search_by_title("Beyond"), I get a JSON formatted return containing the following:

[
   {
      "title":"_Beyond Band of Brothers: The War Memoirs of Major Dick Winters_",
      "link":"[[Beyond Band of Brothers (334)]]",
      "type":"book",
      "authors":[
         {
            "author":"Winters, Richard",
            "authorFirstLast":"Richard Winters",
            "authorLink":"[[Winters, Richard]]",
            "gender":"None"
         }
      ],
      "source":"",
      "date":""
   },
   {
      "title":"_Beyond Apollo_",
      "link":"[[Beyond Apollo (58)]]",
      "type":"book",
      "authors":[
         {
            "author":"Malzberg, Barry N",
            "authorFirstLast":"Barry N Malzberg",
            "authorLink":"[[Malzberg, Barry N]]",
            "gender":"m",
            "alternateNames":[
               {
                  "name":"Barry, Mike",
                  "nameLink":"[[Barry, Mike]]"
               }
            ]
         }
      ],
      "source":"",
      "date":""
   },
   {
      "title":"_Beyond the Blue Event Horizon_",
      "link":"[[Beyond the Blue Event Horizon (259)]]",
      "type":"book",
      "authors":[
         {
            "author":"Pohl, Frederik",
            "authorFirstLast":"Frederik Pohl",
            "authorLink":"[[Pohl, Frederik]]",
            "gender":"None"
         }
      ],
      "source":"",
      "date":""
   }
]

The results so far

I’ve linked all of this together using my master reading list note. This note contains a list of everything I have read since 1996 and serves as a kind of index to my reading:

A sample from my reading list index note.

A big part of the way Obsidian works is that it can show you the relationships between your notes. While I am still working on importing all of the notes I have in my Kindle, I can already see a a network of relationships when I view the graph of my Obsidian vault:

A graph of the relationships between all of my notes.

Most of my notes are book and reading-related at this point. That big dot in the center is the master reading list illustrated above. If I highlight it, this is what I see:

Sample of a highlighted node on the graph.

From there, you can see other nodes and relationships that have started to form. For instance, if I hover over one of the Alan Lightman books I finished yesterday, In Praise of Wasting Time, you can see a little network of links coming off that book:

Some of those links point to annotation files. Another points back to the note for Alan Lightman. And a few of the annotation links point to seemingly unrelated notes.

Here is another example. One of the big nodes is for John W. Campbell, editor of Astounding Science Fiction in the late 1930s through his head in the early 1970s. I read many of those old issue when I was taking my Vacation in the Golden Age of Science Fiction. So Campbell shows up a lot on my master reading list:

Highlighting an author node on the graph.

You can see that Campbell is linked to all of the issues of Astounding that I have read. I have started to bring my notes in for those issues. If we look at the July 1939 issue, for instance, you can see this is related to all of the stories and articles and authors in that issue:

Currently, the notes for each story are part of the story note, but I plan on breaking those out into their own Zettelkasten-style notes as I’ve done for my other notes and annotations.

Conclusions

Keep in mind, that this is all being done with plain text files, something that I like because the format is compatible virtually anywhere. This could be done as easily on a Windows machine as a Mac. It could be done easily on a Linux machine. The openness and longevity of plain text (which has been around for fifty years now) is a big part of what I like about this system.

The linking that Obsidian provides from within its application makes all of this useful. But once established, those links are just as useful outside Obsidian with a little coding–as I’ve done with my API for books and authors. And this API is extensible. This week, I plan to add capability for the API to return any annotations when returning a “book” object. So in addition to what is returned by the JSON format illustrated above, that will soon contain a node for annotations related to that book.

Mostly, I am satisfied that I now have a simple way of keeping my reading notes in a useful form. These are easily searchable, they are easily linked. I can continue to capture highlights and brief notes as a I read. The import function allows a nice step to expand on my annotations as I review them after they’ve been pulled into Obsidian.

It did take me some time to get the infrastructure in place, but now that it is there, I am able to focus on reading, notes, and let the system organize them for me.

Automating My Daily Notes with Obsidian

ETA: 4/1/2021: I have now posted my code for my daily notes automation script on GitHub. See this post for more details.

For nearly a week now my Daily Notes file in Obsidian is generated automatically each night after midnight. I wrote a Python script that creates the note, and having the script running as a Launchd job–which these days is recommended on Mac OS over cron jobs. So far, this has been working out beautifully for me. Of course, this is in large part because Obsidian itself works so well for me.

I use my daily notes in much the same way as I used daily notes in my Bullet Journal days. Indeed, I store my daily notes in my Obsidian vault in a folder called “Bullet Journal”. In addition to having reference information right up front, I use the Today’s Notes section of my notes for jotting various notes down throughout the day, that serve as a kind of log of activity, ideas, and things to do.

The basis of the design of my Daily Note drew inspiration from this post on the Obsidian forums.

Here is a look at my annotated Daily Notes file for today, Monday, February 8.

An annotated version of a daily notes files automatically generated by my scripts.

1. Note title

My Python script generates a note title for the note in yyyy.mm.dd.www format. I used this format back when my Bullet Journal was on paper and so I’ve kept it going here.

2. Navigation/weather bar

My script generates a navigation bar to the previous day’s notes, and to tomorrow’s note (which doesn’t exist yet, of course). It also make a call to a weather app to pull in a very short version of the weather for the day for my location.

3. Agenda

Next, the script generates the “Agenda” section of my note. It does this by calling a command-line program called “icalBuddy” (for Mac OS) which allows you to grab information from your calendar. The published version of icalBuddy had an issue with accessing calendars from a cron job or launchd, but I found post where a person was provided with an updated version of the binary that fixes the problem. The only place I’ve been able to find that binary is in that post, but it worked perfectly once I replaced the original binary.

Each night, a call to icalBuddy is made and the resulting events on that day’s calendar are returned. I then render those events in the Agenda section of my Daily Note.

4. To-Do

This section lists all of the tasks I have that have not yet been checked off. This was a little tricky. The script searches for all tasks in my notes and splits them into 2 groups, completed and incomplete. It then hashes the tasks and removes any “incomplete” tasks that appear in the “completed” task list. This prevents seeing uncompleted tasks that were completed on a subsequent day. The result is that I only see tasks that I haven’t completed.

4a. Illustrates what happens when tasks are carried over day-to-day without being completed. My script appends a note indicating how long the task has been open. For instance, that “Add functionality for notes timeline” task has been open for 2 days (since Saturday).

Throughout the day, I’ll check off tasks here and add new ones and those get incorporated into the next day’s runs.

5. Reading

I try to read at least one feature article from the various magazines I subscribe to every day. This is where I record what I read. I’ve actually developed a fairly elaborate reading and book notes system in Obsidian that I will write about in a future post. What I do here is jot down the article title, author, and source in a specific format (e.g., “Article Title” by Some Authors | Source: Magazine (Month Year)) and a separate script adds this information to my Library notes.

6. Today’s notes

This is where my bullet journal-style daily notes go. These can be virtually anything that I think is noteworthy. As you can see, when I opened the paper this morning, I saw that George Shultz died, and noted that here.

I’ll put to-do items here. If I am working on a something, I’ll make notes about what i am working on. Ideas get logged here. Funny remarks I hear throughout the day might find their way here. It can be quite a mish-mash of stuff, but it ends up being a useful reference.


I suspect that some people will ask if I can share my scripts. I am happy to do that, but not quiet yet. I tend to write scripts the way I write stories, the first draft is quick and dirty and functional. The second draft is cleaned up and organized. Once I get the “second draft” written, I’ll post what I’ve written on GitHub and anyone who wants can make use of them.

Some of this functionality is built into existing Obsidian plug-ins, but it is either not automated, or not quite the way I wanted it, so the scripts that I have written are tailored to my specific needs and behaviors. It’s nice to wake up in the morning, open Obsidian, and see my daily notes file waiting for me.

Notes with Obsidian: My Initial Impressions

Now that I have been using Obsidian fairly exhaustively for the last 5 days, I think I’ve got enough experience with it to share my initial impressions.

I mentioned earlier that Obsidian was the notetaking tool I’ve been looking for all my life. That begs the question: what is it that I’ve been looking for in notetaking software to begin with? Here are a few of my requirements. (Your mileage may vary.)

My requirements

Notes should be plain text

There are two kinds of notes I deal with:

  1. Ephermeral notes, or those notes that I am jotting something quickly for later use, but that can ultimately get thrown away because they will be transformed into something else. Examples of this might include a person’s name, or phone number, or a rough set of notes from a phone conversation.
  2. Permanent notes: the kind that I’ll continue to use, update, and search into the forseeable future. Examples of these include how-to notes, daily notes, notes and clippings from reading. Drafts of blog posts, lists I maintain.

When I think of notes, I think of the need for versatility. I keep a Field Notes notebook in my pocket at all times to capture many of the type 1 notes above. I’ve found over the years that is much quicker than pulling out my phone, unlocking it, finding a notes app, and tapping out what I want to capture. To that end, the most versatile file format for digital notes, for me, anyway, is plain text. Plain text has many advantages:

  • I don’t have to worry about the file format. Plain text files created in the 1980s can still be read today with any old text editor.
  • Plain text files are easy vehicles for automation. Automation is a big reason why I’d want to keep notes in digital form in the first place.
  • Plain text files are cross-platform compatible.
  • Plain text files are easy to search and are the perfect targets for advances searches with regular expressions.
  • With Markdown, even plain text files can be rendered with formatting.

Notes should be rendered as Markdown

When I have looked to convert all my notes (and even my writing) to plain text in the past, one thing that has help me back is the lack of a good tool for rendering the Markdown in the files. It never looked like how I wanted it. Most text editors are really for writing code, and they bias their features in that direction.

This is one place where Obsidian excels. It has a great UI, one that is designed for notetaking (instead of code) and one that renders Markdown well. It is also flexible enough to allow custom themes so that really, you can make your plain text notes look pretty much however you want.

Notes should be easily linkable

A few years back, I tried to put together a system that allowed me to create links between my notes. I understood intuitively how useful this would be. But it seemed like a difficult problem, and really, I didn’t want to spend my time writing code, I wanted to work with my notes.

Obsidian’s note-linking capability–one of its central functions–eliminates this problem, and removes all the roadblocks to linking notes. Not only have I jumped on this feature, I have even started down the Zettelkasten rabbit hole (something I will discuss in a future post). Being able to easily link notes, follow those links, and see the relationships they form is a huge plus in favor of Obsidian.

Notes should be kept locally

I like that underneath it all, these are just plain text files sitting on my local file system. Of course, I can store them in iCloud or Dropbox or some other cloud service if I want to, but the actual files are local. This just helps with automation.

My initial impressions of Obsidian

Where to begin?

  • I just love the UI. It is clean, and easy to use. It makes me want to live in the app when I am working on the computer.
  • The linking is fantastic.
  • I also love that is has a folding feature built in which allows you to easily show and hide sections of a document.
  • I love that moving notes around (which I do often for organizational purposes) keeps the links up-to-date. I don’t have to think about it, it just work.
  • I love not having to save the notes. They save as I type. Important since more and more tools work this way and I’ve gotten use to not having to manually save documents.
  • Great searching capabilities! I can even search regular expressions, which comes in hand.
  • The Daily Notes feature is awesome and I’ve started using it to create a kind of digital bullet journal.

I’ve liked what I’ve seen enough that I became an Obsidian Supporter. I find the forums and documentation both useful.

Some use cases to illustrate and illuminate

Let me illustrate some of what I like about Obsidian with some real use cases that I’ve got just in my first five days using the tool.

Reading notes

I take notes when I read. I mark up books (real and digital). Ultimately, I like to get those notes together in some useful fashion. (This is the one area in which I have really started to explore Zettelkasten, and I’ll have more to say about it in the future.) Here is one simple example of what I have started to do in Obsidian.

I try to read one feature article from the various magazines I get every day. When I finish an article, I create a note for the article with information about the author(s), source, date, a link to the online version, if available. I also tag the note.

In order to build a kind of searchable index, I have started to create a note for each author. On the article note, I’ll link to the author note, like this, for example:

An Obsidian note for an article in Smithsonian by Glenn Adamson

Note that I have linked to the the author on this note. So when I go and look at the note I have for that author, I see this:

Obsidian note for author with backlinks section listing all of the backlinks

The “Backlinks”section on the note is created automatically through some automation I’ve written. Again, I’ll talk about how I automated this in a future post. I know that I can just go to the backlinks section on the sidebar to see backlinks, but I like seeing the actual references in the note itself.

Presenting with notes

Block folding is incredibly useful. That is, being able to close some blocks and keep others open while looking at a note. Here is an example of this post (yes, written in Obsidian) so far with most of the blocks closed.

Obsidian note with folded headers

I find this allows me to use my notes as presentations in meeting. I’ll jot down the topics I want to discuss in the meeting. List out the points under those topics, and then, before the meeting starts, close all of the headings, so that I can walk through them one by one, sharing my notes on the screen. It allows us to focus only on the open block at hand, while still having a context for the rest of the discussion. No need for slides!

Searching for daily notes

One test I have for any good editor is how well it can search. In addition to the great capabilities that Obsidian has for searching already, it also passes my test for being able to do regular expression searches. Here is a regular express search I have ato surface all of my Daily Notes:

A regular expression search for my daily notes: /^\d{4}\.\d{2}/

I have actually gotten quite a bit of mileage already out of the daily notes features, especially after I added some automation to make it even better.

How I’ve got my Obsidian configured

I’ve already had people asking me thing like what themes I am using, so here is a summary of how I have Obsidian configured for my Macs:

![[Pasted image 20210131113120.png]]

Obsidian configured with my preferences.
My Obsidian configuration

Theme

I’m using Pisum (from the community themes). Incidentally, I did spend time loading every community theme to find one that fit well. I kept a note open and moved the ones I liked to the top of the list as I discovered ones I liked better.

Plug-ins

I’m currently using the following plug-ins:

  • Calendar
  • Daily Notes
  • Zettelkasten prefixer

Other settings I’m using

  • Fold Heading
  • Tab size: 8

Questions I’m still pondering

There are some things I am still trying to figure out, and I’ve been searching the forums to see how others approach these issues:

  1. What should be in my vault. When I started, I created my vault as a Notes folder within my Documents folder. From there, a sub-folder structure began to evolve. But I’ve found that I want to link to documents that are not in the vault, so I am beginning to wonder if really, the vault should be my entire documents folder. I’m looking for advice on the scope of a vault in Obsidian.
  2. Zettelkasten. This seems like overkill for a lot of the notes I have. Put it seems like the perfect solution for how I have envisioned organizing notes from my reading. I used to have a note for a book. All my comments and highlights would be in that note. Now, I have a note for a book, but highlights each get their own note with (a) a link back to the book in question, and (b) a reason for the link. I’ve even started linking some of these extracts to other notes, formiing the web of relationships that Zettelkasten is all about.
  3. Best format for my Daily Notes. I’ve actually done some automation here and getting happier with what I’ve got. But I’m still figuring it out. Here is an example of my Daily Note for today:
An example of a daily note
An example of a Daily Note

Automation preview

So far, I have automated 3 things using Obsidian notes and the plain-text framework:

  1. Collect “to-read” references into a single note. This collects tags marked “to-read” throughout my notes into a single “To Read” note so that I have a single list to look at. My script will ignore items tagged to read if they have already been checked off.
  2. Update People index. This is the script that scrapes notes for backlinks to specific people notes and adds the “Backlinks” section to that person’s notes.
  3. Daily Note generation. This genarates the daily note for the day (just after midnight). I pulls in the agenda from my iCloud calendars. It pulls the to-dos from incomplete to-dos found in other notes, and from any incomplete items in my Apple Reminders. It then formats the other sections and creates the note.

I’ll write about these automations in more detail in a future post.


So there you have it, my initial impressions of Obisian. I’m looking forward to exploring it more. And I’m eager for that mobile app that’s “coming soon.”

Automation and the Power of Process Improvement

Three recent experiences remind me that automation for the sake of automation doesn’t really do much. But if automation can be used to improve processes, eliminate repetition, and redundancy, then it is well worth investing the time to improve the automation. It is a personal pet peeve of mine whenever I have to supply the same piece of information more than one in a given transaction, especially when that data is available somewhere else. Here are two small failures, and one small success to illustrate where we stand with automation and process improvement in day-to-day tasks.

“Would you like to apply for a Target Red Card?”

The complex we live in borders a Target and Safeway shopping center. This has been very convenient. We can walk to the store. It means we probably go more frequently than we need to. And because Target has pretty much everything, we go there quite a bit. Eventually, we decided to get one of their Red Card credit cards because we save 5% on every purchase. Once we had the card, I set it up to pay the bill in full each month. There is no point in saving 5% on purchases if you are paying 19% interest. So we get a nice benefit on every purchase we make from Target. So far, so good.

Recently, however, it seems that Target is really pushing the use of the Red Card to the exclusion of all logic. For instance, on several occasions, I’d put my items on the conveyor, slide my Red Card and wait for the total.

“Would you like to apply for a Target Red Card?” the cashier asked.

I blinked. “Well,” I said, holding up my Target Red Card, “I just paid with mine. Do I need another?”

The cashier laughed and we each went about our day.

But it happened again on the next visit with a different cashier. And then again. And again.

Finally, I said to the cashier, “You guys have been asking this quite a bit. I pay with my Red Card every time I come here and you always ask me if I want to apply for a Red Card. Isn’t there something on your screen that tells you that I have paid with a Red Card?”

The cashier said, “We are told to ask everyone, even if you already have a Red Card?”

“What sense does that make? If I say, ‘yes’ and apply for a second Red Card what would happen?”

The cashier just shrugged.

I don’t mind being asked this once or twice, but every single time I come to Target, and when I have already swiped my Red Card? That seems like a major breakdown, not just in a logical process, but in customer service.

“Can you fill out these forms?”

I took the Little Man to the doctor the other day. There is a nice touch-screen system to check in, and pay your co-pay, if one is due. It’s a nice piece of automation. But it failed this time. The system told me to see the desk. So I saw the receptionist and he told me that since it was the Little Man’s first visit this year, I had to fill out some forms. He handed me three forms.

The forms were all standard information. Parent names, addresses, phone numbers, insurance company information. I filled them out at a slow burn because I knew what would happen. I’d turn the papers back to the receptionist and he would key in my responses to the central system. So not only was I entering the information, but he was entering the information. He had to parse my handwriting, increasing the chance that some of the data would be entered incorrectly.

Read more