Welcome to my blog series, “Practically Paperless with Obsidian.” For an overview of this series, please see Episode 0: Series Overview.
A couple of months ago, in Episode 15, I wrote about how I use daily notes as an index to my life. I was impressed by a post I’d read on someone who kept all of his daily notes in a single file, and I aimed to try to reproduce that in Obsidian. Since writing that post, I’ve tweaked things somewhat–enough that I felt it was worth reporting back on. Since this series is an ongoing experiment in going practically paperless, it isn’t unusual for me to change the way I do things when I find ways and methods that work better.
Two competing requirements
In the case of daily notes, I found myself pinned in by two competing requirements:
- I want the ability to see more than just “today’s” note. I like to be able to easily scroll through past daily notes. This is a quick and easy way to review things without having to search through a bunch of files, or open a bushel of notes windows to reference something I am looking for. This is what attracted me to the single daily notes file in the first place.
- I want to use the native daily notes functionality that comes with Obsidian. This functionality, however, is based on a single-note-per-day, and I’ve found it just doesn’t work nearly as well when all daily notes are in a single file. The main problem was that if I linked to another note from my daily note file, the backlink referred to the daily note file, but not the date in the file under which the link was listed.
So: I want to use a single file for all of my daily notes, but I want the daily notes functionality that is based on one note per day. What to do?
First solution
In order to address this conundrum, I decided to go back to the single-note-per-day. That immediately gained me the native daily note functionality in Obsidian. In a single-note-per-day model, if I link to another note, the backlink shows the date of because it is the name of the daily note file itself. For instance, I list out the blog posts I write on a given day within my daily notes file. In the case of this post, the backlink shows what day I wrote the post:
This is much more useful than when I used a single note for all my daily notes. In that situation, the backlink would show up as follows:
and “Daily” doesn’t tell me the date. That is a big drawback of the single-file model. As I pointed out in Episode 15, one thing that would help would be if the section path that the link was under was displayed as part of the backlink — that at least would pick up the date. I even submitted a feature request for this. But it doesn’t exist now and I have to work with what I’ve got.
One thing I did to attempt to get the best of both worlds was to setup my workspace so that the previous day’s note always appeared in a window above the current day’s note. That way I could easily scroll through the previous day’s note if I needed to reference something there. I pin the current day’s note, but leave the previous day’s note unpinned so I can quickly switch to another day using the calendar.
I played around with this model and it worked okay, but it just wasn’t as useful as being able to scroll through all of my daily notes in a single file. It did, however, get me the benefit of the native daily note functionality in Obsidian, so that was a partial win.
Second solution
After giving it some thought, I came up with another solution that works even better: a single daily notes file made up of transcluded links (embeds) to individual daily notes. This is achieved as follows:
- Daily notes are created normally, one-file-per-day, using native daily note functionality in Obsidian.
- At the end of each day, I add a transcluded link to the current day’s daily note to the top of a note file called “Daily”
The “Daily” note file looks like this in source mode:
When viewed in Live Preview mode, however (which is my default), this same file looks as follows:
This allows me to scroll through all of my individual daily notes as if they are in a single file, even though the daily notes themselves are each in a separate file. It allows me to work in a workspace where I keep the Daily file pinned at all times, even while having today’s daily note open in a separate window, like this:
In this setup, I can easily scroll through all of my daily notes going back to when I started intending to keep them in a single file (December 28, 2021). What’s more, as I add new items to my current daily note (the window on the lower-right), they appear instantly in the file on the left.
One change I made in the “Daily” file from my previous attempt at a single file is that I reverse sort the entries so that the most recent day appears at the top of the file. I found that I was constantly scrolling to the bottom of the single file to add new notes, and while I like the strict chronology, it was more practical to reverse sort it. Several readers pointed out to me that this is what they did, and it made a lot of sense to me.
While the embedded text within the file is not searchable in source mode or Live Preview mode, it is searchable in Read mode. And I can leave the file in read mode because I only have to made an update to it once a day, to add the current embedded daily note file. That means, if I hit Cmd-F to search in the file, results show for all of the embedded files. For instance, if I search for the term “- read” here is what the results look like:
Note that the search results span multiple days in this view, meaning they span multiple embedded files. This is what I was hoping for when I began looking into this solution.
The best of both worlds?
This solution — using individual daily note files and a separate Daily file with embedded notes via transcluded links — is better and gets me closer to the two competing requirements I’ve been aiming for. I can use native daily note functionality; I can see references to daily notes in backlinks, and I can scroll through the daily notes as if they are all in a single file. But it is not perfect. There is still at least one challenge to overcome, and a relatively easy one at that.
Challenge: Automating the Daily file
As I said, currently, at the end of each day, I add a new transcluded link to the current daily note at the top of my Daily note file. If I forget to do this, I won’t see the previous day’s note. This isn’t all that cumbersome, but it is something that I want to automate.
A simple shell or Python script can take care of that. The script will add the trancluded link to the new daily note at the beginning of each day so that I can see it there at the start of the day, instead of just at the end. This really is a simple script and when I get it written, I’ll post a link to it on GitHub1.
Daily summary and Dataview
There is one other adjustment I’ve been playing around with in my daily notes. Now that each day is in a separate file, my daily notes template includes YAML frontmatter for capturing some information that I care about. Currently, that template looks as follows:
---
locations: ["Arlington, VA"]
sleep: 1
summary: Good writing day, busy work day. Upgraded Obsidian. Good response to latest PP post
---Here is what this data is for:
- Locations: a list of places that I was at on a given day. Not every place I go, but rather where I was based on that day. On days I travel, this list contains multiple entries, like [“Arlington, VA”, “Wisp Resort, McHenry, MD”]
- Sleep: a rating of the quality of my previous nights sleep on a scale of -2 to 2, where 0 is an average night, -2 is a absolutely terrible sleep, and 2 is a dreamless, perfect sleep. For those curious, I learned of this method listening to Jim Collins describe this method on the Tim Ferris Show podcast.
- Summary: A short, one line summary of my day, written at the end of the day.
I use this in a Dataview table that provides a kind of index to my daily notes. At a glance I can see where I was on a given day. I can also use the summary to help remember what happened on that day. I can then drill into the specific note for more details. I have a note called “Daily Notes Index” and within that note, use the following dataview query:
```dataview
TABLE summary, locations FROM "Daily Notes"
SORT file.name DESC```Here is what this Daily Notes Index looks like for March 2020 so far:
This hasn’t replaced my Daily file containing embedded daily notes. Instead, I find this file more useful as a high-level review of my days, something I can scan through at the end of the month or end of the year, or a place I can go to quickly see where I was or what I did on a certain day without flipping through the details of the individual daily notes file.
Putting it all together
To sum up the current state of my daily notes:
- I have gone back to using a single-note-per-day to get the most benefit from Obsidian’s native daily note functionality. I always have “today’s” note open in a pinned window.
- I maintain a “Daily” note that emulates the all-daily-notes-in-one-file that I started with back in Episode 15. This allows me to easily scroll through all of my daily notes and search them without having to open multiple files. Given the way that I work, this is more efficient for me.
- I use the YAML frontmatter in my daily notes to populate a dataview table in a Daily Notes Index file that I use periodically for skimming my days
This is as close as I’ve been able to get to the best of both worlds: each day in its own note, and all notes in a single file. So far, it seems to be working for me, but as always, I am open to ideas an suggestions for improvment.
In next week’s post, I’m going to change pace a bit and talk about my personal views on information security with respect to my notes in Obsidian. I’ve been asked questions about security quite a bit. This was also try when I was writing about using Evernote a decade ago. The post will summarize how I think about security, some tools I use to protect my data, and what I do and won’t worry about. See you back here next week!
Prev: Episode 21: Tags in Theory and Tags in Practice (And Never the Twain Shall Meet?)
Next: Episode 23: Protecting My Data in Obsidian
Written on March 9 and 11, 2022.
Did you enjoy this post?
If so, consider subscribing to the blog using the form below or clicking on the button below to follow the blog. And consider telling a friend about it. Already a reader or subscriber to the blog? Thanks for reading!
- I suppose I could try to make this a plug-in but (a) this is too niche a use case, and (b) I’d have to deal with all of the overhead involved in a plug-in. A Python script on a scheduled job is much simpler for this. ↩
RSS - Posts
Hi Jamie,
You must realize that this is getting very complicated for us “ordinary mortals”.
Image if you are new to Obsidian and want to use it for journaling! After reading your blog, you will get second thoughts. That’s why I looked for a solution without coding and I must say that my solution nearly gives the same result but you only have to invest in an application. I don’t have shares in it; I am just a testpilot.
Just want to use your blog to share my solution.
I use the Agenda application as a front-end where I work in from day-to-day with a Daily note template as a base. I keep the Daily notes in a Project (Journal) with all the other Daily notes of this year in reverse (current date first). In Agenda it’s possible to fold notes, so you then only see the header (which is a date, in my case).
I now have only this year’s Daily notes in Agenda (besides information of other notes from other projects I do). On the left of the work area there is a list of all my projects and on the right there is a list of all entries from my Calendar and Reminders (both Apple applications that come free with the hardware). Also able to edit those entries in Agenda.
So that gives a complete overview of your business.
Now when a note or project is finished, you can archive it within Agenda, but I export a note or project (also journaling notes) to a Markdown- or textfile or PDF file in the file system. If you wish, in the Obsidian directory.
Agenda works on all Apple system platforms and syncs through clouds.
So this gives a lot of flexibility, no coding and a great overview.
Just wanted to show my solution to the other “ordinary mortals”!
Still look forward to other solutions.
Thanks.
Jaap, with this post in particular I wondered if it was getting too much in the weeds. But I thought it was only fair to show how my process has evolved. And none of this requires any coding (yet). The reference to automation was a “nice-to-have” but by no means required. Thanks for sharing your process, though, because I think it is really helpful to see this from different perspectives.
Personally I am thoroughly enjoying reading about the evolution of Jamie’s system as I am going through much the same thing myself and it feels like I have a weekly companion on my journey. I never could bring myself to use the ‘all daily notes in one file’ approach though and I feel like now that Jamie has gone back to one note per day, everything is right with the world 🙂
@Jaap I haven’t spotted anything in what Jamie has written here which would require any coding (other than the optional basic automation he mentioned, where he has also offered to share the code). Having said that, I would expect the blog of a fellow software developer to have at least some technical elements.
This had me laughing out loud!
In Jaap’s defense, things get messy when I experiment, and that can lead to confusion. With this follow-up post in particular I worried that it wasn’t as clearly written as some of my other posts. I almost cut the entire “First solution” section at the last minute thinking that by focusing on the end result things would be more clear. But that old admonition from my teachers–“show your work”–kept coming back to me. And I thought that there was value in seeing how I got from A-to-B-to-C.
In episode 23-28 I think things will be simpler and more focused on practical, everyday use cases. That doesn’t mean there won’t be technical ones in the future–some may even have code associated with them. But I do get where Jaap is coming from.
I understand that sometimes you have to go *through rough patches”, but I didn’t think that Jamie only wrote this blog for “fellow software developers”. A lot of people are journaling and want to better themselves in it. Anyway, he promised to “improve”!
Weeds?…. I would call it woods! ;-))
I’d love to get the transcluded single file of all your daily notes when you make the code available! It would be really nice to have it auto create links to every daily note so you don’t have to edit the file every day. Editable trancluded sections would be even nicer and prevent needing to enter read mode for searching, and there is already a feature request for that on which I mentioned this article: https://forum.obsidian.md/t/edit-transcluded-embedded-notes-blocks-in-place-likely-requires-wyswyg-first/15339/18?u=ryanwwest.
Hello, interesting post. Thank you very much for your work here! Many good ideas. I think / I want the same function that the app logseq has. There you automatically have a journal with all notes in a chronological list. Each note can be a single note. Then you have both. I don’t like all my thoughts in one big and long note. But I like a journal with a scroll function and the possibility to search through everything. Loqseq also has the advantage that it lists all the notes in the journal and not just the Daily Notes. There is an Obsidian Plug In “Daily Notes Viewer”. This offers a partial solution. But only for Daily Notes. Searching the displayed notes is not possible either and you can set how many notes should be displayed in the viewer. So not all notes but e.g. the last 60 days.
Thank you for the “Read mode” search-ability tip! I was getting a bit frustrated with transcluding and dataview because I couldn’t find the text these blocks were displaying when searching the note. Turns out that was just an editing-view shortcoming/misunderstanding.
I’m curious what your higher-level thoughts are on dataview in combination with your desire to have a set of notes that are text-readable in the future via any ol’ text editor? The moment I think of getting too wild with dataview, I remember that I will not see what dataview outputs in the backing text file. Do you think of it as just “bonus content” for your notes rather than something to deeply depend on? Or do you throw caution to the wind when dataview is providing you value right now?
Barry, your question is timeline, since I address it in the most recent Episode, which went live today. The short version is that I have taken to hedging. One downside to surfacing notes through a dataview table is that the table itself does not create actual links between notes. What I have been doing is including those links in the notes outside the meta-data. In my writing templates, for instance, there is a footer (you can see in today’s post) that links a draft to a project, or a submissions to a project, or contract to a project, etc. The “footer” contains an actual Obsidian link, added as part of the template. Thus, even though I can use the meta-data in the note for dataviews, I can also still see the linking happening in the graph view, or though backlinks. The nice part is that this is pretty much automated through the templates I use.
Thank you for sharing those details and directing me to your latest post. It does seem like a balancing act and your templating+footer does that well!
Here is a short shell script that can be used to automate creation of the embedded link in the daily summary. It mimics the date format Jamie uses. It uses the MacOS version of sed and inserts the link in the third line of the file, assuming Year and Month headers are used (this won’t overwrite them.)
# Creates a new embedded link to today's daily note in daily summary notetoday=$(date -j +'%Y-%m-%d %a')
sed -i '' "3s/^/![[$today]]\n/" ~/Mainframe/daily.md
This is great! And bonus points for using sed!
Here is a one liner that can be used to create the initial daily summary page, assuming you already have a folder of daily notes.This works in both bash and zsh on MacOS (should work in any bash shell.) Edit the file manually after creation to add year and month headers if desired. Replace “~/Mainframe/test.md” below with the location and name of your desired file. Note this this will wipe and replace that file with each run so its really only good for populating the list initially.
output=~/Mainframe/test.md;:>$output;for f in *.md;do file="$f";name="${file%%.*}";echo -e '!'"[[$name]]\n$(cat $output)">$output;done
Thanks for providing such a valuable resource. Been following since your Evernote days. There are only a few sites I return to over and over again that continue to resonate and add clarity to what I am striving to accomplish and this one stands above them all.
Thank you for writing this series, it’s an amazing resource to help me dig deeper into all that Obsidian has to offer!
I love the use of transcluded links to collect all the daily notes into one big list but I do have a question:
How do you manage to display the date/daily note title in your “Daily” entry? If I use transcluded links in my setup the same way you show in your source view, I just get the content itself. To see a date in the list, I’d either have to add a heading in the daily note or write out the date again in the “Daily” list. Is there a workaround for that or do I miss something?
Johanna, thank you for your kinds words on the series. To answer your question, the date that shows up in my Daily file (which is a just a series of transcluded links) comes from the title of each note. My individual daily notes use the title YYYY-MM-DD ddd, or “2022-05-04 Wed”, for example. When a note link is embedded in the daily file, both the title and content of the note is displayed. So the date you are seeing is just the title of the embedded note, which happens to be very convenient in this case.
Thanks for the quick answer! That’s what I couldn’t figure out – I embedded the notes the same way you did and the title didn’t show up.
Then I switched to another theme just to get a different look – and then suddenly the titles appeared. So apparently the theme was at fault! Sorry for taking your time!
Thanks so much for this entire series, it’s been so helpful!!
On the reverse sort on your daily file, I’m not seeing a sort option for data inside a note or file.
How are you accomplishing this, please?
I am looking for similar solution while I am migrating my daily notes from Logseq to Obsidian, and I found one plugin might help, not sure if you have checked it out https://github.com/Johnson0907/obsidian-daily-notes-viewer
So essentially you have your “source” data and two derivative daily notes files – one that is comprehensive, and an abbreviated index – right?