Going Paperless: Prototype: Automatically Send Kindle Notes and Highlights to Evernote

I promised this year that once a month, I’d post a more advanced “going paperless” automation post. Since this is my 100th Going Paperless post, I have a special one planned for today. It is more than just an automation post, it is a kind of call to action for those interested in extending what I have done here. I have tried to explain the process below methodically, and I apologize in advance for places where things get unclear. What follows is for advanced users, although I hope that with help, we can make it a solution that just about anyone can use.

---

If you’ve been following along, you know that I’ve often despaired that there is no good way to automatically send Kindle notes and highlights to Evernote. This seems like a huge gap just waiting to be filled. Michael Hyatt has good instruction for one method for getting Kindle notes into Evernote, but it requires manual effort, and what I am looking for is something that is automated, that just works as part of the normal routine without taking any special steps. In fact, what I want is something like this:

I read a book on my Kindle device, and make highlights along the way. Like this:

When I plug my Kindle into my computer to charge it, some process automatically updates Evernote with the notes and highlights I’ve taken since the last update. I get one note for each book, and the note looks something like this:

Well, actually, I’ve created some automation that does exactly that. The above screenshots were captured from that automation, and I will share how it works below. I did it because I didn’t want to wait any longer. But I did not have time to build a robust, end-to-end solution that will work smoothly and easily for everyone. In other words, I hacked together a solution, and I present here as a prototype of what is possible, with the hope that others will take what I’ve done, and vastly improve upon it.

What I have done, is:

• Written a Python script that can parse the My Clippings.txt file on a Kindle device.
• Add or append notes in Evernote via email from the results of the parsed My Clippings.txt file.
• Automate this through the use of a Keyboard Maestro macro on the Mac.

Before I get into the details, let me talk about the limitations:

Limitations of the prototype

• The prototype does not use the Evernote API. Instead, it uses Evernote’s email functionality to email notes to your Evernote account, either creating new notes, or appending to existing notes.
• One note is created per title on the Kindle, if said title has notes and highlight.
• The solution only works with Kindle devices. It does not work with Kindle Apps, like the iPad or iPhone app.
• It only works for notes taken on a Kindle device. If you take notes on, say, the Kindle App on your iPad, and then sync across devices, the My Clippings.txt file on the Kindle device is not updated with those notes. They are stored separately in the cloud.
• For automating the process, it requires a tool that allows you to detect when a USB device has been connected to your computer, and take an action when that happens. I use Keyboard Maestro for this on the Mac, but I don’t know what tool to use for this on Windows.

Once again, this is a prototype, a proof-of-concept that the automation can work. Having demonstrated that it can work, I’m hoping others can improve upon it.

What you need to get started

1. A Kindle device.
2. The kindle-to-evernote source code. I’ve made this code available on GitHub. Note once again that it is a prototype. It works pretty well for me but there are still bugs that need to be worked out. These bugs can be tracked as part of the GitHub repository.
3. If you are on a Mac or Linux system, you already have Python installed. If you are on a Windows machine, you will need to install Python.
4. I use several Python packages that weren’t among the default packages installed on Windows. These include the Six-1.5.2 package and the python-dateutils-2.2 package. These appear to be installed by default on Mac systems. (And I presume Linux as well.)
5. For automation on the Mac, I use Keyboard Maestro. This software allows me to trigger a macro when a USB device is connected to my Mac. The macro then runs the kindle2en python script. I don’t know what the equivalent software would be on Windows or Linux, but am certainly open to suggestions. This piece is only needed if you want the process to be fully automated.
6. A Gmail account for sending messages to Evernote.
7. An Evernote account.

How the solution works

When properly installed and configured, the solution is actually pretty simple. A configuration file stores information about where to look for the My Clippings.txt file, as well as login information for a Gmail account. I store the authentication credentials in the configuration file because that file should only ever be on your local machine.

The first time the kindle2en.py script is run, it creates a “semaphore” file in your home directory called .kindle2en_sem. This file is used to capture the last time the script was run. It stores the date and time of the last run. The first time the script is run, the date is set to January 1, 1990 so that it captures all of the notes and highlights on the Kindle.

When the kindle2en.py script is run, it reads the My Clippings.txt file on your Kindle device and looks for any notes or highlights added since the date contained in the semaphore file. It collects these notes and highlights into buckets, one bucket per title. Once it has processed the file, it sends one email message per title to Evernote. The email messages are formatted so that highlights appear as Evernote highlights in the note body. Notes appear as plain text. The email messages are sent to your Evernote account via Gmail. The Subject line of each message is the title of the work followed by the author, like this:

On Writing (King, Stephen)

The script uses the + at the end of the subject line to append the notes and highlights to and existing note of the same title. If no such note exists, it creates a new note.

You can run the script manually at the command line whenever your Kindle device is attached to your computer simply by typing:

python kindle2en.py

(assuming the script is installed in your path).

The command has several options. If you type

python kindle2en.py -h

you will see the available options, as follows:

Usage: kindle2en.py [options]

Options:

-f       specific location of configuration file other than home dir

-h       display help

-v       verbose output

-V       output version information and exit

The command can also be called from another process. On my Mac, I use a Keyboard Maestro macro to execute kindle2en.py whenever my Kindle device connects to my Mac’s USB port.

Part of the beauty of the solution is that once the note is in Evernote, you can add addition notes directly to the note in Evernote. And if you highlight more on your Kindle, those updates will be appended to your note.

Installing the solution

Here is how I have the solution setup on my iMac. Things will vary slightly from machine-to-machine, and platform-to-platform.

Setting up Keyboard Maestro

Here is what the macro I use in Keyboard Maestro looks like:

1. My Macro. This is the name of the macro I created, followed by the trigger for the macro. Most macros are triggered by a keyboard command, but in this case, you can see it is triggered by a USB device.

2. Trigger conditions. This trigger conditions for this macro are pretty simple. The macro is triggered when a USB device named “Kindle” is mounted on my machine. You will need to check the name of your Kindle device and update the Macro accordingly. Case matters, so make sure that the name you put here matches your device name exactly. The trigger occurs when the Kindle “is attached” to my machine. This means when I plug it in to the USB cable, the macro will run.

3. Optional. This part is entirely optional. I included in my macro so that I have a visual indication that the script is running. Whenever my Kindle is attached, Growl will send a message to the screen telling me that my Kindle is attached. This way I know the script is working.

4. Verification. I could just run the action, but I added an additional verification step here to make sure the device is still connected when I run the action. If it is still connected, then it executes my Python script.

5. Execution. This is where I tell the macro to run a shell script. I have to include the full path of the scripts that I want to run. And since I wasn’t sure if the script runs with my environment or not, I used the -f switch for the kindle2en.py which tells the program where to look for my configuration file. Your paths may vary.

That’s it. That’s what tells the system to trigger the script. If you are running this on a Windows or Linux machine, you’ll need to find some software that detects when a USB device is connected and can execute a script on such an event.

Installing/configuring the kindle2en.py script

Installing the script is pretty straight-forward. I put the script in my /usr/local/bin folder. You want to make sure the script is executable so you’ll run the following command on it:

chmod +x kindle2en.py

Next, you will put the .kindle2en.cfg file in your home directory. This file then needs to be edited with some information that the script uses to find your Kindle notes and send them to evernote. The configuration file looks like this, out of the box:

# File Locations
CLIPPINGS_FILE=<path to your clippings.txt file>

# Mail Settings
GMAIL_USERNAME=<your gmail address>
GMAIL_PASS=<gmail password>
GMAIL_SERVER=smtp.gmail.com
EN_ADDRESS=<evernote email address>

Gmail is used to send the notes and highlights to Evernote via email. The password is stored in your settings file on your local machine. This is relatively safe if your machine is your machine and not a shared machine. One future improvement would be to use the Google API directly and OAuth authentication, but I was trying to keep things simple here.

If you use 2-factor authentication with Google, as I do, then you’ll need to create an Application Password for this app and use that password in your settings file, instead of your regular password.

The Clipping File location is the path to your Kindle device clipping file when the device is attached. This will vary based on the name of your Kindle device. For me, this line looks as follows:

CLIPPINGS_FILE=/Volumes/Kindle/documents/My\ Clippings.txt

Additional Python libraries

It is possible that you won’t have all of the required Python libraries installed by default on your machine, but they are easy to get. I know for a fact that when I installed this on my Windows machine, I had to add the following libraries to Python:

• Six-1.5.2
• python-dateutil-2.2

What libraries you have by default will depend on your installation.

Once you’ve got these things setup, the script should run each time you connect your Kindle device to your computer.

A call to action for future improvements

As I’ve said, I’ve been using this for two week now, and it works pretty well for me. I’ve run it manually from a Windows machine, and have automated it (via Keyboard Maestro). But there is a lot of room for improvement. Things are kludgey right now and can be made much more robust. The installation process can be automated and simplified. I’ve shared the source code on GitHub with the hope that others who have also been looking for this functionality will use what I’ve put there as a jumping off point.

Some suggestions for improvements in the short-term:

• Replace the email-to-Evernote functionality with Evernote API calls. Evernote provides a rich API for developers. Part of the reason I chose Python is because it is one of the languages for which Evernote provides an API. A much more seamless integration would be for the script to use the Evernote API to update notes instead of email. I used email because it was quick and dirty. It allowed me to “prove the concept.” The API would be much more robust, and give you much more control. It would also allow you to get rid of the Gmail component.
• Identify tools that allow automation on Windows and Linux. I’m sure there are ways of detecting when a USB device on Windows and Linux machines, but I didn’t investigate them because my ecosystem is primarily Mac, and I wanted to prove the concept. The concept works, so others, with more experience on Window and Linux might be able to suggest ways to automate things there.

ETA: Be sure to see Nathan’s comment below. He points out an iPhone app that does all of this and more–it gets all notes and highlights from Amazon regardless of device, and you can send them to Evernote or Dropbox. The app is called Snippefy, and it does 95% of what I was hoping for, and better than what I have outlined here. (Although, I still think what I have outlined is a pretty decent example of trying to come up with a solution.)

---

I’m sort of amazed that I’ve written 100 of these posts, and I’m looking forward to starting the next hundred. For this post in particular, I hope it will be a collaborative effort. Share thoughts and ideas in the comments, but also be sure to check out the code on GitHub and see what you can make it do.

---

If you have a suggestion for a future Going Paperless post, let know me. Send it to me at feedback [at] jamietoddrubin.com. As always, this post and all of my Going Paperless posts is also available on Pinterest.

Last week’s post: Append to Existing Notes in Evernote via Email.