Using Hypothes.is to quickly place notes into my digital notebook

It is not so much the option to set your notes to “private” that counts for me. It is the impossibility to hide them from Hypothes.is personnel/machines.

There are workable options for all of these criticisms. I would suggest that @Klaas, @JkNML, and others not dismiss the platform so quickly out of hand.

In my experience, the Hypothes.is team—working on a non-profit, education-based platform—are some of the most ethical designers and developers I’ve run across in a long time. They’re not doing some of the highly questionable things with people’s data that one sees many venture backed or for-profit companies doing.

Hypothes.is generally does make it easy to have one’s data and there are many methods in addition to their API. However, this does take a bit of technical work, particularly if you want it as a stream of data in real time, which is why I would advocate for that sort of functionality to be made into a plugin for projects like Obsidian. They could likely create the support in a few hours for the benefit of their thousands of users who are far less likely to do that work individually.

There are two broad “versions” of Hypothes.is (and possibly others in the wild).

One is the publicly run instance that offers their service for free (with some universities and educational institutions paying for additional levels of support and integration as a means of supporting their enterprise). Because the education market is their primary business, they take privacy and security of users data very seriously. (This is also primarily why they indicate in their terms of service that they can delete data that is abusive or outside of societal norms. They’re doing that to protect against abuses in school settings. I’ve not heard any cases of them having done so.)

All of one’s annotations can be marked as either private or public with the default being private. If they’re marked public, then they go into their “public” timeline, but in many years of experience doing primarily only public annotations, I suspect that very few people are reading, much less commenting on my annotations. (Part of this is because of the niche, academic nature of the platform.) They do also offer private groups—primarily for classrooms—so that students could annotate pages with their teachers and classmates in a private setting. Naturally this could be screenshot and shared, but I haven’t seen or heard of this sort of leakage in the wild. And of course the average user may never need or want a private group anyway.

And of course if this doesn’t ameliorate your level of concern, the second “version” of the project is an open source one so you have the option to download it and run it on your own server. This way, if you don’t like the company’s policy on data, privacy, or other options you can manage them for yourself.

All this said, they’ve put a huge amount of engineering and design work into making their platform work seamlessly on the majority of the web. This is something that any note-taking project should love to take advantage of and support. It’s unlikely that a project like Obsidian would be able to devote the time, effort, staff, or other resources to support this sort of functionality themselves.

Apologies if my piece seemed like an advertisement to some, I hope it shouldn’t. I definitely spent longer laying out the problem, but my goal was to bring everyone up to speed on my viewpoint of a very specific UI problem that I (and many others) face and the details of a very specific, flexible, and elegant solution, which many users are unlikely to have the level of experience with that I do.

The other “hidden” portion relating to Webmention is also incredibly useful in that it could also apply to the thousands of social media and other online platforms that users use to generate data, and which could be used to pipe that easily into a personal notebook for direct use. As an example, a bookmarking service like Pinboard is nice and has a useful tagging and search function, but is less usable to me because none of that data is usable the way it would be in a bi-directionally linked setting like Obsidian. I also may need to search multiple platforms (Pocket, Pinterest, Instapaper, Diigo, etc.) to remember which one I saved a particular web page to. Imagine if all of the online data and content you generated over decades were available in a personal private/public notebook?

9 Likes

Love this idea! I’ve starting borrowing your workflow for Hypothesis using IFTT to drop new notes into my Obsidian inbox.

It works surprisingly well and works great across platforms (since I use Mac/iPhone but have an Android e-reader tablet for heavy reading stuff).

It can also be used as a web clipper by highlighting stuff you want to save for later.

Thanks for the inspiration, @chrisaldrich.
Ray

2 Likes

Thanks for this suggestion. I’ve just created the IFTTT recipe and, while it’s not 100% what I’m looking for, it’s the closest thing I’ve found to getting me there. Much appreciated.

@Solari I also own an Onyx Boox Android eReader and would love to know how you use Hypothes.is with your e-reading device. I have been using Instapaper and have found it very underwhelming.

1 Like

Inspired by all this, and wanting to see if I could come up with something that didn’t use IFTTT, I searched Github to see what kind of Hypothesis export scripts were out there, and I’ve cobbled together a workflow (blogged here ). What follows was created in the context of working on a Mac; PC will be broadly similar.

Basically, if you have python installed on your machine:

Get your annotations

Install ‘Hypexport’ from GitHub - karlicoss/hypexport: Export/access your Hypothes.is data: annotations and profile info . Install it with

pip3 install --user git+https://github.com/karlicoss/hypexport

Then, create a new text file; call it secrets.py and put into it your Hypothesis username and your developer token (which is underneath your username when you have hypothesis open) like so:

username = "USERNAME"
token = "TOKEN"

Save that file.

Now, you can grab all of your annotations with:

python3 -m hypexport.export --secrets secrets.py > annotations.json

Turn json into markdown

Now we need to turn that json into markdown.

( Incidentally, if you want to turn your annotations into a csv, get jq and run something like this

jq -r '.annotations[] | [.text, .tags, .updated, .uri] | @csv' annotations.json > annotations.csv

)

So, here’s a json to markdown script: GitHub - PolBaladas/torsimany: 💡✏️️ ⬇️️ JSON to Markdown converter - Generate Markdown from format independent JSON . Pip install that:

pip install torsimany

but then find where it’s located on your machine (search for torsimany.py) and change this line

data = f.read().decode('ascii', 'ignore')

to just

data = f.read()

and then run

torsimany annotations.json

at the command prompt (in the folder where you downloaded your annotations to), and after a bit you’ll have a file called annotations.markdown. (Incidentally, you can modify the torsimany.py script with your own preferences for how the different elements should be rendered as markdown, if you don’t like the defaults).

Split into individual markdown files

Last thing – we want to split that up into separate markdown files, to drop into the obsidian vault. cpslit, split, awk, etc, all of those things will probably work; here’s some perl (Mac’s have perl installed by default). Copy it into a text file, save with .pl, and if you’re on a mac, change its permissions with

chmod +x split.pl

so you can run it. Here’s the file (sourced from stackoverflow):

#!/usr/bin/perl

undef $/;
$_ = <>;
$n = 0;

for $match (split(/(?=### Title)/)) {
      open(O, '>temp' . ++$n);
      print O $match;
      close(O);
}

then run

./split.pl annotations.markdown

and you’ll have a whoooole lotta files you can drop into your obsidian vault. Ta da!

Fix the lack of file extensions

Now, you’ll have to add the .md file extension, which can be done as a batch with this one liner on a mac:

for file in *; do mv "$file" "${file%}.md"; done

and there we go.

So, not perhaps as elegant as using IFTTT, but you can see every step in the process. I’ll eventually get around to making this all just a single script, but I’m learning as I go. No doubt there are far more elegant ways of making this all work.

Also, it is possible to install your own hypothesis server so that you’re not part of the larger Hypothesis system; I intend to figure that out eventually and will post it on my blog if ever I do. I am personally satisfied with how Hypothesis treats user data, but I understand the concerns of folks who might like more control.

7 Likes

thanks for sharing this! was hoping someone would do it :slight_smile:

I’ve written some code that formats the title, adds tags (when they exist) and joins highlights and annotations onto a single markdown file if that’s of interest.

github repo here.

will add some stuff so it can be updated without duplicating previously added notes.

4 Likes

Nice!

Hi all. Like everyone here I’m an obsidian user but I’ve also been using Remnote for my Spaced Repetition workflows and have decided to use hypothes.is as the means of getting content into remnote for further refinement. There’s a way to do this via IFTTT and Webhooks, my problem in the case of hypothes.is is that it seems to only work for public annotations. Would anyone know how to make this workflow work on private ones? I just need to be able to create an RSS for the latter and it should work with my Remnote webhook… Thanks for any ideas!

@zendude - I’ve not found anything simple. They all include scripting and server-side setups. I believe they all include warnings that your security is exposed when using these methods.

If you’re interested, here are two I’ve bookmarked in case I get the itch to try them out.
This one is the method Hypothes.is points to from their site.
…and see this fairly detailed setup from the forums.

If you have any success, I’d be happy to hear of it. Good luck!

This reminded to type up some recent Discord messages about hypothes.is markdown export:

Facet allows exporting annotations to HTML - You search for your username, select the group (if it’s private you’d need an API token generated from https://hypothes.is/account/developer) and that lists all your annotations which you can then export as HTML. With the markdownload browser plugin, you can right-click copy tab as markdown.

Also, quick plug for gooseberry for command-line users - bulk tagging + export with customizable folder structure and markdown templates.

thanks all. I tried Facet’s python which I think was suppose to generate an XML file which I coul’dve then turned into an RSS and then via Webhook to Remnote. unfortunately it didn’t work. as for the rest, it doesn’t seem to do it in a (for a lack of better word) “dynamic” kind of way ala Readwise… where when annotating something new, it automatically creates a corresponding page that syncs your annotations in readwise and then roam. I prefer the extracts to go directly to Remnote, Obsidian, etc… since I have a habit of forgetting that I’ve annotated something at all when it’s not in my work inboxes

Folks have seen this? Uses templates plugin , works nicely.

1 Like

Can we export annotations through RSS and IFTTT if the notes are posted privately ? What if the annotations are saved to a particular group ?

Alas one would need some sort of authentication to do private posts/groups via RSS and none currently exists. Here’s some further details on RSS/Atom feeds: Atom & RSS Feeds for Annotations : Hypothesis

You might try a work around using View and export Hypothesis annotations.

http://are.na is a good option for this. Their API is robust and you can export all your content super easy.

1 Like

Did you do any good findings? I myself am ordering a Boox Note Air, and I am also very curious to find out how to work with annotations there. Be it in their own reader app or another app or something web based.

Thought I’d throw this in the mix as well— Readwise will import your Hypothes.is notations. I’ve started using Readwise pretty heavily for importing tweets, and realized the other day that it can do hypothesis as well! Readwise has a pretty robust manual export as well as an API.

Another thought, for those who were concerned about privacy… they don’t have a detailed guide, afaict, but you could probably (definitely?) host your own instance of hypothesis! Everything they produce is open-source. The “h” repo is the server code.

Hi, I just got my Obsidian-Hyphothesis plugin approved and now listed in the Community Plugin. Feel free to check it out. Any feedback would be greatly appreciated as well.

8 Likes