Clipboard snippets in your Inbox (for later review), even when Obsidian closed!

Clipboard snippets in your Inbox (for later review), even when Obsidian closed!

This is for Linux but may work elsewhere. It can be used system-wide, even when you don’t even use Obsidian.

Important bugfix: Please upgrade to v1.5+ to prevent errors when using spaces in your file paths!

Inspiration!

Talking with @koala on the Discord today, he gave me an inspiration. How about having something that …

  • would allow copying from any program, web page, code editor, terminal window, etc.
  • then add the text, HTML, or image into a “Clipboard” note in your vault’s Inbox,
  • store copied images in an assets folder (subfolder in Inbox, or even the Inbox folder itself,
  • and even use Pandoc (if installed) to convert HTML to Markdown on the fly, and store the Markdown in your “Clipboard” note?
  • More secret sauce (v1.1): If desired, clipit can now also create .pdf, .odt, .docx clippings from HTML sources. These will be stored in the assets folder and linked in the Clipboard file. Read the comments in the source.
  • Pandoc experts can now really fine-tune the output in v1.3: Extensions and Options supported.
  • V1.5 shows how to make a Daily Clipboard, much like Daily Notes.
  • V1.6 handles some Pandoc edge cases, and also reports Pandoc errors to the CLIPFILE so they don’t go unnoticed.
  • V1.7 now adds a YAML frontmatter header!

Solution!

Well, we have Linux, we can do anything! (Or maybe MacOS or the Windows WSL, where it might work—no guarantees.)

So I created a bash script, clipit, which can be stored in your ~/bin, ~/.local/bin or /usr/local/bin folder, chmod +x’ed and used.

You should modify the CLIPFILE and CLIPASSETS variables to suit your needs. Other options allow for a different date format, image file name, and Markdown dialect.

Almost all Linux systems (with a Desktop Environment) have a means to set keyboard shortcuts and associate applications with those. In my case, on Linux Mint 20.2, I used the Super+V key combo, since it’s kinda “Super Paste”, and Ctrl+V is usually the “paste” command.

(The “Super” key is labelled with either a “flying Windows” or “Tux penguin” on most keyboards.)

So now we’ve set that up, try copying something into the clipboard, and in a terminal try the command clipit. This should work and create a new clipboard file within your vault.

If you get errors, especially Pandoc errors, read the explanations in the code! Most errors come from using an old Pandoc version or some settings that don’t play well together!

If that works, try out copying something and using the Super+V key combination, and it should make the next entry in your clipboard file. Easy, eh?

For the users of outliners and “heading link” people, each entry begins with a H2 header and the date, in ISO format.

Then follows the clipped text, converted HTML, or link to the downloaded image.

Each entry has a horizontal line --- at the end, for easy visual separation.

Requirements

  • Almost any Linux (maybe macOS) system having a Desktop Environment installed.
  • X11 display server (sorry Wayland users!)
  • A bash shell installed.
  • xclip must be installed (it often already is), otherwise use your package manager to install.
    • On Debian derivatives (i.e., Debian, Ubuntu, Linux Mint):
      sudo apt install xclip
      
    • On MacOS, using Homebrew:
      brew install xclip
      
  • Pandoc should be installed and in the PATH, for HTML → Markdown conversion.
  • If Pandoc complains about missing rsvg-convert, install that using your package manager, i.e., on Debian-type systems:
    sudo apt install librsvg2-bin
    

The CODE you’ve been waiting for!

#!/bin/bash

# clipit -- Quickly store clipboard from an X selection into the Obsidian Inbox, for later review.
# Brought to you by Matthias C. Hormann (aka Moonbase59), based on an idea by @koala/kometenstaub.
#
# Requirements:
#   - X11 display server (sorry Wayland users!)
#   - xclip must be installed (it often already is, otherwise use your package manager to install)
#   - Pandoc should be installed and in the PATH, for HTML → Markdown conversion.
#   - If Pandoc complains about missing "rsvg-convert", install that using your package manager,
#     i.e. "sudo apt install librsvg2-bin" on Debian-type systems.
#
# IMPORTANT: Before building your perfect workflow, test this thoroughly using the "clipit" command
# on the command line, especially when using Pandoc! Pandoc has so many versions and different options,
# and you will only see the errors on the command line.
#
# Use with a keyboard shortcut (available in Linux under Settings → Keyboard → Keyboard Shortcuts)
# I’ve opted for SUPER+V, which is not normally used and this is a "super" paste ;-)
#
# Set up CLIPFILE to point to your desired Clipboard Markdown file.
# Each clipit invocation will generate a new, time-stamped entry.
#
# Set up CLIPASSETS (with a trailing /) to point to a folder where images should be stored.
# If the clipboard contains an image that can be saved as PNG, it will store a time-stamped
# "Pasted Image YYYYMMDDhhmmss.png" file in the CLIPASSETS folder, much like Obsidian does.
# An embed link to that image will be written into the CLIPFILE.
#
# v1.0 -- 2021-08-19
#   - Initial release
#   - Intentionally didn’t restrict xclip to _one_ clipboard, so it also works on console
# v1.1 -- 2021-08-19
#   - Added option for other output formats (pdf, odt, docx), written to assets folder
# v1.2 -- 2021-08-20
#   - Added XSELECTION option, to define the X11 selection to take
# v1.3 -- 2021-08-20
#   - Added EXTENSIONS and OPTIONS for Pandoc, so the output can be fine-tuned.
# v1.4 -- 2021-08-20
#   - Added good defaults and MUCH more comments. READ THEM.
# v1.5 -- 2021-08-21
#   - Bugfix: CLIPFILE and CLIPASSETS paths with SPACES in them now work correctly.
#   - Code optimizations to get rid of the many "echo" statements.
#   - Added a comment on how to make a CLIPFILE PER DAY ("YYYY-MM-DD Clipboard.md").
# v1.6 -- 2021-08-20
#   - Added Pandoc error reporting to CLIPFILE so setting problems don’t go unnoticed.
#   - Bugfix: When copying from LibreOffice, a NUL byte gets added that confuses other programs.
#     This now gets removed by clipit.
#   - Code rework to handle edge cases for Pandoc output, i.e. "Font Name" with spaces.
# v1.7 -- 2021-08-26
#   - Upon initial creation of a CLIPFILE, a YAML HEADER is now written.
#     It has "date: YYYY-MM-DD" and user-settable "tags:"

# define me
me=`basename "$0"`

# --- USER CUSTOMIZATION: File and Assets Path ---

# This is the clipboard file (should point to a file within your vault’s Inbox)
# Use an absolute path here, and prefer $USER over $HOME, the latter isn’t always available.
# You CAN make a CLIPFILE PER DAY, using:
CLIPFILE="/home/$USER/Dokumente/Obsidian/Knowledgebase/-Inbox/$(date +"%F") Clipboard.md"

# If you just want ONE CLIPFILE, use this:
#CLIPFILE="/home/$USER/Dokumente/Obsidian/Knowledgebase/-Inbox/-Clipboard.md"

# The assets folder for images (must not necessarily be a subfolder of Inbox).
# Use a trailing "/"!
CLIPASSETS="/home/$USER/Dokumente/Obsidian/Knowledgebase/-Inbox/Anhänge/"

# tags for the YAML header: use "[tag1, tag2]" notation, NO # here!
#TAGS="" # no tags
TAGS="[clipboard]"

# --- USER CUSTOMIZATION: Extras, normally no need to change ---
# --- THINGS CAN BREAK HERE, but you might want to make it perfect. TEST ON COMMANDLINE! ---

# Date format YYYY-MM-DDThh:mm:ss+00:00
#TIMESTAMP=$(date +"%FT%T%:z")
TIMESTAMP=$(date +"%F %H:%M")
# Image name, make it unique! "Pasted Image YYYYMMDDhhmmss.png" here, like Obsidian.
IMAGENAME="Pasted Image $(date +"%Y%m%d%H%M%S").png"
# Base name for non-Markdown conversions (file output, extension comes from OUTPUTEXT)
OTHERNAME="Pasted Clip $(date +"%Y%m%d%H%M%S")"

# Pandoc’s list of Markdown dialects (my preference is "markdown_strict")
MARKDOWNDIALECTS=(commonmark, commonmark_x, gfm, markdown, markdown_github,
  markdown_mmd, markdown_phpextra, markdown_strict)

# Markdown dialect to use for HTML conversion.
# Use one from above list! (markdown = Pandoc’s Markdown)
# If you really want to output docx, odt, pdf instead,
# you MUST set the correct OUTPUTEXTENSIONS, OPTIONS and OUTPUTEXT!
# Check Pandoc documentation for the version you have and TEST ON COMMANDLINE!
OUTPUTFORMAT="markdown_strict"
# Example for PDF output:
#OUTPUTFORMAT="pdf"

# Pandoc Extensions (+ to include, - to exclude, no spaces in between!)
# These must be allowed for the selected OUTPUTFORMAT, respectively, or you’ll get an error.
# Check Pandoc documentation for the version you have!
OUTPUTEXTENSIONS="+backtick_code_blocks+strikeout+footnotes" # for Markdown
# Example for PDF output:
#OUTPUTEXTENSIONS=""

# Pandoc options, set all headings ATX style (old: "--atx-headers", new: "--markdown-headings=atx")
# These can usually be left in: "--markdown-headings=atx -V papersize:a4". Check Pandoc docs!
# Need to put these into an array so the quoting works. (See BashFAQ/050.)
OPTIONS=(--atx-headers)
# Example for PDF output (for UTF-8, must use xelatex and font supporting all characters used):
#OPTIONS=(--markdown-headings=atx --pdf-engine=xelatex -V papersize:a4 -V mainfont:"DejaVu Sans")
# can use papersize:letter

# If using non-Markdown file output, you MUST specify the extension (with a leading dot).
# Get the list of possible output formats:
#   pandoc --list-output-formats
OUTPUTEXT=".md"
# Example for PDF output:
#OUTPUTEXT=".pdf"

# --- DO NOT MODIFY BELOW THIS LINE! ---

# Which X11 selection to take?
# Use either of:
#   primary -- default; primary selection (copy of last selection; mouse selection)
#   secondary -- application-specific, seldom used
#   clipboard -- Ctrl+C/X/V and context menu clipboard
XSELECTION="primary"

# check if xclip is installed
command -v xclip >/dev/null 2>&1 || \
  { echo >&2 "$me requires \"xclip\" but it's not installed. Aborting."; exit 2; }

# check if we have Pandoc available
PANDOC=true
command -v pandoc >/dev/null 2>&1 || \
  { PANDOC=false; echo >&2 "Pandoc is not installed."; \
  echo >&2 "$me will work, but you're missing out big time!"; }

# Should you HAVE Pandoc but really don’t want to use it, set ...
# PANDOC=false

# tr should be available, no need to check here
# (it’s in GNU coreutils so should be in all distros)

# function: if the CLIPFILE doesn’t yet exist, create it & write a YAML header
init_clipfile () {
  if [ ! -e "${CLIPFILE}" ] ; then
    echo $'---\n'"date: $(date +"%F")"$'\n'"tags: ${TAGS}"$'\n---\n' >> "${CLIPFILE}"
  fi
}

# check available targets and go process!
case $(xclip -o -t TARGETS) in

  *image/png*)
    # an image that can be converted to PNG
    #echo "Found image, storing in ${CLIPASSETS}${IMAGENAME}"

    init_clipfile
    echo "## ${TIMESTAMP}"$'\n' >> "${CLIPFILE}"
    xclip -selection ${XSELECTION} -o -t image/png > "${CLIPASSETS}${IMAGENAME}"
    echo "![[${IMAGENAME}]]"$'\n\n---\n' >> "${CLIPFILE}"
    ;;

  *text/html*)
    # found HTML text, try to convert to Markdown
    #echo "Found HTML text"

    init_clipfile
    echo "## ${TIMESTAMP}"$'\n' >> "${CLIPFILE}"
    if [ "$PANDOC" = true ] ; then
      if [[ "${MARKDOWNDIALECTS[@]}" =~ "${OUTPUTFORMAT}" ]] ; then
        # ok, it’s Markdown, put into Clipboard file
        # uses tr to remove unwanted NUL byte at end if copying from LibreOffice
        xclip -selection ${XSELECTION} -o -t text/html | LC_ALL=C tr -d '\000' \
          | pandoc -r html -w ${OUTPUTFORMAT}${OUTPUTEXTENSIONS} "${OPTIONS[@]}" \
          >> "${CLIPFILE}"
        ERROR=$?
        if [ $ERROR -gt 0 ] ; then
          echo "**Pandoc error ${ERROR}: Check \`${me}\` settings!**" >> "${CLIPFILE}"
          echo "[List of Pandoc exit codes](https://pandoc.org/MANUAL.html#exit-codes)" >> "${CLIPFILE}"
        fi
      else
        # some other format requested, write file to assets folder & create a link
        # uses tr to remove unwanted NUL byte at end if copying from LibreOffice
        xclip -selection ${XSELECTION} -o -t text/html | LC_ALL=C tr -d '\000' \
          | pandoc -r html -w ${OUTPUTFORMAT}${OUTPUTEXTENSIONS} "${OPTIONS[@]}" \
          -o "${CLIPASSETS}${OTHERNAME}${OUTPUTEXT}"
        ERROR=$?
        if [ $ERROR -gt 0 ] ; then
          echo "**Pandoc error ${ERROR}: Check \`${me}\` settings!**" >> "${CLIPFILE}"
          echo "[List of Pandoc exit codes](https://pandoc.org/MANUAL.html#exit-codes)" >> "${CLIPFILE}"
        else
          echo "[[${OTHERNAME}${OUTPUTEXT}]]" >> "${CLIPFILE}"
        fi
      fi
    else
      xclip -selection ${XSELECTION} -o >> "${CLIPFILE}"
    fi
    echo $'\n\n---\n' >> "${CLIPFILE}"
    ;;

  *TEXT*|*STRING*)
    # found TEXT, COMPOUND_TEXT, STRING or UTF8_STRING
    #echo "Found text"

    init_clipfile
    echo "## ${TIMESTAMP}"$'\n' >> "${CLIPFILE}"
    xclip -selection ${XSELECTION} -o >> "${CLIPFILE}"
    echo $'\n\n---\n' >> "${CLIPFILE}"
    ;;
esac

Happy clipping!

Fun fact: You can even have your clipboard note open in Obsidian and watch it grow! :slight_smile:

Notes

This is not meant to be a replacement for full-fledged webpage clippers like @death.au’s wonderful MarkDownload browser add-on. It’s just a convenient and fast tool to clip and collect text, HTML and images from everywhere, without the need of starting Obsidian everytime.

Thanks for the good idea, @koala!

Changelog

  • v1.0 – 2021-08-19
    • Initial release
    • Intentionally didn’t restrict xclip to one clipboard, so it also works on console
  • v1.1 – 2021-08-19
    • Added option for other output formats (pdf, odt, docx), written to assets folder. This only works from HTML clippings!
  • v1.2 – 2021-08-20
    • Added XSELECTION option, to define the X11 selection to take. (For edge cases.)
  • v1.3 – 2021-08-20
    • Added EXTENSIONS and OPTIONS for Pandoc, so the output can be fine-tuned.
  • v1.4 – 2021-08-20
    • Added good defaults and MUCH more comments. READ THEM.
  • v1.5 – 2021-08-21
    • Bugfix: CLIPFILE and CLIPASSETS paths with SPACES in them now work correctly.
    • Code optimizations to get rid of the many “echo” statements.
    • Added a comment on how to make a CLIPFILE PER DAY (“YYYY-MM-DD Clipboard.md”).
  • v1.6 – 2021-08-20
    • Added Pandoc error reporting to CLIPFILE so setting problems don’t go unnoticed.
    • Bugfix: When copying from LibreOffice, a NUL byte gets added that confuses other programs. This now gets removed by clipit.
    • Code rework to handle edge cases for Pandoc output, i.e. “Font Name” with spaces.
  • v1.7 – 2021-08-26
    • Upon initial creation of a CLIPFILE, a YAML HEADER is now written.
      It has “date: YYYY-MM-DD” and user-settable “tags:”. Tag default is “#clipboard”.
8 Likes

The actual clip it made from the above (using markdown_strict style)

This went way too long, so here a screenshot of a part of it, unmodified, as is in my Clipboard file:

Note this is from version 1.1—there might have been updates since, always check first post!

3 Likes

Important bugfix and some usage notes

  • There was an error when your CLIPFILE and/or CLIPASSETS path contains spaces. This has been fixed in v1.5+, please upgrade!
  • Daily use has shown that clippings accumulate rather fast, making one big Clipboard file unwieldy. I have added a comment in the code on how to have clipit automatically create Daily Clipboards (“YYYY-MM-DD Clipboard.md”), much like the Daily Notes.
  • The question has come up if this tool somehow interacts with Obsidian or could be used standalone.
    • No, it doesn’t require Obsidian, and can be used standalone! You can adjust CLIPFILE and the CLIPASSETS path to anything you want; there’s no requirement to point these into your vault (although that’s really helpful).
    • It is a system-wide tool and can be used to clip (almost) anything from anywhere. Think web page snippets, code snippets from your editor, text from your preferred text editor, clips from the terminal, etc.

Hi thanks for the script.

But its possible create a github repo? from follow all updates.

Regards,

2 Likes

This is great, thanks a lot @Moonbase59

Unfortunately, I couldn’t get it to work on macOS though. I’ve installed all required dependencies such as xclip and pandoc, but ran into issues with xclip. I’ve installed X-Quartz for the X11 requirement using Homebrew:

brew install --cask xquartz

And then debugged the script after adding set -x, by executing it directly within my terminal, which resulted in:

+ XSELECTION=primary
+ command -v xclip
+ PANDOC=true
+ command -v pandoc
+ case $(xclip -o -t TARGETS) in
++ xclip -o -t TARGETS
Error: Can't open display: (null)

I noticed there are other people who ran into issues with xclip on macOS: Does anyone use xclip successfuly on Mac? How? · Issue #52 · astrand/xclip · GitHub, but perhaps I’m just missing something. I’m also not really familiar with how X11 works - after looking into pbpaste, having xclip would be important though as pbpaste can’t handle images and GitHub - jcsalterego/pngpaste: Paste PNG into files, much like pbpaste does for text. wouldn’t work as we can’t loop over the clipboard content to determine whether it’s an image, text or a combination of both.

Has anyone found a good way to get it to work on macOS?

Windows users can use Ditto to capture the clipboard and this python script I made that converts Ditto’s database to a markdown file.