In the last 10 years I used twitter as my main channel to share interesting information with my friends and followers and to discuss with them. I sent more than 50.000 tweets with thousands of quotes, links and images that are very hard to organize and search now.
I’m not sure what is the best way to do it and if there is already a community plugin or tool for it.
I plan to repeat this process every month to add all my new tweets to my Obsidian Vault.
My ideal goal is to
have all my tweets in Obsidian in a specific folder
have a separate note for every tweet I sent (except threads)
have all the tweets in a thread (what is a thread) in a unique note to improve readability
import all the images I tweeted in Obsidian in a “Attachments” subfolder and include them in the tweet note
tag all the content coming from twitter with a specific tag to filter them out in the search if needed
tag all my tweet replies with a specific tag to differentiate them from my main tweets I sent. Usually tweet replies contain less interesting information and more conversational messages.
be sure that every month when I will import the full archive again it will not duplicate the notes
I’m not a developer but I know how to run a script if needed. Is there anything covering at least part of my needs already available?
Thanks for the links but both the tools just convert tweet links to markdown.
I have a tweet archive (a huge json file and attachments in different folders), not tweet links, as a source.
One could possibly parse out the 50K tweets using CLI tools like sed & jq. Then, evaluate those tweets & determine if they are threads or one-off tweets, replies, etc. Build out the markdown format of the tweet or thread and save to md notes for Obsidian. Also, determine which images go with each tweet and create embed code to attach them to the md files. All the above could be accomplished with a bash script using various CLI apps.
- or -
One could register for a Twitter app API bearer token, as described in the aforementioned @kbravh’s Github repo kbravh/tweet-to-markdown’s README. Then, same as above, evaluate those tweets & determine if they are threads or one-off tweets, replies, etc. Then, in a bash script loop thru those tweet URLs and let ttm do all the heavy lifting. ttm would build out the markdown files and (optionally) download the original HQ images or alternatively, link to the already downloaded images you have already exported. One would be wise to rate-limit this script, as to not have Twitter revoke one’s API access for hammering their service. One could use the sleep command to limit how quickly the script makes requests. With 50K+ tweets to archive, one would have to limit the script to only a portion of those since Twitter has a monthly request limit of 50K.
In both scenarious, one still needs to engineer the script to be re-used each time you export your Tweets. The script would have to keep a record of which tweet ID is the latest imported into Obsidian. In the interest of the longevity of script, option 2 would not require you to download the archive of your images since ttm would have that option.
I would have an interest in helping you solve this challenge. My intent would be to work with you to create a script as described above, preferring option 2. I see this as something that other Twitter/Obsidian users could benefit from, and quite frankly could be marketed as a SAAS (Software As AService) for others w/ the same desires to archive their Tweets for their own personal needs.
One immediate question I would have is … Does the archive contain videos and/or animated GIFs probably in MP4 format?
I just created a test twitter account, tweeted a little and requested the archive. It will be ready in 24 hours and I’ll add the link to the thread.
I would personally prefer to have a obsidian plugin doing the import instead of relying on a third party service for both security and performance reasons. I also had very bad experiences with twitter APIs in the past. They are not really reliable and the limitations they set change quite often.