here is a link to the post I made on stack overflow but I will copy the text below.
TL;DR - link canvas cells/cards to jupyter notebook cells to allow dynamic code running.
I use python as part of my daily workflow mostly for modeling and data analysis and I’ve been dying to use some system similar to the one I outlined in this picture (here I am using obsidian canvas as an example). My dream is to have something like this with one click snapshot to save current cell layouts and code for quick and dirty version control with simple notes. Are there any existing systems that come close to this? I have not messed with apache airflow but seems to be similar but much more production oriented instead of proof of concept/prototyping oriented
ideal software would
- be lightweight
- allow drag and drop cells and flow lines
- click into the cell to open up code
- one click snapshots with timestamp and user note (ie changes made notes)
- only allow variable inheritance from upstream cell blocks.
The idea here is that I have to quickly experiment with so many variations of data, filtering, modeling approaches, dataset validations etc. So I want to remember what has worked so far without worrying about making perfect/permanent code. Once proof of concept is established I move from notebooks to python package etc.
