Learning Data Science/Machine Learning and connecting concepts

If anyone has any experience of Learning Data Science using Obsidian, please help me.

I am just 3 days into using Obsidian, really great map view but want to know how do you connect ides in Data Science?
Basically everything in my map is just connected so harder to get the bigger picture. Still learning the second brain concept, so will be updating the forums if I find any better ideas.

3 Likes

Iā€™ve been in the same boat as you. I started using Obsidian when I was starting to study about Machine Learning. And I have to admin that I did not get much value out of the graph view, except for the fact that it looks pretty. Since concepts like Parameters are used in every single note, it now has like 100ā€™s of connections and make the graph view kinda clumsy.

But writing down notes, then looking for places where I can link it to and finally trying to organize them using MOCs really helped me flush out the concepts. So just start taking notes, donā€™t overthink about process of Knowledge Management in the begining, you will get there. Just start taking notes.

3 Likes

Hi buddy,
I can learn from your methods, can we connect on any other platform were we can have a quick chat?

I am currently taking down notes in Evernote and then trying to connect it in Obsidian, I am unable to understand the graph but it looks really pretty, not useful though.

Hi,
Iā€™m a mathematician and preparing for a new position in ML validation etc.

Since ML / DSci was not my primary area of research Iā€™ve begun developing a library of useful approaches to common problems that will be useful in my new position ā€” along with relevant examples, links to corresponding libraries, and methods in sklearn etc.

Iā€™d be curious of your structure for dependencies in knowledge, I.e., how you handle the fact that a confusion matrix and decision boundaries are prerequisite to ROC. Also do you use a template for concepts/techniques etc?

I have a structure note corresponding to the problem/task type: ā€œclassification performance metricsā€. This structure note links to notes each of which are approaches to solve the problemā€” i.e., I have a note linked to in this structure note called ā€œclassification performance via ROCā€, and ā€œclassification performance via f-betaā€

Currently I have three note types and corresponding templates:

  1. Problem / Task ā€”a structure note which specifies the task and links toā€¦
  2. Approaches / Solutionsā€” a note which codified one approach to solving the problem in the associated structure/problem note. If the approach requires knowledge of ML concepts about which I am unfamiliar then I have
  3. Object / Concept noteā€” a simple note with object definition, applications, example, quick facts, and relevant references.

Iā€™m hoping that a core set of tasks/problems with solutions exampleā€™s and references will be helpful.

If you have any suggestions for alternative structures for this particular task-solution-reference use-case Iā€™d be grateful to hear your thoughts.

Happy to share my templates if anyone is interested. Please do the same if you are working with a task/problem-approach/solution sort of structure.

4 Likes

Iā€™ve been studying a quantitative masterā€™s degree in the last few months and all my notes are in Obsidian, even lots of LaTeX equations. I have a few hundred notes and the graph view has been incredibly useful for me to understand concepts on my course. Iā€™m pleased with results Iā€™ve had from my recent assessments so itā€™s gone well; if I did ā€˜linearā€™ notes I wouldnā€™t have done that well.

Two of the biggest modules of my degree have been biostats (:red_circle:) and epidemiology (:yellow_circle:). Over time it really became clear when I was taking notes how they related to each other, from the graph view. I have one folder per module in my vault and use those for the colour code.

My longest notes have ~1k words. The typical note has ~250 words. My knowledge base currently totals over 55k words. I wasnā€™t expecting to write so muchā€¦ :grinning_face_with_smiling_eyes: I initially thought it would be very tough to revise that amount of content, but didnā€™t need to read much for exam preparation. The note taking process definitely helped my comprehension of concepts.

Backlinks are the most powerful thing IMO (not the main graph view), especially when you have dozens of notes. Backlinks are what I used most for navigating between concepts and understanding how things relate to each other, in a rapid way. If a note didnā€™t have the backlinks that Iā€™d have expected, it was a sign I needed to revisit other notes and add wikilinks to that note.

General summary of how I have done note-taking:

  • I do one note per concept. If something sounds like a concept that may come up, Iā€™ll create a wikilink for it. I havenā€™t felt need to make notes for literature.
  • I donā€™t do MOCs. I wouldnā€™t even know how to structure MOCs for my knowledge graph! :upside_down_face: The closest thing I have is a few notes that have many wikilinks, e.g. ā€˜Study designsā€™ note has a table of the different study designs in epidemiology, ā€˜Hypothesis testingā€™ note has sections for different types of tests e.g. goodness of fit, tests of independence. I created those notes to help synthesise the concepts - they arenā€™t really tables of content, they do have a fair amount of text.
  • If a note has many sections or gets quite large (~750 words), that is an indication I could benefit from splitting out the content into more notes. I have a note for ā€˜Biasā€™, which is around 750 words and is focused on the statistical aspects of bias, but early on it also had content on types of bias that affect study design (e.g. ā€˜Selection biasā€™). Once I had more detail on those concepts in later lectures, I would rework the notes. It ended up quite organic.
  • I do tagging to help me flag which notes I need to rework, where I need to spend more time writing compared to what I managed during lectures. Typically it would be something like ā€˜#REVISITā€™ - I would blast through about 30 of these on a good day. It was actually quite a nice way to gamify my learning. I now have about ~60 of these tags in my graph so I figure I need around 3 solid days of reworking my notes before my next term starts.
  • I tend to keep code blocks and equations in notes for concepts. I donā€™t split it out into separate notes for equations, libraries, etc.

Iā€™m doing modules next term on the machine learning, I can see from the greyed out notes where some of the concepts fit in my current graph, so that also gives me ideas of which notes I should revise ahead of that.

3 Likes

i would love to to see your templates! You inspired my to try out your note types for my cloud computing knowledge management.

Happy to, Iā€™ll add to a github repo
BRB

Iā€™ve shared/introduced the templates https://forum.obsidian.md/t/task-solution-oriented-templates-for-reference-during-ml-data-science-tasks-link-included/30318

Direct link:

https://github.com/BradfordFournier/problem-solution-markdown-templates

1 Like

Hello there,

Learning data science / ML too. My problem is that most of my notes are inside Jupyter notebooks because I need to explore some concepts interactively and not only with mathematics. Imagine youā€™re studying NLP. Okay there are tons of maths and concepts to write onā€¦ but eventually you want to test it by yourself in a notebook. You want some graphs, to see the process, the resultsā€¦

And Iā€™m struggling on how to integrate those into Obsidian. What would you recommend?

1 Like

Have you looked at foam in VSCode. Should work fine on the markdown files in your vault. I use Jupyter NBs in VSCode and at least itā€™s convenient to have the foam extension for accessing Obsidian notes. Foam | A personal knowledge management and sharing system for VSCode

Obsidian Jupyter Plugin works great for me. You can use tag system in markdown part of your notes, i.e. #scikit_learn #decision_tree #classification/binary etc. or links to other existing notes like this [[decision tree]]. You will be building a working concept/knowledge map in no time.

Like many Iā€™ve been taking notes on AI/ML and deep learning concepts. I was struggling also to have a good overview.

So I tried something which was using the new Canvas Plug-in and also had ChatGPT help out in providing an outline on what it thought the best way of structuring a Map of Content of these topics.

It was quite surprising how it unlocked it for me. Something that I had been struggling to formulate.

Using Canvas I also think is key since it allows you to have full input on how you want to map relationships between your notes.

I think once a couple helper plugins come out for it, it will be the go to way to structure concepts in my vault.

Iā€™m actually using Obsidian for studing data science.

I have a folder called Zettlers, in where I store all my notes, when I read a book, i just create a new note about the section of the book about what Iā€™m understand for about I just read.

My notes structures are the title, the keywords and a reference section, in the keywords section i connect the note to a note related to that subseject, for example, if the note is about python, I will connect to the python note, if the note is about data science too, I will add a connection to the data section note. For each book I read, i create a note about that book, and for each note what I create from that book I connect it to the note of the book.

In the particular case of Data science, Iā€™m doing the google advance course from coursea, and for each video, exercice and text, I create a new note about it, in the keywords section I put a connection to the note related to the course, but I reference the a subheader related to the part of the course of what is about, for example the last note I create is called GoogleAdvace####Present a story. In the body section I put what I understand about the material of the course, and in the reference section I put the text and video of the part of the course.

The only limitation is that Obsidian can not read jupyter notebooks, so I just download the jupyter in .ipynb and in PDF and I put both in the note of where the notebook belong to.

here an example of a note of the google course from coursea.