I’m dealing with a memory explosion with dataview. After using about 4gigs of memory, obsidian crashes to blank and releases memory. I’m on windows 10 64bit, 32gb memory
I’m trying to export a csv of media records.
I’m using dataview, tasks, and csv exporter for this. A single media record as a task might look like this:
- [x] SCP Archives podcast [sub::SCP-049: "The Plague Doctor"],[numeric::5],[publish::2019-04-09], ✅2020-07-17
I keep my records together in a file per media source (in this case, the SCP podcast) or a catch-all podcast page, and these tasks helpfully inherit metadata from the parent file. So far so good.
for reasons (driving some gdoc charts) I want to periodically export csv, so I have a dataview table as follows:
TABLE WITHOUT ID Tasks.completion AS date, regexreplace(Tasks.text, "[\[✅].*$", "") AS media, Tasks.numeric AS numeric, Tasks.sub AS sub, Tasks.published AS published, media-type AS type, Tasks.media-type AS unittype, Tasks.priority AS "pri" FROM #_media WHERE file.tasks FLATTEN file.tasks AS Tasks WHERE Tasks.completion > date(2022-12-31) LIMIT 10000
this dataview looks for all #_media tagged notes and flattens every task within into a table that gives me useful info - especially date, media name, and media type.
Then I can use table to csv exporter.
This had been going well until I finished ingesting my backlog of media records.
Now when I try to generate the above dataview, the flatten operation causes the memory to balloon to 4gb, and then obsidian crashes.
I’d love to know if there’s a way to let the process use near to my full 32 gb of memory to brute force the problem, or if there is a way to get this data transformation to use far less memory without me having to reorganise a bunch of data by hand.
- lowered the LIMIT - has no effect, probably would just be filtering the results
- removed the regexreplace - no discernible difference
- excluded a number of files - when I excluded a large portion of my media notes, I got the process to peak at about 3gb then successfully complete, which then released the memory back down to about 400mb.
- checked that I am in fact using the 64 bit version of obsidian
Any help would be greatly appreciated!