Crawling Obsidian publish

Wondering if anyone ever coded something to crawl obsidian publish? I had issue last time I tried for some reason (i want to crawl my own obsidian publish btw)

related Publish: let the user provide a custom robot.txt (Block crawling, LLM Content protection) (but in my case I WANT TO CRAWL)