I want to support some math notations,replace $$,$$ to \[, \]

I tried to use MarkdownPostProcessor, but text and markdown is mixed together, so if there other way to separate text and [ … ] ?
by the way, how to use MarkdownRenderer.render method?

For your second query, here is an example where I am rendering the HTML code returned by the render method inside an HTML div :

	const componentRef = useRef<Component | null>(null);
	useEffect(() => {
		// Initialize Obsidian Component on mount
		componentRef.current = new Component();
		componentRef.current.load();

		return () => {
			// Cleanup the component on unmount
			componentRef.current?.unload();
		};
	}, []);

		const rawTextToRender = "#This is H1\nThis is a simple line\n- [ ] This is a Task";
		renderDivRef.current.innerHTML = ''; // clean up old content
		const renderDivRef = useRef<HTMLDivElement>(null);
		await MarkdownRenderer.render(
			app,
			rawTextToRender,
			renderDivRef.current,
			filePathOfTheContent,
			componentRef.current
		);

	return (
		<div className="MarkdownRenderSection" ref={renderDivRef} />
	);

Using the above method, you can render either in normal View or also inside a Modal. (I am redering inside modal, hence I had to create the componentRef).

For getting the hover preview and links inside your renderedText to work, go through this topic : Internal Links don't work in Custom view - #3 by Tu2_atmanand

For your first question, why not use a regular Expression to replace the pattern and then writing it back to the file ?

Is there anything specific you are trying to achieve ?

This seems a bit of an xy problem. Is what you’re after access to the mathjax API/config? Some of the TeX plugins already access it, perhaps have a look there.