I basically do what @elmsfeuer suggests. My literature notes are most often in the language of the text. If it’s a language I’m not very good at, then sometimes I take notes in English because it’s easier (it’s not my native, but that’s what I do my academic writing in). If the source text is in a dead language (which I happen to work with pretty often), then again the notes are often in English, or sometimes mixed (e.g., paraphrases in Latin and other thoughts in English). I have not found it difficult to navigate these.
Overall, I would think the most important factor is what language you want your “output” to be in. What is your output? Papers, blog posts, lectures, notes for yourself…? What language do you want them to be in, primarily?
But of course it also matters how easy it is to take notes in English, and how easy it is for you to translate. I’m terrible at translating, but think very easily in English these days. If you translate easily, then probably the note taking language does not matter that much anyway.