When quickly moving canvas objects or click on colors and selection buttons, you can accidentally click on remove button. The need to be careful reduces the speed of work.
Proposed solution
To delete objects you need to click on the remove button 2 times.
This is not a matter of convenience, but of psychology, no one want even the smallest chance that something will go wrong. In a note any changes are immediately noticeable. But in canvases, where can be hundreds of notes?