I can understand your point, but I don't think it's as big of a deal as you might think. Imagine it this way, if you have a photo of a deceased loved one hanging on your wall or a keepsake of theirs, you certainly wouldn't say you worship them. If someone questioned your purpose of having the picture/keepsake of someone that is no longer with us, logically you would not have an answer, logically there is no reason. There are many other abstract reasons such as sentimental, hope, perserverance, memories, love, respect, things you really cannot explain. It would be no different than visiting the grave of a loved one to pay your respects, you are not worshipping that person. I see Mary the same way, not as a deity to worship, but as the mother of Jesus.
I am not asking to argue what I have posted, I am trying to help you understand what it is like from say my perspective. You may find that our beliefs are really probably not much different.