Seeking to become free of the fear of what others think, takes time; perhaps a lifetime of discipline and effort to reach the goal desired. Best to do what one wants, than to not do it out of fear of others thoughts. People will think what they will, nothing to be done about it.
It can be very difficult to do this; I am of course speaking from experience. It is the little things that are important, doing what is right, instead of what others think should be done. Little by little, this freedom grows, and joy and freedom are possible. The fruit of this kind of practice is that compassion and acceptance of others grow.
Taking care of self is important, no one else will. If someone takes advantage of me, it is my fault, I allow it by not speaking up and letting others see me in a new light; this is an injustice to those who have to deal with me. Unless there is a place to stand from, others will not know how to treat me. Not taking responsibility for ones life, is to become an unwitting victim of others, who do not want that power.
To speak ones mind respectfully, may cause others discomfort at times, or even anger, but in the end respect is gained, and one is seen in a new way, and treated accordingly. Expressing oneself raises the interaction with others from the unconscious to the conscious mind, causing growth and change, and not an endless cycle of misunderstandings, based on the lack of ability to clearly say what is wanted or needed.
If one lacks the courage to do this, then a life of frustration is lived. Freedom has to be gained one step at a time, one choice at a time, taking a chance, one situation at a time. To not do the above, is to become a prisoner of ones own making. To reject this, leads to greater and greater difficulty in communication, since it is not based on truth, but on fear, and perhaps only saying, or doing, what one thinks others want. It is often called passive/aggressive behavior, and shame is often the root of it.