So if the cloak were to bend light around the subject, it would be impossible for them to see while invisible. An additional sort of "invisibility goggles" would have to be invented as well in order to let a small amount of light through. Otherwise a person wearing it would need eye-holes and just look like a pair of eyes floating around.
Additionally, the suit would have to be constantly clean and used only in dust-free dry environments, otherwise a thin film would appear on the surface making the figure look like a ghost. While it would be difficult to spot someone using the cloak, countermeasures based on infrared could be set up at installations that sprayed glowing or glittering dust on anyone who tripped an infrared detector. Since heat emanates from the human body, subjects within the invisibility suit would likely show up on infrared.
Additionally, an invisibility cloak user's gloves and boots would be the most visible things, as they would track in dust and particles from the environment. The eventual system would only be 100% effective in environments which were enclosed and free of floating particles.
To get around this, microscopic vacuum pumps working on silent might be able to channel some particles around the cloak just as light is channeled while a thermal suit could make the subject less likely to trip infrared. There has been research that certain types of robot can operate without giving off recognizable heat signatures, but I can't remember what they called them. Additionally, a robot could have the "cloak" installed in its protective shell rather than a fabric-like material.
The best/worst part of invisibility cloaks? A nation would only have to have one and say it's deployed, and it's the perfect tool of psychological warfare to inspire anxiety and paranoia.