It isnt just the strongest men that are picked for special forces...

That is correct. My group (a SOG) consisted of a dozen men. Our training started with 104, and a dozen made it. The interesting part is that in subsequent training programs, the same dozen finished, and well over a thousand others who wanted in washed out. Many of those were stronger than us, larger than us: more a matter of mind, I think. In my case, the challenge was made, and (being something of an ass) I took it to mean "there is no way you can make me quit!" and so always made it.
I had one friend who was SF and a tunnel rat. He was about 5'6" and 130 lb wet. But I would rather have him on my side than not: he was truly a bad one, not mean, not nasty, but terribly competent. Another friend was a SF blaster, 275 lb, 5'11", strong as an ox, and deadly. Myself, 225 and 6'3". Tobias (NO one called him Toby) was a SEAL, 200 at 5'10". I don't consider myself in the same class as those guys, but did have their respect, why I don't quite understand. And others.
My guess is that probably there are women who could have done.

Its not about physical strenght as much as it is about endurance. You're not going to be expected to like, pick up a helicopter and throw it at the enemy in the SF.