Not sure, if we think of the same story. I know that I made a few comments lately on a story that I think is supposed to be con, but reads to me like non-con. Anyway - to your question and my thoughts:

I think I'm pretty much like you in this area.
That's why for me personally it's more difficult to read stories that are tagged "dub-con" than ones that are clearly tagged "non-con".
I like it, if the level of consent is a black or white thing.

A lot of times I have the feeling with "dub-con" that while it reads to me like non-con, the writer might have thought of it as "not really(?) non-con" or consentional.
To name just two examples that I think are often tagged like that (and please, I don't want to offend anyone who might have written something along those lines, I'm just stating how it feels to me personally)
... prostitution - does it count as consent, if the person chose to get money that way, even if they didn't see any other options? Is it really a free choice or does take desperation that choice away?
... stories where someone is kidnapped/abused/raped and suddenly falls in love with the abuser/rapist - to me it cries Stockholm-Syndrome, to some others it still says epic love story.

I don't know...which I guess, is what it comes down to when tagging it dub-con ...suspicion it might not be con, but not sure about it? The grey area.

If we wanted to get really philosophical, we could even widen the consent-theme to not sex related themes. canon even.
Like, angels need their vessels consent to use them as a meat-suit.
But is a YES from Dean or Sam, when given as a result of torture or because there is no other way to stop the end of the world really consent?