What the fuck did I do?

I can't shake this question out of my head.  The answer for me is no.  But the opposite question intrigues me, which is Have you ever not said I love you when you would have meant it. 

I've been in love with four girls/women over the last thirty years (omigod that makes me sound soo old!) and I only told two of them.  I never told my first love and just when I was good and ready to, she broke my heart by losing her virginity to a holiday romance.  It took years to get over that, and even now, writing this I can feel the pain.  And yet, with hindsight she would have been wrong for me.

I told the second and I married her (and we're still married now after 22 years which ain't bad I guess).  I told the third and it nearly ended my marriage, but I chose to stay and sort things out.  Number 3 was a beauty, five years younger than me.  Ironically, in light of my recent bad behaviour we never had sex (although she did introduce me to phone sex, and it was brilliant with her).  Number 4 I never told because it would have done no good, for either of us, because neither of us would have acted upon it. I don't know if she knew, but I suspect she did and that's why she ended it.