Yes but some one can break into whatever server is hosting the source and add a backdoor withought anyone noticing for quite a while, do you go through all the code you compile? Can't think of any specific examples right now but i remember this happening a few times

You can rather easily see the deltas. It's not perfect, but it's at least possible. That's another line of defense Windows users lack (and, usually, don't care about)