How can you "totally understand", yet think size (or more precisely, lack thereof) shouldn't be a deterrent.  It's really a no-brainer, all other things being equal you go with the bigger player every time. 

For a QB, the heightened (no pun intended) ability to see over the defenders is huge.  The release point of the pass is also naturally higher, making it more difficult to block/deflect. 

As an offensive coordinator you look for the mismatches, getting the 6'4" guy, one-on-one, against the 5'10" guy.

On the other side of the ball, you want the 6'5' DE over the 6'0" guy.

I'm not saying a short player can't be just as good, but in this sport there's a natural advantage in size

---