I believe golf takes a very special set of skills and quite a bit of athletisicm. You have to be very good with your hand-eye coordination. Strength and stamina play an important part as well. How well could you play other sports if your hand eye coordination were not good, even if you are lean and have speed? Maybe Soccer?
I think the reason many believe golf is not athletic is because almost everyone can play it and enjoy it, but how many can play it well. Not many. I have tried several sports and consider myself a pretty decent athlete, and golf is the definitely the most difficult sport I have ever tried to be good at. Sure, you don't have to run, but you have to use your physical skills along with your mental toughness and determination to be successful, and it takes a lot of practice (or at least it does for me).
By the way, it does irritate me to a degree to look at the heavy guys on tour like the Stadlers, Herron, De Jonge, Petterson, etc and know I couldn't touch them on the course...and I am 2 Hcp. It has taken me several years to get to this level. But that is part of the beauty of the game that should give us "athletes" hope to improve, because surely we should be better than those fat guys...