According to a recent interview between, Edge magazine, Super Mario Galaxy director Yoshiaki Koizum. Koizum explains that the development had some trouble figuring out the camera issue with a 3d Mario game. 

After working on titles such as Super Mario Sunshine I began to wonder if it was possible to continue to implement the themes of surprise and ease of play with the increasing complexity of camera controls,” Yoshiaki said.

“Until I could find a solution I decided to close off the idea of any future Mario games.” 

I have to say I think they figured it out, it may of taken them 5 years, but they figured it out.