I also agree with this idea. The details of an event could change (such as time or location) and you might forget to invite some people when you first add it.

Also I think it shoud only be in the calendars and Upcoming Event Notification emails for people that are invited... similar to how only the birthdays and anniversarys for close relatives show up in one's calendars.

This is unless a member has his/her settings set to "all events" instead of the default. In this case these members who have this setting can receive notifications of the even even if not invited.