By default everyone in your account can see them. If you want your notes to be private, you have two options:
1) Add one or more custom fields to the Contacts module, assign the custom fields only to you, and enter your private notes into those fields instead of the general Notes field.
2) Turn on the Personal Contacts Only option for your other users. Then they will only see the phone, email and address of the first contact attached to each event, but won't be able to see the full contact records or browse the complete Contacts list.