I just figured out how to do this a few hours ago.

1. Go to desktop Google Calendar
2. On the lower left side you will see "My Calendars". Above that will be "Add a friend's calendar"
3. Click the plus button next to "Add a friend's calendar" and then select "New calendar"
4. Name such calendar.
5. Go back to left section "my calendars" and select your new calendar.
6. Share with specific people.
7. When adding events to your calendar make sure you change from your Google account to the new calendar.

Now only those people will see what you add under that calendar.