Can Google still see contacts if not using their contacts app?

Probably. Android has a contacts database with which your contacts app interacts. And Google Play Services, which you can’t disable in stock Android, has access to everything, including this database.

Plus they can use location to see who you meet up with, and get their info and their contacts’ info from their phones. One way or another, Google can build up a pretty thorough profile of your social circle.