Can I build a product on top of CiviCRM and offer it as an ASP without having to provide a download of the source code to that separate product?

Yes, so long as CiviCRM is a separate and independent program that your program interacts with solely through its API, the download provision of the AGPL does not apply to your separate program. If, however, CiviCRM files are modified in any way, then that modified version of CiviCRM is subject to the AGPL and its download requirements.