No, you should have been on frequency for "the ATC facility providing air traffic services," which is something us lowly pilots are not privy to. It might be Tower, it might be Tower for a neighboring airport, it might be Approach, or it might even be Center. It simply is not true that all of Class D airspace is under the control of the tower inside. It's very common for bits to get traded with neighboring airports, or the top 1000 feet to get handled by Approach. There doesn't even have to be a tower.
We have to depend on handoffs, or ask. We might be able to guess under some circumstances, but not upon entering Class D. The regs do get a bit clearer if you are in the pattern of an airport with an operating control tower, but that happens well after entering Class D (some student patterns notwithstanding).