Cost was insane. IIRC SAC was faced with keeping one wing of these operational or 2-3 wings of B-52s. Combined with the limited operational niche and less flexibility, they ended up retired. Still a real technical achievement from the period.
At one point, it was no object during the Cold War. Hence the B-58 and B-70. The B-58 was evidently deemed vulnerable to SAM defenses as well-- Still debatable, I think when you're carting a weapons pod with a several megaton city-buster as well as 4 more in the later upgrades of the plane and all it would take would be a few to get through.