Not just infuriating, but potentially dangerous. If a website uses my audio to play a 100% volume sound without my consent and I had my headphones up high to hear a subtle recording, I might get permanent hearing damage.
Every time the concept of a "Web Bill of Rights" comes up, I want Article I to be: no use of audio without express user consent. That consent might be stored for a time on a per domain basis, but only for 1st party content.
This seems at odds with the conventional wisdom around here RE: adblock. You're simply requesting bytes from a server, what your computer does with them is up to you (and therefore your responsibility). You should probably configure your system so website have no ability to autoplay.