Buying goods from the US in a business based in France: the business refuses to remove the VAT. Can I get the VAT reimbursed?

Question

I live in California, USA. I plan to purchase ~500 EUR worth of goods by phone from a French business that will send me the goods by mail.

I am aware that the business may choose not to charge me VAT, since I live outside the European Union.

However, the said business tells me that:

1. European-Union-based businesses may opt to charge me VAT even though I live outside the European Union.
2. It is an administrative pain not to charge me VAT, because both them and me have to fill some forms, and be potentially audited.
3. Many/most of their main competitors would charge me VAT.

As a result, the said business will charge me VAT.

Can I get the VAT reimbursed?