Using the physical card or not are two different scenarios, namely "Card Present" and "Card Not Present" (also known as MOTO as in Mail Order / Telephone Order). They may involve different contracts, different rates, different risks, and different equipment.
Some contracts will simply not allow Card Not Present transactions. You need to actually use the card in the terminal, either by swiping it, or by using the chip (and ideally pin). This adds an additional layer of verification (mostly if you use chip & pin, but even the magnetic stripe has info that is not available by reading the card), and the network and card issuer know if the card was actually used or not.
Likewise, some terminals will not enable you to do a card not present transaction. Even if it has a keypad, it may simply not have any feature allowing the manual entry of a card.
Since the merchant does not see the card, and none of the security features available with a payment terminal can be used, there is also an additional risk. This may involve higher fees for the merchant and/or a higher risk of a chargeback. Usually the risk lies with the bank if the transaction used one of the secure modes (chip + pin, or 3D secure when used online), while the risk lies with the merchant in other cases.
So, as a summary:
- he may just not be able to (contract or terminal won't allow it)
- it may cost him more (higher fees)
- it may involve a higher risk
Or he may just be grumpy :-)