It's known that any multi-qubit quantum gate can be represented as a product of a number of CNOT and single-qubit gates. The total number of these simple gates required is exponential (in the number of qubits) on average, and thus we can say that an average complexity for a random gate is exponential. Thus, there can be a big difference between some two multi-qubit gates. But this is just from one perspective.
In reality a quantum gate is a physical device. Why, from physics perspective, one quantum gate should be more complex than another one?
As an analogy, think of the distinction between classical CPU and specialized hardware, like FPGA, ASIC, GPU, TPU, LPU, etc. Specialized hardware is much faster for specific tasks, while they are not supposed to be universal computing devices like CPUs.
So, would it be possible for a random quantum gate to physically construct a device that will compute it with a speed comparable to, say, the computation of the Walsh-Hadamard gate?
Perhaps, this question sounds more reasonable when we have just one quantum system with $d$ degrees of freedom and look for a complexity of a random unitary acting on it.