I've seen derivations of Lorentz transformation but are they really derivations or are we just teasing out the formula using some special cases and then assuming it to be valid for all the cases.
Is this really a derivation as in the mathematical sense or are Lorentz transformations a law in themselves apart from conservation of the spacetime metric.
By Lorentz transformation, I mean the relation between space and time 'intervals' for two inertial observers.
This is dangerously close to being a matter of opinion, but for what it's worth I would regard the Lorentz transformations as a special case rather than anything fundamental.
The fundamental principle in special relativity is the invariance of the proper time, which is presumably what you mean by conservation of the spacetime metric. The Lorentz transformations give you the coordinate transformations for the special case of relative motion at constant velocity. However in any system where the velocity is changing the Lorentz transformations don't apply.
In the past this has lead to the common claim that special relativity doesn't apply to accelerating systems. This is of course nonsense as the invariance of the proper time can be used to describe motion in a circle or accelerated motion in a straight line, or indeed any arbitrarily accelerated motion.
My point is that the Lorentz transformations describe only a subset of the motions that can be described using special relativity, and therefore they can't be a fundamental law.
I believe you are thinking of the Lorentz velocity boost (and rotation) transformations as the 4x4 matrices which leave invariant the metric diag(1,-1,-1,-1) (ie: they leave invariant the proper time). In this sense, perhaps they seem secondary to the SR principle of invariance of the proper time.
However, these 4x4 matrices are just one representation (called the fundamental rep) of the Lorentz Group O(1,3) or its covering group SL(2,C). There are many other representations by matrices of different dimensions. EVERY particle or object in the universe transforms under rotations and boosts as some dimensional representation of the Lorentz group. You are probably most familiar with this for rotations (subgroup of the Lorentz group). Here each different dimensional representation is labelled by the spin of the particle being rotated. For example, a spin j=1/2 particle has (2j+1)=2 states and is rotated by 2x2 matrices. There is nothing more fundamental in physics than this. Some purely abstract math SL(2,C), can be put in a correspondence with how every particle in the universe must behave! This isn't even an equation. There was no formula to tease out. For example, there was no equation to pull out of a hat by setting curvature equal to stress-energy density because it gave Newton's law in the limit.
Whereas understanding how every object in the universe transforms under rotations and boosts (a major part of quantum mechanics) is a fundamental big deal, the Lorentz group may be just the tip of the group iceberg. Rotations and boosts are not the only continuous transformations that can be done to an object. We also can do strain, space translations, and time translations. The Poincare group extends the Lorentz group by adding abelian translations. If a different group were used in which translations did not commute with each other, perhaps it would be more interesting. Raising and lowering operators would be available and much like non-commuting rotations provided the quantization of angular momentum, non-commuting translations would provide the quantization of mass. Yes, Lorentz transformations are very fundamental for physics.
Is this really a derivation as in the mathematical sense or are Lorentz transformations a law in themselves apart from conservation of the spacetime metric.
The Lorentz transformations do not form a law in themselves, but are derived mathematically from the conservation of the Minkowski metric $\eta$. In the derivation, one considers which transformations would leave this special metric invariant, specifically:
$X \cdot X = X^\mathrm{T} \eta X = {X'}^\mathrm{T} \eta {X'}$
where $X'$ was transformed by transformation $\Lambda$:
$X' = \Lambda X$
The question then arises as to exactly what form this $\Lambda$ would have to take in order for $\eta$ to remain invariant. The derivation comes to the conclusion that $\Lambda$ must be part of the Lorentz group for the above condition to apply. (further information on this can be found, for example, in the Wikipedia article quite far down).
This Lorentz group of possible transformations also allows time or space inversions, however. The Lorentz boosts you are referring to form a subgroup of these possible transformations if you omit such inversions (called proper Lorentz transformations).
As explained by the previous answers, however, these transformations only apply between unaccelerated systems and are not general in this sense. Their special meaning comes from the fact that it is assumed that two unaccelerated systems should be physically indistinguishable and thus all physical laws/equations should be invariant under Lorentz transformations. When comparing (more general) accelerated systems, one expects the physics to change during the transformation, and thus the laws of physics may also change (e.g. due to an added centripetal force).
In fact, this is closely related to rotations in Euclidean space. In classical mechanics with Galilean transformations, one of the requirements is the isotropy of space: i.e. the laws of physics should be formulated in such a way that they are invariant, regardless of how the experiment is rotated/oriented.
