Any answer to my questions may assume arbitrary knowledge of differential geometry, which I will be happy to learn in order to understand the most appropriate type of formulation for the theory.
I would like to consider Special Relativity as the theory of a general Lorentz manifold $(M,g)$ where $M$ is diffeomorphic to (the standard diff. structure of) $\mathbb{R}^4$ and the metric (of constant Lorentzian signature $(-+++)$) is constant over all of $M$ in the coordinate-free sense. I do not, however, want to assume that $M=\mathbb{R}^4$ and that $g=(\text{usual matrix})$ because I find it unphysical to apply "linear transformations" to positions in spacetime.
In particular, I demand that (for this discussion) the theory be formulated with arbitrary smooth coordinates (in the sense of differential geometry) where we can of course choose global coordinates, but no single coordinate system is in any sense "more fundamental" than another or "a priori innertial". This is as opposed to the case of $\mathbb{R}^4$, where one chooses as innertial the coordinate system $id$, which returns positions in spacetime as the tuples of numbers that they are, together with any other coordinate system which is connected to $id$ by a Poincarré transformation.
I interpret physically the manifold as the set of all events in spacetime, and the metric as designating causality; in particular the metric dictates what (smooth) curves in $M$ are space-, time-, and lightlike. In order to specify past and future we assume the existence of a global vector field $X\in TM$, which is defined to be future-directed at each point. Together $X$ and $g$ specify the past and future light cones for each point $p\in M$ as subsets of $M$ (which is not considered as a vectorspace) as well as all(?!) other concepts relevant to causality, such as Cauchy surfaces and the like.
My questions:
Question 1. In the next step of this theoretical setup, which is not completely clear to me, one defines a linear action of the Lorentz group$^1$ (I don't think Poincarré makes sense here?!) on each tangent space $T_pM$. How does one set up this representation and justify physically why both the group and the chosen representation are meaningfull and the correct choice?
$ $
Question 2. Is it possible with the above prerequisites to meaningfully define when a diffeomorphism $x:M\to\mathbb{R}^4$ is an "innertial coordinate system"? Would this be equivalent to requiring that all "$g$-Levi-Civita-connection-straight" curves be mapped by $x$ to straight curves in $\mathbb{R}^4$?
$ $
Question 3. Assuming the above makes sense, what is the connection of the Poincarré group with coordinate changes between innertial coordinate systems?
Footnotes:
$^1$ I consider all these groups just as mathematical objects - Lie groups with a priori no physical significance. I want to include the physical significance by arguing why these groups with their appropriate actions on the appropriate objects are physically meaningfull.