There is never a double-slit interference pattern in the total pattern of signal photons at D0, such a pattern at D0 can only be found by taking a coincidence count with some detector that detected the idler photons. See my answer here for details--I think your question is basically a duplicate of that one.
Incorporating some discussion in the comments into this answer, I'll add that complementarity between which-path information and interference (defined formally in a 1979 paper by Wootters and Zurek, the paper itself is not online but see this pdf for a discussion) only applies to the coincidence count between the signal photon and idlers that have been detected at a particular location (like detectors D3 or D4 in the diagram from my answer above). There's never double-slit interference in the total probabilities for the signal photons to land at different points on the screen (i.e. the probabilities you'd calculate if you don't know what happened to the idler, see this answer for an elaboration if it's not clear) if they have been entangled with idlers that could even potentially have been measured in a way that would determine the which-path information of the signal photon, regardless of what actually happens to the idlers. See for example the last page of "Induced Coherence and Indistinguishability in Optical Interference" by Zou et al., available as a pdf here, which says:
The disappearance of the interference pattern here is ... a
consequence of the fact that the two possible photon paths ... have
become distinguishable ... Whether or not this auxiliary measurement
... is actually made ... appears to make no difference. It is
sufficient that it could be made, and that the photon path would then
be identifiable, in principle, for the interference to be wiped out.
Likewise, page S275 of the paper "Quantum effects in one-photon and two photon-interference" by L. Mandel, online here, says:
If the different possible photon paths from source to detector are
indistinguishable, then we have to add the corresponding probability
amplitudes before squaring to obtain the probability. This results in
interference terms as in Eq.(3). On the other hand, if there is some
way, even in principle, of distinguishing between the possible photon
paths, then the corresponding probabilities have to be added and there
is no interference.
And page S279 discusses a particular example, and says (emphasis mine):
However, when $i_1$ is blocked, $D_i$ provides information about the
source of the signal photon detected by $D_s$.For example, if the
detection of a signal photon by $D_s$ is accompanied by the
simultaneous detection of an idler photon by $D_i$, a glance at Fig. 6
shows immediately that the signal photon (and the idler) must have
come from NL2. On the other hand, if the detection of a signal photon
by $D_s$ is not accompanied by the simultaneous detection of an idler
by $D_i$, then the signal photon cannot have come from NL2 and must
have originated in NL1.
...
Needless to say, it is not necessary actually to carry out the
auxiliary measurement with $D_i$; the mere possibility, in principle,
that such a measurement could determine the source of the signal
photon is sufficient to kill the interference of $s_1$ and $s_2$.
