In this article here, the writer claims that a new type of neural net is required to deal with data that is both continuous, and also sparsely sampled.
It was my understanding that this was the entire purpose of techniques that use neural nets, to make assumptions about a system with a non-continuous data set.
So why do we need to switch to a non-layered design to deal with these data sets better?
They struggle because if your network have an inductive bias towards modeling datasets which are described with ODEs well, you will learn faster, and with smaller dataset. I think, this what the authors of the original article meant.
In a similar way, CNNs recognize images much better, because their features are translation invariant whereas fully connected net needs to learn to recognize a cat in each different position from scratch.
