A good deep learning architecture is one that introduces correct priors about the problem at hand -- in particular, about the structure of the correlations found in the data.
For instance, convolution is preferable to dense layers for data dimensions that are locally autocorrelated and translation-invariant. And depthwise separable convolutions are superior to convolution for any data that is locally autocorrelated and where channels are decorrelated.
(Which is naturally the case past the first layer of any convnet.)
