Typically "movement" is the first cause of those type of grout issues. The very fact that grout when installed sorta locks itself into the cavaties and stays in place is what would make it hard to swallow that it is simply falling out in a matter of days. Just falling out really isn't all that possible.
What brand/type of grout are we talking about? Did they use Polyblend.
Grout improperly mixed by using too much water will ruin grout and cause it to erode when the showwer is being used. This will happen immediately or within days.
Have you seen these "big pieces"?
You are correct that not sealing the grout has absolutely nothing to do with the grout falling out so soon. I can assure you this is going to be installer error if Polyblend (or any Portland grout) was used.
