>>535065057
It's actually quite hard to get it unless you're looking for it. There's 9 separate times you get to dive in the spirit world, and if you don't collect the memories, the next day they will still be available. It only takes opening the map once in the 9 chances you get to see there are more memories than the obvious ones. Even if you miss all of them, as long as you find one on the last day, you can collect them all right there.
You'd have to be completely blind and never check the map, not even out of curiosity for what Ruka's hometown looks like, on 9 distinct days in order to accidentally trigger the bad ending. I can only imagine how funny it would be if that happens to someone though.