The "Chance of all tests passing" at the CPAN dependencies and test results checker is not a scientific measure - but it is great in making it visible that installing from CPAN is still a game of chance with quite big odds of failure. The linked above Catalyst dependencies page quotes the 'probability' of installation failure as 22%, and that is for the most advertised Perl web framework. The important thing in that picture is how this relatively big failure chance comes from the individual small failure chances of the Catalyst prerequisites.
So what can we do? One thing that I would propose is to stop using optional features and dependencies because they really get into the way of automatic installs. We can also simplify the dependency graph of some more important modules, make better statistics about the failure rates and they origins, but probably the most effective way will be to be just more vigilant about the test failures in our modules and in the prerequisites. The important lesson here is that a 1% failure rate seems like nothing important - but it can make our module inadequate to become a prerequisite for some more complex library.