First guy to hit the field was Matt Raible (1 2 3 4), ending up with Drupal, Joomla, Magnolia, OpenCms and MeshCMS being runner-ups.
Then there is OpenAdvantage that tries out a handful (Drupal, Exponent CMS, Lenya, Mambo, and Silva), including Plone which they use for their own site (funny/annoying that the entire site has no RSS-feeds, nor is it possible to comment on the articles), following Matt's approach by exluding many CMS that seem not to fit the criteria. It is somewhat strange that OpenAdvantage cuts away Magnolia because it "Requires J2EE server; difficult to install and configure; more of a framework than CMS", and proceed to include Apache Lenya in the full evaluation. Magnolia does not require a J2EE server. It runs on Tomcat just like Lenya does (maybe it's an idea to bundle Magnolia with Jetty to make it seem more lightweight). I'm still sure that OpenAdvantage would 'fail' Magnolia for being too complicated as Matt did.
All websites have a different need, and these two evaluators value ease of setup, use and design-modification (not functional modification). A more enterprise-ish review has been done by Optaros, evaluating for different website needs (brochure, periodical, collaboration, wiki and community). Elegant observation:
Open source content management software is most frequently used in small to medium sized web sites with very common requirements (such as corporate identity websites and departmental intranet sites or online magazines rather than large product websites with hundreds of thousands of pages) and as a foundation for building unique, highly-customized solutions (such as Amazon.com which uses open source components such as Perl, MySQL, and the Mason templating engine).The paper provides an in-depth evaluation of three or four CMS'es in each of the five categories. I am left with the feeling that the landscape of Java-CMS'es are very far behind the other ones, but still I would prefer to work with a Java-based CMS as it is my language of choice, and because I've fallen pretty much in love the the Java Content Repository. Nonetheless, the paper is an excellent starting point for a small or medium-sized business considering an open source CMS. I wonder if there is a common content repository interface (not Java-dependant). Jackrabbit has (or are on their way to) implemented a PHP-interface for their repository (but still no .net, perl, ruby and python interfaces). In the mean time the closest you get to platform independant content is a database (which is not so good).