I will have to choose some sort of formal process for comparing information systems (more specifically, CMS-es). The only real one I've encountered so far is the Analytical Hierarchy Process (AHP), as mentioned in on of the earlier article reviews. It seems a bit heavy, but there are alot of factors inside that I like, and it is also good for comparing few systems (pair-wise), so I can use that as an excuse for comparing fewer systems ;)
Some googling showed that there are lots of research which have been done with the aid of AHP. There are also many other frameworks or methods (decision support tools) that can be used which I might have to take a look at before settling with AHP.
I also have to choose what I want to compare. Let's say I have to pick four CMS'es. Needless to say, these would have to be of the same level, not some enterprise big thingie, but rather a couple of open source ones and a couple of proprietary smaller ones. I allready have one proprietary (Primetime Portal, or Primetime 2), so I might have to find another.
Perhaps one of the ugly-license-OSS will do..