Academia - What to do with questions asking to evaluate commercial online services?
Historically, this community has had a strong policy of staying away from assessments of (most) specific organizations, commercial or otherwise. I believe that this is a good policy for several reasons:
- Assessments are often highly relative and based on perspective
- There is a temptation for advocacy, whether for personal ("Go Tech! Beat State!") or financial ("Buy our widgets!") reasons.
- Complementarily, people are likely to become upset if others place harsh judgement on an organization that is important to them.
- Many organizations (especially new entrants to a field) will change quickly in their nature, impact, and significance, and answers will tend to become rapidly obsolete.
- Allowing any assessments of organizations opens the door to a potentially unbounded flood of requests to assess other organizations.
I think that this should apply no matter how large the organization, when the question is about assessing the organization.
Instead, I notice that most well-regarded questions about organizations seem to fall into two classes:
Questions about established places like LinkedIn, ResearchGate, Facebook, Google Scholar, IEEE, etc., which already assume that organization is notable and legitimate but which are asking advice about how to manage some aspect of one's interactions with it with respect to some aspect of academia. Thus, we take the organization for granted and ask for experts in it to share their experiences.
Questions about whether to trust a possibly sketchy organization. These are often something that can be generalized to a class of organizations, like how to assess whether a conference or journal is predatory.
In neither of these cases do we need to assess an individual organization, and thus we avoid the tar-pit of associated problems.
Thus, if a question cannot be edited into one of these two classes, I think that it should be closed. I think the Academia.edu question might be able to be turned into a question in the first class, but I don't think the Peer.us question can be.
TL;DR: Evaluating an organization is not OK. Evaluating a class of organizations is OK.