Skewed Ratings

Cyxult · Apr 30, 2007, 10:24 PM

Before you start reading, I want you to know, this is not a hardcore rant, it's more of an open discussion, and hopefully a way to start helping RMS become a more credible source of RO server ratings.

Am I the only person that seems to notice the abundance of 100/100 ratings on server ratings, or the lack of decent ratings on a server? It seems that RMS is used by people for one of the following two reasons. One, the server owner is paying players with in-game rewards for 100 ratings, which gives people a skewed perception of how "good" this server actually is. Two, people are using RMS as a way to exact revenge on a server owner, because they were banned. I know that the administrators of RMS do their best to weed out those people that abuse the system to make a server look bad, but what about the load of 100/100 ratings? I get the feeling that a lot of people rate these servers for reasons other than informing others about how it really is.

Don't get me wrong here, I love RMS, and I think it's one of the most inventive creations the private RO community has for voting and advertisement, but I think that the rating system needs to be tweaked, or the rules need to be adjusted. I honestly do not believe that there is a perfect 100/100 server out there, no matter what anyone thinks. Anyone have any sort of ideas to make the rating system more credible? Perhaps, certified people that have their ratings stick out from the others, or maybe some way to enforce rules upon people who pay their members off for good ratings...

I'm just throwing ideas out there, I hope nobody takes any offense. If you have any thoughts/ideas, feel free to respond.

EDIT: While browsing, I got mixed up, wrong section... please move it to the according place (suggestions).

Jayne Cobb · May 01, 2007, 01:25 AM

like a "featured reviewer"?

Max · May 01, 2007, 02:35 AM

Secretly delete all 100/100 ratings every few weeks?

Pandora · May 01, 2007, 08:36 AM

I would tend to agree actually. I've seen some server get 100 on their opening day, as if class balance, economy, guild, etc. could really deserve 10/10 after so little time.

I think some players give 100 because they like the server they play, not for any reward in particular, probably to attrack new players to their server. At least that's how I see it, my server has gotten some 100 before, and I don't give any reward to motivate people to give high score. I don't think any server deserve 100 either, there's always room for improvement.

yC · May 01, 2007, 12:33 PM

yes Cyxult i am fully aware of the situation and the problem. Yet there isn't much can be done because it should be a "fair" system so i have to be fair on my part. It might be the age group of the users that ... i shouldn't say more to make my users look bad.

So yea, suggestion is welcome but no guarantee that it can be done.

simply we can't remove all the 100/100 reviews lol, nor does all the 10/100 not deserve their scores.

Cyxult · May 01, 2007, 08:26 PM

Quote from: yC on May 01, 2007, 12:33 PM
yes Cyxult i am fully aware of the situation and the problem. Yet there isn't much can be done because it should be a "fair" system so i have to be fair on my part. It might be the age group of the users that ... i shouldn't say more to make my users look bad.

So yea, suggestion is welcome but no guarantee that it can be done.

simply we can't remove all the 100/100 reviews lol, nor does all the 10/100 not deserve their scores.

That's true, and for the record, I wasn't expecting some sort of dramatic over-night change.

Perhaps, a reviewing guideline should pop up when someone clicks the "Write a Review" button. Maybe that would help deter the stream of insanely high ratings?

Max · May 02, 2007, 05:13 AM

Maybe create a hidden rating which excludes all 100/100 and 10/100 scores.

Glenn · May 03, 2007, 05:33 PM

Ok _M_; let's be honest putting in a hidden exclusion system wouldn't work. People would find out and then the next thing we will see is 99's/11's

Secondly, I've thought of 2 ways to help make RMS more accurate. I didn't say these are EASY ways but, bare with me.

1. Allow other users to rate reviews, if they found them helpful or not. Then only use the top 10 highest rated(not rated rated, but rated by other players) reviews. So like say your reading a servers reviews and you see one like "zomg this server suxxoz" You would probably rate that review a 1. That would average with everyone elses ratings for that review, and it would probably eventually drop off the top 10 reviews. (like I said only have the top10 rated reviews goto the server score)

2. Have a team of RMS reviewers that you trust.(ie. Me) Go around rating servers to give them an 'RMS Review' and have a seperate score that players cant alter. The you can put both scores in a list. Sort of like one of those Expert Review/ Player review systems.

Two decent ideas that could ultimatly make reviewing more accurate.

Pandora · May 04, 2007, 08:40 AM

Method 1 brings us to the same problem, that is ppl giving 10/10 to reviews they like and 1/10 to reviews they dont like.

Method 2 sounds better but harder to do.

All in all it will not be possible to get anything 100% fair, I think people can judge for themselves by reading the text of the review, like if the 100/100 review reads "omg tihs servar is sooo leet!!!!11!!!1!"....the best way will still be to check it out yourself, after all RMS is there to help you find a server but the best is still to try it and forge your own opinion.

Glenn · May 04, 2007, 04:30 PM

Well for one you would have it so people can't rate thier own review, and they can only rate it once.

More then likely people will only rate a review good, if it's helpful or inciting. You don't even neccisarily need a 'rating' for it. Perhaps somehting like a did you find this review helpful question with 'Yes/No' The ones with the best yes to no ratio goes to the top of the list. It's like so other players can review the reviews. I mean people aren't gonna vote yes for "This server suxxorz". Voting once was implied.

bulbasteve · Jul 11, 2007, 02:42 PM

Well there is another aspect of how ratings are skewed, and that is in terms of the actual things that it is rated on. If it is a no-GM server then what do you give it, a 10? a 0? If there is a shopping mall for items and as such the game isn't about an economy then what would you rate that? Heck if the server isn't a about guild wars but totally about PvM what do you rate it? Or if it is the "SinX is best" server? So the skewing isn't just people who rate 100 or not 10 but people who because of the way ratings work give it a 60 or 70, where if they were to give their own overall rating would probably give it a 80 or 90. Or a bad review where they really feel that overall the game should get a 10 but have to rate it higher because just because the GMs suck and it's a horrible game you can't argue that it is up 24/7

Quote from: Cyxult on Apr 30, 2007, 10:24 PM
Anyone have any sort of ideas to make the rating system more credible? Perhaps, certified people that have their ratings stick out from the others, or maybe some way to enforce rules upon people who pay their members off for good ratings...

I'm just throwing ideas out there, I hope nobody takes any offense. If you have any thoughts/ideas, feel free to respond.

I suppose one could have a seperate "editorial reviews" section, where members of the RMS staff go to various servers and give their own in-depth review of the server, which could certainly give people are more reliable review and help out other smaller servers who don't pay people to review and sorta brute force their way to the top of the lists.

Flip · Jul 11, 2007, 03:47 PM

I agree with the editorials on servers, but it'd definitely take their toll on the RMS staff.

For one thing, just how many servers are out there? Lol. :p

Then again, maybe we could get volunteers or something o.o;

Mieu · Jul 11, 2007, 04:31 PM

Or readers could like, read the reviews, disregard the 100/100 or like, 99/100 ones, and use the nifty drop-down feature that breaks down exactly why the user rated the server the way they did. If it seems credible that a 'new' server get rated a 50 or so on their economy or WoE competition, and that other values seem logical, one can most likely take that person's review to heart.

If you can take the time to read each individual testimonial, then you definitely have enough time to browse the detailed rating system that RMS already has. Just as there is an overabundance of people who make unfair or untruthful ratings, there exists the same number of server admins that will defend themselves against a 'bad,' yet truthful rating until they're blue in the face.

The old saying 'caveat emptor' still works, 'let the buyer beware.' There is still no better way to know the quality of a server than to play on it for a few days. You may find a diamond in the rough, or a wolf in sheep's clothing.

yC · Jul 11, 2007, 05:40 PM

i could just be lazy (like always) and do it the hmm Olympic way ... for every 10 reviews remove the highest and lowest ... do nothing for those that has less than 10 reviews ... so if u have 20 reviews 2xhighest and 2xlowest will be stripped.

(stripped as in not count the score but still show the review)

Down side is ... spammer has more reason to make more accounts to spam ... i am already sick of ppl spamming good/bad reviews why can't they just do by the rule and make one review only.

Slaw · Jul 11, 2007, 05:44 PM

I believe the idea of volunteers taking care of it is a good idea. Only problem is finding people and make sure they're being fair. Hence I present an "improvement" of sorts to the idea; Why not put a team of 1-3 people in charge of recruiting and moderating a team of review moderators? Wouldn't really be more work for the admins, And I believe quite a bunch of the people on this forum would enjoy helping out.