Proposal to improve the rating system

[yC]

Okay, the same sets of servers are always at the top and nothing change for months and not likely to change in the coming months.

I propose to:

1)  Review will valid for a set amount of time [currently thinking 120 days = quarterly TO/OR 180 days = semi-annually].  After the valid period ends, the review will be on an archive section.  That means it will not count toward the score and the ranking but will be visible at another page linked from the server's listing or simply at the end of all the currently valid listing.

Reason for change:
Simply because server changes from good to bad and bad to good over time, a server can change a lot in some 3 months to/or 6 months period.  Thus review shouldn't be valid for years.  If the user want to re-rate the server of course they can delete their archived review and write a new one.

2)  With respect to 1), server with no active review (meaning all reviews are archived or no review for the set period given) will be moved to an archived server list.  This should help a little in keeping the list from dead (+ rotten) server.  When an archived server gets a new review it will be bumped back to the normal listing.

3)  Increase the space allowed for review, not a lot, maybe up to 300-350 characters.  However more strict rules are put on reviews (it's already in actually).  No more "best server ever!" 100/100 -- think about it, how often do you see people give reviews like "best keyboard ever!", "this monitor rocks!" etc on amazon or bestbuy.  They do not tell the reader anything informative really.

4)  Select server by language (I wish ...) but might take longer to do I'll just note it here.

5)  The banner resizing thing skotlex suggested -- do not resize banner if size is less than limit, noting it here as well.

6)  I heard about the rate high rate low a lot that suggest how to fight the 10/100 spam, i have 2 suggestion on this ... it's gonna take a long time and i am lazy but w/e just noting:

6a) Use the Olympic standard, make the highest and lowest for every 10 reviews won't count toward the total score and ranking.  But are spammer not going to give up easily?

6b) IF 1) is going to get implemented, make each user can rate review up or down.  Each up will add x days to the life of the review, each down will minus x days to the life of the review ... damn the edit/delete part will be hell for me ... unless after u add/minus u can't change it ever

7) With all these changes we need a long due FAQ page to make ppl understand the system.

** 8 ) IF all of them are going to be implemented I can see a lot more queries / checking usage meaning I will need to move to a more powerful server ... $ ... the current serer load is above 1 and below 2 at non-peak time and go beyond 2 at peak hours =/

Any other suggestion to the server listing section? 

Do we really want to ask the average amount of players the servers has?  You know ... it might be an embarrassing question and kids will put 134328934023843 wouldn't they? ... that's why i hesitate to add that question ...

Lastly, please comment.

[bulbasteve]

Ok I'll bite.

1. The problem with this is it seems to be targetted towards large servers. Small servers which may be active but just not tell their users to go review the site would be hurt by this and find themselves back down in ranking. If this were to be the case I would say it would have to be only for servers with enough reviews to justify clearing them every so often. Which basically just means the 60-70+ reviewed servers.

2. Same reasoning as number 1 for me. There is no reason to code in some archive system when you guys can just do it by hand. Just because your server is small and doesn't review often doesn't mean it is dead. If the website is down and you can't get any information about the server put it on a probationary page and if in a month or so it is still down kick it over to a dead server list. And I guess pop an email to the server creator and put a "revive" option for a creator if the reports of their deaths were premature.

3. More space is good.

4. Are non-english servers even on here? But yeah good idea.

5. Of course.

6a. Again, only for higher reviewed servers. I really don't want to see unpopular servers screwed over.

6b. Well I don't like the whole lifespan idea anyway. But even the general idea worries me because you are going to just going to just change review spam to review rating spam. Just another flavor of the same crappy practice by these guys.

7. Heck even without it you need one!

8. uuuh..yeah, guess you would.

9?. Don't add the average number of users. It changes with time and you don't want a server owner to have to go in every month to change the number. And agian it's all about the little guy and you don't want to scare people away from a server just cause they see a low number. Not to mention like you said the whole lying thing...

Other suggestions, well I still think there should be some sort of "featured reviews" page. Since not everything is about how nice the gms are and how good the economy is but have people (staff or users) write some in-depth reviews about servers which they feel have interesting features that set it apart from the rest.

As well I posted before I think there should either be a "tilt" system to offset bad marks or even have it so the server creator can decide to take off certain things from the rankings because they don't apply to them. Some servers have no GMs/Events/Guild Competiton/Economy as a integral part of the server. So It's pretty bad to impose an artificial standard of what makes a server good or not. The reason these 100 and 10 votes are so bad isn't so much that someone thinks it is the best/worst server ever but because they have to lie about certain aspects of the server to get to that position.

Certainly you should either add a donation or no donation option in the server listing and hell probably just totally separate donation from non-donation servers and give them two totally different lists (each also having the 4 separate rate listings), and have in the donation servers an option if its donation system is no-reward, no-stat equip or stat items to further clarify the type of donation. This is obviously the biggest and most controversial part about private servers and probably as important to a player joining a server as even the exp rate. And as a side bonus it will make the individual lists smaller which helps the little guy, which is always a plus. And hey going through all the servers to figure out if they are donation or not is also a good way to start weeding out the dead ones

Latest reviews should probably not be the only way that something appears on the front page (heck it probably shouldn't show the server names at all there and just keep it as a link on the side to list them all), I think it just gives people one more reason to review spam so there name is on the front page. But in its place I guesss have a little section be a randomly generated server description of something.

[BRabbit]

Add the possibility of "not ranking" some parts. Like if someone hasn't gone into WoE yet but likes/hates the server and wants to rate, he can leave Guild Competition "blank".

(from previous post, worded differently but I agree with him) At the list of server features "healer" "resetter" etc. you could add "Donation rewards" with the choices "Yes" / "No" / "Not Unbalancing". The third would mean that the server does not give MVP cards, equips with great stats for donations. Like you have a fighting chance without donating. Name changes, guild leader changes etc. would count as "not unbalancing". Items that make a character look cooler, but doesn't give stat bonuses, can also be considered not unbalancing.
You may also forget about the third option and just put in a Yes/No option, like there's on the others.

On the "Default" server listing, make the amount of reviews matter less. AnimaRO is the lowest rated low rate server, but it has the most reviews. It's on 5th place in the LR listing. That's totally wrong. The formula needs some improvement.

Like mentioned, increase the space. Maybe even more, up to 400 characters. There's so many things to rate. I tried to make the best review I could, but the space wasn't enough.

You don't need to add the amount of players on the server. People looking for popular servers can go to a server's website and click on their Control Panel. Most popular servers have the amount of users in there, even if you don't log in.

[bulbasteve]

Seems you have done me one better on every point

Your right that not ranking would be much better than tilting or server creators not having things up for rating. If your server has no WoE than you may as well just have the people reviewing it leave guild competition blank instead of having to make some new complicated system.

Though I guess the natural aftereffect of this is that there would probably have to be a one time edit option for rankings (though I suppose not the text of the review).

Although I have to say I still like the idea of giving the servers some breathing room by making them a seperate list for donation the idea of making them another check box is also much better. Though I think the wording should be different, I don't know any server owner who would admit that their rewards are unbalanced :p

And to a certain extent, maybe I'm just cynical, but simply allowing donations I think should be included here. Because there seems to be a lot of servers, while they don't do rewards, will say "hey I need x number of dollars to pay for the server by the end of the month or the server is history". Which really can be just as bad for a server as some godly kaho horns. Although I guess obviously it wouldn't need to be another check box if having donations itself made it appear on a seperate list...

While the formula does need improvement, I think it only should be done if the "not ranking" part is added. Because like I said some servers are missranked just because of what we rank on, and in that case then sheer number is probably a good thing to be be weighed more highly. But obviously either way, Anima should be way down on page 2, not in the top 5, that's just crazy.

[Glenn]

1. I don't think it's really neccisary.. and think it would also hurt smaller off servers

2. Same as 1, won't effect large servers, will effect small ones.. wich aren't neccicarily bad/dead.. just new~

3. Do this! @_@

4. I don't think it would be EXTREMELY usefull, but it wouldn't hurt.

5. mhmmm

6. Ok I'll bring up what I suggested a while back

1. Allow other users to rate[+/-] reviews, if they found them helpful or not. Then only use the top 5 or 10 highest rated[+/-] reviews in the overall score. So like say your reading a servers reviews and you see one like "zomg this server suxxoz" You would probably rate that review a [-]. That would average with everyone elses ratings for that review, and it would probably eventually drop off the top 5/10 reviews. (like I said only have the top5/10 rated reviews goto the server score)

I think this would kind of go with number 1, while not hurting servers

7. How well did our last FAQ go? XD j/k ... I'm sure we'd be able to muster one up.

8. maybe?

9?) average isn't consistant enough...

@BRabbit
Yes at the Not ranking, some of the categorys don't apply to servers. (ie. roleplaying servers don't have woe, economy is low on new servers) Though, would people abuse this and leave them all blank(perfect score?) I guess that would make it a [-] for my system though.

@bulbasteve

I agree with donator/non-donator option. (There's allot more servers without donations then people think @BRabbit)

[Skotlex]

My two cents regarding the review-score:
- As others said, let any given field be 0 or N/A, not 1 (worst), but 0 (non-existant, does not care about it)
- Add another rating dimension: The Heart. In other words, this should be the single most important (and biased) review dimension. Basicly it is "how much you like the server?" While all other dimensions can be measured objectively, this is the one that should sum up how the player actually feels about the server.
- I would be inclined to say that then, overall server rating should be based on the heart alone. You could use a weight system where more recent reviews have a higher weight than the older ones, that is true. I do wonder how would you manage then to prevent a new server with 1 review which is 100% heart to go to the top... a server should not only be liked the most, it should also have some history to confirm its stability. Maybe rate based on the longest string of positive reviews? I don't know, just throwing ideas out here. I do believe that the most important rating factor should be The Heart, though (people should be able to manually pick the dimension by which the list or ordered, though, I am just speaking about the default method).

PS: I don't agree with removing/archiving old reviews, on a long-living server they would give players an idea of how the server has changed with time, and whether it is for the worst or the better. If the player is smart enough to read the reviews, he would also be smart enough to realize the reviews he/she/it is reading are old! If you want to help with that, change the color scheme so it becomes a darker color the more years old the review is so it becomes apparent on sight if the review you are reading is very old or not.

[Glenn]

My top 5 suggestion, I didn't mean to take them off completely. I just meant the top 5 pointed reviews would lead to the overall score, other reviews would remain.. just wouldn't effect the score unless they bumped off another review by getting more total 'points' then it. Clicking + would add 1 point, and - would subtract a point. Thought I should explain my idea a little more.

[yC]

1+2) It's not just for large server.  If a server that don't have a single review every 3-6 months or even one review to begin with.  Can we assume it's dead already or the server is not interested in using our service anymore?  Thus we push them to the other list until it give some care (of course we'll email them when we push them away).  I think it's a good solution to get rid of dead server.  Otherwise we (just me actually ><) will always have to hand pick dead server it's not efficient ...

As I said we need to have a more competitive ranking, not always the same server on the same spot.  Because as of now, if one server spam enough they will be in the front page forever, now that's not fair to small server as well. 

What else is there to accomplish these?
1) make list more competitive overtime
2) get rid of dead server fast


Other:

The least blank idea is OKAY but ... i have to think how to do this ... need to change a lot of things.  so if they don't want to rate it ... it will be n/a and then how we store the score??? make it out of 90 80 70 etc? and take the %?  Then we need to have a minimum, maybe pick at least 5 to rate because if they n/a everything it will be 0/0 and what is this score? 0? need to think more on this...

"Though I guess the natural aftereffect of this is that there would probably have to be a one time edit option for rankings (though I suppose not the text of the review)."

- ahhh don't make me do this ... i think they can just delete and rewrite come on ...


The "Donation reward" option

- i think is too detailed, options only have true/false values.  Plus a new server (always) start with no donation then add them later once they lure enough players. ... and yea ... "I don't know any server owner who would admit that their rewards are unbalanced" ... even anima say they are balanced ... lol


"On the "Default" server listing, make the amount of reviews matter less."

-Well, make the amount of review matter less means make the score matter more.  From the beginning i know if more weight is put into the score than servers spam themselves with all 100 will be at the top even the server only has a few people ... that's why i introduce #1 and #2 to lower the effect of large server getting more reviews over time making them at the top forever.


Glenn:
"1. Allow other users to rate[+/-] reviews, if they found them helpful or not. Then only use the top 5 or 10 highest rated[+/-] reviews in the overall score. So like say your reading a servers reviews and you see one like "zomg this server suxxoz" You would probably rate that review a [-]. That would average with everyone elses ratings for that review, and it would probably eventually drop off the top 5/10 reviews. (like I said only have the top5/10 rated reviews goto the server score)"

- really don't know how this can be of help.  I mean who's going to rate the reviews ... it's the same bunch of people.  Or if there's no one rating any then the effect will be the same as now.  More detail?  ... and damn another complicated system please don't try to kill me i am lazy i think ways that are easier on the coding side

Skotlex:
- dont know where the "heart" can fit ... thinking
- old reviews aren't going to be removed, using a different color skim is nice idea if I want to have them on the same page, just not counting score and ranking.  Now i think i could have the same user rate the server again after the user's first review expired without deleting the first review.  In order to keep the historical record.


Major problem remain:
- how to deal with spam, low rater aiming to hurt score, high rater cheat.  But these are daily

routine don't worry about them.

[bulbasteve]

1+2: But the "service" for the game creator is just to list their server and that is all, the service for the game player is to rate them. There are different account types after all. It really isn't the server creators job to get people to review. Their only responsiblity according to the user agreement was to was to keep their server up. I say just make a dead server thread on the forums, I'm sure the users will be more than happy to post a server they find that is dead (heck give them good karma or whatever wonky system we got here).

I don't think there is any way aside from a total revamp to make them really competative. I mean big servers will always be on top, even if you revamp the system to get anima off the front page that doesn't mean that a small server will ever be number 1, it's always going to be a large donation server that frankly already gets more than enough traffic from just having a large user base to spread the word, paying for banners and being on the top of voting sites too. Ok...maybe I'm just bitter

I guess forcing to pick 5 will work. Though I gotta say it's cutting it close, a no GM server will have 3 NAs by default, and probably events as a NA too, not to mention anyother unique NAs they might have.  But yeah just do it as a percentage of what is actually there. Hmm though I'll make an even more bold suggestion, it may be prudent to make it so that a majority NA vote on a server will make the entire rating for that part NA. Since it would kinda suck if one guy either in the past or after the system is in votes 1 for say events where everyone else voted NA and suddenly your stuck with a 1 vote for events in the general listing. Yikes!

Well if there is no donation reward option I still say just split the servers in two sets of lists. And if not a forced yes/no for donation is good enough, and while sure they may not start out as one, you can change them if people talk about them changing into a donation system on the forum.

Well a requirement of  "detailed reviews" for very high or very low scores really would help solve spam. So if they do want to spam with 100 then they gotta write a even more than 400 character review that you mods look over to see if its up to snuff to justify that rating. These guys will be too dumb and lazy to make a long review for it. (though eep it is biased against non-english speakers....), maybe the only english they know is "this server sucks" . But it really should be taken out of users hands and to you guys, because this WILL turn into a review rating war. I can easily see servers encouraging their users to up-vote positive review and down-vote negative reviews and it will just be another variation of the same problem you are having.

Color just helps people know things might be out of date without making a section which basically screams "reviews you probably wont read or care about and don't count for anything!"

Skots heart idea is basically just to totally change the actual ranking to "film review style" rather than "gamespot review style". While people can still rate on the individual thing the actual ranking is only based on the "heart" score given to the game (which I guess really just means star since thats what the site uses ;p) Which, to be quite frank, is how it is done in practice anyway. If someone wants to give a server an 85 they will fudge any individual ranking they can to give it an 85. Which you know, they are probably right. Hell then you won't even need to make some complicated system for NA, they can NA everything if they want and just give what "heart" score they think is best.

Gotta say I agree with the "how much of a string of positive reviews you have" would probably help. Because it is sometimes odd that a server that has not had any "rotten" reviews can still be lower than a server with 3 or 4 because it has more users reviews or the general rating was higher (probably because of lame 100 votes).

[Skotlex]

One more point about N/A. Do like mysql does, consider those null values, they do no affect the average. 10 + N/A + N/A = 10. Just use the values where people have voted. Perhaps if there is a large amount of N/As then the final ranking on that area should be N/A too. Hmm, or you could just show the average of the people WHO voted and a number in parenthesis next to it to indicate how many people have not rated that area. Just don't make a N/A vote alter the final average.

[Glenn]

Just average them, and multiply by 10 as opposed to adding them all togther.

The heart system seems more easily spammable to trolls/flamers and server promoters than it is now. I'd say put it as an option that DOESN'T effect score, so people will put more honesty into it, as opposed to trying to be reasonable to bring the server up.

[bulbasteve]

Depends how you do it, if you are strict over what warrents something above 90 or below 20 or something. Because really this hearts system kinda IS what we have now. There are only a scant few people probably who actually follow what there honest ratings for each section would be. Everyone gives a few extra points or takes some way to various things cause they want a 85 and not a 81, or of course really just plain lie about something like uptime when making a bad review (though the NA system would solve that in itself).

[kopono]

I have to be honest with you... I've only read the first post and all of Skotlex's posts. "My two cents regarding the review-score:" - Quoted from Skotlex

I agree with everything you have said. The thing is, many people who start now are interested for the money, just like Pre-RO. I am willing to sacrifice my position no matter what it is to get rid of servers like Pre-RO who have bribed their players to make fake reviews.

HOWEVER, I believe "120 days" is TOO LONG. And as for the newer servers, there has to be a server out there which maybe the "one" which we all haven't found yet. I'm not going into the technical stuff because I know Skotlex, Glenn, bulbasteve probably have said most of the stuff AND they look intelligent enough to give you the right advice.

I say go for it =) and best of luck!