Letter from the Producer (January 20, 2006)

This is a letter from EverQuest producer Chris Lena regarding the recent player rollbacks, as seen on. The situation stems from this patch on Wednesday. ====================================================== Hello everyone, We have now completed our last group of character restores in regards to the update issue that occurred on the morning of January 18th. Before I go into the details let me start by giving our apologies to all of those that were unknowingly affected by the correction of our error. We did feel that we needed to be aggressive in dealing with this issue. I also hope that everyone can understand that we needed to be vague on the details while our actions were being taken. However, now that it is complete let me take a moment and explain our solutions. The summary is that we rolled back any character that had traded with the Beta merchants from our Prophecy of Ro testing server. We also rolled back characters that had traded with those ground zero characters as well as any subsequent traders along down the line. Taking a step back I will start with the morning of the update. During the beta testing of an expansion we will have a number of NPCs available to facilitate the testing of new tradeskill recipes. Since the point is to actually test recipes and combines the items are free or extremely cheap. Unfortunately, due to a data error, these 25 NPCs with 1,530 items ended up on the live servers. As soon as we realized the issue we removed the NPCs from the Plane of Knowledge. We decided not to take down the servers until we fully understood the situation and its ramifications. This was the approach that we continued to take throughout the day – slow, steady, and methodical. After researching and tracking both the players and items we crafted a solution that would not require server downtime and would only affect a small percentage of our overall population. We began with the characters that purchased items directly from the Beta merchants taking the time to exclude those people that only purchased harmless items such as water. We also excluded anyone who only sold to those merchants. After that we expanded to those that traded with those characters and iterated down the chain to include all trades. We felt that we had to widen the web in this way to prevent a number of secondary methods of dealing with these ill-gotten items such as trading the results of tradeskill combines. At this point we understand that we are directly affecting players that might have had no knowledge of the source of these items or results of these items. However, we believe it was necessary to take this aggressive step to maintain the integrity of the economy. Soon thereafter we expanded our investigation to the other sources such as guild bank, shared bank, bazaar transactions, and other more obscure possibilities. This afternoon we ended with a final sweep of all areas to ensure that we conducted as thorough of a clean up as possible. We did this in a step by step manner so that we could keep an eye on the number of characters affected, maintain a high level of thoroughness, and minimize any chance of errors. I have also seen a number of people suggest that we should have taken down the servers and rolled back every active character on every server. That was my first thought as well but once you get into the details you can see that this would not have been the best course of action. Primarily, every character would have lost their play time instead of the small percentage that was involved. Also, keep in mind that any rollback or restore has a small chance of an error and rolling back that number of characters would have magnified those exceptions to an unacceptable degree. After going through this process over the last 24 hours and seeing the results I am confident that we took the best course of action under the circumstances. The result is that the integrity of the economy has been maintained while touching less than 1% of the character population of EverQuest. The majority of the EverQuest population was able to enjoy the game normally with zero server downtime and no loss of play time. We do understand that this might not matter to those of you that were rolled back and for that we apologize. In conclusion, there are two known issues with the rollbacks regarding guild membership status and missing corpse issues that may have affected some customers. Customer Service will be assisting with these specific known issues, but will not be assisting with requests for reimbursement of items, coin, experience, AA’s, or flags that were removed due to the character rollbacks. If your character is having problems joining or being removed from a guild or if your character appears to be missing a corpse, please submit a petition under the appropriate category and they will assist you as quickly as possible. Thank you for your patience, Chris Lena Producer EverQuest

Comments

« Previous 1 2
Post Comment
EQ versus WOW
# Feb 26 2006 at 10:10 AM Rating: Default
WoW is no more a childs game than EQ is. They are equivalent in many ways.

Blizzard does not put your level 65 spells on raid god mobs and force players to navigate the ignorant selfish eight year old ego's of greedy guild leading punks to get your level 65 spell scrolls.

WoW graphics > Sony
Blizzard support has always been > Sony
Wow XP too fast - Sony XP WAY WAY too slow

I actually perked up when they did double XP weekend. I figured if they went full time with DBL XP I might come back but then they killed it.

EQ - needs vendors in POK or somewhere that EQ that dish out KEI for 10pp or something.

EQ - need to DBL XP and half XP loss from death.

EQ needs to put out a $9.99 all inclusive patch to get all the people with 3-9 expansions up to the full ten or twelve or however many ridiculous number of expansions have ben out since we all quit.

EQ needs to put all the spells and some good cheap mid-grade quality armor on vendors for all those people who are SO far behind now from being away that their gear leaves them being asked if they were Ebay'd when they try to return.

EQ needs to stop listening to all of the current EQ players and listen to some of the ones that left. If they can try to overlook the hostile, childish, arrogant, selfish, OBNOXIOUS players, then Sony should be willing to try to help them enjoy better CASUAL gameplay and not have to rely on those hard core no life losers that stayde behind which are IN THE WAY of many people coming back.

CASUAL PLAY means I can SOLO or 'six-man' to get all my spells or some decent armor upgrades.

CASUAL PLAY means I we are not all no-life losers who live with our MOMS and cannot afford to put game ahead of real life.

CASUAL PLAY means if you stick some pompous jerk GL in charge of if we ever get out level 65-70 spells we will never have them. Sad people stop sending you money as you have seen by now so FIX IT!

Give everyone a WAY to come back and play your game. Leave it like this and they will continue to move to WoW and the next summer WoW release will squash what little chance EQ has of any REAL future life.

Ignorant flames are not required thank you. Thats my opinion. Chortles and obnoxious comebacks are understood and not AT ALL needed Thank You. Your earlier flames of IF YOU DONT LIKE IT GO PLAY SOMETHING ELSE were sufficient.

WE DID!
for thoses wow players
# Feb 01 2006 at 5:29 AM Rating: Decent
27 posts
Just to say WoW is not a MMO for adults. Wow is a child game ... go seek mummy and dad and dont disturb us anymore.
only lost LAA
# Jan 25 2006 at 10:20 PM Rating: Decent
only thing i lost at all was a LAA. hmm how the hell does LAA get affected by this didnt lose any aa or plat or any gear or anything just a LAA
My 2cp
# Jan 25 2006 at 6:17 AM Rating: Decent
**
308 posts
Everyone that has a problem with EQ should come play WOW. The game is what EverQuest aspired to be oh those many years ago. I sit as my elf on the docks (waiting for the boat! the BOAT!) fishing (!!!) and I'm simply awestruck at how a game can give me a sense of nostalgia so strong.

I don't play EQ anymore, I actually uninstalled it from my hard drive when they started releasing an expansion a year seems like. Plus with most of the community leaving for EQ2, WoW, and GW it just felt like time to move on. Where am I going with this? If you aren't happy with the game, just go cold turkey and pick up something else. YOU pay THEM to play their game, not the other way around. Think about that next time a GM won't respond or they ***** up the coding yet again.

Just my 2cp.
tbh,
# Jan 28 2006 at 7:32 PM Rating: Decent
Avatar
**
654 posts
To be honest, Clokver's got a point. Personally I've seen MANY people leave EQ for EQ2, WoW and some few for Guild wars. But from what I've heard they left because they were bored, and ended up 3/4 of them coming BACK to EverQuest.

So sometimes a little change is a good thing. Go ahead and check out what's out there. Everyone does sometime. Just seems like EQ's where the heart is. /shrug

Just me 2cp. =)
____________________________
http://www.magelo.com/eq_view_profile.html?num=744646
----------------
ShadowSoldier
Antonius Bayle
Rollbacks
# Jan 24 2006 at 5:11 PM Rating: Excellent
I was thinking of devious ways someone could gain from the rollbacks, say 10 stacks of the Metallic Drake scales for example.
1. put the ill gotten items in the guild bank - they said they fixed that.
2. hide the items on a corpse - they thought of that
3. hide the items through a series of trades between alts on different accounts - they thought of that.
4. put the items in the guild bank, have an alt get them out, trade them to a diff alt, have that alt put them back in the bank and a different alt get them out. THen duel the alt and loot them as duel winnings a few times to different alts. (no trade window?) then sell the items in your guild hall to the merchant there (shh dont tell anybody or else make enormous threats to naughty guild members and be sure to have some un-tainted alts stand around in the guild hall guarding the trader with bad AFK messages to warn ppl and so the trader doesnt reset!

Hows that for laundering money?

Edited, Tue Jan 24 16:13:54 2006
RE: Rollbacks
# Jan 26 2006 at 2:28 PM Rating: Decent
rofl you thought this out didn't u?
Produce Delvelopment Manager
# Jan 24 2006 at 4:47 PM Rating: Default
Please welcome our newest manager to Sony Online Entertainment. His name is Leeroy Jenkins and he will be responsible for new expansion testing and migrations to the live servers. He has some gaming experience or experiences anyway.

Thank you SOE.
New Produce Development Manager
# Jan 24 2006 at 4:42 PM Rating: Decent
This is SoE.
Please welcome our newest Product Devleopment Manager to our company. His name is Leeroy Jenkins and he has some experience in gaming. Leeroy will be responsible for migration and testing of code for our new expansions.

/golfclap
What a joke
# Jan 24 2006 at 4:41 PM Rating: Decent
Sage
Avatar
**
756 posts
Here's one for you. A friend who has been away since December 16, logged in today to find several chars a level lower and many items missing. It's IMPOSSIBLE for her chars to have been involved in ANY part of this, yet they were rolled back a long ways.....
customer service ??
# Jan 24 2006 at 10:42 AM Rating: Decent
Well Alls well but I got no benefit from this and have played 4 years some help to us that couldnt play last week ..
Quit the PMCing
# Jan 24 2006 at 9:48 AM Rating: Default
SoE finally does something that resembles Customer Service after years of ignoring its client base, and most of what I read here is Pissing Moaning and Complaingin!

How many of you Code writers have applied to SoE?
If the rest of you had anything that resembled a life, it no longer shows because your so wrapped up in what you yourself refer to as a game.

Was I rolled back? Maybe, maybe not? I am not even sure I was effected in anyway by it.

Did I take advantage of SoE's attempt at customer service? OH HECK YEA!!!!

Am I sucking up? No, just acknowledging SoE's attempt at giving back to its customer base, instead of treating us like stamps. {licking us sticking us and sending us away}.
stunned
# Jan 23 2006 at 7:58 PM Rating: Decent
First off, I wasn't affected by the rollbacks. I am just in complete disbelief that Sony would waste all the time and money on something that, according to them, affected less than 1% of the population. But the buying and selling of characters and pp affects 100% of the population, and has destroyed the "integrity of the economy", and they do nothing. The reasons stated for what was done is just BS.
so long
# Jan 23 2006 at 6:51 PM Rating: Decent
i am so frustrated i am leaving eq, and i have been playing for 6.7 years. i don t give a **** about a week end double exp, the exp is too easy with all the new Monster Missions ANYWAY.
You say you went through all the logs to check for items that were bought from beta merchants, u even checked people that did not interact with those npcs, but did you check the rest of the logs? no because if you did, you would have rolled back and then refixed the things that were not linked to the transactions.
BUT you are lazzy. you do half the work. and we pay you for that and expect us to stay and say ok rape us and we will say we are happy?
a double week end exp is not enought for what i lost in that roll back. we give you our time to give us good time, but you **** up with us like that and expect us to keep paying you. i guess you though the 1% people that were goin to get rolled back were not important enought to be at least warned or contacted by a gm. u just ran a freakin script looking for strings and tada roll back. lame. 6 years i have been playing and i am quittin for that little error you did.
but i m not worried, u still have 99.999999% of your customers, right? well keep up this politic of 1% each time u **** up, and you will see them all leave for other mmorpgs.
typical SOE
# Jan 23 2006 at 4:47 PM Rating: Decent
I test/write software apps. and whoever let this "mistake" happen,shouldn't be allowed near production code.

well ok, it's only a game, though ppl. do spend a lot of time and money, effort, etc developing their toons.

so the 1st mistake was compounded, by what should have been done, either a instant rollback of all servers, and they seemed to know quite early on, about the "mistake", and/or severely punished, the "ground zero" toons.

next they compounded the issues by not only rolling back once, but anyone who was rolled back, was rolled back twice... so they couldn't even get that right.
( I sold some old scrolls & junk on my trader that had been on him for at least 6 months, for next to nothing, and "unkown to me, was purchased with tainted plat... all 100-150pp of it", and foolish me moved some plat to my main toon from trader),was rolled back at least a week, as some quests I'd either completed of was mostly done, need to be done all over, lost some AA ).
so, ok SOE realizes they really pi**ed off a lot of people unfairly, for what was, and is thier faulty release process, if they even have a process in writing, and make any attempt to follow it... they say.. hey, will give; everyone 'including the ground zero ppl, who if anyone else besides SOE is to blame for this mess, would be them; everyone else who were rolled back, for "some unknown reason"; and everyone else.
everyone is like, hey way cool, gonna log in 1st thing sat. and play all weekend.. this is just so cool... then login 1st thing sat... hear /ooc is anyone getting 2x exp? because we sure are not F**ING SOE !!%$^*$%; of course a lot of us were not that suprised that they couldn't even get that right... so then 4PM sat, they take the servers down yet one more time, and more or less actually get it right.. so now many ppl. will miss out even on this little bit of a refund, as they are working, or in school, etc.... way to go SOE.

3 cheers for SOE! ( sarcasm if you missed that part). They might want to send thier programmers to some classes on software release methodology, instead of blaming thier customers for all of their mistakes.
RE: typical SOE
# Jan 23 2006 at 5:04 PM Rating: Decent
LOL almost forgot this little gem:

was in a HHK mission last night, one of the ppl in our little group had lost his guild tag during this mess. ( we were just about to the point of spawning the named for a win).... a GM pops into out instance.. OMG a GM, they're not just a myth; cool he's here to fix his guild tag. we're standing by the exit waiting for the named to pop. the last prisoner runs right past me, watching the ALT-Q mission tasks.. WTF 0/5 prisoners, and I just watched him run by with my own eyes. We /tell /ooc the GM "did you perhaps mess up the instance".. no reply sent tell to person he's working with. GM talking to him, but also 'no response' to him, on this. or if he could somehow spawn the named. a few minutes later, when GM's confirmed guild tag is back, GM "who doesnt even showup on /w all GM, logs out, as /tell nolonger works".
silly us, we see the instance respawn, and figure what the heck, we start over, and get to where the prisoners are, they didn't respawn, just the mobs do.. ok there's a wasted hour or so doing it twice, for notta. Even when they come to fix something, they break more things along the way, and then wonder why people are unhappy with them.
Flamers ...
# Jan 23 2006 at 12:53 PM Rating: Decent
My sympathies to those who lost things. My absolute contempt to those who flame people for getting angry. They have a right to vent. They have a right to get angry over losing their virtual possessions.

Flaming people who spent hours of their lives doing things that are then arbitrarily taken from them is pointless and just mean-spirited. Shame on the flamers ... may you all meet with a similar disaster so you can truly understand the frustration of these other folks.

That all said, Sony did it's best to fix things in a fair way. It sucks if it hurt you but it's life. May you be more fortunate in the future.
Why rollback to a number of hours before the patch
# Jan 22 2006 at 8:54 PM Rating: Default
I got rolled back to about 2 hours prior to the patch. I didn't purchase anything from the affected merchants or people involved with them but still got rolled back. Fine but why to a couple of hours before the patch considering I did a fair bit of tradeskilling then? (I lost some skill ups and my GM smith hammer which was my 1st success)
EQ Is An Amazing Game
# Jan 22 2006 at 7:05 PM Rating: Decent
[b]Everquest is amazing.
[b]
Anyone who complains about the occasional hiccup or miscode should be punched in the arm, or at the very least given a purple nurple.
RE: EQ Is An Amazing Game
# Feb 27 2006 at 3:37 AM Rating: Decent
I'm sorry you feel that way; I have heard of people being marginalized by Sony. Honestly though, I have never had a problem with them, and I doubt I ever will.
RE: EQ Is An Amazing Game
# Jan 23 2006 at 8:47 AM Rating: Decent
I will agree with you that EQ is a great game to play or none of us would play it, BUT it isn't free. It isn't a service they offer out of the kindness of their hearts for us to use. If it was and this happenned I would be saying, well it is a free game and stuff happens. They don't have the money to pay for top of the line equipment and employees. This is not a free game. We pay every month to play it. SONY makes a small fortune from this game. They CAN afford the equipment and the people to fix this the correct way. I feel like I am paying every month to be treated like dirt every time this sort of thing happens.(Which, on my server is becoming quite regularly) When you are a paying customer you should be treated as a customer, not an inconvenience. That is my biggest gripe. I work in customer service in a sense and would never in a million years treat people the way SONY does. I won't quit completely because I do still enjoy playing. I probably won't keep all my accounts up though. Why should I reward SONY for being dinks about this? Maybe this clarifies this for you?
It's a Game!
# Jan 22 2006 at 12:29 PM Rating: Default
All I can say is, "Somebody call the wahhhmbulance!" This is a game, people! Get over your bad selves and get on with it. If you are threatening to leave the game, do it. Otherwise, just suck it up and start over again. I'd hate to have to associate with some of you in real life when something happened you didn't like.
RE: It's a Game!
# Jan 26 2006 at 7:59 PM Rating: Decent
**
308 posts
'd hate to associate with you in any sense if every time someone has a personal hardship all you can think of to say is "Somebody call the Wahhhmbulance!"
RE: It's a Game!
# Jan 27 2006 at 1:52 PM Rating: Decent
"I'd hate to associate with you in any sense if every time someone has a personal hardship all you can think of to say is "Somebody call the Wahhhmbulance!"

I'd never describe losing a few imaginary items or arbitrary points in a video game as a personal hardship.
What EQ needs to do
# Jan 22 2006 at 5:39 AM Rating: Decent
You after reading all the crying and whineing from you people makes me laugh. Sorry you lost stuff levels and items but damn thats funny to me no I'm not evil just never took this game that serious. Next people will start shooting themselves over it and blame EQ and the game will get shur down or something else will happen. Come on people it's just a game you pay $15 a month to waste time on something to pass the time. Hell I play EQ because I live in a small town next to nothing where counting flies is the best thing to happen to ice cream. I could play Eq or go to the local bar or even watch TV until my eyes rotted. Sure EQ/Sony screws up here and there and we all feel it but it could be worse. They could just say well you lost crap tough ****. I'm not the uber player that spends 20 hours a day selling stuff or playing so I can be a god like and kill anything sony throws at me. I play to have fun and waste time here to get away from the RL.
But I think EQ needs to get rid of selling plat and toons completely. From what I know EQ owns all of our toons and the plat so really you shouldn't be selling eith one. If I was them I would ban all accounts selling plat or toons and any other accounts on that credit card.
I would get rid of MMs completely sence they come out the out pour of people going from level 1 to 70 is insane people have toon that can't even cast a simple heal spell or even fight but they take these toons sell them on the net to people for money. Now I might keep them if they reasied the level to 45 and made the items able to go with you as you level and also drop the exp by 3/4.
Come on we all see people doing MMs all day and thats it no more groups of old have you seen some of the old world zones? Like Dreadlands no one is there or HHK nope no one there or even hunting Icy orcs in Everfrost nope no one there either.
I think Eq should make a pre Luclin server for some people that love the old days where we WORKED for our stuff. So keep on ******** and whineing please I love laughing at you people.
Pasquinell Icewulf
68BST
Kingdom of the Blades
Rodcet server
RE: What EQ needs to do
# Jan 22 2006 at 11:36 AM Rating: Decent
None of the other problems with EQ have anything to do with this SONY programming mistake.

The solution was crude, thoughtless, and similar to using dynamite to get rid of a fly.

If the merchants were only up for 20 minutes then you roll back the servers 21 minutes. Simpler and fairer to all involved. Then give the xp bonus to make up for the down time since a pile of people would be affected.

Sure, some quests,raids, etc. get messed but... that is far less than taking levels away from people that 7 characters down the chain "were involved in this".

To all the people defending the mishandling of this... since most seem to have been unaffected that are defending sony's approach... imagine you lost 1-3 levels, 2-11 aa, 3-6 items some of which are no drop and had 1-2 quests broken with partial turn-ins now in your inventory meaning you have to redo stuff that may have taken you months.

It doesn't matter if you were casual or hardcore, put your self in the situation and then tell me you like how it was dealt with.

For the record, my characters were not affected and I think the whole situation was completely flubbed.
No Double XP for Stromm
# Jan 22 2006 at 2:36 AM Rating: Default
Niiiice, So they give us double experience the first day then our server goes down for 15min. When it comes back up - Normal Experience. Gee Thanks SOE, Thanks for another mess up. Hope they give Stromm and the other servers who crashed an extra day or 2 of double xp.
SOE using overkill
# Jan 21 2006 at 6:34 PM Rating: Decent
*
66 posts
The people who are saying tis is no big deal get over it either dont understand the scope of the problem...or possibly work for SOE.

Trading IN ANY WAY with a tainted character bought you a rollback. If you bought a mob dropped item that couldnt possibly have been on those beta merchants, you could have been rolled back.

Case in point:

Shammy on our guild has been leveling hard the last few weeks to try and get to where she could go one guild groups, finally broke over 60. On thursday she logs on and counts up her pp, she has been saving to buy a ring that drops from oow. She takes the pp from here alt, adds it to the shammy and JOY!, has just enough if its still for sale. The same merchant has had it for sale for weeks in the bazaar.

Shammy goes to bazaar, buys the ring, goes and solos until its time for guild night. The wave of rollbacks Thurs night sends the shammy back to wed morning stats...the ring is gone.

So is the pp she transfered from her alt.

Her alt doesnt get rolled back, only the shammy...becasue only the shammy bought the ring from the merchant who was somehow tainted.

The shammy wasnt online on wed, didnt buy anything else at all on thursday...loses more than a month of her pp. No recourse.


NOTHING soe said suggested this should happen yet they did it. And they are doing a form letter *sorry no refunds*


This series of rollbacks was nothing short of insane. They used a bulldozer to do fine work.


For those who say its just time, fine...next time you go to work tell your boss you dont want to be paid since its just some time you are putting in there.


Sliggoth, druid of 7th Hammer


PS Doesnt SOE have anyone who understands basic PR????
RE: SOE using overkill
# Jan 23 2006 at 12:31 AM Rating: Decent
***
1,117 posts
So petition and explain the situation to SOE. Odds are they will roll back the other toon as well if you ask POLITLELY and explain why. Might take a few days, and maybe even a couple of petitions/emails, but they will probably do it.
What to do, what to do...
# Jan 21 2006 at 4:49 PM Rating: Decent
To all those that were unfairly rolledback, I'm really sorry for you. That truly, truly,sux.

Now that being said, here is one more.

What happened is a direct resemblance to what happens in real life all around the world every day 24/7.

Life can be tough, you work hard, you do what little things you can to make your life, your family's life, your friends (if you have any) life just a little better/easier.

But then life kicks you in the nuts and nocks you back 10feet.

Do you cry and shout Unfair?

HELL YES YOU DO!!

But does that change one little stinking thing?
Does that mean its never going to happen again?

HELL NO IT DOESNT!!

LIFE IS UNFAIR.

Everyday somewhere somebody is getting the crap kicked out of them for no good reason.
Thats the way life is. Dont like it?
Well, you can quit life.
Thats your perogative.
It sux but that's just the way life is.
You either deal and move on or take a dirt nap.

***** all you like its your right as a human being to voice your feelings.

But is there truly one place in your life where you havent taken it in the shorts at some point or another.
If so then your life is blessed from where I'm sitting.

And yes this error is just another fat log on the burning pile of lifes 'up yours' fire.

Sorry,life goes on.
Sofurious.
RE: What to do, what to do...
# Jan 21 2006 at 4:56 PM Rating: Decent

I may be taking this the wrong way... But are you saying that if someone kicks the crap out of me and tramples all over my rights.. I should just lay down and take it because "thats life"?

That's a very sad outlook on things my friend.
Thank you Sony - Really !
# Jan 21 2006 at 4:28 PM Rating: Decent
Thank you Chris Lena for the detailed explanation and for your teams work to fix an innocent but complex problem. EQ is a great game and the time lost to patches, downed servers and other problems is completely insignificant to the amount of uninterrupted enjoyable game time. Please keep up the great work.

Edited, Sat Jan 21 15:30:38 2006
« Previous 1 2
Post Comment

Free account required to post

You must log in or create an account to post messages.