NeverWinter AD Exploit

If you came here after searching for an exploit, you’re going to be disappointed.  I’m not going to say what the exploit was, I’m going to talk about my reaction to Cryptic’s reaction to the exploit.  You can talk about your reaction to that reaction in the comments :P

As you’ve probably heard by now, Neverwinter had some issues with exploiters over the weekend.  Early Sunday morning, the exploiting hit levels that were bad enough, Cryptic ended up taking the game down to investigate/resolve the issue.  When they brought it the game back up late last night, the Auction House and AD Exchange were still offline.

Without going too much into detail about the exploit, it basically allowed unscrupulous folks to clone AD (Astral Diamonds, one of the in-game currencies in Neverwinter).  AD is the currency used on the in-game Auction House… so, as happens when people have way too much of an ingame currency… prices on the AH skyrocketed.. the more people exploited the more damaged the economy got (or, would get).  Initially Cryptic’s plan was to just take the AH/AD Exchange offline, and look into the matter further.  It appears that once they got into their research, they saw how rampant the exploit was…and shut everything down to address it.

Here’s the post the NWN team was using throughout the day yesterday to update everyone:

Hey All,

The Astral Diamond Exchange and the Auction House have been temporarily taken offline to resolve a high priority bug.

At this time we are still investigating the details regarding the issue and will post more information as the situation progresses.

As the issue involves in-game currency, we understand that there may be questions regarding the follow up actions needed to ensure that the economy remains fair and balanced. At this time we are still in the process of collecting the data, but plan to share details regarding follow up actions once the investigation concludes. Thank you for your patience and understanding while we continually smooth out issues during open beta.

Sincerely,

Dezstravus and the Neverwinter Team

 

Update 12PM Pacific: We are still investigating the situation. There is a high likelihood that we will be performing character-specific rollbacks on any accounts that were found to be utilizing the exploit. Additionally, we are investigating the extent of what can additional steps can be taken to remove any illegitimately obtained Astral Diamonds that have entered into the economy. We are also investigating the possibility of a shard-wide rollback.

Update 12:15PM Pacific: We’re bringing the shards down temporarily to better investigate the issue. Estimated downtime is 1 hour. Thank you!

Update 12:45PM Pacific: We’re still investigating the issue and will have more information for you as soon as possible. Thank you for your patience!

Update 12:55PM Pacific: The shard downtime will be extended for two more hours as we address the AD/AH issue. More details soon – thank you!

Update 1:30PM Pacific: We are identifying the accounts that have utilized the AH exploit and will be taking action against said accounts. We are also still investigating the severity of the issue and its impact on the game, and from this information we will take corrective action. Estimated downtime is at least two more hours.

Update 5:15 PM Pacific: We have most of the information in-hand, but are waiting on a few final details to determine our plan of action. To clarify one thing I’ve seen around on Twitter, whatever we ultimately decide to do, a full character wipe will not happen. We’re doing everything we can to minimize the impact of this fix, so that it changes what must change, but not much else. Thank you all for your patience, we appreciate it and will have you back in-game as soon as possible.

Update 6 PM Pacific: We’re still finalizing the plan of action. Once everything is locked in, we’ll be posting everything, so keep your eyes on Facebook, on Twitter and on our forums — we’ll make sure it goes everywhere. Thank you all again, we appreciate your support and patience.

Update 6:35 PM Pacific: We have a plan of action. We’re verifying the exact timeline with the Neverwinter team, then we’ll be triple-checking the planned forum post with all key stakeholders before posting. We’re not finished yet, folks, but we’re getting closer.

Update 9 PM Pacific: The post about our plan of action is being confirmed — stay tuned, folks, we’re getting closer.

 

 

The communication was good, started out more frequent then it ended, but, still seemed to be a good attempt to keep everyone in the loop (the 2 hours from the last update, to when they posted their ‘plan’ was a bit long between updates, but, depending on who they needed to find at 9PM on a Sunday, to approve things.. partially understandable.  I’d have preferred to see more frequent updates between 9 & 11, to tell us where in the approval process things were.. X out of Y people approved..etc..).

The 11:00 update, brought us the plan (hint: it didn’t involve a full wipe), to get everyone back in game ASAP (after the 7 hour roll back).

Dearest Beta Testers and Bravest Adventures,

In the dark hours of this Sunday morning an in-game bug was discovered and taken advantage of by a very small group of villainous Nashers intent on exploiting Neverwinter’s Open Beta Gateway and Auction House systems for unearned Astral Diamonds.

Thanks in no small part to the efforts of our continually amazing Beta community, we were able to quickly identify the exploit and the perpetrators. Once identified, we took immediate action, calling in the entire development and publishing teams to lock down the Neverwinter OBT as we sought out a solution.

Rest assured, the issue has been corrected and we have taken appropriate action against all players who took advantage of the bug, including but not limited to enforcing permanent bans.

Sadly, the damage to the economy was done.

Rather than let the malicious efforts of a few unsavory players linger and continually impact the game’s economy and balance as we progress through these later stages of Open Beta, we have made the extremely difficult decision to rollback Neverwinter to a time shortly before the abuse and exploitation began.

This means that roughly seven hours of progress made between 5:20 AM and 12:20 PM Pacific Time will be lost to all players. For that, we do sincerely apologize.

Following an extensive QA check, Neverwinter will begin again as soon as possible. We hope to have the servers available again soon, and will continue to post updates on the forums and other channels.

When the servers do come back up, a few systems will not be active: the Auction House and the Astral Diamond exchange. When the servers are back up and running, we will be performing extensive testing to ensure that the issues related to exploitation are completely resolved.

Any attempt to use the Astral Diamond exchange will give an error message. The Auction House, however, wont give an error — it simply will fail to post the item. This is the known behavior, and we’re working to bring these systems back up as soon as possible.

As a token of our appreciation for you, the community, and all that you’ve done for the game throughout these days of Beta, we will be sending a thank-you gift to players and opening ourselves up on the forums and various other channels to answer any and all questions we can.

For more exact details on the rollback and other efforts related to it, please see our comprehensive forum FAQ.

With sincerest thanks for the continued support and feedback,

Your Neverwinter Team

PS: This thread is closed to keep things organized, but we welcome comments about this on the FAQ related to this issue.

 

Now, here is where I think the Cryptic/PWE Neverwinter team did it wrong, while, at the same time, doing things right.

Quick list of the ‘right’ and ‘wrong’ (in my opinion, obviously) of this solution/announcement.

Good:

Gave vague details of the exploit.

Acknowledged the damage done to the economy.

Laid out specifics of the fix.

Talk about ‘appropriate action’ against the exploiters.

Acknowledge inconvenience to the player base, and offer a ‘oops, our bad’ gift.

 

Wrong:

7 Hour Rollback, instead of a full wipe.

Now, I know you’re asking why I’d advocate for a full wipe of a game over the results of a bug “discovered Sunday morning”.  Let me explain.  This exploit existed long before Sunday morning, Cryptic/PWE even acknowledge this fact in the FAQ that goes with the announcement:

Q: This bug existed for more than 7 hours. What are you doing about the fact that players took advantage of this before the rollback window?

A: We are performing additional log searches and investigations to ensure that the appropriate actions are taken against any accounts that performed the exploit before the rollback window. Exploits that were performed prior to our rollback window make up a fraction of a percent of total impact, so we will be dealing with those issues selectively as not to cause greater impact on those legitimate testers.

That means that people have been impacting the economy due to this exploit for a much longer period (potentially going back to Closed Beta, if some of the posts on the NWN forums are accurate).  To allow long term manipulation of the economy by exploiters, and not take actions to rectify that damage, in a game labelled BETA (note, they keep referring to us as ‘testers’ not players)… is inexcusable.  Beta (open, or closed), exists to find bugs, Beta testers also expect (or should) wipes of their progress, and restoral of in-game purchases.

In my opinion, the correct solution to this exploit, was to do a full wipe, refund all Zen purchases made since OB started, and permanently ban the exploiters (and all their alt accounts).  The fact that Cryptic/PWE didn’t do this wipe, tells me that they do not actually consider NWN in Beta, this is a released game for them.  Treating the game as released vs Beta, explains the rollback decision vs a wipe (as has happened in numerous betas, repeatedly to address issues).

All in all, I’m ok with how Cryptic/PWE handled this, could it have been better?  Yes, should they remove the beta tag, since they obviously don’t consider this a beta? Yes.

I do wonder if Cryptic/PWE will answer some of the outstanding questions that I’ve seen asked (and I’m curious about as well), like:

1. If this is a beta game, why not rollback, it would have saved a lot of heartache, possibly got the servers back up sooner, and (based on forum polls) was a desirable solution to a lot of testers players?
2. Is this the same exploit that occured in Star Trek Online’s Auction House back when the Gateway was added to STO?  If so, how did it ever make it live (or in Beta) in NWN? Shouldn’t the fixed code from STO have been ported to NWN with the rest of the AH code?
3. You say a “very small” group of exploiters are responsible, how small is “very small” how many of that group received permanent bans, and how confident are you that you caught their ‘alt’ accounts that they shipped goods/diamonds off to?
4. Assuming you miss some ‘mules’/’alts’ used to hide goods/AD, what are your plans to prevent these hidden caches of millions of AD, from being reintroduced into the economy, and causing further damage?

It will be interesting to see how Cryptic/PWE follows up in the next few days over this.  I also wonder how many players they lost as a result of this.