ERMO logo ERMO Puzzle Available for

Nightmares in the weekend

I want to write a story about the development for Windows.

Few weeks ago, I published a little puzzle game on Xbox One. Friday evening a player mailed me with a suggestion.

I was happy to include it and I did, I published a new version. But, my super effective release-a-new-build script went wrong and I mistakenly published a corrupted package. My mistake.

I noticed myself few hours later. And soon I attempted to publish again.

And here that the story begins.

Foreword

The previous update, the bad guy, was in releasing state. Stuck in releasing state! I couldn’t do anything. No cancel, no update, nothing.

First, I thought they caught the package was corrupted (fast crash just on open) and halted the process, yeah, let me check the error logs…


 ^
 |
 |
+-+
 |
 |
+-+
 |
 |
+-+
 |                                              XX  
 |                                            XXX 
+-+                                         XXX
 |                                         XX
 |                                      XXXX
+-+                                  XXXX
 |     XX                        XXXXX
 |   XXXXXXX      XXXX  XXX    XX
 +---X------XXXXXXX--XXXX-XXXXX-------------------->
 
Like my ASCII graph? Courtesy of a non working dashboard health panel.

Few hours later.


 ^
 |
 |
+-+                                    XXXXXX
 |                                  XXX
 |                                XX
+-+                              XX
 |                              X
 |                             XX
+-+                           XX
 |                           XX
 |                          XX
+-+                        XX
 |                       XXX
 |                     XXX
+-+                 XXX
 |    XXX       XXXX
 |  XX  XX    XXX
 +--X----XXXXXX----------------------------------->
  
Like my ASCII graph? Courtesy of a non working dashboard health panel.

You got the idea.

The package went through, with so many player happily enjoying the nice experience of a white splash screen and boom.

Chapter 1: Hack

I started to try everything to unstuck the previous release, with no success.

I tried to inject some js commands via the console, nothing, the backend was categorical you have a pending release. Yeah, pity though it was not so pending.

Ok, let’s hack with the Store API, I’m a clever guy. There’s a nice submission management endpoint that fits exactly this use case.

Nothing.

Wow, they did the homework about checking inputs against the actual state.

It’s 4am, very late, let’s fall asleep, tomorrow will be good.

Chapter 2: Sentence

Rays filter through the window left open, finally ceasing my tormented rest.


                                 X
                               XXXX         XXXX
 ^           XXXXXXXXX      XXXX  XX      XXX  XX
 |          XX       XX     X       X    XX     XX
 |         XX         XX   XX        XXXXX        XXX
+-+        X           XX XX
 |        XX            XXX
 |      XXX
+-+  XXXX
 |XXX
 |
+-+
 |
 |
+-+
 |
 |
+-+
 |
 |
 +------------------------------------------------>
  
Like my ASCII graph? Courtesy of a non working dashboard health panel.

Oh nice, I like roller coaster in the morning.

Well, let’s try again, they did something of course, for sure they have a huge operation constantly monitoring the production.

Let me open the dev center, let me click on submission…

Releasing. Stuck.

Oh the cache! Clear cache! When you’re desperate you start losing your mind…

F5, hard refresh, tick on disable cache when devtools are open… and lots of hope.

Releasing. Stuck.

Meanwhile, some players mailed me about the crashes and, later in the day, 1-star reviews started flowing in.

Chapter 3: Introspection

ERMO isn’t a unique game, it’s not peculiar, just a common logic puzzle, but I put a lot of hard work to make it and to port to the Xbox One. I was very happy for the few that bought the “premium” levels, the first spare bucks I earn with gamedev, after I had quit my previous enterprise-grade lead engineer wage.

When you work in a giant company delivering world-scale products, production issues are common, you get quite accustomed with. Nothing affects your mind.

A new bug-ticket came in, set the gravity, put an ETA, assign to someone you know or some weird-named dev team over there, put in the queue. If you’re the (un)fortunate assignee, dig through logs and tons of monospace chars you never saw before, roll up your sleeves, you solve in just one day with no prior knowledge, congrats yourself, you’re a hack ninja, push to test, push to preprod, package and issue a request-to-install-to-prod ticket, and back again to the start of the cycle.

Day after day, a ticket becomes an abstract thing, like triangles or circles, no more than that.

How many tickets today? How many polygons? How many yellow scarfs? Sound exactly the same.

There’s no more connection with the real world, people facing bugs and not be able to use your company product, after acquiring the pricey (but worthy : ) license.

I think it’s kind of a self defense, you can’t work if you know personally all the users and all stuff involved, all the work halted, all the money lost, all the ramification a single critical bug can cause to a person using your company product on the other side of the planet. The total amount of pressure will freeze anyone has any electric impulse roaming in his mind.

To get the job done, you don’t need the complete context, just a row in a table(state=open), logs and a code repository. And sometimes they alone are overwhelming too. Your company product, simply is a product.

But when it comes to solo, the time flows slower, the world becomes smaller and despite all the efforts to keep your head up, your eyes wide open, all the times you say yourself you’re in a big landscape, there’s a lot out there with so many things happening, moving, mutating, even though the experience taught you the quick rise and slow fall of all issues, how common is to be overtaken by that turbulent and restless wave… yet, solo-day after solo-day, your mind still keeps tricking yourself, the horizon shrinks down to your limited perception, and it happens that, in the whole mega big world, there are only you and three players that bought your game yesterday and cannot play today.

And it happened.

Chapter 4: Fight

Look at the top bar, right corner, oh, there’s a help button. I’m an experienced dev, I don’t need any help.

Refresh. The issue’s still here.

Mmm… help, … there’s a help, mmm… no, no, I never used help menus before.

Refresh. Refresh.

Pending.

Mail in, Ohi, ERMO crashed on start screen just after the Xbox login…

Refresh.

Reply, Working on, just publishing an update.

Refresh. Yet, Pending.

Ok, just a click, let’s see what happens. A question mark can’t cause big damage and I’m just woke up, my HP bar is full.

Click. Oh three items, Support, Feedback, About.

About first. I don’t want contact support, maybe there’s something in the About.

Nothing. Just Microsoft Developer. I know, I know that. What I don’t know is what’s going on and why I can’t publish anything!

Ok, lots of time passes to make two clicks, so, Refresh.

Stuck.

Question Mark, Feedback now.

Type of feedback? Kudos? No kudos for you this time. Suggestion? I don’t think I can suggest you to put back the New Submission button. Problem? Oh yeah, I have a problem. Just a tiny one, not so important, but a problem is.

Please allow to cancel a submission or create a new one, where the actual is stuck on releasing state.

Mail in, Ehi, Xbox, login, BOOOOM.

Reply, I just published an update, store processing… it will arrive soon.

Oh, so many keystrokes, for sure, a lot of time passed. Refresh.

Releasing…, yeah for sure.

Refresh. Refreeeesh.

Ok, ok, I realize I can’t do anything.

But, it’s Microsoft, a megacorp, there must be some info about devcenter issues somewhere. Let’s write some chars on a competing service:

Dev center status page: no result.
Microsoft store status: wrong search.
developer dashboard windows status: no result.

Ok, no page.

Let’s have a look on twitter, probably there’s an handle there texting about all the magic stuff the windows dev center offers.

Nothing.
And, oh yeah, under 3 minutes I can bring a legacy app to windows store, cool, but my app is already in the store and I need to update that.

I miss a Refresh.
Refresh.

I miss the New Submission button.
Refresh.

No New Submission button.

Ok, I’m totally in the dark. Probably this is an isolated issue, happening only to my game. (Not true, just convincing myself).

I have no other choice, I don’t want, I never did, but I have to, they forced me.

Question mark, Support.

Support is available Monday-Friday 9:am-9:pm EDT.

Ok, I’m not a child anymore, I’m a grown up man now, I can’t lose control.
Shouting doens’t help, neither crying.

I surrender. Politely. I recognize your prowess. I did my best, but you won on every ground.

The sun sets behind who has been won, and the shadows of my defeat are invading a mutinous finger still lingering on a F and a 5.

Chapter 5: Harmless

Sunday passed, crashes and crashes logged, bad reviews, lots of Refresh, lots of Pending.

Monday afternoon.

The so badly desired New Submission button came to existence, square, enabled, shiny light gray on white background.

So much beauty I couldn’t resist. The temptation was too strong.
I clicked and I released the update, ending the nightmare and this story.

Not quite true, some errors happened, some endpoints kept responding 5*10^2+3, but solved in a couple of hours.
Still reviews and health page are unavailable now that I’m writing, I couldn’t take screenshots of the errors chart.
Update: Today (12 sept) the dev center dashboard is totally unavailable.

Afterwords

What still boggles me now, in this moment of regained lucidity, is the total absence of communication.

An issue on a critical path, blocking all the publishing services and no sign of any activity? No alert, no message, no status, nothing for almost two days straight.

We must have faith someone noticed the problem and act accordingly?

What’s going on?
Is this the way to do business with many aspiring developers for the platform?
Aspiring might remain aspiring or become leaving, if some serious countermeasures aren’t made.

I love the platform, I love the tools, I love the ecosystem, I love C#, I love typescript (oh so much love, I can’t imagine typing javascript : ) anymore), but I don’t want to be alone.

Let be honest, when it comes to games, the Windows Store is totally skipped. All devs and players I talked to said, you have to release on Steam or here and there, no one cares about Windows Store.

Ok, I’ll do some day, but I care about Windows ecosystem.
I like to have one codebase that runs on all Windows devices and Xbox, I like the services that run the same (almost), I like the UWP API, not so much the docs (the shining old days of the offline msdn are gone forever sigh), and I just want a better experience.

The entire world doesn’t boil down to the issue I had this weekend and my negligible game. And I don’t want make it bigger than it really is. But the details matter.

To be pragmatic, I talk about my experience and my little headaches, hence few friendly tips.

Start with a simple service status page.
This is the 3rd time in a month, I experience similar issues.

Then proactively communicate and talk with the developers.

I have sort of a mystical and urticant fascination for the word proactively. When I was hired, I clearly remember the big big big boss saying me, you do not have to do just your job or what you are assigned to, you have to proactively look for issues and where you find one, turn it in an opportunity to show your skills.

Well, water has passed under bridges and wind through trees, I hope it holds true for Microsoft too.

Keep Coding.

Wrong English? No problem, tell me, not native, I’ll learn

Follow this devblog

If you want you can stay in touch:

This website uses cookies to improve your experience