2014/03/31

Hello again. Since my last blog entry, we shipped the game and took a bit of a vacation! Let me catch you up on our technical progress since last time.

The code base blossomed mightily with all the new features required for the Saga. The Zeno tool got lots of new panels and features, as did the game engine itself. Some massive performance improvements were made. Many memory leaks and bloats were fixed. If you compare the Strand scene, or battle scenes, in the Saga with those in Factions, you should be able to see and feel a difference. Control-Shift-P should display the performance graph, which shows Frametime in Factions and Frametime + Memory usage in Saga.

During the last few months of development, we relied heavily upon Sculpin QA, an outsource QA company in Canada. Going back and forth every day with them we were able to pretty thoroughly test the game and prioritize fixes and improvements.

We met our self-imposed deadline of December 20 for ĎGold Masterí, which means the game was finished and ready to ship. After our holiday breaks, and during the 2 weeks leading up to launch, we addressed a select few additional items and bugs, to put the final polish on. Because of this schedule, we ended up with something that is pretty much how we want it to be, and very stable and bug-free. We knew that when everyone piled in, unexpected issues would crop up, so we set aside the 2 weeks post-launch for triage and responding to issues.

We also had an emergent problem with our website on launch day. A remarkable number of Kickstarter backers appeared at launch time, and our website host could not handle it. Our host had no viable upgrade/scale path for us, so we put the website into maintenance mode and got busy creating an Amazon AWS EC2-based webserver. This allowed us to provision plenty of resources, and will let us scale it up and down in the future in anticipation of demand. The forums were still slow, primarily due to database access times to our old hosting provider. After the dust cleared, I put the forums in maintenance again, and moved the databases into Amazonís cloud, yielding massive performance improvements.

Another thing that went wrong when thousands of people appeared at once to register on the forums is that our outgoing SMTP mail server started silently failing. We use our Google apps service for sending out email, but when it hits a certain, low, limit, it starts silently failing to send out mail. This means that people registering and waiting for their confirmation email were not receiving them, and therefore were unable to get their Steam key quickly. Exacerbating the problem was the fact that after registration, the forum page didnít (and doesnít) really make it clear that you need to confirm your registration before you can get your key. We did LOTS of manual customer support, bolstered by some round-robin mail server tricks I pulled and some SQL scripts that we could run to fix up everybodyís permissions. At any rate, we really appreciate the patience of our Kickstarter backers as we worked long hours to deal with the high demand.
During the 2 weeks after launch we fixed everything that came up, and added in subtitle support for the VO in the game, based on player demand for it. Also during the time around launch I worked on getting all the engine updates integrated into Factions (new music, optimizations, animations, user interface improvements, etcÖ). I havenít been able to finish that work yet, so it has been shelved for a while as we move forward on porting, localization, and the next part of the Saga.

After our first patch, I got busy with several different things. Iíve tried to support the intrepid modders in the quest to figure out the game data formats. To that end I created a small decompiler app that lets you peek inside and modify compiled game data in-place.

Iíve worked on several things that should show up in the next patch:
  • Save profile slot support. There are 5 save profiles that save games can be put in.
  • Zeno editor included with the game download.
  • Zeno decompilation support. This should let you decompile the entire asset set into their uncompressed forms, which Zeno can then re-compile back into the game.


Iíve been hard at work on the iOS port. Currently the game loads each of its GUI SWFs dynamically at runtime. Some of these are loaded at startup, and some are only loaded when needed. On iOS, the AIR runtime wonít let you do this -- all SWFs have to be loaded in at compile time, essentially. Iíve been laboriously converting the game to compile in all its GUIs rather than load them dynamically. This will affect the PC/Mac version as well. This process is almost complete. I had to create a system that allows big, texture-heavy SWFs to load their big textures at runtime, though. Adobe AIR provides some compilation support for stripping static data out of SWFs but I would rather avoid additional reliance on the black-box Adobe tools if possible. The game runs remarkably well on the new iPad Air. I havenít tried it on older/slow iOS devices yet. The biggest chunk of work remaining for the iOS port is converting the FMOD integration into iOS code. Then of course the testing and optimization. Iíve been searching around for programming contractors with Adobe AIR Native Extension experience, but to no avail so far.

I spent a good bit of time prepping the game for localization. There are quite a few English strings hard-coded into the GUIs. I had to go through and search all those out one by one, and move them into the string file. The conversation Inkle files have the English text embedded directly into them, so I had to write a system that provides the translated text to them at runtime. I needed to add some support to let the game auto-detect your Steam language and use that. I still need to add an in-game menu for switching languages. We evaluated about 4 different translation companies. We looked at their competencies, portfolio, and price. We finally settled on one, and rather than doing only our top 3 languages (German, Russian, French), we decided to splurge and do 3 more at the same time (Spanish, Portuguese, Polish). As I write this we are still waiting for the translations to complete. I worked with the translators to determine the best way to deliver our string data. I built a tool into Zeno that will export the entire gameís strings into a single CSV file which makes life easier for the translators.

Iíve been spending time investigating options for porting to Linux, Playstation, and Xbox. I spent a week at GDC making contacts and talking to folks, and our plan is currently to leverage ScaleForm to do the ports. I am working on following up with some contractor leads that I might be able to hire to do the ports. The cool thing about this approach is that it also works on PC, Mac, iOS, Android, the works. There is a distinct possibility that the game might run more smoothly and faster than it currently does, in which case I would be happy to abandon Adobe AIR entirely, but that remains to be seen. I do have the nagging concern that my iOS AIR Native Extension work will end up being obsolete and deprecated later this year, but so be it.

Since returning from vacation, SXSW, and GDC, my main focus has been on prepping TBS2. Basically it involves making Zeno and the game engine more flexible so that there are no hard dependencies on anything specific to the first game (Saga1 as we call it internally). Iíve been improving Zeno to handle more data sets, and spending a great deal of time streamlining and abstracting our Ant build scripts. Ideally I can get Alex and Arnie in full production mode while I go forward with managing the porting and localization efforts. We have several improvements planned, and we have a laundry list of enhancements that weíd like to get into Zeno. Iím looking to hire a programmer contractor with experience in Apache Flex to upgrade and improve Zeno.

Since the last time I gave you a snapshot of the code, things have grown dramatically. At this point we are at well over 250k lines of code. From the graph below you can see how things slowed down immediately after Dec 20 lockdown, and then again after the patch.



Until next time!
JW

Previous: Technical Blog #4