Burst & Record

It took me much longer than I expected to implement the record feature in Weapon M's terminal. Things have been blowing up at work and I have very little code juice to spare. In contrast to how I've felt in the past, I don't merely need a break from work projects; I need a break from coding entirely. I'm glad the holidays are coming up. When I get back to it, I plan to tie up a few loose ends with regard to opening and saving databases. Then I'll begin reviewing all the data lexer rules from version 1.

Bursting with Awesomeness

I've been working on the burst feature for Weapon M's terminal. I actually had it working and then decided I wanted to add more features, so I mocked up a new version:

The burst panel is a small panel that pops up at the bottom of the terminal window when you press Ctrl-B. The combo box will allow you to input a new burst or select from a configurable number of bursts that were previously sent. You'll be able to send the burst as many times as you like, either by pressing enter or clicking the "Send" button repeatedly, or by setting the number of repeats in the spinner. The burst panel will be hidden when you press escape.

I'm combining the burst feature with a SWATH-like "quick macro" recording feature. If you press Ctrl-R while the terminal has focus, it will display the burst panel with a blinking "record" icon but keep the focus on the terminal. Any keystrokes sent to the game will also be appended to the combo box. To stop recording, press Ctrl-R again or focus one of the widgets in the burst panel using the mouse. The burst panel will then behave as though it had been shown with Ctrl-B.

JTX Selection Control

I've been working on the long-awaited selection control for JTX. I made some non-backward-compatible API changes to support it, so I branched version 1.x and bumped the trunk to 2.0-SNAPSHOT. And since I'm breaking the API anyway, I took the opportunity to make a bunch of other incompatible changes, including removing the synchronization from Display and ScrollbackBuffer and changing all the package names to reflect my domain name. I may also change the parameter order of some of the methods in the Buffer interface.

This is what it looks like in action:

I still need to add methods to copy the selection out of the control. I also plan to provide a means of converting the selection into various useful formats, such as ASCII bytes or the Unicode characters that correspond to the glyphs of IBM code page 437.

Dark Matter Update

The development of Dark Matter is proceeding swiftly. It functions as a terminal now. I'm copying classes in from Weapon M and reviewing them as I go. I've already fixed a few stupid bugs in the old code. There's currently a bug in JTX where it doesn't scroll to the bottom when the scroll pane's client first becomes larger than the viewport. This happened occasionally in Weapon M, but it seems to happen all the time in Dark Matter. I expect it'll be an easy fix.

The maintainer of DockingFrames does not want to maintain a Maven repo, so I forked it and deployed docking-frames-core, docking-frames-common, and docking-frames-ext-glass in my own repo. I also deployed Steffen Kux's glasslib. Dark Matter can now be built with Maven, but I haven't written the assembly module yet, so the resulting jars cannot be run without a bunch of dependencies on the classpath. Soon!

Dark Matter

I'm finally getting around to working on the new version of Weapon M that I mentioned back in July. It doesn't do much yet, but I'm really excited about it. It's going to solve a lot of the problems that prevented the original version from achieving its full potential. And it's going to look super sweet, which is critically important. The early code is up on GitHub.

The Database Revisited

I've been working on Weapon M again. It feels great to work on my own code, even if it's only for a few hours a week. The difference between programming as a hobby and programming as a job is like the difference between painting a canvas and painting a house.

I've made two decisions regarding the database. First, as much as I'd like an excuse to learn how to use a graph database, Weapon M isn't it. I can't see any benefit to it when the entire graph of POJOs is already accessible to scripts, and serialization works just fine for persistence. And second, I've decided to simplify the synchronization to a single lock shared by the entire database.

Crazy, you say? A huge performance bottleneck? That's what I initially assumed. But the simplicity of a single lock was so alluring that I had to write a simulation to test my assumption. It consisted of two small programs, each of which starts and then joins with two threads. One thread writes random values to the fields of mutable objects in an array, while the other reads them. In one program, the threads contend for a single lock. In the other, they synchronize on the individual object they are reading or writing.

Each program performed one million reads and one million writes on one hundred objects. The program in which the threads were contending for a single lock consistently runs in 140-170 ms; the other runs in 90-100 ms. That's a pretty significant difference, percentage-wise. But remember, we're talking about two million field accesses in a fraction of a second. Things do not happen that fast in Trade Wars! And this is on a G555 Celeron, a fairly low-end processor by today's standards. When the programs are modified to perform only ten thousand reads and writes, the performance difference is completely lost in the overhead of setting up the threads. And increasing the number of objects being manipulated to decrease the lock contention and give the program using individual locks more of an advantage does not significantly affect the results.

My conclusion is that there would be no significant performance penalty to using a single lock. The benefit would be significantly simpler code, eliminating the possibility of any lock-order bugs that could lead to deadlock.

Swing Docking Frameworks

I've been evaluating Swing docking frameworks in preparation for a major overhaul of Weapon M's UI. I wrote small demo programs using InfoNode Docking Windows, VLDocking, and Docking Frames to get a feel for their quality and the difficulties of working with them. All three exhibited problems managing keyboard focus on dockable components. InfoNode was a strong contender, and it's probably the quickest and simplest way to write a nice docking UI. But I'm declaring Docking Frames the winner on the basis of the project still being actively developed (or at least, as the developer prefers to say, actively maintained) and the fact that the Eclipse theme with the Glass Extension looks incredibly sleek and professional.

I'm actually thinking about a lot more than a UI overhaul. I've started planning a complete rewrite to incorporate lessons learned from 1.x. Version 2.0 will be a single, modular Maven project including the scripts and zip assembly. It will have a unified parser-to-UI event system, as suggested by a user. And it will address weaknesses like issue #22.

The biggest decision I have yet to make is whether I should replace the crude serialized object graph with a real embedded database. If I do, it'll be an OODBMS or a NOSQL graph database, but I've only barely started comparing the available options.

Issue 22

In case anyone has noticed that the build date for Weapon M 1.0 keeps changing, it's because I've been tweaking the build scripts. I've decided that I won't change the version number for a re-packaging that doesn't include any significant functional changes in the program itself.

That said, I think I'm done messing with the build scripts for now. As of last night, I'm starting work on 1.1-SNAPSHOT. The first major thing I'm going to take on is issue 22.

Whateva! I do what I want!

My projects are now all deploying with Maven, except when scp stalls out. I'm still troubleshooting that. Each project's wiki page displays an infobox including links to the latest Maven build. I've been slowly writing other wiki content. And I cast my first line toward getting Weapon M into classrooms. I emailed three professors at the University of Arizona and a blogger who writes about computer science education.

I could write a whole post about the subjective differences between commercial and open-source development. But the biggest difference is that as long as nobody is giving me feedback on my open source projects, my only guide when it comes to what to work on is what I feel like working on. I'm the only person with input on project goals, but nobody holds me accountable if I don't stick to my own plans.

And lately, despite my commitment to moving ahead with JPlex, I'm obsessed with the idea of writing a better lexer and parser generator. So maybe I will. Maybe I will...

My urgent to-do list

I fixed a design flaw in JPlex this weekend. The fix is not backward-compatible, but updating code that uses JPlex only requires changing the imports for the exceptions. I could add an option to make the old behavior optional, but I'll do that only if it's requested.

The problem was that JPlex was generating its boilerplate exceptions in the krum.jplex package, which could lead to those exceptions being duplicated in the classpath. This wasn't creating any conflicts yet because the implementation of those exceptions has never changed. But if it ever did, it could lead to version mismatches. Now the exceptions are generated in each lexer's own package, so it can reliably refer to the version it was generated to use.

Speaking of JPlex... I've recommitted to using it in Weapon M's data parser. I was frustrated by the difficulty of parsing some of the more complex patterns in the game, such as sector displays. (This actually held up the public release by a couple weeks, and it's still not finished.) I would love to use a tree-building parser for the game data, but no suitable parser exists. Every tree-building parser I've looked at waits until the end of input to construct its parse tree, which makes it useless for parsing a continuous stream of text. Writing a forest-building stream parser sounds like a really exciting project, so it's something I'll probably eventually undertake. But for now, I will make the best of the tools I have.

What this means is that I'm ready to begin an extensive audit of the JPlex spec for the data parser. The first step will be to create a comprehensive list of player-entered text that can appear to other players. A complete understanding of the context surrounding player-entered text will allow me to reduce the amount of context matched by the lexer rules without introducing opportunities for spoofing. This will simplify the hand-coded parsing, reduce the compile time of the lexer spec, and increase runtime performance. Second, I will make a side-by-side comparison of all game text in TWGS versions 1 and 2 and rewrite the lexer rules to handle both versions.

I noticed that I didn't publish the source of the scripts I included with Weapon M. I'm going to correct that oversight as soon as I decide where to put it.

Last but not least, I recognize the need for a document describing how to build Weapon M. Maven is becoming more appealing, but I still need to learn more about it.

Too tired to think of a clever title.

It's been two weeks since I released my projects to the public. It feels more like two months. The new year came with a new job, and I've been spending ten hours a day fixing bugs in other people's code. You might expect that the last thing I'd want to do after stumbling home from the bus stop in the freezing darkness is to fire up the computer and stare at Eclipse some more. Not so; I've actually been eager to start fixing bugs in Weapon M. I just haven't had the time.

SourceForge has been pretty flaky lately, so I'm moving my projects over to Github. I'm also converting them to build with Maven. Automaton, JPlex, and JTX are up already, but I still need to get Prefuse converted before moving on to Weapon M itself.

As expected, there's plenty to fix. Tweety has proven especially adept at finding bugs, and mrdon has already fixed a few of them and even added some new features to his local builds. Thanks to them, I should be able to put out a bug fix release by the end of the month.

Public Release Tonight!

I plan to upload the first public release of Weapon M when I get home from work tonight, sometime after 6 PM MST. There are still many features left unfinished, including a few that I consider essential. But even where it has not met my expectations, it's demonstrated the capability to do so. More importantly, I think I've already achieved my main objective of creating a client that will attract more players to the game. The map alone should draw some interest.

Surprisingly, I've received very little feedback from the project supporters. Either they're not actually using the program, or they're playing their cards very close to their chests when it comes to what kinds of scripts they're working on. I've actually received more comments and suggestions from players who have seen nothing but the early screenshots and scripting tutorials. So I won't be surprised if a large number of bugs are discovered in the first public release.

Going forward, I'll prioritize enhancements and bug fixes based on some combination of what users tell me they care about and what's easiest to deliver. I'll respond to questions here and on the official TradeWars Helpers & Scripts forum, and track bugs and feature requests in the project's ticket system.

Dodging Spoofing Bullets Left And Right

Writing a helper has alerted me to many quirks of the game text that I never noticed as a casual player. For instance, the "Warping to sector..." message does not appear when you move using the 'M' key. And the "Transporter Engaged!" message only appears when you bwarp blind; when you bwarp with a lock, the destination and distance message is repeated instead. These inconsistencies make detecting ship movement far more complicated than it ought to be.

Even sector fighters aren't completely straightforward. Thankfully, fighters that are genuinely yours or your corp's are displayed with a purple owner name. But there are still opportunities for mischief. The color difference is easy to overlook, so a trader named "your Corp" could set personal fighters that would fool many players. And with only one exception that I know of, fighters that aren't yours are always displayed with a yellow owner name. So a player with a name like "The Ferrengi" or "Corp#2, Stupid Corp" could lure players into unwanted alignment changes and keep them wondering who the fighters they encounter really belong to.

If I've learned one thing from this project, it's how to make a text interface a real pain in the ass to parse!

Release Notes, Version 20021212

I sent out another update to the backers a couple days ago. The focus of the next release cycle will be getting planets into the database.

Here are the notes for the latest update:

Tooltip Heaven

I've added some nifty icons to indicate SD and Class 0 ports, and a custom edge renderer to color the warps between Organics-Equipment PPT and Equipment-Equipment SSM pairs so you can identify them at a glance. Sector and warp tooltips also contain handy information. These features will be in the next RC that goes out to the backers. (The map controls don't all work yet, but hopefully they will by then.)

I also did some work this week on JTX, the terminal system I wrote for Weapon M. I expect JTX to be a product of much broader appeal than Weapon M itself, and there's a link to Weapon M in the JTX docs. My hope is that JTX will draw attention to Weapon M and thereby draw players to the game.

Are We There Yet?

I keep making significant improvements, yet I still feel no closer to being "done." I guess that's the nature of a complex project like this. It will probably never be "done." But I won't let perfect become the enemy of good. I'm going to release whatever I've got to the $10 backers tomorrow.

One small improvement I made today was to color only the dashed part of one-ways red, so their direction is more evident when the map is zoomed way out:

I can't get over how sweet the map looks, and I've barely started working on it. Once you understand it, Prefuse is (mostly) a joy to work with, and the rendering is fast and looks so damn good. Static screenshots don't do justice to the zooming, panning, and node-dragging controls that can be added to a Display with just one line of code.

One of the few weaknesses of Prefuse is that it stubbornly holds onto references to everything you put in its Tables, and therefore tends to leak memory. There are dozens of threads about memory leaks on the Prefuse forums. I spent several hours this evening tracking down object references in VisualVM, but I think I've finally got it licked. I had to null all the parts of the map except the Display and the filter and layout actions, call setVisualization(null) on those three things, and then call reset() on the Display before a closed database would be released to the garbage collector.

I'm pretty happy with what I'm seeing in the profiler. With a 1K universe loaded, the app uses about 30 MB of heap, smoothly climbing to around 45 MB before the GC runs. With a 30K universe, heap usage peaks around 90 MB.

I've been so intensely focused for so long on all the little ways in which this project hasn't met my expectations that it's a bit of a shock to step back and look at what I've accomplished. I wrote a freakin' TradeWars helper! If it doesn't work exactly the way I want, I can change it myself. This is something I've dreamed of since I was fifteen years old. All told, it's taken me over fourteen months to get to this point, but most of the work was done in the last three. It probably goes without saying that the best way to find all the little things that need fixed in a program is to actually use it, and I'm looking forward to doing exactly that.

I Can Smell It

It's close. Real close.

It's taken me far longer than I planned, but now it's pretty much down to finishing touches. I'm busting my ass to get the release candidate out to the backers by 1 December so I can release it to the public on New Year's Eve.

Reader challenge: there's at least one bug evident in this screenshot. Can you spot it?

Backer Survey Results

I thought today would be an appropriate day to post the results of my Kickstarter backer surveys. I asked the $25 and $50 backers what stretch goals they would have voted for if any stretch goals had been funded. With 89% of surveyed backers reporting, the winners are the scripting tutorial and essential script packs:

Weighting the results by giving the $50 backers three votes changes the proportions slightly, but doesn't change the outcome.

The good news for the minority is that of all the listed enhancements, TWGSv1 support is probably the most tedious but also the simplest. In other words, it's something that interested third parties could easily contribute to once the source is released. I could write a tutorial describing what needs to be done. As for extracting the terminal component, that's already partly done; it just needs a few enhancements (copy & paste being the big one) and better documentation.

Cake or Death?

I obviously didn't meet my goal of being ready to release to the $10 backers this weekend. However, I did get a new kitten. I'm going to send out one more update to the $25+ backers tomorrow, and hopefully publish to the $10 backers in a few days.

One thing I did this week was to audit the script and database API for naming consistency. For example, instead of referring to "Figs" in one place and "Fighters" in another, now everything is nice and uniform. This may have broken some scripts, but in the long run it will make the API easier to read and learn.

In the process, I caught and corrected a few mistakes in my modeling of the game data, such as psychic probes being an unnecessary property of ship types. I improved the default behavior of Script#getScriptName() and Script#getMenuPath() to better organize scripts for which they're not overridden. And there's one more thing I don't think I mentioned:

I finally finished the default script parameters dialog. Any parameters registered in Script#initScript() are neatly displayed with the appropriate input widgets. As you can see, this user has chosen wisely.

Home Stretch

The core features of Weapon M should be finished within a week. I'm eager to share it with a wider audience and start using it in live games myself!

I've given it the ability to auto-load scripts. I've also changed the script manager so that a script is not unloaded if it tries to load another script and the other script's initialization fails. And I gave scripts the ability to load other scripts by name instead of requiring a compile-time reference to their class. These last two changes should make it easier to write a bot that coordinates many other scripts.

I'm considering a few other enhancements, such as per-database auto-loading and giving scripts a chance to recover if a script they require fails after initialization. But these changes will take a back seat to finishing the core features.