Vizsage

Rails Lessons Learned the Hard Way

2008-01-27T19:42:00.000-06:00

Things I've learned the hard way in Rails:

Layouts run inside views, not the other way round. Set an instance variable in app/views/monkeys/show.html.erb and it will be defined in app/views/layouts/monkey.html.erb but not vice versa.
- set instance vars in view
  @foo_val = find_foo_val
- pass variables to partials using
  <%= render :partial => "root/license", :locals => { :foo => @foo_val } -%>
- use the instance var freely in the layout; it will take the value defined in the view
Dump an object for bobo debugging through the console or log:
$stderr.puts tag_list.to_yaml

In a migration, if you define a unique index on an attribute, make sure both the index AND attribute are :unique => true, or else you'll get no uniqueness validation from Rails:


   create_table  :monkeys do |t|
     # set :unique here
     t.string :name, :default => "", :null => false, :unique => true
   end
   # if you have :unique here
   add_index :datasets, [:name], :name => :name,  :unique => true

If you scaffold a User or other object with private data, MAKE SURE you strip out fields you don't want a user setting or viewing:
- Set attr_accessible, which controls data coming *in* -- prevents someone setting an attribute by stuffing in a form value.
- In each view (.html.erb &c) and render method (to_xml), strip out fields you don't want anyone to see using the :only => [:ok_to_see, :this_too] parameter.
- Set filter_parameter_logging, which controls what goes into your logs. (Logs should of course be outside the public purview, but 'Defense in Depth' is ever our creed.)
Using the the restful-authentication generator as an example:
- In the model, whitelist fields the user is allowed to set (this excludes things like confirmation code or usergroup):
  attr_accessible :login, :email, :password, :password_confirmation
- In the controller file, whitelist only the fields you wish to xml serialize:
  format.xml { render :xml => @user.to_xml(:only => [:first_name, :last_name]) }
- Obviously,In the show.html.erb and edit.html.erb strip out fields that shouldn't be seen.
- In the model file, blacklist fields from the logs:
  filter_parameter_logging :password, :salt, "activation-code"
I won't even tell you how often this happens to me: If you edit or install code in a plugin, restart the server.

Parsing Names with Honorifics

2008-01-27T18:34:00.000-06:00

In Railscast #16, Ryan Bates goes over Virtual Attributes in Rails, using the standard example of storing first and last names but getting/setting full names. He uses the following simple snippet:


def full_name=(name)
  split = name.split(' ', 2)
  self.first_name = split.first
  self.last_name = split.last
end

Which -- given that the focus was on virtual attributes -- is fine for explanation. However, that snippet will fail on names like "Franklin Delano Roosevelt" (last name of "Delano Roosevelt"). Here's a method which our 32d President will like better:


def clean(n, re = /\s+|[^[:alpha:]\-]/)
 return n.gsub(re, ' ').strip
end

# Returns [first_name, last_name] (or '' if there isn't any).
# Leading/trailing spaces ignored.
def first_last_from_name(n) 
    parts    = clean(n).split(' ')
    [parts.slice(0..-2).join(' '), parts.last]
end

names = [
    "Bill! Merkin,PhD.",
    "Jim               Thurston Howell III   ",
    "Charo", 
    "Heywood Jablowmie",
    "Sergei Rodriguez-Ivanoviv",
    "Polly Romanesq. ",
    "   ", 
    "",
    ]
p names.map { |n| first_last_from_name n }
# => [["Bill", "Merkin,PhD"], ["Jim Thurston Howell", "III"], ["", "Charo"], ["Heywood", "Jablowmie"], ["Sergei", "Rodriguez-Ivanoviv"], ["Polly", "Romanesq"], ["", nil], ["", nil]]

A regex is more extensible, and makes more sense for Perl refugees like me.


# Returns [first_name, last_name] (or nil if there isn't any).
# Leading/trailing spaces ignored.
def first_last_from_name_re(n)
    n = clean(n); 
    (n =~ / /) ? (n.scan(/(.*)\s+(\S+)$/).first) : [nil, n]     
end

p names.map { |n| first_last_from_name_re n }
# => [["Bill", "Merkin,PhD"], ["Jim Thurston Howell", "III"], [nil, "Charo"], ["Heywood", "Jablowmie"], ["Sergei", "Rodriguez-Ivanoviv"], ["Polly", "Romanesq"], [nil, ""], [nil, ""]]

However, as someone who can't check in at the automatic kiosks in airports because -- no joke -- the credit card thinks my last name is "IV", I like this version better.


# Returns [first_name, last_name, appendix] 
# (first name and appendix are nil if there isn't any).
# Leading/trailing spaces ignored.
# 
def first_last_appendix_from_name_re(n, appendix = nil)
    n = clean(n)
    appendix_re ||= %q((I|II|III|IV|(?:jr|sr|m\.?d|esq|Ph\.?D)\.?))
    if (n !~ / /) then
        [nil, n, nil]           # with no spaces return n as last name
    else
        n.scan(
          /\A(.*?)\s+           # everything up to the last name
           (\S+?)               # last name is last stretch of non-whitespace
           (?:                  # But! there may be an appendix.  Look for an optional group
             (?:,\s*|\s+)       #   that is set off by a comma or spaces
             #{appendix_re}     #   and that matches any of our standard honorifics.
             )?                 # but if not, don't worry about it.
           \Z/ix).first         # scan gives array of arrays; \A..\Z guarantees exactly one match
    end
end

p names.map { |n| first_last_appendix_from_name_re n }
# => [["Bill", "Merkin", "PhD"], ["Jim Thurston", "Howell", "III"], [nil, "Charo", nil], ["Heywood", "Jablowmie", nil], ["Sergei", "Rodriguez-Ivanoviv", nil], ["Polly", "Romanesq", nil], [nil, "", nil], [nil, "", nil]]

All three versions might make Japanese (and other "FamilyName GivenNames" cultures) sad.

Copyright Disputes are usually Failures of Imagination

2008-01-22T21:15:00.000-06:00

Hasbro is trying to shut down Scrabulous, a successful online Scrabble game -- perhaps the most successful Facebook app to date.

On the one hand, I think that Hasbro is completely within their rights: it's a clear infringement.

On the other hand, it's a departure from form (they've for a long time licensed gray-market implementations), and a failure of imagination that doesn't account for important subtleties in software engineering and social networks.

On the software engineering end, all of the interesting computer Scrabble implementations I know of were created independently and *then* brought into the fold, to both parties' mutual benefit. Hasbro is a board game company: It doesn't, and shouldn't employ brilliant independent software engineers who create a new entry in the scrabble ecosystem. The other thing to note is that Scrabulous solves some difficult problems in a way no previous product has.

Here's a brief history of the important scrabble programs I know of. The first ones let you play against a computer; this requires a powerful artificial intelligence (AI) engine and an unobtrusive interface. (The hard part is the AI; note that Scrabulous was written in Flash, a very constrained programming environment). Maven was the first Scrabble program that played at an expert level (at one time it was the best scrabble player in the world). Though developed independently, it was purchased by Hasbro (or their licensee) and adopted as the AI agent in the official Hasbro Scrabble software. I don't believe that the official software has been updated for some years, it was Windows-only, and the official scrabble site has no link to it. ACbot, another early implementation, was independently developed by James A Cherry and could play at a low-expert level.. A current offering is Quackle, a free scrabble robot developed by a student at MIT. Its AI engine is extremely strong (also one of the best players in the world) and its front end, while /quite/ rough, is useable and works on Windows/OSX/Linux. All of these programs were written outside Hasbro's aegis. They were developed by experts in computer artificial intelligence and game theory and are far superior to anything that was or could be developed in-house by a board game company.

Another approach lets you play against a person using the network in real time. One of the first was MarlDOoM -- a primitive (text only, pre-web technology) free online scrabble bulletin board. It was developed by John Chew, who at the time was simply a scrabble enthusiast but is now on the official Nat'l Scrabble Association's dictionary committee and the webmaster for their site -- I believe that implementing MarlDOoM helped bring this about. There are modern programs and websites that are officially licensed and let you compete remotely. However, their price or subscription fee exceeds the cost of the physical version, and they require that *both* parties pay for the game, which the physical version does not.

A third approach is 'scrabble by mail' -- one move every day or so, with as much or as little time commitment and deliberation as you care to devote. If there are licensed products that allow this I'm not aware of them.

In all, here's what you'd like a compelling software version of a board game to offer:

Play from any computer, anywhere; simple to acquire, install and use.
Reasonable price compared to the physical game
Skill level:
- Enjoyable for an expert player
- Enjoyable for a casual player
- A casual player and a strong player may enjoy a game where their focus is on socializing and not gameplay
Time commitment:
- Play for 10 minutes at a time -- an quick diversion.
- Play for an hour at a time -- a leisure activity
- Play without having to meet at the same time
Social play:
- Play remotely against a friend, in real time (complete a game at one sitting)
- Play remotely in a "Chess by Mail" context: make a move every day or so, when you have time.
6. Competitve play:
- Compete remotely against a skill-matched stranger, in real time or move-a-day
- Track durable competitive rankings
- Tie those ratings to a reputation system to prevent gaming the rating mechanism.

None of the licensed programs or sites, as far as I know, cost less than the one-time-only, one-person-plays price of a physical Hasbro scrabble. Scrabulous is free, requires only a browser, and is available from any computer anywhere. It provides a simple experience that my computer-incompetent mom can enjoy. (As far as she knows, facebook *is* a scrabble program.)

Scrabulous is the first solution that enables me (an intermediate tournament-level player) to play remotely against any of my casual-level friends -- friends would never pay for, or seek out, or regularly visit, a scrabble-only site. My friend Jen lives in Shanghai -- no previous approach that I'm aware lets me play on my lunch break against her on her lunch break,. None let me *easily* discover when a casual friend is on: all require that you go to their sandbox when you want to play, and that all the people you'd like to compete with patronize the same sandbox. None of them let me jump in / jump out for a quick 10-minute timewaster. Since Scrabulous/Facebook is part of a compelling portal, it's natural to check in and meet friends; it understands my social network; and the play-by-turns feature lets my scale the time commitment and schedule.

No previous approach effectively prevent a cheater from manipulating his rating. However, in Facebook you are a person: you have friends, you have a name, you are part of a community. It's still feasible to be a troll or a sock-puppet or any of the other strategies to game or disrupt a community rating, but there are barriers and consequences for doing so.

If Hasbro shuts down -- rather than licenses -- Scrabulous it will be a business failure. They should be ecstatic that people are integrating scrabble into their social lives, and should see a modest halo effect in board game sales. The revenue stream from Scrabulous' share of Facebook advertising is, I believe, quite significant -- enough for Hasbro and Scrabulous to both enjoy while keeping the game free.

More importantly, Social Network research consistently highlights the importance of "Network Effects" in technology adoption (http://en.wikipedia.org/wiki/Metcalfe%27s_law). There are many, many social games on Facebook, and if Scrabulous is taken down the large body of casual users will move to another entry in this niche. After all, these games are only interesting if your friends also play. Any Hasbro implementation must not only match the quality of Scrabulous' implementation, but must build a network of friends who select it for their social gameplay arena -- and they must build that network against the ill-will that will accrue from shutting Scrabulous down.

Creating a software program (and more importantly a community) like Scrabulous has is HARD: look at all of the previous attempts that have failed to get millions of people to play online. It's hard because there are subtle and serious software engineering challenges, and it's hard because there are subtle and serious community building challenges. If Hasbro shuts down the Scrabulous guys there's no reason to think they'll be able to reproduce their success.

The power of a good visualization

2008-01-16T19:22:00.001-06:00

I just found a program called Grand Perspectivethat present your disk usage as an interactive mipmap (see pic on right). Helping web nerds save hard drive space isn't finding hidden heart defects or keeping planes in the air, but I was struck by how well this program demonstrates the power of intelligent data exploration tools. Here are the Tufte criteria for information presentation:

Documentary · Comparative · Causal · Explanatory · Quantified · Multivariate · Exploratory · Skeptical

Each box is a file, and each top-level directory takes a continuous rectangular portion of the view. Scanning a 350GB disk with a /lot/ of tiny files (5+ million for just the far top left corner, the MLB gameday dataset) took < 5 minutes. You may highlight any box in a segment and navigate "down" to make that segment fill the screen, and may choose to color files by location, depth, name or extension (exploratory, multivariate).

The giant orange box in the top left was 15GB of pure junk -- apparently a CGI-script generating some page I was screenscraping went crazy and sent me 15GB of junk data, the same line repeated almost billions of times. I had /no/ idea it was sitting there. That dataset was supposed to be huge, so I had never drilled into the directory beyond my standard du -sc | sort -n on the containing directory. The picture, however, showed at a glance what a table of numbers dramatically failed to do: that the directory consumed twice as much as it should. The simple metaphor of diskspace=area and the whole-disk view (explanatory, documentary) - highlighted something important I'd never noticed. The giant cluster in the bottom right corner is a huge (~51GB) collection of video ephemera I only kinda cared about. I planned, someday, to sort them -- but for that effort and 51GB usage, it was clearly not worth it. By enforcing comparisons, the data display made me reconsider the value vs. resource consumption of that project and make a more sound decision. In all, I freed up almost 100GB and put a few bucks in his tip jar. Joe Bob says Check it out. (Similar programs exist for Linux (Baobab) and Windows (WinDirStat) too.)

The 2007 Feltron Annual Report

2008-01-14T20:08:00.000-06:00

The 2007 Feltron Annual Report is available now. In a series of elegant infographics, see the ambit of places he walked to in Brooklyn and Manhattan, review how many albums Mr. Feltron bought in the year (12 CDs, 1LP and 98 download tracks), and how often he visited bars in October (6 times; he made 57 total bar visits in the year, down 39% from 2006). My print copy is on the way. (last year's report). Metadata is the new Eyeballs (which is the old Interaction).

More things I wish someone else will write

2008-01-08T12:48:00.001-06:00

More random software ideas:

Google search, restricted to find bug reports only. You'd crawl usenet, sourceforge/google code, debian etc. build farms, open issue trackers, mailing list archives and blogs; extract things in 'pre' tags, and look for repeated stanzas (these indicate where bug was pasted in).
NTP server along the lines of the procrastinator's clock, that would dither the time (by extending or delaying each second) by up to a set amount fast, and never slow. You'd have to be careful with rsync, server logs, kerberos/cookie session stores/other authentication... or maybe just use it at the app level, if your clocks will use NTP themselves.

Reference Cards

2008-01-07T09:56:00.000-06:00

Here are some pretty reference cards I made a while back:

Scale Landmarks: What's something you're familiar with that is about 10 nm big? How do the speed of continental drift, a raindrop, a champion sprinter and an SR-71A Blackbird compare? What is the range between the least massive (electron) and most massive (universe) objects science can describe?
Periodic Table
Periodic Table, Flat -- material properties as a table and not as Mendeleev puts it.
Mechanical, Geometric and Material properties of Screws, Bolts and Fasteners - probably the most useful among these, this gives thread geometry, decimal inch/screw/metric equivalents, mechanical strengths, torque ratings and more. Super handy for machining or general shop use.
A similar table for AN Hardware (milspec fasteners used in airplanes, racecars and hot rods).
A flat table of decimal equivalents: decimal and fractional inch, metric, and standard (US) screw sizes.
ASCII Chart - easily index up hex, octal, ascii, symbol font/latin font/DOS font values for characters.

Owning my Metadata

2008-01-06T00:10:00.001-06:00

Dear Lazyweb,

I'd like someone to invent a 'Metadata reclaimer': a program to screenscrape all my amazon ratings, flickr tags, facebook posts, etc.

I try, as far as possible, to only use apps that let me keep ownership of my metadata. As our friend pud has remarked, all successful internet enterprises share the same business model: either

People pay to Enter Data into your Database (eBay, Google AdWords, Flickr, Second Life, World of Warcraft, IMDB pro, Craigslist), or less defensible,
People Enter Data into Your Database For Free while Other People Pay to Get it Out (rapidshare, iTunes Music Store, Pud's Internal Memos; with youtube, myspace, epinions etc viewers pay with the tenuous currency of their ad brain).

There's nothing wrong with that; all these companies levelled their playing field in some fundamental and important way. (Well, nothing wrong unless you're the loathsome gracenote.com (formerly cddb), who turned an open community-generated resource into a closed database, without even the courtesy of a copy to fork from.)

But it's fair to ask that I be able to export my copy of the data I've added to their business asset, and to do so easily.

Sites that play well with others:

my del.icio.us tags and bookmarks
my bloglines/google reader feeds
my librarything.com everything
my last.fm history
my iTunes playcounts, tags and ratings: mostly, I think?
Firefox bookmarks and history

Sites with an 'I gave up my metadata and all I got was this stupid webpage' policy:

facebook posts, friends, photos, everthing
flickr tags &c
amazon recommendations
Google calendar mostly no (at least, the last time I tried to sync my address books it was a Giant Pain in the Ass: nothing was durably id'd and recurring events were semantically incorrect. (Yes, I'd love to have 96 separate entries for my Grandmother's birthday!)
eBay bids, purchases, ratings
Blogger: Blogs, yes if you remote host your site. However, you can't even /list/ the blogger comments you've made, let alone export them.
I believe Myspace's engineers can't even spell XML

(I could be wrong about any of these except the last one).

I'm picturing something with a plugin architecture -- the main app handles the screenscraping, authentication, form submission, web crawling and file export details; the plugin supplies URL wildcards and regexp's the data back into semantic structure. With XML export, a motivated plugin author or well-itched user could supply a decent XSLT stylesheet to represent that metadata in a useful local fashion (and with helpful links back to the main site). It would be useful to have plugins (trivial) and stylesheets (no more or less so) even for sites like Last.fm and Library Thing that Do The Right Thing by granting transparent access to your metadata.

Much of this may exist in some form or another; for example the Aperture/iPhoto plugin will apparently sync your flickr and iPhoto tags, and embed the result into the app database. But going from XML => app is more flexible -- and possibly easier -- than the other way 'round.

I one off'ed this a while back for my Amazon ratings, but I just saw where I'd gone from ~350 to ~650 'things rated' since then. I'm hoping the LazyWeb has solved my problem, since I'm not sure where I put those scripts. (Ironic, considering my previous post.)

50 years of Baseball Play-by-play data mashed with 50 years of hourly weather data FTW

2008-01-05T12:38:00.000-06:00

Note: I found this sitting in my drafts folder, unpublished. It actually dates from October.

I've had two interesting realizations from the Retrosheet Baseball data vs. Hourly Weather information mashup I've implemented. The first is how my two favorite scripting languages (Python and Perl) compare. The second is how the hard parts of this process is actually the stupidest part... there's four steps in doing an interesting visualization of open data. In order of steps as well as decreasing difficulty and decreasing stupidity:

Bring the data from behind its bureaucratic barriers
Unlock it into a universal format
Process and digest the data
Actually explore, visualize and share the data

The hardest and least justifiable steps are the first two, a problem we have to fix.[Edit: this is why I'm starting infochimp.org]

Here's a longer description of how I did the baseball games / weather data mashup.

Several significant parts of this project were written in Perl, for its superior text handling and for the ease of XML::Simple (which I love); several other parts were done in Python, for its more gracious object-orientation.

To suck in the Hourly Weather Data files, you have to click through a 4-screen web form process to prepare a query. Although it sends the final form submission as a POST query, the backend script does accept a GET url (you know, where the data is sent in the URL form.pl?param=val&param2=val&submit=yay instead of in the HTTP request). There's an excellent POST to GET bookmarklet that will take any webpage form and make the parameters appear in the URL. No guarantees that the backend script will accept this, but it's always worth a twirl for screenscraping webpages or just trying to understand what's going on behind the curtain.

Now I need to know what queries to generate. First I needed the location of each major league baseball stadium: Brian Foy posted a Google Earth index of Major League Stadiums, a structured XML file with latitude, longitude and other information. I used the Perl XML::Simple package to bring in this file. These simple routines just pull in the XML files and create a data structure (hashes and arrays of hashes) that mirror the XML tree. The stream-based (SAX) parsers are burlier and more efficient, but for this one-off script, who cares?

Next I needed the locations of all the weather stations. Perl and Python both have excellent flat-file capabilities. The global weather station directory is held in a flat file (meaning that each field is a fixed number of characters that line up in columns). Here's the column header, a sample entry, and numbers showing the width of each field:

USAF   NCDC  STATION NAME                  CTRY  ST CALL  LAT    LON     ELEV*10
010010 99999 JAN MAYEN                     NO JN    ENJA  +70933 -008667 +00090
123456 12345 12345678901234567890123456789 12 12 12 1234  123123 1234123 123456

To break this apart, you just specify an 'unpack' format string. 'A' means an (8-bit) ASCII character; 'x' means a junk character:

A6    xA5   xA29                          xA2xA2xA2xA4  xxA6    xA7     xA6

The result is an array holding each interesting (non-'x') field. The Perl code snippet:

    # Flat file format
    my $fmt    = "A6x    A5x   A29x                          A2xA2xA2xA4xx  A6x  A7x   A6";
    my @fields = qw{id_USAF id_WBAN name region country state callsign lat lng elev};
    # Pull in each line
    for my $line () {
        next if length($line) < 79; chomp $line;
        # Unpack flat record
        my @flat = unpack($fmt, $line);
        # Process raw record 
        ...
    }

I also grabbed the station files for Daily weather reports, since that data goes back much farther (generally, we have since ~1945 for Hourly and since ~1900 for Daily).

Then I score each station by (Proximity and Amount-of-Date), and select the five best stations for each stadium.

Now, I could of course use Perl to generate the POST request using the HTTP modules, but it was simpler to mindlessly just control click on a dozen links at a time and then answer each form. and spit out an HTML file with a big matrix of URLs for each station, for a subset of years. P (You can see the linkdump file here: http://vizsage.com/apps/baseball/results/weather/ParkWeatherGetterDirectory.html)

I also use perl to clean up the XML generated by the MySQL Query Browser -- which returns a flat XML file with all fields as content, not attributes. I just suck the file in with XML::Simple, walk down the resultant hash to create a saner (and semantic) data structure, then spit back out as XML.

The python parts are not terribly interesting. I pull in the flat file, clean up a few data fields and convert in-band NULLs into actual NULLs (they use 99999 to represent a null value in a 5-digit field, for instance) then export the data as a CSV file (for a MySQL LOAD DATA INFILE query). I chose python for this part because I find its object model cleaner -- it's easier to toss structured records around -- and the CSV module is a tad nicer.

The idea I find most interesting is that we're starting to get enough rich data on the web to make these cross-domain data mashups easy and fun -- I did all this in less than a week. With the effortless XML handling and text processing of modern scripting languages (and relieved from any efficiency concerns) it's easy to see forward to a future where we'll have all these datasets sitting at our fingertips. This data set lets you examine ideas such as "How does the break distance of curveballs change with atmospheric temperature and pressure for a full baseball season?" "Effectiveness of pitchers against gametime temperature, stratified by age of pitcher or inning?" "Batting average on fly balls vs. ground balls against % of total cloud cover?". It's easy to come up with a variety of other "This Rich Dataset vs. That Rich Dataset" opportunities. Stock price and Earnings of Harley-Davidson vs. average household income, unemployment and percent of the population that has reached retirement age? Year-by-year movie attendance at comedies compared to dramas, Attendance at Baseball Games, and Sales of Fast Food vs. Consumer Satisfaction Index, national Suicide Rate, and Persons treated for mental health/substance abuse? Presidential approval rating vs. gasoline prices and Consumer Price Index? Amazon.com sales rank, # mentions on Technorati blogs and # of mentions in mainstream media vs. time?

The hard part is actually the stupidest part: to unlock the data from behind bureaucratic barriers (the first script I described), then to convert into a universal semantically rich data format (the second set of scripts I described). Once one person has unlocked this data, however, it's there for the whole world to enjoy, and tools will evolve to capitalize on this bounty of rich, semantically tagged and freely available information.

Time Machine is neat-o, but I want a Time and Space Machine for my files

2008-01-03T22:36:00.000-06:00

I've long wished for a versioned home directory, but the svn/ish seem too heavyweight, and it's nice to have a live copy and not an opaque DB-ball. The right answer the stunningly elegant Time Machine. The idea isn't actually new, and it can be approximated across platforms and remotely using standard Unix tools. Now you just need a landing spot.

Some remote backup solutions have come to the fore lately. $Zero/year gets you 2GB remote backup from Mozy. $60/year buys unlimited backup space at Mozy, where 'unlimited' means the ~40 GB/month you'll see by leaving your pipe fully saturated 24/7. A price between free and $100/year gets you the more intriguing CrashPlan. These are slick and easy. If you don't know what ssh is, you want one of these. If you do have ssh and you don't set your mom's computer up with the free Mozy account then you're a bad person.

For the uber-uber-nerds, use what I'm using: an $85/year bluehost account. You may think of it as a 600GB ssh'able rsync'able remote backup host that, by the way, can also act as a webserver. You can install svn (as I have) to do versioning over svn+ssh. Two years of bluehost costs the same as a 500GB hard drive+cheap enclosure, and they regularly increase your diskspace allowance at the same monthly price. [Note: the preceding bluehost link will give me a kickback if you sign up through it. Hit bluehost.com directly if that rubs you the wrong way.]

As described below, all my various project shards get sync'ed back to my desktop PC. The desktop then pushes my files out to a bluehost account via rsync, versioned with an rsync-as-poor-man's-time machine script. This gives me a live, versioned backup, accessible to me from anywhere by ssh, on Bluehosts' offsite, secure, RAID-UPS-and-diesel generator protected colocation, and at the end of a fat pipe. After about a week or so for the initial ~50GB backup to roll in, daily incrementals will take an hour or two each day (bandwidth choked to 25 kBps). When I leave town for a week next month I'll start pushing my music collection into its own unversioned directory.

This is only for the stuff I've created or can't replace: not for system software and not for music/movies/media (apart from my iTunes.xml, inbox, and various bookmarks/pref's/stickiesDB/MySQLCachedQueries folders). Unlike most people, I don't worry too much about backing up my system software. Maybe I'm damaged from my Windows upbringing, but just reinstall from scratch if your OS gets hosed (I keep install disks and images around). The nascent defect may be present in the restore; the accumulated OS cruft certainly is. You're already kinda screwed; better to take a certain two days and finish with a clean system than to fight a flaky restore and then spend those two days again. Yes, you are allowed to come point and laugh if this happens to me.

Beyond the backup, I have various levels of defense-in-depth for my data -- data that is created, changes daily and is essential for my economic well-being has four or more levels of redundancy. Data that is intransigently huge but can be sourced elsewhere has no redundancy. (There's no reason backing up my processed wikipedia dump, for instance: only the scripts that process it.) Right now my fileverse is spread across

Desktop computer ~ 1TB
Flivo (my homebrew Tivo computer ~0.5TB
School account ~3GB
sourceforge account
Four webservers holding sites I operate or caretake ~ 1-5GB each
GMail account ~4GB
Flickr account ~6GB
iPhone, Google Calendar, Yahoo Address book, Plaxo
blogger/twitter/facebook/etc

This is organized as

All the stuff I'm currently messing with is in a 'now' folder. This is what sync's to my school account, and every time I go on a trip I burn a DVD of the now folder to toss in my backpack. (You never know when you'll want a file, and it enforces an occasional hardcopy backup).
Within the now folder, the stuff I develop for work is versioned with svn. I house a private repository on my bluehost account and connect over svn+ssh. I'm reasonably good about checking in every few hours or when I shift conceptual gears.
Most of the 'now' folder I keep in sloppy sync with the school account. ('sloppy' because I sync when I think of it and not through a cron script as I should).
Each year I move everything that isn't under current work out of the now folder and into an 'archive/YEAR' folder; there it sits and changes almost never.
GMail holds all my mail (sync'ed with IMAP)
Flickr holds an incomplete and poorly correlated segment of my photos (they own my metadata, and yes that bugs me).
iPhone sync handles the address book; gcal is still quite difficult.
The rest of the little metacontent is trapped, meh.
Each webserver's content is replicated to the desktop. For small changes I'll sometimes diddle the file on the live server and then sync back later (tsk tsk); for heavier work I proceed locally and then deploy.

The usage breaksdown like this:

== Work space -- changes ~ daily ==
  2   GB         vizsage project software       Desktop, svn, school, vizsage, bkup
  9   GB         infochimp site & working data  Desktop, svn, school, infochimp, bkup
== Work resources -- changes ~ weekly ==
100   GB         'huge' datasets                Desktop, infochimp
 60   GB         live local DB of datasets      Desktop
 30   GB         infochimp website DB           infochimp, bkup
== Slowly changing -- changes ~ semiyearly ==
  3   GB         Other projects, docs, stuff    Desktop, bkup
  4   GB         Archive - doesn't change       Desktop, bkup
                 (~ 300 MB / year for 11 years) 
  3   GB         Library (prefs,caches,etc)     Desktop
== Metacontent -- changes daily-weekly ==
 12   GB         Photos                         Desktop, Flickr
  3   GB         Mail                           Desktop, GMail, bkup
  ~   MB         iPhone/Addr Book/Calendar      Desktop, iPhone, GCal, Yahoo AddrBk
  ~   MB         this blogger blog              vizsage, Blogger
== Websites ==
  1   GB         website1                       Desktop, website1, bkup 
  2   GB         website2                       Desktop, website2, bkup
  6   GB         website3                       Desktop, website3, bkup
== Media ==
many  GB         music                          Desktop, some on iPhone, some on DVD
many  GB         recorded tv shows, movies, etc Desktop, data DVD
== System Software ==
some  GB         OS & installed programs        on each machine

I'd like to live in a world where I wouldn't have to worry about how these are partitioned across machines. Changes made to 'website1', say, or to a project in 'now', would lazily propagate to each interested shard as well as to the remote time-machineish versioned backup. At any time I could force an immediate sync, whether to deploy a change, to repair a mistake, or to satiate an OCD twinge, if I don't want to wait for automatic syncronization.

I'm actually pretty close to having this out of a McGuyvered patchwork of rsync, svn, time machine, IMAP/Aperture+Flickr and distributed file systems, all enforced by cron. I'm planning to soon waste a weekend buttoning up my sync scripts, getting everything to run daily and being superattentive in case I screw it up.

But it sure would be nifty to augment (a cross-platform) Time Machine into a Time and Space Machine. I'd see an overview of my distributed fileverse (versioned in time, distributed in shards according to how I use it), and I could delegate various live realizations, svn/diff-versioned backups or hard-link-versioned backups to each local or remote instance. No single machine would necessarily hold the entire fileverse: note that a few things up there don't propagate back to my main desktop. And hopefully the whole thing would have polished Apple Fit And Finish instead of Mad Max Homebrew Itworksithinkihope.

Old-School Shop Guide

2007-12-13T09:51:00.001-06:00

ShopGuideFront300dpi.png

I rediscovered this super-compact reference-and-tool-and-measuring device while looking for a tool. It is jam-packed with handy information for anyone doing things mechanical or woodworking. I got this from a family friend of a family friend -- I bought their lathe after her husband, an avid (and skilled) woodworker, had passed away. She wanted his tools to go to someone who would love them and use them, which was me, which I do. The lathe is good, but I've discovered after the fact that the throwins were the best part. The chisels are *top notch*, but still pale in comparison to getting "his old woodworking magazines." This turned out to be almost every issue of Fine Woodworking magazine, beginning in its first year of publication; somewhere in the stack was this nifty Shop Guide.

ShopGuideBack300dpi.png

I think this thing is so neat -- so much information in such a small space. My own mechanical data reference table (more here) has more numbers but less intrinsic functionality ... What's really neat about this shop guide is how they used the shape of the guide itself as a tool. Print this onto heavy cardstock and punch brother punch with care... enjoy!

Leveraging the Bittorrent Underground for semantic data and media

2007-12-13T00:33:00.000-06:00

I just ran across a pretty interesting site called coverbrowser.com, which uses a variety of image APIs to pull in comic book, game, book, music, movie and other cover art. (Read the technical details here).

It reminded me of an idea I had while back but which I will never get around to implementing --- maybe you will, or for all I know someone's already been doing for years. (Sidenote: I've had some people express interest in this, and have worked out some parts of it, but just don't have the time to complete it right now. If you'd like to help develop it get in touch).

Many of the movie and music torrents on the, ahem, "Unauthorized Evaluation Copy" bittorrent sites contain hi-res scans of their cover art, and all of the major bittorrent sites maintain topic-specific RSS feeds.

As long as the torrent indexes the files individually (as not as an opaque .zip or .rar) -- and most do index individually -- you can target specific files within the torrent. I don't know whether you could chop all the large-file-size copyright-problematic files that you don't want out of the torrent, or whether you'd have to hack Azureus or other bittorrent client (instructing it to get only *.{png,gif,jpg,jpeg,bmp,tiff,tff} or what have you). Either way, you would then only be pushing out the bandwidth required to grab the photos and not the accompanying multi-megabyte file, and you would only be getting the information to which you assumedly have fair use rights for.

So you'd set up a daemon process that would

watch the Movies and the Music RSS feeds off whichever or all of the sites,
identify albums whose cover art you lack,
pull in the bittorrent,
but download only the cover art
and perhaps also process any of the accompanying semantic data

You might have to get yourself a seedbox to make this work, but they're not unaffordable.

I think this would lead to a large stream of incoming cover art for music and other media files, complete with a reasonable amount of semantic information.

There's probably a lot of other crowdsourced semantic data flowing through the underground, if someone actually created such a torrenting robot. (And yes, I feel yucky using "crowdsourced" and "semantic data" in the same sentence).

Moving from Perl to Python with XML and Templating

2007-12-05T16:17:00.000-06:00

Mr. XKCD is correct in this. (My friend Dr. Larsson has been saying this all along). As I'm moving from data munging to data working-with, I've been moving from perl to python. Recommended:

lxml is a beautiful interface for dealing with XML in Python. You get XPath and validation and namespaces and all that hooha but you don't have to think hard and you don't have to write SAX stream parsers or walk a DOM path. You just say crap like

from lxml    import etree   
from urllib2 import urlopen
# Load file
uri   = "http://vizsage.com/apps/baseball/results/parkinfo/parkinfo-all.xml"
parks = etree.ElementTree(file=urlopen(uri))
# for each park (<park> tag anywhere in document)
for (idx, park) in enumerate(parks.xpath('//park')): 
  # dump its id, time of service and name (@attr is XPath for 'corresponding attribute')
  print ' -- '.join(
    [ s+': '+','.join(park.xpath('@'+s)) 
      for s in ('parkID', 'beg', 'end', 'games', 'name',) 
    ])

and you get this in return

parkID: MIL01 -- beg: 1878-05-14 -- end: 1878-09-14 -- games: 25   -- name: Milwaukee Base-Ball Grounds
parkID: MIL02 -- beg: 1884-09-27 -- end: 1885-09-25 -- games: 14   -- name: Wright Street Grounds
parkID: MIL03 -- beg: 1891-09-10 -- end: 1891-10-04 -- games: 20   -- name: Borchert Field
parkID: MIL04 -- beg: 1901-05-03 -- end: 1901-09-12 -- games: 70   -- name: Lloyd Street Grounds
parkID: MIL05 -- beg: 1953-04-14 -- end: 2000-09-28 -- games: 3484 -- name: County Stadium
parkID: MIL06 -- beg: 2001-04-06 -- end: NULL       -- games: 486  -- name: Miller Park

lxml.objectify is the replacement for perl's XML::Simple we've all been looking for. You just say gimme and it pulls in an XML file as the corresponding do-what-I-mean data structure (identical elements become arrays, tree leaves become atoms, tree structures become maps).
Kid Templating is a great solution for XML transmogrifying, and I think I like it much better than XSLT. It looks perfect for your "Anything => XML" purposes, which is the hard part. I suppose XSLT can do the "XML => anything" tasks but those always look like stunts; the whole point of XML is that "Turn XML into whatever" tasks are easy, especially given a simple API like lxml or lxml.objectify.

Hourly Weather data for each Retrosheet game

2007-10-26T18:03:00.001-05:00

I noticed some suspect entries for game conditions in the eventfiles and realized I could not only fix it but add a pretty useful dimension to the retrosheet collection. The National Climate Data Center makes available "Global Hourly Surface Data" -- several dozen physical and observational characterizations of the current weather, taken hourly. This data goes back to the forties and sometimes to the start of the century.

Please enjoy this preliminary dataset giving the hourly weather data for each game in Fenway since 1957: http://vizsage.com/apps/baseball/results/weather/

(open the WeatherData-BOS07.* file of your choice) I don't have all the data in hand yet, but I thought I'd get your thoughts and see if anyone would like to help with some of the drudge work.

I'm excited about doing some fun things with the data, like see knuckleball effectiveness vs. humidity or elderly pitchers vs. temperature. Combined with the MLB gameday pitch trajectory info you could do physics "experiments": show the break distance of all curveballs vs. atmospheric pressure.

Email me back if you're interested or with comments.

-----------------------
DATA FIELDS AVAILABLE
-----------------------


The fields I've spit out are

-- game_ID, gamedate, gamenum_in_day, start_time, daygame_flag from
the cwgame output.
- temp deg C
The temperature of the air in degrees Celsius.
- press_atmos HPa
The atmospheric pressure at the observation point.
- press_sealvl HPa

The air pressure relative to Mean Sea Level (MSL).
- press_altim HPa
The pressure value to which an aircraft altimeter is set so that it
will indicate the altitude relative to mean sea level of an aircraft
on the ground at the location for which the value was determined.
- press_chg_3hr_del HPa
The absolute value of the quantity of change in atmospheric pressure
measured at the beginning and end of a three hour period.
- press_chg_3hr_obs --

The code that denotes the characteristics of an
ATMOSPHERIC-PRESSURE-CHANGE that occurs over a period of
three hours.
- wind_dir deg
The angle, measured in a clockwise direction, between true north and
the direction from which the wind is blowing.
- wind_obs --
The code that denotes the character of the WIND-OBSERVATION.
- wind_speed m/s

The rate of horizontal travel of air past a fixed point.
- wind_gust_speed m/s
The rate of speed of a wind gust.
- cloud_cover_low (frac)
The code that represents the fraction of the celestial dome covered
by all low clouds present. If no low clouds are present; the code
denotes the fraction covered by all middle level clouds present.
- vis_dist m
The horizontal distance at which an object can be seen and identified.

- sunshine_time min
The quantity of time sunshine occurred over the reporting period.
- wea_pr_m_obs_1 --
The code that denotes a specific type of weather observed manually.
- wea_pr_m_obs_2 --
The code that denotes a specific type of weather observed manually.
- wea_pr_m_obs_3 --
The code that denotes a specific type of weather observed manually.
- groundcond --

The code that denotes a type of Ground condition
- precip_hist_contin bool
The code that denotes whether precipitation is continuous (true) or
intermittent (false).
- precip_lq1_depth mm
The depth of LIQUID-PRECIPITATION that is measured at the time of an
observation. Unit:Millimeters
- precip_lq1_period hours
The quantity of time over which the LIQUID-PRECIPITATION was measured.

---------- WHAT I DID ----------

I used Brian Foy's Google Earth index of Major League Stadiums:
http://www252.pair.com/comdog/google_earth/major_league_baseball_stadiums.kml and the NCDC ISH-HISTORY file (gives locations for each weather station) ftp://ftp.ncdc.noaa.gov/pub/data/inventories/
to find the closest station with continuous data. (Turns out I could have saved a ton of trouble by just using the nearest airport -- in almost every case it was the best match.)

Then I pulled down data sets from http://cdo.ncdc.noaa.gov/pls/plclimprod/poemain.accessrouter?datasetabbv=DS3505 (If you're interested in replicating any of this I have a script that sends a GET url to help automate the weather data collection.) The last step is to match games with stadiums with locations, and dates and times with hourly observations.

I could be clever and subtle and use the start time and game duration to grab only the hours of gameplay, but instead I just pull in the records from 10:00am to 11:59pm for day games, and 5:00pm to 11:59pm for night games. I suppose I'll fix it to see if a game overhangs midnight and get the post-12am data for those only.

----------------------- WHAT YOU CAN DO TO HELP -----------------------

Geolocation for the rest of the stadiums

Inspect the data for consistency and correctness

If you have access to a computer at a .edu or .k12.us, or fancy GIS data, help me grab the rest of the weather files.

Email me if you'd like to help.

Retrosheet Eventfile Inconsistencies II

2007-10-26T18:01:00.000-05:00

I've found a few more inconsistencies and minor inaccuracies in the retrosheet event files and game logs.

I made a diff (applied using the 'patch' tool) to mechanically recreate these corrections: http://vizsage.com/apps/baseball/results/rseventfiles_20070923_patch.diff

I pulled these out by whipping up a few simple scripts (one-liners, mostly) that extracts all unique values for each event file field. For example, the only values for the "info,pitches" field are 'count, 'none' and 'pitches' -- just as promised in the documentation. The "info,temp" field, however, has not only normal temperatures ("78", or "104", or "0" for [unknown]) but also spurious values of '670' and '700' (wrong), '8/7' (ill-formed) and '' (differs with the format documentation).

I'll posting all the dubious entries (event files version 2007 Sep 23) I find at http://vizsage.com/blog/2007/10/retrosheet-eventfile-inconsistencies.html as comments.

==================== Incorrect Data ====================

In 1993MIL.EVA:
info,start,spieb001,"Bim| Spiers",1,9,4
should be
info,start,spieb001,"Bill Spiers",1,9,4

These temperatures need fixing:
1988MON.EVN,info,temp,670
1988MON.EVN,info,temp,700
1964NYA.EVA,info,temp,8/7

I looked at a few suspiciously short games (< 60 minutes):
This should be 1:58, according to the NYT box score:

http://select.nytimes.com/gst/abstract.html?res=FB0614F73D59107B93C4A8178FD85F4C\
8585F9
1958BOS.EVA,info,timeofgame,58
These two are correct:
1971BAL.EVA,info,timeofgame,48 BAL197107300 -- Game called due to rain
1976BOS.EVA,info,timeofgame,57 BOS197609100 -- Game called due to rain
Another thing to look at would be suspicious game length/number of
outs ratio, but I haven't done this yet.

I also checked a few games with attendance below 1000, but these seem
to be very cold or rescheduled days. I'll taka a peak sometime soon at
"game attendance less than two and a half standard deviations from
that year's average attendance" to see what sticks out. (I also
peeked at 2.5+ above -- those look like bandwagon game)

==================== Badly Formatted ====================

These are probably correct but just ill-formatted:
1959CHN.EVN,info,timeofgame,0158
2001PIT.EVN,info,attendance, 34915
1962BOS.EVA,info,daynight,day,
1966ATL.EVN,info,howscored,"park"
1966HOU.EVN,info,howscored,"park"
1970CHA.EVA:data,er,roung101,4#
1958PIT.EVN:data,er,wills102,1y

In these files, the "howscored" field is spelled "howentered":
1990BOS.EVA,info,howentered,game
1990DET.EVA,info,howentered,game
1990DET.EVA,info,howentered,game
1990DET.EVA,info,howentered,game
1990DET.EVA,info,howentered,game
1990HOU.EVN,info,howentered,game
1990HOU.EVN,info,howentered,game
1990LAN.EVN,info,howentered,game
1990MON.EVN,info,howentered,game
1990MON.EVN,info,howentered,game
1990PIT.EVN,info,howentered,game
1990SFN.EVN,info,howentered,game
1990SFN.EVN,info,howentered,game
1990SLN.EVN,info,howentered,game
1990TEX.EVA,info,howentered,game
1990TEX.EVA,info,howentered,game

There are no "info,edittime" records -- is this purposeful?

==================== Inconsistent with Documentation ====================

In the 2003TBA.EVA file, the umpires are given by name and not by ID.

These are supposed to use 0 as the unknown value but in a few places
use a blank.
1990NYA.EVA,info,temp,
1978ATL.EVN,info,attendance,
1978NYA.EVA,info,attendance,
1979SDN.EVN,info,attendance,
2000PIT.EVN:info,windspeed,

There are some "info,ump[...],(None)" fields, and there are some
"info,ump[...]," fields. Does one indicate "unknown" and the other
indicate "none"? Or is this a formatting inconsistency?

These files have a bunch of "info,windspeed,unknown" fields (the dox
say "An unknown windspeed is indicated by -1."):
1969ATL.EVN 1969HOU.EVN 1969MON.EVN 1969PIT.EVN 1969SDN.EVN
1970ATL.EVN 1970HOU.EVN
These files have an "info,temp,unknown" field (the dox say "An unknown
temp is indicated by 0."):
1969ATL.EVN 1969HOU.EVN 1969MON.EVN 1969PIT.EVN 1969SDN.EVN 1970ATL.EVN
1970HOU.EVN 1990NYA.EVA

These lines have trailing spaces, which is harmless but still
shouldn't be there:
1958CHA.EVA:info,save,
1957BOS.EVA:com,"xwas a lot of action. Had this game been played
today, it no doubt"
1957BRO.EVN:com,"$In addition to 12,559 paid, 6000 knothole,"
1957CLE.EVA:com,"xCC4 changed E9/F.2-3;BX2(9)# to 9/F.2-3(E9)#"
1957MLN.EVN:com,"xCC4 per film, TSN 26 is DP"
1958CLE.EVA:com,"$ Strong wind to left; cool"
1958KC1.EVA:com,"xScoresheet scores DP as 142. I Checked with newspaper"
1958NYA.EVA:com,"$Total attendance: 13323"
1958SFN.EVN:com,"$paper box and Cin s/s has Cepeda and Sauer reversed"
1958SFN.EVN:com,"$paper box has stats that match SF s/s not Cin s/s"

Here are all the well-formed windspeed values:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
23 24
25 26 27 28 29 30 31 32 33 35 36 37 38 40 59 60 66 67 68 69 74 78 87
What are the units on these? If this is in MPH, 39 is Gale force
("Difficult to walk against wind. Twigs and small branches blown off
trees."), 55 is Storm ("Trees uprooted, structural damage likely") and 64
is ("Trees uprooted, structural damage likely").

Here are games with windspeeds over 40:
id,CHA197408270|windspeed,67
id,MIN198008190|windspeed,87
id,TOR198208030|windspeed,68
id,CHN198307042|windspeed,74
id,TOR198307270|windspeed,87
id,LAN199006050|windspeed,78
id,DET199506160|windspeed,87
id,CLE199609141|windspeed,69
id,COL199606150|windspeed,59
id,DET199704300|windspeed,66
id,TEX200104220|windspeed,40
id,SLN200610010|windspeed,60

The SLN200610010 event file gives a wind speed of 60mph (from baseball-reference and ESPN), but a) that's crazy and b) the weather report from that day doesn't confirm it:

http://www.wunderground.com/history/airport/KSTL/2006/10/1/DailyHistory.html?req\ _city=NA&req_state=NA&req_statename=NA Which gives 83F, 9mph SSW wind, clear

See also my next message, about getting weather data for each game.

The BGAME.exe documentation says "WindSpeed: 0 Unknown, 1 Known, other value is the wind speed" but I think it should be "WindSpeed: -1 Unknown other value is the wind speed in miles per hour".

The Asdrubal Carrera Hall of Fame

2007-10-26T04:33:00.000-05:00

Inspired by one of Tim McCarver's flights of fancy during the ALCS, I present The Asdrubal Carrera Hall of Fame, open to anyone in unique possession of a particular first name among Major League baseball players: LIST.

You may be familiar with Honus Wagner, Eppa Rixey, Boog Powell or Yogi Berra. But have you heard the storied diamond exploits of Firpo Mayberry, Zoilo Versalles, Pi Schwert or Bevo LeBourveau? OK, then how about Mysterious Walker, The Only Nolan, or Phenomenal Smith?

For some dinnertime fun over the holidays, discuss the relative merits of Urban Shocker, Twink Twining, Pussy Tebeau, Bris Lord, Boob Fowler, Crazy Schmit, Creepy Crespi, Cuddles Marshall, Vinegar Bend Mizell, and Buttercup Dickerson. (Unfortunately, 12 other players keep Rusty Kuntz off this list.)

Other stunningly yclept combatants include Ambiorix Burgos, Alamazoo Jennings, Welcome Gaston, Chicken Hawks, Sixto Lezcano, Wheezer Dell, Yam Yaryan, Yo-Yo Davalillo, Admiral Schlei, Boss Schmidt, Brick Smith, Brickyard Kennedy, Broadway Jones, Cannonball Titcomb, Baby Doll Jacobson, Sweetbreads Bailey, Zaza Harvey, Bubbles Hargrave, Pickles Dillhoefer, Double Joe Dwyer, Cowboy Jones, Coot Veal, Mul Holland, Live Oak Taylor, Skyrocket Smith, Kaiser Wilhelm, Kewpie Pennington, Possum Whitted, Snooks Dowd, and Mox McQuery.

See the list for links to each player's Baseball Reference page. Nerds may additionally view the generating mySQL query here.

Retrosheet Eventfile Inconsistencies

2007-10-17T17:01:00.000-05:00

Here are a few inconsistent records in the retrosheet.org event files of 2007 Sep 23. I'm using chadwick and not the retrosheet DOS utils, but I think I've source all these to the original event files. Weird Attendance in gamelog GL1941.TXT:

  WS1194107220 (WS1 vs DET) has '1500 e' as its attendance

Weird Start Time in eventfiles: Many daynight records lack an AM or PM. I assume the time mapping of times are as follows:

   daynight  start_time   24hr Time
   D or N    0            Unknown
    D        1000..1259   1000h to 1259h
    D        100..459     1300h to 1659h
    N        500..1150    1700h to 1359h

In that case, here are some weird start times reported by cwgame:

  - Negative start time:
      2003 D 0  -195 SEA 2003 04 15        SEA200304150    info,starttime,-2:05PM   info,daynight,day
  - No daynight flag:
      1998 D 0   506 LAN 1998 08 30        LAN199808300    info,starttime,5:06      -- no daynight --
  - Plainly inconsistent daynight flag:
      1985 D 1   605 CIN 1985 06 21        CIN198506211    info,starttime,6:05PM    info,daynight,day
      1960 N 0   135 BOS 1960 04 19        BOS196004190    info,starttime,1:35PM    info,daynight,night
  - Second half of a double header, listed as a day game despite 5pm or later start:
      1966 D 2   507 BAL 1966 10 02        BAL196610022    info,starttime,5:07PM    info,daynight,day
      2001 D 2   500 PHI 2001 05 27        PHI200105272    info,starttime,5:00PM    info,daynight,day
      2001 D 2   519 PIT 2001 06 03        PIT200106032    info,starttime,5:19PM    info,daynight,day
      2001 D 2   625 MIN 2001 05 26        MIN200105262    info,starttime,6:25PM    info,daynight,day
      2001 D 2   719 CHA 2001 09 04        CHA200109042    info,starttime,7:19PM    info,daynight,day
      2001 D 2   738 CHN 2001 08 20        CHN200108202    info,starttime,7:38PM    info,daynight,day
      2001 D 2   752 PIT 2001 09 03        PIT200109032    info,starttime,7:52PM    info,daynight,day
      2001 D 2   753 SLN 2001 08 03        SLN200108032    info,starttime,7:53PM    info,daynight,day
  - Start times that appear to be after midnight (this could be correct):
      1996 N 1    35 CIN 1996 06 25        CIN199606251    info,starttime,0:35      info,daynight,night
      1998 N 0   105 LAN 1998 06 13        LAN199806130    info,starttime,1:05      info,daynight,night
      1966 N 2  1207 BAL 1966 06 08        BAL196606082    info,starttime,12:07AM   info,daynight,night

These eventfile games have more than one "info,daynight" record

  ATL197004150    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197004160    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197005260    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197006191    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197006192    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197006200    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197006210    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197007031    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197007032    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197007050    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197009220    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197009230    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197009240    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197009250    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197009260    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  ATL197009270    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197006220    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197008031    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197008032    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197008040    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197009010    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197009110    info,starttime,0:00PM   info,daynight,day       info,daynight,night
  HOU197009130    info,starttime,0:00PM   info,daynight,day       info,daynight,night

This eventfile game is missing an "info,daynight" record:

  LAN199808300    info,starttime,5:06

File Structure in eventfile 2001HOU.EVN:

  2001HOU.EVN lacks a trailing newline (unix commands hate this).

Here are the unix commands I used to dump all that info. Sorry for the one-linerism.

# How many have a negative starttime?
grep 'info,starttime,-' *.EV*

# How many have missing or extra "info,daynight" fields?
# -- pull out the info, daynight and starttime records in order
# -- slurp the whole file as one giant string with internal linebreaks;
# -- split each stretch following an id,XXXX record into one line
# -- dump lines that have none or more than one daynight record
  cat *.EV* | egrep '^(id,|info,daynight|info,starttime)' | \
    perl -e '$_ = join(" ",<>); s/[\r\n]+/!!!/g; @games= (split /id,/, $_);
      shift @games;
      for $game (@games) {
          $game =~ s/!!!/\t/g; print "$game\n" if (($game !~ m/daynight/) || ($game =~ m/daynight.*daynight/));
      }'

# How many have a start_time and daynight_flag that disagree?
# -- use cwgame to pull off the gameID,start_time,daynight_flag records;
#    put it into a temporary file    
# -- Use a big stupid regex to find
#    . start_time that is >  500 and marked day
#    . start_time that is <  500 and marked night 
#    . start_time that is > 1200 and marked night 
#    . start_time that is <  100 
#    . start_time that is negative
( for ((year=1957;$year<=2006;year++)) ; do \
     for teamfile in ${year}*.[Ee][Vv]* ; do \
     cwgame -y $year -f '0-0,4-4,6-6' $teamfile 2>/dev/null ; \
     done; \
  done ) > /tmp/starttimeIDs.txt
cat /tmp/starttimeIDs.txt | \
  perl -ne '(m/"(\w\w\w)(\d\d\d\d)(\d\d)(\d\d)(\d)",(12\d\d|[1234]\d\d|\d\d|[1-9]|-\d+),"(N)"/ ||
    m/"(\w\w\w)(\d\d\d\d)(\d\d)(\d\d)(\d)",((?:5|6|7)\d\d|.*-.*|\d\d|[1-9]),"(D)"/)    &&
    printf "%s %s %5d %s %s %s %s\n", $7, $5, $6, $1, $2, $3, $4;' | sort

Rules of thumb for Rack Leave in Scrabble

2007-09-07T16:02:00.000-05:00

This isn't exactly within the ambit of this blog but at least it's about data. While I should have been doing work, I instead made an awesome spreadsheet to find rules of thumb for what the best Scrabble rack leaves are. (rules of thumb below, tables here: http://vizsage.com/other/scrabble/RackLeaveRules.html) The computer program Quackle is one of the strongest scrabble players in the world. It uses the following 'Superleave' valuation:

http://web.mit.edu/~jasonkb/Public/scrabble/superleaves
(warning: huge-assed file)
http://vizsage.com/other/data/superleaves.xls
(The above, sorted by value, only up-to-4 leaves)

To find "synergies" and "anti-synergies" (dysphoria?), I calculated the marginal valuation for each combo. Basically, how much of the value for each two-leave is explained by the valuation of the component one-leaves, etc? For example,
- From S (7.35) and M (0.08), the joint valuation of MS is 7.44, a marginal gain of 0.1: the joint valuation is almost entirely from S&M. (<-- will lead to interesting google hits). This combination has no synergy.
- From Q (-9.0) and U (-5.1), the joint valuation of QU is 0.2, a marginal gain of 14.3. This is by far the largest synergy; next is ZO at +3.2.

I also played with three-letter synergies -- 3-leave valuations marginally different from the most explanatory 2-leave. General Lessons:

Get a feel for the 1-leave list, and the learn these:
- Synergy: QU OZ JU CH GN WY IN DE JK ER EV GIN JKY JKU ERS KWY HWY ?IN EST JOW ?AL ?EL ?IL IST
- Anti-Synergy: BP CG FP MV PV CW CQ QS SX LQ BV SZ QR BC CZ VZ MQ RX GQ + most things with blank BPV CGQ BCG LQR FPV LNQ SVZ CMQ CLQ BCV BNV KTV LMQ GKT CFV GMQ FSV LNR DGT
Worth keeping with a blank: The letters in "Lei an orc DTM" + the following digrams IN AL IL EL CI AN ER EN AC AR IT NO QU ET DE CO AT OR LO GN OT AM DI CE IM IR DO MO GI AB AG
double letters are bad (duh), except FF, which is good.

See the spreadsheet at http://vizsage.com/other/data/superleaves.xls). Don't go betting the house on these results....

Tables (including 1-tile-leave values) are available here.

as3mathlib (formerly WIS math libraries)

2007-09-01T02:49:00.000-05:00

I've just imported the WIS mathematics library -- an excellent collection of mathematics routines -- onto Google Code. (You'll find the Actionscript 2 version of the library at its original site)

This library carries a BSD-ish license and includes support for

Geometric Objects and Intersection calculations
Integral and Differential equation calculations
Bezier, Quadric, Polynomial, Complex, Vector and Matrix calculations
Symbolic expression parsing

I'm converting the library to Actionscript 3 from Actionscript 2 as time and necessity allow. (That's converting as in getting it to work, and converting as in getting it to be object/pattern oriented). Right now it builds without errors and only a few warnings, but I haven't applied any of the unit tests or checked it for correctness or compatibility.

If you see the value of updating this well-thought out collection of functions, please get in touch and I will add you as a developer. The code is quite modular: it will be straigforward to take modest chunks and get them working independently. I wrote the original author and maintainer, who responded "By all means, continue in the evolution/integration of my library to support AS3" -- but please let me know of any other efforts to update this code, or if a similar or superior math library exists, so that I don't waste my time :).

Email me [flip at the mrflip with the dot and the com] or comment on this post if you'd like to pitch in!

Subway Geography and Geometry

2007-08-27T10:06:00.000-05:00

I've written an applet that lets you reimagine the geography of greater Washington, DC area with "distance" measured by subway-travel-time, measured by subway-travel-cost, or as the standard clarified subway wall map would deform it. This was in large part inspired by Oskar Karlin's beautifully rendered Isochronic Elephant-Castle map of the London Underground and the interactive tube mapplet from Tom Carden. Subway Maps of the world all on the same scale is pretty interesting, as is this directory of remixed London Underground maps. There's a few interesting images on wiki commons, like this geographical map within this gallery. Also, you can download the image files (very large, register with each other) from Wikipedia:

Patches to the AS3 Cookbook Code

2007-08-24T15:08:00.000-05:00

The Actionscript 3 Cookbook is a very helpful reference, and the example code that came with it has many good examples. Unfortunately there's a modicum of bitrot in the code: compiler warnings and errors when compiling under strict mode.

Here is a patch against the version of the code I downloaded on 2007-08-24:

files/AS3CB-patch-2007-08-24.diff

Apply it thusly.

How to make a patch using diff

2007-08-24T10:44:00.000-05:00

I always forget the command to use, and google is strangely devoid of helpful/correct advice. Therefore I'm posting this for my own and future generations' reference.

How to Generate a Patch from Standalone Code

For non-(svn|cvs),

You should have a directory holding the original source (i.e. "dir") and a directory holding the modified (i.e. "dir-orig").
Obviously, don't modify dir-orig (that is, it should match the author's). If you don't trust yourself, do a chmod -R a-w dir-orig to recursively mark the directory read-only.
Generate the patch by going to the parent directory (holding dir-orig and dir) and running the command
diff -Nuwr dir-orig dir > /tmp/my-happy-patch.diff
- dir and dir-orig should be paths to the dirs in question, obviously.
- -N creates newly added files (treats absent files as empty files)
- -u creates a "unified" diff -- it's hunam readable and works well with patch
- -w ignores whitespace, which is polite if your (clearly superior) formatting policy differs from the original.
- -r recursively descends the source tree.
Other helpful options:

-p (applied to C or C++ code) shows the function the new code appears in. Only use this with C or C++ code (i.e. YMMV)
-u6 (or any number following the -u) gives that many lines of context (the default is 3, which should be fine for code that isn't changing like the star of a Tootsie stage performance)
-x ".??*" ignores .DS_Store, Eclipse and other hidden-file turds, if you have those. Emacsen should add -x "*~".

Sanity-check the change less /tmp/my-happy-patch.diff

How to Generate a Patch from Subversion or CVS

To generate a patch from cvs or svn, (advice horked from X-Smiles.org):

Make sure you are synchronized with the latest sources: $ cvs update src (or wherever your changes are; use a directory that spans all the changed modules or the trunk directory.)
Sanity-check the change: $ cvs diff src
Generate the patch (replace -Nuwr with whatever you decided works for you from the options above). $ svn --diff-cmd=diff -x-Nuw src > /tmp/my-happy-patch.diff
(Rather than include -x "" args, you should be adding turd files to your ~/.subversion/config, for instance global-ignores = *.o *.lo *.la #*# .*.rej *.rej .*~ *~ .#* .DS_Store or however you pronounce that in a ~/.cvsrc or .cvsignore.)
Sanity-check the patch less /tmp/my-happy-patch.diff

(By the way, if your broken OS lacks a command line, you might be able to add one).

How to use a Patch

To apply such a patch,

Download the patch and save it somewhere intelligent.
If you're not using version control, !!make a copy of the source tree!!
Change directories into that new folder --the one holding the unmolested (or svn trunk) code.
Sanity check: run the command cat /tmp/my-happy-patch.diff | patch -p1 --dry-run
If you see results like patching file foo/bar/my-happy-file.as ... you're good to go: cat /tmp/my-happy-patch.diff | patch -p1
Pitfalls:

If you get a patch taken from within the modified directory, change -p1 to -p0.
If you get a patch with the original and modified dirs reversed, add a --reverse flag.

Flex Demo: Matrix Math (and an error in the Actionscript docs)

2007-08-20T12:09:00.000-05:00

I'm working on something that uses (an algorithm similar to) texture mapping, for which I want to precalculate the .invert() of a whole bunch of .transform.matrix objects. I'll post something about that in the next coupla days. Meanwhile, I found something perplexing in the Actionscript documentation but the possibility exists that I am just a dope so please point out an error in my reasoning. As you may know, you can represent any arbitrary combination of 2-D scalings, skews, rotations and translations using standard matrix operations. The Actionscript docs for the Matrix class mention this in passing, but has elements .b and .c switched: it should be

and not .

I whipped up a MatrixMathDemo in flex to demonstrate the issue:

and a writeup on Mathematical matrices and the actionscript Matrix transformations [PDF]. The 2d, 3d, 4th tabs compare the Matrix methods concat(), invert() and (deltaP/p)ointTransform() respectively with an explicit calculation of the corresponding Matrix operation: they show that in fact the documentation has b and c switched. (The code is free to reuse or modify (but give credit) in case that's useful.) Some references:

This flash-specific tutorial at senocular.com is good, though the matrices are transposed from what is typically presented.
The posts on the flashcoders list A little matrix.invert() mystery and followup are explained by b and c being switched; they link to post on a french blog that is helpful if you parlez.
To brush up on matrix mathematics, please see the Matrix and Quaternion FAQ or Wikpedia.

Emacs modes for Flex

2007-07-25T04:51:00.000-05:00

Emacs modes for Flex:

XML: nXML-mode for Emacs from James Clark Using Emacs for XML documents
Actionscript: actionscript-mode.el for editing actionscript files in emacs.
At least right now it seems you want this xml mode and this actionscript-mode.el.

Then, add

(setq auto-mode-alist (append (list
 '("\\.as\\'"   . actionscript-mode)
 '("\\.\\(xml\\|xsl\\|rng\\|xhtml\\|mxml\\)\\'" . nxml-mode)
 ;; add more modes here
 ) auto-mode-alist))

;;
;; ------------------ Magic for XML Mode ----------------
;;

(setq nxml-mode-hook
    '(lambda ()
 (setq tab-width        2
       indent-tabs-mode nil)
       (set-variable 'nxml-child-indent     2)
       (set-variable 'nxml-attribute-indent 2)
       ))

You can use M-x customize-group RET nxml-highlighting-faces RET to fix your colors the way you like 'em.

Setting up asdoc to work within Flex Builder:

First, install ant support (Ant is an offshoot of apache and is like Makefile only more betterer.)
Then set up a build.xml in your docs/ directory to build the documentation set.
I had to modify mine a bit: I added <property name="Templates.dir" location="${FlexSDK.dir}/asdoc/templates/"/> <arg line='-templates-path ${Templates.dir}'/>

I also linked the flex home to a no-funny-characters dir:

    ln -s "/Applications/Applications/Adobe Flex Builder 2" /work/ProgramStores/Flex
   cd /work/ProgramStores/Flex
   ln -s "Flex SDK 2" FlexSDK

Then I exported the location for the asdoc file:

    export FLEX_HOME=/work/ProgramStores/Flex/FlexSDK

or else I got the error message:

    Exception in thread "main" java.lang.NoClassDefFoundError: Flex

Adobe Flex and Custom Namespace / manifest.xml

2007-07-24T00:54:00.000-05:00

Create a file "manifest.xml" and add the following:

<?xml version="1.0"?>
<?componentpackage>
<?!--
URI http://vizsage.com/vzg
namespace gg
package com. vizsage
-->
<component id="widget1" class="com.vizsage.controls.widget1"/>
<component id="widget2" class="com. vizsage.controls.widget2"/>
<?/componentpackage>

Notes:

The class part should give the full path (with / turned into .) to the corresponding .as files.
You don't need one <component/> for each .as file, just one for each component.
The comment part, like most comments (and many goggles), does nothing: all you need is a <component/> for each widget.
You don't have to follow the tld.domain.clevername.widgetname format, but it's what all the cool kids are doing. Just make sure the dotted path matches your files' path.
The dotted path and the namespace URL don't have anything to do with each other.
In fact, the namespace URL is completely made up: it doesn't have to exist; the compiler doesn't look for it; hell, adobe's URL doesn't even exist. It's just a tag for uniquely identifying a namespace. All that matters is that the namespace in your compiler flags and your mxml files match up.

If you use Flex Builder, go into the library project's properties, into the "Library Compiler" field -- add the namespace and manifest.xml into the respective fields. If you use the standalone package, you'll have to add an option for the Component Compiler

-include-namespaces="http://vizsage.com/vzg"  -namespace "http://vizsage.com/vzg" manifest.xml

You need to include the namespace and define it
The -namespace flag takes two arguments (a namespace and a manifest.xml)
The URI here has to match the ns:URL in your .mxml file.

Now your .mxml files (which can be anywhere, and not in that project) start off like

<?xml version="1.0" encoding="utf-8"?>
<mx:Application
xmlns:mx="http://www.adobe.com/2006/mxml"
xmlns:gg="http://vizsage.com/vzg"
layout="absolute" width="100%" height="100%"
viewSourceURL="srcview/index.html">
<?!-- ... your mxml file ... -->

Make damn sure the xmlns URI matches what you used before. I spent 30 minutes figuring out that http://www.vizsage.com/vzg and http://vizsage.com/vzg weren't the same thing.
In Flex Builder 2, you need to get your project's properties, go into "Flex Build Path", then the "Library Path" pane, and "Add SWC" (the one you built with your custom components).
For the command-line tools, add a flag
```
-library-path+=/abs/olute/path/to/library.swc
```
Make sure that's a += there.
Either way, applications (as opposed to libraries) don't need any compiler flags or manifest.xml nothing. The library uniquely identifies itself within a namespace, and provides files in the right .com.foo.bar hierarchy. When your .mxml file (asserts a namespace) and (includes the file) everything turns out right.

For more about namespaces see here, with one caveat: I think you're better off using the -load-config+= trick (to just tack on your changes) than hacking stuff into the main flex-config.xml file.