ID3A update

At this point it's all in the hands of the beta testers.
One found an issue with XServer and FXRuby on MacOS because I had set the row header width to 0.

I confirmed on my eeepc and changed it to 1 pixel.
Problem finished.

The expectation is that I have a tech demo tomorrow so I pushed through the big changes version v.02 which boiled down to a restructuring of the layout, more unit tests, and a batch query process.

On the Drails front, I've renamed it to FXCess.

Using the template that I started with PaperTrade, I was able to quickly build some unit tests which confirmed that I can not only use ActiveRecord to interact with a small sqlite3 db but, also build using TDD which, for a project in which I have a very clear idea of what's going to happen is the fastest way to build.

So far:

C:\fxcess\test>ruby test_all.rb
Loaded suite test_all
Started
.....
Finished in 0.531 seconds.

5 tests, 7 assertions, 0 failures, 0 errors

ID3A is born

Well, after several weeks of hitting my head against the wall of my imposed deadline, I am releasing ID3A (ID3 Analyser) to beta testers tonight.

There are a lot of lessons learned in this first rush to beta and I'm going to jot them down now so I don't forget:

• Ruby rules in just about every way. The ease of the programming language provided me the flexibility to bounce the code around and find out where things went wrong faster than any other language I've used before.
• While I was kvetching about the lack of an IDE last time, I have to point out that FXRuby is so straightforward that one only needs to use google and the API to put pieces together. Case in point, I needed a file dialog form and initially I was afraid that I would have to cobble one together but, the FXFileDialog widget was already created and packed more functionality than I really needed.
• requires matter - as in where to put the 'require' statement. This should be top level and make sure that the "require 'fox 16' " is the top line. Otherwise you'll have a nice mess in the stdout as FXRuby reloads your requirements a second time
• Ocra is a funny beast. It's a handy replacement for rubyscript2exe now that the latter is broken due to some changes in the Ruby internals however, it has some of it's own "gotchas" such as: location matters when compiling the Ruby application. Since I was coding on a virtual Windows XP system, I found that if I compiled on the desktop, I could only run it that desktop. Finally, I compiled in the root folder (C:\) and found that it would work in my Vista system's desktop.

I'm releasing a beta first as I want to take another whack at the code and "pretty it up" before releasing the actual source. Since I'm using three OSS apps (ai4r, fxruby, ocra) it feels correct to release ID3A's source code at some point.

I'm grateful that one of my beta testers is none other than Sergio Fierens (author of ai4r) and I found him to be a great resource for a functionality that I wanted to add to my app which wasn't explicitly built into ai4r.

Right now the majority of the beta testers are stock market traders so I expect that there is a strong probability that there may be changes to the application skewed towards that contingent.

While this might seem a far afield from 'rails' it's not - I am beginning to expect that it would be a very simple thing to 'port' RoR onto the desktop.

I hereby dub that project "Drails" (Desktop Rails).

Beta Cut 1 of ID3 app

Getting laid off was something that really impacted my development process as I've had to focus the marjority of my time on finding another job.

While that has not borne fruit as of yet, I have completed two drafts of a white paper on creating classification trees (a la ID3) and thinking about how to build a Windows app in Ruby.

Thankfully, there is FXRuby - which once one gets over the annoyance of there not being a sensible IDE to build the interface, the actual coding is pretty straightforward. Coming from a background of creating GUI interfaces in VBA for 8 years, it's a new experience to have to manually type out the location, properties, and relationships for each widget.

Fortunately, Ruby gets the backend portion right making anything cobbled together in Visual Basic (including .NET) to be fairly lacking in robustness.

Having looked at a few of the GUI kits in Java, I am also thankful that the author of FXRuby has adhered to Ruby standards and not twisted it into some Ruby looking, Java accented monstrosity to build around. There are enough Java coders out there doing nasty stuff to Ruby already as it is.

The actual compilation to a Windows app is much easier with Ocra than the previous rubyscript2exe and allows for some niceties such as bundling in specific dlls. Once this project is out to the beta testers, I'm going to take a whack at creating a generic database creation tool with FXRuby and sqlite3. Sort of a poor man's Access if you will.

A slight Change in Dev Environs

Vista 64bit doesn't seem to like me at the moment as I am having issues installing software.

Fortunately, for the project and the timeline I have, I can easily switch to Linux as I am doing currently for the sole purpose of adding graphs.

Now this causes a problem in that I wanted to port this app using the Ruby2exe application, and while that application will create a Windows exe from within Linux - the problem is that the graphing was initially going to be done with Gruff and that requires ImageMagick to be installed in one form or another on the target system.

So, I'll have to root around for another graphing subsystem or perhaps use jRuby (which I would like to avoid at all costs because it's not portable in a way that is simple for the end user).

New Code
module StatTools
def sum(seq)
total = 0
seq.each do |line|
total += line
end

def mean(seq)
return sum(seq) / seq.size.to_f
end

def moving_average(seq, num)
total = []
pad = num - 1
1.upto(pad) do |line|
total << 0
end
num.upto(seq.size) do |line|
total << mean(seq.slice(line - num, num))
end
end

A "hackish" data loader

If I'm going to be dealing with whacks of data, I want a fairly clean way of loading the data into the database in the format I specified.

So, while the first order of business would be the creation of the PaperTrade class (which will be the first gem I release out of this work), I wanted a quick and dirty way to load the files so I had to do a few things first.

0. Create the table on the database.

• So, I created a sqlite3 file called "db.db" (because I like palindromes)
• I used the table definition below (at some point I'll come back and create a migration for this
• CREATE TABLE stocks(
  id integer primary key,
  name varchar(20),
  date integer,
  open float,
  high float,
  low float,
  close float,
  volume integer,
  adj_close float);

1. Creation of the Data Loader class.
There are a few notes to this:

• I use a couple of modules to abstract out the ActiveRecord calls so that the loader script which creates the DataLoad class will not have to deal with anything - the only part that is a bit unwieldy is that there is an "include Tables" to pull in the creation of the abstracted Stocks class. On the whole, I believe there should be a more elegant way of dealing with this but for now, it works. The only time I think it will bite me will be if I build more on to this so that multiple tables are created. That could get overly messy
  
• The date format that the csv data I receive back from Yahoo! is DD/MM/YY which apparently is what it thinks all Canadian localised date formats should be. So, I had to create a twist date method to deal with that and it's highly idiosyncratic to my uses - you may not get the same mileage if your date data does not comply with that format. Also, there is a bit of a "Y19K" thing going on so don't import any data prior to 1919 or use this class after 2019 ;-)
  
• Since I felt so "clever" with my Tables include, I decided to do one for the require statements.
• module Requires
  require 'activerecord'
  end
  
  module Tables
  class Stock < ActiveRecord::Base
  end
  end
  
  class DataLoad
  include Requires
  def initialize(data_file_load, stock_name, db_file_name)
  @file_load = data_file_load
  @name = stock_name
  @db_file_name = db_file_name
  @adapter = 'sqlite3'
  end
  
  def connect_to_db
  # Connect to a database
  ActiveRecord::Base.establish_connection(
  {:adapter => @adapter,
  :database => @db_file_name})
  end
  
  def twist_date(stringer)
  months = ["", "jan", "feb", "mar", "apr", "may", "jun", "jul",
  "aug", "sep", "oct", "nov", "dec"]
  split_string = stringer.split('/')
  short_date = split_string.last
  if short_date.to_i > 19 then
  return_date = '19' + short_date
  else
  return_date = '20' + short_date
  end
  return [return_date.to_i, months[split_string[1].to_i], split_string[0].to_i]
  end
  
  def data_file_load
  @data_load = []
  File.open(@file_load) do |file|
  while line = file.gets
  @data_load << line.chomp!
  end
  @headers = @data_load.shift
  @headers.gsub!(/\"/,'')
  end
  end
  
  def csv_to_db
  @data_load.each do |line|
  nter = Stock.new
  nter.name = @name
  date_twisted = twist_date(line.split(',').first)
  nter.date = Time.local(date_twisted.first, date_twisted[1], date_twisted.last).to_i
  nter.open = line.split(',')[1].to_f
  nter.high = line.split(',')[2].to_f
  nter.low = line.split(',')[3].to_f
  nter.close = line.split(',')[4].to_f
  nter.volume = line.split(',')[5].to_i
  nter.adj_close = line.split(',')[6].to_f
  nter.save
  end
  end
  end #class End

At this point in time, I haven't written any unit tests but that should be in an upcoming post.

2. The data loader script:

• # This is a script to load tables to the database
  
  require 'data_load_class'
  
  puts "enter the name of the csv file to load"
  data_file_load = gets.chomp!
  
  puts "enter the name of the identifer for the stock ex: USO"
  stock_name = gets.chomp!
  
  puts "enter the name of the database file"
  db_file_name = gets.chomp!
  
  a_data_loader = DataLoad.new(data_file_load, stock_name, db_file_name)
  
  a_data_loader.connect_to_db
  
  # We can only create the tables after we've connected
  # to the database
  include Tables
  
  # Get the data
  a_data_loader.data_file_load
  
  # Load 'em up
  a_data_loader.csv_to_db
  

Since I'm blogging the code as it exists today, there are a bunch of unpolished bits.

The parts I'm most unhappy about are:

• The whole creating the ActiveRecord Stock class. I did quite a bit of reading on how to do this and wasn't pleased with any of the current solutions. I think Rails has it's own way of doing this so that's a study task for me.
• Creating my own Y19K bug wasn't too pleasing but, I wanted to deal with the data as I was receiving it and not have ot 'normalise' the data through OOCalc or Excel if at all possible.
• Lack of unit tests - I recognize this is a bit askew from my last post but, the fact of the matter is that I have not yet internalised the whole TDD because I have not made up my mind as to whether or not BDD (a la RSpec) makes more sense to me.
  

See you soon.

A (thumbnail) functional design

See-ell-o (I keep toying with the name) will have two fronts but with one core.

The two fronts will be:

1. Utilizing Rails framework aimed at the inter(tubes)
   
2. Utilizing FXRuby and focused on Windows(tm) and Wine(tm) targets
   

The back end will start out as sqlite3 but may morph into mysql on the web end all things depending on how the development goes.

Rationale
The functional purpose is to leverage as much of the ActiveRecord goodness there is because, for the most part there will be one gargantuan table to pull from as I do not intend to over normalize one bit.

DB Schema (version 0.0)
Let me explain, since the application will be creating various options for reversal to mean genomes in a GA solution candidate, there will not be any real need to create a table for each equity so, for all purposes there will be one table with the following structures layout:

• id
• name
• date (in unixtime)
  
• open
• high
• low
• close
• volume
• adjusted close

The first release will be primary a wizard which walks the user through the process of importing a csv file (from either google or yahoo) and loading up the database with that information.

Then selections such as:

• population size
• generations
• cross over
• mutation
• reproduction selection (how much % of qualified candidates will breed)

and then the application will rip into the data creating the all the necessary ancillary data (moving day averages) on the fly and calculate the fitness function for each candidate.

Fitness function will operate as follows:

1. A class called "Paper Trade" will be created using data from test portion of data range
   
2. The candidate will take the starting equity and using buy and sell rules trade the "tape"
   
3. If the candidate finishes the "tape" with a positive equity then it will survive, if it is negative, then that is immediate grounds for removal as a breeder
   

Finally, (either using Ruport or Gruff) a report will be generated with various levels of peformance metrics.

Testing
I will be using rcov to identify where to build the tests.

So, I intend to provide unit tests to keep my skills sharp in that endeavor but I will not be bound by them except at the point where I will be releasing this application as a gem (for desktop) or a Rails app (for the burgeoning Web 2.x).

Development Tools
My primary development environment will be my Windows Vista 64bit Home Premium system. While I realise this might not be the most sexy environment (aka a Macintosh) or the most robust (Linux Mint excellence) - the fact of the matter is that there are specific challenges I mean to address for those Ruby on Windows users.

Editor => irb, Notepad++
Database => Sqlite3

notes:
I use irb as my main development tool. By adding the irb-history module (notes here) and overworking the Marshal.dump and Marshal.load features - I am almost able to reach the Lisp goodness that allows the entire state to be saved and loaded for development.

Since, irb is my main battleground and joy - it would be overkill to actually install Eclipse/Netbeans/Aptana Studio on my system and I would find it very counter productive.

sqlite3 on Vista 64 bit:
If you found this blog by using the above line as a search criteria, then something has probably gone very wrong with the Google Spider.

What I can tell you, hapless wanderer is what worked for me was to create a directory on the C: drive called sqlite and inside it put both the .dll and the executable.

Then I simply added the folder to the path and voila - instant sqlite goodness on my Windows machine. (It took me the better part of an evening to finally figure that one out)

Why Windows?
Persistent bugger aren't you?
Well, here are a few thoughts on the whole Ruby/Windows mess:

1. Everyone knows that Ruby runs very slow on Windows.
2. Except me - the first production quality app I developed was deployed on Windows and it was a major fail. It wasn't until much later that I discovered this is a long standing problem with Ruby.
   
3. At the end of the day, there is a magnitude or order more Windows users than the other platforms combined so not optimising the code for that platform would be akin to biting off my nose to spite my face.
4. Knowing how Ruby works on Windows should make me more adept at working in multiple environments. With the current economic circumstances, it seems prudent to understand as many possible variations as I can.
5. Also, by supporting Wine as a platform, I can write one set of code and deploy on either Mac or *nix at will.
   
6. I'm not an OS elitist. Case in point, when I was spending my days as a graphic artist on Macs, I was spending my evenings tinkering with Windows 95/98 shareware to do similar work and on weekends was taking classes that used Amigas.
   

Back to the lab.

Let Me Introduce Myself

This blog will be about my adventures in Rubyland, with and without Rails.

And all the other stuff that goes along with trying to create an application that will work the same way offline as it does for an online user.

The application will be called see-ello and, according to the design documents I have in my notebook, it will be a Genetic Algorithm application that will be used to determinethe most successful "reversal to mean" methodologies for trading the SPY ETF.

Here's some definitions for the first time reader:
Genetic Algorithms => wikipedia link
Reversal to Mean => investopedia link

About Me

I am a professional Ruby on Rails developer working in an industry that neither uses evolutionary computing technologies nor exists within the financial trading industry.

That's about all I will say about my professional life here. My opinions are my own, just as my personal life is different from my professional life. Ruby is a passion for me and I am very happy to work with it and be paid to do so but, I have found the best philosophy for myself and for any employer that I have worked for, is to maintain a fundamental firewall between the two
domains.

Yo hablo un poquito Español per no intendo escribir acqui en esta idioma.

Well, that's about it for now.