Rhotoalbum 0.8

A new version of Rhotoalbum, a simple but powerful photo album generator written in Ruby, has been released. The version 0.8 contains a number of features (well, after two years) like RSS support (try with cooliris), options per album, pagination, cool thumbnail effects for photos and album covers and improved album cover managing. The project web page [...]

Command Line Options Parsing in Ruby

One of the most common ways how to control a command line application is to use options, e.g. almost every command line application would understand -h or --help or /? (common in the MS-DOS world) as a request to show some instruction. Ruby (but also other programming languages such as Java) has a basic support for [...]

Access Flickr with Ruby and Flickraw

Flickr is a well known site for sharing photos and it is one of the most popular sites that published their services and allow to use them freely. It is not necessary to work on a photo uploader, photo printing site or a mashup solution to become an user of the services. Sometimes it is interesting [...]

Losing Faith

Lately, I've been going through a lot of frustrations with Ruby (by ruby I don't meant just the language but the whole ecosystem and infrastructure based on Ruby). I am in a finishing phase on several projects. It's a time when the finer details get addressed. When there's no more time to do it later, differently or drop it all together. It's when everything that is supposed to work should actually work. The problem with Ruby is that it's been in this kind of finishing phase pretty much ever since I started using it (2004). If you look at it from far enough it seems like there is everything in it but when you come closer and actually try using it you'll find out that a lot of it is in less then pre-alpha stage (of course labeled as production). No language is an island unto itself. No matter how great the language you probably don't want to create all the basic libraries from scratch.

When I made a switch in 2004 Ruby was fairly new, Rails was fairly new so I expected this kind of problem, but you could feel the change in the air. Pretty much all the industry heavy weights were talking about it - Rails conf had a better key note line up than any other conference around, number of books was written and the whole Ruby ecosystem was created over almost overnight. Sad truth is that many of the heavy weights are still talking about it today but they're yet to do their first project in Ruby / Rails. Despite the explosion like surge in popularity little has changed in terms of actually usable Ruby libraries (actually I have the same feeling about 1.9 as well). This is a stark contrast to some of the uncelebrated languages out there. For example I find myself writing increasingly more and more code in Python. I don't think Python is the next Ruby, but I was quite surprised how easily it addressed things I struggled with for 5 years in Ruby.

An example could be excel libraries. I still use the same excel libraries I did 5 years ago (for export I switched recently to xlwt, for import because of the changes in roo I've been using JXL for years now switching to xlrd). The export is great only it doesn't support much formatting and breaks on spreadsheets over 10MB (actually somewhere over 6MB) and the files sometimes don't work on later versions of MS Office. There's been a rewrite that is able to go over 10MB but still without formatting I would need and with a license that doesn't allow commercial use. Python's xlwt - despite a very crude and unpolished appearance is able to produce a complete excel spreadsheet with all the formatting (numbers, dates) (including page setup) and all that in much shorter time and with less resources (Ruby 1.8 4m 28s, Ruby 1.9 2m 36s and Python 33s). I have quite a few more examples but I really don't mean this as a Ruby - Python comparison. This experience, however, got me thinking if Ruby is right for the job I've been trying to do (ie uncool enterprise systems). There are pretty much 2 things for me in Ruby (okay - 2.5 the 0.5 being RubyMine - I'll write about it in some other post). One of them is expressiveness of the language and the other is Rails. Ruby as a medium is incredibly expressive, reads almost like English and allows to do pretty much anything (iterating/mapping an array in 1 line, etc). The down side - as Robert C. Martin put it - is that it's really easy to make a mess. After several months of going through somebody's else code I couldn't agree more. A nice entanglement of block and yields and procs can challenge most of the PHP spagetti code any day. While this expressiveness is incredibly powerful, many times I opt for more verbose solution - Java programmer's code on Monday morning as Gregg Pollack would say - that I will be able to understand 2 years down the road - or that my junior developer will understand.

The other thing in Ruby for me is Rails. The only thing with Rails is that after the years it's becoming a bit too entangled, while it made a lot of sense initially to bundle everything together (ORM, routing, templating, etc.) initially now it starting to be a reason why Rails slowly falls behind in certain areas (either that or Rails team has simply decided to focus on Web 2.0 start up market and let others handle enterprise problems). As surprising as it may seems the world around has not been snoozing after Rails was created and many people took the best ideas (just like Rails did from J2EE world) replaced what didn't work and the whole industry moved forward. While few years back Ruby and Rails were pretty much necessities now there's a lot of alternatives and in many different language. To channel out my current frustrations I started experimenting with quite a number of frameworks and platforms. After so many years I really know what I want and how it fits with my existing projects and infrastructure. Very briefly -
  • I am looking for a modular framework where I can choose different ORM, different templating engine and so on where each component is developed independently by people who understand that specific area. As each of my project is very different sometimes I need the flexibility to change components. It was a great thing back in my Java times (Spring + Hibernate + Webwork). 
  • I really need an enterprise friendly dynamic language. Life's just too short to be wasting time trying to force a language to do things it's inventors don't consider important. 
  • I need a basic ecosystem of libraries (from XML, JSON, through excel, PDF, OO to image manipulation). 
  • I need a framework that supports TDD out of the box. 
  • I need a platform with (at least some) formal education and certification available. I need it because I employ people and certification really helped me back in Java times. To pass even the first Java certificate required people to study and to really understand the language and they simply took it as an independent measure of their skills. 
  • I need a language with a proper IDE. I use Vim every day - BUT not for programming. I've been using RubyMine for more than a month now and I feel like a blind man that got his eye sight back.
  • I need reasonable deployment option - both scalable and resource efficient. 
  • Increasingly more important - I don't want the language/platform community to offend others
It's almost uncanny how many items point back to Java :-). Actually, it's kind of sad that things I took for granted 5 years ago in Java are still not available in the most celebrated and most wildly endorsed language of today. If you look at it - nothing really substantial changed in the way we do the backend (MC) part of the systems. A lot has changed on the View side, but the models and controllers use the same principles as years back. I've used Python/Pylons quite a lot lately and it scores quite high on a number of areas (only so-so on the IDE, though). It seems much more mature compared to Ruby with more matured libraries supporting the enterprisey stuff (so ostracized in Rails community). On the other hand it's marketing is nowhere near Ruby's - you don't really hear people talking about wishing to work in Python. I am a bit worried about the performance - but that's something I just have to try on a real project. A big issue still remains the education and certification. I went through this with Ruby already - no matter how easy and great and superior the language you still need to spend some time learning and then even more time working in it before you really understand it. While there's quite a number of books on Python not much of training courses (at least not in Singapore) and there doesn't seem to be any certification around.

Extracting Data From MS Sql Server on MacOS

I've had really a rough time lately getting data from legacy systems based on MS Sql Server. In each case I got only the .bak backup file and had to do the rest. The whole thing is a bit easier if you run everything on windows, but as I did the development on Mac I'll cover how to connect to MS Sql from MacOS as well.

Restore database from backup

You will need a windows machine with MS Sql Server running (of course :-) Luckily, you can download it for free from here. Next connect to the database server using the command line utility osql (you will need to have it in your PATH)
cd C:\Program Files\Microsoft SQL Server\MSSQL\Binn
osql -E
You may need to create a new user:
use master
go
EXEC sp_addlogin 'peter', 'pass45'
go
Now let's have a look at the backup file (put it in some easy location so you don't have to type out the path):
Restore FILELISTONLY FROM DISK='c:\AMS.bak'
This query allows us to find out the logical name of the database and log file which is needed to appropriately restore a database to a new path. What you get is basically an original location of the data and the log file. In my case I got:
AMS_Data      D:\Program Files\Microsoft SQL Server 2000\MSSQL\data\AMS_Data.MDF
AMS_Log      D:\Program Files\Microsoft SQL Server 2000\MSSQL\data\AMS_Log.LDF
Which means that the original installation was on drive D:\. As my temp server is on C: I will have to recover with changing the locations of the files
RESTORE DATABASE ams
FROM DISK='c:\AMS.bak'
WITH
MOVE 'AMS_Data' to 'C:\Program Files\Microsoft SQL Server\MSSQL\Data\ams.mdf',
MOVE 'AMS_Log' to 'C:\Program Files\Microsoft SQL Server\MSSQL\Data\ams_log.ldf'
go
The only important thing here is that 'AMS_Data' and 'AMS_Log' names match the ones from the previous query. Now you should hopefully see that your database has been restored. Now, will just get access to the database:
use ams
go
EXEC sp_grantdbaccess 'peter'
go
sp_addrolemember db_owner,peter
go
There's more info on how to use osql here. Now we should be set to connect to the database from development machine. This was quite a number of steps just to restore the database - compared to createdb tmp; psql tmp <>Connecting to MS Sql from MacOS We will need to install and set up a couple of things:
sudo port install rb-dbi +dbd_odbc
sudo port install freetds
set up connection to database
/opt/local/etc/freetds/freetds.conf
Add your server to the end of the file:
[Ams]
 host = 192.168.11.106
 port = 1433
 tds version = 8.0
where the host is IP of your windows machine. After this step you should be already able to connect to MS Sql Server:
tsql -S Ams -U peter -P pass45
You should get the server database prompt (just like the osql on windows machine). You can try some commands like
use ams
go
select * from contacts
go
Now set up ODBC connection. Go to Applications -> Utils -> ODBC Administrator 1) add driver with following descriptions:
Description: TDS
Driver File: /opt/local/lib/libtdsobdc.so
Setup FIle: /opt/local/lib/libtdsobdc.so
Define as:   System
2) add User DNS DSN: Ams Desc: old AMS database server add 2 keyword/value placeholders. To update them click on it and press TAB. Set it to following values:
ServerName: Ams   (or whatever you set in freetds.conf)
Database: ams   (or whatever your database name)
now you should be able to test iODBC. Note that I am using sudo for this as it doesn't seem to work without sudo complaining Data source name not found and no default driver specified. Driver could not be loaded (0) SQLSTATE=IM002.
sudo iodbctest "dsn=Ams;uid=USERNAME;pwd=PASSWORD"
You should be now in interactive terminal again. Once all this works connecting from Ruby is really easy:
require 'rubygems'
require 'dbi'

DBI.connect('DBI:ODBC:ams', 'USERNAME', 'PASSWORD')

Rescuing the data

This is just a short ruby script I use to extract all the data from MS Sql and import it to Postgresql. It's not any functional database conversion - it's just to get the data out to something that's easier to work with and doesn't require windows machine running. It may have an issue with binary fields - it worked on most of them but it did choke on a couple of fields. DBI actually provides more metadata like NOT NULL attribute, primary key, etc. so the script could generate a more precise copy of the original database but this was enough for what I needed. You may run into unknown data type - especially if you try to import it to a different database engine - all you need to do is just update the method update_type to return correct mappings of the data types.
require 'rubygems'
require 'dbi'
require_library_or_gem 'pg'

def escape(text)
 return "NULL" if text.nil?

 text = PGconn.escape("#{text}")

 return "'#{text}'"
end

  def self.update_type(col)
    type = col["type_name"]
    type ||= 'timestamptz'
    type = type.downcase
    
    case type
    when 'clob'
      return 'varchar'
    when 'varbinary'
      return "oid"
    when 'long varbinary'
      return "oid"
    when 'double'
      return 'double precision'
    when 'tinyint'
      return 'smallint'
    when 'char'
      return "char(#{col['precision']})"
    else
      return type
    end
  end



 dbh = DBI.connect("DBI:ODBC:ams", "peter", "pass45")

 sth = dbh.prepare('select name from sysobjects where type = \'U\' ORDER BY NAME;')
 sth.execute
 tables = sth.collect{|row| row[0]}

 tables.each do |table|
   sth = dbh.prepare("select * from #{table} ")
   sth.execute

      create = "CREATE TABLE #{table}(\n"
      create << sth.column_info.collect{|col| "\"#{col['name'].downcase}\" #{update_type(col)}"}.join(",\n")
      create << ");\n\n"

      puts create

      sth.each do |row|
        create << "INSERT INTO #{table} (#{sth.column_info.collect{|column| "\"#{column['name'].downcase}\""}.join(', ')}) VALUES (#{sth.column_info.collect{|col| escape(row[col['name']])}.join(', ')});\n"
      end
      
      create << "\n\n"
      
      output = File.new("data_ams.sql", "ab+")
      output.puts create
      output.close
      
      #puts create
    end
To import the data to Postgresql is as simple as:
createdb ams_backup
psql ams_backup <> noise.txt

Rails *is* (still) a Ghetto

nice_ass.png

While I know the title is both asking for trouble (because of the now anecdotal original article with a similar title) and flamebaity, please read on - my goal is not to get some great stats but rather to know your opinion about the situation and discuss the possible solutions of the problem.

How it all started…

I would not like to re-iterate what has been said on several blogs, just to summarize: Matt Aimonetti, member of the Rails Activists, gave a presentation at GoGaRuCo which contained sexually explicit images (according to some - I am not here to judge whether that’s true, and it doesn’t matter anyway, as you’ll see in the rest of the post).

I am not really discussing whether it’s appropriate to have images of nude chicks in your presentation at a Ruby conference (I think it’s not, it’s unprofessional etc. - but that would be a matter of a different post Update: Someone summed this up in the article’s reddit thread nicely: If you’re a Rails programmer, or a Ruby programmer, and you don’t decry this sort of thing, you have no business calling yourself a professional. It doesn’t matter how large your website is, how easy it was to write, how much better it is over PHP or ASP.NET or J2EE; by definition, you do not belong to a professional community. That’s all there is to it. It’s incumbent on every Ruby programmer to either reject this sort of misogynistic sewage, or accept that you’re never going to advance the promotion of Rails in the public perception because members of the community still think it’s edgy or cool to put pictures of strippers in their public presentations. And here’s a hint: if your decided reaction is to talk about how unimportant this is, how much it doesn’t matter, or how much it doesn’t offend you personally, you probably don’t understand professionalism at all.) because sadly, I think there are far bigger problems here than that - shedding light on them is the real purpose of the article, not talking about pr0n at GoGaRuCo again.

Would You Walk Into a Hindu Temple with Your Shoes on?

hindu_temple.pngI have been living in India for 2 months last summer, working on a Rails startup. Maybe I am odd or something, but I knew that I had to remove my shoes when entering a Hindu temple, and no one had to convince me (what’s more, I didn’t even think about it for a second) wether this is the right thing to do, why is it so, whether I should do otherwise etc. This is a similar situation - I just don’t do X when speaking at a conference, if I suspect that X makes feel even one person in the room uncomfortable, whether because of his gender, race, nationality, Ruby/Rails skills, penis size or what have you - regardless whether I think it’s fine for me, my wife, for other members of the community and/or the majority of the room.

The trick is, how does a hindu feel when I enter a temple in footwear (even if that is perfectly acceptable in my country, culture, family, friends) - it’s perfectly irrelevant how do I feel in the given situation. Using the previous paragraph, try to apply this to a Ruby/Rails conference.

Shit happens…

Until this point in the story, I see no problem at all, and could even agree with the guys asking “what’s wrong with you, don’t make a fuss out of nothing” - the pictures Matt used are non-problematic in my book, and he had no idea they are problematic in anyone’s book - theoretically it could have worked, but the point is, it did not. Some members of the Ruby community got offended, and here our story begins.

…and hits the fan

One of the real problems is that after this has been pointed out, Matt still keeps answering “As mentioned many times earlier, I don’t think my presentation is inappropriate.”. As I mentioned two paragraph above, it doesn’t matter what do you think, unless of course, you don’t care about offending some members of the community. In that case you should not try to apologize at all. However, if you are trying, reciting “I don’t think my presentation is inappropriate” will not put and end to the discussion. It just doesn’t work. Why can’t you just simply apologize, admitting that this was a bad move (because it offended some, not because porn, sexual images or whatever in presentations are bad, per se) and finish the discussion?

Rails is Still a Ghetto

However, in my opinion that’s still not the worst part of the story, or to put it differently, some members of the Rails community still found a way to make things worse, by applauding to all this:

dhh_pr0n_is_great.png

OK, you say, we are all used to DHH’s style, this is just how the guy is. That’s (kind of) cool, but I heard that most of the Rails core team (and obviously Matt himself) has the same opinion - and that’s a much more serious problem, because it means that a Rails activist, backed by DHH and other Rails core members finds all this OK, despite of the fact that numerous people in the community voiced their opinion otherwise.

This is not about being a closed-minded prude, shouting for police and suing everyone using sexually explicit images in a presentation. This is not even about women, as I have seen both males and females on either side of the fence. This is about mutual respect - I don’t agree with you, but respect your feelings. Or not, as demonstrated in this case.

So Rails continues to be the most socially unacceptable framework - associated with arrogance, elitism and whatnot in the past - now add pr0n images in presentations. Thankfully RailsConf is held in Las Vegas, and that should calm down all the people who associate Rails with all this crap :-). The real problem is that people associate you with the tools you are using - think Cobol, PHP, Java… or Rails. By being part of the Rails community people associate me with Railsy stereotypes automatically, which aren’t nice at all right now.

I hear you, dear creme-de-la-creme Rails (core) member, I know you don’t give a shit, and you think this is all prude babbling - because your hourly rate is more than some of us earn in a day, and you’ll be sought after even if Rails will have a much worse image than it has now. But 99.9% of us are not in the ‘circle of trust’ and would be happier if Rails would not be constantly associated with a ghetto.

MINASWUBN

In case you are wondering what does the acronym stand for, it’s “Matz is Nice And So We Used to Be Nice”. Unfortunately, the stuff I don’t like about the Rails community is sneaking into Ruby too, it seems, as the above case demonstrates. Besides this, the count of aggressive comments and reactions on various blog posts is really disturbing to me. Please (at least Rubyists) try to avoid being contaminated by all this shit and stop thinking you are cool because you can swear on a forum (always in anonymity). You don’t have to be a douchebag just because you are a Rubyist / Rails coder, as surprising as this might sound to some.

Conclusion

I think “incidents” like this and getting more and more antisocial members are inevitable by-products of growth in a community. The questions is, whether, and if, how, do we stop them. The problem is that it seems to me the Rails “top management” doesn’t want to stop them (what’s more, even encourages them) in the first place (please prove me otherwise - maybe I don’t see the full story - I’ll be the happiest to admit that I am talking bullshit).

I have to admit I have no clue what would be the right move - burying our heads in the sand and pretending everything is fine is not. Please leave a comment if you have an idea or anything to add.

Slovak Ruby and RoR fans will meet in Bratislava on April 25

Slovak Ruby and Ruby on Rails fans will meet in Bratislava on April 25, 2009. For more details see the Slovak Ruby on Rails site or directly the post about the meeting [all in Slovak].

Another Training

Another training's over. Very different but equally challenging.
Seems that Ruby's been making headway to corporate world as most of the participants this time were there on their employers' budget. When I compare it with our first RoR trainings around 2 years ago with Rails 1.0 just out and more than 30% of the things still on a wish list and the only way to learn RoR was Agile Web Development With Rails...this time it was much more business-like. There's like 20 books out, including a (bit obsolete) video from Linda.Com and pretty much everybody knows about rails.
Another aspect that has changed is Rails versions. With Rails 2.3 and Ruby 1.9 out there's a lot of changes and incompatibilities that complicate the learning. Most of the books and reference materials use Rails 2.0 or early versions of 2.2. This makes it little bit difficult to have training on 2.3 as most of the "further" reading is incompatible. While the incompatibilities may seem easy to solve for experienced developers they can be quite exasperating for a newbies. On the other hand, people are quite reluctant to start learning something that is already obsolete.
Similar is true for Ruby 1.9, even though after burning they're fingers several times until it got finally compatible with Rails most people would still prefer to use 1.8.6 or 1.8.7.

SMS Integration Without Hardware

When I tried to integrate one of our applications with SMS messages (both incoming and outgoing) several years ago we didn't manage to find not even one SMS gateway provider that would be able to pass our simple criteria. Allow sending (da :-) and receiving of messages with some reasonable notification of received messages (simple webservice / forward to email, anything). The replies should preferably be handled by a local number and sending/forwarding should be within reasonable price range :-) Is it too much to ask?

There is a number of online portals for sending (some support also receiving) SMSes like Clickatel, SMSZilla, SMSCountry. Their pricing per message varies from SGD0.05 - 0.1 which is quite reasonable and some provide nice and easy API for sending (when you think about it how complicated can it get to write a form handler with 3 or 4 fields) - some ask you to pay for API extra. For better or worse they would accomplish the sending but the receiving was either very painful or expensive or both. On top of that sending a hundred messages to and Indian or US number shows on your bill so we had to find something better.

For systems that require only sending we mostly used Singtel Ideas or Starhub Gee. Both are just a little short of and actual webservice, nevertheless worked quite well. Here's an example how to send message using StarHub Gee (12345678 is your phone number)

require 'rubygems'
require 'mechanize'

mech = WWW::Mechanize.new
page = mech.get('http://groupsms.starhubgee.com.sg/VCRLS/login.cgi?cb=http%3A%2F%2Fgroupsms.starhubgee.com.sg%2Fcgi-bin%2Fs.pl')

#login first
form = page.form('loginform')
form.action = "https://login.starhubgee.com.sg/eam/login"
form['cb'] = 'http://groupsms.starhubgee.com.sg/cgi-bin/s.pl'
form['vcid'] = 'groupsms'
form['pudn'] = 'starhubgee'
form['domain'] = 'starhubgee'

form['fake_uid'] = '12345678'
form['uid'] = '12345678@starhubgee'
form['password'] = 'password'

page = mech.submit(form)  

page = mech.get("http://groupsms.starhubgee.com.sg/cgi-bin/s.pl")
form = page.form('compose')
form['to']='comma separated recipients numbers'

form['message']= "Your message...."

page = mech.submit(form)

This would take care of sending with much lower cost as it utilizes bundled SMSes in the package (400 - 1000 messages) with nothing else required. The problem, however, was still receiving. A little glimpse of hope was SMS+ introduced by SingTel last year, which allows you to auto forward your received messages to email address. This would have been ideal - if not for a small glitch - it only works with messages sent from SingTel numbers. OH MY GOOOOOD!!!!!!!! I almost cried when I found out that.

Lately I found a nice and simple Singaporean provider www.gsmexchange.com able to handle 2 way communication. Nothing to say about sending - just send a http request with username, password, phone numbers and message and you will receive message ID that you can use for tracking. Replies are slightly more complicated as I didn't find any API for reading replies - nevertheless - there is a list page that you can parse with mechanize to get the responses. There is an option to forward messages to email but for 5c per message it's a bit too expensive. Actually, also pricing for sending of the messages is on the higher end 10c per message which makes it much less attractive for our customers.

Luckily I managed to find a much cheaper solution - running my own SMS server :-) See the next post...

Domain Specific Languages in Ruby

It is probably every developer’s dream to create a programming language. It is a complex and difficult task, so it will remain just an unfulfilled dream for many. But there is a chance to create something simpler - a domain specific language (DSL). A language, that covers a specific (and limited) domain. Ruby helps here a lot [...]