I’ve recently found about the Dunning-Kruger effect on a blog post by Steve Yegge. Despite the fact I do like Joel’s book and even have used his technique on a few recruiting process, I’ve found that for everyday businesses, even Smart&Get-things-done guys can be too hard to get.

Anyway, the point I want to talk about is a side effect of Dunning-Kruger. While in college, in some classes, I would get to a point in which I knew almost nothing about something but was very creative about it, getting lot’s of ideas to do things that I didn’t knew in that moment were hard or impossible to do. The DK effect. But some of those ideas were worth trying for the learning. And even some of them, although they would be harder to execute than it initially appeared, were good ideas indeed. Because of DK, you don’t know the limitations and thus your brain is free to create. When you get to know a lot about something, limitations and restrictions are evident and thus your brain restrain some of those ideas. Anyway, the truth is that some of those ideas could be accomplished and be good.

Once, on a Digital Image Processing class, I had the idea of processing photographs of rooms, aisles, buildings, etc along with some makers. Then make a 3D projection of the markers to “infere” the dimensions of the room, making measurement of spaces simple as a click. At first, the idea sound great, and started working on the math. Then I passed through a disenchantment phase: it was like 1994, digital cameras weren’t readily available, the math started to be harder than I knew and computing processing was not so easy to get. Then I quit the idea and moved forward. Now, I know better and maybe it was a good idea. I’ve seen some 3D projections based on 2D images and are awesome.

That said, I will give a chance to those naïve ideas. Maybe someone could be good after all…

This is the simplest way I’ve found to get pretty permalinks and/or URLs for this blog (for SEO optimization): use a 37signals-like standard prefix the id of the post to the prettyfied version of the title of the blog. Use the following to Find the post:

 

def self.find_by_permalink(permalink)
  return(nil) unless permalink
  self.find(permalink.split("-")[0])
end

 

Use the following to build the permalink:

 

def permalink
    "#{self.id}-#{self.title.gsub(/[^a-z0-9]+/i,'-')}"
end

 

 

http://www.amazon.com/Culture-Code-Ingenious-Understand-People/dp/0767920562|Clotaire Rapaille|

Dr. Rapaille describes in this book his discoveries on Archetypes applied to marketing. Gives a very general description of his method to uncover them using focus groups in extended sessions. One of the dissapointments about this book was my expectation to understand different cultures as it focus absolutely on the United States and a few topic on the French, English and German cultures. Some peeks to the Japanese and Italian. But that’s it. Patethically Euro-American centered. Also, it’s quite dissapointing the feeling left behind the attitude of being “more American than the Americans” o más gringo que los gringos, as the most common words are US, WE, OURS… It gets to a point in which is hard to say if an archetype, or “Code”, is mostly a reflection of Dr. Rapaille’s feelings.

Anyway, the insights given by the book are very interesting, as the sex/violence, food/fuel, money/proof… Most Codes are instructing and give us a peek on the perceived ingenuity, greed and results oriented attitudes we have of the americans.

[Hope I can extend this post as I re-scan the book]

 

For this little blog app, I needed to truncate the post length on the home. Of course it can’t be done by a simple truncate since it can break HTML tags. I found this awesome hack by Mike Burns.

http://www.amazon.com/Little-Schemer-Daniel-P-Friedman/dp/0262560992|Learn to think in Scheme|

This is one of the best programming books ever. Refresh your mind by learning to think again.. this time, in Lisp.

http://www.fusionmxsoftware.com/casa|en México D.F.

http://www.ted.com|Ideas

Ruby ERB is a great library that let’s you process a file and embed Ruby code within it using <% ... %> tags. As the file is processed, the code is evaluated and content may be replaced.

This is generaly used with HTML (or RHTML as generally known). Rails does this.

I’m now coding a generator for ASP scripts using ERB. As both ERB and ASP uses the <% <%= tags, I modified the original erb.rb file to use <@ <@= tags so the ASP code can be left as usual. Get the modified file here. If you want to use a diffent tag, I recommend to diff my file with the original erb.rb

The .NET Framework and ASP.NET have some aspects that sometimes really annoys me. One of these is the infamous __VIEWSTATE used to track the state of all the controls in the page.  What is annoying about it is that it’s both a security breach and a performance issue. Also, makes easy things hard.

I wanted to remove it from an app I’m working on, so I started disabling it as suggested by MS. The VIEWSTATE is effectively not used but it stills yields the hidden input field to detect the Postback. This is a nag since this form is performing a GET action and the __VIEWSTATE is passed as part of the QueryString. After some Googling I found a method suggested by Scott Hanselman to move the position of the input field. I just adapted to plainly remove the complete thing as shown below:

       protected override void Render(HtmlTextWriter writer)
        {
            var sw = new StringWriter();
            var htmlWriter = new HtmlTextWriter(sw);
            base.Render(htmlWriter);
            string html = sw.ToString();
            int start = html.IndexOf(“            if (start > 0)
            {
                int end = html.IndexOf(“/>”, start) + 2;
                html = html.Remove(start, end – start);
            }
            writer.Write(html);
        }

    }

All this code just to get a simple, common, standard query string.

^{This post is on construction.}

I’m working on a Recommendations engine. As some processes may be lengthy, I’m also implementing a simple MapReduce framework using a Ruby/DRb schema. I tried Skynet for a while but it looks inmature, and I needed something simple and fast to implement.

The Server – Worker

The DRb server was pretty simple to implement. I just wrapped my Recommendations engine on a single class and created a function for starting it as a DRb server:

def start_server
  uri="druby://0.0.0.0:#{ARGV[0]||'5850'}"
  trap("INT"){puts("Interrupted"); DRb.thread.exit}
  DRb.start_service(uri, RecommendationsServer.new)
  puts("Listening #{uri}")
  DRb.thread.join
end

Just run the server on every processor to which you can distribute some processing load.

The Client – Controller

Next, I crafted a simple program that implemented the MapReduce model by segmenting an array with the data to process, distribute the segments to the workers and yielding the disered operation (which must be defined on the worker server class).

#map
# data  : hash to be mapped to @workers
# worker_object and segment are yielded
def map(data)
  #data = @apps #version lineal (sin procedimiento)
  threads=[]
  result={}
  inicio = 0
  #sec_len = data.keys.length / @workers.length #cargas iguales
  @workers.each do |worker_uri,porcentaje_carga|
    threads << Thread.new( 
          #:prefs=>prefs, #carga igual 
          :segmento=>segmento(data.keys,inicio,porcentaje_carga),
          #:selection=>@apps.keys[worker_n*sec_len..(1+worker_n)*sec_len], #carga igual entre procesos
          :worker_uri=>worker_uri) do |p|
      msg("Spawning worker with #{porcentaje_carga}% load @ #{p[:worker_uri]} ")
      worker_object = DRbObject.new(nil,p[:worker_uri])
      result["result_#{worker_uri}"] = yield(worker_object,p[:segmento])
      #result["result_#{worker_uri}"] = worker_object.calculated_matrix(:prefs=>p[:prefs], :selection=>p[:selection], :type=>:similar, :print=>true)
    end
    inicio += long_segmento(data.keys.length,porcentaje_carga)
  end
  threads.each{|t| t.join}
  return result
end

def reduce(result_set,output_filename=nil,result_set_name=nil)
  r = {}
  result_set.each_value{|worker_result| r.merge!(worker_result) if worker_result}

  if output_filename
    File.open(output_filename,"w") do |f|
      f.write("#{result_set_name||'result'}=#{r.inspect}")
    end
  end

  return r
end

Simplify the process of calling a whole MapReduce cycle:

#args:
#  :prefs=> conjunto de datos
#  :type=> tipo de matriz a calcular
#  :output_filename => archivo de salida
def map_reduce(args)
  msg(args[:process_name]) if args[:process_name]

  #map
  start = Time.now
  msg("Mapping...")
  result_set = map(args[:data]){|worker_object,segmento| 
    #worker_object.calculated_matrix(:prefs=>args[:data],:selection=>segmento,:type=>args[:type],:print=>:true)
    yield(worker_object,segmento)
  }
  msg_ok(start)

  #reduce
  msg("Reducing...")
  start_red = Time.now
  r=reduce(result_set,args[:output_filename],args[:result_set_name])
  msg_ok(start_red)

  msg_ok(start,"#{args[:process_name]} finished") if args[:process_name]
  return r
end

Finally, calling the MapReduce complete process is as simple as:

  ub_recs = map_reduce(
    :data=>@apps,
    :process_name=>'Calculating user based recommendations',
    :output_filename=>'__ub_recs.rb',
    :result_set_name=>'@@ub_recs') do |worker_object,segmento|
      worker_object.calculated_matrix(:prefs=>@apps,:selection=>segmento,:type=>:recommendations,:print=>true)
  end

The yield keyword passes the segmented data and operation parameters to the worker process.

Later, I’ll post on how to load-balance between workers, which proved to be both critical and trickier than first apperared.