foldRight is king

This morning traffic was exceptionally bad. I didn’t happen to choose the right podcast to listen to and it didn’t manage to divert my mind well enough, so I lived through the whole of the extended commute. The nice thing was thought that I didn’t have many meetings, so after a couple of weeks I finally had some uninterrupted hours to focus on coding. I started refactoring a bit that was horrendously embarrassing. While doing this, I bumped into the age-old Scala problem that is caused by some 20-year old bad decisions JVM designers made. I needed to partition a Map based on the key’s actual runtime type. In an ideal world it would’ve looked something like this:

trait Base
case class Derived1
case class Derived2

val map = Map[Base, ValueType](dervied1 -> v1, dervied2 -> v2)
val (d1, d2) = map.partition {
   case (Derived1, _) => true
   case _ => false
}

This of course doesn’t work, as generics lose their type information because of erasure, so the Scala compiler complains about fruitless type checks. This is when I got the artillery and went to fold for help. If you have given it enough thought, it’s actually easy to understand:

val (d1, d2) = map.foldRight(
           (Map[Derived1, ValueType](), Map[Derived2, ValueType]()
         )
{ 
  case(m, (d1, d2) => m match {
          case (k @ Derived1(), v @ _) => (d1 + (k -> v), d2)
          case (k @ Derived2(), v @ _) => (d1, d2 + (k -> v))
       }
  }

Which is pretty cool. If I remember correctly, there’s a theorem that fold can be used to achieve everything that the higher-order functions on the collections provide; they’re merely there for convenience :)

Small wins

I recently moved to a new team within the firm. I was “hired” as an architect/infrastructure designer: this is a team with close to the business and they’re lacking some hard-core technical/technological people. I was a really good match for what they’re trying to achieve. They (and now we) have an old calculation engine that has all the shortcomings from the last decade of software design and development: uses XML heavily, has too many abstractions and it tries to be too smart by allowing too generic things. I’ll be looking into redesigning this while also upgrading the overall architecture of the systems we own. I’ thrilled to do this as I think this is where I want to lead my career.
The first couple of days have been sent with getting familiar with the environment, people and everything. Just out of fun and interest I spawn up a JProfiler and run some huge calcs through the engine. Surprisingly, the top three hotspots were very simple functions – that were called way too many times. One of these was actually just returning the same value 97% of the time, but delegated all the calls back to the parents in the hierarchy. It took about 80 seconds while the calc only used 200 seconds CPU time. It was too apparent that this is the place to start. So I analyzed the code and added some caching where I knew the value can’t change, sparing the huge call chains. I managed to reduce the 80 seconds to around 1-2, dropping CPU usage by 40% for huge calcs (the optimization is not that significant or smaller ones, actually).
When I explained this to more senior people in the team, they were surprised that this was a hotspot – they never imagined it would even come up in a profiling. Then they really like the caching idea. Then the news started to spread, and when I got to talk to the head of the wider team, he already mentioned that he heard that I already cracked the engine and that we’re saving a lot of time :) I explained to him what happened and how it works and that it won’t be reducing time of calcs by 40% as we’re still waiting a lot for IO.
I think this was an easy small win for me: I knew what to do, did it and already made my name heard in the team. I’m not saying people should start playing the cowboy in their new positions, but it’s always worth it to show what you can do iff you have the chance.

My experience with Apple products that Just Work™

Today I wanted to install Windows on my MacBook Air. The wife needs to run a software on the laptop that’s not available on anything else than Windows. It requires proper 3D acceleration, so virtualization is out of question. I didn’t have a choice but to have a native Windows installation. This would’ve been a joy ride, considering Apple created Boot Camp, and things they do just work. Right?
First of all I needed a USB drive for some reason. I didn’t understand why it couldn’t create a temp partition with the install image. First, I tried using an SD card (which is, from the software’s perspective is just a USB drive), but Boot Camp was implemented by more clever people, so it simply rejected the card as a viable medium. OK, I dug up my cheap 64G flash drive from the bottom of the drawer, created the install disk and copied the Apple drivers to it.
The next step should’ve re-partitioned the disk so I have a new one that can be used by Windows. Boot Camp failed at this step. It didn’t tell me why exactly, so I thought that I could do this manually, I could even name the partition BOOTCAMP, hoping they wouldn’t check for anything else. Started the Disk Utility that comes with Mac OS, selected my drive and clicked the Partition button. To my surprise, the product created by the Kings of UX doesn’t handle me changing the partition size in the numeric spinbox, only by moving the handle in the pie chart. *SIGH*. OK, I’ll use that even though it’s much harder to size it exactly to what I want. After hitting Apply, it ran a quick disk check but found errors. Of course those can’t be fixed while you run the OS booted from the drive, so I rebooted in recovery mode to run fsck. After starting the partitioning process again (using the stupid handle to precisely resize) and hitting Apply, the check finished successfully, but. But it simply couldn’t resize the partition. That’s it, no meaningful error details. This is when I said, screw it.
I started looking around for third-party software that would accomplish this hard task. Unfortunately these ranged from $30 to $50 in price, but I didn’t intend to pay for something that I’d only use once and should be available fro free. A hint I read is that probably the volume was fragmented and this wasn’t handled by Apple’s own software. Hurray! I’d need to defrag the drive first, then re-partition. Eventually I decided to revert to something I knew, GParted. It can’t run on Mac OS natively, so I needed a Live USB drive to boot. Went to look for a second USB drive in the drawer and I created the Live GParted disk. I booted it and using the ugly-but-working UI I resized the partition (I could type in numbers that meant megabytes!), created a new one, formatted it to NTFS and I started to feel the accomplishment. Oh, how wrong I was.
Apple silently converted all simple GPT disks to CoreStorage, aka a Logical Volume Manager. This was needed so they can implement FileVault and these hybrid HDD/SSD disks. It kind of makes sense, although nobody asked me if I wanted this or not, or even if I’d ever need it. Not only me, but GParted was also not aware. Changing things around basically broke all the meta data this thing needs. The MacBook started not to boot up. I tried to analyze the kernel’s log, but googling the error message didn’t reveal much. I went over about 10-15 blog posts/Apple forum posts, but nothing really helped. I slowly started to give up on the data on my original partition. Then I started to look around in the list of available commands the command-line version of diskutility supports. I’ve found an interesting one: ‘corestorage revert’, which should revert a CoreStorage Logical Volume to regular GPT partitions. I said what the heck, I can’t lose my stuff even more, I ran the command. Everything related to CoreStorage was gone from my drive. I rebooted.
Voila, the OS managed to start up. I started to feel a little bit better. I went back to Boot Camp to finally finish the installation. I was naive, haha. It said that it couldn’t partition the disk because it needed it to have only one partition. What?! Great. I need to add the LVG back, so I can arrange the thing to only have one partition logically. So I did. The diskutility can do this in a non-destructive manner. I issued the command. I rebooted again to make the new Logical Group online, reran Boot Camp again, got the same error. I went to Disk Utility, and tried to completely remove the new partition (I planned to give Boot Camp only the space, not the partition), but when I deleted it, it always resized the HFS one to occupy the full available space. A quick round of googling didn’t help, nobody had this issue. So I rebooted the GParted USB drive once again, hoping I wouldn’t need to replay the mess with CoreStorage. I deleted the partition and for some reason this time CoreStorage didn’t go haywire and I could straight up boot into Mac OS. Gave Boot Camp another chance. I think you can guess if it succeeded or not. Crap!
I got angry at all those very clever engineers behind Boot Camp, so I thought that I’d simply install Windows on my partition without that stupid thing. I did a google search that revealed that this should work. So I rebooted from the Windows USB drive and installed Windows. Everything succeeded, even the drivers added by Boot Camp installed correctly. The last test remaining was checking if I could still start Mac OS. I could. Pheww, such a relief!
This is my story with the great Apple User Experience. It reminds me of the time when I had a course at the university about J2EE. It all works perfectly until it simply doesn’t, and you can’t possibly fix it because of the endless layers of crap (abstraction!) added to it. Apple managed to close the gap to Sun in this regard, for sure.

My laptop has an uptime of 451 days

This means that I’m probably the worst kind of user. I don’t care for any security updates, maintenance or anything else, I just want to use the machine. Ok, it doesn’t need to handle very high workloads, mostly browsing and the occasional compiling when I’m presenting or on meetings at the university. Still, I think all sys admins and ITsec people would frown upon me for this.

How I got into programming

It started a long time ago, but as you can guess, it was a fundamental experience, so I remember some details. I was in elementary school, 4th grade. My family is not religious -I’m not even christened!- and at fourth grade everyone in my class, except for a select few, started going to bible class. My parents asked me if I wanted to and since I didn’t see the point, they didn’t force it. I’m not really sure if my lack of religiousness would’ve allowed me to even participate. Anyways, this meant that the school had to ‘deal with us, outliers’. There was this old man (Mr. Endre) who handled the not-so-huge computational needs of the school and the town council. He helped where he could, so he was tasked with teaching a Computer Class. The school had a bunch of Commodore 64s and some small TVs, and most of the time Mr Endre simply let us play with them. He also brought some games on 5.25″ floppy disks, taught us how to load the games, etc.
At home we had a Commodore VIC-20 with a 16k cartridge extension. It was still far from the 64’s 64k, but it was more than enough. We didn’t have a floppy drive though, only a cassette drive (called a datasette). I asked my father to hook it up to the TV, and I started playing at home too. It wasn’t as convenient as the school machines, and I couldn’t play those fancy games. They kind of started to bore me.
One day I’ve found my dad’s book about Basic. I was fascinated by it, although a lot of things didn’t make sense. I’m not sure why my dad had it, to be honest, he never picked up programming. Anyways, I started reading the book and basically typed in some things that were programs. I didn’t fully understand what was happening, but this mystery lead me on. I couldn’t get enough. I remember one of the greatest revelations was figuring out conditionals – just the fact that I could GOTO to different lines based on some condition opened up a huge space. I also found a way to get input from the user (thinking about it now, these three concepts (conditions, variable assignment and GOTO) meant I’ve found Turing completeness). Combining these things I wrote my first program: a joke teller. It asked if you wanted to hear a joke. If you said yes, it told you one and asked if you liked it. It was a very basic conversational program. I remember gluing the pieces together took me weeks – multiple instances of the computer class plus my free time at home. It wasn’t like I had long continuous, uninterrupted intervals when I could focus on the problems. I had to think about things for days, come up with theories, test them, then wait some more days until I had the chance to touch the computer again. As a fourth-grader, I had to deal with a lot of other responsibilities :)
In a nutshell, this is how it started for me. I think I decided that I wanted to be a programmer then – in fourth grade.

The Internet is a beautiful thing

I just spent 14 minutes watching a dude fix a puncture in a $30.000 tire:

I envy C/C++ programmers a little

I have found this post on HN, and while I read through it, I had a strange thought. While I do coding professionally (aka do my job), I take care of much higher-level problems much more quickly (you could say in a bigger hurry). I realized that I envy people who can afford spending so much time and effort investigating and solving such a seemingly trivial problem. I’m not being condescending by all means here, I just find it interesting that some people has the blessing of actually handcrafting so little parts of a software system. I never get to spend a lot of time on such little pieces. Yes, I do optimize, I do try to be smart and there are times when I have the luxury of coding features for days uninterruptedly, but it’s usually not the case. I think about my code as a woodworking master who designs and creates furniture: I pay attention to little details, but use a lot of bigger pieces. Sometimes I miss playing with small Legos.