Just to give you an idea, I’ll enumerate the HTTP requests the BiliBid server is constantly receiving:

• The pages. They are the requests for the front page, winners page, single bid page, redeem PIN page, etc. It is requested once per pageview.
• The static files. The pages contain static files (Javascript and LESS files). Multiple requests per pageview, only requested when the page is loaded.
• Balance requests. When a user is logged in, the user’s browser requests for the bid balance once every two seconds.
• Timer requests. When a user is in the front page or a single bid page, the user’s browser requests for the timers of the items in the page once every second.
• Bid requests. Everytime a user places a bid, a request is sent to the server.

Multiply that to the number of users online!

Switch from Apache to nginx + php-fpm

One day around early September, I tried researching on nginx (engine-x) since I’ve heard that it has a smaller and more predictable memory (RAM) footprint. I also read stuff on php-fpm. I tried to weigh things since I’ll have to lose some Apache functionality if I switch to this one. This means no more .htaccess for me, but obviously I can convert my .htaccess file to something that nginx would understand. Fortunately, I found out that it wasn’t that hard to do it.

The same day, I installed nginx on the server. Before killing Apache and starting nginx, I fixed all my configuration files first. Then came the moment of truth, I killed Apache and started nginx. I was surprised that everything went so well that I couldn’t believe that BiliBid was already running in nginx.

Routing problems

Then came late October, people started reporting having high latencies with BiliBid. I think it was also the same period when I decided to put the Connection Quality indicator (before I did it, I know no other penny auction site with that feature – until I received a tip from a bidder that a penny auction site in the Philippines [starts with B and ends with A] implemented it, too). Before the problem occurred, I made no major change on BiliBid therefore I concluded that there might have been routing problems from some of our bidders’ end to our server.

This time, I decided to move our server to a nearer location to reduce latency and hopefully solve the routing problem. The move had trade-offs. It was more cost-efficient but to the expense of number of usable processor threads. On our previous server, I had 16 processor threads compared to four (4) on the new server.

I decided to review most of the SQL queries I was making and optimize it. I was able to reduce the execution time of my complex queries, some went down to up to 50%.

Everything went well until we started our Facebook campaigns. We received double the amount of concurrent online users. Our server load went crazy to up to 20.00, then one time it went as high as 70.00 during the auction of a big item. We had to postpone one of the big items to distribute the load over time. I tried experimenting different configurations on nginx, php-fpm and MySQL but to no avail.

Going back to our old server

This time, I decided to go back to our old server (16 usable processor threads). The problem with the server load disappeared. Everything was okay until recently, we received an overwhelming number of reports that translate to latency problems. I checked them one by one, around 1/2 were valid while the other third I consider invalid – just riding with the flow (and I hate it).

The most recent server switch, adding another server, MariaDB, etc

So I decided to go back to nearer server with four (4) processor threads. Before switching, I tried reading stuff to optimize the server to prevent high loads. I decided to change SQL server from MySQL to MariaDB. I think I first heard of MariaDB during a DevCon in UP. I experimented on the different storage engines offered by MariaDB until I settled with Aria.

Then came the switch. The latency was significantly lowered down (reduced by more than 50% on most users). The only problem was the higher-than-usual server load (as compared to our old server with 16 processor threads) which I fear would slow down the SQL queries (which is bad for a realtime app like a penny auction site) on peak hours.

Fearful of the slowdown, I decided to get another server on the same location. I configured the second server as the SQL server while the other one will be used solely as a web server. Since they are on the same location, I set up a private network for the two machines so that they exist in one logical network and one LAN.

With the new set-up (dedicated machine for SQL and another one for web serving), I was able to reduce the server load on the web server. The SQL server was also running very fine, in fact based on my tests the execution time for my most complex query was reduced to as much as 40%.

Other stuff I did

Since BiliBid is a time-sensitive web application, there is no room for slowdown. I had to:

• Optimize my cron job that runs every minute. I had to tweak it to only update information that needs to be updated, ignore others. Previously, it processes all my entities. Since the number of entities grow over time (new signups daily, new auctions, etc), I found it inefficient.
• Remove all system cron jobs. On a fresh install of most Linux distros, it automatically adds cron jobs for maintenance tasks. I had to remove them since a maintenance task could possibly slow down BiliBid. This equates to disaster if it runs on a peak hour or if there’s an active auction. I decided to manually run this maintenance tasks during off-peak hours.
• Made RRDTool graphs on the latency to make monitoring easier.

I really hope the stuff I did recently will work well. Until now, I continue to experiment different configurations to check the most efficient one.

As BiliBid continues to grow, I foresee new problems that may arise. Most of them are due to concurrency (users accessing the same resource at the same moment). I have to tweak and optimize my code and queries from time to time to make it as efficient as possible, while maintaining correctness.

Testing the changes is even harder. It’s hard to replicate a potential problem, since first and foremost it does not entail a single request only. It’s a combination of continuous requests (for the timer, messages, and bid history) and user-triggered requests (placing a bid) from multiple clients. At times, there’s no assurance that the changes I’ve made will prevent those problems from happening. What I do is just convince myself that the changes I’ve made were for the better. Once convinced, I deploy it to the production server and hope for the best.

Maintaining BiliBid is not as easy as it may seem. It requires attention especially during the first bidding night after changes were committed. I have to trust on my gut feeling to decide whether to rollback the changes or not.

On the lighter note, I see this strenuous task of maintaining BiliBid as a learning experience. I learned a lot from BiliBid alone. For the love of BiliBid, I’ve read a lot of articles on server hardening, tweaking database and web servers and other related stuff.

As I was skimming the Top 500 Non-individual Taxpayers of 2009 earlier tonight, I happened to notice Golden Arches Development Corporation at Rank 95. Golden Arches is the corporation behind McDonald’s Philippines. Because of that, my stomach began growling but as far as I knew then I only had 20 pesos in my pocket.

As time passed, my hunger intensified. Hungry, I searched for the room for coins. I found coins on the floor, my bag, the cabinet, among others. I was able to collect 11 5-peso coins, 146 1-peso coins and 8 25-centavo coins (which one of them I later discovered to be a 1964 5-centavo coin).

Crazy as it may sound, even though I can already afford to eat at McDonald’s already I decided to sort the coins according to year before eating. First thing I did was to create a MySQL database for it. Then I wrote the PHP app where I’ll encode the coins. After which I started encoding data for my 1-peso coins. I made sure that the data in the database are correct by checking it on my physical count every ten coins. I failed to encode the year of one 1-peso coin since it was already mutilated to the point the year is not recognizable already.

Here’s the result in coin-graph:

And here’s the dump from MySQL sorted by year and by frequency:

As you may notice, among the 145 1-peso coins I had no 2005 1-peso coin. Oldest coins minted back in 1995 whilst my latest coins in 2010. As expected, 2010 coins took the lead. I can’t seem to understand why I got a lot of 2004 coins. I speculate that the Central Bank over-minted coins in 2004 to the point that they halted production for 2005. Another observation was that all of my coins minted from 2004 onwards were magnetic. I also got 3 magnetic 2003 1-peso coins out of the 10 I have in possession.

Enough of those coin stuff. Way back in high school, some of my classmates and even teachers suspect that I have ADHD (Attention Deficit Hyperactivity Disorder). I tried running through the symptoms of ADHD in Wikipedia and now I’m diagnosing myself with ADHD. Hahaha.

Here’s the list of symptoms (those highlighted in blue are the things that apply to me [I think]):

• Be easily distracted, miss details, forget things, and frequently switch from one activity to another
• Have difficulty maintaining focus on one task
• Become bored with a task after only a few minutes, unless doing something enjoyable [sorting coins is enjoyable!!]
• Have difficulty focusing attention on organizing and completing a task or learning something new or trouble completing or turning in homework assignments, often losing things (e.g., pencils, toys, assignments) needed to complete tasks or activities
• Not seem to listen when spoken to [rarely]
• Daydream, become easily confused, and move slowly
• Have difficulty processing information as quickly and accurately as others
• Struggle to follow instructions
• Fidget and squirm in their seats
• Talk nonstop [do I need to elaborate on this???]
• Dash around, touching or playing with anything and everything in sight [and everyone, LOL]
• Have trouble sitting still during dinner, school, and story time
• Be constantly in motion
• Have difficulty doing quiet tasks or activities
• Be very impatient
• Blurt out inappropriate comments, show their emotions without restraint, and act without regard for consequences
• Have difficulty waiting for thingI think my friends would agree on me s they want or waiting their turns in games

There you go. If you know me well I think you’ll have to agree with me. On the Wikipedia article, it also says “Untreated adults with ADHD often have chaotic lifestyles, may appear to be disorganized and may rely on non-prescribed drugs and alcohol to get by.” LOL.

Sleep Cycle iPhone app. I also have my own sleep graph, as seen way below this one

Before I left Manila for GenSan, I read this article from Lifehacker about the Five Best Mobile Alarm Apps. One alarm caught my attention, the Sleep Cycle. Sadly, there wasn’t an Sleep Cycle for Android; only for iPhone. Being obsessed with data, the only thing I wanted was the raw data in order to make a graph. And so I decided to make a prototype application for it.

Developing it was pretty easy. I just googled for a sample code in detecting movements using an Android phone’s accelerometer. The one I found was pretty much straightforward but I had to modify some equations most especially the ones described as the speed.

I had to experiment with different movement threshold values for the sensitivity until I found the one I think would suit my needs. I also had to experiment on the time intervals between recording of raw data. In storing the raw data, I had to decide whether to store only the timestamp of the start time then the raw values after each interval OR store the timestamp and the raw value in each time interval. I decided to implement the first one for my first dry-run.

Once I thought everything was alright (I started at 2am and ended by around 7am leaving me with no night sleep), I decided to give it a shot by actually sleeping with the app. When I woke up, I excitedly tried to analyze the data but sadly the phone measured way more data than expected. I decided to just discard the data and debug the app. This was the time I decided to use the latter in storing the data. Store the timestamp plus raw data for each time interval.

So that night (actually last night), I decided to sleep with the app again. Upon waking up earlier today, I tried to check the data and it seemed that it recorded the data the way I expected it to. I immediately tried to learn how to use the RRDtool (Round Robin Database tool) so that I can feed it with the data and finally come up with a graph. In no time, I got a visualization of my sleep, as seen above.

You can see from the graph that I entered sleep at around 12:20am and woke up at 4:50am. Also evident are the parts where my movements were higher than usual, probably the times I was dreaming. For the next few sleeps, I’m planning to feed them the data directly on my laptop instead of having it stored on the phone then later transferred to my laptop. I will make a portal where my sleeping activity is readily available on the Internet in realtime using PHP, a cron job and Dynamic DNS for domain resolution to my network.

I haven’t felt this geek for a while and I’m happy to be one again even just for this project. I am looking forward to making more geek stuff, preferably with Android.

I spent most of my summer taking summer classes (Physics 72 and PE 2 Weight Training). Aside from that, I spent a some chunk of my time in DilNet Help Desk answering phone techs, fixing networking problems on-site and fooling around with my co-applicants and student assistants. I also attended some interest-related events such as the UnPLUG Precinct Mapping Party and Ubuntu 10.04 Release Party.

The final leg of my summer, for me, was the most exciting. I accidentally got involved in a project while eating with my DilNet friends at a food stall near the Main Library. We had to develop an application to be used by a unit in UP.

Developing the app was exciting. Although I had previous experience with CodeIgniter (a PHP web-app framework that uses the MVC architectural pattern), it was my first time to use it for an actual project. It was my first time to use jQuery since the project required some realtime checking of constraints.

I also experienced roadblocks in making my part of the project. Three days before the deadline everything I coded for around a week got lost. After doing some research, I can attribute the loss of my folder to the way ext4 writes a file. I also found some solutions on how to recover my folder but I decided to do it again from scratch since recovering it will entail more work than actually rewriting the code. Here’s a more detailed explanation on what might have happened to me.

To catch up, I had to do some sort of coding marathon. I was awake for some 57 hours minus some short naps. During that time, I learned a lot from my co-developer.

It was a very fun experience and I am looking forward to working on more projects in the future.

And up to now, I am still looking forward for a beach outing. XD

The new official results page is now non-searchable, I wonder why. But that’s not a problem because I made a searchable version of the one’s in the UP website.

I also made a super lightweight web app that checks if your 2010 counterpart passed UPCAT or not. In my case, 201041437 either didn’t pass, still has deficiences or a pending case. His or her name was not on the list.

Aside from this, I also made another one last night which is not related to UPCAT. It’s a searchable database of UP student numbers. I got those student numbers from various sources from the Internet, including past UPCAT results. Hahaha.

I really missed scripting. All those stuff above are hosted on my laptop therefore they’ll be down whenever my laptop is sleeping. LOL.

Several hours ago was the 2009 ACM-ICPC Asia Regionals Manila Site hosted by the Ateneo de Manila University. ACM-ICPC is a programming contest in which college students are to solve a certain number of problems in a limited number of time, in yesterday’s contest ten problems in five hours.

UP Diliman had ten teams in total, seven SOGO teams from the University of the Philippines Programming Guild and three teams from the UPD Department of Computer Science. I was a member of one of the SOGO teams, the SOGO Virgins.

Results, results and results!!

Our team only ranked 26th out of the 55 teams, which was quite disappointing. Among the 10 problems given, we only had three accepted solutions out of the five problems we attempted.

The champion was from our guild (UPPG), the Mga SOGO ni E.T.. They solved all 10 problems in 1 hour and 59 minutes. They will be competing for the World Finals this February 1 to 6, 2009 at Harbin University in China. Other teams who solved all 10 problems sorted by time were teams from Ho Chi Minh City University of Science, National University of Singapore, Hong Kong University of Science and Technology and another team from UP Programming Guild, SOGOng Long Ong Moy Poso. The freshie team (2 Young 2 SOGO) from our guild beat us as they solved 5 problems in all. Woot! For detailed results, you may check out the contest director’s blog.

Problems encountered!

Experience is the best teacher. We encountered problems on some problems.

Problem A. Given three points on a parabola, get the maximum height of the parabola. Since we don’t have a mathematician in the team, we had a hard time finding an equation with degree one for all variables. We had to force one of our teammate (out of 3) to squeeze all his knowledge on parabolas. When he found one, I immediately coded Gaussian Elimination Method for two variables and two equations. Gladly, our solution was correct thus accepted by the judges.

On the summation problem, we got correct answers for the sample inputs but it was too slow that testing it on borderline cases (worst case scenarios) won’t yield a result in a reasonable amount of time. Since no one in our team was a mathematician, we had trouble limiting our search space. Our program was so inefficient but right that we were not able to have our solution accepted.

The other problem we attempted was the A(H1N1) problem. Given a set of N people in a community with their locations and radius of contact and a person infected by A(H1N1), we had to compute the number of levels of transmission until the virus is contained. We recognized that it was a graph problem; people as nodes then connect an edge to nodes within the radius of contact of an infected person. We just have to perform a BFS (Breadth-First Search) on the graph and find the  level distance from the root to find the maximum level of transmission.

The algorithm above was correct but I made a mistake in coding BFS. I colored the node after dequeuing it from the queue instead of coloring it while putting it on the queue as seen in my code below.

Damn. I admit that it was my fault, not anyone else’s. Hmmmmf. Waaaaaa. What a shame. On the other side, it still yielded the correct answer for the sample input.

Lessons Learned

• Never spend too much time on a single problem. We spent too much time debugging the A(H1N1) problem.
• A team needs a mathematician. There were quite a number of Math problems. We were only able to solve one out of all the Math problems.

Random Stuff

Even though we did not rank well, I was still happy of the outcome. Because of this contest, I will never forget BFS anymore! Hahahaha.

I am looking forward to joining the same contest for the next years if fate will permit.

After two sessions of training on our new “programming guild”, I’ve encountered tons of intellectually challenging programming problems. The spectrum of problems were very wide; from counting descendants of a person in a family tree, to beating a greedy opponent on a game, to getting the height of a rocket model using three ground observers using a theodolite, and the list goes on and on.

I admit those problems were new to me and I think it was the best answer for my hunger for knowledge. Those problems squeezed my neurons to the highest level, thus greasing up my brain. Those were the problems that made me realize I still have a lot lot lot more to learn. I mean, A LOT!

To all other geeks out there who have a background on any programming language (preferably C or Java), I dare you to solve the problems on real ACM International Collegiate Programming Competition (ACM-ICPC). There are a lot of problems available there from ACM-ICPC contests around the globe.

So far, I have only solved four ACM-ICPC problems. My goal is to solve at least one per night. Good luck to me and have fun with those problems.

The only criterion for membership in Mensa, is to score within the top 2 percent (98th percentile and up) on an accepted and properly supervised qualifying test. -Mensa Philippines website

Earlier today, the unofficial result of the 2.22 exam was released via web. Mensa posted the good and bad news. To maintain a level of privacy, they posted the codenames we picked from a fishbowl before the exam. My lucky word was MAGELLAN. Waaaaaaaa.

After checking the super cool good and bad news, I found out that I belong to the world’s top 3% IQ-wise. Hahahaha. Yes, I am a genius.

Yes I am a genius.

I am a genius.

Yeah, right!

Anyway, proceeding to the next item I’ll be sharing. I’ve recently shared to you that I have a hunger for knowledge and today, I was fortunate enough to learn something new. I love learning new things! It was a file transfer thru SSH crash course.

My orgmates asked me to generate reports of the recently concluded Computerized University Elections here in UP Diliman. All they gave me were the login credentials of the server where the database was stored and the I.P. address.

The account they gave me did not have any write access to the www folder so uploading the script thru FTP and dumping the results thru HTTP was not an option. I asked for more options and one of them suggested that I use SSH. What the heck?! I’ve never used SSH for uploading and downloading files, seriously. I never thought it was possible.

Since the only thing I knew was to login to the server thru SSH, I had to ask for help. Gladly, kuya Prem extended his help despite his busy schedule. I was also able to apply my newly acquired (last year) Linux shell skills like dumping something to a file. It was so much fun.

Yey, I am a genius. Yes I am a genius! I AM A GENIUS!! Mikko is a GENIUS!

(from XKCD)

Offtopic: In line with the comics above, I always find my self in the same situation. I love to do this with plate numbers.

These past few months were the most dormant periods of my geekiness. I badly need to expand my knowledge.

I have hunger for knowledge. Some people may find it weird but it’s actually true. It’s fun to have passion for learning stuff you’re interested to. Don’t think of me as a college nerd reading all those Bio, Chem and Physics books. I’m not interested on those stuff  except those rare moments when I felt geek enough to make simulation of scientific concepts (so far, I’ve made a simulation for freefall and projectile, road traffic and matrix multiplication).

The latest bit of geekiness I got was from my Computer Science class. It’s only this semester when I got a pretty good picture of real Object Oriented Programming. I knew about it before but I only had the chance to get a clearer picture of it on our class.

Currently, I’m interested on studying Python, Ruby on Rails and JSP. I’m tired of PHP, seriously. I’ve been obsessed with the language since I learned it three or four years ago. I’m also interested on making an application using any API. I’m thinking of making a web application that has something to do with geographical information (so that I can use Google Maps) like generating a map, like a Map of Misfortune [I'll discuss this later].

The BIG problem now is where to start. Do I start with Ruby on Rails or what? That I still do not know. I need help.

To the other geeks, feeling geeks (like me), aspiring geeks, retired geeks and non-geeks out there, I need your suggestions. It could be a programming language or an API to explore. Please respond thru the comments section. I am THIS pathetic to expand my knowledge. Waaaaaaaaaaaaa.