Pictures:

^ The user interface from run time (click to enlarge)

^ My logo, put at the top of the file

Skills used:

Computer Science (Matlab, C++, Libcurl, Ruby, various rubygems)

 

Description:

It’s about time I finished this.

While I label this as a project, this was more of a learning experience than anything else. I came up with this idea near the end of the Fall 2013 semester. Virginia Tech, like many other schools, has a system referred to as “Drop-Add,” during which a student may check the timetable of classes for courses that they wanted to take, and add those CRNs (Course Reference Numbers) to their schedule. The problem is that this process is live, and as such, requires the student to camp the refresh button until the F5 key is worn out. I was adequately good at MATLAB, which I had learned from my ENGE 1114 class, and decided to make a program that would do this process for me. I figured that, in order to increase my knowledge of computer science (which I had not thought about before coming to tech) this would be a quite simple program to do.

I don’t think, to date, I have ever been so wrong. This turned out to be one of the most difficult things I have ever done on a computer.

Starting with the basics, I looked at the Virginia Tech timetable of classes, and noticed that the full classes featured a grey background (the color #F3F3F3 on the HTML), while the open classes featured a white background. I also knew that I could send GET requests and return the HTML with MATLAB using urlread() and a query string tacked onto the end of the URL. “Huzzah,” I thought to myself, and got to work writing the code.

However, the first problem came when I tried to hit the submit button. I figured that since I could enter in fields with a string like blahblahblah?CRN=32559&TERMYEAR=201409, I should be able to append the submit button on there somewhere too. After trying things like “Find Class Sections = true” or “Button = submit,” I started looking at the actual HTML and realized that they were all under a POST form.  I knew that MATLAB could send POST requests, but after trying fruitlessly, and with the spring semester coming up fast, I put the project on the back burner.

Jumping forward a short while (long enough to learn the basics of C++ in ECE 1574), I was feeling guilty for giving up so easy, and noticed that C++ was intensive enough to provide the options that MATLAB would not. So, I rebuilt the framework, and started doing research. First, I ran across winsock and wininet, which are used by windows for internet interface. After getting nowhere fast, I decided to look into external libraries, and ran across libcurl, a library used for POSTs, GETs, and retrieval of source code in C. After bumbling through copious amounts of stack overflow, youtube, and google, I managed to link libcurl to my compiler, and as a test, successfully got the HTML of www.google.com. Then came the next problem: libcurl doesn’t take urls that don’t have “www” in them, and since the timetable of classes was at a page on banweb.banner.vt.edu, libcurl failed me.

By now, I was 75% of the way through spring, and I was worrying about finals. However, I would still work on this when I had the chance. One day, during research, I ran across a code on Github by a person known as “mil,” who had done… well, exactly what I wanted… but in ruby. Although I pushed it off, I eventually gave in and decided to rewrite the code (again) in ruby.

This time was different. Since straight-up copying his code was obviously no way to learn anything, I started learning basic ruby code, and read up on mechanize and nokogiri, the two main gems used in mil’s code. After getting rubyinstaller down pat and writing some shaky loops, I began remaking the bruteforcer. Since I had a working code to lean on in case I needed guidance, I was able to learn how to do it, and I even changed the code up, changing the page that it gets and searching for the color of the background that was going to use back when I tried it in MATLAB, as well as adding a more precise error handling system. On May 22, 2014, after over 40 hours of hard work and failure, I managed to do a successful run from start to finish.

The main flowchart is as follows (click to enlarge):

 

The greens have been done, and the yellows are next on my list. To put it simply, the code takes inputs, logs in to verify, then checks if the class for the given CRN is full. If it is full, it will check again, and again, and again, until if finds an open seat. Once it finds one, it will add that course to the schedule. This process is automated, and is intended to run 24/7 during the drop-add period. On a low-traffic day, the system can check the course in 3 seconds, and add it in 6. This can probably be paired down a little bit, and I intend to work on this a little bit more later on.

(The DBF.rb file for this project has been made available here.)

Pros:

• The user can get whatever class they want without having to fight people on drop-add.
• Since I get whatever classes I want, I can plan my schedule more effectively.
• In the process of making this, I learned so much, including but not limited to: how to use external libraries in C++,  how to use basic web interface in MATLAB, how to use basic ruby and rubygems (woohoo! something else to throw on my resume), and how to interface with the internet with mechanize.
• This is one of the more useful inventions I’ve made.
• Learning how to automate POST form submissions has huge potential in more than just this instance.

Cons:

• It is not a force-adder. The job of this is to run during the drop-add period to catch drops, but if a class is never dropped, then the user is out of luck.
• I did not figure out how to use the interfacing on my own, and I was dependent on Mil’s code to figure out how to make it work.
• It is a .rb file, which means it requires Rubyinstaller to run, as opposed to being an executable file, which requires nothing additional to run.
• The whole process seems really, really sketchy. After spending hours looking for a VT regulation that disallows it, I’ve come up with nothing, so i guess it’s alright, but its not something to abuse.

 

Opportunities for improvement:

With this specific application, speed is always an issue. When attempting to get a seat in a class with some fifty people eyeing the same thing, even 10 seconds is a long time. For every little thing I can trim off the program it runs that little bit faster, and gives that little bit higher chance to get the seat. As such, cleaning up is pretty high on my list of TODOs.

For 1.3 I am going to try to add SMS/email functionality, so that the code will send a confirmation text message to tell the user that the class has been added/an error occurred. I am eyeballing the “sms_fu” rubygem. However, I have run into the problem that “sms_fu” is dependent on “pony,” which in turn is dependent on “mail,” which is dependent on “mime-types” version 1.16, while “mechanize,” the main web interface, depends on “mime-types” version 2.0. The bottom line is that I cannot use both “mechanize” and “mail” (or anything on “mail”). There are a number of ways around this, I just have to pick one and go with it.

My dream is to one day create an auto-planner, where you input times you are not available and the classes you need, and it automatically creates a series of possible schedules that you can pick from, then it will go off and add them. That is quite a challenge, but this is a significant start.

 

Conclusion:

I am rather disappointed that I couldn’t fully figure out on my own, but I am glad I did it. From learning more about three different programming languages, to rebuilding it and having it actually work was quite the experience. This was one of my most time consuming projects to date, and it finally paid off. Now to add more functionality…