Webscraping with Selenium [Ruby, Firefox, OSX]
This article requires knowledge of Ruby, the concept of webscraping, and HTML.
Make sure you have Ruby installed (use RVM if you don’t have it yet) — I’m using 2.3.1 right now.
If you want a reference sheet, I’m in the process of writing one :P. Link will be updated when I get around to do so since there is so little Ruby help out there.
Why Use Selenium?
One nice thing due to the created browser interface is that you can actually see the elements on the webpage, which makes debugging a lot easier.
So in this tutorial we’ll be using Firefox (although I use PhantomJS afterwards since it’s a lot faster since it’s headless — you can google what that means :P).
In order to do so, we need to install a gem gem install selenium-webdriver .
You can test the following by either copying this into a ruby file or running irb (interactive ruby terminal) on your browser and copying and pasting the code below.
driver = Selenium::WebDriver.for :firefox, marionette: true
driver.execute_script("alert(\"hello from Kevin Fang!\");")
driver.find_element(:name, 'q').send_keys "hello world"
submit = driver.find_element(:name, "btnK")
#You can also find stuff with :id, :class, :tag_name, :link_text
#driver.find_elements(:class, "padding") #Returns array
If you get any errors, make sure your gems, Firefox, and Ruby versions are up to date.
Helpful reference websites: